Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc
OpenRouter's response `usage.cost` is preserved as canonical decimal dollars without
floating-point conversion; rounded micro-dollars remain a display/index field.
Endpoints that omit cost retain token-only accounting with unavailable provenance.
- `credentials.rs` and `login.rs`: local device-login storage and session lifecycle.
Renewable credentials bind refresh and logout to a canonical issuing server, and the
inference bearer is accepted only for its stored API base. Legacy credentials infer the
public issuer only when their stored API base is the canonical Postil endpoint. Atomic
owner-only writes stage newly issued and overwritten session families in a
pending-revocation queue before replacing the active credential. Background retries
contact only each family's persisted issuer. Logout retains the active revocation handle
until the issuing server accepts revocation.
- `review.rs`: orchestration; enforces acquisition, model-aware context, request,
provider-attempt, output-token, and worst-case token-exposure budgets before calls;
one UTF-8 byte counts as one projected token rather than using an optimistic ratio.
Expand Down Expand Up @@ -145,17 +153,16 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc
object mode, path, blob object ID, and gitlink path and object ID. Symlink blobs are searched as
link text and are never followed. A snapshot containing a gitlink remains incomplete because
content inside the referenced repository is outside the snapshot.
- `respond.rs`: interactive bot (`postil respond`): answers an @postil mention on a PR
or issue, grounded in the diff/issue, and posts one reply. Review-and-answer only; it
never opens PRs or pushes commits.
- `sarif.rs`: envelope → SARIF 2.1.0 for code-scanning ingestion (`--sarif`).
- `config.rs`: precedence (flags > env > .postil.* > .coderabbit.yaml > defaults),
- `config.rs`: model precedence (flags > env > .postil.* > stored login > defaults),
`deny_unknown_fields` so typos fail loudly. Also resolves `guardrails` and
`content_policy`, the two prompt-injected repo policy sources (see below).
Translated `.coderabbit.yaml` settings supply review policy but do not select a model.
Exception to precedence: `model.apiBase` from a config file is ignored by
default (a repo could redirect the base URL that receives the inference
credential); honored only with `POSTIL_ALLOW_CONFIG_API_BASE=1`. The
`POSTIL_API_BASE` environment variable is always applied. `model.apiFormat` and
`POSTIL_API_BASE` environment variable is applied to BYOK requests. A different
endpoint cannot receive a stored login bearer. `model.apiFormat` and
`POSTIL_API_FORMAT` select `openai-compatible` (default) or `anthropic`.
Hosted admission matches the complete ordered generator/scorer configuration and
consensus width to one immutable qualification profile. Each profile binds benchmark
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ similar = "3.1.1"
syn = { version = "2", features = ["full", "visit"] }
tempfile = "3.27.0"
time = { version = "0.3.54", features = ["parsing", "formatting"] }
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "process", "time"] }
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "process", "signal", "time"] }
toml = "1.1.2"
yaml_serde = "0.10.4"

Expand Down
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,17 @@ Authenticate once for hosted inference against your organization's entitlement,
```sh
postil login # zero-config: stores a renewable login for hosted inference
# or: export MODEL_API_KEY=... # OpenRouter is the default endpoint
# export REVIEW_MODEL=provider/qualified-model
postil doctor # validate the endpoint and repository
postil review --staged # review the staged change
postil review # review staged, branch, or tracked working-tree changes
postil review --staged # explicitly review the staged change
postil review --base origin/main # review the branch
postil review --bounded --base origin/main # cap large reviews at five source batches
postil hook install # add a pre-push review
```

`postil review` exits `0` when the gate passes, `1` when it fails, and `2` when it cannot produce a review envelope.
Bare local review selects staged changes first, then committed changes since the current branch's locally known default branch, then tracked working-tree changes, and finally an empty clean diff. Symbolic remote HEAD plus conventional `main`, `master`, and `trunk` refs are recognized without fetching. If no default branch can be resolved for a repository with commits, Postil fails closed and asks for `--base`, `--staged`, or `--diff-file` instead of reporting a partial review as clean. It sends the selected diff to the configured inference endpoint, but does not modify the working tree, index, or refs and does not write comments or checks to a forge unless `--publish` is supplied.

`postil models` answers model-selection questions offline without reading credentials or contacting a provider. No model setting is required: it identifies the embedded default, explains the model-ID contracts for OpenAI-compatible and native Anthropic endpoints, separates local protocol compatibility from hosted qualification, and gives exact `postil doctor` and override commands. The Luna reviewer uses `low` reasoning effort. Local scoring stays disabled until `REVIEW_SCORER_MODEL` names a scorer; the hosted scorer candidate uses `none`. `postil review` exits `0` when the gate passes, `1` when it fails, and `2` when it cannot produce a review envelope. Interactive terminals show concise progress; `--no-progress` or `POSTIL_NO_PROGRESS=1` replaces animation with concise static milestones, while `--verbose` enables complete provider, retry, and batch telemetry.

## Review a pull request

Expand All @@ -48,7 +50,7 @@ For GitHub Actions, use [`postil-action`](https://github.com/postil-dev/postil-a

## Configuration

`postil init` writes `.postil.yaml`. Flags override environment variables, which override repository configuration and defaults.
`postil init` writes `.postil.yaml`. Flags override environment variables, which override trusted repository configuration, stored login routing, and embedded defaults. `postil config` prints both the resolved model and its winning source.

```yaml
ignore:
Expand All @@ -62,10 +64,12 @@ reviewer:
gate:
failOn: error
onError: block
model:
reasoningEffort: low
scorerReasoningEffort: none
```

Unknown keys are rejected. Repository configuration cannot redirect a deployment credential to another API host unless the operator explicitly permits that behavior.
The CLI has no implicit model roster. Review and scorer models must be selected explicitly from models qualified by the benchmark.
Unknown keys are rejected. Repository configuration cannot redirect a deployment credential to another API host unless the operator explicitly permits that behavior. The embedded local preset uses `openai/gpt-5.6-luna`; local scoring is disabled until explicitly configured. Hosted profile selection remains service-controlled.

## Documentation

Expand All @@ -74,7 +78,7 @@ The CLI has no implicit model roster. Review and scorer models must be selected
| [Configuration](docs/configuration.md) | Policy, precedence, model selection, and environment variables |
| [Model providers](docs/model-providers.md) | OpenAI-compatible, Anthropic, and local endpoints |
| [Code forges](docs/forges.md) | GitHub, GitLab, Bitbucket, and Azure DevOps |
| [Automation](docs/automation.md) | SARIF, incremental review, envelopes, planning, and interactive replies |
| [Automation](docs/automation.md) | SARIF, incremental review, envelopes, planning, and usage receipts |
| [Architecture](ARCHITECTURE.md) | Trust boundaries and review pipeline |
| [Benchmarks](bench/README.md) | Model evaluation harness |

Expand Down
7 changes: 0 additions & 7 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
- Two-check semantics (postil/review advisory, postil/gate blocking); `gate.onError:
block|advisory` for fail-open on provider outages.
- Incremental re-review with resolved/carried reconciliation.
- Interactive bot (`postil respond`): replies to @postil mentions across every forge.
GitHub and GitLab cover PRs/MRs and issues; Bitbucket and Azure DevOps cover pull
requests. Review-and-answer only — never opens PRs or pushes.
- SARIF 2.1.0 output (`--sarif`) for code-scanning ingestion.
- Repo guardrails file (`.postil/guardrails.md`) injected into the prompt; violations
surface as `kind: guardrail` findings that quote the rule.
Expand Down Expand Up @@ -48,10 +45,6 @@
large PRs.
- Learning from dismissals: feed comment-resolution outcomes from the hosted platform
back into per-repo suppression hints.
- `postil respond` on the Bitbucket issue tracker and Azure DevOps work items (today
scoped to PRs there; those comment endpoints use a different base/version we have not
confirmed against a live host), and a visible error reply when the hosted bot exhausts
its retries (today a dead respond job is only logged).
- An `/evidence` benchmark: Postil's own silence rate and confirmed-finding rate on
public OSS PRs, with raw envelopes.

Expand Down
2 changes: 1 addition & 1 deletion bench/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"schemaVersion": 1,
"corpus": {
"fixtureCorpusSha256": "8e4c2cb9ad5a7efdfe6a875566d20133e905155b6f693a873595adf6c069e065",
"evaluatorSha256": "ebc72a8d7d60c08bf79759a0dfc5dbf821a9e6f79f4503c08ea9660dddece4d7"
"evaluatorSha256": "afadb5b9097433000a0c0ee9a81ccb23ecd01cfeb92829421f3e1acb68015c0d"
},
"profiles": {
"z-ai/glm-5.2": {
Expand Down
2 changes: 1 addition & 1 deletion bench/review-contract-sources.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@
"src/main.rs",
"src/output.rs",
"src/plan.rs",
"src/progress.rs",
"src/prompt.rs",
"src/repository_search.rs",
"src/resolve.rs",
"src/respond.rs",
"src/review.rs",
"src/sarif.rs"
]
4 changes: 2 additions & 2 deletions bench/src/harness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -443,7 +443,7 @@ export const envelopeV1 = z.object({
.array(
z.object({
model: z.string().min(1),
role: z.enum(["reviewPlanner", "reviewGenerator", "findingScorer", "mentionResponder"]).optional(),
role: z.enum(["reviewPlanner", "reviewGenerator", "findingScorer"]).optional(),
phase: z.enum(["initial", "schemaRepair", "semanticRetry"]).optional(),
callOrdinal: z.number().int().positive().optional(),
attempt: z.number().int().positive().optional(),
Expand All @@ -459,7 +459,7 @@ export const envelopeV1 = z.object({
modelIncidents: z
.array(
z.object({
phase: z.enum(["planner", "review", "scorer", "respond"]),
phase: z.enum(["planner", "review", "scorer"]),
category: z.enum(["providerError", "invalidOutput", "timeout", "deadline"]),
recovered: z.boolean(),
recovery: z.enum(["repair", "fallback"]).optional(),
Expand Down
2 changes: 1 addition & 1 deletion bench/src/livemodels-score.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ export interface AttributionEvidenceReference {

export interface UsageCostEvidence {
model: string;
role: "reviewPlanner" | "reviewGenerator" | "findingScorer" | "mentionResponder" | null;
role: "reviewPlanner" | "reviewGenerator" | "findingScorer" | null;
phase: "initial" | "schemaRepair" | "semanticRetry" | null;
callOrdinal: number | null;
attempt: number | null;
Expand Down
85 changes: 48 additions & 37 deletions bench/src/livemodels.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -330,7 +330,12 @@ describe("pair qualification configuration", () => {
"structured_outputs",
"temperature",
]],
["provider/generator-fallback", ["max_tokens", "temperature"]],
["provider/generator-fallback", [
"max_tokens",
"reasoning",
"reasoning_effort",
"temperature",
]],
["provider/scorer-fallback", [
"max_tokens",
"reasoning",
Expand Down Expand Up @@ -605,57 +610,66 @@ describe("pair qualification configuration", () => {

});

test("rejects runtime-shaped exact pair exposure above the hard cap", async () => {
test("isolates runtime-shaped preflight and routes any provider escape to loopback", async () => {
const root = await mkdtemp(resolve(tmpdir(), "postil-runtime-preflight-"));
const cases = fixtureInputs.map((input) => benchmarkCase.parse(input));
const inheritedModelKey = process.env.MODEL_API_KEY;
const runtimePair: QualificationPair = {
generatorModel: "z-ai/glm-5.2",
scorerModel: "openai/gpt-5-mini",
};
// Priced an order of magnitude above any admissible profile so the plan is
// refused on projected exposure. Realistic bounds project inside the
// admission ceiling and would exercise the happy path instead.
const binary = resolve(root, "fixture-postil");
const cases = fixtureInputs.slice(0, 2).map((input) => benchmarkCase.parse(input));
const runtimePair = normalizeQualificationPairs([pair])[0]!;
const pricing = new Map([
[runtimePair.generatorModel, {
providerIdentity: "PinnedProvider",
promptUsdPerToken: 0.0028434,
completionUsdPerToken: 0.0089364,
inputMicrosPerMillionTokens: 2_843_400,
outputMicrosPerMillionTokens: 8_936_400,
promptUsdPerToken: 0.000001,
completionUsdPerToken: 0.000002,
inputMicrosPerMillionTokens: 1_000_000,
outputMicrosPerMillionTokens: 2_000_000,
}],
[runtimePair.scorerModel, {
providerIdentity: "PinnedProvider",
promptUsdPerToken: 0.0028434,
completionUsdPerToken: 0.0089364,
inputMicrosPerMillionTokens: 2_843_400,
outputMicrosPerMillionTokens: 8_936_400,
promptUsdPerToken: 0.000001,
completionUsdPerToken: 0.000002,
inputMicrosPerMillionTokens: 1_000_000,
outputMicrosPerMillionTokens: 2_000_000,
}],
]);
await writeFile(binary, `#!/usr/bin/env node
const fail = (message) => { console.error(message); process.exit(97); };
if (process.env.POSTIL_QUALIFICATION_PLAN_ONLY !== "1") fail("plan-only mode was not set");
if (process.env.POSTIL_QUALIFICATION_CAPTURE_API_BASE !== "http://127.0.0.1:9") {
fail("plan-only preflight escaped loopback capture");
}
if (process.env.POSTIL_API_KEY !== "postil-plan-only-fixture") {
fail("the explicit fixture credential was not forwarded");
}
for (const name of ["MODEL_API_KEY", "LLM_API_KEY", "OPENROUTER_API_KEY", "POSTIL_ENDPOINT_AUTH_VALUE"]) {
if (process.env[name] !== undefined) fail("ambient credential reached plan-only preflight: " + name);
}
console.log(JSON.stringify({
version: 1, summary: "", silent: true, findings: [], resolved: [],
counts: { info: 0, warn: 0, error: 0, suppressed: 0, ungrounded: 0 },
confidenceBuckets: [0, 0, 0, 0, 0],
gate: { failOn: "error", failing: false, blockOnKinds: [] },
modelUsed: "none (qualification plan)", usage: { promptTokens: 0, completionTokens: 0 },
reviewCoverage: { mode: "exhaustive", selectedBatches: 1, totalBatches: 1 },
reviewAdmission: { providerAttempts: 6, serializedInputBytes: 5000, outputTokens: 180, projectedCostMicros: 1000000 },
durationMs: 0, baseSha: null, headSha: null, sinceSha: null
}));
`);
await chmod(binary, 0o700);
try {
process.env.MODEL_API_KEY = "postil-plan-only-fixture";
const cargoTarget = process.env.CARGO_TARGET_DIR;
const binary = process.env.POSTIL_BIN === undefined
? cargoTarget === undefined
? resolve(import.meta.dir, "..", "..", "target", "release", "postil")
: resolve(cargoTarget, "release", "postil")
: resolve(process.env.POSTIL_BIN);
const normalizedPair = normalizeQualificationPairs([runtimePair])[0]!;
await expect(assertRuntimeShapedQualificationPreflight({
binary,
rootDir: root,
cases,
pairs: [normalizedPair],
repeats: 9,
pairs: [runtimePair],
repeats: 3,
pricing,
apiBase: normalizeApiBase("https://openrouter.ai/api/v1"),
apiFormat: "openai-compatible",
costCapUsdDecimal: "70",
costCapUsdDecimal: "1",
upstreamProvider: "PinnedProvider",
})).rejects.toThrow("micro-dollar admission projection cap");
credentialEnvironment: { POSTIL_API_KEY: "postil-plan-only-fixture" },
})).rejects.toThrow("runtime-shaped qualification spend");
} finally {
if (inheritedModelKey === undefined) delete process.env.MODEL_API_KEY;
else process.env.MODEL_API_KEY = inheritedModelKey;
await rm(root, { recursive: true, force: true });
}
}, 60_000);
Expand All @@ -664,7 +678,6 @@ describe("pair qualification configuration", () => {
const root = await mkdtemp(resolve(tmpdir(), "postil-runtime-preflight-failure-"));
const binary = resolve(root, "fixture-postil");
const startsPath = resolve(root, "starts.log");
const inheritedModelKey = process.env.MODEL_API_KEY;
await writeFile(binary, `#!/usr/bin/env node
import { appendFileSync, closeSync, openSync } from "node:fs";
import { resolve } from "node:path";
Expand Down Expand Up @@ -692,7 +705,6 @@ process.exit(0);
`);
await chmod(binary, 0o700);
try {
process.env.MODEL_API_KEY = "postil-plan-only-fixture";
const cases = fixtureInputs.slice(0, 8).map((input) => benchmarkCase.parse(input));
const pricing = new Map([
[pair.generatorModel, {
Expand Down Expand Up @@ -721,6 +733,7 @@ process.exit(0);
apiFormat: "openai-compatible",
costCapUsdDecimal: "55",
upstreamProvider: "PinnedProvider",
credentialEnvironment: { POSTIL_API_KEY: "postil-plan-only-fixture" },
})).rejects.toThrow("deliberate preflight failure");
const starts = (await readFile(startsPath, "utf8")).trim().split("\n");
const finishes = (await readFile(resolve(root, "finishes.log"), "utf8"))
Expand All @@ -733,8 +746,6 @@ process.exit(0);
expect(finishes.every((identity) => starts.includes(identity))).toBe(true);
await expect(lstat(resolve(root, "preflight"))).rejects.toThrow();
} finally {
if (inheritedModelKey === undefined) delete process.env.MODEL_API_KEY;
else process.env.MODEL_API_KEY = inheritedModelKey;
await rm(root, { recursive: true, force: true });
}
}, 20_000);
Expand Down
Loading