diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 48dfd3a..07ece20 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -56,6 +56,14 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc OpenRouter's response `usage.cost` is preserved as canonical decimal dollars without floating-point conversion; rounded micro-dollars remain a display/index field. Endpoints that omit cost retain token-only accounting with unavailable provenance. +- `credentials.rs` and `login.rs`: local device-login storage and session lifecycle. + Renewable credentials bind refresh and logout to a canonical issuing server, and the + inference bearer is accepted only for its stored API base. Legacy credentials infer the + public issuer only when their stored API base is the canonical Postil endpoint. Atomic + owner-only writes stage newly issued and overwritten session families in a + pending-revocation queue before replacing the active credential. Background retries + contact only each family's persisted issuer. Logout retains the active revocation handle + until the issuing server accepts revocation. - `review.rs`: orchestration; enforces acquisition, model-aware context, request, provider-attempt, output-token, and worst-case token-exposure budgets before calls; one UTF-8 byte counts as one projected token rather than using an optimistic ratio. @@ -145,17 +153,16 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc object mode, path, blob object ID, and gitlink path and object ID. Symlink blobs are searched as link text and are never followed. A snapshot containing a gitlink remains incomplete because content inside the referenced repository is outside the snapshot. -- `respond.rs`: interactive bot (`postil respond`): answers an @postil mention on a PR - or issue, grounded in the diff/issue, and posts one reply. Review-and-answer only; it - never opens PRs or pushes commits. - `sarif.rs`: envelope → SARIF 2.1.0 for code-scanning ingestion (`--sarif`). -- `config.rs`: precedence (flags > env > .postil.* > .coderabbit.yaml > defaults), +- `config.rs`: model precedence (flags > env > .postil.* > stored login > defaults), `deny_unknown_fields` so typos fail loudly. Also resolves `guardrails` and `content_policy`, the two prompt-injected repo policy sources (see below). + Translated `.coderabbit.yaml` settings supply review policy but do not select a model. Exception to precedence: `model.apiBase` from a config file is ignored by default (a repo could redirect the base URL that receives the inference credential); honored only with `POSTIL_ALLOW_CONFIG_API_BASE=1`. The - `POSTIL_API_BASE` environment variable is always applied. `model.apiFormat` and + `POSTIL_API_BASE` environment variable is applied to BYOK requests. A different + endpoint cannot receive a stored login bearer. `model.apiFormat` and `POSTIL_API_FORMAT` select `openai-compatible` (default) or `anthropic`. Hosted admission matches the complete ordered generator/scorer configuration and consensus width to one immutable qualification profile. Each profile binds benchmark diff --git a/Cargo.toml b/Cargo.toml index b4041d0..a5c7ee9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -41,7 +41,7 @@ similar = "3.1.1" syn = { version = "2", features = ["full", "visit"] } tempfile = "3.27.0" time = { version = "0.3.54", features = ["parsing", "formatting"] } -tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "process", "time"] } +tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "process", "signal", "time"] } toml = "1.1.2" yaml_serde = "0.10.4" diff --git a/README.md b/README.md index 16c82e2..ceac62b 100644 --- a/README.md +++ b/README.md @@ -25,15 +25,17 @@ Authenticate once for hosted inference against your organization's entitlement, ```sh postil login # zero-config: stores a renewable login for hosted inference # or: export MODEL_API_KEY=... # OpenRouter is the default endpoint -# export REVIEW_MODEL=provider/qualified-model postil doctor # validate the endpoint and repository -postil review --staged # review the staged change +postil review # review staged, branch, or tracked working-tree changes +postil review --staged # explicitly review the staged change postil review --base origin/main # review the branch postil review --bounded --base origin/main # cap large reviews at five source batches postil hook install # add a pre-push review ``` -`postil review` exits `0` when the gate passes, `1` when it fails, and `2` when it cannot produce a review envelope. +Bare local review selects staged changes first, then committed changes since the current branch's locally known default branch, then tracked working-tree changes, and finally an empty clean diff. Symbolic remote HEAD plus conventional `main`, `master`, and `trunk` refs are recognized without fetching. If no default branch can be resolved for a repository with commits, Postil fails closed and asks for `--base`, `--staged`, or `--diff-file` instead of reporting a partial review as clean. It sends the selected diff to the configured inference endpoint, but does not modify the working tree, index, or refs and does not write comments or checks to a forge unless `--publish` is supplied. + +`postil models` answers model-selection questions offline without reading credentials or contacting a provider. No model setting is required: it identifies the embedded default, explains the model-ID contracts for OpenAI-compatible and native Anthropic endpoints, separates local protocol compatibility from hosted qualification, and gives exact `postil doctor` and override commands. The Luna reviewer uses `low` reasoning effort. Local scoring stays disabled until `REVIEW_SCORER_MODEL` names a scorer; the hosted scorer candidate uses `none`. `postil review` exits `0` when the gate passes, `1` when it fails, and `2` when it cannot produce a review envelope. Interactive terminals show concise progress; `--no-progress` or `POSTIL_NO_PROGRESS=1` replaces animation with concise static milestones, while `--verbose` enables complete provider, retry, and batch telemetry. ## Review a pull request @@ -48,7 +50,7 @@ For GitHub Actions, use [`postil-action`](https://github.com/postil-dev/postil-a ## Configuration -`postil init` writes `.postil.yaml`. Flags override environment variables, which override repository configuration and defaults. +`postil init` writes `.postil.yaml`. Flags override environment variables, which override trusted repository configuration, stored login routing, and embedded defaults. `postil config` prints both the resolved model and its winning source. ```yaml ignore: @@ -62,10 +64,12 @@ reviewer: gate: failOn: error onError: block +model: + reasoningEffort: low + scorerReasoningEffort: none ``` -Unknown keys are rejected. Repository configuration cannot redirect a deployment credential to another API host unless the operator explicitly permits that behavior. -The CLI has no implicit model roster. Review and scorer models must be selected explicitly from models qualified by the benchmark. +Unknown keys are rejected. Repository configuration cannot redirect a deployment credential to another API host unless the operator explicitly permits that behavior. The embedded local preset uses `openai/gpt-5.6-luna`; local scoring is disabled until explicitly configured. Hosted profile selection remains service-controlled. ## Documentation @@ -74,7 +78,7 @@ The CLI has no implicit model roster. Review and scorer models must be selected | [Configuration](docs/configuration.md) | Policy, precedence, model selection, and environment variables | | [Model providers](docs/model-providers.md) | OpenAI-compatible, Anthropic, and local endpoints | | [Code forges](docs/forges.md) | GitHub, GitLab, Bitbucket, and Azure DevOps | -| [Automation](docs/automation.md) | SARIF, incremental review, envelopes, planning, and interactive replies | +| [Automation](docs/automation.md) | SARIF, incremental review, envelopes, planning, and usage receipts | | [Architecture](ARCHITECTURE.md) | Trust boundaries and review pipeline | | [Benchmarks](bench/README.md) | Model evaluation harness | diff --git a/ROADMAP.md b/ROADMAP.md index fd75921..ade0d13 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -7,9 +7,6 @@ - Two-check semantics (postil/review advisory, postil/gate blocking); `gate.onError: block|advisory` for fail-open on provider outages. - Incremental re-review with resolved/carried reconciliation. -- Interactive bot (`postil respond`): replies to @postil mentions across every forge. - GitHub and GitLab cover PRs/MRs and issues; Bitbucket and Azure DevOps cover pull - requests. Review-and-answer only — never opens PRs or pushes. - SARIF 2.1.0 output (`--sarif`) for code-scanning ingestion. - Repo guardrails file (`.postil/guardrails.md`) injected into the prompt; violations surface as `kind: guardrail` findings that quote the rule. @@ -48,10 +45,6 @@ large PRs. - Learning from dismissals: feed comment-resolution outcomes from the hosted platform back into per-repo suppression hints. -- `postil respond` on the Bitbucket issue tracker and Azure DevOps work items (today - scoped to PRs there; those comment endpoints use a different base/version we have not - confirmed against a live host), and a visible error reply when the hosted bot exhausts - its retries (today a dead respond job is only logged). - An `/evidence` benchmark: Postil's own silence rate and confirmed-finding rate on public OSS PRs, with raw envelopes. diff --git a/bench/baseline.json b/bench/baseline.json index 63f5b0c..ae61525 100644 --- a/bench/baseline.json +++ b/bench/baseline.json @@ -2,7 +2,7 @@ "schemaVersion": 1, "corpus": { "fixtureCorpusSha256": "8e4c2cb9ad5a7efdfe6a875566d20133e905155b6f693a873595adf6c069e065", - "evaluatorSha256": "ebc72a8d7d60c08bf79759a0dfc5dbf821a9e6f79f4503c08ea9660dddece4d7" + "evaluatorSha256": "afadb5b9097433000a0c0ee9a81ccb23ecd01cfeb92829421f3e1acb68015c0d" }, "profiles": { "z-ai/glm-5.2": { diff --git a/bench/review-contract-sources.json b/bench/review-contract-sources.json index 1403436..8ce5eb1 100644 --- a/bench/review-contract-sources.json +++ b/bench/review-contract-sources.json @@ -29,10 +29,10 @@ "src/main.rs", "src/output.rs", "src/plan.rs", + "src/progress.rs", "src/prompt.rs", "src/repository_search.rs", "src/resolve.rs", - "src/respond.rs", "src/review.rs", "src/sarif.rs" ] diff --git a/bench/src/harness.ts b/bench/src/harness.ts index 155491d..ddaa211 100644 --- a/bench/src/harness.ts +++ b/bench/src/harness.ts @@ -443,7 +443,7 @@ export const envelopeV1 = z.object({ .array( z.object({ model: z.string().min(1), - role: z.enum(["reviewPlanner", "reviewGenerator", "findingScorer", "mentionResponder"]).optional(), + role: z.enum(["reviewPlanner", "reviewGenerator", "findingScorer"]).optional(), phase: z.enum(["initial", "schemaRepair", "semanticRetry"]).optional(), callOrdinal: z.number().int().positive().optional(), attempt: z.number().int().positive().optional(), @@ -459,7 +459,7 @@ export const envelopeV1 = z.object({ modelIncidents: z .array( z.object({ - phase: z.enum(["planner", "review", "scorer", "respond"]), + phase: z.enum(["planner", "review", "scorer"]), category: z.enum(["providerError", "invalidOutput", "timeout", "deadline"]), recovered: z.boolean(), recovery: z.enum(["repair", "fallback"]).optional(), diff --git a/bench/src/livemodels-score.ts b/bench/src/livemodels-score.ts index 23ccd3d..6fd41f5 100644 --- a/bench/src/livemodels-score.ts +++ b/bench/src/livemodels-score.ts @@ -94,7 +94,7 @@ export interface AttributionEvidenceReference { export interface UsageCostEvidence { model: string; - role: "reviewPlanner" | "reviewGenerator" | "findingScorer" | "mentionResponder" | null; + role: "reviewPlanner" | "reviewGenerator" | "findingScorer" | null; phase: "initial" | "schemaRepair" | "semanticRetry" | null; callOrdinal: number | null; attempt: number | null; diff --git a/bench/src/livemodels.test.ts b/bench/src/livemodels.test.ts index f39949f..c7db75f 100644 --- a/bench/src/livemodels.test.ts +++ b/bench/src/livemodels.test.ts @@ -330,7 +330,12 @@ describe("pair qualification configuration", () => { "structured_outputs", "temperature", ]], - ["provider/generator-fallback", ["max_tokens", "temperature"]], + ["provider/generator-fallback", [ + "max_tokens", + "reasoning", + "reasoning_effort", + "temperature", + ]], ["provider/scorer-fallback", [ "max_tokens", "reasoning", @@ -605,57 +610,66 @@ describe("pair qualification configuration", () => { }); - test("rejects runtime-shaped exact pair exposure above the hard cap", async () => { + test("isolates runtime-shaped preflight and routes any provider escape to loopback", async () => { const root = await mkdtemp(resolve(tmpdir(), "postil-runtime-preflight-")); - const cases = fixtureInputs.map((input) => benchmarkCase.parse(input)); - const inheritedModelKey = process.env.MODEL_API_KEY; - const runtimePair: QualificationPair = { - generatorModel: "z-ai/glm-5.2", - scorerModel: "openai/gpt-5-mini", - }; - // Priced an order of magnitude above any admissible profile so the plan is - // refused on projected exposure. Realistic bounds project inside the - // admission ceiling and would exercise the happy path instead. + const binary = resolve(root, "fixture-postil"); + const cases = fixtureInputs.slice(0, 2).map((input) => benchmarkCase.parse(input)); + const runtimePair = normalizeQualificationPairs([pair])[0]!; const pricing = new Map([ [runtimePair.generatorModel, { providerIdentity: "PinnedProvider", - promptUsdPerToken: 0.0028434, - completionUsdPerToken: 0.0089364, - inputMicrosPerMillionTokens: 2_843_400, - outputMicrosPerMillionTokens: 8_936_400, + promptUsdPerToken: 0.000001, + completionUsdPerToken: 0.000002, + inputMicrosPerMillionTokens: 1_000_000, + outputMicrosPerMillionTokens: 2_000_000, }], [runtimePair.scorerModel, { providerIdentity: "PinnedProvider", - promptUsdPerToken: 0.0028434, - completionUsdPerToken: 0.0089364, - inputMicrosPerMillionTokens: 2_843_400, - outputMicrosPerMillionTokens: 8_936_400, + promptUsdPerToken: 0.000001, + completionUsdPerToken: 0.000002, + inputMicrosPerMillionTokens: 1_000_000, + outputMicrosPerMillionTokens: 2_000_000, }], ]); + await writeFile(binary, `#!/usr/bin/env node +const fail = (message) => { console.error(message); process.exit(97); }; +if (process.env.POSTIL_QUALIFICATION_PLAN_ONLY !== "1") fail("plan-only mode was not set"); +if (process.env.POSTIL_QUALIFICATION_CAPTURE_API_BASE !== "http://127.0.0.1:9") { + fail("plan-only preflight escaped loopback capture"); +} +if (process.env.POSTIL_API_KEY !== "postil-plan-only-fixture") { + fail("the explicit fixture credential was not forwarded"); +} +for (const name of ["MODEL_API_KEY", "LLM_API_KEY", "OPENROUTER_API_KEY", "POSTIL_ENDPOINT_AUTH_VALUE"]) { + if (process.env[name] !== undefined) fail("ambient credential reached plan-only preflight: " + name); +} +console.log(JSON.stringify({ + version: 1, summary: "", silent: true, findings: [], resolved: [], + counts: { info: 0, warn: 0, error: 0, suppressed: 0, ungrounded: 0 }, + confidenceBuckets: [0, 0, 0, 0, 0], + gate: { failOn: "error", failing: false, blockOnKinds: [] }, + modelUsed: "none (qualification plan)", usage: { promptTokens: 0, completionTokens: 0 }, + reviewCoverage: { mode: "exhaustive", selectedBatches: 1, totalBatches: 1 }, + reviewAdmission: { providerAttempts: 6, serializedInputBytes: 5000, outputTokens: 180, projectedCostMicros: 1000000 }, + durationMs: 0, baseSha: null, headSha: null, sinceSha: null +})); +`); + await chmod(binary, 0o700); try { - process.env.MODEL_API_KEY = "postil-plan-only-fixture"; - const cargoTarget = process.env.CARGO_TARGET_DIR; - const binary = process.env.POSTIL_BIN === undefined - ? cargoTarget === undefined - ? resolve(import.meta.dir, "..", "..", "target", "release", "postil") - : resolve(cargoTarget, "release", "postil") - : resolve(process.env.POSTIL_BIN); - const normalizedPair = normalizeQualificationPairs([runtimePair])[0]!; await expect(assertRuntimeShapedQualificationPreflight({ binary, rootDir: root, cases, - pairs: [normalizedPair], - repeats: 9, + pairs: [runtimePair], + repeats: 3, pricing, apiBase: normalizeApiBase("https://openrouter.ai/api/v1"), apiFormat: "openai-compatible", - costCapUsdDecimal: "70", + costCapUsdDecimal: "1", upstreamProvider: "PinnedProvider", - })).rejects.toThrow("micro-dollar admission projection cap"); + credentialEnvironment: { POSTIL_API_KEY: "postil-plan-only-fixture" }, + })).rejects.toThrow("runtime-shaped qualification spend"); } finally { - if (inheritedModelKey === undefined) delete process.env.MODEL_API_KEY; - else process.env.MODEL_API_KEY = inheritedModelKey; await rm(root, { recursive: true, force: true }); } }, 60_000); @@ -664,7 +678,6 @@ describe("pair qualification configuration", () => { const root = await mkdtemp(resolve(tmpdir(), "postil-runtime-preflight-failure-")); const binary = resolve(root, "fixture-postil"); const startsPath = resolve(root, "starts.log"); - const inheritedModelKey = process.env.MODEL_API_KEY; await writeFile(binary, `#!/usr/bin/env node import { appendFileSync, closeSync, openSync } from "node:fs"; import { resolve } from "node:path"; @@ -692,7 +705,6 @@ process.exit(0); `); await chmod(binary, 0o700); try { - process.env.MODEL_API_KEY = "postil-plan-only-fixture"; const cases = fixtureInputs.slice(0, 8).map((input) => benchmarkCase.parse(input)); const pricing = new Map([ [pair.generatorModel, { @@ -721,6 +733,7 @@ process.exit(0); apiFormat: "openai-compatible", costCapUsdDecimal: "55", upstreamProvider: "PinnedProvider", + credentialEnvironment: { POSTIL_API_KEY: "postil-plan-only-fixture" }, })).rejects.toThrow("deliberate preflight failure"); const starts = (await readFile(startsPath, "utf8")).trim().split("\n"); const finishes = (await readFile(resolve(root, "finishes.log"), "utf8")) @@ -733,8 +746,6 @@ process.exit(0); expect(finishes.every((identity) => starts.includes(identity))).toBe(true); await expect(lstat(resolve(root, "preflight"))).rejects.toThrow(); } finally { - if (inheritedModelKey === undefined) delete process.env.MODEL_API_KEY; - else process.env.MODEL_API_KEY = inheritedModelKey; await rm(root, { recursive: true, force: true }); } }, 20_000); diff --git a/bench/src/livemodels.ts b/bench/src/livemodels.ts index 64de418..bf34591 100644 --- a/bench/src/livemodels.ts +++ b/bench/src/livemodels.ts @@ -97,6 +97,7 @@ export const QUALIFICATION_MAX_AGE_DAYS = 30; export const QUALIFICATION_MAX_AGE_SECONDS = QUALIFICATION_MAX_AGE_DAYS * 24 * 60 * 60; const MAX_QUALIFICATION_SOURCE_BYTES = 16 * 1024 * 1024; const MANAGED_OPENROUTER_API_BASE = "https://openrouter.ai:443/api/v1"; +const PLAN_ONLY_CAPTURE_API_BASE = "http://127.0.0.1:9"; export const MANAGED_OPENROUTER_PROVIDER_IDENTITY = "openrouter:managed-routing"; export const LIVE_MODELS_REPORT_SCHEMA_VERSION = 3; export const LIVE_MODELS_PRIVATE_EVIDENCE_SCHEMA_VERSION = 1; @@ -1507,6 +1508,7 @@ export async function assertRuntimeShapedQualificationPreflight(args: { apiFormat: "openai-compatible" | "anthropic"; costCapUsdDecimal: string; upstreamProvider: string; + credentialEnvironment?: NodeJS.ProcessEnv; }): Promise { let projectedMicros = 0n; const planRoot = join(args.rootDir, "preflight"); @@ -1551,6 +1553,8 @@ export async function assertRuntimeShapedQualificationPreflight(args: { args.apiBase, args.apiFormat, profilePath, + PLAN_ONLY_CAPTURE_API_BASE, + args.credentialEnvironment, ); env.POSTIL_QUALIFICATION_PLAN_ONLY = "1"; const { stdout } = await execFile( @@ -1774,6 +1778,7 @@ export function liveEnv( apiFormat: "openai-compatible" | "anthropic" = "openai-compatible", candidateProfilePath?: string, qualificationCaptureApiBase?: string, + sourceEnvironment: NodeJS.ProcessEnv = process.env, ): NodeJS.ProcessEnv { const env: NodeJS.ProcessEnv = { PATH: process.env.PATH, @@ -1809,12 +1814,12 @@ export function liveEnv( } env.POSTIL_QUALIFICATION_CAPTURE_API_BASE = qualificationCaptureApiBase; } - const endpointAuth = endpointAuthFromEnvironment(apiFormat); + const endpointAuth = endpointAuthFromEnvironment(apiFormat, sourceEnvironment); if (endpointAuth) { env.POSTIL_ENDPOINT_AUTH_HEADER = endpointAuth.header; env.POSTIL_ENDPOINT_AUTH_VALUE = endpointAuth.value; } - const allowPrivate = process.env.POSTIL_ALLOW_PRIVATE_API_BASE; + const allowPrivate = sourceEnvironment.POSTIL_ALLOW_PRIVATE_API_BASE; if (allowPrivate !== undefined) { if (allowPrivate !== "1" && allowPrivate.toLowerCase() !== "true") { throw new Error("POSTIL_ALLOW_PRIVATE_API_BASE must be 1 or true when set"); @@ -1824,7 +1829,7 @@ export function liveEnv( // Forward the selected inference-key variable without logging or placing the // value on argv. Neutral aliases are also mirrored into POSTIL_API_KEY so // older binaries can run from the same benchmark harness. - forwardApiKey(env); + forwardApiKey(env, sourceEnvironment); return env; } @@ -2405,7 +2410,7 @@ export function qualificationRequiredParameters( }; for (const pair of pairs) { for (const model of qualificationGeneratorModels(pair)) { - add(model, ["max_tokens", "temperature"]); + add(model, ["max_tokens", "reasoning", "reasoning_effort", "temperature"]); } for (const model of qualificationScorerModels(pair)) { add(model, [ @@ -2423,9 +2428,10 @@ export function qualificationRequiredParameters( export function endpointAuthFromEnvironment( apiFormat: "openai-compatible" | "anthropic", + sourceEnvironment: NodeJS.ProcessEnv = process.env, ): { header: string; value: string } | null { - const rawHeader = process.env.POSTIL_ENDPOINT_AUTH_HEADER; - const rawValue = process.env.POSTIL_ENDPOINT_AUTH_VALUE; + const rawHeader = sourceEnvironment.POSTIL_ENDPOINT_AUTH_HEADER; + const rawValue = sourceEnvironment.POSTIL_ENDPOINT_AUTH_VALUE; const header = rawHeader?.trim() || undefined; const value = rawValue === "" ? undefined : rawValue; if (header === undefined && value === undefined) return null; diff --git a/bench/src/verify-admission.test.ts b/bench/src/verify-admission.test.ts index 61cfdef..5b41b23 100644 --- a/bench/src/verify-admission.test.ts +++ b/bench/src/verify-admission.test.ts @@ -61,6 +61,27 @@ describe("admission attestation verification", () => { )).toBe("provisional"); }); + test("binds reviewer and scorer reasoning efforts to the model-defaults digest", async () => { + const repositoryRoot = join(import.meta.dir, "..", ".."); + const original = await readFile(join(repositoryRoot, "config.toml"), "utf8"); + for (const altered of [ + original.replace('reasoning_effort = "low"', 'reasoning_effort = "high"'), + original.replace('reasoning_effort = "none"', 'reasoning_effort = "low"'), + ]) { + const directory = await temporaryDirectory(); + const manifest = join(directory, "qualified-models.json"); + const config = join(directory, "config.toml"); + await writeFile(manifest, await readFile(join(repositoryRoot, "qualified-models.json"))); + await writeFile(config, altered); + + await expect(verifyProvisionalRelease( + manifest, + config, + join(repositoryRoot, "provisional-models.json"), + )).rejects.toThrow("empty qualification manifest does not match embedded model defaults"); + } + }); + test("rejects duplicate models within either provisional chain", async () => { const repositoryRoot = join(import.meta.dir, "..", ".."); const directory = await temporaryDirectory(); diff --git a/bench/src/verify-admission.ts b/bench/src/verify-admission.ts index 3f17034..bb9fbc2 100644 --- a/bench/src/verify-admission.ts +++ b/bench/src/verify-admission.ts @@ -36,10 +36,20 @@ const boundedIdentifierSchema = z.string().trim().min(1).refine( "identifier must not contain line breaks", ); const optionalIdentifierSchema = z.union([z.literal(""), boundedIdentifierSchema]); +const reasoningEffortSchema = z.enum([ + "max", + "xhigh", + "high", + "medium", + "low", + "minimal", + "none", +]); const modelDefaultsSchema = z.object({ version: z.number().int().positive(), default_model: optionalIdentifierSchema, + reasoning_effort: reasoningEffortSchema, cascade: z.array(boundedIdentifierSchema), consensus: z.number().int().positive(), api_base: z.literal("https://openrouter.ai/api/v1"), @@ -47,6 +57,7 @@ const modelDefaultsSchema = z.object({ scorer: z.object({ enabled: z.boolean(), default_model: optionalIdentifierSchema, + reasoning_effort: reasoningEffortSchema, fallback: optionalIdentifierSchema, qualification_candidates: z.array(boundedIdentifierSchema), }).strict(), diff --git a/config.toml b/config.toml index 6b8566b..1d70b02 100644 --- a/config.toml +++ b/config.toml @@ -1,5 +1,6 @@ version = 3 default_model = "openai/gpt-5.6-luna" +reasoning_effort = "low" cascade = [] consensus = 1 api_base = "https://openrouter.ai/api/v1" @@ -8,5 +9,6 @@ api_format = "openai-compatible" [scorer] enabled = true default_model = "openai/gpt-5.6-luna" +reasoning_effort = "none" fallback = "" qualification_candidates = ["openai/gpt-5.6-luna"] diff --git a/docs/automation.md b/docs/automation.md index 1c75407..230cd8e 100644 --- a/docs/automation.md +++ b/docs/automation.md @@ -8,6 +8,8 @@ The JSON and YAML envelope contains the summary, findings, resolved findings, po `--sarif ` writes SARIF 2.1.0 for code-scanning ingestion. +Machine data written to stdout never includes progress controls. Interactive terminals collapse routine telemetry into concise progress only when stdout is human-readable. `--output-file` and `--sarif` keep that human progress because their machine artifacts go to files. `--no-progress` or `POSTIL_NO_PROGRESS=1` replaces animation with concise static milestones. `--verbose` enables complete provider, retry, and batch telemetry. Non-TTY and machine-output contexts retain complete telemetry automatically. + ## Incremental review ```sh @@ -32,19 +34,6 @@ postil plan --envelopes .cache/envelopes --config .postil.candidate.yaml The report shows which findings become visible or suppressed and which gate results change. -## Interactive replies - -`postil respond` answers a mention on a pull request. GitHub and GitLab also support issue mentions. It does not open pull requests or push commits. -Posting the reply requires `--publish`; without it, the reply is printed locally. -CI detection and environment variables do not authorize a forge write. - -```sh -POSTIL_COMMENT='@postil is this safe?' \ -postil respond --repo owner/repository --pr 123 --publish -``` - -Pass comment text through `POSTIL_COMMENT` in automation because command-line arguments are visible to other local processes. - ## Usage receipts Hosted workers can set `POSTIL_USAGE_RECEIPT_PATH` to a worker-owned path. A successful response writes a mode-`0600` version 2 JSON receipt before forge delivery. Every provider attempt includes its role, phase, operation-wide call ordinal, transport attempt, token counts, accounting-completeness flag, cost source, canonical provider-reported decimal cost when present, and rounded micro-dollar display value. The caller owns receipt deletion. Receipt consumers must accept the additional `costProviderDecimal` field before deploying this CLI. diff --git a/docs/configuration.md b/docs/configuration.md index f2f54e7..3163fad 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,13 +1,15 @@ # Configuration -Postil resolves settings in this order: +Postil resolves model settings in this order: 1. Command-line flags 2. Environment variables 3. `.postil.yaml`, `.postil.yml`, or `.postil.json` -4. Translated `.coderabbit.yaml` settings +4. Stored `postil login` routing 5. Built-in defaults +Translated `.coderabbit.yaml` settings can supply review policy but do not select a model. + Unknown keys fail validation so a misspelling cannot silently weaken a review. ## Repository policy @@ -33,8 +35,10 @@ contentPolicy: enabled: true model: name: provider/qualified-model + reasoningEffort: low # max, xhigh, high, medium, low, minimal, or none cascade: [] # qualified fallbacks only # scorer: provider/qualified-scorer + scorerReasoningEffort: none consensus: 1 ``` @@ -47,20 +51,23 @@ Ignore patterns remove matching paths before grounding, batching, and large-revi | Variable | Purpose | | --- | --- | | `POSTIL_API_KEY`, `OPENROUTER_API_KEY`, `MODEL_API_KEY`, `LLM_API_KEY` | Model provider credential, checked in that order | -| `POSTIL_LOGIN_SERVER` | Postil web app used by `postil login`, refresh, and `postil logout`; defaults to `https://postil.dev` | -| `POSTIL_API_BASE` | Model endpoint selected by the operator | +| `POSTIL_LOGIN_SERVER` | Postil web app used to issue a new login; defaults to `https://postil.dev`. Refresh and logout use the stored issuing server and reject a conflicting explicit value. | +| `POSTIL_API_BASE` | Model endpoint selected by the operator. A value that differs from a stored login's endpoint requires an explicit API key. | | `POSTIL_API_FORMAT` | `openai-compatible` or `anthropic` | | `POSTIL_ENDPOINT_AUTH_HEADER`, `POSTIL_ENDPOINT_AUTH_VALUE` | Additional private-gateway authentication | | `POSTIL_ALLOW_PRIVATE_API_BASE` | Explicitly permit a local or private-network endpoint | | `POSTIL_ALLOW_CONFIG_API_BASE` | Honor a repository-controlled `model.apiBase` | | `POSTIL_IGNORE_REPOSITORY_MODEL_CONFIG` | Keep trusted local/hosted model selection independent of repository model fields | | `REVIEW_MODEL` | Primary model override | +| `REVIEW_REASONING_EFFORT` | Reviewer reasoning effort: `max`, `xhigh`, `high`, `medium`, `low`, `minimal`, or `none` | | `REVIEW_MODEL_CASCADE` | Comma-separated fallback models | | `REVIEW_MODEL_CONSENSUS` | Number of models from the generator chain to run concurrently; agreeing findings are retained | | `REVIEW_SCORER_MODEL` | Scorer model override | +| `REVIEW_SCORER_REASONING_EFFORT` | Scorer and adjudication reasoning effort: `max`, `xhigh`, `high`, `medium`, `low`, `minimal`, or `none` | | `REVIEW_SCORER_MODEL_CASCADE` | One scorer fallback model | | `POSTIL_UNCERTAINTY_RESOLUTION` | Override uncertainty resolution with `true`/`false` or `1`/`0` | | `POSTIL_CONCISE_FINDINGS` | Override concise findings with `true`/`false` or `1`/`0` | +| `POSTIL_NO_PROGRESS` | Disable animated progress and retain concise human milestones when set | | `POSTIL_LLM_REQUEST_TIMEOUT_SECS` | Per-attempt model request timeout; defaults to 480 seconds | | `POSTIL_LLM_TOTAL_TIMEOUT_SECS` | Optional total local-review model deadline | | `POSTIL_DETAILS_URL` | HTTP(S) details link for forge check runs | @@ -74,7 +81,11 @@ postil login postil logout ``` -`postil login` authenticates against postil.dev over a device-authorization flow (open the printed URL, enter the code) and stores a renewable credential at `${XDG_CONFIG_HOME:-~/.config}/postil/credentials.json`, mode `0600` in a `0700` directory. That credential is a fallback: it is used only when none of `POSTIL_API_KEY`, `OPENROUTER_API_KEY`, `MODEL_API_KEY`, or `LLM_API_KEY` is set. When it is used, Postil rotates the access credential before it expires and persists the replacement; explicit API keys never trigger a refresh. Its `apiBase` and model select the request unless `POSTIL_API_BASE`/`REVIEW_MODEL` are set, which still win. `postil logout` revokes the stored refresh credential server-side and removes the local file even if that call fails. A legacy access-only login, an expired refresh inactivity window, or a rejected refresh produces one instruction to run `postil login` again rather than a provider authentication error. Temporary refresh failures retain the stored credential and ask the user to try again. +`postil login` authenticates against the configured login server over a device-authorization flow (open the printed URL, enter the code) and stores a renewable credential at `${XDG_CONFIG_HOME:-~/.config}/postil/credentials.json`, mode `0600` in a `0700` directory. An ambiguous polling failure retries the same device code so the service can recover the already-issued pair during its 60-second recovery window. The credential records its canonical issuing server. Refresh and logout use that stored issuer; a conflicting explicit `POSTIL_LOGIN_SERVER` fails before any login credential is sent. Issuer-free credentials from older CLI versions infer `https://postil.dev` only when their normalized `apiBase` is the canonical Postil inference endpoint. An issuer-free credential for any custom endpoint requires `postil login` again before refresh or logout. + +The stored credential is a fallback used only when none of `POSTIL_API_KEY`, `OPENROUTER_API_KEY`, `MODEL_API_KEY`, or `LLM_API_KEY` is set. Postil rotates it before access expiry and persists the replacement; explicit API keys never trigger a refresh. Its `apiBase` is the only endpoint that may receive its bearer. A different `POSTIL_API_BASE` fails before network access unless an explicit API key is set. `REVIEW_MODEL` may still select a model at the stored endpoint. A legacy access-only login, an expired refresh inactivity window, or a rejected refresh instructs the user to run `postil login` again. A valid `Retry-After` value on a temporary refresh rate limit is reported in seconds. Missing or malformed values use a generic retry message, and all temporary refresh failures retain the credential. + +`postil logout` removes the active local credential only after remote revocation succeeds. A failed revocation retains its sole retry handle and asks the user to run `postil logout` again. A new login stages the newly issued family and any overwritten family in a private pending-revocation queue before replacing the active credential. A local replacement failure therefore retains the new remote family's revocation handle. Pending retries never block normal review work and never follow `POSTIL_LOGIN_SERVER` overrides. ## Inspect and initialize @@ -84,4 +95,6 @@ postil config postil doctor ``` -`postil config` prints the resolved non-secret configuration and its sources. `postil doctor` validates endpoint reachability, credential acceptance, and repository setup without printing credential values. Both commands identify renewable logins, access expiry, refresh inactivity expiry, and legacy access-only logins. +`postil config` prints the resolved non-secret configuration and separate provenance for the model, reviewer reasoning effort, and scorer reasoning effort. `postil doctor` validates endpoint reachability, credential acceptance, and repository setup without printing credential values. Both commands identify renewable logins, access expiry, refresh inactivity expiry, and legacy access-only logins. + +Use `--reasoning-effort` and `--scorer-reasoning-effort` for one review. These flags override the matching environment variables, which override `model.reasoningEffort` and `model.scorerReasoningEffort`. The built-in reviewer and scorer defaults are `low` and `none`, respectively. Every request carries the resolved value, including retries and repair calls. diff --git a/docs/model-providers.md b/docs/model-providers.md index 985c560..e023bcd 100644 --- a/docs/model-providers.md +++ b/docs/model-providers.md @@ -4,9 +4,9 @@ Postil speaks either an OpenAI-compatible chat-completions interface or the nati ## Model admission -The CLI has no implicit model or fallback chain. Set `REVIEW_MODEL` to a model that passes the repository benchmark for the intended review profile. Set `REVIEW_MODEL_CASCADE` only to other qualified models. Scoring is disabled unless `REVIEW_SCORER_MODEL` names a qualified scorer. `REVIEW_SCORER_MODEL_CASCADE` accepts one qualified scorer fallback. +Local review defaults to the embedded `openai/gpt-5.6-luna` reviewer with no fallback chain or scorer, so no model setting is required. Reviewer requests use `low` reasoning effort. Local scoring is disabled unless `REVIEW_SCORER_MODEL` names a scorer. The embedded hosted scorer candidate is `openai/gpt-5.6-luna` at `none` effort. `postil models` reports these roles, the accepted endpoint model-ID contracts, embedded qualification status, accepted effort values, and exact `postil doctor` and override commands without reading credentials or contacting a provider. Local use has no fixed model-ID allowlist: Postil passes a non-empty model ID unchanged to the configured OpenAI-compatible or native Anthropic endpoint. Endpoint protocol compatibility does not imply hosted qualification. Hosted deployments admit only complete qualified profiles. `REVIEW_SCORER_MODEL_CASCADE` accepts one qualified scorer fallback. -Hosted deployments admit only complete profiles listed in the embedded `qualified-models.json` qualification manifest. A profile binds the canonical API base, provider API format, ordered generator chain, consensus width, ordered scorer chain, exact Bun evaluator runtime, qualification source commit, a 30-day authority window, at least three complete runs, and SHA-256 digests for the review contract, evaluator contract, fixture set, and benchmark report. The manifest also binds the exact `config.toml` digest. Hosted configuration must match one unexpired profile exactly; the expiry second is excluded. An empty, degraded, stale, or mismatched profile rejects inference. +Hosted deployments admit only complete profiles listed in the embedded `qualified-models.json` qualification manifest. A profile binds the canonical API base, provider API format, ordered generator chain, consensus width, ordered scorer chain, exact Bun evaluator runtime, qualification source commit, a 30-day authority window, at least three complete runs, and SHA-256 digests for the review contract, evaluator contract, fixture set, and benchmark report. The manifest also binds the exact `config.toml` digest, including the reviewer and scorer reasoning defaults. Hosted configuration must match one unexpired profile exactly, so an effort override that differs from those qualified defaults is rejected. The expiry second is excluded. An empty, degraded, stale, or mismatched profile rejects inference. Every nonempty manifest has a committed `qualified-models.attestation.json` bundle. CI and release validation use [`gh attestation verify`](https://cli.github.com/manual/gh_attestation_verify) to require SLSA provenance from the exact `postil-dev/postil-cli` admission workflow, `refs/heads/main` source ref, source and signer commit, issued by GitHub OIDC through public Sigstore on a GitHub-hosted runner. The source commit must be an ancestor of the candidate commit, and only the manifest and bundle may differ. A verified Sigstore timestamp must match the signed issue time and remain within the authority window. The bundle authenticates the exact manifest bytes. Checksums inside the report and manifest provide integrity evidence, not producer authentication. The binary metadata exposes the manifest issue time, exclusive expiry time, and maximum age as top-level values. An empty manifest exposes null authority metadata, needs no bundle, and admits no model. @@ -33,6 +33,8 @@ postil doctor These requests authenticate with `Authorization: Bearer`. +OpenAI-compatible requests carry the [OpenRouter reasoning configuration](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) as `reasoning.effort`. The accepted values are `max`, `xhigh`, `high`, `medium`, `low`, `minimal`, and `none`. This field is sent to OpenRouter and custom OpenAI-compatible endpoints on every request. Postil does not drop or downgrade it, so custom endpoints must implement the extension. + ## Anthropic Messages API ```sh @@ -43,7 +45,7 @@ export REVIEW_MODEL=provider-qualified-anthropic-model postil doctor ``` -Native Anthropic requests use `x-api-key` and `anthropic-version`. Set `REVIEW_SCORER_MODEL` to a model available through the same endpoint when scoring is required. +Native Anthropic requests use `x-api-key` and `anthropic-version`. Reviewer and scorer values from `max` through `low` map to `output_config.effort`; incompatible non-default temperature values are omitted from those requests. `none` maps to disabled thinking and retains the request temperature. The Anthropic request format has no `minimal` mapping, so Postil rejects that value locally before provider access. Individual Anthropic models can support a subset of the remaining effort values. Set `REVIEW_SCORER_MODEL` to a model available through the same endpoint when scoring is required. ## Local endpoints diff --git a/qualified-models.json b/qualified-models.json index f0b8b50..5602481 100644 --- a/qualified-models.json +++ b/qualified-models.json @@ -4,6 +4,6 @@ "qualificationIssuedAtUnixSeconds": null, "qualificationExpiresAtUnixSeconds": null, "qualificationMaxAgeDays": null, - "modelDefaultsSha256": "2a2fc3a8732b833ae25e60e76abe6bdafeb3df9c9f9e283c19f2a94dac5c2df5", + "modelDefaultsSha256": "b581d37af77c48552412c383cebcab55129c1d8c73d3f6dfa00bf3c7ff04ab3f", "profiles": [] } diff --git a/src/api_key.rs b/src/api_key.rs index 84dbf06..d5329b6 100644 --- a/src/api_key.rs +++ b/src/api_key.rs @@ -22,6 +22,14 @@ pub(crate) fn names_text() -> String { API_KEY_ENV_VARS.join(", ") } +pub(crate) fn credential_help() -> String { + format!( + "default model is {}; run `postil login` for hosted inference or set one of {}; see `postil models` for supported model IDs, embedded defaults, qualification status, and override syntax", + crate::config::default_model(), + names_text() + ) +} + pub(crate) fn resolve_from_process_env() -> Option { resolve_with(|name| std::env::var(name).ok()) } @@ -115,6 +123,7 @@ mod tests { fn stored_credential(expires_at: &str) -> Credentials { Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some("https://postil.dev".to_string()), token: "pcli_stored-token-not-a-real-secret".to_string(), expires_at: expires_at.to_string(), refresh_token: None, @@ -122,6 +131,7 @@ mod tests { api_base: "https://postil.dev/api/inference/v1".to_string(), org: "runatlas-is".to_string(), model: "z-ai/glm-5.2".to_string(), + pending_revocations: Vec::new(), } } diff --git a/src/brevity.rs b/src/brevity.rs index 5eb8941..66b1d0a 100644 --- a/src/brevity.rs +++ b/src/brevity.rs @@ -60,7 +60,9 @@ pub(crate) async fn compress_findings( pass.model_incidents .extend_from_slice(error.model_incidents()); pass.usage_accounting_complete &= error.usage_accounting_complete(); - eprintln!("postil: finding compression failed open and kept the original body"); + crate::progress::notice(format_args!( + "postil: finding compression failed open and kept the original body" + )); None } }; diff --git a/src/cli.rs b/src/cli.rs index 3014326..366c55f 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -77,6 +77,9 @@ pub enum Command { config: Option, }, /// Review a diff: a PR/MR on a forge, or local changes. + #[command( + after_help = "Examples:\n postil review\n postil review --staged\n postil review --base origin/main\n\nBare local review selects, in order: staged changes; committed changes since the current branch remote's default branch; tracked working-tree changes; then an empty clean diff. Locally known symbolic remote HEAD, main, master, and trunk refs are recognized without fetching. If no default branch can be resolved, Postil fails closed and asks for --base, --staged, or --diff-file. Explicit source flags are mutually exclusive and keep their exact behavior.\n\nUse `postil models` to see supported model-ID contracts, embedded defaults, qualification, and override syntax." + )] Review { /// Code host for remote review. Inferred as github when --repo is set. #[arg(long, value_enum)] @@ -94,13 +97,13 @@ pub enum Command { #[arg(long)] base_sha: Option, /// Review staged changes (git diff --cached). - #[arg(long)] + #[arg(long, conflicts_with_all = ["base", "diff_file"])] staged: bool, /// Review changes since a base ref (git diff base...HEAD). - #[arg(long)] + #[arg(long, conflicts_with_all = ["staged", "diff_file"])] base: Option, /// Review a unified diff from a file. - #[arg(long)] + #[arg(long, conflicts_with_all = ["staged", "base"])] diff_file: Option, /// Existing advisory check-run id to complete (hosted callers). #[arg(long)] @@ -135,6 +138,18 @@ pub enum Command { /// Model override (else REVIEW_MODEL, else config, else default). #[arg(long)] model: Option, + /// Reviewer reasoning effort: max|xhigh|high|medium|low|minimal|none (else REVIEW_REASONING_EFFORT, else config, else low). + #[arg(long, value_name = "EFFORT")] + reasoning_effort: Option, + /// Scorer reasoning effort: max|xhigh|high|medium|low|minimal|none (else REVIEW_SCORER_REASONING_EFFORT, else config, else none). + #[arg(long, value_name = "EFFORT")] + scorer_reasoning_effort: Option, + /// Keep detailed provider, retry, and batch telemetry in interactive terminals. + #[arg(long)] + verbose: bool, + /// Disable animation while keeping concise human progress milestones. + #[arg(long)] + no_progress: bool, /// Use deterministic semantic synthesis and model-assisted risk selection to cap large reviews at five source batches; report bounded coverage. #[arg(long)] bounded: bool, @@ -186,37 +201,8 @@ pub enum Command { )] publication_input_identity: Option, }, - /// Reply to an @postil mention on a pull request or issue (interactive bot). - Respond { - /// Code host. GitHub and GitLab support PRs/MRs and issues; Bitbucket - /// and Azure DevOps support pull requests only. - #[arg(long, value_enum, default_value = "github")] - forge: ForgeArg, - /// Repository as owner/name. - #[arg(long)] - repo: Option, - /// Pull request number the mention is on. - #[arg(long)] - pr: Option, - /// Issue number the mention is on. - #[arg(long)] - issue: Option, - /// The maintainer's message text (the mention body). Falls back to the - /// POSTIL_COMMENT environment variable — prefer that for automation: - /// argv is visible in `ps` and clap would reject text starting with `-`. - #[arg(long, allow_hyphen_values = true)] - comment: Option, - #[arg(long)] - config: Option, - #[arg(long)] - model: Option, - /// Post the reply to the selected forge. Replies are printed locally by default. - #[arg(long, conflicts_with = "no_post")] - publish: bool, - /// Deprecated compatibility flag. Replies are local-only by default. - #[arg(long, hide = true, conflicts_with = "publish")] - no_post: bool, - }, + /// Explain supported model IDs, embedded defaults, qualification, and overrides. + Models, /// Replay stored envelopes under a candidate config: what would change? Plan { /// Directory of envelope JSON files from previous reviews. @@ -247,14 +233,14 @@ pub enum Command { #[command(subcommand)] action: HookAction, }, - /// Authenticate against postil.dev for zero-config hosted inference. + /// Authenticate through a Postil login server. Login { /// Organization to select during approval. The browser approval page /// is authoritative for membership; this only pre-fills a hint. #[arg(long)] org: Option, }, - /// Remove the stored login credential and revoke it server-side. + /// Revoke the stored login server-side, then remove it locally. Logout, } @@ -327,6 +313,88 @@ mod tests { )); } + #[test] + fn review_help_makes_bare_selection_and_model_discovery_explicit() { + let help = Cli::command() + .find_subcommand_mut("review") + .expect("review subcommand") + .render_long_help() + .to_string(); + assert!(help.contains("postil review\n")); + assert!(help.contains("staged changes; committed changes")); + assert!(help.contains("postil models")); + let root = Cli::command().render_long_help().to_string(); + assert!(root.contains("models")); + assert!(!root.contains("respond")); + } + + #[test] + fn review_accepts_role_specific_reasoning_effort_flags() { + let parsed = Cli::try_parse_from([ + "postil", + "review", + "--reasoning-effort", + "xhigh", + "--scorer-reasoning-effort", + "none", + ]) + .unwrap(); + let Command::Review { + reasoning_effort, + scorer_reasoning_effort, + .. + } = parsed.command + else { + panic!("expected review command"); + }; + assert_eq!(reasoning_effort.as_deref(), Some("xhigh")); + assert_eq!(scorer_reasoning_effort.as_deref(), Some("none")); + + let help = Cli::command() + .find_subcommand_mut("review") + .expect("review subcommand") + .render_long_help() + .to_string(); + assert!(help.contains("--reasoning-effort ")); + assert!(help.contains("--scorer-reasoning-effort ")); + assert!(help.contains("max|xhigh|high|medium|low|minimal|none")); + } + + #[test] + fn local_review_sources_are_mutually_exclusive() { + for arguments in [ + vec!["postil", "review", "--staged", "--base", "main"], + vec!["postil", "review", "--staged", "--diff-file", "change.diff"], + vec![ + "postil", + "review", + "--base", + "main", + "--diff-file", + "change.diff", + ], + ] { + assert!(Cli::try_parse_from(arguments).is_err()); + } + } + + #[test] + fn no_progress_keeps_concise_human_milestones() { + let parsed = Cli::try_parse_from(["postil", "review", "--no-progress"]).unwrap(); + let Command::Review { no_progress, .. } = parsed.command else { + panic!("expected review command"); + }; + assert!(no_progress); + + let help = Cli::command() + .find_subcommand_mut("review") + .expect("review subcommand") + .render_long_help() + .to_string(); + assert!(help.contains("--no-progress")); + assert!(help.contains("concise human progress milestones")); + } + #[test] fn review_publication_requires_an_explicit_flag() { let parsed = Cli::try_parse_from([ diff --git a/src/config.rs b/src/config.rs index 7cbe312..ea40ae5 100644 --- a/src/config.rs +++ b/src/config.rs @@ -1,7 +1,8 @@ //! Resolved review configuration. //! -//! Precedence: CLI flags > environment > `.postil.{yaml,yml,json}` > -//! `.coderabbit.yaml` (translated) > defaults. +//! Model precedence: CLI flags > environment > `.postil.{yaml,yml,json}` > +//! stored login routing > embedded defaults. Translated `.coderabbit.yaml` +//! settings do not select a model. //! //! Exception: `model.apiBase` from a config file is repo-controlled, and the //! resolved base URL receives the deployment's bearer key. It is ignored by @@ -14,9 +15,10 @@ //! valid, unexpired credential exists at //! `${XDG_CONFIG_HOME:-~/.config}/postil/credentials.json`, //! `login::resolve_stored_token` supplies its bearer key and its `apiBase`/ -//! `model` replace the values resolved above -- unless `POSTIL_API_BASE` / -//! `REVIEW_MODEL` are set, which still win, applied first in this same -//! function. An expired or unreadable stored credential is left alone here; +//! `model` provide the baseline below trusted project configuration and +//! environment overrides. A resolved endpoint override is accepted only with +//! an explicit API key; a stored login remains bound to its issuing endpoint. +//! An expired or unreadable stored credential is left alone here; //! `resolve_api_key` reports it as one actionable "run `postil login` again" //! error, while `postil config` reports the stored-login state without needing //! a live key. @@ -117,17 +119,61 @@ impl ApiFormat { } } +pub const REASONING_EFFORT_VALUES: &str = "max|xhigh|high|medium|low|minimal|none"; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReasoningEffort { + Max, + Xhigh, + High, + Medium, + Low, + Minimal, + None, +} + +impl ReasoningEffort { + pub fn parse(field: &str, value: &str) -> Result { + match value.trim().to_ascii_lowercase().as_str() { + "max" => Ok(Self::Max), + "xhigh" => Ok(Self::Xhigh), + "high" => Ok(Self::High), + "medium" => Ok(Self::Medium), + "low" => Ok(Self::Low), + "minimal" => Ok(Self::Minimal), + "none" => Ok(Self::None), + _ => anyhow::bail!( + "invalid {field} {value:?}; accepted values: {REASONING_EFFORT_VALUES}" + ), + } + } + + pub fn as_str(self) -> &'static str { + match self { + Self::Max => "max", + Self::Xhigh => "xhigh", + Self::High => "high", + Self::Medium => "medium", + Self::Low => "low", + Self::Minimal => "minimal", + Self::None => "none", + } + } +} + #[derive(Debug, Clone, PartialEq, Eq)] pub struct ModelDefaults { pub version: u64, pub source_sha256: String, pub default_model: String, + pub reasoning_effort: ReasoningEffort, pub cascade: Vec, pub consensus: usize, pub api_base: String, pub api_format: ApiFormat, pub scorer_enabled: bool, pub scorer_model: String, + pub scorer_reasoning_effort: ReasoningEffort, pub scorer_fallback: String, pub scorer_qualification_candidates: Vec, } @@ -239,6 +285,7 @@ const MAX_SAFE_JSON_INTEGER: u64 = 9_007_199_254_740_991; const QUALIFICATION_CANDIDATE_PROFILE_ENV: &str = "POSTIL_QUALIFICATION_CANDIDATE_PROFILE"; const BENCH_SCREEN_PROFILE_ENV: &str = "POSTIL_BENCH_SCREEN_PROFILE"; const BENCH_FORCE_BOUNDED_SELECTION_ENV: &str = "POSTIL_BENCH_FORCE_BOUNDED_SELECTION"; +const IGNORE_REPOSITORY_MODEL_CONFIG_ENV: &str = "POSTIL_IGNORE_REPOSITORY_MODEL_CONFIG"; const PROVISIONAL_HOSTED_ROSTER_ENV: &str = "POSTIL_PROVISIONAL_HOSTED_ROSTER"; #[derive(Debug, Deserialize)] @@ -246,6 +293,7 @@ const PROVISIONAL_HOSTED_ROSTER_ENV: &str = "POSTIL_PROVISIONAL_HOSTED_ROSTER"; struct ModelDefaultsFile { version: u64, default_model: String, + reasoning_effort: String, cascade: Vec, consensus: usize, api_base: String, @@ -258,6 +306,7 @@ struct ModelDefaultsFile { struct ScorerDefaultsFile { enabled: bool, default_model: String, + reasoning_effort: String, fallback: String, qualification_candidates: Vec, } @@ -277,10 +326,18 @@ pub fn default_cascade() -> &'static [String] { model_defaults().cascade.as_slice() } +pub fn default_reasoning_effort() -> ReasoningEffort { + model_defaults().reasoning_effort +} + pub fn default_scorer_model() -> &'static str { model_defaults().scorer_model.as_str() } +pub fn default_scorer_reasoning_effort() -> ReasoningEffort { + model_defaults().scorer_reasoning_effort +} + pub fn default_scorer_fallback() -> &'static str { model_defaults().scorer_fallback.as_str() } @@ -406,6 +463,9 @@ fn admitted_profile_for( } pub fn admitted_profile_for_config(config: &Config) -> Option<&'static QualificationProfile> { + if !reasoning_efforts_match_embedded_defaults(config) { + return None; + } let generator_chain = config.model_chain(); let scorer_chain = config.scorer_chain(); let api_base = normalize_api_base(&config.api_base).ok()?; @@ -418,6 +478,12 @@ pub fn admitted_profile_for_config(config: &Config) -> Option<&'static Qualifica }) } +fn reasoning_efforts_match_embedded_defaults(config: &Config) -> bool { + let defaults = model_defaults(); + config.reasoning_effort == defaults.reasoning_effort + && config.scorer_reasoning_effort == defaults.scorer_reasoning_effort +} + fn evaluator_runtime_identity() -> String { #[derive(Deserialize)] #[serde(rename_all = "camelCase")] @@ -720,6 +786,9 @@ fn validate_qualification_authority(manifest: &QualificationManifest, now: u64) fn parse_model_defaults(raw: &str) -> Result { let file: ModelDefaultsFile = toml::from_str(raw)?; + let reasoning_effort = ReasoningEffort::parse("reasoning_effort", &file.reasoning_effort)?; + let scorer_reasoning_effort = + ReasoningEffort::parse("scorer.reasoning_effort", &file.scorer.reasoning_effort)?; anyhow::ensure!( file.version > 0, "model defaults version must be greater than zero" @@ -785,12 +854,14 @@ fn parse_model_defaults(raw: &str) -> Result { version: file.version, source_sha256: sha256_hex(raw), default_model: file.default_model, + reasoning_effort, cascade: file.cascade, consensus: file.consensus, api_base: file.api_base, api_format: file.api_format, scorer_enabled: file.scorer.enabled, scorer_model: file.scorer.default_model, + scorer_reasoning_effort, scorer_fallback: file.scorer.fallback, scorer_qualification_candidates: file.scorer.qualification_candidates, }) @@ -979,8 +1050,14 @@ pub struct Config { /// also meet the calibrated 0.30 gate confidence floor. Default: [HumanEscalation]. pub block_on_kinds: Vec, pub model: String, + pub reasoning_effort: ReasoningEffort, + /// Winning source for reviewer reasoning effort, shown by `postil config`. + pub reasoning_effort_source: String, pub cascade: Vec, pub scorer: String, + pub scorer_reasoning_effort: ReasoningEffort, + /// Winning source for scorer reasoning effort, shown by `postil config`. + pub scorer_reasoning_effort_source: String, pub scorer_fallback: String, /// Embedded scoring remains disabled until a candidate passes the repeated /// qualification gate. BYOK can select a scorer and one fallback. @@ -1002,6 +1079,8 @@ pub struct Config { pub content_policy_disabled: bool, /// Where the config came from, for `postil config` provenance. pub source: String, + /// Winning source for the generator chain, shown by `postil config`. + pub model_source: String, } impl Default for Config { @@ -1022,9 +1101,13 @@ impl Default for Config { gate_fail_on: GateLevel::Severity(Severity::Error), gate_on_error: OnError::Block, block_on_kinds: vec![Kind::HumanEscalation], - model: String::new(), - cascade: Vec::new(), + model: defaults.default_model.clone(), + reasoning_effort: defaults.reasoning_effort, + reasoning_effort_source: "embedded default".to_string(), + cascade: defaults.cascade.clone(), scorer: String::new(), + scorer_reasoning_effort: defaults.scorer_reasoning_effort, + scorer_reasoning_effort_source: "embedded default".to_string(), scorer_fallback: String::new(), scorer_enabled: false, api_base: defaults.api_base.clone(), @@ -1034,6 +1117,7 @@ impl Default for Config { content_policy: Some(BUILTIN_CONTENT_POLICY.to_string()), content_policy_disabled: false, source: "defaults".to_string(), + model_source: "embedded default".to_string(), } } } @@ -1083,8 +1167,10 @@ pub struct GateSection { #[serde(deny_unknown_fields, rename_all = "camelCase")] pub struct ModelSection { pub name: Option, + pub reasoning_effort: Option, pub cascade: Option>, pub scorer: Option, + pub scorer_reasoning_effort: Option, pub api_base: Option, pub api_format: Option, pub consensus: Option, @@ -1107,27 +1193,29 @@ impl Config { /// Resolve config for a repo root. `explicit` (from --config) bypasses discovery. pub fn load(root: &Path, explicit: Option<&Path>) -> Result { #[cfg(test)] - let _environment_guard = tests::env_lock().lock().unwrap(); - let mut cfg = if let Some(path) = explicit { - let mut c = Self::from_postil_file(path) + let _environment_guard = crate::test_env_lock().lock().unwrap(); + let mut cfg = Config::default(); + // Login supplies a local baseline, below trusted project configuration + // and environment overrides but above the embedded provider defaults. + cfg.apply_stored_login_credential(); + if let Some(path) = explicit { + let file = Self::read_postil_file(path) .with_context(|| format!("reading config {}", path.display()))?; - c.source = path.display().to_string(); - c + cfg.apply_file(file)?; + cfg.source = path.display().to_string(); } else if let Some(path) = find_first(root, &[".postil.yaml", ".postil.yml", ".postil.json"]) { - let mut c = Self::from_postil_file(&path) + let file = Self::read_postil_file(&path) .with_context(|| format!("reading config {}", path.display()))?; - c.source = rel_name(&path); - c + cfg.apply_file(file)?; + cfg.source = rel_name(&path); } else if let Some(path) = find_first(root, &[".coderabbit.yaml", ".coderabbit.yml"]) { - let mut c = Self::from_coderabbit(&path) + let translated = Self::from_coderabbit(&path) .with_context(|| format!("translating {}", path.display()))?; - c.source = format!("{} (translated)", rel_name(&path)); - c - } else { - Config::default() - }; + cfg.apply_translated_config(translated); + cfg.source = format!("{} (translated)", rel_name(&path)); + } cfg.apply_env()?; validate_benchmark_bounded_selection(&cfg)?; // Repo guardrails are a separate file so they can be long-form prose. @@ -1158,7 +1246,7 @@ impl Config { Ok(cfg) } - fn from_postil_file(path: &Path) -> Result { + fn read_postil_file(path: &Path) -> Result { let raw = std::fs::read_to_string(path)?; let file: FileConfig = if path.extension().is_some_and(|e| e == "json") { serde_json::from_str(&raw)? @@ -1166,9 +1254,7 @@ impl Config { parse_yaml_value(&raw)?; yaml_serde::from_str(&raw)? }; - let mut cfg = Config::default(); - cfg.apply_file(file)?; - Ok(cfg) + Ok(file) } pub fn apply_file(&mut self, f: FileConfig) -> Result<()> { @@ -1249,12 +1335,18 @@ impl Config { } if let Some(m) = f.model { if hosted_mode { - eprintln!( + crate::progress::notice(format_args!( "postil: ignoring repository model configuration in hosted mode; hosted inference selects the provider and model roster" - ); + )); } else { if let Some(n) = m.name { self.model = n; + self.model_source = "trusted project config".to_string(); + } + if let Some(effort) = m.reasoning_effort { + self.reasoning_effort = + ReasoningEffort::parse("model.reasoningEffort", &effort)?; + self.reasoning_effort_source = "trusted project config".to_string(); } if let Some(c) = m.cascade { self.cascade = c; @@ -1264,6 +1356,11 @@ impl Config { self.scorer_fallback.clear(); self.scorer_enabled = true; } + if let Some(effort) = m.scorer_reasoning_effort { + self.scorer_reasoning_effort = + ReasoningEffort::parse("model.scorerReasoningEffort", &effort)?; + self.scorer_reasoning_effort_source = "trusted project config".to_string(); + } if let Some(b) = m.api_base { // `model.apiBase` from `.postil.yaml` is repo-controlled, and the // resolved base URL receives the deployment's bearer key. Honoring @@ -1275,11 +1372,11 @@ impl Config { if allow_api_base { self.api_base = b; } else { - eprintln!( + crate::progress::notice(format_args!( "postil: ignoring model.apiBase from config ({b:?}); set \ POSTIL_ALLOW_CONFIG_API_BASE=1 to honor it, or use the \ POSTIL_API_BASE environment variable" - ); + )); } } if let Some(format) = m.api_format { @@ -1346,6 +1443,12 @@ impl Config { Ok(cfg) } + fn apply_translated_config(&mut self, translated: Config) { + self.enabled = translated.enabled; + self.ignore = translated.ignore; + self.min_confidence = translated.min_confidence; + } + fn apply_env(&mut self) -> Result<()> { if let Ok(value) = std::env::var("POSTIL_CONCISE_FINDINGS") && !value.trim().is_empty() @@ -1369,6 +1472,19 @@ impl Config { ), }; } + if let Ok(value) = std::env::var("REVIEW_REASONING_EFFORT") + && !value.trim().is_empty() + { + self.reasoning_effort = ReasoningEffort::parse("REVIEW_REASONING_EFFORT", &value)?; + self.reasoning_effort_source = "environment".to_string(); + } + if let Ok(value) = std::env::var("REVIEW_SCORER_REASONING_EFFORT") + && !value.trim().is_empty() + { + self.scorer_reasoning_effort = + ReasoningEffort::parse("REVIEW_SCORER_REASONING_EFFORT", &value)?; + self.scorer_reasoning_effort_source = "environment".to_string(); + } if hosted_mode() { if let Some(profile) = admitted_profile_for(model_defaults(), qualification_manifest()) { @@ -1380,6 +1496,7 @@ impl Config { self.scorer_enabled = !profile.scorer_chain.is_empty(); self.api_base = profile.api_base.clone(); self.api_format = profile.api_format; + self.model_source = "hosted qualification profile".to_string(); } else if provisional_hosted_roster_enabled() { let profile = provisional_hosted_profile(); self.model = profile.generator_chain[0].clone(); @@ -1390,6 +1507,7 @@ impl Config { self.scorer_enabled = !profile.scorer_chain.is_empty(); self.api_base = profile.api_base.clone(); self.api_format = profile.api_format; + self.model_source = "hosted provisional profile".to_string(); } return Ok(()); } @@ -1404,12 +1522,14 @@ impl Config { self.scorer_enabled = true; self.api_base = profile.api_base; self.api_format = profile.api_format; + self.model_source = "qualification candidate profile".to_string(); return Ok(()); } if let Ok(m) = std::env::var("REVIEW_MODEL") - && !m.is_empty() + && !m.trim().is_empty() { self.model = m; + self.model_source = "environment".to_string(); } if let Ok(c) = std::env::var("REVIEW_MODEL_CASCADE") && !c.is_empty() @@ -1465,16 +1585,15 @@ impl Config { { self.api_format = ApiFormat::parse(&format)?; } - self.apply_stored_login_credential(); Ok(()) } - /// A first-time `postil login` user gets working defaults with zero - /// `.postil.yaml`: when no explicit key env var is set, a stored - /// credential supplies `apiBase` and `model`, unless `POSTIL_API_BASE`/ - /// `REVIEW_MODEL` (just applied above) already overrode them. The - /// cascade is cleared along with `model`, since a leftover BYOK cascade - /// names models the hosted gateway does not operate. + /// A stored `postil login` credential supplies a local fallback API base + /// and model. Project configuration and environment variables are applied + /// after this method and therefore retain their documented precedence. + /// Runtime credential resolution rejects an endpoint override when no + /// explicit API key is set. The cascade is cleared with the model because + /// a BYOK fallback chain does not describe the hosted gateway. /// /// This applies even when the credential is expired: routing /// (`apiBase`/`model`) is not a secret, and populating it here is what @@ -1483,23 +1602,21 @@ impl Config { /// resolution in `resolve_api_key` is the sole place that enforces and /// reports expiry -- it still refuses the token regardless of what /// happens here. - fn apply_stored_login_credential(&mut self) { + fn apply_stored_login_credential(&mut self) -> bool { if api_key::resolve_from_process_env().is_some() { - return; + return false; } let Ok(path) = credentials::default_path() else { - return; + return false; }; let Ok(Some(creds)) = credentials::read(&path) else { - return; + return false; }; - if std::env::var("POSTIL_API_BASE").is_err() { - self.api_base = creds.api_base; - } - if std::env::var("REVIEW_MODEL").is_err() { - self.model = creds.model; - self.cascade.clear(); - } + self.api_base = creds.api_base; + self.model = creds.model; + self.cascade.clear(); + self.model_source = "stored login".to_string(); + true } /// All models to try, in order, deduplicated. @@ -1540,9 +1657,20 @@ impl Config { return Ok(()); } let generator_chain = self.model_chain(); + if self.api_format == ApiFormat::Anthropic { + anyhow::ensure!( + self.reasoning_effort != ReasoningEffort::Minimal, + "model.reasoningEffort minimal is unsupported by the Anthropic request format; accepted Anthropic values: max|xhigh|high|medium|low|none" + ); + anyhow::ensure!( + self.scorer_reasoning_effort != ReasoningEffort::Minimal, + "model.scorerReasoningEffort minimal is unsupported by the Anthropic request format; accepted Anthropic values: max|xhigh|high|medium|low|none" + ); + } anyhow::ensure!( !generator_chain.is_empty(), - "no review model is configured; pass --model, set REVIEW_MODEL, or set model.name in a trusted local config" + "no review model is configured; remove the empty override to use default {}, or run `postil models` for supported model IDs, qualification status, and override syntax", + default_model() ); if hosted { let manifest = qualification_manifest(); @@ -1553,6 +1681,7 @@ impl Config { && profile.consensus == self.consensus && profile.scorer_chain == scorer_chain && profile.api_format == self.api_format + && reasoning_efforts_match_embedded_defaults(self) && normalize_api_base(&self.api_base).ok().as_deref() == Some(profile.api_base.as_str()) }) || (provisional && provisional_hosted_profile_for_config(self).is_some()), @@ -1759,7 +1888,8 @@ pub(crate) fn benchmark_screening_profile_for_config( && profile.consensus == config.consensus && profile.scorer_chain == config.scorer_chain() && profile.api_base == api_base - && profile.api_format == config.api_format, + && profile.api_format == config.api_format + && reasoning_efforts_match_embedded_defaults(config), "benchmark screening profile does not exactly match the resolved review configuration" ); Ok(Some(profile)) @@ -1858,7 +1988,8 @@ pub(crate) fn qualification_candidate_profile_for_config( && profile.consensus == config.consensus && profile.scorer_chain == config.scorer_chain() && profile.api_base == api_base - && profile.api_format == config.api_format, + && profile.api_format == config.api_format + && reasoning_efforts_match_embedded_defaults(config), "qualification candidate profile does not exactly match the resolved review configuration" ); Ok(Some(profile)) @@ -1903,6 +2034,7 @@ fn provisional_hosted_profile_for_config(config: &Config) -> Option bool { hosted_runtime_mode() || benchmark_screening_mode() || benchmark_force_bounded_selection_requested() - || std::env::var("POSTIL_IGNORE_REPOSITORY_MODEL_CONFIG") + || std::env::var(IGNORE_REPOSITORY_MODEL_CONFIG_ENV) .map(|value| value == "1" || value.eq_ignore_ascii_case("true")) .unwrap_or(false) } -fn normalize_api_base(value: &str) -> Result { +pub(crate) fn normalize_api_base(value: &str) -> Result { let url = reqwest::Url::parse(value).context("model API base must be an absolute URL")?; anyhow::ensure!( matches!(url.scheme(), "http" | "https"), @@ -2037,8 +2169,10 @@ gate: model: name: __DEFAULT_MODEL__ + reasoningEffort: __DEFAULT_REASONING_EFFORT__ # max | xhigh | high | medium | low | minimal | none cascade: __DEFAULT_CASCADE__ # scorer: provider/model # explicit BYOK opt-in; embedded scoring is disabled + scorerReasoningEffort: __DEFAULT_SCORER_REASONING_EFFORT__ # apiBase: https://openrouter.ai/api/v1 # OpenAI-compatible or Anthropic endpoint base URL. # apiFormat: openai-compatible # openai-compatible (default) or anthropic. # # Ignored from config by default (a repo could redirect @@ -2064,6 +2198,14 @@ pub fn starter_config() -> &'static str { .collect::(); STARTER_CONFIG_TEMPLATE .replace("__DEFAULT_MODEL__", &yaml_scalar(&defaults.model)) + .replace( + "__DEFAULT_REASONING_EFFORT__", + defaults.reasoning_effort.as_str(), + ) + .replace( + "__DEFAULT_SCORER_REASONING_EFFORT__", + defaults.scorer_reasoning_effort.as_str(), + ) .replace("__DEFAULT_CASCADE__", &cascade) }) } @@ -2079,8 +2221,8 @@ pub const BUILTIN_CONTENT_POLICY: &str = include_str!("builtin-content-policy.md #[cfg(test)] mod tests { use super::*; + use crate::test_env_lock as env_lock; use std::io::Write; - use std::sync::Mutex; struct EnvRestore { name: &'static str, @@ -2100,6 +2242,12 @@ mod tests { std::env::set_var(name, value); } } + + fn remove(name: &str) { + unsafe { + std::env::remove_var(name); + } + } } impl Drop for EnvRestore { @@ -2115,9 +2263,44 @@ mod tests { } } - pub(super) fn env_lock() -> &'static Mutex<()> { - static LOCK: OnceLock> = OnceLock::new(); - LOCK.get_or_init(|| Mutex::new(())) + /// Pin tests that exercise local repository model configuration to the + /// local runtime contract. The production wrapper intentionally enters + /// hosted mode when deployment environment says so, therefore these tests + /// must not inherit that process-wide state from another test or harness. + struct LocalModelConfigEnv { + _hosted: EnvRestore, + _candidate: EnvRestore, + _screening: EnvRestore, + _bounded: EnvRestore, + _ignore_repository_models: EnvRestore, + _review_reasoning_effort: EnvRestore, + _scorer_reasoning_effort: EnvRestore, + } + + impl LocalModelConfigEnv { + fn clear() -> Self { + let environment = Self { + _hosted: EnvRestore::capture("POSTIL_HOSTED_MODE"), + _candidate: EnvRestore::capture(QUALIFICATION_CANDIDATE_PROFILE_ENV), + _screening: EnvRestore::capture(BENCH_SCREEN_PROFILE_ENV), + _bounded: EnvRestore::capture(BENCH_FORCE_BOUNDED_SELECTION_ENV), + _ignore_repository_models: EnvRestore::capture(IGNORE_REPOSITORY_MODEL_CONFIG_ENV), + _review_reasoning_effort: EnvRestore::capture("REVIEW_REASONING_EFFORT"), + _scorer_reasoning_effort: EnvRestore::capture("REVIEW_SCORER_REASONING_EFFORT"), + }; + for name in [ + "POSTIL_HOSTED_MODE", + QUALIFICATION_CANDIDATE_PROFILE_ENV, + BENCH_SCREEN_PROFILE_ENV, + BENCH_FORCE_BOUNDED_SELECTION_ENV, + IGNORE_REPOSITORY_MODEL_CONFIG_ENV, + "REVIEW_REASONING_EFFORT", + "REVIEW_SCORER_REASONING_EFFORT", + ] { + EnvRestore::remove(name); + } + environment + } } fn write(dir: &Path, name: &str, content: &str) { @@ -2180,6 +2363,10 @@ mod tests { assert_eq!(parsed.source_sha256, sha256_hex(MODEL_DEFAULTS_TOML)); assert_eq!(parsed.source_sha256.len(), 64); assert_eq!(parsed.default_model, raw["default_model"].as_str().unwrap()); + assert_eq!( + parsed.reasoning_effort.as_str(), + raw["reasoning_effort"].as_str().unwrap() + ); assert_eq!( parsed.cascade, raw["cascade"] @@ -2193,6 +2380,10 @@ mod tests { parsed.scorer_model, raw["scorer"]["default_model"].as_str().unwrap() ); + assert_eq!( + parsed.scorer_reasoning_effort.as_str(), + raw["scorer"]["reasoning_effort"].as_str().unwrap() + ); assert_eq!( parsed.scorer_fallback, raw["scorer"]["fallback"].as_str().unwrap() @@ -2218,19 +2409,37 @@ mod tests { let defaults = model_defaults(); assert_eq!(default_model(), defaults.default_model); assert_eq!(super::default_cascade(), defaults.cascade); + assert_eq!(default_reasoning_effort(), defaults.reasoning_effort); assert_eq!(default_scorer_model(), defaults.scorer_model); assert_eq!(default_scorer_fallback(), defaults.scorer_fallback); + assert_eq!( + default_scorer_reasoning_effort(), + defaults.scorer_reasoning_effort + ); assert_eq!( scorer_qualification_candidates(), defaults.scorer_qualification_candidates ); } + #[test] + fn reasoning_effort_parser_matches_the_openrouter_contract() { + for value in ["max", "xhigh", "high", "medium", "low", "minimal", "none"] { + let parsed = ReasoningEffort::parse("effort", value).unwrap(); + assert_eq!(parsed.as_str(), value); + } + assert_eq!( + ReasoningEffort::parse("effort", " XHIGH ").unwrap(), + ReasoningEffort::Xhigh + ); + } + #[test] fn malformed_model_defaults_fail_loudly() { let err = parse_model_defaults( r#"version = 1 default_model = "example/model" +reasoning_effort = "low" cascade = ["example/fallback"] consensus = 1 api_base = "https://openrouter.ai/api/v1" @@ -2240,6 +2449,7 @@ unexpected_key = "typo" [scorer] enabled = false default_model = "example/scorer" +reasoning_effort = "none" fallback = "example/scorer-fallback" qualification_candidates = ["example/scorer"] "#, @@ -2256,41 +2466,45 @@ qualification_candidates = ["example/scorer"] ( "version = 0\n\ default_model = \"example/model\"\n\ + reasoning_effort = \"low\"\n\ cascade = [\"example/fallback\"]\n\ consensus = 1\n\ api_base = \"https://openrouter.ai/api/v1\"\n\ api_format = \"openai-compatible\"\n\ - scorer = { enabled = false, default_model = \"example/scorer\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", + scorer = { enabled = false, default_model = \"example/scorer\", reasoning_effort = \"none\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", "version must be greater than zero", ), ( "version = 1\n\ default_model = \"\"\n\ + reasoning_effort = \"low\"\n\ cascade = [\"example/fallback\"]\n\ consensus = 1\n\ api_base = \"https://openrouter.ai/api/v1\"\n\ api_format = \"openai-compatible\"\n\ - scorer = { enabled = false, default_model = \"example/scorer\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", + scorer = { enabled = false, default_model = \"example/scorer\", reasoning_effort = \"none\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", "cascade must be empty when defaultModel is empty", ), ( "version = 1\n\ default_model = \"example/model\"\n\ + reasoning_effort = \"low\"\n\ cascade = [\"\"]\n\ consensus = 1\n\ api_base = \"https://openrouter.ai/api/v1\"\n\ api_format = \"openai-compatible\"\n\ - scorer = { enabled = false, default_model = \"example/scorer\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", + scorer = { enabled = false, default_model = \"example/scorer\", reasoning_effort = \"none\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", "cascade entries must not be empty", ), ( "version = 1\n\ default_model = \"example/model\"\n\ + reasoning_effort = \"low\"\n\ cascade = [\"example/fallback\"]\n\ consensus = 1\n\ api_base = \"https://openrouter.ai/api/v1\"\n\ api_format = \"openai-compatible\"\n\ - scorer = { enabled = false, default_model = \"\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", + scorer = { enabled = false, default_model = \"\", reasoning_effort = \"none\", fallback = \"example/scorer-fallback\", qualification_candidates = [\"example/scorer\"] }\n", "scorer configuration must be empty when scorer.defaultModel is empty", ), ]; @@ -2307,11 +2521,12 @@ qualification_candidates = ["example/scorer"] fn model_defaults_reject_duplicate_generator_and_scorer_entries() { let duplicate_generator = r#"version = 1 default_model = "provider/model" +reasoning_effort = "low" cascade = ["provider/model"] consensus = 2 api_base = "https://openrouter.ai/api/v1" api_format = "openai-compatible" -scorer = { enabled = true, default_model = "provider/scorer", fallback = "", qualification_candidates = [] } +scorer = { enabled = true, default_model = "provider/scorer", reasoning_effort = "none", fallback = "", qualification_candidates = [] } "#; assert!( parse_model_defaults(duplicate_generator) @@ -2322,11 +2537,12 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "", qua let duplicate_scorer = r#"version = 1 default_model = "provider/model" +reasoning_effort = "low" cascade = [] consensus = 1 api_base = "https://openrouter.ai/api/v1" api_format = "openai-compatible" -scorer = { enabled = true, default_model = "provider/scorer", fallback = "provider/scorer", qualification_candidates = [] } +scorer = { enabled = true, default_model = "provider/scorer", reasoning_effort = "none", fallback = "provider/scorer", qualification_candidates = [] } "#; assert!( parse_model_defaults(duplicate_scorer) @@ -2346,11 +2562,16 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid } #[test] - fn provider_guide_requires_qualified_explicit_models() { + fn provider_guide_documents_local_default_and_hosted_admission() { let readme = include_str!("../README.md"); assert!(readme.contains("docs/model-providers.md")); let provider_guide = include_str!("../docs/model-providers.md"); - assert!(provider_guide.contains("no implicit model or fallback chain")); + assert!(provider_guide.contains("openai/gpt-5.6-luna")); + assert!(provider_guide.contains("postil models")); + assert!(provider_guide.contains("reasoning.effort")); + assert!(provider_guide.contains("`minimal`")); + assert!(provider_guide.contains("`none`")); + assert!(provider_guide.contains("output_config.effort")); assert!(provider_guide.contains("qualification manifest")); } @@ -2537,19 +2758,215 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid } #[test] - fn defaults_require_an_explicit_model_roster() { + fn defaults_use_the_embedded_generator_roster_without_a_scorer() { let c = Config::default(); let defaults = model_defaults(); - assert!(c.model.is_empty()); - assert!(c.cascade.is_empty()); - assert!(c.scorer.is_empty()); + assert_eq!(c.model, defaults.default_model); + assert_eq!(c.cascade, defaults.cascade); + assert_eq!(c.model_source, "embedded default"); + assert_eq!(c.reasoning_effort, ReasoningEffort::Low); + assert_eq!(c.reasoning_effort_source, "embedded default"); + assert_eq!(c.scorer_reasoning_effort, ReasoningEffort::None); + assert_eq!(c.scorer_reasoning_effort_source, "embedded default"); assert!(c.concise_findings); assert!(!c.scorer_enabled); assert!(c.scorer_chain().is_empty()); - assert!(c.model_chain().is_empty()); + assert_eq!(c.model_chain(), vec![defaults.default_model.clone()]); assert_eq!(c.api_base, defaults.api_base); } + #[test] + fn empty_model_environment_does_not_claim_provenance() { + let _lock = env_lock().lock().unwrap(); + let _local_environment = LocalModelConfigEnv::clear(); + let review_model = EnvRestore::capture("REVIEW_MODEL"); + let review_cascade = EnvRestore::capture("REVIEW_MODEL_CASCADE"); + EnvRestore::set("REVIEW_MODEL", " "); + EnvRestore::set("REVIEW_MODEL_CASCADE", ""); + let mut config = Config::default(); + + config.apply_env().unwrap(); + + assert_eq!(config.model, model_defaults().default_model); + assert_eq!(config.model_source, "embedded default"); + EnvRestore::set("REVIEW_MODEL", "environment/model"); + config.apply_env().unwrap(); + assert_eq!(config.model, "environment/model"); + assert_eq!(config.model_source, "environment"); + drop(review_cascade); + drop(review_model); + } + + #[test] + fn reasoning_effort_precedence_and_provenance_are_role_specific() { + let _lock = env_lock().lock().unwrap(); + let _local_environment = LocalModelConfigEnv::clear(); + let file: FileConfig = yaml_serde::from_str( + "model:\n reasoningEffort: medium\n scorerReasoningEffort: high\n", + ) + .unwrap(); + let mut config = Config::default(); + config.apply_file(file).unwrap(); + assert_eq!(config.reasoning_effort, ReasoningEffort::Medium); + assert_eq!(config.reasoning_effort_source, "trusted project config"); + assert_eq!(config.scorer_reasoning_effort, ReasoningEffort::High); + assert_eq!( + config.scorer_reasoning_effort_source, + "trusted project config" + ); + + EnvRestore::set("REVIEW_REASONING_EFFORT", "xhigh"); + EnvRestore::set("REVIEW_SCORER_REASONING_EFFORT", "minimal"); + config.apply_env().unwrap(); + assert_eq!(config.reasoning_effort, ReasoningEffort::Xhigh); + assert_eq!(config.reasoning_effort_source, "environment"); + assert_eq!(config.scorer_reasoning_effort, ReasoningEffort::Minimal); + assert_eq!(config.scorer_reasoning_effort_source, "environment"); + } + + #[test] + fn invalid_reasoning_effort_lists_every_accepted_value() { + let _lock = env_lock().lock().unwrap(); + let _local_environment = LocalModelConfigEnv::clear(); + let file: FileConfig = yaml_serde::from_str( + "model:\n reasoningEffort: turbo\n scorerReasoningEffort: none\n", + ) + .unwrap(); + let error = Config::default().apply_file(file).unwrap_err(); + assert!(error.to_string().contains("model.reasoningEffort")); + assert!(error.to_string().contains(REASONING_EFFORT_VALUES)); + + EnvRestore::set("REVIEW_SCORER_REASONING_EFFORT", "turbo"); + let error = Config::default().apply_env().unwrap_err(); + assert!(error.to_string().contains("REVIEW_SCORER_REASONING_EFFORT")); + assert!(error.to_string().contains(REASONING_EFFORT_VALUES)); + } + + #[test] + fn native_anthropic_rejects_minimal_effort_before_provider_access() { + let reviewer = Config { + api_format: ApiFormat::Anthropic, + reasoning_effort: ReasoningEffort::Minimal, + ..Config::default() + }; + let error = reviewer.require_model_for(false, false).unwrap_err(); + assert!(error.to_string().contains("reasoningEffort minimal")); + assert!(error.to_string().contains("max|xhigh|high|medium|low|none")); + + let scorer = Config { + api_format: ApiFormat::Anthropic, + scorer_reasoning_effort: ReasoningEffort::Minimal, + ..Config::default() + }; + let error = scorer.require_model_for(false, false).unwrap_err(); + assert!(error.to_string().contains("scorerReasoningEffort minimal")); + assert!(error.to_string().contains("max|xhigh|high|medium|low|none")); + } + + #[test] + fn hosted_profile_provenance_replaces_ignored_project_provenance() { + let _lock = env_lock().lock().unwrap(); + let hosted = EnvRestore::capture("POSTIL_HOSTED_MODE"); + let provisional = EnvRestore::capture(PROVISIONAL_HOSTED_ROSTER_ENV); + EnvRestore::set("POSTIL_HOSTED_MODE", "1"); + EnvRestore::set(PROVISIONAL_HOSTED_ROSTER_ENV, "1"); + let mut config = Config::default(); + let file: FileConfig = + yaml_serde::from_str("model:\n name: ignored/project-model\n").unwrap(); + + config.apply_file(file).unwrap(); + assert_eq!(config.model_source, "embedded default"); + config.apply_env().unwrap(); + + assert_eq!( + config.model, + provisional_hosted_profile().generator_chain[0] + ); + assert_eq!(config.model_source, "hosted provisional profile"); + drop(provisional); + drop(hosted); + } + + #[test] + fn explicit_provider_keys_do_not_inherit_stored_login_routing() { + let _lock = env_lock().lock().unwrap(); + let directory = tempfile::tempdir().unwrap(); + let xdg = EnvRestore::capture("XDG_CONFIG_HOME"); + let saved_keys = crate::api_key::API_KEY_ENV_VARS.map(EnvRestore::capture); + EnvRestore::set("XDG_CONFIG_HOME", directory.path().to_str().unwrap()); + let path = credentials::default_path().unwrap(); + credentials::write( + &path, + &credentials::Credentials { + version: credentials::CREDENTIALS_VERSION, + issuer: Some("https://postil.dev".into()), + token: "pcli_fixture-not-a-real-secret".into(), + expires_at: "2999-01-01T00:00:00.000Z".into(), + refresh_token: None, + refresh_expires_at: None, + api_base: "https://postil.dev/api/inference/v1".into(), + org: "fixture".into(), + model: "hosted/model".into(), + pending_revocations: Vec::new(), + }, + ) + .unwrap(); + for name in crate::api_key::API_KEY_ENV_VARS { + for other in crate::api_key::API_KEY_ENV_VARS { + unsafe { std::env::remove_var(other) }; + } + EnvRestore::set(name, "provider-fixture-key"); + let mut cfg = Config::default(); + assert!( + !cfg.apply_stored_login_credential(), + "{name} must suppress login routing" + ); + assert_eq!(cfg.api_base, model_defaults().api_base, "{name}"); + assert_eq!(cfg.model, model_defaults().default_model, "{name}"); + } + drop(saved_keys); + drop(xdg); + } + + #[test] + fn project_model_provenance_replaces_stored_login_provenance() { + let _lock = env_lock().lock().unwrap(); + let directory = tempfile::tempdir().unwrap(); + let xdg = EnvRestore::capture("XDG_CONFIG_HOME"); + let saved_keys = crate::api_key::API_KEY_ENV_VARS.map(EnvRestore::capture); + EnvRestore::set("XDG_CONFIG_HOME", directory.path().to_str().unwrap()); + for name in crate::api_key::API_KEY_ENV_VARS { + unsafe { std::env::remove_var(name) }; + } + credentials::write( + &credentials::default_path().unwrap(), + &credentials::Credentials { + version: credentials::CREDENTIALS_VERSION, + issuer: Some("https://postil.dev".into()), + token: "pcli_fixture-not-a-real-secret".into(), + expires_at: "2999-01-01T00:00:00.000Z".into(), + refresh_token: None, + refresh_expires_at: None, + api_base: "https://postil.dev/api/inference/v1".into(), + org: "fixture".into(), + model: "hosted/model".into(), + pending_revocations: Vec::new(), + }, + ) + .unwrap(); + let mut config = Config::default(); + + assert!(config.apply_stored_login_credential()); + assert_eq!(config.model_source, "stored login"); + let file: FileConfig = yaml_serde::from_str("model:\n name: project/model\n").unwrap(); + config.apply_file(file).unwrap(); + assert_eq!(config.model, "project/model"); + assert_eq!(config.model_source, "trusted project config"); + + drop(saved_keys); + drop(xdg); + } + #[test] fn postil_yaml_wins_over_coderabbit() { let dir = tempfile::tempdir().unwrap(); @@ -2638,13 +3055,17 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid let mut c = Config::default(); c.apply_file(f).unwrap(); assert_eq!(c.max_findings, 20); - assert!(c.model.is_empty()); - assert!(c.cascade.is_empty()); + assert_eq!(c.model, model_defaults().default_model); + assert_eq!(c.cascade, model_defaults().cascade); + assert_eq!(c.reasoning_effort, ReasoningEffort::Low); + assert_eq!(c.scorer_reasoning_effort, ReasoningEffort::None); assert!(c.scorer.is_empty()); } #[test] fn model_scorer_parses_from_postil_config() { + let _lock = env_lock().lock().unwrap(); + let _local_environment = LocalModelConfigEnv::clear(); let f: FileConfig = yaml_serde::from_str("model:\n scorer: custom/scorer\n").unwrap(); let mut c = Config::default(); c.apply_file(f).unwrap(); @@ -2656,7 +3077,7 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid #[test] fn trusted_runtime_can_ignore_the_complete_repository_model_section() { let f: FileConfig = yaml_serde::from_str( - "model:\n name: anthropic/claude-opus-4.1\n cascade:\n - attacker/fallback\n scorer: anthropic/claude-haiku-4.5\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", + "model:\n name: anthropic/claude-opus-4.1\n reasoningEffort: max\n cascade:\n - attacker/fallback\n scorer: anthropic/claude-haiku-4.5\n scorerReasoningEffort: high\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", ) .unwrap(); let mut config = Config::default(); @@ -2665,7 +3086,20 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid config.apply_file_inner(f, true, true).unwrap(); assert_eq!(config.model_chain(), expected.model_chain()); + assert_eq!(config.reasoning_effort, expected.reasoning_effort); + assert_eq!( + config.reasoning_effort_source, + expected.reasoning_effort_source + ); assert_eq!(config.scorer, expected.scorer); + assert_eq!( + config.scorer_reasoning_effort, + expected.scorer_reasoning_effort + ); + assert_eq!( + config.scorer_reasoning_effort_source, + expected.scorer_reasoning_effort_source + ); assert_eq!(config.scorer_chain(), expected.scorer_chain()); assert_eq!(config.api_base, DEFAULT_API_BASE); assert_eq!(config.api_format, ApiFormat::OpenaiCompatible); @@ -3119,12 +3553,14 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid version: 1, source_sha256: "a".repeat(64), default_model: "provider/generator".into(), + reasoning_effort: ReasoningEffort::Low, cascade: vec!["provider/fallback".into()], consensus: 1, api_base: "https://models.example/v1".into(), api_format: ApiFormat::OpenaiCompatible, scorer_enabled: true, scorer_model: "provider/scorer".into(), + scorer_reasoning_effort: ReasoningEffort::None, scorer_fallback: "provider/scorer-fallback".into(), scorer_qualification_candidates: Vec::new(), }; @@ -3230,6 +3666,8 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid #[test] fn native_anthropic_has_no_implicit_scorer_and_honors_explicit_byok_models() { + let _lock = env_lock().lock().unwrap(); + let _local_environment = LocalModelConfigEnv::clear(); let mut config = Config { api_format: ApiFormat::Anthropic, ..Config::default() @@ -3286,6 +3724,12 @@ scorer = { enabled = true, default_model = "provider/scorer", fallback = "provid assert_eq!(profile.upstream_provider_identity, "Azure"); assert_eq!(profile.generator_chain, vec!["openai/gpt-5.6-luna"]); assert_eq!(profile.scorer_chain, vec!["openai/gpt-5.6-luna"]); + let drifted = Config { + reasoning_effort: ReasoningEffort::High, + ..config.clone() + }; + assert!(provisional_hosted_profile_for_config(&drifted).is_none()); + assert!(drifted.require_model_for(true, true).is_err()); assert_eq!( profile.model_price_bounds, vec![ModelPriceBound { diff --git a/src/credentials.rs b/src/credentials.rs index a8d7414..4c26abf 100644 --- a/src/credentials.rs +++ b/src/credentials.rs @@ -5,7 +5,7 @@ //! developer secret bound to one machine, not project configuration. The //! bearer token inside spends an organization's hosted-inference //! entitlement, so both the containing directory (0700) and the file itself -//! (0600) get their final permission bits set at creation time -- never +//! (0600) get their final permission bits set at creation time, never //! `chmod`'d afterward, so there is no window where the token is briefly //! world-readable. @@ -19,17 +19,33 @@ use anyhow::{Context, Result}; use fs2::FileExt; use serde::{Deserialize, Serialize}; -pub const CREDENTIALS_VERSION: u32 = 2; +pub const CREDENTIALS_VERSION: u32 = 3; pub const LEGACY_CREDENTIALS_VERSION: u32 = 1; +pub const LEGACY_REFRESH_CREDENTIALS_VERSION: u32 = 2; +const PENDING_REVOCATIONS_VERSION: u32 = 1; // The refresh exchange has separately bounded send and body-read phases. A // second local process waits long enough for both before asking the caller to // retry, so concurrent agent runs converge on one token rotation. const CREDENTIAL_LOCK_WAIT: std::time::Duration = std::time::Duration::from_secs(45); const CREDENTIAL_LOCK_RETRY: std::time::Duration = std::time::Duration::from_millis(25); +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct PendingRevocation { + pub issuer: String, + pub token: String, + #[serde( + default, + rename = "refreshToken", + skip_serializing_if = "Option::is_none" + )] + pub refresh_token: Option, +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct Credentials { pub version: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub issuer: Option, pub token: String, #[serde(rename = "expiresAt")] pub expires_at: String, @@ -49,6 +65,18 @@ pub struct Credentials { pub api_base: String, pub org: String, pub model: String, + #[serde( + default, + rename = "pendingRevocations", + skip_serializing_if = "Vec::is_empty" + )] + pub pending_revocations: Vec, +} + +#[derive(Debug, Serialize, Deserialize)] +struct PendingRevocations { + version: u32, + revocations: Vec, } impl Credentials { @@ -118,11 +146,20 @@ pub fn read(path: &Path) -> Result> { anyhow::ensure!( matches!( credentials.version, - LEGACY_CREDENTIALS_VERSION | CREDENTIALS_VERSION + LEGACY_CREDENTIALS_VERSION | LEGACY_REFRESH_CREDENTIALS_VERSION | CREDENTIALS_VERSION ), "unsupported credentials version {}; run `postil login` again", credentials.version ); + anyhow::ensure!( + credentials.version != CREDENTIALS_VERSION + || credentials + .issuer + .as_deref() + .is_some_and(|issuer| !issuer.trim().is_empty()), + "credentials version {} is missing its issuer; run `postil login` again", + credentials.version + ); Ok(Some(credentials)) } @@ -130,11 +167,77 @@ pub fn read(path: &Path) -> Result> { /// bits already set, filled, `fsync`'d, then renamed into place, so a reader /// never observes a partially written or briefly world-readable file. pub fn write(path: &Path, credentials: &Credentials) -> Result<()> { - let parent = path + anyhow::ensure!( + matches!( + credentials.version, + LEGACY_CREDENTIALS_VERSION | LEGACY_REFRESH_CREDENTIALS_VERSION | CREDENTIALS_VERSION + ), + "unsupported credentials version {}", + credentials.version + ); + anyhow::ensure!( + credentials.version != CREDENTIALS_VERSION + || credentials + .issuer + .as_deref() + .is_some_and(|issuer| !issuer.trim().is_empty()), + "credentials version {} is missing its issuer", + credentials.version + ); + write_private_json(path, credentials, "credentials") +} + +pub fn read_pending(credentials_path: &Path) -> Result> { + let path = pending_path(credentials_path)?; + let raw = match fs::read_to_string(&path) { + Ok(raw) => raw, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(error) => { + return Err(error).with_context(|| format!("reading {}", path.display())); + } + }; + let pending: PendingRevocations = + serde_json::from_str(&raw).with_context(|| format!("parsing {}", path.display()))?; + anyhow::ensure!( + pending.version == PENDING_REVOCATIONS_VERSION, + "unsupported pending revocations version {}", + pending.version + ); + Ok(pending.revocations) +} + +pub fn write_pending(credentials_path: &Path, revocations: &[PendingRevocation]) -> Result<()> { + let path = pending_path(credentials_path)?; + if revocations.is_empty() { + return remove(&path); + } + write_private_json( + &path, + &PendingRevocations { + version: PENDING_REVOCATIONS_VERSION, + revocations: revocations.to_vec(), + }, + "pending revocations", + ) +} + +fn pending_path(credentials_path: &Path) -> Result { + let parent = credentials_path .parent() .context("credentials path must have a parent directory")?; + Ok(parent.join("pending-revocations.json")) +} + +fn write_private_json(path: &Path, value: &T, description: &str) -> Result<()> { + let parent = path + .parent() + .with_context(|| format!("{description} path must have a parent directory"))?; create_private_dir(parent)?; - let temp_path = parent.join(format!(".credentials.{}.tmp", std::process::id())); + let file_name = path + .file_name() + .and_then(|name| name.to_str()) + .with_context(|| format!("{description} path must have a file name"))?; + let temp_path = parent.join(format!(".{file_name}.{}.tmp", std::process::id())); let _ = fs::remove_file(&temp_path); let mut options = OpenOptions::new(); options.write(true).create_new(true); @@ -143,19 +246,15 @@ pub fn write(path: &Path, credentials: &Credentials) -> Result<()> { let result = (|| -> Result<()> { let mut file = options .open(&temp_path) - .context("creating private credentials file")?; - anyhow::ensure!( - matches!( - credentials.version, - LEGACY_CREDENTIALS_VERSION | CREDENTIALS_VERSION - ), - "unsupported credentials version {}", - credentials.version - ); - serde_json::to_writer_pretty(&mut file, credentials).context("serializing credentials")?; - file.write_all(b"\n").context("writing credentials")?; - file.sync_all().context("syncing credentials")?; - fs::rename(&temp_path, path).context("installing credentials file")?; + .with_context(|| format!("creating private {description} file"))?; + serde_json::to_writer_pretty(&mut file, value) + .with_context(|| format!("serializing {description}"))?; + file.write_all(b"\n") + .with_context(|| format!("writing {description}"))?; + file.sync_all() + .with_context(|| format!("syncing {description}"))?; + fs::rename(&temp_path, path).with_context(|| format!("installing {description} file"))?; + sync_parent_dir(parent).with_context(|| format!("syncing {description} directory"))?; Ok(()) })(); if result.is_err() { @@ -164,6 +263,17 @@ pub fn write(path: &Path, credentials: &Credentials) -> Result<()> { result } +#[cfg(unix)] +fn sync_parent_dir(parent: &Path) -> Result<()> { + File::open(parent)?.sync_all()?; + Ok(()) +} + +#[cfg(not(unix))] +fn sync_parent_dir(_parent: &Path) -> Result<()> { + Ok(()) +} + /// A kernel-backed, cross-process lock shared by every stored-login mutation. /// Keeping the lock file in place avoids stale-file recovery: the operating /// system releases the exclusive lock when the handle closes or its process @@ -218,7 +328,6 @@ fn open_private_lock_file(path: &Path) -> Result { } Ok(file) } - /// Idempotent: removing an already-absent file is success, matching the /// server-side logout endpoint's own idempotence. pub fn remove(path: &Path) -> Result<()> { @@ -250,6 +359,7 @@ mod tests { fn sample(expires_at: &str) -> Credentials { Credentials { version: CREDENTIALS_VERSION, + issuer: Some("https://postil.dev".to_string()), token: "pcli_test-token-not-a-real-secret".to_string(), expires_at: expires_at.to_string(), refresh_token: Some("fixture-refresh-not-a-credential".to_string()), @@ -257,6 +367,7 @@ mod tests { api_base: "https://postil.dev/api/inference/v1".to_string(), org: "runatlas-is".to_string(), model: "z-ai/glm-5.2".to_string(), + pending_revocations: Vec::new(), } } @@ -337,7 +448,6 @@ mod tests { fn unparsable_expiry_fails_closed_as_expired() { assert!(sample("not-a-timestamp").is_expired()); } - #[test] fn reads_a_v1_access_only_credential() { let dir = tempfile::tempdir().unwrap(); @@ -349,8 +459,42 @@ mod tests { .unwrap(); let credentials = read(&path).unwrap().unwrap(); assert_eq!(credentials.version, 1); + assert!(credentials.issuer.is_none()); assert!(credentials.refresh_token.is_none()); assert!(credentials.refresh_expires_at.is_none()); + assert!(credentials.pending_revocations.is_empty()); + } + + #[test] + fn reads_a_v2_refresh_credential_without_an_issuer() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("credentials.json"); + fs::write( + &path, + r#"{"version":2,"token":"pcli_test-token-not-a-real-secret","expiresAt":"2999-01-01T00:00:00.000Z","refreshToken":"fixture-refresh-not-a-credential","refreshExpiresAt":"2999-12-01T00:00:00.000Z","apiBase":"https://postil.dev/api/inference/v1","org":"runatlas-is","model":"z-ai/glm-5.2"}"#, + ) + .unwrap(); + let credentials = read(&path).unwrap().unwrap(); + assert_eq!(credentials.version, LEGACY_REFRESH_CREDENTIALS_VERSION); + assert!(credentials.issuer.is_none()); + assert!(credentials.can_refresh()); + } + + #[test] + fn rejects_a_current_refresh_credential_without_an_issuer() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("credentials.json"); + fs::write( + &path, + format!( + r#"{{"version":{CREDENTIALS_VERSION},"token":"pcli_test-token-not-a-real-secret","expiresAt":"2999-01-01T00:00:00.000Z","refreshToken":"fixture-refresh-not-a-credential","refreshExpiresAt":"2999-12-01T00:00:00.000Z","apiBase":"https://postil.dev/api/inference/v1","org":"runatlas-is","model":"z-ai/glm-5.2"}}"# + ), + ) + .unwrap(); + + let error = read(&path).expect_err("current credentials must carry their issuer"); + assert!(error.to_string().contains("missing its issuer")); + assert!(error.to_string().contains("postil login")); } #[test] @@ -359,12 +503,14 @@ mod tests { let path = dir.path().join("postil").join("credentials.json"); let mut credentials = sample("2999-01-01T00:00:00.000Z"); credentials.version = 1; + credentials.issuer = None; credentials.refresh_token = None; credentials.refresh_expires_at = None; write(&path, &credentials).unwrap(); let raw = fs::read_to_string(path).unwrap(); assert!(raw.contains("\"version\": 1")); assert!(!raw.contains("refreshToken")); + assert!(!raw.contains("issuer")); } #[test] @@ -402,4 +548,39 @@ mod tests { "stored login lock file must be mode 0600, got {mode:o}" ); } + + #[test] + #[cfg(unix)] + fn pending_revocations_are_private_and_atomically_replaced() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let first = PendingRevocation { + issuer: "https://postil.dev".to_string(), + token: "pcli_first-access-not-a-real-secret".to_string(), + refresh_token: Some("fixture-first-refresh-not-a-credential".to_string()), + }; + let replacement = PendingRevocation { + issuer: "https://login.example.test".to_string(), + token: "pcli_second-access-not-a-real-secret".to_string(), + refresh_token: None, + }; + + write_pending(&credentials_path, std::slice::from_ref(&first)).unwrap(); + write_pending(&credentials_path, std::slice::from_ref(&replacement)).unwrap(); + + assert_eq!(read_pending(&credentials_path).unwrap(), vec![replacement]); + let pending_path = credentials_path + .parent() + .unwrap() + .join("pending-revocations.json"); + let mode = fs::metadata(&pending_path).unwrap().permissions().mode() & 0o777; + assert_eq!( + mode, 0o600, + "pending revocations file must be mode 0600, got {mode:o}" + ); + let temp_name = format!(".pending-revocations.json.{}.tmp", std::process::id()); + assert!(!pending_path.parent().unwrap().join(temp_name).exists()); + } } diff --git a/src/diff.rs b/src/diff.rs index e63c097..2f28dfe 100644 --- a/src/diff.rs +++ b/src/diff.rs @@ -1987,57 +1987,6 @@ fn hosted_risk_tokens(line: &str) -> Vec { tokens } -/// Build one bounded, line-grounded context for an interactive answer. Every -/// source window contributes semantic evidence; large inputs are reduced with -/// the same bounded fan-in used by review synthesis rather than byte truncation. -pub fn bounded_respond_context( - snapshot: &DiffSnapshot, - max_bytes: usize, - max_manifest_bytes: usize, -) -> Result<(String, bool)> { - let mut prepared = prepare_review(snapshot)?; - let reserved_anchor = prepared.reserved_anchor; - let mut batches = spool_model_batches(&mut prepared, max_bytes, max_manifest_bytes, false)?; - if batches.count == 0 { - return Ok((String::new(), reserved_anchor)); - } - let mut first: Option = None; - let mut digests = Vec::with_capacity(batches.count.min(1_024)); - while let Some(batch) = batches.next_batch()? { - if let Some(previous) = first.take() { - if digests.is_empty() { - digests.push(semantic_digest(&previous)); - } - digests.push(semantic_digest(&batch)); - } else if digests.is_empty() { - first = Some(batch); - } else { - digests.push(semantic_digest(&batch)); - } - } - if digests.is_empty() { - return Ok((first.expect("one batch was counted"), reserved_anchor)); - } - let mut level = 1usize; - loop { - let chunks = pack_semantic_digests(&digests, level, max_bytes)?; - if chunks.len() == 1 { - return Ok(( - chunks.into_iter().next().expect("checked above"), - reserved_anchor, - )); - } - anyhow::ensure!( - chunks.len() < digests.len(), - "interactive context synthesis did not reduce its bounded fan-in" - ); - digests = chunks; - level = level - .checked_add(1) - .context("interactive synthesis level overflowed")?; - } -} - pub fn spool_model_batches( prepared: &mut PreparedReview, max_batch_bytes: usize, @@ -6964,36 +6913,6 @@ diff --git a/src/multi.rs b/src/multi.rs assert!(first < second && second < third); } - #[test] - fn interactive_context_streams_distant_evidence_without_byte_truncation() { - use std::fmt::Write as _; - - let mut source = String::from( - "diff --git a/src/flow.rs b/src/flow.rs\n--- a/src/flow.rs\n+++ b/src/flow.rs\n@@ -0,0 +1,400 @@\n", - ); - for line in 1..=400 { - if line == 1 { - source.push_str("+let checked = validate_pair(input);\n"); - } else if line == 400 { - source.push_str("+dangerous_sink(original);\n"); - } else { - writeln!( - source, - "+let padding_{line} = ordinary; // {}", - "x".repeat(1_000) - ) - .unwrap(); - } - } - let snapshot = DiffSnapshot::from_bytes(source.as_bytes()).unwrap(); - let (context, reserved) = bounded_respond_context(&snapshot, 120_000, 24_000).unwrap(); - assert!(!reserved); - assert!(context.len() <= 120_000); - assert!(context.contains("validate_pair(input)")); - assert!(context.contains("dangerous_sink(original)")); - assert!(!context.contains("diff truncated")); - } - #[test] fn malformed_and_unsupported_lockfiles_use_bounded_source_review() { let malformed = "diff --git a/Cargo.lock b/Cargo.lock\n--- a/Cargo.lock\n+++ b/Cargo.lock\n@@ -1,5 +1,5 @@\n-checksum = \"old-one\"\n-source = \"old-two\"\n-checksum = \"old-interior\"\n-source = \"old-four\"\n-checksum = \"old-five\"\n+checksum = \"new-one\"\n+source = \"new-two\"\n+checksum = \"new-interior\"\n+source = \"new-four\"\n+checksum = \"new-five\"\n"; diff --git a/src/doctor.rs b/src/doctor.rs index 915a74b..1477fe1 100644 --- a/src/doctor.rs +++ b/src/doctor.rs @@ -62,7 +62,7 @@ pub async fn run(cfg: &Config) -> Result> { Ok(Some(key)) } else { match credentials_path.as_ref() { - Ok(path) => login::resolve_stored_token(path).await, + Ok(path) => login::resolve_stored_token(path, &cfg.api_base).await, Err(error) => Err(anyhow::anyhow!("{error:#}")), } }; @@ -220,7 +220,6 @@ pub fn print_report(checks: &[Check]) -> bool { } all_ok } - #[cfg(test)] mod tests { use super::*; @@ -229,6 +228,7 @@ mod tests { fn stored_login() -> Credentials { Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some("https://postil.dev".to_string()), token: "pcli_test-access-not-a-real-secret".to_string(), expires_at: "2999-01-01T00:00:00.000Z".to_string(), refresh_token: Some("fixture-refresh-not-a-credential".to_string()), @@ -236,6 +236,7 @@ mod tests { api_base: "https://postil.dev/api/inference/v1".to_string(), org: "runatlas-is".to_string(), model: "z-ai/glm-5.2".to_string(), + pending_revocations: Vec::new(), } } diff --git a/src/envelope.rs b/src/envelope.rs index f76008a..061d808 100644 --- a/src/envelope.rs +++ b/src/envelope.rs @@ -1533,7 +1533,6 @@ pub enum ModelUsageRole { ReviewPlanner, ReviewGenerator, FindingScorer, - MentionResponder, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] @@ -1557,7 +1556,6 @@ pub enum ModelIncidentPhase { Planner, Review, Scorer, - Respond, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] diff --git a/src/forge/azure.rs b/src/forge/azure.rs index eb973eb..8aa2cee 100644 --- a/src/forge/azure.rs +++ b/src/forge/azure.rs @@ -17,8 +17,8 @@ use serde_json::json; use std::io::Write; use super::{ - CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, ThreadKind, check_summary, - check_title, untracked_review_publication_receipt, + CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, check_summary, check_title, + untracked_review_publication_receipt, }; use crate::diff::{DiffSnapshot, DiffSpool, WorkspaceBudget}; use crate::envelope::{Envelope, Finding}; @@ -473,7 +473,9 @@ impl Forge for Azure { return Ok(receipt); } if !self.snapshot_is_current(snapshot).await? { - eprintln!("postil: azure review delivery skipped because the pull request changed"); + crate::progress::notice(format_args!( + "postil: azure review delivery skipped because the pull request changed" + )); return Ok(receipt); } let summary = self.review_summary(envelope); @@ -536,7 +538,9 @@ impl Forge for Azure { _annotate_findings: bool, ) -> Result<()> { if !self.snapshot_is_current(snapshot).await? { - eprintln!("postil: azure status delivery skipped because the pull request changed"); + crate::progress::notice(format_args!( + "postil: azure status delivery skipped because the pull request changed" + )); return Ok(()); } let head = envelope.head_sha.clone().unwrap_or_default(); @@ -585,47 +589,6 @@ impl Forge for Azure { .await? == expected.base_sha) } - - /// Title and description of a PR. Work-item comments live under a different - /// `_apis/wit` base and api-version than the git endpoints this client uses, - /// so respond is scoped to pull requests. - async fn fetch_thread(&self, _number: u64, kind: ThreadKind) -> Result<(String, String)> { - // TODO(respond): Azure work-item comments - // (`_apis/wit/workItems/{id}/comments?api-version=7.0-preview.3`, body - // field `text`) use a different base/version; scope respond to PRs. - if kind == ThreadKind::Issue { - return Err(anyhow!( - "postil respond on Azure DevOps supports --pr only (work items not supported)" - )); - } - let pr = self.pr().await?; - Ok((pr.title, pr.description)) - } - - /// Post a top-level PR comment as a non-anchored thread (the bot's reply to - /// a mention), using the same threads endpoint the reviewer uses for its summary. - async fn post_comment(&self, number: u64, kind: ThreadKind, body: &str) -> Result<()> { - // TODO(respond): Azure work-item comments are unverified here; scope to PRs. - if kind == ThreadKind::Issue { - return Err(anyhow!( - "postil respond on Azure DevOps supports --pr only (work items not supported)" - )); - } - let resp = self - .request( - reqwest::Method::POST, - self.url(&format!("/pullRequests/{number}/threads"), ""), - ) - .json(&json!({ - "comments": [{ "parentCommentId": 0, "commentType": 1, "content": body }], - "status": 1, - })) - .send() - .await - .context("posting comment")?; - Self::check_ok(resp, "comment post").await?; - Ok(()) - } } /// Both merge commits, or an actionable error: Azure omits them on PRs with diff --git a/src/forge/bitbucket.rs b/src/forge/bitbucket.rs index 37c13b5..2b66c0e 100644 --- a/src/forge/bitbucket.rs +++ b/src/forge/bitbucket.rs @@ -21,8 +21,8 @@ use sha2::Digest; use std::io::Write; use super::{ - CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, ThreadKind, check_summary, - check_title, untracked_review_publication_receipt, + CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, check_summary, check_title, + untracked_review_publication_receipt, }; use crate::diff::{DiffSnapshot, DiffSpool, WorkspaceBudget}; use crate::envelope::{Envelope, Finding}; @@ -519,7 +519,9 @@ impl Forge for Bitbucket { return Ok(receipt); } if !self.snapshot_is_current(snapshot).await? { - eprintln!("postil: bitbucket review delivery skipped because the pull request changed"); + crate::progress::notice(format_args!( + "postil: bitbucket review delivery skipped because the pull request changed" + )); return Ok(receipt); } let summary = self.review_summary(envelope); @@ -584,7 +586,9 @@ impl Forge for Bitbucket { _annotate_findings: bool, ) -> Result<()> { if !self.snapshot_is_current(snapshot).await? { - eprintln!("postil: bitbucket status delivery skipped because the pull request changed"); + crate::progress::notice(format_args!( + "postil: bitbucket status delivery skipped because the pull request changed" + )); return Ok(()); } let head = envelope @@ -634,42 +638,6 @@ impl Forge for Bitbucket { .await?; Ok(merge_base == expected.base_sha) } - - /// Title and description of a PR. Bitbucket Cloud's issue tracker is a - /// separate, often-disabled product with a different object shape we cannot - /// verify against a live instance, so respond is scoped to pull requests. - async fn fetch_thread(&self, _number: u64, kind: ThreadKind) -> Result<(String, String)> { - // TODO(respond): Bitbucket issue-tracker comments - // (`/issues/{id}` + `/issues/{id}/comments`) are unverified; scope to PRs. - if kind == ThreadKind::Issue { - return Err(anyhow!( - "postil respond on Bitbucket supports --pr only (issue tracker not supported)" - )); - } - let pr = self.pr_meta().await?; - Ok((pr.title, pr.summary.map(|s| s.raw).unwrap_or_default())) - } - - /// Post a top-level comment on a PR (the bot's reply to a mention). - async fn post_comment(&self, number: u64, kind: ThreadKind, body: &str) -> Result<()> { - // TODO(respond): Bitbucket issue-tracker comments are unverified; scope to PRs. - if kind == ThreadKind::Issue { - return Err(anyhow!( - "postil respond on Bitbucket supports --pr only (issue tracker not supported)" - )); - } - let resp = self - .request( - reqwest::Method::POST, - self.url(&format!("/pullrequests/{number}/comments")), - ) - .json(&json!({ "content": { "raw": body } })) - .send() - .await - .context("posting comment")?; - Self::check_ok(resp, "comment post").await?; - Ok(()) - } } #[cfg(test)] diff --git a/src/forge/github.rs b/src/forge/github.rs index f7472c1..31344d5 100644 --- a/src/forge/github.rs +++ b/src/forge/github.rs @@ -26,8 +26,7 @@ use super::{ PublicationPlanReviewCreateOutcome, PublicationPlanReviewCreatePayload, PublicationPlanReviewSummaryCase, PublicationPlanSnapshot, PublicationPlanTerminalOperation, PublicationPlanTerminalOutcome, ReviewPublicationReceipt, ReviewPublicationSummary, - SummaryContext, ThreadKind, check_summary, check_title, only_operational_findings, - valid_details_url, + SummaryContext, check_summary, check_title, only_operational_findings, valid_details_url, }; use crate::diff::{Diff, DiffIndex, DiffSnapshot, DiffSpool, WorkspaceBudget}; use crate::envelope::{Envelope, Finding, Severity}; @@ -37,6 +36,18 @@ pub const EXPECTED_REPOSITORY_ID_ENV: &str = "POSTIL_EXPECTED_GITHUB_REPO_ID"; const GITHUB_MAX_ANNOTATIONS_PER_REQUEST: usize = 50; const FILE_LEVEL_COMMENT_MARKER: &str = ""; +macro_rules! github_telemetry { + ($($argument:tt)*) => { + crate::progress::telemetry(format_args!($($argument)*)) + }; +} + +macro_rules! github_notice { + ($($argument:tt)*) => { + crate::progress::notice(format_args!($($argument)*)) + }; +} + // Filtering caps visible findings before forge publication, so one completed // check update always fits GitHub's annotation request limit. Raising the // product cap requires implementing multi-request annotation delivery first. @@ -195,7 +206,7 @@ impl GitHub { .await .is_err() { - eprintln!( + github_notice!( "postil: github operation=review-summary-update status=incomplete recovery=truthful-initial-summary" ); } @@ -406,7 +417,7 @@ impl GitHub { let response = match response { Ok(response) => response, Err(error) => { - eprintln!( + github_telemetry!( "postil: github operation={} attempt={}/{} category=transport-error elapsed_ms={} budget_remaining_ms={}", what.replace(' ', "-"), attempt, @@ -432,7 +443,7 @@ impl GitHub { let status = response.status(); let request_id = github_request_id(response.headers()).unwrap_or_else(|| "none".to_string()); - eprintln!( + github_telemetry!( "postil: github operation={} attempt={}/{} status={} elapsed_ms={} request_id={} rate_remaining={} retry_after_secs={}", what.replace(' ', "-"), attempt, @@ -454,7 +465,7 @@ impl GitHub { if delay.is_zero() { return Ok(response); } - eprintln!( + github_telemetry!( "postil: github operation={} retrying_in_secs={} retry={}/{}", what.replace(' ', "-"), delay.as_secs(), @@ -726,48 +737,11 @@ impl GitHub { Ok(receipt) } - async fn comment_exists(&self, number: u64, marker: &str) -> Result { - const PAGE_SIZE: usize = 100; - const MAX_PAGES: usize = 20; - for page in 1..=MAX_PAGES { - let response = self - .send_retryable( - self.request( - reqwest::Method::GET, - self.url(&format!( - "/issues/{number}/comments?per_page={PAGE_SIZE}&page={page}&sort=created&direction=desc" - )), - ), - "comment reconciliation", - ) - .await?; - let comments: Vec = super::bounded_response_json( - Self::check_ok(response, "comment reconciliation").await?, - "GitHub comment reconciliation", - ) - .await?; - let page_len = comments.len(); - if comments - .into_iter() - .any(|comment| comment.body.contains(marker)) - { - return Ok(true); - } - if page_len < PAGE_SIZE { - return Ok(false); - } - } - Err(anyhow!( - "GitHub comment reconciliation exceeded {MAX_PAGES} pages; refusing an unsafe retry" - )) - } - /// Finding markers already carried by inline comments on this pull request. /// /// A finding's id is a hash over the head SHA, kind, path, line, and title, /// so an identical marker is the same finding at the same place on the same - /// commit. Two reviews of one head — a push review followed by an `@postil` - /// mention, say — legitimately re-detect the same issue, and without this + /// commit. Two reviews of one head legitimately re-detect the same issue, and without this /// the second run posts a second copy of every comment the first already /// left. /// @@ -920,47 +894,6 @@ impl GitHub { unreachable!("bounded GitHub file-level comment loop always returns") } - async fn post_comment_reconciled(&self, number: u64, body: &str, marker: &str) -> Result<()> { - const RETRIES: u32 = 2; - for retry in 0..=RETRIES { - self.verify_repository_identity_before_write().await?; - let response = self - .request( - reqwest::Method::POST, - self.url(&format!("/issues/{number}/comments")), - ) - .json(&json!({ "body": body })) - .send() - .await; - match response { - Ok(response) if response.status().is_success() => return Ok(()), - Ok(response) - if github_retryable_response(response.status(), response.headers()) => - { - if self.comment_exists(number, marker).await? { - return Ok(()); - } - if retry == RETRIES { - Self::check_ok(response, "comment post").await?; - } - } - Ok(response) => { - Self::check_ok(response, "comment post").await?; - } - Err(error) => { - if self.comment_exists(number, marker).await? { - return Ok(()); - } - if retry == RETRIES { - return Err(error).context("posting comment after reconciliation"); - } - } - } - tokio::time::sleep(github_transport_retry_delay(retry)).await; - } - unreachable!("bounded GitHub comment loop always returns") - } - async fn pull_files(&self, expected: usize) -> Result> { const PAGE_SIZE: usize = 100; const MAX_FILES: usize = 3_000; @@ -2007,11 +1940,6 @@ struct PublishedReviewComment { commit_id: Option, } -#[derive(Deserialize)] -struct PublishedComment { - body: String, -} - enum ReviewDelivery { Response(reqwest::Response), Reconciled(PublishedReview), @@ -3298,12 +3226,12 @@ impl Forge for GitHub { )); } - eprintln!( + github_telemetry!( "postil: github operation=review-post status=422 category=unresolved-line request_id={} recovery=placement-ladder", request_id, ); } else { - eprintln!( + github_telemetry!( "postil: github operation=review-reconciliation status=partial recovery=placement-ladder" ); } @@ -3428,7 +3356,7 @@ impl Forge for GitHub { && !line_findings.is_empty() && github_review_rejected_line(&body) { - eprintln!( + github_telemetry!( "postil: github operation=placement-fallback status=422 category=unresolved-line request_id={} recovery=file-comments", request_id, ); @@ -3688,32 +3616,6 @@ impl Forge for GitHub { } Ok(()) } - - /// Title and body of an issue or PR (the issues API covers both, so `kind` - /// is not needed here). - async fn fetch_thread(&self, number: u64, _kind: ThreadKind) -> Result<(String, String)> { - let resp = self - .send_retryable( - self.request(reqwest::Method::GET, self.url(&format!("/issues/{number}"))), - "issue fetch", - ) - .await?; - let v: serde_json::Value = super::bounded_response_json( - Self::check_ok(resp, "issue fetch").await?, - "GitHub issue", - ) - .await?; - let title = v["title"].as_str().unwrap_or_default().to_string(); - let body = v["body"].as_str().unwrap_or_default().to_string(); - Ok((title, body)) - } - - /// Post a top-level comment on an issue or PR (the bot's reply to a mention). - async fn post_comment(&self, number: u64, _kind: ThreadKind, body: &str) -> Result<()> { - let marker = comment_marker(number, body); - self.post_comment_reconciled(number, &append_marker(body, &marker), &marker) - .await - } } fn valid_object_id(value: &str) -> bool { @@ -4519,17 +4421,6 @@ fn required_review_body(body: &str, marker: &str) -> Result { Ok(marked) } -fn comment_marker(number: u64, body: &str) -> String { - let mut digest = Sha256::new(); - digest.update(number.to_be_bytes()); - digest.update(body.as_bytes()); - let hash = digest.finalize(); - format!( - "", - hash[0], hash[1], hash[2], hash[3], hash[4], hash[5] - ) -} - #[cfg(test)] mod tests { use super::{ @@ -4873,37 +4764,6 @@ mod tests { assert!(matches!(response, super::ReviewDelivery::Reconciled(_))); } - #[tokio::test] - async fn github_comment_reconciles_before_retrying_uncertain_post() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/repos/owner/repo/issues/9/comments")) - .respond_with(ResponseTemplate::new(500)) - .expect(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/owner/repo/issues/9/comments")) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!([{ - "body": "reply\n\n" - }])), - ) - .expect(1) - .mount(&server) - .await; - let github = test_github(&server); - - github - .post_comment_reconciled( - 9, - "reply\n\n", - "", - ) - .await - .unwrap(); - } - fn test_github(server: &MockServer) -> GitHub { GitHub { http: reqwest::Client::new(), @@ -7728,56 +7588,6 @@ mod tests { } } - #[tokio::test] - async fn github_repository_identity_fence_rejects_a_renamed_repository() { - let server = MockServer::start().await; - Mock::given(method("GET")) - .and(path("/repos/owner/repo")) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": 42, - "full_name": "owner/renamed" - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/owner/repo/issues/9/comments")) - .respond_with(ResponseTemplate::new(201)) - .expect(0) - .mount(&server) - .await; - - let error = fenced_test_github(&server, 42) - .post_comment_reconciled(9, "reply", "marker") - .await - .unwrap_err(); - assert!(crate::forge::is_repository_identity_failure(&error)); - } - - #[tokio::test] - async fn github_repository_identity_fence_rejects_a_fork_id_for_base_publication() { - let server = MockServer::start().await; - Mock::given(method("GET")) - .and(path("/repos/owner/repo")) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": 42, - "full_name": "owner/repo" - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/owner/repo/issues/9/comments")) - .respond_with(ResponseTemplate::new(201)) - .expect(0) - .mount(&server) - .await; - - let error = fenced_test_github(&server, 99) - .post_comment_reconciled(9, "reply", "marker") - .await - .unwrap_err(); - assert!(crate::forge::is_repository_identity_failure(&error)); - } - #[tokio::test] async fn github_repository_identity_fence_publishes_only_for_the_same_pr_snapshot() { let server = MockServer::start().await; diff --git a/src/forge/gitlab.rs b/src/forge/gitlab.rs index add8e71..bf907a0 100644 --- a/src/forge/gitlab.rs +++ b/src/forge/gitlab.rs @@ -10,8 +10,8 @@ use serde_json::json; use std::io::Write; use super::{ - CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, ThreadKind, check_summary, - check_title, untracked_review_publication_receipt, + CheckRunIds, CheckState, Forge, PrMeta, ReviewPublicationReceipt, check_summary, check_title, + untracked_review_publication_receipt, }; use crate::diff::{DiffSnapshot, DiffSpool, WorkspaceBudget}; use crate::envelope::{Envelope, Finding}; @@ -494,7 +494,9 @@ impl Forge for GitLab { } let mr = self.mr().await?; if !mr_matches_snapshot(&mr, snapshot) { - eprintln!("postil: gitlab review delivery skipped because the merge request changed"); + crate::progress::notice(format_args!( + "postil: gitlab review delivery skipped because the merge request changed" + )); return Ok(receipt); } let summary = self.review_summary(envelope); @@ -557,7 +559,9 @@ impl Forge for GitLab { ) -> Result<()> { let current = self.mr().await?; if !mr_matches_snapshot(¤t, snapshot) { - eprintln!("postil: gitlab status delivery skipped because the merge request changed"); + crate::progress::notice(format_args!( + "postil: gitlab status delivery skipped because the merge request changed" + )); return Ok(()); } let head = envelope @@ -596,52 +600,6 @@ impl Forge for GitLab { let current = self.mr().await?; Ok(mr_matches_snapshot(¤t, expected)) } - - /// Title and description of an issue or MR. GitLab's issue and merge-request - /// objects both expose `title` and `description`, so only the resource path - /// differs by `kind`. - async fn fetch_thread(&self, number: u64, kind: ThreadKind) -> Result<(String, String)> { - let resource = match kind { - ThreadKind::Pull => "merge_requests", - ThreadKind::Issue => "issues", - }; - let resp = self - .request( - reqwest::Method::GET, - self.url(&format!("/{resource}/{number}")), - ) - .send() - .await - .context("fetching thread")?; - let v: serde_json::Value = super::bounded_response_json( - Self::check_ok(resp, "thread fetch").await?, - "GitLab thread", - ) - .await?; - let title = v["title"].as_str().unwrap_or_default().to_string(); - let body = v["description"].as_str().unwrap_or_default().to_string(); - Ok((title, body)) - } - - /// Post a top-level note on an issue or MR (the bot's reply to a mention). - /// Both resources expose `/{resource}/{number}/notes` with a `body` field. - async fn post_comment(&self, number: u64, kind: ThreadKind, body: &str) -> Result<()> { - let resource = match kind { - ThreadKind::Pull => "merge_requests", - ThreadKind::Issue => "issues", - }; - let resp = self - .request( - reqwest::Method::POST, - self.url(&format!("/{resource}/{number}/notes")), - ) - .json(&json!({ "body": body })) - .send() - .await - .context("posting note")?; - Self::check_ok(resp, "note post").await?; - Ok(()) - } } #[cfg(test)] diff --git a/src/forge/mod.rs b/src/forge/mod.rs index a89848c..b8fe49c 100644 --- a/src/forge/mod.rs +++ b/src/forge/mod.rs @@ -409,17 +409,6 @@ pub struct CheckRunIds<'a> { pub gate: &'a str, } -/// What a `respond` thread number points at. GitHub's issues API covers both, -/// so it ignores this; GitLab/Bitbucket/Azure key issues and PRs on different -/// endpoints, so they branch on it. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ThreadKind { - /// A pull request / merge request. - Pull, - /// An issue / work item on the forge's issue tracker. - Issue, -} - /// Versioned result of one review publication attempt. Finding outcomes are /// keyed by the envelope's stable finding ID so the hosted service can join /// the immutable delivery result to later thread lifecycle observations. @@ -1509,15 +1498,6 @@ pub trait Forge { && current.base_sha == expected.base_sha && current.target_sha == expected.target_sha) } - - /// Title and body of the issue/PR/MR a maintainer mentioned Postil on, used - /// to ground the answer (`postil respond`). `kind` disambiguates the number - /// for forges whose issues and pulls live on different endpoints. - async fn fetch_thread(&self, number: u64, kind: ThreadKind) -> Result<(String, String)>; - - /// Post a top-level comment (Postil's reply to a mention). `kind` selects the - /// issue- vs pull-level endpoint where the forge separates them. - async fn post_comment(&self, number: u64, kind: ThreadKind, body: &str) -> Result<()>; } /// GitHub rejects a check-run `output.summary` over 65535 chars and a `title` diff --git a/src/lib.rs b/src/lib.rs index 93c4b7b..0ddb27a 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -21,9 +21,15 @@ pub mod login; pub(crate) mod machine_claim; pub mod output; pub mod plan; +pub(crate) mod progress; pub mod prompt; pub(crate) mod repository_search; pub(crate) mod resolve; -pub mod respond; pub mod review; pub mod sarif; + +#[cfg(test)] +pub(crate) fn test_env_lock() -> &'static std::sync::Mutex<()> { + static LOCK: std::sync::OnceLock> = std::sync::OnceLock::new(); + LOCK.get_or_init(|| std::sync::Mutex::new(())) +} diff --git a/src/llm.rs b/src/llm.rs index ae14d25..f53a038 100644 --- a/src/llm.rs +++ b/src/llm.rs @@ -19,9 +19,13 @@ use time::Date; use crate::adjudication::{AdjudicationResult, MAX_ADJUDICATION_OUTPUT_TOKENS}; use crate::api_key; + +macro_rules! eprintln { + ($($argument:tt)*) => { crate::progress::telemetry(format_args!($($argument)*)) }; +} use crate::config::{ ApiFormat, Config, HOSTED_ADMISSION_PROJECTION_CAP_MICROS, HOSTED_OPERATION_COST_CAP_MICROS, - ModelPriceBound, + ModelPriceBound, ReasoningEffort, }; use crate::envelope::{ Finding, Kind, ModelIncident, ModelIncidentCategory, ModelIncidentPhase, ModelIncidentRecovery, @@ -130,15 +134,6 @@ struct RawAtomicAttribution { reason: String, } -#[derive(Debug, Clone)] -pub struct Answer { - pub content: String, - pub model_used: String, - pub usage: Usage, - pub models: Vec, - pub usage_accounting_complete: bool, -} - #[derive(Debug)] pub struct ModelError { error: anyhow::Error, @@ -402,10 +397,6 @@ pub(crate) fn add_usage(total: &mut Usage, usage: Usage) { }; } -fn has_billable_usage(usage: Usage) -> bool { - usage.prompt_tokens > 0 || usage.completion_tokens > 0 || usage.provider_cost.is_some() -} - fn add_response_usage( usage: &mut Usage, prompt_tokens: u64, @@ -455,20 +446,6 @@ fn safe_model_error_category(error: &ModelError) -> &'static str { } } -fn safe_anyhow_category(error: &anyhow::Error) -> &'static str { - if error.downcast_ref::().is_some() { - "deadline" - } else if reqwest_error(error).is_some_and(reqwest::Error::is_timeout) - || error.downcast_ref::().is_some() - { - "timeout" - } else if error.downcast_ref::().is_some() { - "provider" - } else { - "invalid-output" - } -} - /// Raw shape we ask the model for. Findings are validated leniently here and /// strictly (grounding, ranges) by the caller. #[derive(Debug, Deserialize)] @@ -632,6 +609,8 @@ pub struct LlmClient { struct RequestDecorations { api_base: String, api_format: ApiFormat, + reasoning_effort: ReasoningEffort, + scorer_reasoning_effort: ReasoningEffort, require_openrouter_privacy: bool, hosted_price_bounds: Option>>, pinned_upstream_provider: Option, @@ -662,11 +641,14 @@ struct LlmTimeouts { /// Retries per model on transient provider errors before the cascade moves on. pub(crate) const TRANSIENT_RETRIES: u32 = 2; -/// A fresh request can recover when the caller's request timeout, rather than -/// the provider's response, ended an otherwise viable routed completion. The -/// shared total deadline remains authoritative, so this cannot extend a hosted -/// review beyond its worker budget. -const TIMEOUT_RETRIES: u32 = 1; +/// Maximum total transient retries available to a budgeted review. +/// The fixed allowance uses the first two, leaving ten budget-funded retries +/// and at most thirteen attempts for one logical review call. +pub(crate) const MAX_BUDGETED_TRANSIENT_RETRIES: u32 = 12; +const MIN_BUDGETED_RETRY_WAIT: Duration = Duration::from_secs(1); +const MAX_SYNTHESIZED_RETRY_BACKOFF: Duration = Duration::from_secs(20); +/// The minimum time reserved for the request funded by a budgeted wait. +const RETRY_ATTEMPT_RESERVE: Duration = Duration::from_secs(15); const EMPTY_RESPONSE_RETRIES: u32 = 1; const EXHAUSTED_OUTPUT_RETRIES: u32 = 1; const TIMEOUT_RETRY_CAP_SECS: u64 = 90; @@ -675,7 +657,6 @@ pub(crate) const REVIEW_MAX_OUTPUT_TOKENS: u32 = 16_000; const EXHAUSTED_OUTPUT_RETRY_MAX_TOKENS: u32 = REVIEW_MAX_OUTPUT_TOKENS; const MAX_REVIEW_RETRY_PREVIOUS_BYTES: usize = 16_384; const MAX_REVIEW_RETRY_REASON_BYTES: usize = 16_384; -const PROVIDER_RETRY_DELAY_CAP_SECS: u64 = 30; /// Unqualified models receive a bounded review budget. A larger bound belongs /// in explicit admitted-model metadata after that model proves it needs one. @@ -708,11 +689,13 @@ pub(crate) struct ReviewPlanSchedule<'a> { pub planner: Option<(&'a str, usize)>, pub batch_concurrency: usize, } -pub(crate) const MAX_PROVIDER_ATTEMPTS: usize = 216; pub(crate) const MAX_REPORTED_TOKEN_SPEND: usize = 20_000_000; const MAX_PROVIDER_INPUT_BYTES: usize = 64 * 1024 * 1024; pub(crate) const MAX_PROVIDER_REQUEST_BYTES: usize = 256 * 1024; -pub(crate) const MAX_PROVIDER_OUTPUT_TOKEN_EXPOSURE: usize = 2_000_000; +// Review admission includes the full thirteen-attempt transport path for each +// logical review call. Keep the output ceiling above the five-batch hosted +// plan while the aggregate token and cost caps remain authoritative. +pub(crate) const MAX_PROVIDER_OUTPUT_TOKEN_EXPOSURE: usize = 10_000_000; // Hosted planning reserves one initial and one correction call for every // selected model request plus the maximum enabled resolution and compression // passes. The selected-batch limit is derived after all model fan-out is known. @@ -722,8 +705,12 @@ const MAX_LOGICAL_CALLS_PER_SCORER_MODEL: usize = 2; const MAX_LOGICAL_CALLS_PER_RESOLUTION_MODEL: usize = 2; const MAX_LOGICAL_CALLS_PER_BREVITY_MODEL: usize = 1; const MAX_TRANSPORT_ATTEMPTS_PER_CALL: usize = TRANSIENT_RETRIES as usize + 1; +const MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL: usize = MAX_BUDGETED_TRANSIENT_RETRIES as usize + 1; +pub(crate) const MAX_PROVIDER_ATTEMPTS: usize = + MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG * MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL; const _: () = assert!( - MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG * MAX_TRANSPORT_ATTEMPTS_PER_CALL <= MAX_PROVIDER_ATTEMPTS + MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG * MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL + <= MAX_PROVIDER_ATTEMPTS ); const MAX_MODEL_RESPONSE_BYTES: usize = 512 * 1024; const SCORER_BASE_MAX_TOKENS: u32 = 256; @@ -731,10 +718,6 @@ const SCORER_MAX_TOKENS_PER_FINDING: u32 = 144; const SCORER_REPAIR_BYTES_PER_OUTPUT_TOKEN: usize = 4; pub(crate) const SCORER_MAX_FINDINGS: usize = 20; const REPAIR_ERROR_MAX_BYTES: usize = 1_024; -// The publication contract targets 1,200 characters and hard-stops at 2,400. -// Keep generation bounded too, so an invalid model cannot spend an article's -// worth of output tokens before the validator rejects it. -const RESPOND_MAX_TOKENS: u32 = 1024; const PLANNER_MAX_TOKENS: u32 = 1024; fn planned_review_models(cfg: &Config) -> Vec { if cfg.consensus > 1 { @@ -1298,6 +1281,16 @@ fn base_request_body( impl RequestDecorations { fn from_config(cfg: &Config) -> Result { + if cfg.api_format == ApiFormat::Anthropic { + ensure!( + cfg.reasoning_effort != ReasoningEffort::Minimal, + "model.reasoningEffort minimal is unsupported by the Anthropic request format; accepted Anthropic values: max|xhigh|high|medium|low|none" + ); + ensure!( + cfg.scorer_reasoning_effort != ReasoningEffort::Minimal, + "model.scorerReasoningEffort minimal is unsupported by the Anthropic request format; accepted Anthropic values: max|xhigh|high|medium|low|none" + ); + } let screening_profile = crate::config::benchmark_screening_profile_for_config(cfg)?; let hosted_price_bounds = if crate::config::hosted_runtime_mode() { ensure_hosted_provider_contract(cfg.api_format, &cfg.api_base)?; @@ -1347,6 +1340,8 @@ impl RequestDecorations { Ok(Self { api_base: cfg.api_base.trim_end_matches('/').to_string(), api_format: cfg.api_format, + reasoning_effort: cfg.reasoning_effort, + scorer_reasoning_effort: cfg.scorer_reasoning_effort, require_openrouter_privacy: is_canonical_openrouter_base(&cfg.api_base) && (crate::config::hosted_runtime_mode() || crate::config::benchmark_screening_mode() @@ -1377,6 +1372,10 @@ impl RequestDecorations { max_tokens, temperature, ); + apply_openrouter_reasoning( + &mut body, + phase.reasoning_effort(self.reasoning_effort, self.scorer_reasoning_effort), + ); apply_openrouter_privacy(&mut body, self.require_openrouter_privacy); let canonical_openrouter = is_canonical_openrouter_base(&self.api_base); if let Some(provider) = self.pinned_upstream_provider.as_deref() { @@ -1399,14 +1398,21 @@ impl RequestDecorations { } body } - ApiFormat::Anthropic => base_request_body( - self.api_format, - model, - system, - user, - max_tokens, - temperature, - ), + ApiFormat::Anthropic => { + let mut body = base_request_body( + self.api_format, + model, + system, + user, + max_tokens, + temperature, + ); + apply_anthropic_reasoning( + &mut body, + phase.reasoning_effort(self.reasoning_effort, self.scorer_reasoning_effort), + ); + body + } } } } @@ -1525,12 +1531,65 @@ fn provider_retry_delay(retry: u32) -> Duration { } fn provider_retry_delay_with_sample(retry: u32, sample: u64) -> Duration { - let ceiling_ms = 2_000_u64.saturating_mul(u64::from(retry.max(1))); + let ceiling_ms = if retry <= TRANSIENT_RETRIES { + 2_000_u64.saturating_mul(u64::from(retry.max(1))) + } else { + let doublings = retry - TRANSIENT_RETRIES; + 2_000_u64 + .saturating_mul(u64::from(TRANSIENT_RETRIES)) + .saturating_mul(1_u64 << doublings.min(6)) + } + .min(MAX_SYNTHESIZED_RETRY_BACKOFF.as_millis() as u64); let floor_ms = ceiling_ms / 2; let jitter_ms = sample % (ceiling_ms - floor_ms + 1); Duration::from_millis(floor_ms + jitter_ms) } +fn budgeted_review_retry(phase: LlmPhase) -> bool { + matches!(phase, LlmPhase::Review) +} + +fn max_transport_attempts(phase: LlmPhase) -> u32 { + if budgeted_review_retry(phase) { + MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u32 + } else { + MAX_TRANSPORT_ATTEMPTS_PER_CALL as u32 + } +} + +fn budgeted_retry_wait(retry_after: Option, retry: u32) -> Duration { + retry_after + .unwrap_or_else(|| provider_retry_delay(retry)) + .max(MIN_BUDGETED_RETRY_WAIT) +} + +fn transient_retry_wait(retry_after: Option, phase: LlmPhase, retries: u32) -> Duration { + if budgeted_review_retry(phase) && retries >= TRANSIENT_RETRIES { + budgeted_retry_wait(retry_after, retries + 1) + } else { + retry_after.unwrap_or_else(|| provider_retry_delay(retries + 1)) + } +} + +/// Whether the next transient retry can be funded without consuming the time +/// needed to issue the request it is intended to recover. +fn transient_retry_affordable( + phase: LlmPhase, + retries: u32, + wait: Duration, + remaining: Option, +) -> bool { + if retries < TRANSIENT_RETRIES { + return true; + } + budgeted_review_retry(phase) + && retries < MAX_BUDGETED_TRANSIENT_RETRIES + && remaining.is_some_and(|left| { + wait.checked_add(RETRY_ATTEMPT_RESERVE) + .is_some_and(|required| required <= left) + }) +} + fn timeout_status(status: u16) -> bool { matches!(status, 408 | 504) } @@ -1553,7 +1612,6 @@ enum LlmPhase { Adjudication, #[cfg_attr(not(feature = "qualification-candidate"), allow(dead_code))] Attribution, - Respond, #[cfg_attr(not(test), allow(dead_code))] Total, } @@ -1581,7 +1639,6 @@ impl LlmPhase { Self::Scorer { .. } => "scorer", Self::Adjudication => "finding-adjudication", Self::Attribution => "attribution", - Self::Respond => "respond", Self::Total => "total", } } @@ -1595,7 +1652,19 @@ impl LlmPhase { Self::Scorer { .. } => ModelUsageRole::FindingScorer, Self::Adjudication => ModelUsageRole::FindingScorer, Self::Attribution => ModelUsageRole::FindingScorer, - Self::Respond => ModelUsageRole::MentionResponder, + } + } + + fn reasoning_effort( + self, + reviewer: ReasoningEffort, + scorer: ReasoningEffort, + ) -> ReasoningEffort { + match self { + Self::Planner | Self::Review | Self::Resolution | Self::Brevity | Self::Total => { + reviewer + } + Self::Scorer { .. } | Self::Adjudication | Self::Attribution => scorer, } } @@ -1737,7 +1806,6 @@ impl std::fmt::Display for DeadlineExceeded { | LlmPhase::Scorer { .. } | LlmPhase::Adjudication | LlmPhase::Attribution - | LlmPhase::Respond | LlmPhase::Total => f.write_str("LLM total deadline exceeded"), } } @@ -1829,6 +1897,7 @@ impl LlmClient { temperature: f64, phase: LlmPhase, ) -> Result { + let max_attempts = max_transport_attempts(phase) as usize; let initial_body = self.request_body(model, system, user, initial_max_tokens, temperature, phase); let initial_bytes = serialized_provider_request_bytes( @@ -1842,10 +1911,7 @@ impl LlmClient { let expanded_tokens = phase.exhausted_output_retry_max_tokens(initial_max_tokens); if expanded_tokens == initial_max_tokens { return Ok(PlannedRequestPath { - attempts: vec![ - (initial_bytes, initial_max_tokens as usize); - MAX_TRANSPORT_ATTEMPTS_PER_CALL - ], + attempts: vec![(initial_bytes, initial_max_tokens as usize); max_attempts], }); } let expanded_body = @@ -1861,11 +1927,11 @@ impl LlmClient { expanded_bytes >= initial_bytes, "expanded output request serialized smaller than its initial request" ); - let mut attempts = Vec::with_capacity(MAX_TRANSPORT_ATTEMPTS_PER_CALL); + let mut attempts = Vec::with_capacity(max_attempts); attempts.push((initial_bytes, initial_max_tokens as usize)); attempts.extend(std::iter::repeat_n( (expanded_bytes, expanded_tokens as usize), - MAX_TRANSPORT_ATTEMPTS_PER_CALL.saturating_sub(1), + max_attempts.saturating_sub(1), )); Ok(PlannedRequestPath { attempts }) } @@ -2191,35 +2257,6 @@ impl LlmClient { ReviewAdmission::try_from(exposure) } - pub(crate) fn preflight_respond_plan( - &self, - cfg: &Config, - system: &str, - user: &str, - ) -> Result { - let Some(bounds) = &self.request_decorations.hosted_price_bounds else { - return Ok(ReviewAdmission::default()); - }; - let models = cfg.model_chain(); - ensure!(!models.is_empty(), "hosted respond has no admitted model"); - let mut exposure = PlannedExposure::default(); - for model in models { - let price = bounds.get(&model).ok_or_else(|| { - anyhow!("hosted respond model {model:?} has no admitted price bound") - })?; - let path = self.planned_request_path( - &model, - system, - user, - RESPOND_MAX_TOKENS, - 0.1, - LlmPhase::Respond, - )?; - exposure.add_path(&path, price)?; - } - self.validate_hosted_exposure("respond", &exposure) - } - pub async fn plan_review_batches( &self, cfg: &Config, @@ -2444,9 +2481,10 @@ impl LlmClient { logical_calls <= MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG, "complete hosted review needs {logical_calls} logical model calls, exceeding the {MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG}-call watchdog plan" ); - // Admission includes every executable logical call and the exact - // three-attempt transport path. Output expansion replaces a retry; it - // does not add a fourth request. + // Admission includes every executable logical call and its exact + // phase-specific transport path. Review calls can use thirteen + // attempts; non-Review paths use three. Output expansion replaces a + // retry; it does not add another request. let mut exposure = PlannedExposure::default(); for model in &review_models { let price = bounds @@ -2605,7 +2643,7 @@ impl LlmClient { /// Local and interactive clients have no built-in total deadline. They only /// get one when POSTIL_LLM_TOTAL_TIMEOUT_SECS is explicitly set. pub async fn from_env(cfg: &Config) -> Result { - let api_key = resolve_api_key().await?; + let api_key = resolve_api_key(&cfg.api_base).await?; let timeouts = LlmTimeouts::from_env(DEFAULT_REQUEST_TIMEOUT_SECS, None)?; let total_deadline = timeouts.total.map(|duration| Instant::now() + duration); Self::build( @@ -2624,7 +2662,7 @@ impl LlmClient { default_review_timeout: Duration, default_total_timeout: Duration, ) -> Result { - let api_key = resolve_api_key().await?; + let api_key = resolve_api_key(&cfg.api_base).await?; let timeouts = LlmTimeouts::from_env( default_request_timeout.as_secs(), Some(default_total_timeout.as_secs()), @@ -3012,102 +3050,6 @@ impl LlmClient { } } - /// Interactive answer validated by the caller's publication contract. - /// Invalid model content consumes and preserves its usage before the next - /// configured model is tried. - pub async fn answer( - &self, - cfg: &Config, - system: &str, - user: &str, - validate: F, - ) -> Result - where - F: Fn(&str) -> Result, - { - let mut usage = Usage::default(); - let mut models = Vec::new(); - let mut last_err = None; - let mut usage_accounting_complete = true; - let chain = cfg.model_chain(); - let chain_len = chain.len(); - for (index, model) in chain.into_iter().enumerate() { - let mut model_usage = Usage::default(); - let mut call_usage = Vec::new(); - let mut model_accounting_complete = true; - match self - .chat( - &model, - system, - user, - &mut model_usage, - &mut call_usage, - &mut model_accounting_complete, - RESPOND_MAX_TOKENS, - LlmPhase::Respond, - LlmCallPhase::Initial, - ) - .await - { - Ok(content) => { - usage_accounting_complete &= model_accounting_complete; - add_usage(&mut usage, model_usage); - models.append(&mut call_usage); - match validate(&content) { - Ok(content) => { - return Ok(Answer { - content, - model_used: model, - usage, - models, - usage_accounting_complete, - }); - } - Err(error) => { - let disposition = if index + 1 < chain_len { - "trying the next model" - } else { - "no fallback models remain" - }; - eprintln!( - "postil: model {} produced an invalid reply; {disposition} category={}", - log_text(&model), - safe_anyhow_category(&error), - ); - last_err = Some(error.context("model reply failed publication checks")); - } - } - } - Err(e) => { - // Usage parsed from a provider response is complete even - // when the response has no usable answer. A transport - // failure with no response usage is ambiguous. - if !model_accounting_complete || !has_billable_usage(model_usage) { - usage_accounting_complete = false; - } - eprintln!( - "postil: model {} failed category={}", - log_text(&model), - safe_anyhow_category(&e) - ); - // Provider failures that report no tokens have no billable - // usage to attribute. Omit them rather than emitting a - // misleading accounting entry; token-bearing failures are - // retained and priced by the hosted control plane. - if has_billable_usage(model_usage) { - add_usage(&mut usage, model_usage); - } - models.append(&mut call_usage); - if e.downcast_ref::().is_some() { - return Err(e); - } - last_err = Some(e); - } - } - } - Err(last_err.unwrap_or_else(|| anyhow!("empty model chain"))) - } - pub async fn score_findings( &self, cfg: &Config, @@ -4361,10 +4303,16 @@ impl LlmClient { Self::ensure_atomic_attribution_request_size(&body)?; } let mut retries = 0u32; - let mut timeout_retries = 0u32; let mut empty_response_retries = 0u32; let mut exhausted_output_retries = 0u32; let mut attempt_timeout = self.request_timeout; + let attempt_limit = if budgeted_review_retry(phase) + && (self.review_deadline.is_some() || self.review_model_deadline.is_some()) + { + MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u32 + } else { + MAX_TRANSPORT_ATTEMPTS_PER_CALL as u32 + }; loop { if matches!(phase, LlmPhase::Review | LlmPhase::Adjudication) { self.ensure_review_request_fits_context(model, &body, request_max_tokens)?; @@ -4379,7 +4327,7 @@ impl LlmClient { phase.as_str(), log_text(model), retries + 1, - TRANSIENT_RETRIES + 1, + attempt_limit, elapsed_text(timeout), remaining .map(elapsed_text) @@ -4402,18 +4350,22 @@ impl LlmClient { *usage_accounting_complete = false; call_usage .push(self.model_usage_event(model, phase, call_phase, attempt, None)); - if timeout_retries < TIMEOUT_RETRIES && retries < TRANSIENT_RETRIES { + let wait = transient_retry_wait(None, phase, retries); + let remaining = self.remaining_budget(phase)?; + if transient_retry_affordable(phase, retries, wait, remaining) { retries += 1; - timeout_retries += 1; - let wait = provider_retry_delay(retries); eprintln!( "postil: model {} hit a request timeout after {}, retrying in {} \ - (timeout retry {timeout_retries}/{TIMEOUT_RETRIES}; retry {retries}/{TRANSIENT_RETRIES})", + (retry {retries}/{attempt_limit}; budget_remaining={})", log_text(model), elapsed_text(attempt_started_at.elapsed()), - elapsed_text(wait) + elapsed_text(wait), + remaining + .map(elapsed_text) + .unwrap_or_else(|| "unbounded".to_string()) ); - self.sleep_with_budget(phase, wait).await?; + self.sleep_with_captured_budget(phase, wait, remaining) + .await?; attempt_timeout = self.timeout_retry_timeout; continue; } @@ -4634,25 +4586,26 @@ impl LlmClient { *usage_accounting_complete = false; } let status = response.status; - if timeout_status(status.as_u16()) - && timeout_retries < TIMEOUT_RETRIES - && retries < TRANSIENT_RETRIES - { - retries += 1; - timeout_retries += 1; - let wait = response - .retry_after - .unwrap_or_else(|| provider_retry_delay(retries)); - eprintln!( - "postil: model {} returned timeout HTTP {status} after {}, retrying in {} \ - (timeout retry {timeout_retries}/{TIMEOUT_RETRIES}; retry {retries}/{TRANSIENT_RETRIES})", - log_text(model), - elapsed_text(attempt_started_at.elapsed()), - elapsed_text(wait) - ); - self.sleep_with_budget(phase, wait).await?; - attempt_timeout = self.timeout_retry_timeout; - continue; + if timeout_status(status.as_u16()) { + let wait = transient_retry_wait(response.retry_after, phase, retries); + let remaining = self.remaining_budget(phase)?; + if transient_retry_affordable(phase, retries, wait, remaining) { + retries += 1; + eprintln!( + "postil: model {} returned timeout HTTP {status} after {}, retrying in {} \ + (retry {retries}/{attempt_limit}; budget_remaining={})", + log_text(model), + elapsed_text(attempt_started_at.elapsed()), + elapsed_text(wait), + remaining + .map(elapsed_text) + .unwrap_or_else(|| "unbounded".to_string()) + ); + self.sleep_with_captured_budget(phase, wait, remaining) + .await?; + attempt_timeout = self.timeout_retry_timeout; + continue; + } } if timeout_status(status.as_u16()) { let detail = provider_http_status_detail( @@ -4662,21 +4615,26 @@ impl LlmClient { ); return Err(anyhow::Error::new(ProviderHttpFailure(status)).context(detail)); } - if retryable_status(status.as_u16()) && retries < TRANSIENT_RETRIES { - retries += 1; - let wait = response - .retry_after - .unwrap_or_else(|| provider_retry_delay(retries)); - eprintln!( - "postil: model {} returned retryable HTTP {status} after {}, retrying in {} \ - (retry {retries}/{TRANSIENT_RETRIES})", - log_text(model), - elapsed_text(attempt_started_at.elapsed()), - elapsed_text(wait) - ); - self.sleep_with_budget(phase, wait).await?; - attempt_timeout = self.request_timeout; - continue; + if retryable_status(status.as_u16()) { + let wait = transient_retry_wait(response.retry_after, phase, retries); + let remaining = self.remaining_budget(phase)?; + if transient_retry_affordable(phase, retries, wait, remaining) { + retries += 1; + eprintln!( + "postil: model {} returned retryable HTTP {status} after {}, retrying in {} \ + (retry {retries}/{attempt_limit}; budget_remaining={})", + log_text(model), + elapsed_text(attempt_started_at.elapsed()), + elapsed_text(wait), + remaining + .map(elapsed_text) + .unwrap_or_else(|| "unbounded".to_string()) + ); + self.sleep_with_captured_budget(phase, wait, remaining) + .await?; + attempt_timeout = self.request_timeout; + continue; + } } return Err(anyhow::Error::new(ProviderHttpFailure(status)).context( provider_http_status_detail( @@ -4686,45 +4644,55 @@ impl LlmClient { ), )); } - Err(error) - if reqwest_error(&error).is_some_and(reqwest::Error::is_timeout) - && timeout_retries < TIMEOUT_RETRIES - && retries < TRANSIENT_RETRIES => - { + Err(error) if reqwest_error(&error).is_some_and(reqwest::Error::is_timeout) => { *usage_accounting_complete = false; call_usage .push(self.model_usage_event(model, phase, call_phase, attempt, None)); - retries += 1; - timeout_retries += 1; - let wait = provider_retry_delay(retries); - eprintln!( - "postil: model {} hit a request timeout after {}, retrying in {} \ - (timeout retry {timeout_retries}/{TIMEOUT_RETRIES}; retry {retries}/{TRANSIENT_RETRIES})", - log_text(model), - elapsed_text(attempt_started_at.elapsed()), - elapsed_text(wait) - ); - self.sleep_with_budget(phase, wait).await?; - attempt_timeout = self.timeout_retry_timeout; + let wait = transient_retry_wait(None, phase, retries); + let remaining = self.remaining_budget(phase)?; + if transient_retry_affordable(phase, retries, wait, remaining) { + retries += 1; + eprintln!( + "postil: model {} hit a request timeout after {}, retrying in {} \ + (retry {retries}/{attempt_limit}; budget_remaining={})", + log_text(model), + elapsed_text(attempt_started_at.elapsed()), + elapsed_text(wait), + remaining + .map(elapsed_text) + .unwrap_or_else(|| "unbounded".to_string()) + ); + self.sleep_with_captured_budget(phase, wait, remaining) + .await?; + attempt_timeout = self.timeout_retry_timeout; + continue; + } + return Err(RequestTimedOut.into()); } - Err(error) - if reqwest_error(&error).is_some_and(reqwest::Error::is_connect) - && retries < TRANSIENT_RETRIES => - { + Err(error) if reqwest_error(&error).is_some_and(reqwest::Error::is_connect) => { *usage_accounting_complete = false; call_usage .push(self.model_usage_event(model, phase, call_phase, attempt, None)); - retries += 1; - let wait = provider_retry_delay(retries); - eprintln!( - "postil: model {} hit a retryable connection error after {}, retrying in {} \ - (retry {retries}/{TRANSIENT_RETRIES})", - log_text(model), - elapsed_text(attempt_started_at.elapsed()), - elapsed_text(wait) - ); - self.sleep_with_budget(phase, wait).await?; - attempt_timeout = self.request_timeout; + let wait = transient_retry_wait(None, phase, retries); + let remaining = self.remaining_budget(phase)?; + if transient_retry_affordable(phase, retries, wait, remaining) { + retries += 1; + eprintln!( + "postil: model {} hit a retryable connection error after {}, retrying in {} \ + (retry {retries}/{attempt_limit}; budget_remaining={})", + log_text(model), + elapsed_text(attempt_started_at.elapsed()), + elapsed_text(wait), + remaining + .map(elapsed_text) + .unwrap_or_else(|| "unbounded".to_string()) + ); + self.sleep_with_captured_budget(phase, wait, remaining) + .await?; + attempt_timeout = self.request_timeout; + continue; + } + return Err(error.context("request to model endpoint failed")); } Err(error) => { *usage_accounting_complete = false; @@ -5137,7 +5105,7 @@ impl LlmClient { | LlmPhase::Scorer { .. } | LlmPhase::Adjudication | LlmPhase::Attribution => self.scorer_deadline.or(self.total_deadline), - LlmPhase::Respond | LlmPhase::Total => self.total_deadline, + LlmPhase::Total => self.total_deadline, }; let deadline = if matches!(phase, LlmPhase::Planner | LlmPhase::Review) { match (phase_deadline, self.review_model_deadline) { @@ -5170,7 +5138,18 @@ impl LlmClient { } async fn sleep_with_budget(&self, phase: LlmPhase, duration: Duration) -> Result<()> { - let Some(remaining) = self.remaining_budget(phase)? else { + let remaining = self.remaining_budget(phase)?; + self.sleep_with_captured_budget(phase, duration, remaining) + .await + } + + async fn sleep_with_captured_budget( + &self, + phase: LlmPhase, + duration: Duration, + remaining: Option, + ) -> Result<()> { + let Some(remaining) = remaining else { tokio::time::sleep(duration).await; return Ok(()); }; @@ -5260,6 +5239,33 @@ fn apply_openrouter_price_ceiling(body: &mut serde_json::Value, bound: &ModelPri ); } +fn apply_openrouter_reasoning(body: &mut serde_json::Value, effort: ReasoningEffort) { + body["reasoning"] = json!({ "effort": effort.as_str() }); +} + +fn apply_anthropic_reasoning(body: &mut serde_json::Value, effort: ReasoningEffort) { + match effort { + ReasoningEffort::None => { + body["thinking"] = json!({ "type": "disabled" }); + } + ReasoningEffort::Minimal => { + unreachable!("Anthropic minimal effort is rejected while resolving request settings") + } + _ => { + if body + .get("temperature") + .and_then(serde_json::Value::as_f64) + .is_some_and(|temperature| temperature != 1.0) + { + body.as_object_mut() + .expect("Anthropic request body is an object") + .remove("temperature"); + } + body["output_config"] = json!({ "effort": effort.as_str() }); + } + } +} + fn apply_openrouter_scorer_contract(body: &mut serde_json::Value, expected_len: usize) { debug_assert!(expected_len <= SCORER_MAX_FINDINGS); let provider = body @@ -5271,10 +5277,7 @@ fn apply_openrouter_scorer_contract(body: &mut serde_json::Value, expected_len: .as_object_mut() .expect("provider routing configuration is an object") .insert("require_parameters".to_string(), json!(true)); - body["reasoning"] = json!({ - "effort": "none", - "exclude": true, - }); + body["reasoning"]["exclude"] = json!(true); body["response_format"] = json!({ "type": "json_schema", "json_schema": { @@ -5334,7 +5337,7 @@ fn apply_openrouter_atomic_attribution_contract( .as_object_mut() .expect("provider routing configuration is an object"); provider.insert("require_parameters".to_string(), json!(true)); - body["reasoning"] = json!({ "effort": "none", "exclude": true }); + body["reasoning"]["exclude"] = json!(true); body["response_format"] = json!({ "type": "json_schema", "json_schema": { @@ -5421,7 +5424,7 @@ fn retry_after_duration_at(headers: &HeaderMap, now: std::time::SystemTime) -> O .unwrap_or_default() }; - Some(duration.min(Duration::from_secs(PROVIDER_RETRY_DELAY_CAP_SECS))) + Some(duration) } fn safe_header_value(value: Option<&HeaderValue>) -> Option { @@ -5652,7 +5655,7 @@ impl LlmTimeouts { } } -async fn resolve_api_key() -> Result { +async fn resolve_api_key(api_base: &str) -> Result { // Checked first, and independently of the credentials-path lookup below, // so an explicit key still works in a minimal environment with no HOME // or XDG_CONFIG_HOME -- exactly what worked before login existed. @@ -5660,16 +5663,14 @@ async fn resolve_api_key() -> Result { let stored_login = async { let credentials_path = crate::credentials::default_path() .context("resolving the postil login credentials path")?; - crate::login::resolve_stored_token(&credentials_path).await + crate::login::resolve_stored_token(&credentials_path, api_base).await }; match api_key::resolve_explicit_or_stored(explicit_key, stored_login).await? { Some(key) => Ok(key), - None => { - let key_names = api_key::names_text(); - Err(anyhow!( - "no API key: set {key_names}, or run `postil login`. Postil never proxies your inference without one of these." - )) - } + None => Err(anyhow!( + "no API key: {}. Postil never proxies your inference without one of these.", + api_key::credential_help() + )), } } @@ -6571,9 +6572,112 @@ mod tests { Duration::from_secs(4) ); } + + #[test] + fn retry_attempt_limits_are_phase_specific() { + let fixed_phases = [ + LlmPhase::Planner, + LlmPhase::Resolution, + LlmPhase::Brevity, + LlmPhase::Scorer { expected_len: 1 }, + LlmPhase::Adjudication, + LlmPhase::Attribution, + ]; + for phase in fixed_phases { + assert_eq!(max_transport_attempts(phase), 3, "phase={phase:?}"); + } + assert_eq!(max_transport_attempts(LlmPhase::Review), 13); + } + + #[test] + fn budgeted_waits_have_a_floor_and_provider_hints_remain_authoritative() { + assert_eq!( + budgeted_retry_wait(Some(Duration::ZERO), TRANSIENT_RETRIES + 1), + MIN_BUDGETED_RETRY_WAIT + ); + assert_eq!( + budgeted_retry_wait(Some(Duration::from_secs(37)), TRANSIENT_RETRIES + 1), + Duration::from_secs(37) + ); + assert_eq!( + provider_retry_delay_with_sample(MAX_BUDGETED_TRANSIENT_RETRIES, 10_000), + MAX_SYNTHESIZED_RETRY_BACKOFF + ); + } + + #[test] + fn budgeted_retry_reserves_the_next_attempt_and_unbudgeted_review_stops_at_three() { + let phase = LlmPhase::Review; + assert!(transient_retry_affordable( + phase, + TRANSIENT_RETRIES, + Duration::from_secs(1), + Some(RETRY_ATTEMPT_RESERVE + Duration::from_secs(1)), + )); + assert!(!transient_retry_affordable( + phase, + TRANSIENT_RETRIES, + Duration::from_secs(2), + Some(RETRY_ATTEMPT_RESERVE + Duration::from_secs(1)), + )); + assert!(!transient_retry_affordable( + phase, + TRANSIENT_RETRIES, + Duration::from_secs(1), + None, + )); + assert!(!transient_retry_affordable( + phase, + MAX_BUDGETED_TRANSIENT_RETRIES, + Duration::from_secs(1), + Some(Duration::from_secs(86_400)), + )); + } + + #[test] + fn hosted_exposure_prices_review_and_non_review_phases() { + let config = Config { + model: "provider/model".into(), + scorer_enabled: false, + uncertainty_resolution: false, + concise_findings: false, + ..Config::default() + }; + let client = LlmClient::build( + &config, + "test-key".into(), + Duration::from_secs(1), + None, + None, + ) + .unwrap(); + let price = ModelPriceBound { + model: config.model.clone(), + input_micros_per_million_tokens: 1, + output_micros_per_million_tokens: 1, + }; + let review = client + .planned_review_request_paths(&config.model, "system", "user", REVIEW_MAX_TOKENS) + .unwrap(); + let resolution = client + .planned_request_path( + &config.model, + "system", + "user", + 1_024, + 0.0, + LlmPhase::Resolution, + ) + .unwrap(); + assert_eq!( + request_paths_exposure(&review, &price).unwrap().attempts, + 26 + ); + assert_eq!(resolution.attempts.len(), 3); + } use crate::config::Config; use crate::envelope::{Kind, Severity}; - use std::sync::{Mutex, OnceLock}; + use crate::test_env_lock as env_lock; use wiremock::matchers::{body_string_contains, method, path}; use wiremock::{Mock, MockServer, ResponseTemplate}; @@ -6743,11 +6847,6 @@ mod tests { } } - fn env_lock() -> &'static Mutex<()> { - static LOCK: OnceLock> = OnceLock::new(); - LOCK.get_or_init(|| Mutex::new(())) - } - #[test] fn provider_error_downcast_survives_additional_context() { // review.rs's fail-open/fail-closed classifier does @@ -7174,12 +7273,14 @@ mod tests { .unwrap(); assert_eq!( admission.output_tokens, - u64::from(REVIEW_MAX_TOKENS + 2 * REVIEW_MAX_OUTPUT_TOKENS) + (u64::from(REVIEW_MAX_TOKENS) + + (MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u64 - 1) + * u64::from(REVIEW_MAX_OUTPUT_TOKENS)) * MAX_LOGICAL_CALLS_PER_REVIEW_MODEL as u64 + u64::from(MAX_ADJUDICATION_OUTPUT_TOKENS) * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 ); - assert_eq!(admission.provider_attempts, 9); + assert_eq!(admission.provider_attempts, 29); let plan = client .plan_review_batches( &config, @@ -7861,50 +7962,6 @@ mod tests { })); } - #[test] - fn hosted_respond_preflight_accounts_for_every_fallback_before_calls() { - let config = Config { - model: "provider/primary".into(), - cascade: vec!["provider/fallback".into()], - ..Config::default() - }; - let mut client = LlmClient::build( - &config, - "test-key".into(), - Duration::from_secs(1), - None, - None, - ) - .unwrap(); - client.request_decorations.hosted_price_bounds = Some(Arc::new(HashMap::from([ - ( - "provider/primary".into(), - ModelPriceBound { - model: "provider/primary".into(), - input_micros_per_million_tokens: 1, - output_micros_per_million_tokens: 1, - }, - ), - ( - "provider/fallback".into(), - ModelPriceBound { - model: "provider/fallback".into(), - input_micros_per_million_tokens: 1, - output_micros_per_million_tokens: 1, - }, - ), - ]))); - let admission = client - .preflight_respond_plan(&config, "system", "bounded user context") - .unwrap(); - assert_eq!(admission.provider_attempts, 6); - assert_eq!( - admission.output_tokens, - u64::from(RESPOND_MAX_TOKENS + 2 * RESPOND_MAX_TOKENS * 2) * 2 - ); - assert_eq!(client.admission.lock().unwrap().attempts, 0); - } - #[test] fn exhausted_output_retry_ceiling_is_phase_aware() { assert_eq!( @@ -7915,10 +7972,6 @@ mod tests { LlmPhase::Planner.exhausted_output_retry_max_tokens(PLANNER_MAX_TOKENS), PLANNER_MAX_TOKENS * 2 ); - assert_eq!( - LlmPhase::Respond.exhausted_output_retry_max_tokens(RESPOND_MAX_TOKENS), - RESPOND_MAX_TOKENS * 2 - ); let scorer_tokens = scorer_max_tokens(SCORER_MAX_FINDINGS).unwrap(); assert_eq!( LlmPhase::Scorer { @@ -8011,7 +8064,9 @@ mod tests { }, ) .unwrap(); - let review_per_model = u64::from(REVIEW_MAX_TOKENS + 2 * REVIEW_MAX_OUTPUT_TOKENS) + let review_per_model = (u64::from(REVIEW_MAX_TOKENS) + + (MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u64 - 1) + * u64::from(REVIEW_MAX_OUTPUT_TOKENS)) * MAX_LOGICAL_CALLS_PER_REVIEW_MODEL as u64; let scorer_per_model = u64::from(scorer_max_tokens(SCORER_MAX_FINDINGS).unwrap()) * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 @@ -8033,8 +8088,8 @@ mod tests { + u64::from(MAX_ADJUDICATION_OUTPUT_TOKENS) * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 ); - assert_eq!(admission.provider_attempts, 117); - assert_eq!(admission.projected_cost_micros, 234); + assert_eq!(admission.provider_attempts, 157); + assert_eq!(admission.projected_cost_micros, 314); } #[test] @@ -8087,12 +8142,12 @@ mod tests { assert_eq!( admission.output_tokens, u64::from(REVIEW_MAX_OUTPUT_TOKENS) - * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 + * MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u64 * MAX_LOGICAL_CALLS_PER_REVIEW_MODEL as u64 + u64::from(MAX_ADJUDICATION_OUTPUT_TOKENS) * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 ); - assert_eq!(admission.provider_attempts, 9); + assert_eq!(admission.provider_attempts, 29); } #[test] @@ -8188,7 +8243,9 @@ mod tests { .unwrap(); assert_eq!( admission.output_tokens, - u64::from(REVIEW_MAX_TOKENS + 2 * REVIEW_MAX_OUTPUT_TOKENS) + (u64::from(REVIEW_MAX_TOKENS) + + (MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL as u64 - 1) + * u64::from(REVIEW_MAX_OUTPUT_TOKENS)) * MAX_LOGICAL_CALLS_PER_REVIEW_MODEL as u64 * crate::review::MAX_HOSTED_SELECTED_BATCHES as u64 + u64::from(PLANNER_MAX_TOKENS + 2 * PLANNER_MAX_TOKENS * 2) @@ -8204,7 +8261,7 @@ mod tests { + u64::from(MAX_ADJUDICATION_OUTPUT_TOKENS) * MAX_TRANSPORT_ATTEMPTS_PER_CALL as u64 ); - assert_eq!(admission.provider_attempts, 84); + assert_eq!(admission.provider_attempts, 184); assert_eq!(client.admission.lock().unwrap().attempts, 0); } @@ -8296,8 +8353,8 @@ mod tests { assert!(logical_calls <= MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG); assert!(logical_calls * MAX_TRANSPORT_ATTEMPTS_PER_CALL <= MAX_PROVIDER_ATTEMPTS); assert_eq!( - MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG * MAX_TRANSPORT_ATTEMPTS_PER_CALL, - 192 + MAX_HOSTED_PLANNED_CALLS_BY_WATCHDOG * MAX_REVIEW_TRANSPORT_ATTEMPTS_PER_CALL, + MAX_PROVIDER_ATTEMPTS ); } @@ -8453,13 +8510,13 @@ mod tests { output_micros_per_million_tokens: 1, }, )]))); - let prompts = vec!["candidate".to_string(); 25]; + let prompts = vec!["candidate".to_string(); 7]; let output_tokens = vec![REVIEW_MAX_TOKENS; prompts.len()]; client .preflight_review_plan_with_output_limits( &config, - 24, + 6, "system", ReviewPreflightPrompts { first_users: &prompts, @@ -8470,14 +8527,14 @@ mod tests { }, ReviewPlanSchedule { planner: None, - batch_concurrency: 4, + batch_concurrency: 1, }, ) .unwrap(); let error = client .preflight_review_plan_with_output_limits( &config, - 25, + 7, "system", ReviewPreflightPrompts { first_users: &prompts, @@ -8488,7 +8545,7 @@ mod tests { }, ReviewPlanSchedule { planner: None, - batch_concurrency: 4, + batch_concurrency: 1, }, ) .unwrap_err(); @@ -9014,7 +9071,7 @@ mod tests { } #[test] - fn provider_retry_after_parses_delta_seconds_and_caps_local_waits() { + fn provider_retry_after_preserves_provider_delta_seconds() { let mut headers = HeaderMap::new(); headers.insert(reqwest::header::RETRY_AFTER, HeaderValue::from_static("0")); assert_eq!(retry_after_duration(&headers), Some(Duration::ZERO)); @@ -9031,7 +9088,7 @@ mod tests { ); assert_eq!( retry_after_duration(&headers), - Some(Duration::from_secs(PROVIDER_RETRY_DELAY_CAP_SECS)) + Some(Duration::from_secs(999)) ); } @@ -9062,14 +9119,12 @@ mod tests { headers.insert( reqwest::header::RETRY_AFTER, - HeaderValue::from_str(&httpdate::fmt_http_date( - now + Duration::from_secs(PROVIDER_RETRY_DELAY_CAP_SECS), - )) - .unwrap(), + HeaderValue::from_str(&httpdate::fmt_http_date(now + Duration::from_secs(999))) + .unwrap(), ); assert_eq!( retry_after_duration_at(&headers, now), - Some(Duration::from_secs(PROVIDER_RETRY_DELAY_CAP_SECS)) + Some(Duration::from_secs(999)) ); headers.insert( @@ -9481,7 +9536,7 @@ mod tests { } #[test] - fn canonical_openrouter_scorer_uses_no_reasoning_effort_and_requires_strict_schema() { + fn canonical_openrouter_uses_role_defaults_and_scorer_strict_schema() { let client = LlmClient::build( &Config::default(), "test-key".into(), @@ -9553,7 +9608,7 @@ mod tests { 0.1, LlmPhase::Review, ); - assert!(generator.get("reasoning").is_none()); + assert_eq!(generator["reasoning"], json!({"effort": "low"})); assert!(generator.get("response_format").is_none()); assert!(generator["provider"].get("require_parameters").is_none()); } @@ -9576,7 +9631,7 @@ mod tests { 0.0, LlmPhase::Resolution, ); - assert!(resolution.get("reasoning").is_none()); + assert_eq!(resolution["reasoning"], json!({"effort": "low"})); assert!(resolution.get("response_format").is_none()); assert!(resolution["provider"].get("require_parameters").is_none()); } @@ -9748,6 +9803,118 @@ mod tests { assert_eq!(client.admission.lock().unwrap().reported_token_spend, 27); } + #[tokio::test] + async fn budgeted_review_retries_source_and_synthesis_past_the_fixed_allowance() { + for route in [ReviewRequestRoute::Source, ReviewRequestRoute::Synthesis] { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "0")) + .with_priority(1) + .up_to_n_times(3) + .expect(3) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": [{ + "finish_reason": "stop", + "message": {"content": "{\"summary\":\"\",\"findings\":[]}"} + }], + "usage": {"prompt_tokens": 10, "completion_tokens": 2} + }))) + .with_priority(2) + .expect(1) + .mount(&server) + .await; + let config = Config { + api_base: server.uri(), + api_format: ApiFormat::OpenaiCompatible, + model: "provider/model".into(), + scorer_enabled: false, + uncertainty_resolution: false, + concise_findings: false, + ..Config::default() + }; + let deadline = Instant::now() + Duration::from_secs(20); + let client = LlmClient::build( + &config, + "test-key".into(), + Duration::from_secs(1), + Some(deadline), + Some(deadline), + ) + .unwrap(); + *client.http.lock().unwrap() = Some(reqwest::Client::new()); + + let result = client + .review_validated_with_safe_output_limit( + &config, + "system", + "user", + REVIEW_MAX_TOKENS, + route, + |_| Ok(()), + ) + .await + .unwrap(); + assert!(result.findings.is_empty()); + assert_eq!(server.received_requests().await.unwrap().len(), 4); + assert_eq!( + result + .model_usage + .iter() + .map(|event| event.attempt) + .collect::>(), + vec![Some(1), Some(2), Some(3), Some(4)] + ); + } + } + + #[tokio::test] + async fn unbudgeted_source_review_keeps_three_transient_attempts() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "0")) + .expect(3) + .mount(&server) + .await; + let config = Config { + api_base: server.uri(), + api_format: ApiFormat::OpenaiCompatible, + model: "provider/model".into(), + scorer_enabled: false, + uncertainty_resolution: false, + concise_findings: false, + ..Config::default() + }; + let client = LlmClient::build( + &config, + "test-key".into(), + Duration::from_secs(1), + None, + None, + ) + .unwrap(); + *client.http.lock().unwrap() = Some(reqwest::Client::new()); + + let error = client + .review_validated_with_safe_output_limit( + &config, + "system", + "user", + REVIEW_MAX_TOKENS, + ReviewRequestRoute::Source, + |_| Ok(()), + ) + .await + .unwrap_err(); + assert!(!error.is_timeout()); + assert_eq!(server.received_requests().await.unwrap().len(), 3); + } + #[tokio::test] async fn a_persistently_missing_identity_echo_stays_terminal_after_the_transient_budget() { let server = MockServer::start().await; @@ -9973,7 +10140,7 @@ mod tests { } #[test] - fn byok_openai_and_direct_anthropic_request_bodies_have_no_hosted_routing_fields() { + fn byok_openai_and_direct_anthropic_requests_send_role_reasoning_without_hosted_routing() { let _lock = env_lock().lock().unwrap(); let _env = EnvRestore::capture(&["POSTIL_HOSTED_MODE"]); EnvRestore::remove("POSTIL_HOSTED_MODE"); @@ -9999,7 +10166,7 @@ mod tests { LlmPhase::Scorer { expected_len: 1 }, ); assert!(openai_body.get("provider").is_none()); - assert!(openai_body.get("reasoning").is_none()); + assert_eq!(openai_body["reasoning"], json!({"effort": "none"})); assert!(openai_body.get("response_format").is_none()); let anthropic = LlmClient::build( @@ -10024,11 +10191,118 @@ mod tests { LlmPhase::Scorer { expected_len: 1 }, ); assert!(anthropic_body.get("provider").is_none()); - assert!(anthropic_body.get("reasoning").is_none()); + assert_eq!(anthropic_body["thinking"], json!({"type": "disabled"})); + assert!(anthropic_body.get("output_config").is_none()); + assert_eq!(anthropic_body["temperature"], 0.0); assert!(anthropic_body.get("response_format").is_none()); assert_eq!(anthropic_body["system"], "system"); } + #[test] + fn every_openrouter_phase_uses_its_role_specific_effort() { + let client = LlmClient::build( + &Config { + reasoning_effort: ReasoningEffort::Xhigh, + scorer_reasoning_effort: ReasoningEffort::Medium, + ..Config::default() + }, + "test-key".into(), + Duration::from_secs(1), + None, + None, + ) + .unwrap(); + + for phase in [ + LlmPhase::Planner, + LlmPhase::Review, + LlmPhase::Resolution, + LlmPhase::Brevity, + LlmPhase::Total, + ] { + let body = client.request_body("provider/model", "system", "user", 100, 0.0, phase); + assert_eq!(body["reasoning"], json!({"effort": "xhigh"})); + } + for phase in [ + LlmPhase::Scorer { expected_len: 1 }, + LlmPhase::Adjudication, + LlmPhase::Attribution, + ] { + let body = client.request_body("provider/model", "system", "user", 100, 0.0, phase); + assert_eq!(body["reasoning"]["effort"], "medium"); + } + } + + #[test] + fn native_anthropic_maps_supported_efforts_and_rejects_minimal() { + let client = LlmClient::build( + &Config { + api_format: ApiFormat::Anthropic, + api_base: "https://api.anthropic.com/v1".into(), + reasoning_effort: ReasoningEffort::High, + scorer_reasoning_effort: ReasoningEffort::None, + ..Config::default() + }, + "test-key".into(), + Duration::from_secs(1), + None, + None, + ) + .unwrap(); + let reviewer = client.request_body( + "provider/model", + "system", + "user", + 100, + 0.0, + LlmPhase::Review, + ); + assert_eq!(reviewer["output_config"], json!({"effort": "high"})); + assert!(reviewer.get("thinking").is_none()); + assert!(reviewer.get("temperature").is_none()); + let reviewer_with_default_temperature = client.request_body( + "provider/model", + "system", + "user", + 100, + 1.0, + LlmPhase::Review, + ); + assert_eq!(reviewer_with_default_temperature["temperature"], 1.0); + assert_eq!( + reviewer_with_default_temperature["output_config"], + json!({"effort": "high"}) + ); + let scorer = client.request_body( + "provider/model", + "system", + "user", + 100, + 0.0, + LlmPhase::Scorer { expected_len: 1 }, + ); + assert_eq!(scorer["thinking"], json!({"type": "disabled"})); + assert!(scorer.get("output_config").is_none()); + assert_eq!(scorer["temperature"], 0.0); + + let error = LlmClient::build( + &Config { + api_format: ApiFormat::Anthropic, + api_base: "https://api.anthropic.com/v1".into(), + reasoning_effort: ReasoningEffort::Minimal, + ..Config::default() + }, + "test-key".into(), + Duration::from_secs(1), + None, + None, + ) + .err() + .expect("unsupported effort must fail before a request is built"); + assert!(error.to_string().contains("reasoningEffort minimal")); + assert!(error.to_string().contains("max|xhigh|high|medium|low|none")); + } + #[test] fn hosted_projected_and_reported_costs_cannot_cross_the_service_reservation() { let config = Config { diff --git a/src/local.rs b/src/local.rs index 4020f7b..e5e4001 100644 --- a/src/local.rs +++ b/src/local.rs @@ -4,6 +4,7 @@ use std::io::{Read, Write}; use std::process::Stdio; use anyhow::{Context, Result, anyhow}; +use sha2::{Digest, Sha256}; use tokio::process::Command; use crate::diff::{DiffSnapshot, DiffSpool}; @@ -15,6 +16,106 @@ pub enum LocalSource { Base(String), /// A unified diff already on disk. DiffFile(std::path::PathBuf), + /// Read-only staged selection for bare `postil review`. + AutoStaged { fence: LocalStateFence }, + /// A clean branch range has an immutable HEAD revision. A dirty branch + /// range includes the working tree and deliberately has no revision. + AutoBranch { + base_ref: String, + base_oid: String, + include_worktree: bool, + fence: LocalStateFence, + }, + /// Tracked working-tree changes only, never untracked files. + WorkingTree { fence: LocalStateFence }, + /// A clean repository has a valid empty review without a model request. + Clean { fence: LocalStateFence }, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LocalStateFence { + index_tree: String, + tracked_worktree_sha256: String, + tracked_worktree_dirty: bool, + untracked_paths: String, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct DefaultBranch { + reference: String, + oid: String, +} + +pub struct AutoSelection { + pub source: LocalSource, + pub warning: Option<&'static str>, +} + +/// Select the least surprising local diff without fetching or changing Git +/// refs or the index. Capturing the index may materialize its immutable tree. +pub async fn auto_select( + repository_root: &std::path::Path, + head_sha: Option<&str>, +) -> Result { + let fence = capture_local_state(repository_root).await?; + let warning = (!fence.untracked_paths.trim().is_empty()).then_some( + "untracked files were not reviewed; add the files you want reviewed, then rerun `postil review`", + ); + let index_base = match head_sha { + Some(head_sha) => head_sha.to_string(), + None => empty_tree(repository_root).await?, + }; + if git_has_changes( + repository_root, + &[ + "diff", + "--quiet", + index_base.as_str(), + fence.index_tree.as_str(), + "--", + ], + ) + .await? + { + return Ok(AutoSelection { + source: LocalSource::AutoStaged { fence }, + warning, + }); + } + let default_branch = if head_sha.is_some() { + Some(default_branch(repository_root).await?.ok_or_else(|| { + anyhow!( + "could not determine the repository default branch without fetching; rerun with `--base `, or use `--staged` or `--diff-file ` for an explicit review source" + ) + })?) + } else { + None + }; + if let (Some(base), Some(head_sha)) = (default_branch, head_sha) { + let range = format!("{}...{head_sha}", base.oid); + if git_has_changes(repository_root, &["diff", "--quiet", &range, "--"]).await? { + let include_worktree = fence.tracked_worktree_dirty; + return Ok(AutoSelection { + source: LocalSource::AutoBranch { + base_ref: base.reference, + base_oid: base.oid, + include_worktree, + fence, + }, + warning, + }); + } + } + if fence.tracked_worktree_dirty { + return Ok(AutoSelection { + source: LocalSource::WorkingTree { fence }, + warning, + }); + } + Ok(AutoSelection { + source: LocalSource::Clean { fence }, + warning, + }) } pub struct LocalReviewSnapshot { @@ -35,32 +136,376 @@ pub async fn acquire( diff: DiffSnapshot::from_path(path)?, repository_revision: None, }), - LocalSource::Staged => { - let base_revision = match head_sha { - Some(head_sha) => head_sha.to_string(), - None => git_output(repository_root, &["mktree"]).await?, - }; - let index_tree = git_output(repository_root, &["write-tree"]).await?; - if !crate::repository_search::valid_full_object_id(&index_tree) { - return Err(anyhow!("git write-tree returned an invalid object id")); - } - Ok(LocalReviewSnapshot { - diff: git_diff( - repository_root, - &["diff", "--no-color", &base_revision, &index_tree, "--"], - ) - .await?, - repository_revision: Some(index_tree), - }) + LocalSource::Staged => acquire_index_snapshot(head_sha, repository_root).await, + LocalSource::AutoStaged { fence } => { + acquire_automatic( + repository_root, + head_sha, + fence, + None, + acquire_index_tree_snapshot(head_sha, &fence.index_tree, repository_root), + ) + .await } LocalSource::Base(base) => { - let range = format!("{base}...{}", head_sha.unwrap_or("HEAD")); + let base_oid = resolve_commit_oid(repository_root, base).await?; + let range = format!("{base_oid}...{}", head_sha.unwrap_or("HEAD")); Ok(LocalReviewSnapshot { diff: git_diff(repository_root, &["diff", "--no-color", &range]).await?, repository_revision: head_sha.map(str::to_string), }) } + LocalSource::AutoBranch { + base_ref, + base_oid, + include_worktree, + fence, + } => { + let head_sha = head_sha.context("automatic branch review requires a captured HEAD")?; + anyhow::ensure!( + crate::repository_search::valid_full_object_id(head_sha), + "automatic branch review captured an invalid HEAD object id" + ); + acquire_automatic( + repository_root, + Some(head_sha), + fence, + Some((base_ref.as_str(), base_oid.as_str())), + async { + let merge_base = git_output( + repository_root, + &["merge-base", base_oid.as_str(), head_sha], + ) + .await?; + let range = format!("{base_oid}...{head_sha}"); + let diff = if *include_worktree { + git_diff( + repository_root, + &["diff", "--no-color", merge_base.as_str(), "--"], + ) + .await? + } else { + git_diff( + repository_root, + &["diff", "--no-color", range.as_str(), "--"], + ) + .await? + }; + Ok(LocalReviewSnapshot { + diff, + repository_revision: (!include_worktree).then(|| head_sha.to_string()), + }) + }, + ) + .await + } + LocalSource::WorkingTree { fence } => { + acquire_automatic(repository_root, head_sha, fence, None, async { + Ok(LocalReviewSnapshot { + diff: git_diff(repository_root, &["diff", "--no-color", "--"]).await?, + repository_revision: None, + }) + }) + .await + } + LocalSource::Clean { fence } => { + acquire_automatic(repository_root, head_sha, fence, None, async { + Ok(LocalReviewSnapshot { + diff: DiffSnapshot::from_bytes(b"")?, + repository_revision: head_sha.map(str::to_string), + }) + }) + .await + } + } +} + +async fn acquire_automatic( + repository_root: &std::path::Path, + captured_head: Option<&str>, + fence: &LocalStateFence, + base: Option<(&str, &str)>, + acquisition: impl std::future::Future>, +) -> Result { + ensure_automatic_state( + repository_root, + captured_head, + fence, + base, + "after automatic review selection", + ) + .await?; + let value = acquisition.await?; + ensure_automatic_state( + repository_root, + captured_head, + fence, + base, + "while acquiring the automatic review", + ) + .await?; + Ok(value) +} + +async fn acquire_index_snapshot( + head_sha: Option<&str>, + repository_root: &std::path::Path, +) -> Result { + let index_tree = capture_index_tree(repository_root).await?; + acquire_index_tree_snapshot(head_sha, &index_tree, repository_root).await +} + +async fn acquire_index_tree_snapshot( + head_sha: Option<&str>, + index_tree: &str, + repository_root: &std::path::Path, +) -> Result { + let base_revision = match head_sha { + Some(head_sha) => head_sha.to_string(), + None => empty_tree(repository_root).await?, + }; + if !crate::repository_search::valid_full_object_id(index_tree) { + return Err(anyhow!("git write-tree returned an invalid object id")); } + Ok(LocalReviewSnapshot { + diff: git_diff( + repository_root, + &["diff", "--no-color", &base_revision, index_tree, "--"], + ) + .await?, + repository_revision: Some(index_tree.to_string()), + }) +} + +async fn capture_local_state(repository_root: &std::path::Path) -> Result { + let index_tree = capture_index_tree(repository_root).await?; + let tracked = git_diff( + repository_root, + &[ + "diff", + "--no-color", + "--binary", + "--full-index", + "--no-ext-diff", + "--no-textconv", + "--", + ], + ) + .await?; + let tracked_worktree_dirty = !tracked.as_bytes().is_empty(); + let tracked_worktree_sha256 = Sha256::digest(tracked.as_bytes()) + .iter() + .map(|byte| format!("{byte:02x}")) + .collect(); + let untracked_paths = git_succeeds( + repository_root, + &["ls-files", "--others", "--exclude-standard"], + ) + .await?; + Ok(LocalStateFence { + index_tree, + tracked_worktree_sha256, + tracked_worktree_dirty, + untracked_paths, + }) +} + +async fn capture_index_tree(repository_root: &std::path::Path) -> Result { + let index_tree = git_output(repository_root, &["write-tree"]).await?; + anyhow::ensure!( + crate::repository_search::valid_full_object_id(&index_tree), + "git write-tree returned an invalid object id" + ); + Ok(index_tree) +} + +async fn empty_tree(repository_root: &std::path::Path) -> Result { + git_output(repository_root, &["mktree"]).await +} + +async fn ensure_automatic_state( + repository_root: &std::path::Path, + captured_head: Option<&str>, + fence: &LocalStateFence, + base: Option<(&str, &str)>, + timing: &str, +) -> Result<()> { + ensure_captured_head(repository_root, captured_head, timing).await?; + if let Some((base_ref, base_oid)) = base { + let current_base = resolve_commit_oid(repository_root, base_ref).await.ok(); + anyhow::ensure!( + current_base.as_deref() == Some(base_oid), + "default branch {base_ref} changed {timing}; rerun `postil review`" + ); + } + let current = capture_local_state(repository_root).await?; + anyhow::ensure!( + current.index_tree == fence.index_tree, + "index changed {timing}; rerun `postil review`" + ); + anyhow::ensure!( + current.tracked_worktree_sha256 == fence.tracked_worktree_sha256, + "tracked working tree changed {timing}; rerun `postil review`" + ); + anyhow::ensure!( + current.untracked_paths == fence.untracked_paths, + "untracked file set changed {timing}; rerun `postil review`" + ); + Ok(()) +} + +async fn ensure_captured_head( + repository_root: &std::path::Path, + captured_head: Option<&str>, + timing: &str, +) -> Result<()> { + let current_head = head_sha(repository_root).await?; + anyhow::ensure!( + current_head.as_deref() == captured_head, + "HEAD changed {timing}; rerun `postil review`" + ); + Ok(()) +} + +async fn git_succeeds(repository_root: &std::path::Path, args: &[&str]) -> Result { + let output = Command::new("git") + .arg("-C") + .arg(repository_root) + .args(args) + .output() + .await + .context("running git")?; + if !output.status.success() { + return Err(anyhow!( + "git {} failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + String::from_utf8(output.stdout).context("git output is not UTF-8") +} + +async fn git_has_changes(repository_root: &std::path::Path, args: &[&str]) -> Result { + let output = Command::new("git") + .arg("-C") + .arg(repository_root) + .args(args) + .output() + .await + .context("running git")?; + match output.status.code() { + Some(0) => Ok(false), + Some(1) => Ok(true), + _ => Err(anyhow!( + "git {} failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr).trim() + )), + } +} + +async fn default_branch(repository_root: &std::path::Path) -> Result> { + let current = git_output(repository_root, &["branch", "--show-current"]) + .await + .ok() + .filter(|branch| !branch.is_empty()); + let configured_remote = if let Some(current) = current.as_deref() { + git_output( + repository_root, + &["config", "--get", &format!("branch.{current}.remote")], + ) + .await + .ok() + } else { + None + }; + let known_remotes = git_output(repository_root, &["remote"]) + .await + .unwrap_or_default() + .lines() + .map(str::trim) + .filter(|remote| !remote.is_empty()) + .map(str::to_string) + .collect::>(); + let mut remotes = Vec::new(); + if let Some(remote) = configured_remote.filter(|remote| !remote.is_empty() && remote != ".") { + remotes.push(remote); + } + if known_remotes.iter().any(|remote| remote == "origin") + && !remotes.iter().any(|remote| remote == "origin") + { + remotes.push("origin".to_string()); + } + for remote in known_remotes { + if !remotes.contains(&remote) { + remotes.push(remote); + } + } + for remote in remotes { + let mut candidates = Vec::new(); + if let Ok(symbolic) = git_output( + repository_root, + &[ + "symbolic-ref", + "--quiet", + "--short", + &format!("refs/remotes/{remote}/HEAD"), + ], + ) + .await + { + candidates.push(symbolic); + } + candidates.extend([ + format!("refs/remotes/{remote}/main"), + format!("refs/remotes/{remote}/master"), + format!("refs/remotes/{remote}/trunk"), + ]); + if let Some(branch) = first_resolved_branch(repository_root, candidates).await? { + return Ok(Some(branch)); + } + } + first_resolved_branch( + repository_root, + ["main", "master", "trunk"] + .into_iter() + .filter(|candidate| current.as_deref() != Some(*candidate)) + .map(str::to_string), + ) + .await +} + +async fn first_resolved_branch( + repository_root: &std::path::Path, + candidates: impl IntoIterator, +) -> Result> { + for candidate in candidates { + if let Ok(oid) = resolve_commit_oid(repository_root, &candidate).await { + return Ok(Some(DefaultBranch { + reference: candidate, + oid, + })); + } + } + Ok(None) +} + +async fn resolve_commit_oid(repository_root: &std::path::Path, reference: &str) -> Result { + let oid = git_output( + repository_root, + &[ + "rev-parse", + "--verify", + "--quiet", + &format!("{reference}^{{commit}}"), + ], + ) + .await?; + anyhow::ensure!( + crate::repository_search::valid_full_object_id(&oid), + "git rev-parse returned an invalid commit object id" + ); + Ok(oid) } async fn git_output(repository_root: &std::path::Path, args: &[&str]) -> Result { @@ -132,15 +577,32 @@ async fn git_diff(repository_root: &std::path::Path, args: &[&str]) -> Result Option { - let out = Command::new("git") - .args(["rev-parse", "HEAD"]) +pub async fn head_sha(repository_root: &std::path::Path) -> Result> { + let output = Command::new("git") + .arg("-C") + .arg(repository_root) + .args(["rev-parse", "--verify", "--quiet", "HEAD"]) .output() .await - .ok()?; - out.status - .success() - .then(|| String::from_utf8_lossy(&out.stdout).trim().to_string()) + .context("running git rev-parse")?; + match output.status.code() { + Some(0) => { + let head = String::from_utf8(output.stdout) + .context("git rev-parse output is not UTF-8")? + .trim() + .to_string(); + anyhow::ensure!( + crate::repository_search::valid_full_object_id(&head), + "git rev-parse returned an invalid HEAD object id" + ); + Ok(Some(head)) + } + Some(1) => Ok(None), + _ => Err(anyhow!( + "git rev-parse HEAD failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + )), + } } #[cfg(test)] @@ -183,8 +645,530 @@ mod tests { commit } + fn commit_from_head(root: &Path) -> String { + run_git(root, &["add", "-A"]); + let tree = run_git(root, &["write-tree"]); + let parent = run_git(root, &["rev-parse", "HEAD"]); + let output = std::process::Command::new("git") + .arg("-C") + .arg(root) + .args(["commit-tree", &tree, "-p", &parent, "-m", "fixture"]) + .env("GIT_AUTHOR_NAME", "Fixture") + .env("GIT_AUTHOR_EMAIL", "fixture@example.invalid") + .env("GIT_COMMITTER_NAME", "Fixture") + .env("GIT_COMMITTER_EMAIL", "fixture@example.invalid") + .output() + .unwrap(); + assert!(output.status.success()); + let commit = String::from_utf8(output.stdout).unwrap().trim().to_string(); + run_git(root, &["update-ref", "HEAD", &commit]); + commit + } + + fn repository_with_main_and_feature(root: &Path) -> String { + run_git(root, &["init", "--quiet"]); + std::fs::write(root.join("tracked.txt"), "base\n").unwrap(); + let base = create_commit(root); + if run_git(root, &["branch", "--show-current"]) != "main" { + run_git(root, &["branch", "main", &base]); + } + run_git(root, &["checkout", "--quiet", "-b", "feature"]); + base + } + + #[tokio::test] + async fn bare_selection_prefers_one_immutable_staged_snapshot() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "staged\n").unwrap(); + run_git(root, &["add", "tracked.txt"]); + std::fs::write(root.join("tracked.txt"), "working\n").unwrap(); + + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::AutoStaged { .. })); + let snapshot = acquire(&selection.source, Some(&head), root).await.unwrap(); + assert!(snapshot.diff.as_str().contains("+staged")); + assert!(!snapshot.diff.as_str().contains("+working")); + let revision = snapshot.repository_revision.as_deref().unwrap(); + assert_eq!( + run_git(root, &["show", &format!("{revision}:tracked.txt")]), + "staged" + ); + } + + #[tokio::test] + async fn bare_selection_uses_current_remote_symbolic_default_before_main_fallback() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let base = run_git(root, &["rev-parse", "main"]); + run_git(root, &["update-ref", "refs/remotes/origin/trunk", &base]); + run_git( + root, + &[ + "symbolic-ref", + "refs/remotes/origin/HEAD", + "refs/remotes/origin/trunk", + ], + ); + run_git(root, &["config", "branch.feature.remote", "origin"]); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + let head = commit_from_head(root); + + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!( + matches!(selection.source, LocalSource::AutoBranch { ref base_ref, include_worktree: false, .. } if base_ref == "origin/trunk") + ); + let snapshot = acquire(&selection.source, Some(&head), root).await.unwrap(); + assert!(snapshot.diff.as_str().contains("+committed")); + assert_eq!(snapshot.repository_revision.as_deref(), Some(head.as_str())); + } + + #[tokio::test] + async fn bare_selection_uses_remote_default_without_a_branch_upstream() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let base = run_git(root, &["rev-parse", "main"]); + run_git(root, &["update-ref", "refs/remotes/origin/trunk", &base]); + run_git( + root, + &[ + "symbolic-ref", + "refs/remotes/origin/HEAD", + "refs/remotes/origin/trunk", + ], + ); + run_git( + root, + &[ + "remote", + "add", + "origin", + "https://example.invalid/repository.git", + ], + ); + assert!( + std::process::Command::new("git") + .args([ + "-C", + root.to_str().unwrap(), + "config", + "--get", + "branch.feature.remote" + ]) + .output() + .unwrap() + .stdout + .is_empty() + ); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + let head = commit_from_head(root); + + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!( + matches!(selection.source, LocalSource::AutoBranch { ref base_ref, include_worktree: false, .. } if base_ref == "origin/trunk") + ); + let snapshot = acquire(&selection.source, Some(&head), root).await.unwrap(); + assert!(snapshot.diff.as_str().contains("+committed")); + assert_eq!(snapshot.repository_revision.as_deref(), Some(head.as_str())); + } + #[tokio::test] - async fn staged_review_binds_diff_and_repository_to_one_index_tree() { + async fn bare_selection_uses_remote_trunk_without_a_symbolic_remote_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let base = run_git(root, &["rev-parse", "main"]); + run_git( + root, + &[ + "remote", + "add", + "origin", + "https://example.invalid/repository.git", + ], + ); + run_git(root, &["update-ref", "refs/remotes/origin/trunk", &base]); + run_git(root, &["branch", "-D", "main"]); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + let head = commit_from_head(root); + + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!( + matches!(selection.source, LocalSource::AutoBranch { ref base_ref, include_worktree: false, .. } if base_ref == "refs/remotes/origin/trunk") + ); + let snapshot = acquire(&selection.source, Some(&head), root).await.unwrap(); + assert!(snapshot.diff.as_str().contains("+committed")); + assert_eq!(snapshot.repository_revision.as_deref(), Some(head.as_str())); + } + + #[tokio::test] + async fn bare_selection_fails_closed_when_the_default_branch_is_unresolved() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + run_git(root, &["init", "--quiet"]); + run_git(root, &["symbolic-ref", "HEAD", "refs/heads/feature"]); + std::fs::write(root.join("tracked.txt"), "committed\n").unwrap(); + let head = create_commit(root); + + let error = match auto_select(root, Some(&head)).await { + Ok(_) => panic!("an unresolved default branch must not produce a partial review"), + Err(error) => error, + }; + let message = error.to_string(); + assert!(message.contains("could not determine the repository default branch")); + assert!(message.contains("--base ")); + assert!(message.contains("--staged")); + assert!(message.contains("--diff-file ")); + } + + #[tokio::test] + async fn bare_selection_rejects_the_current_local_default_branch_without_a_remote() { + for branch in ["main", "trunk"] { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + run_git(root, &["init", "--quiet"]); + run_git( + root, + &["symbolic-ref", "HEAD", &format!("refs/heads/{branch}")], + ); + std::fs::write(root.join("tracked.txt"), "committed\n").unwrap(); + let head = create_commit(root); + + let error = match auto_select(root, Some(&head)).await { + Ok(_) => panic!( + "the checked-out local {branch} branch must not be its own automatic base" + ), + Err(error) => error, + }; + let message = error.to_string(); + assert!(message.contains("could not determine the repository default branch")); + assert!(message.contains("--base ")); + assert!(message.contains("--staged")); + assert!(message.contains("--diff-file ")); + } + } + + #[tokio::test] + async fn configured_remote_main_precedes_another_remote_symbolic_default() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let base = run_git(root, &["rev-parse", "main"]); + run_git( + root, + &[ + "remote", + "add", + "upstream", + "https://example.invalid/upstream.git", + ], + ); + run_git( + root, + &[ + "remote", + "add", + "origin", + "https://example.invalid/origin.git", + ], + ); + run_git(root, &["update-ref", "refs/remotes/upstream/main", &base]); + run_git(root, &["update-ref", "refs/remotes/origin/trunk", &base]); + run_git( + root, + &[ + "symbolic-ref", + "refs/remotes/origin/HEAD", + "refs/remotes/origin/trunk", + ], + ); + run_git(root, &["config", "branch.feature.remote", "upstream"]); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + let head = commit_from_head(root); + + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!( + selection.source, + LocalSource::AutoBranch { + ref base_ref, + include_worktree: false, + .. + } if base_ref == "refs/remotes/upstream/main" + )); + } + + #[tokio::test] + async fn clean_automatic_branch_review_rejects_a_changed_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("captured.txt"), "captured\n").unwrap(); + let captured_head = commit_from_head(root); + + let selection = auto_select(root, Some(&captured_head)).await.unwrap(); + assert!(matches!( + selection.source, + LocalSource::AutoBranch { + include_worktree: false, + .. + } + )); + + std::fs::write(root.join("later.txt"), "later\n").unwrap(); + let later_head = commit_from_head(root); + assert_ne!(captured_head, later_head); + + let error = match acquire(&selection.source, Some(&captured_head), root).await { + Ok(_) => panic!("a clean automatic branch review must reject a moving HEAD"), + Err(error) => error, + }; + assert!(error.to_string().contains("HEAD changed")); + } + + #[tokio::test] + async fn dirty_automatic_branch_review_rejects_a_changed_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("captured.txt"), "captured\n").unwrap(); + let captured_head = commit_from_head(root); + std::fs::write(root.join("tracked.txt"), "dirty\n").unwrap(); + + let selection = auto_select(root, Some(&captured_head)).await.unwrap(); + assert!(matches!( + selection.source, + LocalSource::AutoBranch { + include_worktree: true, + .. + } + )); + + commit_from_head(root); + let error = match acquire(&selection.source, Some(&captured_head), root).await { + Ok(_) => panic!("a dirty automatic review must not follow a moving HEAD"), + Err(error) => error, + }; + assert!(error.to_string().contains("HEAD changed")); + } + + #[tokio::test] + async fn automatic_staged_review_rejects_a_changed_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "staged\n").unwrap(); + run_git(root, &["add", "tracked.txt"]); + let captured_head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&captured_head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::AutoStaged { .. })); + + commit_from_head(root); + let error = match acquire(&selection.source, Some(&captured_head), root).await { + Ok(_) => panic!("an automatic staged review must reject a moving HEAD"), + Err(error) => error, + }; + assert!(error.to_string().contains("HEAD changed")); + } + + #[tokio::test] + async fn automatic_worktree_review_rejects_a_changed_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "working\n").unwrap(); + let captured_head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&captured_head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::WorkingTree { .. })); + + commit_from_head(root); + let error = match acquire(&selection.source, Some(&captured_head), root).await { + Ok(_) => panic!("an automatic worktree review must reject a moving HEAD"), + Err(error) => error, + }; + assert!(error.to_string().contains("HEAD changed")); + } + + #[tokio::test] + async fn automatic_clean_review_rejects_a_changed_head() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let captured_head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&captured_head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::Clean { .. })); + + std::fs::write(root.join("later.txt"), "later\n").unwrap(); + commit_from_head(root); + let error = match acquire(&selection.source, Some(&captured_head), root).await { + Ok(_) => panic!("an automatic clean review must reject a moving HEAD"), + Err(error) => error, + }; + assert!(error.to_string().contains("HEAD changed")); + } + + #[tokio::test] + async fn automatic_acquisition_rejects_a_head_change_after_reading_the_source() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let captured_head = run_git(root, &["rev-parse", "HEAD"]); + let fence = capture_local_state(root).await.unwrap(); + + let error = acquire_automatic(root, Some(&captured_head), &fence, None, async { + let acquired_head = run_git(root, &["rev-parse", "HEAD"]); + std::fs::write(root.join("later.txt"), "later\n").unwrap(); + commit_from_head(root); + Ok(acquired_head) + }) + .await + .unwrap_err(); + assert!( + error + .to_string() + .contains("HEAD changed while acquiring the automatic review") + ); + } + + #[tokio::test] + async fn automatic_branch_review_rejects_a_moved_base_ref() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + let head = commit_from_head(root); + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::AutoBranch { .. })); + + run_git(root, &["update-ref", "refs/heads/main", &head]); + let error = match acquire(&selection.source, Some(&head), root).await { + Ok(_) => panic!("an automatic branch review must reject a moved base ref"), + Err(error) => error, + }; + assert!(error.to_string().contains("default branch main changed")); + } + + #[tokio::test] + async fn automatic_staged_review_rejects_an_index_change() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "staged-one\n").unwrap(); + run_git(root, &["add", "tracked.txt"]); + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + + std::fs::write(root.join("tracked.txt"), "staged-two\n").unwrap(); + run_git(root, &["add", "tracked.txt"]); + let error = match acquire(&selection.source, Some(&head), root).await { + Ok(_) => panic!("an automatic staged review must reject an index change"), + Err(error) => error, + }; + assert!(error.to_string().contains("index changed")); + } + + #[tokio::test] + async fn automatic_worktree_review_rejects_a_tracked_change() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "working-one\n").unwrap(); + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + + std::fs::write(root.join("tracked.txt"), "working-two\n").unwrap(); + let error = match acquire(&selection.source, Some(&head), root).await { + Ok(_) => panic!("an automatic worktree review must reject a tracked change"), + Err(error) => error, + }; + assert!(error.to_string().contains("tracked working tree changed")); + } + + #[tokio::test] + async fn automatic_clean_review_rejects_a_new_tracked_change() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + + std::fs::write(root.join("tracked.txt"), "later\n").unwrap(); + let error = match acquire(&selection.source, Some(&head), root).await { + Ok(_) => panic!("an automatic clean review must reject a tracked change"), + Err(error) => error, + }; + assert!(error.to_string().contains("tracked working tree changed")); + } + + #[tokio::test] + async fn bare_selection_combines_committed_branch_and_tracked_worktree_changes() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("branch.txt"), "committed\n").unwrap(); + commit_from_head(root); + std::fs::write(root.join("tracked.txt"), "working\n").unwrap(); + + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!( + selection.source, + LocalSource::AutoBranch { + include_worktree: true, + .. + } + )); + let snapshot = acquire(&selection.source, Some(&head), root).await.unwrap(); + assert!(snapshot.diff.as_str().contains("+committed")); + assert!(snapshot.diff.as_str().contains("+working")); + assert!(snapshot.repository_revision.is_none()); + } + + #[tokio::test] + async fn bare_selection_falls_back_to_tracked_worktree_then_clean_diff() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("tracked.txt"), "working\n").unwrap(); + let head = run_git(root, &["rev-parse", "HEAD"]); + let working = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!(working.source, LocalSource::WorkingTree { .. })); + + std::fs::write(root.join("tracked.txt"), "base\n").unwrap(); + let clean = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!(clean.source, LocalSource::Clean { .. })); + assert!( + acquire(&clean.source, Some(&head), root) + .await + .unwrap() + .diff + .as_str() + .is_empty() + ); + } + + #[tokio::test] + async fn bare_selection_warns_about_untracked_files_without_claiming_them_reviewed() { + let directory = tempfile::tempdir().unwrap(); + let root = directory.path(); + repository_with_main_and_feature(root); + std::fs::write(root.join("new.txt"), "untracked\n").unwrap(); + let head = run_git(root, &["rev-parse", "HEAD"]); + let selection = auto_select(root, Some(&head)).await.unwrap(); + assert!(matches!(selection.source, LocalSource::Clean { .. })); + assert!( + selection + .warning + .unwrap() + .contains("untracked files were not reviewed") + ); + } + + #[tokio::test] + async fn automatic_staged_review_matches_explicit_diff_and_searches_the_same_index_tree() { let directory = tempfile::tempdir().unwrap(); let root = directory.path(); run_git(root, &["init", "--quiet"]); @@ -193,19 +1177,23 @@ mod tests { std::fs::write(root.join("config.yaml"), "version: staged-one\n").unwrap(); run_git(root, &["add", "config.yaml"]); - let snapshot = acquire(&LocalSource::Staged, Some(&head), root) + let selection = auto_select(root, Some(&head)).await.unwrap(); + let automatic = acquire(&selection.source, Some(&head), root).await.unwrap(); + let explicit = acquire(&LocalSource::Staged, Some(&head), root) .await .unwrap(); + assert_eq!(automatic.diff.as_str(), explicit.diff.as_str()); + assert_eq!(automatic.repository_revision, explicit.repository_revision); std::fs::write(root.join("config.yaml"), "version: staged-two\n").unwrap(); run_git(root, &["add", "config.yaml"]); - let revision = snapshot.repository_revision.as_deref().unwrap(); + let revision = automatic.repository_revision.as_deref().unwrap(); assert_eq!( run_git(root, &["show", &format!("{revision}:config.yaml")]), "version: staged-one" ); - assert!(snapshot.diff.as_str().contains("+version: staged-one")); - assert!(!snapshot.diff.as_str().contains("staged-two")); + assert!(automatic.diff.as_str().contains("+version: staged-one")); + assert!(!automatic.diff.as_str().contains("staged-two")); let finding = crate::envelope::Finding { path: "config.yaml".into(), diff --git a/src/login.rs b/src/login.rs index 38fb95b..f864d7b 100644 --- a/src/login.rs +++ b/src/login.rs @@ -13,7 +13,8 @@ use anyhow::{Context, Result, anyhow}; use serde::Deserialize; use serde::de::DeserializeOwned; -use crate::credentials::{self, Credentials}; +use crate::config::normalize_api_base; +use crate::credentials::{self, Credentials, PendingRevocation}; use crate::llm::secure_http_client; /// Overrides the Postil web app the device-flow calls target. Distinct from @@ -21,6 +22,7 @@ use crate::llm::secure_http_client; /// inference gateway itself, not the auth endpoints. const LOGIN_SERVER_ENV: &str = "POSTIL_LOGIN_SERVER"; const DEFAULT_LOGIN_SERVER: &str = "https://postil.dev"; +const DEFAULT_LOGIN_API_BASE: &str = "https://postil.dev/api/inference/v1"; /// The server caps polling at 200 attempts and then returns 410 regardless; /// this is only a client-side backstop against a stuck loop if that cap is @@ -29,27 +31,138 @@ const MAX_POLL_ATTEMPTS: u32 = 220; const ACCESS_REFRESH_MARGIN: Duration = Duration::from_secs(5 * 60); const REFRESH_RESPONSE_MAX_BYTES: usize = 16 * 1024; const REFRESH_REQUEST_TIMEOUT: Duration = Duration::from_secs(20); +const REVOCATION_REQUEST_TIMEOUT: Duration = Duration::from_secs(20); -fn login_server() -> String { - std::env::var(LOGIN_SERVER_ENV) - .ok() - .map(|value| value.trim().to_string()) - .filter(|value| !value.is_empty()) - .unwrap_or_else(|| DEFAULT_LOGIN_SERVER.to_string()) +fn canonicalize_issuer(value: &str) -> Result { + anyhow::ensure!( + !value.trim().is_empty(), + "login server URL must not be empty" + ); + let mut url = + reqwest::Url::parse(value.trim()).context("login server must be an absolute URL")?; + anyhow::ensure!( + matches!(url.scheme(), "http" | "https"), + "login server must use HTTP or HTTPS" + ); + anyhow::ensure!( + url.username().is_empty() && url.password().is_none(), + "login server must not contain credentials" + ); + anyhow::ensure!( + url.host_str().is_some_and(|hostname| !hostname.is_empty()), + "login server must include a hostname" + ); + anyhow::ensure!( + url.query().is_none() && url.fragment().is_none(), + "login server must not contain a query or fragment" + ); + let path = url.path().trim_end_matches('/').to_string(); + url.set_path(if path.is_empty() { "/" } else { &path }); + Ok(url.as_str().trim_end_matches('/').to_string()) +} + +fn login_server_override() -> Result> { + let Some(_) = std::env::var_os(LOGIN_SERVER_ENV) else { + return Ok(None); + }; + let value = std::env::var(LOGIN_SERVER_ENV) + .with_context(|| format!("{LOGIN_SERVER_ENV} must contain a valid URL"))?; + canonicalize_issuer(&value) + .with_context(|| format!("invalid {LOGIN_SERVER_ENV}")) + .map(Some) +} + +fn login_server() -> Result { + login_server_override()?.map_or_else(|| canonicalize_issuer(DEFAULT_LOGIN_SERVER), Ok) +} + +fn stored_issuer(credentials: &Credentials) -> Result { + if let Some(issuer) = credentials.issuer.as_deref() { + return canonicalize_issuer(issuer) + .context("the stored login issuer is invalid; run `postil login` again"); + } + + let stored_api_base = normalize_api_base(&credentials.api_base) + .context("the stored login API base is invalid; run `postil login` again")?; + let canonical_api_base = normalize_api_base(DEFAULT_LOGIN_API_BASE) + .expect("the embedded Postil API base must be valid"); + anyhow::ensure!( + stored_api_base == canonical_api_base, + "the stored legacy login has no issuing server and its API base is not the canonical Postil endpoint; run `postil login` again" + ); + canonicalize_issuer(DEFAULT_LOGIN_SERVER) +} + +fn require_matching_issuer( + credentials: &Credentials, + configured_issuer: Option<&str>, +) -> Result { + let issuer = stored_issuer(credentials)?; + if let Some(configured_issuer) = configured_issuer { + let configured_issuer = canonicalize_issuer(configured_issuer) + .with_context(|| format!("invalid {LOGIN_SERVER_ENV}"))?; + anyhow::ensure!( + configured_issuer == issuer, + "{LOGIN_SERVER_ENV} does not match the stored login issuer; unset it to use this login, or run `postil login` to replace the login for that server" + ); + } + Ok(issuer) +} + +fn require_matching_api_base(credentials: &Credentials, resolved_api_base: &str) -> Result<()> { + let stored_api_base = normalize_api_base(&credentials.api_base) + .context("the stored login API base is invalid; run `postil login` again")?; + let resolved_api_base = + normalize_api_base(resolved_api_base).context("the resolved model API base is invalid")?; + if stored_api_base != resolved_api_base { + let key_names = crate::api_key::names_text(); + anyhow::bail!( + "the stored postil login is bound to {}; the resolved model API base is {}. Set an explicit key with {}, or remove the API base override to use this login", + credentials.api_base, + resolved_api_base, + key_names + ); + } + Ok(()) } pub async fn run_login(org: Option) -> Result { - let server = login_server(); + let server = login_server()?; let client = secure_http_client(&server).context("building the postil login HTTP client")?; let path = credentials::default_path()?; login_with(&client, &server, org.as_deref(), &path).await } pub async fn run_logout() -> Result { - let server = login_server(); - let client = secure_http_client(&server).context("building the postil logout HTTP client")?; + let configured_issuer = login_server_override()?; let path = credentials::default_path()?; - logout_with(&client, &server, &path).await + logout_with_mode(ClientMode::Secure, configured_issuer.as_deref(), &path).await +} + +#[derive(Clone, Copy)] +enum ClientMode<'a> { + Secure, + #[cfg(test)] + Provided(&'a reqwest::Client), + Reuse { + client: &'a reqwest::Client, + issuer: &'a str, + }, +} + +impl ClientMode<'_> { + fn for_issuer(self, issuer: &str) -> Result { + match self { + #[cfg(test)] + Self::Provided(client) => Ok(client.clone()), + Self::Reuse { + client, + issuer: client_issuer, + } if client_issuer == issuer => Ok(client.clone()), + Self::Secure | Self::Reuse { .. } => secure_http_client(issuer) + .context("building an HTTP client for stored login issuer"), + } + } } #[derive(Debug, Deserialize)] @@ -103,6 +216,7 @@ async fn login_with( org: Option<&str>, credentials_path: &Path, ) -> Result { + let server = canonicalize_issuer(server)?; let mut start_body = serde_json::json!({ "clientVersion": env!("CARGO_PKG_VERSION") }); // The device/start contract takes no org field today; sending it as an // extra, ignorable JSON key is a forward-compatible hint only. Org @@ -111,7 +225,7 @@ async fn login_with( if let Some(org) = org { start_body["org"] = serde_json::Value::String(org.to_string()); } - let start_url = format!("{}/api/cli/device/start", server.trim_end_matches('/')); + let start_url = format!("{server}/api/cli/device/start"); let start_response = client .post(&start_url) .json(&start_body) @@ -139,21 +253,24 @@ async fn login_with( eprintln!("postil: waiting for approval..."); let interval = Duration::from_secs(interval_secs(start.interval)); - let token_url = format!("{}/api/cli/device/token", server.trim_end_matches('/')); + let token_url = format!("{server}/api/cli/device/token"); for _ in 0..MAX_POLL_ATTEMPTS { tokio::time::sleep(interval).await; - let response = client + let response = match client .post(&token_url) .json(&serde_json::json!({ "deviceCode": start.device_code })) .send() .await - .context("polling postil login status")?; + { + Ok(response) => response, + Err(_) => continue, + }; match response.status().as_u16() { 200 => { - let approved: DeviceTokenApproved = response - .json() - .await - .context("parsing postil login approval response")?; + let approved: DeviceTokenApproved = match response.json().await { + Ok(approved) => approved, + Err(_) => continue, + }; let DeviceTokenApproved { token, expires_at, @@ -163,10 +280,11 @@ async fn login_with( org, model, } = approved; - let renewable = match (refresh_token, refresh_expires_at) { + let mut renewable = match (refresh_token, refresh_expires_at) { (Some(refresh_token), Some(refresh_expires_at)) => { let credentials = Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some(server.clone()), token, expires_at, refresh_token: Some(refresh_token), @@ -174,13 +292,15 @@ async fn login_with( api_base, org: org.slug, model, + pending_revocations: Vec::new(), }; validate_refresh_response(&credentials) .context("validating postil login approval response")?; credentials } (None, None) => Credentials { - version: credentials::LEGACY_CREDENTIALS_VERSION, + version: credentials::CREDENTIALS_VERSION, + issuer: Some(server.clone()), token, expires_at, refresh_token: None, @@ -188,14 +308,23 @@ async fn login_with( api_base, org: org.slug, model, + pending_revocations: Vec::new(), }, _ => anyhow::bail!( "the postil login approval response has incomplete renewable credentials" ), }; let renewable_login = renewable.refresh_token.is_some(); - let _lock = credentials::CredentialLock::acquire(credentials_path).await?; - credentials::write(credentials_path, &renewable)?; + let pending_count = persist_approved_login_with( + ClientMode::Reuse { + client, + issuer: &server, + }, + credentials_path, + &mut renewable, + credentials::write, + ) + .await?; if renewable_login { eprintln!( "postil: logged in to {} ({}): renewable until inactive after {}", @@ -212,9 +341,13 @@ async fn login_with( org.name, renewable.org ); } + if pending_count > 0 { + eprintln!("postil: a previous login is pending revocation and will be retried"); + } return Ok(0); } 428 => continue, + 408 | 429 | 500..=599 => continue, 403 => { eprintln!("postil: login request was denied"); return Ok(1); @@ -239,64 +372,134 @@ fn interval_secs(server_interval: u64) -> u64 { server_interval.clamp(1, 30) } +#[cfg(test)] async fn logout_with( client: &reqwest::Client, - server: &str, + configured_issuer: &str, + credentials_path: &Path, +) -> Result { + logout_with_mode( + ClientMode::Provided(client), + Some(configured_issuer), + credentials_path, + ) + .await +} + +async fn logout_with_mode( + client_mode: ClientMode<'_>, + configured_issuer: Option<&str>, credentials_path: &Path, ) -> Result { let _lock = credentials::CredentialLock::acquire(credentials_path).await?; - match credentials::read(credentials_path) { - Ok(Some(creds)) => { - let logout_url = format!("{}/api/cli/logout", server.trim_end_matches('/')); - let request = client.post(&logout_url).bearer_auth(&creds.token); - let request = if let Some(refresh_token) = creds - .refresh_token - .as_deref() - .filter(|token| !token.trim().is_empty()) - { - request.json(&serde_json::json!({ "refreshToken": refresh_token })) - } else { - request - }; - match request.send().await { - Ok(response) if response.status().is_success() => {} - Ok(response) => eprintln!( - "postil: logout request returned {}; removing the local credential anyway", - response.status() - ), - Err(_) => eprintln!( - "postil: could not reach postil to revoke the credential; removing the local credential anyway" - ), - } + let Some(mut stored) = credentials::read(credentials_path)? else { + let mut pending = credentials::read_pending(credentials_path)?; + if pending.is_empty() { + eprintln!("postil: not logged in"); + return Ok(0); } - Ok(None) => eprintln!("postil: not logged in"), - Err(_) => eprintln!("postil: stored credentials were unreadable; removing them anyway"), + let remaining = + drain_pending_without_active_locked(client_mode, credentials_path, &mut pending) + .await?; + if remaining == 0 { + eprintln!("postil: pending session revocations completed; no login is stored"); + return Ok(0); + } + eprintln!("postil: a previous session could not be revoked; run `postil logout` again"); + return Ok(1); + }; + + let issuer = require_matching_issuer(&stored, configured_issuer)?; + let mut detached = credentials::read_pending(credentials_path)?; + if !detached.is_empty() { + stored.pending_revocations.append(&mut detached); + deduplicate_revocations(&mut stored.pending_revocations); + credentials::write(credentials_path, &stored)?; + } + + let current = revocation_for(&stored)?; + if !revoke(client_mode, ¤t).await { + eprintln!( + "postil: could not revoke the stored login at its issuing server; credentials were kept. Run `postil logout` again" + ); + return Ok(1); } - // Removal happens regardless of what happened above: a developer running - // `postil logout` wants the local secret gone even if the network is down. + + debug_assert_eq!(issuer, current.issuer); + let remaining = + drain_pending_for_active_locked(client_mode, credentials_path, &mut stored).await?; + if remaining == 0 { + credentials::remove(credentials_path)?; + credentials::write_pending(credentials_path, &[])?; + eprintln!("postil: logged out"); + return Ok(0); + } + + credentials::write_pending(credentials_path, &stored.pending_revocations)?; credentials::remove(credentials_path)?; - eprintln!("postil: logged out"); - Ok(0) + eprintln!( + "postil: the current login was revoked, but a previous session is still pending revocation; run `postil logout` again" + ); + Ok(1) } /// Resolves a stored login for a hosted request. Explicit API-key environment /// variables are handled by the caller before this function is reached, so a /// BYOK invocation never reads, locks, or refreshes a local Postil login. -pub(crate) async fn resolve_stored_token(credentials_path: &Path) -> Result> { - let server = login_server(); - let client = secure_http_client(&server).context("building the postil refresh HTTP client")?; - resolve_stored_token_with(&client, &server, credentials_path).await +pub(crate) async fn resolve_stored_token( + credentials_path: &Path, + resolved_api_base: &str, +) -> Result> { + let configured_issuer = login_server_override()?; + let result = resolve_stored_token_with_mode( + ClientMode::Secure, + configured_issuer.as_deref(), + credentials_path, + resolved_api_base, + ) + .await; + if result.is_ok() { + spawn_pending_revocation_drain(credentials_path); + } + result } +#[cfg(test)] async fn resolve_stored_token_with( client: &reqwest::Client, - server: &str, + configured_issuer: &str, credentials_path: &Path, +) -> Result> { + let resolved_api_base = credentials::read(credentials_path)? + .map(|credentials| credentials.api_base) + .unwrap_or_default(); + resolve_stored_token_with_mode( + ClientMode::Provided(client), + Some(configured_issuer), + credentials_path, + &resolved_api_base, + ) + .await +} + +async fn resolve_stored_token_with_mode( + client_mode: ClientMode<'_>, + configured_issuer: Option<&str>, + credentials_path: &Path, + resolved_api_base: &str, ) -> Result> { let Some(credentials) = credentials::read(credentials_path)? else { return Ok(None); }; - if credentials.refresh_token.is_some() && !credentials.can_refresh() { + require_matching_api_base(&credentials, resolved_api_base)?; + if credentials.refresh_token.is_none() { + if credentials.is_expired() { + anyhow::bail!("the stored postil login credential expired; run `postil login` again"); + } + return Ok(Some(credentials.token)); + } + require_matching_issuer(&credentials, configured_issuer)?; + if !credentials.can_refresh() { anyhow::bail!("the stored postil login can no longer be renewed; run `postil login` again"); } if !credentials.expires_within(ACCESS_REFRESH_MARGIN) { @@ -307,7 +510,9 @@ async fn resolve_stored_token_with( let Some(credentials) = credentials::read(credentials_path)? else { return Ok(None); }; - if credentials.refresh_token.is_some() && !credentials.can_refresh() { + require_matching_api_base(&credentials, resolved_api_base)?; + let issuer = require_matching_issuer(&credentials, configured_issuer)?; + if !credentials.can_refresh() { anyhow::bail!("the stored postil login can no longer be renewed; run `postil login` again"); } if !credentials.expires_within(ACCESS_REFRESH_MARGIN) { @@ -317,9 +522,11 @@ async fn resolve_stored_token_with( anyhow::bail!("the stored postil login credential expired; run `postil login` again"); }; - let refreshed = refresh_token_with(client, server, refresh_token).await?; + let client = client_mode.for_issuer(&issuer)?; + let refreshed = refresh_token_with(&client, &issuer, refresh_token).await?; let replacement = Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some(issuer), token: refreshed.token, expires_at: refreshed.expires_at, refresh_token: Some(refreshed.refresh_token), @@ -327,12 +534,191 @@ async fn resolve_stored_token_with( api_base: credentials.api_base, org: credentials.org, model: credentials.model, + pending_revocations: credentials.pending_revocations, }; validate_refresh_response(&replacement)?; credentials::write(credentials_path, &replacement)?; Ok(Some(replacement.token)) } +fn revocation_for(credentials: &Credentials) -> Result { + anyhow::ensure!( + !credentials.token.trim().is_empty(), + "the stored login cannot be revoked because its access credential is invalid; run `postil login` again" + ); + Ok(PendingRevocation { + issuer: stored_issuer(credentials)?, + token: credentials.token.clone(), + refresh_token: credentials + .refresh_token + .as_deref() + .filter(|token| !token.trim().is_empty()) + .map(str::to_string), + }) +} + +fn deduplicate_revocations(revocations: &mut Vec) { + let mut unique = Vec::with_capacity(revocations.len()); + for revocation in revocations.drain(..) { + if !unique.contains(&revocation) { + unique.push(revocation); + } + } + *revocations = unique; +} + +async fn persist_approved_login_with( + client_mode: ClientMode<'_>, + credentials_path: &Path, + approved: &mut Credentials, + write_active: F, +) -> Result +where + F: FnOnce(&Path, &Credentials) -> Result<()>, +{ + let _lock = credentials::CredentialLock::acquire(credentials_path).await?; + let new_revocation = revocation_for(approved)?; + let mut pending = credentials::read_pending(credentials_path)?; + pending.push(new_revocation.clone()); + deduplicate_revocations(&mut pending); + if let Err(write_error) = credentials::write_pending(credentials_path, &pending) { + if revoke(client_mode, &new_revocation).await { + return Err(write_error) + .context("could not retain the new login locally; its remote session was revoked"); + } + return Err(write_error).context( + "could not retain the new login or its revocation handle locally; the remote session may remain active", + ); + } + if let Some(existing) = credentials::read(credentials_path)? { + pending.extend(existing.pending_revocations.iter().cloned()); + pending.push(revocation_for(&existing)?); + } + deduplicate_revocations(&mut pending); + credentials::write_pending(credentials_path, &pending)?; + approved.pending_revocations = pending; + write_active(credentials_path, approved)?; + let pending_count = + match drain_pending_for_active_locked(client_mode, credentials_path, approved).await { + Ok(count) => count, + Err(_) => approved.pending_revocations.len(), + }; + Ok(pending_count) +} + +fn revocation_matches_active(revocation: &PendingRevocation, active: &Credentials) -> Result { + if canonicalize_issuer(&revocation.issuer)? != stored_issuer(active)? { + return Ok(false); + } + if revocation.token == active.token { + return Ok(true); + } + Ok(revocation + .refresh_token + .as_deref() + .is_some_and(|pending| active.refresh_token.as_deref() == Some(pending))) +} + +async fn revoke(client_mode: ClientMode<'_>, revocation: &PendingRevocation) -> bool { + if revocation.token.trim().is_empty() { + return false; + } + let Ok(issuer) = canonicalize_issuer(&revocation.issuer) else { + return false; + }; + let Ok(client) = client_mode.for_issuer(&issuer) else { + return false; + }; + let logout_url = format!("{issuer}/api/cli/logout"); + let request = client.post(&logout_url).bearer_auth(&revocation.token); + let request = if let Some(refresh_token) = revocation + .refresh_token + .as_deref() + .filter(|token| !token.trim().is_empty()) + { + request.json(&serde_json::json!({ "refreshToken": refresh_token })) + } else { + request + }; + matches!( + tokio::time::timeout(REVOCATION_REQUEST_TIMEOUT, request.send()).await, + Ok(Ok(response)) if response.status().is_success() + ) +} + +async fn drain_pending_for_active_locked( + client_mode: ClientMode<'_>, + credentials_path: &Path, + active: &mut Credentials, +) -> Result { + let mut detached = credentials::read_pending(credentials_path)?; + if !detached.is_empty() { + active.pending_revocations.append(&mut detached); + deduplicate_revocations(&mut active.pending_revocations); + credentials::write(credentials_path, active)?; + let _ = credentials::write_pending(credentials_path, &[]); + } + + for revocation in active.pending_revocations.clone() { + match revocation_matches_active(&revocation, active) { + Ok(true) => { + active + .pending_revocations + .retain(|item| item != &revocation); + credentials::write(credentials_path, active)?; + continue; + } + Ok(false) => {} + Err(_) => continue, + } + if revoke(client_mode, &revocation).await { + active + .pending_revocations + .retain(|item| item != &revocation); + credentials::write(credentials_path, active)?; + } + } + Ok(active.pending_revocations.len()) +} + +async fn drain_pending_without_active_locked( + client_mode: ClientMode<'_>, + credentials_path: &Path, + pending: &mut Vec, +) -> Result { + deduplicate_revocations(pending); + for revocation in pending.clone() { + if revoke(client_mode, &revocation).await { + pending.retain(|item| item != &revocation); + credentials::write_pending(credentials_path, pending)?; + } + } + Ok(pending.len()) +} + +fn spawn_pending_revocation_drain(credentials_path: &Path) { + let credentials_path = credentials_path.to_path_buf(); + tokio::spawn(async move { + let _ = drain_pending_revocations(&credentials_path).await; + }); +} + +async fn drain_pending_revocations(credentials_path: &Path) -> Result<()> { + let _lock = credentials::CredentialLock::acquire(credentials_path).await?; + match credentials::read(credentials_path)? { + Some(mut active) => { + drain_pending_for_active_locked(ClientMode::Secure, credentials_path, &mut active) + .await?; + } + None => { + let mut pending = credentials::read_pending(credentials_path)?; + drain_pending_without_active_locked(ClientMode::Secure, credentials_path, &mut pending) + .await?; + } + } + Ok(()) +} + async fn refresh_token_with( client: &reqwest::Client, server: &str, @@ -350,7 +736,24 @@ async fn refresh_token_with( .map_err(|_| anyhow!("could not refresh the stored postil login; try again"))? .map_err(|_| anyhow!("could not refresh the stored postil login; try again"))?; let status = response.status(); - if status.is_server_error() || matches!(status.as_u16(), 408 | 429) { + if status.as_u16() == 429 { + if let Some(retry_after) = response + .headers() + .get(reqwest::header::RETRY_AFTER) + .and_then(|value| value.to_str().ok()) + .filter(|value| { + !value.is_empty() + && value.bytes().all(|byte| byte.is_ascii_digit()) + && value.parse::().is_ok() + }) + { + anyhow::bail!( + "could not refresh the stored postil login; retry after {retry_after} seconds" + ); + } + anyhow::bail!("could not refresh the stored postil login; try again"); + } + if status.is_server_error() || status.as_u16() == 408 { anyhow::bail!("could not refresh the stored postil login; try again"); } if !status.is_success() { @@ -504,6 +907,7 @@ mod tests { assert_eq!(exit, 0); let stored = credentials::read(&credentials_path).unwrap().unwrap(); assert_eq!(stored.token, "pcli_test-token-not-a-real-secret"); + assert_eq!(stored.issuer.as_deref(), Some(server.uri().as_str())); assert_eq!(stored.org, "runatlas-is"); assert_eq!(stored.model, "z-ai/glm-5.2"); assert_eq!( @@ -580,7 +984,7 @@ mod tests { ); let stored = credentials::read(&credentials_path).unwrap().unwrap(); - assert_eq!(stored.version, 1); + assert_eq!(stored.version, credentials::CREDENTIALS_VERSION); assert!(stored.refresh_token.is_none()); assert!(stored.refresh_expires_at.is_none()); } @@ -622,6 +1026,163 @@ mod tests { assert!(credentials::read(&credentials_path).unwrap().is_some()); } + #[tokio::test] + async fn login_retries_an_ambiguous_server_failure_with_the_same_device_code() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/api/cli/device/start")) + .respond_with(ResponseTemplate::new(200).set_body_json(start_body())) + .mount(&server) + .await; + let responder = SequentialTokenResponder { + calls: Arc::new(AtomicUsize::new(0)), + responses: Arc::new(vec![ + (500, serde_json::json!({"status": "uncertain"})), + (200, approved_body()), + ]), + }; + Mock::given(method("POST")) + .and(path("/api/cli/device/token")) + .respond_with(responder.clone()) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + assert_eq!( + login_with( + &reqwest::Client::new(), + &server.uri(), + None, + &credentials_path, + ) + .await + .unwrap(), + 0 + ); + assert_eq!(responder.calls.load(Ordering::SeqCst), 2); + let requests = server.received_requests().await.unwrap(); + let device_codes = requests + .iter() + .filter(|request| request.url.path() == "/api/cli/device/token") + .map(|request| request.body_json::().unwrap()["deviceCode"].clone()) + .collect::>(); + assert_eq!( + device_codes, + vec![ + serde_json::json!("test-device-code"), + serde_json::json!("test-device-code") + ] + ); + } + + #[tokio::test] + async fn login_write_failure_retains_the_new_family_revocation_handle() { + let server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let mut approved = stored_credentials_at(&server.uri(), "2999-01-01T00:00:00.000Z"); + approved.token = "pcli_test-token-not-a-real-secret".to_string(); + approved.refresh_token = Some("fixture-refresh-not-a-credential".to_string()); + let client = reqwest::Client::new(); + let error = persist_approved_login_with( + ClientMode::Provided(&client), + &credentials_path, + &mut approved, + |_path, _credentials| anyhow::bail!("simulated active credential write failure"), + ) + .await + .expect_err("installing the active credential should fail"); + + assert!( + error + .to_string() + .contains("simulated active credential write failure") + ); + assert!(credentials::read(&credentials_path).unwrap().is_none()); + let pending = credentials::read_pending(&credentials_path).unwrap(); + assert_eq!(pending.len(), 1); + assert_eq!(pending[0].issuer, server.uri()); + assert_eq!(pending[0].token, "pcli_test-token-not-a-real-secret"); + assert_eq!( + pending[0].refresh_token.as_deref(), + Some("fixture-refresh-not-a-credential") + ); + } + + #[tokio::test] + async fn relogin_retains_a_failed_old_family_revocation_until_retry_succeeds() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/api/cli/device/start")) + .respond_with(ResponseTemplate::new(200).set_body_json(start_body())) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/api/cli/device/token")) + .respond_with(ResponseTemplate::new(200).set_body_json(approved_body())) + .mount(&server) + .await; + let revocations = SequentialTokenResponder { + calls: Arc::new(AtomicUsize::new(0)), + responses: Arc::new(vec![ + (503, serde_json::json!({"status": "retry"})), + (200, serde_json::json!({"status": "revoked"})), + ]), + }; + Mock::given(method("POST")) + .and(path("/api/cli/logout")) + .respond_with(revocations.clone()) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + credentials::write( + &credentials_path, + &stored_credentials_at(&server.uri(), "2999-01-01T00:00:00.000Z"), + ) + .unwrap(); + let client = reqwest::Client::new(); + + assert_eq!( + login_with(&client, &server.uri(), None, &credentials_path) + .await + .unwrap(), + 0 + ); + let mut stored = credentials::read(&credentials_path).unwrap().unwrap(); + assert_eq!(stored.token, "pcli_test-token-not-a-real-secret"); + assert_eq!(stored.pending_revocations.len(), 1); + assert_eq!( + stored.pending_revocations[0].refresh_token.as_deref(), + Some("fixture-old-refresh-not-a-credential") + ); + assert_eq!(stored.pending_revocations[0].issuer, server.uri()); + + let _lock = credentials::CredentialLock::acquire(&credentials_path) + .await + .unwrap(); + assert_eq!( + drain_pending_for_active_locked( + ClientMode::Provided(&client), + &credentials_path, + &mut stored, + ) + .await + .unwrap(), + 0 + ); + assert!( + credentials::read(&credentials_path) + .unwrap() + .unwrap() + .pending_revocations + .is_empty() + ); + assert_eq!(revocations.calls.load(Ordering::SeqCst), 2); + } + #[tokio::test] async fn login_reports_denial_and_writes_nothing() { let server = MockServer::start().await; @@ -682,11 +1243,27 @@ mod tests { assert!(credentials::read(&credentials_path).unwrap().is_none()); } + #[test] + fn issuer_normalization_removes_default_ports_and_trailing_slashes() { + assert_eq!( + canonicalize_issuer(" HTTPS://Example.COM:443/auth/// ").unwrap(), + "https://example.com/auth" + ); + assert_eq!( + canonicalize_issuer("http://Example.COM:80/").unwrap(), + "http://example.com" + ); + } + #[tokio::test] - async fn logout_removes_credentials_even_when_the_server_call_fails() { - // No mock mounted for /api/cli/logout at all: every request to this - // server 404s, standing in for "the network call failed." + async fn transient_logout_failure_preserves_the_revocation_handle() { let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/api/cli/logout")) + .respond_with(ResponseTemplate::new(503)) + .expect(1) + .mount(&server) + .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); @@ -694,6 +1271,7 @@ mod tests { &credentials_path, &Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some(server.uri()), token: "pcli_test-token-not-a-real-secret".to_string(), expires_at: "2999-01-01T00:00:00.000Z".to_string(), refresh_token: Some("fixture-refresh-not-a-credential".to_string()), @@ -701,6 +1279,7 @@ mod tests { api_base: "https://postil.dev/api/inference/v1".to_string(), org: "runatlas-is".to_string(), model: "z-ai/glm-5.2".to_string(), + pending_revocations: Vec::new(), }, ) .unwrap(); @@ -708,8 +1287,86 @@ mod tests { let exit = logout_with(&reqwest::Client::new(), &server.uri(), &credentials_path) .await .unwrap(); - assert_eq!(exit, 0); - assert!(!credentials_path.exists()); + assert_eq!(exit, 1); + assert!(credentials_path.exists()); + let stored = credentials::read(&credentials_path).unwrap().unwrap(); + assert_eq!( + stored.refresh_token.as_deref(), + Some("fixture-refresh-not-a-credential") + ); + } + + #[tokio::test] + async fn explicit_issuer_conflict_sends_no_refresh_or_logout_credential() { + let issuing_server = MockServer::start().await; + let conflicting_server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let original = stored_credentials_at(&issuing_server.uri(), "2020-01-01T00:00:00.000Z"); + credentials::write(&credentials_path, &original).unwrap(); + let client = reqwest::Client::new(); + let conflicting_issuer = format!("{}/", conflicting_server.uri()); + + let refresh_error = resolve_stored_token_with_mode( + ClientMode::Provided(&client), + Some(&conflicting_issuer), + &credentials_path, + &original.api_base, + ) + .await + .expect_err("a conflicting issuer must stop refresh"); + let logout_error = logout_with_mode( + ClientMode::Provided(&client), + Some(&conflicting_issuer), + &credentials_path, + ) + .await + .expect_err("a conflicting issuer must stop logout"); + + assert!(refresh_error.to_string().contains(LOGIN_SERVER_ENV)); + assert!(refresh_error.to_string().contains("postil login")); + assert!(logout_error.to_string().contains(LOGIN_SERVER_ENV)); + assert!(issuing_server.received_requests().await.unwrap().is_empty()); + assert!( + conflicting_server + .received_requests() + .await + .unwrap() + .is_empty() + ); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + original + ); + } + + #[tokio::test] + async fn api_base_conflict_sends_no_stored_credential() { + let issuing_server = MockServer::start().await; + let other_api = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let original = stored_credentials_at(&issuing_server.uri(), "2020-01-01T00:00:00.000Z"); + credentials::write(&credentials_path, &original).unwrap(); + + let error = resolve_stored_token_with_mode( + ClientMode::Provided(&reqwest::Client::new()), + None, + &credentials_path, + &other_api.uri(), + ) + .await + .expect_err("a stored bearer must remain bound to its API base"); + + let message = error.to_string(); + assert!(message.contains("bound to")); + assert!(message.contains("explicit key")); + assert!(issuing_server.received_requests().await.unwrap().is_empty()); + assert!(other_api.received_requests().await.unwrap().is_empty()); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + original + ); } #[tokio::test] @@ -737,7 +1394,7 @@ mod tests { let credentials_path = dir.path().join("postil").join("credentials.json"); credentials::write( &credentials_path, - &stored_credentials("2999-01-01T00:00:00.000Z"), + &stored_credentials_at(&server.uri(), "2999-01-01T00:00:00.000Z"), ) .unwrap(); let held_lock = credentials::CredentialLock::acquire(&credentials_path) @@ -766,6 +1423,7 @@ mod tests { fn stored_credentials(expires_at: &str) -> Credentials { Credentials { version: credentials::CREDENTIALS_VERSION, + issuer: Some(DEFAULT_LOGIN_SERVER.to_string()), token: "pcli_test-old-access-not-a-real-secret".to_string(), expires_at: expires_at.to_string(), refresh_token: Some("fixture-old-refresh-not-a-credential".to_string()), @@ -773,9 +1431,16 @@ mod tests { api_base: "https://postil.dev/api/inference/v1".to_string(), org: "runatlas-is".to_string(), model: "z-ai/glm-5.2".to_string(), + pending_revocations: Vec::new(), } } + fn stored_credentials_at(issuer: &str, expires_at: &str) -> Credentials { + let mut credentials = stored_credentials(expires_at); + credentials.issuer = Some(canonicalize_issuer(issuer).unwrap()); + credentials + } + fn write_legacy_credentials(path: &Path, credentials: &Credentials) { std::fs::create_dir_all(path.parent().unwrap()).unwrap(); std::fs::write(path, serde_json::to_string(credentials).unwrap()).unwrap(); @@ -802,19 +1467,25 @@ mod tests { let credentials_path = dir.path().join("postil").join("credentials.json"); credentials::write( &credentials_path, - &stored_credentials("2020-01-01T00:00:00.000Z"), + &stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"), ) .unwrap(); - let resolved = - resolve_stored_token_with(&reqwest::Client::new(), &server.uri(), &credentials_path) - .await - .unwrap(); + let client = reqwest::Client::new(); + let resolved = resolve_stored_token_with_mode( + ClientMode::Provided(&client), + None, + &credentials_path, + "https://postil.dev/api/inference/v1", + ) + .await + .unwrap(); assert_eq!( resolved.as_deref(), Some("pcli_test-new-access-not-a-real-secret") ); let stored = credentials::read(&credentials_path).unwrap().unwrap(); + assert_eq!(stored.issuer.as_deref(), Some(server.uri().as_str())); assert_eq!(stored.token, "pcli_test-new-access-not-a-real-secret"); assert_eq!( stored.refresh_token.as_deref(), @@ -836,7 +1507,11 @@ mod tests { .unwrap(); let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - credentials::write(&credentials_path, &stored_credentials(&expires_at)).unwrap(); + credentials::write( + &credentials_path, + &stored_credentials_at(&server.uri(), &expires_at), + ) + .unwrap(); resolve_stored_token_with(&reqwest::Client::new(), &server.uri(), &credentials_path) .await @@ -872,7 +1547,7 @@ mod tests { let credentials_path = dir.path().join("postil").join("credentials.json"); credentials::write( &credentials_path, - &stored_credentials("2020-01-01T00:00:00.000Z"), + &stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"), ) .unwrap(); let client = reqwest::Client::new(); @@ -903,7 +1578,7 @@ mod tests { .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = @@ -929,7 +1604,7 @@ mod tests { .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = @@ -954,7 +1629,7 @@ mod tests { .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = @@ -969,19 +1644,116 @@ mod tests { ); } + #[tokio::test] + async fn refresh_record_without_issuer_is_bound_to_the_public_default() { + let override_server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let mut legacy_refresh = stored_credentials("2020-01-01T00:00:00.000Z"); + legacy_refresh.version = credentials::LEGACY_REFRESH_CREDENTIALS_VERSION; + legacy_refresh.issuer = None; + credentials::write(&credentials_path, &legacy_refresh).unwrap(); + + assert_eq!( + stored_issuer(&legacy_refresh).unwrap(), + DEFAULT_LOGIN_SERVER + ); + let error = resolve_stored_token_with( + &reqwest::Client::new(), + &override_server.uri(), + &credentials_path, + ) + .await + .expect_err("an old refresh credential must not follow an issuer override"); + + assert!(error.to_string().contains(LOGIN_SERVER_ENV)); + assert!(error.to_string().contains("postil login")); + assert!( + override_server + .received_requests() + .await + .unwrap() + .is_empty() + ); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + legacy_refresh + ); + } + + #[tokio::test] + async fn custom_refresh_record_without_issuer_fails_before_network() { + let custom_server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let mut legacy_refresh = stored_credentials("2020-01-01T00:00:00.000Z"); + legacy_refresh.version = credentials::LEGACY_REFRESH_CREDENTIALS_VERSION; + legacy_refresh.issuer = None; + legacy_refresh.api_base = format!("{}/api/inference/v1", custom_server.uri()); + credentials::write(&credentials_path, &legacy_refresh).unwrap(); + + let error = resolve_stored_token_with_mode( + ClientMode::Provided(&reqwest::Client::new()), + None, + &credentials_path, + &legacy_refresh.api_base, + ) + .await + .expect_err("an issuer-free custom refresh credential must require login"); + + assert!(error.to_string().contains("no issuing server")); + assert!(error.to_string().contains("postil login")); + assert!(custom_server.received_requests().await.unwrap().is_empty()); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + legacy_refresh + ); + } + + #[tokio::test] + async fn custom_access_record_without_issuer_refuses_logout_before_network() { + let custom_server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let mut legacy_access = stored_credentials("2999-01-01T00:00:00.000Z"); + legacy_access.version = 1; + legacy_access.issuer = None; + legacy_access.refresh_token = None; + legacy_access.refresh_expires_at = None; + legacy_access.api_base = format!("{}/api/inference/v1", custom_server.uri()); + write_legacy_credentials(&credentials_path, &legacy_access); + + let error = logout_with_mode( + ClientMode::Provided(&reqwest::Client::new()), + None, + &credentials_path, + ) + .await + .expect_err("an issuer-free custom access credential must require login"); + + assert!(error.to_string().contains("no issuing server")); + assert!(error.to_string().contains("postil login")); + assert!(custom_server.received_requests().await.unwrap().is_empty()); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + legacy_access + ); + } + #[tokio::test] async fn expired_v1_login_requires_relogin() { let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); let mut legacy = stored_credentials("2020-01-01T00:00:00.000Z"); legacy.version = 1; + legacy.issuer = None; legacy.refresh_token = None; legacy.refresh_expires_at = None; write_legacy_credentials(&credentials_path, &legacy); let error = resolve_stored_token_with( &reqwest::Client::new(), - "http://127.0.0.1:9", + DEFAULT_LOGIN_SERVER, &credentials_path, ) .await @@ -995,13 +1767,14 @@ mod tests { let credentials_path = dir.path().join("postil").join("credentials.json"); let mut legacy = stored_credentials("2999-01-01T00:00:00.000Z"); legacy.version = 1; + legacy.issuer = None; legacy.refresh_token = None; legacy.refresh_expires_at = None; write_legacy_credentials(&credentials_path, &legacy); let resolved = resolve_stored_token_with( &reqwest::Client::new(), - "http://127.0.0.1:9", + "https://self-hosted.example.test", &credentials_path, ) .await @@ -1022,7 +1795,7 @@ mod tests { let error = resolve_stored_token_with( &reqwest::Client::new(), - "http://127.0.0.1:9", + DEFAULT_LOGIN_SERVER, &credentials_path, ) .await @@ -1043,11 +1816,12 @@ mod tests { let credentials_path = dir.path().join("postil").join("credentials.json"); credentials::write( &credentials_path, - &stored_credentials("2020-01-01T00:00:00.000Z"), + &stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"), ) .unwrap(); - logout_with(&reqwest::Client::new(), &server.uri(), &credentials_path) + let client = reqwest::Client::new(); + logout_with_mode(ClientMode::Provided(&client), None, &credentials_path) .await .unwrap(); assert!(!credentials_path.exists()); @@ -1072,7 +1846,7 @@ mod tests { .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let mut legacy = stored_credentials("2999-01-01T00:00:00.000Z"); + let mut legacy = stored_credentials_at(&server.uri(), "2999-01-01T00:00:00.000Z"); legacy.version = 1; legacy.refresh_token = None; legacy.refresh_expires_at = None; @@ -1094,19 +1868,23 @@ mod tests { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/api/cli/token/refresh")) - .respond_with(ResponseTemplate::new(429).set_body_string("retry later")) + .respond_with( + ResponseTemplate::new(429) + .insert_header("Retry-After", "3527") + .set_body_string("retry later"), + ) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = resolve_stored_token_with(&reqwest::Client::new(), &server.uri(), &credentials_path) .await .expect_err("rate-limited refreshes must fail closed without relogin"); - assert!(error.to_string().contains("try again")); + assert!(error.to_string().contains("retry after 3527 seconds")); assert!(!error.to_string().contains("run `postil login` again")); assert_eq!( credentials::read(&credentials_path).unwrap().unwrap(), @@ -1114,6 +1892,41 @@ mod tests { ); } + #[tokio::test] + async fn malformed_or_missing_retry_after_uses_a_safe_generic_error() { + for retry_after in [None, Some("tomorrow"), Some("18446744073709551616")] { + let server = MockServer::start().await; + let response = ResponseTemplate::new(429).set_body_string("retry later"); + let response = match retry_after { + Some(value) => response.insert_header("Retry-After", value), + None => response, + }; + Mock::given(method("POST")) + .and(path("/api/cli/token/refresh")) + .respond_with(response) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let credentials_path = dir.path().join("postil").join("credentials.json"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); + credentials::write(&credentials_path, &original).unwrap(); + + let error = resolve_stored_token_with( + &reqwest::Client::new(), + &server.uri(), + &credentials_path, + ) + .await + .expect_err("an unusable Retry-After must fail safely"); + assert!(error.to_string().contains("try again")); + assert!(!error.to_string().contains("retry after")); + assert_eq!( + credentials::read(&credentials_path).unwrap().unwrap(), + original + ); + } + } + #[tokio::test] async fn request_timeout_refresh_keeps_the_previous_credential() { let server = MockServer::start().await; @@ -1124,7 +1937,7 @@ mod tests { .await; let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = stored_credentials_at(&server.uri(), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = @@ -1157,7 +1970,8 @@ mod tests { }); let dir = tempfile::tempdir().unwrap(); let credentials_path = dir.path().join("postil").join("credentials.json"); - let original = stored_credentials("2020-01-01T00:00:00.000Z"); + let original = + stored_credentials_at(&format!("http://{address}"), "2020-01-01T00:00:00.000Z"); credentials::write(&credentials_path, &original).unwrap(); let error = resolve_stored_token_with( diff --git a/src/main.rs b/src/main.rs index fbc3c29..c25e224 100644 --- a/src/main.rs +++ b/src/main.rs @@ -5,9 +5,12 @@ use postil_cli::attribution; use postil_cli::cli::{ Cli, Command, ForgeArg, HookAction, publication_enabled, publication_plan_contract_capability, }; -use postil_cli::config::{Config, qualification_metadata, starter_config}; +use postil_cli::config::{ + Config, REASONING_EFFORT_VALUES, default_reasoning_effort, default_scorer_model, + default_scorer_reasoning_effort, qualification_metadata, starter_config, +}; use postil_cli::review::{ForgeKind, ReviewArgs}; -use postil_cli::{doctor, hook, login, plan, respond, review}; +use postil_cli::{doctor, hook, login, plan, review}; #[tokio::main] async fn main() { @@ -61,6 +64,10 @@ async fn dispatch(cli: Cli) -> anyhow::Result { fail_on, config, model, + reasoning_effort, + scorer_reasoning_effort, + verbose, + no_progress, bounded, publish, no_post, @@ -100,6 +107,10 @@ async fn dispatch(cli: Cli) -> anyhow::Result { fail_on, config, model, + reasoning_effort, + scorer_reasoning_effort, + verbose, + no_progress, bounded, no_post: !publication_enabled(publish, no_post)?, defer_gate_check, @@ -109,35 +120,78 @@ async fn dispatch(cli: Cli) -> anyhow::Result { }) .await } - Command::Respond { - forge, - repo, - pr, - issue, - comment, - config, - model, - publish, - no_post, - } => { - let kind = match forge { - ForgeArg::Github => ForgeKind::GitHub, - ForgeArg::Gitlab => ForgeKind::GitLab, - ForgeArg::Bitbucket => ForgeKind::Bitbucket, - ForgeArg::Azure => ForgeKind::Azure, - ForgeArg::Local => ForgeKind::Local, + Command::Models => { + let qualification = qualification_metadata(); + let cascade = postil_cli::config::default_cascade(); + let cascade_text = if cascade.is_empty() { + "none".to_string() + } else { + cascade.join(" -> ") }; - respond::run(respond::RespondArgs { - forge: kind, - repo, - pr, - issue, - comment, - config, - model, - no_post: !publication_enabled(publish, no_post)?, - }) - .await + println!("Postil model support (offline)"); + println!(" No model setting is required."); + println!( + " Embedded local reviewer (Luna): {}", + postil_cli::config::default_model() + ); + println!(" Reviewer source: embedded default"); + println!(" Fallback cascade: {cascade_text}"); + println!( + " Reviewer reasoning effort: {}", + default_reasoning_effort().as_str() + ); + println!(" Local scorer: disabled (set REVIEW_SCORER_MODEL to enable)"); + println!(" Hosted scorer candidate: {}", default_scorer_model()); + println!( + " Hosted scorer reasoning effort: {}", + default_scorer_reasoning_effort().as_str() + ); + println!(" Accepted reasoning efforts: {REASONING_EFFORT_VALUES}"); + println!("\nAccepted local model IDs"); + println!(" Postil does not maintain a fixed local model-ID allowlist."); + println!( + " OpenAI-compatible endpoints accept any non-empty endpoint model ID and pass it unchanged; OpenRouter commonly uses provider/model." + ); + println!( + " Native Anthropic endpoints accept any non-empty Anthropic endpoint model ID and pass it unchanged, such as claude-* IDs." + ); + println!( + " Local compatibility means Postil can construct the endpoint protocol; it does not mean the model is hosted-qualified." + ); + println!("\nHosted qualification"); + if let Some(profile) = qualification.admitted_profile.as_ref() { + println!(" Embedded qualified profile: {}", profile.id); + println!( + " Qualified reviewer IDs: {}", + profile.generator_chain.join(" -> ") + ); + println!( + " Qualified scorer IDs: {}", + if profile.scorer_chain.is_empty() { + "none".to_string() + } else { + profile.scorer_chain.join(" -> ") + } + ); + } else { + println!(" Hosted qualified model IDs: none (no embedded qualified profile)"); + } + println!("\nCheck the configured endpoint and model: postil doctor"); + println!("Override once: postil review --model provider/model"); + println!( + "Override reasoning once: postil review --reasoning-effort high --scorer-reasoning-effort none" + ); + println!("Override persistently: REVIEW_MODEL=provider/model postil review"); + println!( + "Persist reasoning: REVIEW_REASONING_EFFORT=high REVIEW_SCORER_REASONING_EFFORT=none postil review" + ); + println!( + "Native Anthropic config: set model.apiBase, model.apiFormat: anthropic, and model.name, then run postil doctor" + ); + println!( + "Config keys: model.name, model.reasoningEffort, and model.scorerReasoningEffort" + ); + Ok(0) } Command::Plan { envelopes, config } => { let cwd = std::env::current_dir()?; @@ -196,8 +250,23 @@ async fn dispatch(cli: Cli) -> anyhow::Result { } ); println!("model.name: {}", cfg.model); + println!("model.source: {}", cfg.model_source); + println!("model.reasoningEffort: {}", cfg.reasoning_effort.as_str()); + println!( + "model.reasoningEffort.source: {}", + cfg.reasoning_effort_source + ); println!("model.cascade: {:?}", cfg.cascade); println!("model.scorer: {}", cfg.scorer); + println!("model.scorer.enabled: {}", cfg.scorer_enabled()); + println!( + "model.scorerReasoningEffort: {}", + cfg.scorer_reasoning_effort.as_str() + ); + println!( + "model.scorerReasoningEffort.source: {}", + cfg.scorer_reasoning_effort_source + ); println!("model.apiBase: {}", cfg.api_base); println!("model.apiFormat: {}", cfg.api_format.as_str()); println!("model.consensus: {}", cfg.consensus); diff --git a/src/output.rs b/src/output.rs index 5b3b270..b16afb4 100644 --- a/src/output.rs +++ b/src/output.rs @@ -153,7 +153,7 @@ fn csv_field(field: String) -> String { } pub fn print_pretty(envelope: &Envelope) { - let color = std::io::stderr().is_terminal(); + let color = std::io::stderr().is_terminal() && std::env::var_os("NO_COLOR").is_none(); let mut out = String::new(); if envelope.silent { diff --git a/src/progress.rs b/src/progress.rs new file mode 100644 index 0000000..daad824 --- /dev/null +++ b/src/progress.rs @@ -0,0 +1,209 @@ +//! Human-terminal review progress. Machine and hosted paths retain telemetry. + +use std::fmt::Arguments; +use std::io::{IsTerminal, Write}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + +static SUPPRESS_TELEMETRY: AtomicBool = AtomicBool::new(false); +static REDRAW: AtomicBool = AtomicBool::new(false); +static DEGRADED: AtomicBool = AtomicBool::new(false); +static TICK: AtomicUsize = AtomicUsize::new(0); + +pub struct ProgressGuard { + active: bool, +} + +pub struct OutputSuspension { + redraw: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReviewCompletion { + Clean, + AdvisoryFindings, + BlockingFindings, + ProviderIncomplete, + ModelOutputIncomplete, + OperationalIncomplete, +} + +pub fn start_review( + machine_output: bool, + verbose: bool, + hosted: bool, + no_progress: bool, +) -> ProgressGuard { + let ci = std::env::var_os("CI").is_some(); + let debug = + std::env::var_os("RUST_LOG").is_some() || std::env::var_os("POSTIL_DEBUG").is_some(); + let tty = std::io::stdout().is_terminal() && std::io::stderr().is_terminal(); + let dumb = std::env::var("TERM").ok().as_deref() == Some("dumb"); + let no_color = std::env::var_os("NO_COLOR").is_some(); + let static_progress = no_progress + || std::env::var_os("POSTIL_NO_PROGRESS").is_some_and(|value| !value.is_empty()); + let suppress = tty && !machine_output && !verbose && !hosted && !ci && !debug; + SUPPRESS_TELEMETRY.store(suppress, Ordering::Relaxed); + REDRAW.store( + suppress && !static_progress && !dumb && !no_color, + Ordering::Relaxed, + ); + DEGRADED.store(false, Ordering::Relaxed); + if suppress { + if REDRAW.load(Ordering::Relaxed) { + draw(); + } else { + eprintln!("postil: reviewing changes..."); + } + } + ProgressGuard { active: suppress } +} + +pub fn telemetry(message: Arguments<'_>) { + if !SUPPRESS_TELEMETRY.load(Ordering::Relaxed) { + eprintln!("{message}"); + } else if REDRAW.load(Ordering::Relaxed) { + draw(); + } +} + +/// Keep safety-relevant notices visible even while detailed telemetry is +/// collapsed into an interactive progress line. +pub fn notice(message: Arguments<'_>) { + DEGRADED.store(true, Ordering::Relaxed); + if SUPPRESS_TELEMETRY.load(Ordering::Relaxed) { + clear(); + eprintln!("{message}"); + if REDRAW.load(Ordering::Relaxed) { + draw(); + } + } else { + eprintln!("{message}"); + } +} + +/// Remove the transient progress line while durable review output is written. +/// The line resumes afterward if the review still has work to complete. +pub fn suspend_for_output() -> OutputSuspension { + let redraw = SUPPRESS_TELEMETRY.load(Ordering::Relaxed) && REDRAW.load(Ordering::Relaxed); + if redraw { + clear(); + REDRAW.store(false, Ordering::Relaxed); + } + OutputSuspension { redraw } +} + +impl Drop for OutputSuspension { + fn drop(&mut self) { + if self.redraw && SUPPRESS_TELEMETRY.load(Ordering::Relaxed) { + REDRAW.store(true, Ordering::Relaxed); + draw(); + } + } +} + +impl ProgressGuard { + pub fn finish(mut self, completion: ReviewCompletion) { + if self.active { + clear(); + eprintln!( + "postil: {}", + completion_message(completion, DEGRADED.load(Ordering::Relaxed)) + ); + self.active = false; + } + } +} + +impl Drop for ProgressGuard { + fn drop(&mut self) { + if self.active { + clear(); + } + SUPPRESS_TELEMETRY.store(false, Ordering::Relaxed); + REDRAW.store(false, Ordering::Relaxed); + DEGRADED.store(false, Ordering::Relaxed); + } +} + +fn completion_message(completion: ReviewCompletion, degraded: bool) -> String { + let message = match completion { + ReviewCompletion::Clean => "review complete; no findings", + ReviewCompletion::AdvisoryFindings => "review complete; advisory findings reported", + ReviewCompletion::BlockingFindings => "review found blocking findings", + ReviewCompletion::ProviderIncomplete => "review incomplete; model provider unavailable", + ReviewCompletion::ModelOutputIncomplete => { + "review incomplete; model output could not be validated" + } + ReviewCompletion::OperationalIncomplete => { + "review incomplete; an operational error prevented completion" + } + }; + if degraded { + format!("{message}; warnings were also reported") + } else { + message.to_string() + } +} + +fn draw() { + let frames = ["⠋", "⠙", "⠹", "⠸"]; + let frame = frames[TICK.fetch_add(1, Ordering::Relaxed) % frames.len()]; + let _ = write!(std::io::stderr(), "\r\x1b[2K{frame} Reviewing changes..."); + let _ = std::io::stderr().flush(); +} + +fn clear() { + if REDRAW.load(Ordering::Relaxed) { + let _ = write!(std::io::stderr(), "\r\x1b[2K"); + let _ = std::io::stderr().flush(); + } +} + +#[cfg(test)] +mod tests { + use super::{ReviewCompletion, completion_message}; + + #[test] + fn degraded_reviews_preserve_the_specific_completion_state() { + assert_eq!( + completion_message(ReviewCompletion::Clean, true), + "review complete; no findings; warnings were also reported" + ); + assert_eq!( + completion_message(ReviewCompletion::BlockingFindings, true), + "review found blocking findings; warnings were also reported" + ); + assert_eq!( + completion_message(ReviewCompletion::ProviderIncomplete, true), + "review incomplete; model provider unavailable; warnings were also reported" + ); + } + + #[test] + fn completion_states_distinguish_verdicts_from_incomplete_reviews() { + assert_eq!( + completion_message(ReviewCompletion::Clean, false), + "review complete; no findings" + ); + assert_eq!( + completion_message(ReviewCompletion::AdvisoryFindings, false), + "review complete; advisory findings reported" + ); + assert_eq!( + completion_message(ReviewCompletion::BlockingFindings, false), + "review found blocking findings" + ); + assert_eq!( + completion_message(ReviewCompletion::ProviderIncomplete, false), + "review incomplete; model provider unavailable" + ); + assert_eq!( + completion_message(ReviewCompletion::ModelOutputIncomplete, false), + "review incomplete; model output could not be validated" + ); + assert_eq!( + completion_message(ReviewCompletion::OperationalIncomplete, false), + "review incomplete; an operational error prevented completion" + ); + } +} diff --git a/src/prompt.rs b/src/prompt.rs index b53e9dd..3cdcea2 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -326,44 +326,6 @@ pub(crate) fn sanitize_scorer_input(value: &str) -> String { .collect() } -/// System prompt for the interactive bot answering a maintainer's mention. -/// The small JSON envelope keeps generated prose behind a deterministic -/// publication check before it can reach a forge. -pub fn respond_system_prompt(cfg: &Config, current_utc_date: Date) -> String { - let mut p = String::from( - "You are Postil, replying to a maintainer who mentioned you on a pull request or \ - issue. Answer the actual question directly. Ground every claim in the diff or thread \ - you are given, and cite file:line when you reference code. If something cannot be \ - determined from the supplied context, say so plainly rather than guessing. No filler, \ - praise, preamble, or restatement of the question. You do not open pull requests or push \ - commits; if asked to, explain that you review and answer only.\n\ - \n\ - Keep an ordinary reply at or below 1,200 characters. A re-review reply is a compact \ - review, not an article: report only actionable merge risks, give each risk in one \ - concise item with its file:line evidence and next action, and say briefly when no such \ - risk is present. Do not add an overview, implementation tour, correctness section, \ - generic risk inventory, or verdict. Do not use Markdown headings. Use no more than three \ - list items.\n\ - Do not emit active @mentions, raw HTML or HTML comments, details blocks, Markdown \ - tables, or images.\n\ - \n\ - Return ONLY one JSON object with exactly this shape and no markdown fence or surrounding \ - prose:\n\ - {\"answer\":\"concise GitHub-flavored Markdown\",\"diagram\":null}\n\ - The answer must be non-empty and diagram must always be null. Generated diagrams and \ - Mermaid are not accepted. The publication validator rejects output over 2,400 characters \ - or 24 nonblank lines, extra fields, Markdown headings, more than three list items, and \ - unsafe Markdown.", - ); - p.push_str(&trusted_current_date_context(current_utc_date)); - if let Some(rules) = &cfg.guardrails { - p.push_str("\n\nRepository guardrails you may reference:\n"); - let rules = bounded_untrusted_prompt_text(rules, MAX_GUARDRAIL_PROMPT_BYTES / 2); - p.push_str(&rules); - } - p -} - const MAX_PR_BODY_PROMPT_CHARS: usize = 2_000; fn bounded_pr_body(body: Option<&str>) -> Option { @@ -618,24 +580,6 @@ mod tests { assert_eq!(sanitize_scorer_input("a\0b\u{001f}c\n\t"), "a b c\n\t"); } - #[test] - fn respond_prompt_requires_a_compact_structured_reply() { - let p = respond_system_prompt(&Config::default(), trusted_date()); - assert_eq!(p.matches("UTC date 2026-08-10; later=future.").count(), 1); - assert!(!p.contains("UTC date 2026-08-11; later=future.")); - assert!(p.contains("at or below 1,200 characters")); - assert!(p.contains("not an article")); - assert!(p.contains("{\"answer\":\"concise GitHub-flavored Markdown\",\"diagram\":null}")); - assert!(p.contains("diagram must always be null")); - assert!(p.contains("Mermaid are not accepted")); - assert!(!p.contains("When justified")); - assert!(!p.contains("materially clarifies")); - assert!(p.contains("Do not add an overview")); - assert!(p.contains("Do not use Markdown headings")); - assert!(p.contains("no more than three list items")); - assert!(p.contains("Do not emit active @mentions")); - } - #[test] fn system_prompt_omits_content_policy_when_inactive() { let mut cfg = Config::default(); diff --git a/src/resolve.rs b/src/resolve.rs index fe14783..56e60c9 100644 --- a/src/resolve.rs +++ b/src/resolve.rs @@ -106,9 +106,9 @@ pub(crate) async fn resolve_uncertainties( { Ok(Ok(files)) => files, Ok(Err(error)) => { - eprintln!( + crate::progress::notice(format_args!( "postil: uncertainty resolution is continuing with diff evidence after repository file acquisition failed: {error:#}" - ); + )); Vec::new() } Err(_) => { @@ -158,9 +158,9 @@ pub(crate) async fn resolve_uncertainties( pass.model_incidents .extend_from_slice(error.model_incidents()); pass.usage_accounting_complete &= error.usage_accounting_complete(); - eprintln!( + crate::progress::notice(format_args!( "postil: uncertainty resolution failed open and kept the original finding" - ); + )); None } }; @@ -187,12 +187,12 @@ pub(crate) async fn resolve_uncertainties( } pass.suppressed_findings.reverse(); if confirmed > 0 || unresolved > 0 || !pass.suppressed_findings.is_empty() { - eprintln!( + crate::progress::telemetry(format_args!( "postil: uncertainty resolution confirmed={} refuted={} unresolved={}", confirmed, pass.suppressed_findings.len(), unresolved - ); + )); } pass } diff --git a/src/respond.rs b/src/respond.rs deleted file mode 100644 index 0206fa0..0000000 --- a/src/respond.rs +++ /dev/null @@ -1,1080 +0,0 @@ -//! Interactive bot: reply to an @postil mention on a PR or issue. -//! -//! Scope is review and answer only. Postil never opens PRs or pushes commits. -//! Works across every forge the reviewer supports. PR/MR mentions are grounded -//! on the diff; issue mentions on the issue body. GitHub and GitLab cover both -//! issues and pulls; Bitbucket and Azure DevOps are scoped to PRs (their issue -//! trackers / work items use endpoints we cannot verify against a live host). - -use std::fs::{File, OpenOptions}; -use std::io::Write; -use std::os::unix::fs::OpenOptionsExt; -use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicU64, Ordering}; - -use anyhow::{Context, Result, anyhow, ensure}; -use serde::{Deserialize, Serialize}; -use time::OffsetDateTime; - -use crate::config::Config; -use crate::diff; -use crate::envelope::{ModelUsageCostSource, ModelUsagePhase, ModelUsageRole}; -use crate::forge::{ - Forge, ThreadKind, azure::Azure, bitbucket::Bitbucket, github::GitHub, gitlab::GitLab, -}; -use crate::llm::{Answer, LlmClient}; -use crate::prompt; -use crate::review::ForgeKind; - -pub struct RespondArgs { - pub forge: ForgeKind, - pub repo: Option, - /// The PR number, when the mention is on a pull request. - pub pr: Option, - /// The issue number, when the mention is on an issue. - pub issue: Option, - /// The maintainer's comment text (the mention). When None, read from the - /// POSTIL_COMMENT environment variable, the safe path for automation. - pub comment: Option, - pub config: Option, - pub model: Option, - /// Print the answer instead of posting it. - pub no_post: bool, -} - -// Leave enough transport headroom for JSON escaping, the system prompt, PR -// metadata, and the maintainer message under either supported API shape. -const MAX_RESPOND_DIFF_CONTEXT_BYTES: usize = crate::llm::MAX_PROVIDER_REQUEST_BYTES / 12; -const MAX_RESPOND_MANIFEST_BYTES: usize = 24_000; -const MAX_RESPOND_REPO_BYTES: usize = 512; -const MAX_RESPOND_TITLE_BYTES: usize = 2 * 1024; -const MAX_RESPOND_DESCRIPTION_BYTES: usize = 4 * 1024; -const MAX_RESPOND_COMMENT_BYTES: usize = 8 * 1024; -const MAX_RESPOND_ISSUE_BODY_BYTES: usize = 8 * 1024; -const USAGE_RECEIPT_PATH_ENV: &str = "POSTIL_USAGE_RECEIPT_PATH"; -const RESPOND_MAX_CHARS: usize = 2_400; -const RESPOND_MAX_NONBLANK_LINES: usize = 24; -const RESPOND_MAX_LIST_ITEMS: usize = 3; - -#[derive(Deserialize)] -#[serde(deny_unknown_fields)] -struct RespondOutput { - answer: String, - diagram: serde_json::Value, -} - -#[derive(Serialize)] -#[serde(rename_all = "camelCase")] -struct RespondUsageReceipt<'a> { - version: u32, - operation: &'static str, - prompt_tokens: u64, - completion_tokens: u64, - models: Vec>, - usage_accounting_complete: bool, -} - -#[derive(Serialize)] -#[serde(rename_all = "camelCase")] -struct RespondModelUsage<'a> { - model: &'a str, - #[serde(skip_serializing_if = "Option::is_none")] - role: Option, - #[serde(skip_serializing_if = "Option::is_none")] - phase: Option, - #[serde(skip_serializing_if = "Option::is_none")] - call_ordinal: Option, - #[serde(skip_serializing_if = "Option::is_none")] - attempt: Option, - prompt_tokens: u64, - completion_tokens: u64, - #[serde(skip_serializing_if = "Option::is_none")] - cost_micros: Option, - #[serde(skip_serializing_if = "Option::is_none")] - cost_provider_decimal: Option<&'a str>, - #[serde(skip_serializing_if = "Option::is_none")] - cost_source: Option, - accounting_complete: bool, -} - -// PID separates concurrent processes; this sequence separates writers within -// one process. create_new below also fails closed if a path already exists. -static RECEIPT_TEMP_SEQUENCE: AtomicU64 = AtomicU64::new(0); - -struct UsageReceiptWriter { - file: Option, - temp_path: PathBuf, - final_path: PathBuf, -} - -impl UsageReceiptWriter { - fn from_env() -> Result> { - let Some(path) = std::env::var_os(USAGE_RECEIPT_PATH_ENV) else { - return Ok(None); - }; - anyhow::ensure!( - !path.is_empty(), - "{USAGE_RECEIPT_PATH_ENV} must not be empty" - ); - let final_path = PathBuf::from(path); - let parent = final_path.parent().unwrap_or_else(|| Path::new(".")); - let file_name = final_path - .file_name() - .ok_or_else(|| anyhow!("{USAGE_RECEIPT_PATH_ENV} must name a file"))? - .to_string_lossy(); - let sequence = RECEIPT_TEMP_SEQUENCE.fetch_add(1, Ordering::Relaxed); - let temp_path = parent.join(format!( - ".{file_name}.{}.{}.tmp", - std::process::id(), - sequence - )); - let file = OpenOptions::new() - .write(true) - .create_new(true) - // Usage receipts contain private provider-accounting metadata. - .mode(0o600) - .open(&temp_path) - .context("creating private usage receipt temporary file")?; - Ok(Some(Self { - file: Some(file), - temp_path, - final_path, - })) - } - - fn commit(mut self, answer: &Answer) -> Result<()> { - let receipt = RespondUsageReceipt { - version: 2, - operation: "respond", - prompt_tokens: answer.usage.prompt_tokens, - completion_tokens: answer.usage.completion_tokens, - models: answer - .models - .iter() - .map(|model| RespondModelUsage { - model: &model.model, - role: model.role, - phase: model.phase, - call_ordinal: model.call_ordinal, - attempt: model.attempt, - prompt_tokens: model.prompt_tokens, - completion_tokens: model.completion_tokens, - cost_micros: model.cost_micros, - cost_provider_decimal: model.cost_provider_decimal.as_deref(), - cost_source: model.cost_source, - accounting_complete: model.accounting_complete, - }) - .collect(), - usage_accounting_complete: answer.usage_accounting_complete, - }; - let file = self.file.as_mut().expect("usage receipt file is present"); - serde_json::to_writer(&mut *file, &receipt).context("serializing usage receipt")?; - file.write_all(b"\n").context("writing usage receipt")?; - // Publish only after file contents are durable. The directory sync - // below makes the rename durable before stdout or forge delivery. - file.sync_all().context("syncing usage receipt")?; - drop(self.file.take()); - std::fs::rename(&self.temp_path, &self.final_path) - .context("atomically publishing usage receipt")?; - if let Some(parent) = self.final_path.parent() { - File::open(parent) - .and_then(|directory| directory.sync_all()) - .context("syncing usage receipt directory")?; - } - Ok(()) - } -} - -impl Drop for UsageReceiptWriter { - fn drop(&mut self) { - // Drop cannot return cleanup errors. Best-effort removal is safe: an - // unpublished temp file is never treated as a committed receipt, and - // create_new prevents a later writer from clobbering it. - let _ = std::fs::remove_file(&self.temp_path); - } -} - -pub async fn run(args: RespondArgs) -> Result { - let cwd = std::env::current_dir()?; - let mut cfg = Config::load(&cwd, args.config.as_deref())?; - if let Some(m) = &args.model { - cfg.model = m.clone(); - } - cfg.require_model()?; - let repo = args - .repo - .clone() - .or_else(|| std::env::var("GITHUB_REPOSITORY").ok()) - .ok_or_else(|| anyhow!("--repo is required"))?; - let comment = args - .comment - .clone() - .or_else(|| std::env::var("POSTIL_COMMENT").ok()) - .filter(|c| !c.trim().is_empty()) - .ok_or_else(|| anyhow!("the mention text is required: --comment or POSTIL_COMMENT"))?; - let usage_receipt = UsageReceiptWriter::from_env()?; - - // The number the mention is on, and whether it is a PR/MR or an issue. - let (number, kind) = match (args.pr, args.issue) { - (Some(pr), _) => (pr, ThreadKind::Pull), - (None, Some(issue)) => (issue, ThreadKind::Issue), - (None, None) => return Err(anyhow!("one of --pr or --issue is required")), - }; - - // Same flow for every forge; the trait carries the per-host endpoints. The - // forge is monomorphized (the trait uses `async fn` and is not dyn-safe), so - // dispatch by kind here, exactly as the reviewer does. - match args.forge { - ForgeKind::GitHub => { - respond_with( - GitHub::new(&repo, number)?, - &cfg, - &repo, - number, - kind, - &comment, - args.no_post, - usage_receipt, - ) - .await - } - ForgeKind::GitLab => { - respond_with( - GitLab::new(&repo, number)?, - &cfg, - &repo, - number, - kind, - &comment, - args.no_post, - usage_receipt, - ) - .await - } - ForgeKind::Bitbucket => { - respond_with( - Bitbucket::new(&repo, number)?, - &cfg, - &repo, - number, - kind, - &comment, - args.no_post, - usage_receipt, - ) - .await - } - ForgeKind::Azure => { - respond_with( - Azure::new(&repo, number)?, - &cfg, - &repo, - number, - kind, - &comment, - args.no_post, - usage_receipt, - ) - .await - } - ForgeKind::Local => Err(anyhow!("postil respond needs a remote forge, not --local")), - } -} - -#[allow(clippy::too_many_arguments)] -async fn respond_with( - forge: F, - cfg: &Config, - repo: &str, - number: u64, - kind: ThreadKind, - comment: &str, - no_post: bool, - usage_receipt: Option, -) -> Result { - let current_utc_date = OffsetDateTime::now_utc().date(); - let context = build_context(&forge, repo, number, kind).await?; - - let system = prompt::respond_system_prompt(cfg, current_utc_date); - let user = respond_user_prompt(&context, comment); - let client = LlmClient::from_env(cfg).await?; - client.preflight_respond_plan(cfg, &system, &user)?; - let answer = client - .answer(cfg, &system, &user, validate_respond_output) - .await?; - let reply = answer.content.clone(); - - // Hosted execution requires the durable usage receipt before any external - // delivery. Commit it before stdout or forge posting so the control plane - // can reconcile spend and own idempotent delivery. - if let Some(writer) = usage_receipt { - writer.commit(&answer)?; - } - if no_post { - println!("{reply}"); - } else { - forge - .post_comment(number, kind, &reply) - .await - .context("posting reply")?; - eprintln!("postil: replied on {repo}#{number}"); - } - Ok(0) -} - -fn respond_user_prompt(context: &str, comment: &str) -> String { - let comment = prompt::bounded_untrusted_prompt_text(comment.trim(), MAX_RESPOND_COMMENT_BYTES); - format!( - "{context}\n--- Maintainer's message to you ---\n{comment}\n\nReply to the message above." - ) -} - -fn validate_respond_output(raw: &str) -> Result { - if raw.chars().count() > RESPOND_MAX_CHARS { - return Err(anyhow!( - "reply exceeds the 2,400-character publication limit" - )); - } - if nonblank_line_count(raw) > RESPOND_MAX_NONBLANK_LINES { - return Err(anyhow!( - "reply exceeds the {RESPOND_MAX_NONBLANK_LINES}-line publication limit" - )); - } - - let output: RespondOutput = serde_json::from_str(raw.trim()) - .context("reply must be the exact {answer, diagram} JSON object")?; - if !output.diagram.is_null() { - return Err(anyhow!("reply diagram must be null")); - } - let normalized_answer = normalize_newlines(&output.answer); - let answer = trim_outer_blank_lines(&normalized_answer); - if answer.is_empty() { - return Err(anyhow!("reply answer is empty")); - } - let publication_text = mask_markdown_code(&answer); - validate_answer_publication(&publication_text)?; - if contains_mermaid_fence(&answer) - || answer - .lines() - .any(|line| is_mermaid_declaration(line.trim())) - { - return Err(anyhow!("Mermaid must use the diagram field")); - } - if markdown_heading_count(&publication_text) > 0 { - return Err(anyhow!( - "reply contains a Markdown heading; conversational replies do not allow headings" - )); - } - let list_items = markdown_list_item_count(&publication_text); - if list_items > RESPOND_MAX_LIST_ITEMS { - return Err(anyhow!( - "reply contains {list_items} list items; at most {RESPOND_MAX_LIST_ITEMS} are allowed" - )); - } - - let rendered = answer; - - if rendered.chars().count() > RESPOND_MAX_CHARS { - return Err(anyhow!( - "rendered reply exceeds the 2,400-character publication limit" - )); - } - if nonblank_line_count(&rendered) > RESPOND_MAX_NONBLANK_LINES { - return Err(anyhow!( - "rendered reply exceeds the {RESPOND_MAX_NONBLANK_LINES}-line publication limit" - )); - } - Ok(rendered) -} - -fn normalize_newlines(text: &str) -> String { - text.replace("\r\n", "\n").replace('\r', "\n") -} - -fn trim_outer_blank_lines(text: &str) -> String { - let lines: Vec<&str> = text.split('\n').collect(); - let Some(first) = lines.iter().position(|line| !line.trim().is_empty()) else { - return String::new(); - }; - let last = lines - .iter() - .rposition(|line| !line.trim().is_empty()) - .expect("a first nonblank line has a matching last line"); - lines[first..=last].join("\n") -} - -fn validate_answer_publication(publication_text: &str) -> Result<()> { - if contains_active_mention(publication_text) { - return Err(anyhow!("reply contains an active mention")); - } - if contains_raw_html(publication_text) { - return Err(anyhow!("reply contains raw HTML")); - } - if contains_markdown_image(publication_text) { - return Err(anyhow!("reply contains a Markdown image")); - } - if contains_markdown_table(publication_text) { - return Err(anyhow!("reply contains a Markdown table")); - } - Ok(()) -} - -fn mask_markdown_code(text: &str) -> String { - let mut masked = String::with_capacity(text.len()); - let mut fence: Option<(char, usize)> = None; - let mut in_indented_block = false; - let mut previous_line_was_blank = true; - for line_with_newline in text.split_inclusive('\n') { - let (line, newline) = line_with_newline - .strip_suffix('\n') - .map_or((line_with_newline, ""), |line| (line, "\n")); - if let Some((marker, width)) = fence { - let closes = markdown_fence_marker(line).is_some_and(|(candidate, candidate_width)| { - candidate == marker - && candidate_width >= width - && fence_remainder(line, candidate, candidate_width) - .trim() - .is_empty() - }); - masked.extend(std::iter::repeat_n(' ', line.chars().count())); - masked.push_str(newline); - if closes { - fence = None; - previous_line_was_blank = true; - } - continue; - } - if let Some(opening) = markdown_fence_opening(line) { - fence = Some(opening); - in_indented_block = false; - masked.extend(std::iter::repeat_n(' ', line.chars().count())); - masked.push_str(newline); - continue; - } - if is_indented_code_line(line) && (in_indented_block || previous_line_was_blank) { - in_indented_block = true; - previous_line_was_blank = false; - masked.extend(std::iter::repeat_n(' ', line.chars().count())); - masked.push_str(newline); - continue; - } - if line.trim().is_empty() { - previous_line_was_blank = true; - masked.push_str(line); - masked.push_str(newline); - continue; - } - in_indented_block = false; - previous_line_was_blank = false; - masked.push_str(&mask_inline_code(line)); - masked.push_str(newline); - } - masked -} - -fn markdown_fence_marker(line: &str) -> Option<(char, usize)> { - let indentation = line.chars().take_while(|ch| *ch == ' ').count(); - if indentation > 3 { - return None; - } - let content = &line[indentation..]; - let marker = content.chars().next()?; - if !matches!(marker, '`' | '~') { - return None; - } - let width = content.chars().take_while(|ch| *ch == marker).count(); - (width >= 3).then_some((marker, width)) -} - -fn markdown_fence_opening(line: &str) -> Option<(char, usize)> { - let (marker, width) = markdown_fence_marker(line)?; - if marker == '`' && fence_remainder(line, marker, width).contains('`') { - return None; - } - Some((marker, width)) -} - -fn is_indented_code_line(line: &str) -> bool { - line.starts_with(" ") || line.starts_with('\t') -} - -fn fence_remainder(line: &str, marker: char, width: usize) -> &str { - let indentation = line.chars().take_while(|ch| *ch == ' ').count(); - let offset = indentation + marker.len_utf8() * width; - &line[offset..] -} - -fn contains_mermaid_fence(text: &str) -> bool { - text.lines().any(|line| { - markdown_fence_opening(line).is_some_and(|(marker, width)| { - fence_remainder(line, marker, width) - .split_whitespace() - .next() - .is_some_and(|language| language.eq_ignore_ascii_case("mermaid")) - }) - }) -} - -fn mask_inline_code(line: &str) -> String { - let chars: Vec = line.chars().collect(); - let mut masked = chars.clone(); - let mut cursor = 0; - while cursor < chars.len() { - if chars[cursor] != '`' { - cursor += 1; - continue; - } - let width = chars[cursor..].iter().take_while(|ch| **ch == '`').count(); - let mut closing = cursor + width; - let mut found = None; - while closing < chars.len() { - if chars[closing] != '`' { - closing += 1; - continue; - } - let closing_width = chars[closing..].iter().take_while(|ch| **ch == '`').count(); - if closing_width == width { - found = Some(closing + width); - break; - } - closing += closing_width; - } - let Some(end) = found else { - cursor += width; - continue; - }; - masked[cursor..end].fill(' '); - cursor = end; - } - masked.into_iter().collect() -} - -fn contains_active_mention(text: &str) -> bool { - let chars: Vec = text.chars().collect(); - for (index, ch) in chars.iter().enumerate() { - if *ch != '@' - || index - .checked_sub(1) - .and_then(|previous| chars.get(previous)) - .is_some_and(|previous| { - previous.is_ascii_alphanumeric() || matches!(previous, '_' | '-') - }) - { - continue; - } - let mut cursor = index + 1; - let mut name_len = 0; - while chars - .get(cursor) - .is_some_and(|next| next.is_ascii_alphanumeric() || *next == '-') - && name_len < 39 - { - cursor += 1; - name_len += 1; - } - if name_len > 0 - && chars.get(cursor).is_none_or(|boundary| { - !boundary.is_ascii_alphanumeric() && !matches!(boundary, '_' | '-') - }) - { - return true; - } - } - false -} - -fn contains_raw_html(text: &str) -> bool { - if text.contains("Visible text.", - "
MoreHidden text.
", - "A | B\n--- | ---\n1 | 2", - "![diagram](https://example.test/diagram.png)", - "## What this PR does\nA tour.", - "# Correctness\nNarration.", - "### Issues and risks\nInventory.", - "## Verdict\nLooks good.", - ] { - assert!( - validate_respond_output(&structured(answer, None)).is_err(), - "unsafe reply was accepted: {answer}" - ); - } - let email = structured("Email dev@example.test with the result.", None); - assert_eq!( - validate_respond_output(&email).unwrap(), - "Email dev@example.test with the result." - ); - } - - #[test] - fn rejects_all_generated_diagrams_and_mermaid_answers() { - let diagram = structured( - "The request crosses the queue before a worker handles it.", - Some("flowchart LR\n API --> Queue"), - ); - assert!( - validate_respond_output(&diagram) - .unwrap_err() - .to_string() - .contains("must be null") - ); - for answer in [ - "```mermaid\nflowchart TD\n A --> B\n```", - "``` mermaid\nA --> B\n```", - "~~~mermaid\nsequenceDiagram\n A->>B: hello\n~~~", - " ~~~ MERMAID\nA --> B\n ~~~", - "flowchart LR\n A --> B", - "graph TD\n A --> B", - "sequenceDiagram", - "stateDiagram", - "stateDiagram-v2", - "classDiagram\n class A", - "erDiagram", - "journey", - "gantt", - "pie showData", - "mindmap", - "timeline", - "gitGraph {\"showBranches\": true}", - "quadrantChart", - "xychart-beta", - "block-beta columns 3", - "packet-beta", - "architecture-beta", - "kanban", - "sankey-beta", - ] { - assert!(validate_respond_output(&structured(answer, None)).is_err()); - } - } - - #[test] - fn mermaid_detection_does_not_reject_ordinary_prose() { - for answer in [ - "Graph construction is linear in the number of edges.", - "A timeline helps explain the retry sequence.", - "Pie is not relevant to this handler.", - "The journey continues through the queue.", - "Kanban boards are outside this change.", - ] { - assert_eq!( - validate_respond_output(&structured(answer, None)).unwrap(), - answer - ); - } - } - - #[test] - fn rejects_report_shaped_headings() { - for answer in [ - "# Analysis\nLong-form report prose.", - "# Recommendations\nLong-form report prose.", - "# Overview\nLong-form report prose.", - "# Summary\nLong-form report prose.", - "# What this pull request does\nLong-form report prose.", - "# Issue and risk\nLong-form report prose.", - "# Risks\nLong-form report prose.", - "# Assessment\nLong-form report prose.", - "# Review metadata\nLong-form report prose.", - "# Summary)\nLong-form report prose.", - "## Verdict: ###\nLong-form report prose.", - "Correctness!\n------------\nLong-form report prose.", - ] { - let reply = structured(answer, None); - assert!( - validate_respond_output(&reply).is_err(), - "report heading was accepted: {answer}", - ); - } - } - - #[test] - fn contract_vectors_mask_code_and_reject_image_forms() { - for answer in [ - "Use `@maintainer` as the literal test fixture.", - "Unicode before code: é `@maintainer`.", - "`
not HTML here
` is sample text.", - "```markdown\n@maintainer\n
sample
\n![image][ref]\n## Verdict\nA | B\n--- | ---\n```", - " @maintainer", - "\t@maintainer", - ] { - assert_eq!( - validate_respond_output(&structured(answer, None)).unwrap(), - answer - ); - } - - for image in [ - "![inline](image.png)", - "![reference][image-ref]", - "![collapsed][]", - "![shortcut]", - "![multi\nline](image.png)", - ] { - assert!(validate_respond_output(&structured(image, None)).is_err()); - } - - for invalid in [ - r#"{"answer":"ok"}"#, - r#"{"answer":"ok","diagram":null,"extra":true}"#, - r#"{"answer":"ok","diagram":{}}"#, - r#"{"answer":"ok","diagram":""}"#, - ] { - assert!(validate_respond_output(invalid).is_err()); - } - } - - #[test] - fn commonmark_masking_does_not_hide_active_mentions_in_live_prose() { - for answer in [ - "Ask @maintainer to approve this.", - "`unterminated @maintainer", - "Unicode before unmatched code: é `unterminated @maintainer", - "```rust`invalid info string\n@maintainer\n```", - "This is a paragraph.\n @maintainer remains live prose.", - ] { - let error = validate_respond_output(&structured(answer, None)).unwrap_err(); - assert!( - error.to_string().contains("active mention"), - "active mention was hidden by invalid code markup: {answer}" - ); - } - } - - #[test] - fn outer_blank_line_trimming_preserves_indented_code() { - let answer = " @maintainer"; - assert_eq!( - validate_respond_output(&structured("\n \n @maintainer\n\t\n", None)).unwrap(), - answer - ); - } - - #[test] - fn shape_limits_ignore_markers_inside_code() { - for answer in [ - " 1) One\n 2) Two\n 3) Three\n 4) Four\n 5) Five\n 6) Six", - " # One\n ## Two\n ### Three", - "`# One`\n`## Two`\n`### Three`\n`1) One`\n`2) Two`\n`3) Three`\n`4) Four`\n`5) Five`\n`6) Six`", - ] { - assert_eq!( - validate_respond_output(&structured(answer, None)).unwrap(), - answer - ); - } - } -} diff --git a/src/review.rs b/src/review.rs index 8c8990a..3434dc0 100644 --- a/src/review.rs +++ b/src/review.rs @@ -6,7 +6,7 @@ use std::time::{Duration, Instant}; use anyhow::{Context, Result, anyhow}; use futures::StreamExt; -use crate::config::{Config, FindingPresentation, GateLevel, OnError}; +use crate::config::{Config, FindingPresentation, GateLevel, OnError, ReasoningEffort}; use crate::diff; use crate::durable_plan::{DurablePlanRegistrar, DurableReviewPlan}; use crate::envelope::{ @@ -27,6 +27,14 @@ use sha2::{Digest, Sha256}; use std::collections::HashMap; use time::{Date, OffsetDateTime}; +macro_rules! eprintln { + ($($argument:tt)*) => { crate::progress::telemetry(format_args!($($argument)*)) }; +} + +macro_rules! notice { + ($($argument:tt)*) => { crate::progress::notice(format_args!($($argument)*)) }; +} + /// Each model request stays bounded. Large reviews continue through sequential /// source windows; actual provider-attempt, deadline, and spend guards remain /// enforced by `LlmClient` while raw diff size never decides reviewability. @@ -414,6 +422,10 @@ pub struct ReviewArgs { pub fail_on: Option, pub config: Option, pub model: Option, + pub reasoning_effort: Option, + pub scorer_reasoning_effort: Option, + pub verbose: bool, + pub no_progress: bool, pub bounded: bool, pub no_post: bool, pub defer_gate_check: bool, @@ -430,6 +442,11 @@ impl ReviewArgs { self.output } } + + fn writes_machine_stdout(&self) -> bool { + (self.resolved_output_format().is_some() && self.output_file.is_none()) + || self.publication_plan_output.as_deref() == Some(Path::new("-")) + } } struct ReviewInput<'a> { @@ -458,6 +475,51 @@ struct RemoteReviewResult { publication_diff: Option, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct ReviewOutcome { + exit_code: i32, + completion: crate::progress::ReviewCompletion, +} + +impl ReviewOutcome { + fn from_envelope(envelope: &Envelope) -> Self { + use crate::progress::ReviewCompletion; + + let completion = if envelope + .findings + .iter() + .any(|finding| finding.path == crate::envelope::PROVIDER_PATH) + { + ReviewCompletion::ProviderIncomplete + } else if envelope.findings.iter().any(|finding| { + finding.path == crate::envelope::OPERATIONAL_PATH + && matches!( + finding.title.as_str(), + "Model output could not be validated" + | "Model narrated risk without structured findings" + ) + }) { + ReviewCompletion::ModelOutputIncomplete + } else if envelope + .findings + .iter() + .any(|finding| finding.path == crate::envelope::OPERATIONAL_PATH) + { + ReviewCompletion::OperationalIncomplete + } else if envelope.gate.failing { + ReviewCompletion::BlockingFindings + } else if envelope.findings.is_empty() { + ReviewCompletion::Clean + } else { + ReviewCompletion::AdvisoryFindings + }; + Self { + exit_code: i32::from(envelope.gate.failing), + completion, + } + } +} + #[derive(Clone, Copy)] struct PublicationContext<'a> { snapshot: &'a PrMeta, @@ -523,92 +585,126 @@ impl std::error::Error for ReviewFailure {} pub async fn run(args: ReviewArgs) -> Result { let cwd = std::env::current_dir()?; - if args.output_json { - eprintln!("warning: --output-json is deprecated; use --output json instead"); - } - if args.output_file.is_some() && args.resolved_output_format().is_none() { - return Err(anyhow!("--output-file requires --output or --output-json")); - } - if args.publication_plan_output.is_some() { - anyhow::ensure!( - args.forge == ForgeKind::GitHub - && args.repo.is_some() - && args.pr.is_some() - && args.sha.is_some() - && args.base_sha.is_some() - && !args.staged - && args.base.is_none() - && args.diff_file.is_none(), - "--publication-plan-output requires a remote GitHub pull request with explicit --repo, --pr, --sha, and --base-sha" - ); - anyhow::ensure!( - args.no_post - && args.check_run_id.is_none() - && args.gate_check_run_id.is_none() - && !args.defer_gate_check, - "--publication-plan-output cannot be combined with forge mutation options" - ); - let generation = args - .publication_generation - .as_deref() - .context("--publication-plan-output requires --publication-generation")?; - crate::forge::ensure_publication_decimal_identifier("controller generation", generation) + let progress = crate::progress::start_review( + args.writes_machine_stdout(), + args.verbose, + crate::config::hosted_runtime_mode(), + args.no_progress, + ); + let review = async { + if args.output_json { + notice!("warning: --output-json is deprecated; use --output json instead"); + } + if args.output_file.is_some() && args.resolved_output_format().is_none() { + return Err(anyhow!("--output-file requires --output or --output-json")); + } + if args.publication_plan_output.is_some() { + anyhow::ensure!( + args.forge == ForgeKind::GitHub + && args.repo.is_some() + && args.pr.is_some() + && args.sha.is_some() + && args.base_sha.is_some() + && !args.staged + && args.base.is_none() + && args.diff_file.is_none(), + "--publication-plan-output requires a remote GitHub pull request with explicit --repo, --pr, --sha, and --base-sha" + ); + anyhow::ensure!( + args.no_post + && args.check_run_id.is_none() + && args.gate_check_run_id.is_none() + && !args.defer_gate_check, + "--publication-plan-output cannot be combined with forge mutation options" + ); + let generation = args + .publication_generation + .as_deref() + .context("--publication-plan-output requires --publication-generation")?; + crate::forge::ensure_publication_decimal_identifier( + "controller generation", + generation, + ) .context("invalid --publication-generation")?; - let input_identity = args - .publication_input_identity - .as_deref() - .context("--publication-plan-output requires --publication-input-identity")?; - crate::forge::ensure_publication_sha256_identity("input identity", input_identity) - .context("invalid --publication-input-identity")?; - } else { - anyhow::ensure!( - args.publication_generation.is_none(), - "--publication-generation requires --publication-plan-output" - ); - anyhow::ensure!( - args.publication_input_identity.is_none(), - "--publication-input-identity requires --publication-plan-output" - ); - } - let mut cfg = Config::load(&cwd, args.config.as_deref())?; - if let Some(m) = &args.model { - cfg.model = m.clone(); - } - cfg.require_model()?; - if let Some(fo) = &args.fail_on { - cfg.gate_fail_on = - GateLevel::parse(fo).ok_or_else(|| anyhow!("invalid --fail-on {fo:?}"))?; - } - if !args.no_post - && cfg.finding_presentation == FindingPresentation::CheckAnnotations - && args.forge != ForgeKind::GitHub - { - return Err(anyhow!( - "review.findingPresentation checkAnnotations requires GitHub publication" - )); - } - - match args.forge { - ForgeKind::Local => run_local(&args, &cfg, &cwd).await, - ForgeKind::GitHub => { - let repo = require_repo(&args)?; - let forge = GitHub::new(&repo, require_pr(&args)?)?; - run_remote(&args, &cfg, &forge, &repo, RepositorySource::GitHub(&forge)).await + let input_identity = args + .publication_input_identity + .as_deref() + .context("--publication-plan-output requires --publication-input-identity")?; + crate::forge::ensure_publication_sha256_identity("input identity", input_identity) + .context("invalid --publication-input-identity")?; + } else { + anyhow::ensure!( + args.publication_generation.is_none(), + "--publication-generation requires --publication-plan-output" + ); + anyhow::ensure!( + args.publication_input_identity.is_none(), + "--publication-input-identity requires --publication-plan-output" + ); } - ForgeKind::GitLab => { - let repo = require_repo(&args)?; - let forge = GitLab::new(&repo, require_pr(&args)?)?; - run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + let mut cfg = Config::load(&cwd, args.config.as_deref())?; + if let Some(m) = &args.model { + cfg.model = m.clone(); + cfg.model_source = "command line".to_string(); } - ForgeKind::Bitbucket => { - let repo = require_repo(&args)?; - let forge = Bitbucket::new(&repo, require_pr(&args)?)?; - run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + if let Some(effort) = &args.reasoning_effort { + cfg.reasoning_effort = ReasoningEffort::parse("--reasoning-effort", effort)?; + cfg.reasoning_effort_source = "command line".to_string(); } - ForgeKind::Azure => { - let repo = require_repo(&args)?; - let forge = Azure::new(&repo, require_pr(&args)?)?; - run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + if let Some(effort) = &args.scorer_reasoning_effort { + cfg.scorer_reasoning_effort = + ReasoningEffort::parse("--scorer-reasoning-effort", effort)?; + cfg.scorer_reasoning_effort_source = "command line".to_string(); + } + cfg.require_model()?; + if let Some(fo) = &args.fail_on { + cfg.gate_fail_on = + GateLevel::parse(fo).ok_or_else(|| anyhow!("invalid --fail-on {fo:?}"))?; + } + if !args.no_post + && cfg.finding_presentation == FindingPresentation::CheckAnnotations + && args.forge != ForgeKind::GitHub + { + return Err(anyhow!( + "review.findingPresentation checkAnnotations requires GitHub publication" + )); + } + match args.forge { + ForgeKind::Local => run_local(&args, &cfg, &cwd).await, + ForgeKind::GitHub => { + let repo = require_repo(&args)?; + let forge = GitHub::new(&repo, require_pr(&args)?)?; + run_remote(&args, &cfg, &forge, &repo, RepositorySource::GitHub(&forge)).await + } + ForgeKind::GitLab => { + let repo = require_repo(&args)?; + let forge = GitLab::new(&repo, require_pr(&args)?)?; + run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + } + ForgeKind::Bitbucket => { + let repo = require_repo(&args)?; + let forge = Bitbucket::new(&repo, require_pr(&args)?)?; + run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + } + ForgeKind::Azure => { + let repo = require_repo(&args)?; + let forge = Azure::new(&repo, require_pr(&args)?)?; + run_remote(&args, &cfg, &forge, &repo, RepositorySource::Unavailable).await + } + } + }; + let result = tokio::select! { + result = review => result, + _ = tokio::signal::ctrl_c() => Err(anyhow!("review interrupted")), + }; + match result { + Ok(outcome) => { + progress.finish(outcome.completion); + Ok(outcome.exit_code) + } + Err(error) => { + progress.finish(crate::progress::ReviewCompletion::OperationalIncomplete); + Err(error) } } } @@ -625,25 +721,39 @@ fn require_pr(args: &ReviewArgs) -> Result { .ok_or_else(|| anyhow!("--pr is required for remote review")) } -async fn run_local(args: &ReviewArgs, cfg: &Config, repo_root: &Path) -> Result { - let source = if let Some(path) = &args.diff_file { - LocalSource::DiffFile(path.clone()) +async fn run_local(args: &ReviewArgs, cfg: &Config, repo_root: &Path) -> Result { + let head_sha = if args.diff_file.is_some() { + None + } else { + local::head_sha(repo_root).await? + }; + let selection = if let Some(path) = &args.diff_file { + local::AutoSelection { + source: LocalSource::DiffFile(path.clone()), + warning: None, + } } else if args.staged { - LocalSource::Staged + local::AutoSelection { + source: LocalSource::Staged, + warning: None, + } } else if let Some(base) = &args.base { - LocalSource::Base(base.clone()) + local::AutoSelection { + source: LocalSource::Base(base.clone()), + warning: None, + } } else { - return Err(anyhow!( - "local review needs one of --staged, --base , or --diff-file " - )); + local::auto_select(repo_root, head_sha.as_deref()).await? }; - let head_sha = local::head_sha().await; - let local_snapshot = local::acquire(&source, head_sha.as_deref(), repo_root).await?; + if let Some(warning) = selection.warning { + crate::progress::notice(format_args!("postil: {warning}")); + } + let local_snapshot = local::acquire(&selection.source, head_sha.as_deref(), repo_root).await?; let baseline = load_baseline(args)?; let touched_carried_error = cfg.enabled && args.since_sha.is_some() && carried_error_outside_anchor_file_is_touched(&baseline, &local_snapshot.diff); - let supports_full_fallback = matches!(&source, LocalSource::Base(_)); + let supports_full_fallback = matches!(&selection.source, LocalSource::Base(_)); let fail_closed_baseline = (touched_carried_error && !supports_full_fallback).then(|| baseline.clone()); // This is a fail-closed placeholder, not the completed review's trust. @@ -736,7 +846,7 @@ async fn run_remote( forge: &F, repo: &str, repository_source: RepositorySource<'_>, -) -> Result { +) -> Result { let review_started = std::time::Instant::now(); let strict_publication = strict_hosted_github_publication(args); let meta = run_with_hosted_budget( @@ -790,7 +900,7 @@ async fn run_remote( ); } // CI tokens without checks:write still get review + exit code. - eprintln!("postil: cannot create check runs ({e:#}); continuing without"); + notice!("postil: cannot create check runs ({e:#}); continuing without"); None } } @@ -895,7 +1005,7 @@ async fn run_remote( // Emit the envelope and SARIF before delivery. Hosted GitHub runs // require every applicable publication step; other invocations // retain their gate-derived result when the forge is unavailable. - let code = if envelope.gate.failing { 1 } else { 0 }; + let outcome = ReviewOutcome::from_envelope(&envelope); let finish_result = finish( args, cfg, @@ -918,8 +1028,8 @@ async fn run_remote( if strict_publication { Err(post_err) } else { - eprintln!("postil: could not post the error review ({post_err:#})"); - Ok(code) + notice!("postil: could not post the error review ({post_err:#})"); + Ok(outcome) } } }; @@ -1009,7 +1119,7 @@ fn retain_publication_failure( Err(error) if required => Ok(Some(error)), Err(error) if crate::forge::is_repository_identity_failure(&error) => Err(error), Err(error) => { - eprintln!("postil: {warning} ({error:#})"); + notice!("postil: {warning} ({error:#})"); Ok(None) } } @@ -1017,8 +1127,8 @@ fn retain_publication_failure( fn combine_required_publication( check_failure: Option, - finish_result: Result, -) -> Result { + finish_result: Result, +) -> Result { match (check_failure, finish_result) { (None, result) => result, (Some(check_error), Ok(_)) => { @@ -2925,7 +3035,8 @@ async fn finish( hosted_budget_started_at: Option, publication: Option>, strict_publication: bool, -) -> Result { +) -> Result { + let outcome = ReviewOutcome::from_envelope(&envelope); let publication_plan_to_stdout = args.publication_plan_output.as_deref() == Some(Path::new("-")); // Persist artifacts before any forge I/O: a posting hiccup must not @@ -2944,6 +3055,7 @@ async fn finish( if !publication_plan_to_stdout && (args.resolved_output_format().is_none() || args.output_file.is_some()) { + let _progress_suspension = crate::progress::suspend_for_output(); output::print_pretty(&envelope); } @@ -2990,7 +3102,7 @@ async fn finish( } else { crate::forge::write_github_publication_plan(path, &plan)?; } - return Ok(if envelope.gate.failing { 1 } else { 0 }); + return Ok(outcome); } if let Some(forge) = forge @@ -3012,7 +3124,7 @@ async fn finish( } } crate::forge::write_review_publication_receipt_from_env(&receipt)?; - return Ok(if envelope.gate.failing { 1 } else { 0 }); + return Ok(outcome); } if !should_comment { let mut receipt = forge.plan_review_publication(&envelope, expected_snapshot); @@ -3031,7 +3143,7 @@ async fn finish( } } crate::forge::write_review_publication_receipt_from_env(&receipt)?; - return Ok(if envelope.gate.failing { 1 } else { 0 }); + return Ok(outcome); } let freshness = if let Some(started_at) = hosted_budget_started_at { snapshot_is_current(forge, expected_snapshot, started_at).await @@ -3046,10 +3158,10 @@ async fn finish( )); } Ok(false) => { - eprintln!( + notice!( "postil: publication skipped because the pull request snapshot changed after review" ); - return Ok(if envelope.gate.failing { 1 } else { 0 }); + return Ok(outcome); } Err(error) if strict_publication => { return Err(error).context( @@ -3057,10 +3169,10 @@ async fn finish( ); } Err(error) => { - eprintln!( + notice!( "postil: publication skipped because snapshot freshness could not be verified ({error:#})" ); - return Ok(if envelope.gate.failing { 1 } else { 0 }); + return Ok(outcome); } } let posted = run_with_hosted_budget( @@ -3081,11 +3193,11 @@ async fn finish( if strict_publication { return Err(e).context("required hosted review publication failed"); } - eprintln!("postil: could not post review comment ({e:#})"); + notice!("postil: could not post review comment ({e:#})"); } } } - Ok(if envelope.gate.failing { 1 } else { 0 }) + Ok(outcome) } async fn run_with_hosted_budget( @@ -3551,9 +3663,9 @@ mod tests { .unwrap() }; - let local_edge = format!("Benchmark pull request{}", "x".repeat(42)); - let ci_edge = format!("Benchmark pull request{}", "x".repeat(58)); - let below_floor = format!("Benchmark pull request{}", "x".repeat(442)); + let local_edge = format!("Benchmark pull request{}", "x".repeat(13)); + let ci_edge = format!("Benchmark pull request{}", "x".repeat(29)); + let below_floor = format!("Benchmark pull request{}", "x".repeat(413)); let local_budgets = batch_budgets_for_title(&local_edge); assert_eq!(local_budgets.synthesis, 4_116); @@ -3656,6 +3768,10 @@ mod tests { finding.body ); assert!(envelope.gate.failing); + assert_eq!( + ReviewOutcome::from_envelope(&envelope).completion, + crate::progress::ReviewCompletion::OperationalIncomplete + ); } #[test] @@ -3676,6 +3792,10 @@ mod tests { "the recorded scorer error must be preserved: {}", finding.body ); + assert_eq!( + ReviewOutcome::from_envelope(&envelope).completion, + crate::progress::ReviewCompletion::ModelOutputIncomplete + ); } #[test] @@ -3809,6 +3929,12 @@ mod tests { let envelope = error_envelope(&cfg, &error, "head", Some(&pr_meta()), 99); assert_eq!(envelope.findings[0].path, crate::envelope::PROVIDER_PATH); assert!(!envelope.gate.failing); + let outcome = ReviewOutcome::from_envelope(&envelope); + assert_eq!(outcome.exit_code, 0); + assert_eq!( + outcome.completion, + crate::progress::ReviewCompletion::ProviderIncomplete + ); assert_eq!(envelope.model_usage.len(), 2); assert_eq!(envelope.review_admission.unwrap().provider_attempts, 12); } @@ -3826,6 +3952,41 @@ mod tests { assert!(envelope.summary.contains("failing closed")); } + #[test] + fn ordinary_envelopes_distinguish_clean_advisory_and_blocking_completion() { + let mut envelope = error_envelope( + &Config::default(), + &anyhow::anyhow!("fixture operational error"), + "head", + Some(&pr_meta()), + 1, + ); + envelope.findings[0].path = "src/lib.rs".to_string(); + envelope.findings[0].title = "Ordinary review finding".to_string(); + envelope.gate.failing = false; + let advisory = ReviewOutcome::from_envelope(&envelope); + assert_eq!(advisory.exit_code, 0); + assert_eq!( + advisory.completion, + crate::progress::ReviewCompletion::AdvisoryFindings + ); + + envelope.gate.failing = true; + let blocking = ReviewOutcome::from_envelope(&envelope); + assert_eq!(blocking.exit_code, 1); + assert_eq!( + blocking.completion, + crate::progress::ReviewCompletion::BlockingFindings + ); + + envelope.findings.clear(); + envelope.gate.failing = false; + assert_eq!( + ReviewOutcome::from_envelope(&envelope).completion, + crate::progress::ReviewCompletion::Clean + ); + } + #[test] fn invalid_output_anywhere_in_exhausted_scorer_chain_dominates_provider_failure() { let incidents = vec![ @@ -4094,7 +4255,14 @@ mod tests { let retained = retain_publication_failure(true, completion, "fixture warning") .unwrap() .expect("strict hosted publication retains the timeout"); - let error = combine_required_publication(Some(retained), Ok(0)).unwrap_err(); + let error = combine_required_publication( + Some(retained), + Ok(ReviewOutcome { + exit_code: 0, + completion: crate::progress::ReviewCompletion::Clean, + }), + ) + .unwrap_err(); assert!(format!("{error:#}").contains("completing check runs timed out")); assert!( diff --git a/tests/e2e.rs b/tests/e2e.rs index 3b35962..a144e38 100644 --- a/tests/e2e.rs +++ b/tests/e2e.rs @@ -1,8 +1,9 @@ //! End-to-end tests: the real binary against mocked LLM and forge endpoints. use std::collections::BTreeMap; +use std::path::Path; use std::sync::atomic::{AtomicUsize, Ordering}; -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, OnceLock}; use assert_cmd::Command; #[cfg(feature = "qualification-candidate")] @@ -464,26 +465,6 @@ fn llm_text(text: &str) -> Value { }) } -fn respond_payload(answer: &str, diagram: Option<&str>) -> String { - json!({"answer": answer, "diagram": diagram}).to_string() -} - -fn respond_text(answer: &str) -> Value { - llm_text(&respond_payload(answer, None)) -} - -fn respond_article_slop() -> String { - let prefix = "I reviewed the full diff. Here's my assessment.\n\n\ -## What this PR does\n\nA broad implementation tour.\n\n\ -## Correctness\n\nSeveral paragraphs of non-actionable narration.\n\n\ -## Issues and risks\n\n\ -1. First item.\n2. Second item.\n3. Third item.\n\ -4. Fourth item.\n5. Fifth item.\n6. Sixth item.\n\n\ -## Verdict\n\nA long conclusion.\n\n"; - let padding = 7_186 - prefix.chars().count(); - format!("{prefix}{}", "x".repeat(padding)) -} - fn anthropic_content(findings: Value, input_tokens: u64, output_tokens: u64) -> Value { let findings = explicit_repository_context(findings); let summary = if findings.as_array().is_none_or(|items| items.is_empty()) { @@ -1126,43 +1107,168 @@ async fn mount_azure_merge_base(server: &MockServer) { .await; } -fn postil() -> Command { +fn isolated_config_home() -> &'static Path { + static HOME: OnceLock = OnceLock::new(); + HOME.get_or_init(|| tempfile::tempdir().unwrap()).path() +} + +const ISOLATED_POSTIL_ENV: &[&str] = &[ + "CI", + "GITHUB_ACTIONS", + "GITHUB_REPOSITORY", + "REVIEW_MODEL", + "REVIEW_MODEL_CASCADE", + "REVIEW_MODEL_CONSENSUS", + "REVIEW_REASONING_EFFORT", + "REVIEW_SCORER_MODEL", + "REVIEW_SCORER_MODEL_CASCADE", + "REVIEW_SCORER_REASONING_EFFORT", + "POSTIL_DISABLE_SCORER", + "POSTIL_ALLOW_CONFIG_API_BASE", + "POSTIL_UNCERTAINTY_RESOLUTION", + "POSTIL_CONCISE_FINDINGS", + "POSTIL_NO_PROGRESS", + "POSTIL_DEBUG", + "RUST_LOG", + "NO_COLOR", + "POSTIL_HOSTED_MODE", + "POSTIL_PROVISIONAL_HOSTED_ROSTER", + "POSTIL_EXPECTED_GITHUB_REPO_ID", + "POSTIL_QUALIFICATION_CANDIDATE_PROFILE", + "POSTIL_BENCH_SCREEN_PROFILE", + "POSTIL_QUALIFICATION_PLAN_ONLY", + "POSTIL_QUALIFICATION_CAPTURE_API_BASE", + "POSTIL_IGNORE_REPOSITORY_MODEL_CONFIG", + "POSTIL_BENCH_FORCE_BOUNDED_SELECTION", + "POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", + "POSTIL_LLM_REQUEST_TIMEOUT_SECS", + "POSTIL_LLM_TOTAL_TIMEOUT_SECS", + "MODEL_API_KEY", + "LLM_API_KEY", + "OPENROUTER_API_KEY", + "POSTIL_API_KEY", + "POSTIL_API_BASE", + "POSTIL_API_FORMAT", + "POSTIL_ENDPOINT_AUTH_HEADER", + "POSTIL_ENDPOINT_AUTH_VALUE", + "POSTIL_LARGE_REVIEW_PLAN_ENDPOINT", + "POSTIL_LARGE_REVIEW_PLAN_TOKEN", + "POSTIL_ALLOW_PRIVATE_API_BASE", + "POSTIL_DETAILS_URL", + "POSTIL_PREVENTION_HINT", + "POSTIL_PREVENTION_COMMANDS_JSON", + "POSTIL_PUBLICATION_RECEIPT_PATH", + "POSTIL_LOGIN_SERVER", + "POSTIL_PUBLISH", + "POSTIL_NO_POST", + "AZURE_DEVOPS_API_URL", + "AZURE_DEVOPS_TOKEN", + "BITBUCKET_API_URL", + "BITBUCKET_TOKEN", + "BITBUCKET_USER", + "GITHUB_API_URL", + "GITHUB_TOKEN", + "GITHUB_SERVER_URL", + "GITLAB_API_URL", + "GITLAB_TOKEN", + "POSTIL_ENABLE_BITBUCKET_INCREMENTAL", +]; + +fn isolated_postil() -> Command { let mut cmd = Command::cargo_bin("postil").unwrap(); // Isolate from developer environment and repo config discovery. - cmd.env_remove("REVIEW_MODEL") - .env_remove("REVIEW_MODEL_CASCADE") - .env_remove("REVIEW_SCORER_MODEL") - .env_remove("REVIEW_SCORER_MODEL_CASCADE") - .env_remove("POSTIL_DISABLE_SCORER") - .env_remove("POSTIL_UNCERTAINTY_RESOLUTION") - .env_remove("POSTIL_CONCISE_FINDINGS") - .env_remove("POSTIL_HOSTED_MODE") - .env_remove("POSTIL_EXPECTED_GITHUB_REPO_ID") - .env_remove("POSTIL_QUALIFICATION_CANDIDATE_PROFILE") - .env_remove("POSTIL_QUALIFICATION_PLAN_ONLY") - .env_remove("POSTIL_BENCH_FORCE_BOUNDED_SELECTION") - .env_remove("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY") - .env_remove("POSTIL_LLM_REQUEST_TIMEOUT_SECS") - .env_remove("POSTIL_LLM_TOTAL_TIMEOUT_SECS") - .env_remove("MODEL_API_KEY") - .env_remove("LLM_API_KEY") - .env_remove("OPENROUTER_API_KEY") - .env_remove("POSTIL_API_KEY") - .env_remove("POSTIL_API_BASE") - .env_remove("POSTIL_API_FORMAT") - .env_remove("POSTIL_ENDPOINT_AUTH_HEADER") - .env_remove("POSTIL_ENDPOINT_AUTH_VALUE") - .env_remove("POSTIL_LARGE_REVIEW_PLAN_ENDPOINT") - .env_remove("POSTIL_LARGE_REVIEW_PLAN_TOKEN") - .env_remove("POSTIL_ALLOW_PRIVATE_API_BASE") - .env_remove("POSTIL_DETAILS_URL") - .env_remove("POSTIL_PREVENTION_HINT") - .env_remove("POSTIL_PREVENTION_COMMANDS_JSON") - .env_remove("POSTIL_PUBLISH") - .env_remove("POSTIL_NO_POST") - .env_remove("GITHUB_SERVER_URL") - .env_remove("POSTIL_ENABLE_BITBUCKET_INCREMENTAL") - .env("REVIEW_MODEL", "openai/gpt-5-mini") + for name in ISOLATED_POSTIL_ENV { + cmd.env_remove(name); + } + cmd.env("XDG_CONFIG_HOME", isolated_config_home()); + cmd +} + +#[cfg(unix)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ScriptFlavor { + UtilLinux, + Bsd, +} + +#[cfg(unix)] +fn native_script_flavor() -> ScriptFlavor { + if cfg!(target_os = "linux") { + ScriptFlavor::UtilLinux + } else { + ScriptFlavor::Bsd + } +} + +#[cfg(unix)] +fn shell_quote(argument: &str) -> String { + format!("'{}'", argument.replace('\'', "'\"'\"'")) +} + +#[cfg(unix)] +fn script_arguments(flavor: ScriptFlavor, program: &str, arguments: &[String]) -> Vec { + match flavor { + ScriptFlavor::UtilLinux => vec![ + "-qefc".to_string(), + std::iter::once(program) + .chain(arguments.iter().map(String::as_str)) + .map(shell_quote) + .collect::>() + .join(" "), + "/dev/null".to_string(), + ], + ScriptFlavor::Bsd => std::iter::once("-q".to_string()) + .chain(std::iter::once("/dev/null".to_string())) + .chain(std::iter::once(program.to_string())) + .chain(arguments.iter().cloned()) + .collect(), + } +} + +#[cfg(unix)] +fn isolated_script(program: &str, arguments: &[String]) -> std::process::Command { + let mut command = std::process::Command::new("script"); + for name in ISOLATED_POSTIL_ENV { + command.env_remove(name); + } + command + .env("XDG_CONFIG_HOME", isolated_config_home()) + .args(script_arguments(native_script_flavor(), program, arguments)); + command +} + +#[cfg(unix)] +#[test] +fn pty_script_arguments_cover_util_linux_and_bsd_syntax() { + let arguments = vec![ + "review".to_string(), + "--base".to_string(), + "topic's base".to_string(), + ]; + assert_eq!( + script_arguments(ScriptFlavor::UtilLinux, "/tmp/postil binary", &arguments), + vec![ + "-qefc", + "'/tmp/postil binary' 'review' '--base' 'topic'\"'\"'s base'", + "/dev/null", + ] + ); + assert_eq!( + script_arguments(ScriptFlavor::Bsd, "/tmp/postil binary", &arguments), + vec![ + "-q", + "/dev/null", + "/tmp/postil binary", + "review", + "--base", + "topic's base", + ] + ); +} + +fn postil() -> Command { + let mut cmd = isolated_postil(); + cmd.env("REVIEW_MODEL", "openai/gpt-5-mini") .env("MODEL_API_KEY", fixture_credential("provider")) // Mock providers bind loopback. Production and normal CLI invocations // reject private API endpoints unless this explicit local-only escape @@ -1171,6 +1277,147 @@ fn postil() -> Command { cmd } +#[tokio::test] +async fn transient_logout_failure_keeps_the_local_revocation_handle() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/api/cli/logout")) + .respond_with(ResponseTemplate::new(503)) + .expect(1) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let config_home = dir.path().join("config"); + let credentials_dir = config_home.join("postil"); + std::fs::create_dir_all(&credentials_dir).unwrap(); + let credentials_path = credentials_dir.join("credentials.json"); + std::fs::write( + &credentials_path, + serde_json::to_vec(&json!({ + "version": 3, + "issuer": server.uri(), + "token": "pcli_e2e-access-not-a-real-secret", + "expiresAt": "2999-01-01T00:00:00.000Z", + "refreshToken": "fixture-e2e-refresh-not-a-credential", + "refreshExpiresAt": "2999-12-01T00:00:00.000Z", + "apiBase": "https://postil.dev/api/inference/v1", + "org": "example", + "model": "example/model" + })) + .unwrap(), + ) + .unwrap(); + + let assertion = postil() + .env("XDG_CONFIG_HOME", &config_home) + .env("POSTIL_LOGIN_SERVER", format!("{}/", server.uri())) + .arg("logout") + .assert() + .code(1); + let stderr = String::from_utf8(assertion.get_output().stderr.clone()).unwrap(); + + assert!(stderr.contains("credentials were kept")); + assert!(stderr.contains("postil logout")); + assert!(!stderr.contains("logged out")); + let stored: Value = serde_json::from_slice(&std::fs::read(credentials_path).unwrap()).unwrap(); + assert!(stored["refreshToken"].is_string()); +} + +#[tokio::test] +async fn stored_login_refuses_an_api_base_override_before_network() { + let server = MockServer::start().await; + let dir = tempfile::tempdir().unwrap(); + let config_home = dir.path().join("config"); + let credentials_dir = config_home.join("postil"); + std::fs::create_dir_all(&credentials_dir).unwrap(); + std::fs::write( + credentials_dir.join("credentials.json"), + serde_json::to_vec(&json!({ + "version": 3, + "issuer": "https://postil.dev", + "token": "pcli_e2e-access-not-a-real-secret", + "expiresAt": "2999-01-01T00:00:00.000Z", + "refreshToken": "fixture-e2e-refresh-not-a-credential", + "refreshExpiresAt": "2999-12-01T00:00:00.000Z", + "apiBase": "https://postil.dev/api/inference/v1", + "org": "example", + "model": "openai/gpt-5-mini" + })) + .unwrap(), + ) + .unwrap(); + let diff = write_diff(dir.path()); + + let output = postil() + .current_dir(dir.path()) + .env("XDG_CONFIG_HOME", &config_home) + .env("POSTIL_API_BASE", server.uri()) + .env_remove("POSTIL_API_KEY") + .env_remove("OPENROUTER_API_KEY") + .env_remove("MODEL_API_KEY") + .env_remove("LLM_API_KEY") + .args(["review", "--diff-file"]) + .arg(&diff) + .assert() + .failure(); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); + assert!(stderr.contains("stored postil login is bound to")); + assert!(stderr.contains("explicit key")); + assert!(server.received_requests().await.unwrap().is_empty()); +} + +#[tokio::test] +async fn explicit_byok_key_remains_valid_with_an_api_base_override() { + let server = MockServer::start().await; + let provider_key = fixture_credential("provider"); + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(header("authorization", format!("Bearer {provider_key}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": [{ + "finish_reason": "stop", + "message": {"content": "{\"summary\":\"\",\"findings\":[]}"} + }], + "usage": {"prompt_tokens": 10, "completion_tokens": 5} + }))) + .expect(1) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let config_home = dir.path().join("config"); + let credentials_dir = config_home.join("postil"); + std::fs::create_dir_all(&credentials_dir).unwrap(); + std::fs::write( + credentials_dir.join("credentials.json"), + serde_json::to_vec(&json!({ + "version": 3, + "issuer": "https://postil.dev", + "token": "pcli_e2e-access-not-a-real-secret", + "expiresAt": "2999-01-01T00:00:00.000Z", + "refreshToken": "fixture-e2e-refresh-not-a-credential", + "refreshExpiresAt": "2999-12-01T00:00:00.000Z", + "apiBase": "https://postil.dev/api/inference/v1", + "org": "example", + "model": "openai/gpt-5-mini" + })) + .unwrap(), + ) + .unwrap(); + let diff = write_diff(dir.path()); + + postil() + .current_dir(dir.path()) + .env("XDG_CONFIG_HOME", &config_home) + .env("POSTIL_API_BASE", server.uri()) + .env("MODEL_API_KEY", provider_key) + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .success(); +} + #[tokio::test] async fn local_review_stays_local_without_a_forge_token_in_ci() { let server = MockServer::start().await; @@ -1659,6 +1906,8 @@ async fn native_anthropic_review_uses_messages_shape_auth_and_usage() { assert_eq!(body["messages"].as_array().unwrap().len(), 1); assert_eq!(body["messages"][0]["role"], "user"); assert_eq!(body["max_tokens"], 8_000); + assert_eq!(body["output_config"], json!({"effort": "low"})); + assert!(body.get("thinking").is_none()); assert!(body.get("choices").is_none()); } @@ -1887,6 +2136,21 @@ async fn native_anthropic_findings_use_explicit_native_scorer() { assert_eq!(envelope["findings"][0]["scorerConfidence"], 0.82); assert_eq!(envelope["usage"]["promptTokens"], 52); assert_eq!(envelope["usage"]["completionTokens"], 22); + let requests = server.received_requests().await.unwrap(); + let generator: Value = requests + .iter() + .find(|request| String::from_utf8_lossy(&request.body).contains("claude-sonnet-4-6")) + .unwrap() + .body_json() + .unwrap(); + let scorer: Value = requests + .iter() + .find(|request| String::from_utf8_lossy(&request.body).contains("claude-haiku-4-5")) + .unwrap() + .body_json() + .unwrap(); + assert_eq!(generator["output_config"], json!({"effort": "low"})); + assert_eq!(scorer["thinking"], json!({"type": "disabled"})); } #[tokio::test] @@ -1956,7 +2220,14 @@ async fn openai_successful_scorer_with_zero_usage_marks_accounting_incomplete() .env("POSTIL_API_BASE", server.uri()) .env("REVIEW_MODEL", "generator-model") .env("REVIEW_SCORER_MODEL", "scorer-model") - .args(["review", "--diff-file"]) + .args([ + "review", + "--reasoning-effort", + "medium", + "--scorer-reasoning-effort", + "high", + "--diff-file", + ]) .arg(&diff) .arg("--output-json") .assert() @@ -1969,6 +2240,21 @@ async fn openai_successful_scorer_with_zero_usage_marks_accounting_incomplete() assert!(stderr.contains( "llm usage accounting incomplete phase=scorer model=scorer-model attempt=1 reason=nonpositive" )); + let requests = server.received_requests().await.unwrap(); + let generator: Value = requests + .iter() + .find(|request| String::from_utf8_lossy(&request.body).contains("generator-model")) + .unwrap() + .body_json() + .unwrap(); + let scorer: Value = requests + .iter() + .find(|request| String::from_utf8_lossy(&request.body).contains("scorer-model")) + .unwrap() + .body_json() + .unwrap(); + assert_eq!(generator["reasoning"], json!({"effort": "medium"})); + assert_eq!(scorer["reasoning"], json!({"effort": "high"})); } #[tokio::test] @@ -5731,7 +6017,7 @@ fn hosted_config_ignores_repository_model_provider_and_scorer() { let dir = tempfile::tempdir().unwrap(); std::fs::write( dir.path().join(".postil.yaml"), - "model:\n name: anthropic/claude-opus-4.1\n cascade: [attacker/fallback]\n scorer: anthropic/claude-haiku-4.5\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", + "model:\n name: anthropic/claude-opus-4.1\n reasoningEffort: max\n cascade: [attacker/fallback]\n scorer: anthropic/claude-haiku-4.5\n scorerReasoningEffort: high\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", ) .unwrap(); @@ -5750,6 +6036,8 @@ fn hosted_config_ignores_repository_model_provider_and_scorer() { assert!(stdout.contains("model.cascade: []")); assert!(stdout.contains("model.scorer: ")); assert!(!stdout.contains("model.scorer: openai/gpt-5.6-luna")); + assert!(stdout.contains("model.reasoningEffort: low")); + assert!(stdout.contains("model.scorerReasoningEffort: none")); assert!(stdout.contains("model.apiBase: https://openrouter.ai/api/v1")); assert!(stdout.contains("model.apiFormat: openai-compatible")); assert!(stdout.contains("model.consensus: 1")); @@ -5763,7 +6051,7 @@ fn provisional_hosted_config_uses_only_the_baked_roster() { let dir = tempfile::tempdir().unwrap(); std::fs::write( dir.path().join(".postil.yaml"), - "model:\n name: attacker/model\n cascade: [attacker/fallback]\n scorer: attacker/scorer\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", + "model:\n name: attacker/model\n reasoningEffort: max\n cascade: [attacker/fallback]\n scorer: attacker/scorer\n scorerReasoningEffort: high\n apiBase: https://attacker.invalid/v1\n apiFormat: anthropic\n consensus: 3\n", ) .unwrap(); @@ -5782,6 +6070,8 @@ fn provisional_hosted_config_uses_only_the_baked_roster() { assert!(stdout.contains("model.name: openai/gpt-5.6-luna")); assert!(stdout.contains("model.cascade: []")); assert!(stdout.contains("model.scorer: openai/gpt-5.6-luna")); + assert!(stdout.contains("model.reasoningEffort: low")); + assert!(stdout.contains("model.scorerReasoningEffort: none")); assert!(stdout.contains("model.apiBase: https://openrouter.ai:443/api/v1")); assert!(stdout.contains("model.apiFormat: openai-compatible")); assert!(stdout.contains("model.consensus: 1")); @@ -5812,8 +6102,7 @@ async fn local_review_reports_grounded_finding_and_gates() { assert_eq!(env["gate"]["failing"], true); assert_eq!(env["counts"]["error"], 1); // Embedded scoring is disabled unless the user explicitly configures it. - assert_eq!(env["usage"]["promptTokens"], 130); - assert_eq!(env["usage"]["completionTokens"], 60); + // Complete finding adjudication still uses the generator under the scorer role. assert_eq!(env["usageAccountingComplete"], true); let model_usage = env["modelUsage"].as_array().unwrap(); assert_eq!(model_usage.len(), 2); @@ -5823,7 +6112,10 @@ async fn local_review_reports_grounded_finding_and_gates() { assert_eq!(model_usage[0]["attempt"], 1); assert_eq!(model_usage[0]["accountingComplete"], true); assert_eq!(model_usage[1]["role"], "findingScorer"); + assert_eq!(model_usage[1]["phase"], "initial"); assert_eq!(model_usage[1]["callOrdinal"], 2); + assert_eq!(model_usage[1]["attempt"], 1); + assert_eq!(model_usage[1]["accountingComplete"], true); assert_eq!( model_usage .iter() @@ -5834,7 +6126,10 @@ async fn local_review_reports_grounded_finding_and_gates() { let requests = server.received_requests().await.unwrap(); let request: Value = requests[0].body_json().unwrap(); + let adjudication: Value = requests[1].body_json().unwrap(); assert_eq!(request["max_tokens"], 8_000); + assert_eq!(request["reasoning"], json!({"effort": "low"})); + assert_eq!(adjudication["reasoning"], json!({"effort": "none"})); assert_eq!(request["messages"].as_array().unwrap().len(), 2); let prompt_bytes = request["messages"] .as_array() @@ -5856,2835 +6151,2355 @@ fn review_without_an_explicit_model_exits_before_provider_access() { .current_dir(dir.path()) .env_remove("REVIEW_MODEL") .env_remove("REVIEW_MODEL_CASCADE") + .env_remove("MODEL_API_KEY") + .env_remove("LLM_API_KEY") + .env_remove("OPENROUTER_API_KEY") + .env_remove("POSTIL_API_KEY") .args(["review", "--diff-file"]) .arg(&diff) .assert() .code(2); let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("no review model is configured")); + assert!(stderr.contains("no API key")); + assert!(stderr.contains("openai/gpt-5.6-luna")); + assert!(stderr.contains("postil models")); } -#[tokio::test] -async fn weak_human_escalation_remains_visible_without_blocking_gate() { - let server = MockServer::start().await; - mock_review( - &server, - json!([finding_at_with_kind(41, "error", "humanEscalation", 0.05)]), - ) - .await; - +#[test] +fn bare_review_keeps_redirected_output_free_of_terminal_control_sequences() { let dir = tempfile::tempdir().unwrap(); - std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0\n").unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + assert!( + std::process::Command::new("git") + .args(["init", "--quiet"]) + .current_dir(dir.path()) + .status() + .unwrap() + .success() + ); + let output = isolated_postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - - assert_eq!(env["findings"][0]["kind"], "humanEscalation"); - assert_eq!(env["findings"][0]["confidence"], 0.05); - assert_eq!(env["gate"]["failing"], false); + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("NO_COLOR") + .arg("review") + .output() + .unwrap(); + assert!(output.status.success()); + let combined = [output.stdout, output.stderr].concat(); + assert!(!String::from_utf8_lossy(&combined).contains("\x1b[")); + assert!(!String::from_utf8_lossy(&combined).contains("Reviewing changes")); } -#[tokio::test] -async fn human_escalation_at_floor_blocks_even_at_warn_severity() { - let server = MockServer::start().await; - mock_review( - &server, - json!([finding_at_with_kind(41, "warn", "humanEscalation", 0.30)]), - ) - .await; - +#[cfg(unix)] +#[test] +fn bare_review_clears_progress_before_pretty_output_in_a_pty() { let dir = tempfile::tempdir().unwrap(); - std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0\n").unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + assert!( + std::process::Command::new("git") + .args(["init", "--quiet"]) + .current_dir(dir.path()) + .status() + .unwrap() + .success() + ); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + let output = isolated_script(&binary, &["review".to_string()]) .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(1); - let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + .env("TERM", "xterm") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("NO_COLOR") + .output() + .unwrap(); + assert!(output.status.success()); + let rendered = String::from_utf8_lossy(&output.stdout); + let summary = rendered + .find("✓ postil: no merge-relevant findings") + .expect("pretty output was not rendered"); + let before_summary = &rendered[..summary]; + let spinner = before_summary + .rfind("Reviewing changes...") + .expect("progress was not rendered"); + let clear = before_summary + .rfind("\r\x1b[2K") + .expect("progress was not cleared"); + assert!( + clear > spinner, + "pretty output began before the progress line was cleared: {rendered:?}" + ); +} - assert_eq!(env["findings"][0]["severity"], "warn"); - assert_eq!(env["gate"]["failing"], true); +#[cfg(unix)] +#[test] +fn file_artifacts_keep_human_progress_and_pretty_output_in_a_pty() { + let dir = tempfile::tempdir().unwrap(); + assert!( + std::process::Command::new("git") + .args(["init", "--quiet"]) + .current_dir(dir.path()) + .status() + .unwrap() + .success() + ); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + let envelope = dir.path().join("review.json"); + let sarif = dir.path().join("review.sarif.json"); + + for arguments in [ + vec![ + "review".to_string(), + "--output".to_string(), + "json".to_string(), + "--output-file".to_string(), + envelope.display().to_string(), + ], + vec![ + "review".to_string(), + "--sarif".to_string(), + sarif.display().to_string(), + ], + ] { + let output = isolated_script(&binary, &arguments) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("POSTIL_DEBUG") + .env_remove("NO_COLOR") + .env_remove("POSTIL_NO_PROGRESS") + .output() + .unwrap(); + assert!(output.status.success()); + let rendered = String::from_utf8_lossy(&output.stdout); + assert!(rendered.contains("Reviewing changes"), "{rendered:?}"); + assert!( + rendered.contains("✓ postil: no merge-relevant findings"), + "{rendered:?}" + ); + } + + serde_json::from_slice::(&std::fs::read(envelope).unwrap()).unwrap(); + let sarif: Value = serde_json::from_slice(&std::fs::read(sarif).unwrap()).unwrap(); + assert_eq!(sarif["version"], "2.1.0"); } +#[cfg(unix)] #[tokio::test] -async fn scorer_lowers_confidence_and_stores_both_values() { +async fn interactive_progress_controls_animation_separately_from_telemetry() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("generator-model")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), - ) - .mount(&server) - .await; - let mut scorer_response = scorer_content(json!([{ - "confidence": 0.7, - "kind": "risk", - "reason": "Impact depends on the query behavior shown here." - }])); - scorer_response["usage"]["completion_tokens_details"] = json!({"reasoning_tokens": 0}); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with(ResponseTemplate::new(200).set_body_json(scorer_response)) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - let finding = &env["findings"][0]; - assert_eq!(env["scorerModel"], "anthropic/claude-haiku-4.5"); - assert_eq!(env["scorerDisagreements"], 0); - assert_eq!(finding["confidence"], 0.7); - assert_eq!(finding["generatorConfidence"], 0.92); - assert_eq!(finding["scorerConfidence"], 0.7); - assert_eq!(finding["kind"], "risk"); - assert_eq!(finding["generatorKind"], "risk"); - assert_eq!(finding["scorerKind"], "risk"); - assert_eq!(env["usage"]["promptTokens"], 160); - assert_eq!(env["usage"]["completionTokens"], 70); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); - assert_eq!(env["modelUsage"][0]["costMicros"], 123); - assert_eq!(env["modelUsage"][1]["costMicros"], 45); - assert_eq!(env["modelUsage"][2]["costMicros"], 45); - assert_model_usage_matches_aggregate(&env); - assert!(stderr.contains("postil: attempting model: generator-model")); - assert!(stderr.contains("postil: model generator-model responded in")); - assert!(stderr.contains("postil: running scorer with anthropic/claude-haiku-4.5")); - assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 completed successfully in")); - assert!(stderr.contains("reasoning_tokens=0")); - - let requests = server.received_requests().await.unwrap(); - let scorer_request: Value = requests - .iter() - .map(|request| request.body_json::().unwrap()) - .find(|body| { - body["model"] == "anthropic/claude-haiku-4.5" - && body["messages"][0]["content"] - .as_str() - .is_some_and(|system| { - system.contains("Postil's independent second-model scorer") - }) - }) - .unwrap(); - assert_eq!(scorer_request["temperature"], 0.0); - assert_eq!(scorer_request["max_tokens"], 400); - let scorer_prompt_bytes = scorer_request["messages"] - .as_array() - .unwrap() - .iter() - .map(|message| message["content"].as_str().unwrap().len()) - .sum::(); - assert!( - scorer_prompt_bytes <= 17_000, - "qualification scorer prompt bound is too small: {scorer_prompt_bytes} bytes" - ); - let scorer_user = scorer_request["messages"][1]["content"].as_str().unwrap(); - assert!(!scorer_user.contains("0.92")); - assert!(!scorer_user.contains("\"kind\": \"risk\"")); - assert!(scorer_user.contains("diffHunk")); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_repairs_once_with_same_model_and_preserves_raw_evidence() { - let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("provider/scorer")) - .respond_with(SequentialReviewResponder { - calls: calls.clone(), - responses: Arc::new(vec![ - attribution_text("{\"sameDefect\":\"yes\",\"reason\":\"Wrong type.\"}"), - attribution_text("{\"sameDefect\":true,\"reason\":\"Both describe a retry that bypasses idempotency.\"}"), - ]), - }) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let profile = dir.path().join("candidate.json"); - std::fs::write( - &profile, - json!({ - "benchmarkProviderIdentity": postil_cli::config::MANAGED_OPENROUTER_PROVIDER_IDENTITY, - "upstreamProviderIdentity": "test-provider", - "apiBase": postil_cli::config::MANAGED_OPENROUTER_API_BASE, - "apiFormat": "openai-compatible", - "generatorChain": ["openai/gpt-5-mini"], - "consensus": 1, - "scorerChain": ["provider/scorer"], - "modelPriceBounds": [ - {"model": "openai/gpt-5-mini", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000}, - {"model": "provider/scorer", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000} - ] - }) - .to_string(), - ) - .unwrap(); - let input = dir.path().join("attribution.json"); - std::fs::write(&input, json!({ - "model": "provider/scorer", - "expectedProvider": "test-provider", - "target": { - "path": "src/payments.ts", "startLine": 41, "endLine": 41, - "contract": "A retry posts a second debit because the idempotency guard is bypassed." - }, - "candidate": { - "path": "src/payments.ts", "line": 41, "endLine": 41, - "severity": "error", "kind": "risk", - "title": "Retry duplicates the debit", - "body": "The retry skips idempotency and charges the payment again." - } - }).to_string()).unwrap(); - let out = postil() - .current_dir(dir.path()) - .env( - "POSTIL_API_BASE", - postil_cli::config::MANAGED_OPENROUTER_API_BASE, - ) - .env("CI", "true") - .env("GITHUB_API_URL", "http://127.0.0.1:9") - .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") - .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) - .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) - .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .success(); - let output: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(output["sameDefect"], true); - assert_eq!(output["model"], "provider/scorer"); - assert_eq!(output["provider"], "test-provider"); - assert_eq!(output["responseIdentities"].as_array().unwrap().len(), 2); - assert_eq!(output["rawResponses"].as_array().unwrap().len(), 2); - assert_eq!(output["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(calls.load(Ordering::SeqCst), 2); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - assert!(requests.iter().all(|request| { - let body: Value = serde_json::from_slice(&request.body).unwrap(); - body["model"] == "provider/scorer" - && body["temperature"] == 0.0 - && body["provider"]["order"] == json!(["test-provider"]) - && body["provider"]["allow_fallbacks"] == false - && body["provider"]["max_price"] == json!({"prompt": 0.435, "completion": 0.87}) - })); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_never_expands_an_empty_length_retry() { - let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("provider/scorer")) - .respond_with(SequentialReviewResponder { - calls: calls.clone(), - responses: Arc::new(vec![json!({ - "model": "provider/scorer", - "provider": "test-provider", - "choices": [{ - "finish_reason": "length", - "message": {"content": null, "reasoning": "budget exhausted"} - }], - "usage": { - "prompt_tokens": 30, - "completion_tokens": 180, - "cost": 0.000045 - } - })]), - }) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let (profile, input) = write_atomic_attribution_inputs(dir.path()); - postil() - .current_dir(dir.path()) - .env( - "POSTIL_API_BASE", - postil_cli::config::MANAGED_OPENROUTER_API_BASE, - ) - .env("CI", "true") - .env("GITHUB_API_URL", "http://127.0.0.1:9") - .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") - .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) - .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) - .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr( - predicates::str::contains( - "postil:atomic-attribution-terminal:v1:{\"category\":\"output-nonterminal-length\"", - ) - .and(predicates::str::contains("\"providerAttemptCount\":1")) - .and(predicates::str::contains( - "\"usageAccountingComplete\":true", - )), - ); - assert_eq!(calls.load(Ordering::SeqCst), 1); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 1); - for request in requests { - let body: Value = serde_json::from_slice(&request.body).unwrap(); - assert_eq!(body["max_tokens"], 180); - assert!( - serde_json::to_vec(&body).unwrap().len() - <= postil_cli::attribution::MAX_PROVIDER_REQUEST_BYTES - ); - } -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_reports_terminal_length_response() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "model": "provider/scorer", - "provider": "test-provider", - "choices": [{ - "finish_reason": "length", - "message": {"content": "{\"sameDefect\":true"} - }], - "usage": {"prompt_tokens": 30, "completion_tokens": 180, "cost": 0.000045} - }))) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let (profile, input) = write_atomic_attribution_inputs(dir.path()); - postil() - .current_dir(dir.path()) - .env( - "POSTIL_API_BASE", - postil_cli::config::MANAGED_OPENROUTER_API_BASE, - ) - .env("CI", "true") - .env("GITHUB_API_URL", "http://127.0.0.1:9") - .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") - .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) - .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) - .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr(predicates::str::contains( - "postil:atomic-attribution-terminal:v1:{\"category\":\"output-nonterminal-length\"", - )); - assert_eq!(server.received_requests().await.unwrap().len(), 1); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_reports_terminal_provider_http_status() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "0")) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let (profile, input) = write_atomic_attribution_inputs(dir.path()); - postil() - .current_dir(dir.path()) - .env( - "POSTIL_API_BASE", - postil_cli::config::MANAGED_OPENROUTER_API_BASE, - ) - .env("CI", "true") - .env("GITHUB_API_URL", "http://127.0.0.1:9") - .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") - .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) - .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) - .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr(predicates::str::contains( - "postil:atomic-attribution-terminal:v1:{\"category\":\"provider-http-429\"", - )); - assert_eq!(server.received_requests().await.unwrap().len(), 3); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_rejects_oversized_repair_before_second_provider_call() { - let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("provider/scorer")) - .respond_with(SequentialReviewResponder { - calls: calls.clone(), - responses: Arc::new(vec![attribution_text(&format!( - "{{\"sameDefect\":\"yes\",\"reason\":\"{}\"}}", - "x".repeat(4_000), - ))]), - }) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let profile = dir.path().join("candidate.json"); - std::fs::write( - &profile, - json!({ - "benchmarkProviderIdentity": postil_cli::config::MANAGED_OPENROUTER_PROVIDER_IDENTITY, - "upstreamProviderIdentity": "test-provider", - "apiBase": postil_cli::config::MANAGED_OPENROUTER_API_BASE, - "apiFormat": "openai-compatible", - "generatorChain": ["openai/gpt-5-mini"], - "consensus": 1, - "scorerChain": ["provider/scorer"], - "modelPriceBounds": [ - {"model": "openai/gpt-5-mini", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000}, - {"model": "provider/scorer", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000} - ] - }) - .to_string(), - ) - .unwrap(); - let input = dir.path().join("attribution.json"); - std::fs::write( - &input, - json!({ - "model": "provider/scorer", - "expectedProvider": "test-provider", - "target": { - "path": "src/payments.ts", "startLine": 41, "endLine": 41, - "contract": "A retry posts a second debit because the idempotency guard is bypassed." - }, - "candidate": { - "path": "src/payments.ts", "line": 41, "endLine": 41, - "severity": "error", "kind": "risk", - "title": "Retry duplicates the debit", - "body": "The retry skips idempotency and charges the payment again." - } - }) - .to_string(), - ) - .unwrap(); - postil() - .current_dir(dir.path()) - .env( - "POSTIL_API_BASE", - postil_cli::config::MANAGED_OPENROUTER_API_BASE, - ) - .env("CI", "true") - .env("GITHUB_API_URL", "http://127.0.0.1:9") - .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") - .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) - .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) - .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr( - predicates::str::contains("model provider request failed").and( - predicates::str::contains( - "postil:atomic-attribution-terminal:v1:{\"category\":\"provider-request-too-large\"", - ), - ), - ); - assert_eq!(calls.load(Ordering::SeqCst), 1); - assert_eq!(server.received_requests().await.unwrap().len(), 1); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_rejects_off_region_without_provider_call() { - let server = MockServer::start().await; - let dir = tempfile::tempdir().unwrap(); - let input = dir.path().join("attribution.json"); - std::fs::write(&input, json!({ - "model": "provider/scorer", - "expectedProvider": "test-provider", - "target": {"path": "src/payments.ts", "startLine": 41, "endLine": 41, "contract": "A retry posts a second debit."}, - "candidate": {"path": "src/payments.ts", "line": 42, "endLine": 42, "severity": "error", "kind": "risk", "title": "Other issue", "body": "A nearby line has another defect."} - }).to_string()).unwrap(); - postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr(predicates::str::contains( - "candidate anchor inside the exact authored region", - )); - assert!(server.received_requests().await.unwrap().is_empty()); -} - -#[cfg(all(feature = "qualification-candidate", unix))] -#[test] -fn hidden_atomic_attribution_does_not_follow_input_symlinks() { - use std::os::unix::fs::symlink; - - let dir = tempfile::tempdir().unwrap(); - let target = dir.path().join("target.json"); - let link = dir.path().join("attribution.json"); - std::fs::write(&target, "{}").unwrap(); - symlink(&target, &link).unwrap(); - postil() - .current_dir(dir.path()) - .args(["atomic-attribution", "--input"]) - .arg(&link) - .assert() - .failure() - .stderr(predicates::str::contains( - "open atomic attribution input without following links", - )); -} - -#[cfg(feature = "qualification-candidate")] -#[test] -fn hidden_atomic_attribution_rejects_non_regular_input() { - let dir = tempfile::tempdir().unwrap(); - let input = dir.path().join("input-directory"); - std::fs::create_dir(&input).unwrap(); - postil() - .current_dir(dir.path()) - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr(predicates::str::contains( - "atomic attribution input must be a regular file", - )); -} - -#[cfg(feature = "qualification-candidate")] -#[test] -fn hidden_atomic_attribution_rejects_oversized_input_before_parsing() { - let dir = tempfile::tempdir().unwrap(); - let input = dir.path().join("attribution.json"); - std::fs::write(&input, vec![b' '; 4 * 1024 + 1]).unwrap(); - postil() - .current_dir(dir.path()) - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr(predicates::str::contains( - "atomic attribution input exceeds 4096 bytes", - )); -} - -#[cfg(feature = "qualification-candidate")] -#[tokio::test] -async fn hidden_atomic_attribution_rejects_provider_substitution() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "model": "provider/scorer", - "provider": "substituted-provider", - "choices": [{"finish_reason": "stop", "message": {"content": "{\"sameDefect\":true,\"reason\":\"Same defect.\"}"}}], - "usage": {"prompt_tokens": 30, "completion_tokens": 10, "cost": 0.000045} - }))) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let input = dir.path().join("attribution.json"); - std::fs::write( - &input, - json!({ - "model": "provider/scorer", - "expectedProvider": "test-provider", - "target": {"path": "src/payments.ts", "startLine": 41, "endLine": 41, "contract": "A retry posts a second debit."}, - "candidate": {"path": "src/payments.ts", "line": 41, "endLine": 41, "severity": "error", "kind": "risk", "title": "Duplicate debit", "body": "The retry posts another debit."} - }) - .to_string(), - ) - .unwrap(); - postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_SCORER_MODEL", "provider/scorer") - .args(["atomic-attribution", "--input"]) - .arg(&input) - .assert() - .failure() - .stderr( - predicates::str::contains("atomic attribution response identity does not match").and( - predicates::str::contains( - "postil:atomic-attribution-terminal:v1:{\"category\":\"response-identity-mismatch\"", - ), - ), - ); - assert_eq!(server.received_requests().await.unwrap().len(), 3); -} - -#[tokio::test] -async fn same_model_generator_and_scorer_emit_separate_balanced_usage_rows() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("same-model")) - .and(body_string_contains( - "Postil's independent second-model scorer", - )) - .respond_with( - ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ - "confidence": 0.8, - "kind": "risk", - "reason": "The changed line contains the reported unsafe data flow." - }]))), - ) - .with_priority(1) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("same-model")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), - ) - .with_priority(2) - .expect(1) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "same-model") - .env("REVIEW_SCORER_MODEL", "same-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["usage"]["promptTokens"], 160); - assert_eq!(envelope["usage"]["completionTokens"], 70); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); - assert!( - envelope["modelUsage"] - .as_array() - .unwrap() - .iter() - .all(|entry| entry["model"] == "same-model") - ); - assert_eq!(envelope["modelUsage"][0]["role"], "reviewGenerator"); - assert_eq!(envelope["modelUsage"][1]["role"], "findingScorer"); - assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); - assert_eq!(envelope["modelUsage"][0]["callOrdinal"], 1); - assert_eq!(envelope["modelUsage"][1]["callOrdinal"], 2); - assert_eq!(envelope["modelUsage"][2]["callOrdinal"], 3); - assert_model_usage_matches_aggregate(&envelope); -} - -#[tokio::test] -async fn scorer_confidence_below_minimum_is_suppressed_and_nonblocking() { - let server = MockServer::start().await; - let finding = json!({ - "path": "src/orders.rs", - "line": 21, - "severity": "error", - "kind": "risk", - "confidence": 0.92, - "title": "Order update may race without a row lock", - "body": "Verify load_order_for_update issues FOR UPDATE before update_order writes the row.", - "evidence": "let row = load_order_for_update(id);" - }); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("generator-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([finding])))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with( - ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ - "confidence": 0.1, - "kind": "risk", - "reason": "Independent evidence does not support the generator claim." - }]))), - ) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0.6\n").unwrap(); - let diff = dir.path().join("change.diff"); - let diff_text = concat!( - "diff --git a/src/orders.rs b/src/orders.rs\n", - "--- a/src/orders.rs\n", - "+++ b/src/orders.rs\n", - "@@ -20,2 +20,3 @@ fn update_order_record() {\n", - " context line\n", - "+let row = load_order_for_update(id);\n", - " update_order(row);\n", - "diff --git a/tests/orders.rs b/tests/orders.rs\n", - "--- a/tests/orders.rs\n", - "+++ b/tests/orders.rs\n", - "@@ -60,2 +60,3 @@ fn load_order_locks_the_row() {\n", - " let query = load_order_for_update_query(7);\n", - "+assert!(query.contains(\"FOR UPDATE\"));\n", - " assert!(query.contains(\"WHERE id = $1\"));\n", - ); - let parsed = postil_cli::diff::parse(diff_text); - assert!(parsed.complete, "{parsed:#?}\n{diff_text}"); - std::fs::write(&diff, diff_text).unwrap(); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"].as_array().unwrap().len(), 0); - assert_eq!(envelope["silent"], true); - assert_eq!(envelope["gate"]["failing"], false); - assert_eq!(envelope["counts"]["error"], 0); - assert_eq!(envelope["counts"]["suppressed"], 1); - assert_eq!(envelope["suppressedFindings"].as_array().unwrap().len(), 1); - assert_eq!( - envelope["suppressedFindings"][0]["reason"], - "belowConfidence" - ); - assert_eq!( - envelope["suppressedFindings"][0]["finding"]["path"], - "src/orders.rs" - ); - assert_eq!(envelope["scorerModel"], "anthropic/claude-haiku-4.5"); - - let requests = server.received_requests().await.unwrap(); - let scorer_request: Value = requests - .iter() - .map(|request| request.body_json::().unwrap()) - .find(|body| { - body["model"] == "anthropic/claude-haiku-4.5" - && body["messages"][0]["content"] - .as_str() - .is_some_and(|system| { - system.contains("Postil's independent second-model scorer") - }) - }) - .unwrap(); - let scorer_user = scorer_request["messages"][1]["content"].as_str().unwrap(); - assert!(scorer_user.contains("relatedEvidence")); - assert!(scorer_user.contains("### tests/orders.rs")); - assert!(scorer_user.contains("FOR UPDATE")); -} - -#[tokio::test] -async fn malformed_scorer_reason_gets_one_same_model_schema_repair() { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.92)]), - ) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .and(body_string_contains("failed schema validation")) - .respond_with( - ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ - "confidence": 0.75, - "kind": "risk", - "reason": "This is a concrete defect." - }]))), - ) - .with_priority(1) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with( - ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ - "confidence": 0.75, - "kind": "risk", - "reason": "This reason is incomplete" - }]))), - ) - .with_priority(2) - .expect(1) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert_eq!(envelope["scorerModel"], "anthropic/claude-haiku-4.5"); - assert_eq!(envelope["findings"][0]["scorerKind"], "risk"); - assert_eq!(envelope["modelIncidents"][0]["phase"], "scorer"); - assert_eq!(envelope["modelIncidents"][0]["category"], "invalidOutput"); - assert_eq!(envelope["modelIncidents"][0]["recovered"], true); - assert_eq!(envelope["modelIncidents"][0]["recovery"], "repair"); - assert_eq!(envelope["usage"]["promptTokens"], 190); - assert_eq!(envelope["usage"]["completionTokens"], 80); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 4); - assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); - assert_eq!(envelope["modelUsage"][2]["phase"], "initial"); - assert_eq!(envelope["modelUsage"][3]["role"], "findingScorer"); - assert_eq!(envelope["modelUsage"][3]["phase"], "schemaRepair"); - assert_eq!(envelope["modelUsage"][3]["callOrdinal"], 4); - assert_model_usage_matches_aggregate(&envelope); - assert!(stderr.contains("requesting one schema repair")); -} - -#[tokio::test] -async fn generic_provider_repairs_each_malformed_ordered_scorer_shape() { - let valid = json!([{ - "confidence": 0.75, - "kind": "risk", - "reason": "This is a concrete defect." - }]); - let byte_overflow = json!([{ - "confidence": 0.75, - "kind": "risk", - "reason": format!("{}。", "界".repeat(80)) - }]) - .to_string(); - let cases = [ - ( - "unknown-field", - r#"[{"index":0,"confidence":0.75,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), - ), - ( - "negative-confidence", - r#"[{"confidence":-1,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), - ), - ( - "high-confidence", - r#"[{"confidence":5,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), - ), - ( - "raw-nan", - r#"[{"confidence":NaN,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), - ), - ( - "string-nan", - r#"[{"confidence":"NaN","kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), - ), - ("missing-entry", "[]".to_string()), - ( - "duplicate-entry", - r#"[{"confidence":0.75,"kind":"risk","reason":"This is a concrete defect."},{"confidence":0.75,"kind":"risk","reason":"This repeats the same input."}]"#.to_string(), - ), - ( - "edge-whitespace", - r#"[{"confidence":0.75,"kind":"risk","reason":" Leading whitespace is invalid."}]"#.to_string(), - ), - ( - "control-character", - r#"[{"confidence":0.75,"kind":"risk","reason":"A control\u0000character is invalid."}]"#.to_string(), - ), - ( - "missing-punctuation", - r#"[{"confidence":0.75,"kind":"risk","reason":"This reason is incomplete"}]"#.to_string(), - ), - ("byte-overflow", byte_overflow), - ]; - - for (label, malformed) in cases { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.92)]), - ) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("scorer-model")) - .and(body_string_contains("failed schema validation")) - .respond_with(ResponseTemplate::new(200).set_body_json(scorer_content(valid.clone()))) - .with_priority(1) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("scorer-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(scorer_text(&malformed))) - .with_priority(2) - .expect(1) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "scorer-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["scorerModel"], "scorer-model", "case {label}"); - assert_eq!( - envelope["findings"][0]["scorerConfidence"], 0.75, - "case {label}" - ); - assert_eq!( - envelope["modelIncidents"][0]["category"], "invalidOutput", - "case {label}" - ); - assert_eq!( - envelope["modelIncidents"][0]["recovery"], "repair", - "case {label}" - ); - assert_eq!( - envelope["modelUsage"].as_array().unwrap().len(), - 4, - "case {label}" - ); - } -} - -#[tokio::test] -async fn slow_explicit_scorer_times_out_without_unqualified_fallback() { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.92)]), - ) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(scorer_content(json!([]))), - ) - .mount(&server) - .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert!(env.get("scorerModel").is_none()); - assert!(env["scorerError"].as_str().unwrap().contains("timed out")); - assert_eq!(env["usageAccountingComplete"], false); - assert_eq!(env["modelIncidents"][0]["phase"], "scorer"); - assert_eq!(env["modelIncidents"][0]["category"], "timeout"); - assert_eq!(env["modelIncidents"][0]["recovered"], false); - assert!(env["modelIncidents"][0].get("recovery").is_none()); - let models: Vec<_> = env["modelUsage"] - .as_array() - .unwrap() - .iter() - .map(|entry| entry["model"].as_str().unwrap()) - .collect(); - assert!(models.contains(&"generator-model")); - // The timed-out call is retained as an incomplete row with no invented - // token count. - assert!(models.contains(&"anthropic/claude-haiku-4.5")); - assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 timed out after")); - assert!(stderr.contains("no fallback scorers remain")); - assert!(!stderr.contains("openai/gpt-5-mini")); -} - -#[tokio::test] -async fn scorer_kind_escalation_into_configured_blocking_kind_takes_effect() { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.9)]), - ) - .await; - mock_scorer_model( - &server, - "anthropic/claude-haiku-4.5", - json!([{ - "confidence": 0.88, - "kind": "guardrail", - "reason": "The finding violates the configured rule." - }]), - ) - .await; - - let dir = tempfile::tempdir().unwrap(); - std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n blockOnKinds:\n - guardrail\n", - ) - .unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(1); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(env["findings"][0]["kind"], "guardrail"); - assert_eq!(env["findings"][0]["generatorKind"], "risk"); - assert_eq!(env["findings"][0]["scorerKind"], "guardrail"); - assert_eq!(env["gate"]["failing"], true); -} - -#[tokio::test] -async fn scorer_kind_deescalation_from_blocking_kind_is_ignored() { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at_with_kind(41, "warn", "guardrail", 0.9)]), - ) - .await; - mock_scorer_model( - &server, - "anthropic/claude-haiku-4.5", - json!([{ - "confidence": 0.88, - "kind": "risk", - "reason": "The finding is not a guardrail violation." - }]), - ) - .await; - - let dir = tempfile::tempdir().unwrap(); - std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n blockOnKinds:\n - guardrail\n", - ) - .unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(1); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(env["findings"][0]["kind"], "guardrail"); - assert_eq!(env["findings"][0]["generatorKind"], "guardrail"); - assert_eq!(env["findings"][0]["scorerKind"], "risk"); - assert_eq!(env["gate"]["failing"], true); -} - -#[tokio::test] -async fn large_confidence_disagreement_escalates_to_uncertainty_with_default_gate() { - let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 1.0)]), - ) - .await; - mock_scorer_model( - &server, - "anthropic/claude-haiku-4.5", - json!([{ - "confidence": 0.6, - "kind": "risk", - "reason": "The finding has weak supporting evidence." - }]), - ) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .success(); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(env["findings"][0]["confidence"], 0.6); - assert_eq!(env["findings"][0]["kind"], "uncertainty"); - assert_eq!(env["findings"][0]["generatorKind"], "risk"); - assert_eq!(env["findings"][0]["scorerKind"], "risk"); - assert_eq!(env["scorerDisagreements"], 1); -} - -fn uncertainty_finding(body: &str) -> Value { - uncertainty_finding_with_severity(body, "warn") -} - -fn uncertainty_finding_with_severity(body: &str, severity: &str) -> Value { - json!({ - "path": "src/auth.rs", - "line": 41, - "severity": severity, - "kind": "uncertainty", - "confidence": 0.9, - "title": "Resolve the caller contract", - "body": body, - "evidence": "let token = format!(\"{}\", user_input);" - }) -} - -fn concise_finding(body: &str) -> Value { - json!({ - "path": "src/auth.rs", - "line": 41, - "severity": "warn", - "kind": "risk", - "confidence": 0.9, - "title": "Retry bypasses the idempotency guard", - "body": body, - "evidence": "let token = format!(\"{}\", user_input);" - }) -} - -fn enable_uncertainty_resolution(directory: &std::path::Path) { - std::fs::write( - directory.join(".postil.yaml"), - "review:\n uncertaintyResolution: true\n", + mount_github_complete_diff(&server, 7).await; + mount_static_github_pr(&server).await; + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + + let interactive = isolated_script( + &binary, + &[ + "review".to_string(), + "--repo".to_string(), + "acme/api".to_string(), + "--pr".to_string(), + "7".to_string(), + ], ) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("POSTIL_DEBUG") + .env_remove("NO_COLOR") + .output() .unwrap(); -} + assert!(interactive.status.success()); + let rendered = String::from_utf8_lossy(&interactive.stdout); + assert!(rendered.contains("Reviewing changes")); + assert!( + !rendered.contains("postil: github operation="), + "interactive review exposed routine GitHub telemetry: {rendered:?}" + ); -fn commit_uncertainty_fixture(directory: &std::path::Path) { - let run = |args: &[&str]| { - let output = std::process::Command::new("git") - .arg("-C") - .arg(directory) - .args(args) - .output() - .unwrap(); - assert!( - output.status.success(), - "git {} failed: {}", - args.join(" "), - String::from_utf8_lossy(&output.stderr) - ); - String::from_utf8(output.stdout).unwrap().trim().to_string() - }; - run(&["init", "--quiet"]); - run(&["add", "-A"]); - let tree = run(&["write-tree"]); - let output = std::process::Command::new("git") - .arg("-C") - .arg(directory) - .args(["commit-tree", &tree, "-m", "fixture"]) - .env("GIT_AUTHOR_NAME", "Fixture") - .env("GIT_AUTHOR_EMAIL", "fixture@example.invalid") - .env("GIT_COMMITTER_NAME", "Fixture") - .env("GIT_COMMITTER_EMAIL", "fixture@example.invalid") + let machine = postil() + .current_dir(dir.path()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", "--repo", "acme/api", "--pr", "7", "--output", "json", + ]) .output() .unwrap(); - assert!(output.status.success()); - let commit = String::from_utf8(output.stdout).unwrap().trim().to_string(); - run(&["update-ref", "HEAD", &commit]); -} + assert!(machine.status.success()); + let telemetry = String::from_utf8_lossy(&machine.stderr); + assert!( + telemetry.contains("postil: github operation="), + "machine review omitted GitHub telemetry: {telemetry:?}" + ); -fn stage_uncertainty_review(directory: &std::path::Path) { - std::fs::create_dir_all(directory.join("src")).unwrap(); - std::fs::write( - directory.join("src/auth.rs"), - format!( - "{}let token = format!(\"{{}}\", user_input);\nexec_query(&token);\n", - "\n".repeat(40) + for (arguments, no_progress_environment) in [ + ( + vec![ + "review".to_string(), + "--repo".to_string(), + "acme/api".to_string(), + "--pr".to_string(), + "7".to_string(), + "--no-progress".to_string(), + ], + None, ), - ) - .unwrap(); - let output = std::process::Command::new("git") - .arg("-C") - .arg(directory) - .args(["add", "src/auth.rs"]) - .output() - .unwrap(); - assert!(output.status.success()); -} - -#[tokio::test] -async fn uncertainty_resolution_refuted_with_verbatim_evidence_is_suppressed() { - let server = MockServer::start().await; - let original_body = "`src/reference.rs` may omit verification before insertion. Preserve the verification step."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - &json!({ - "resolution": "refuted", - "revisedBody": "", - "evidence": "verification is always inserted" - }) - .to_string(), - 1, - ) - .await; + ( + vec![ + "review".to_string(), + "--repo".to_string(), + "acme/api".to_string(), + "--pr".to_string(), + "7".to_string(), + ], + Some("1"), + ), + ] { + let mut automation = isolated_script(&binary, &arguments); + automation + .current_dir(dir.path()) + .env("TERM", "xterm") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("POSTIL_DEBUG") + .env_remove("NO_COLOR") + .env_remove("POSTIL_NO_PROGRESS"); + if let Some(value) = no_progress_environment { + automation.env("POSTIL_NO_PROGRESS", value); + } + let static_progress = automation.output().unwrap(); + assert!(static_progress.status.success()); + let static_progress = String::from_utf8_lossy(&static_progress.stdout); + assert!( + static_progress.contains("postil: reviewing changes..."), + "{static_progress:?}" + ); + assert!( + static_progress.contains("postil: review complete; no findings"), + "{static_progress:?}" + ); + assert!(!static_progress.contains("\x1b[2K"), "{static_progress:?}"); + assert!( + !static_progress.contains("postil: github operation="), + "static human progress exposed routine GitHub telemetry: {static_progress:?}" + ); + } - let directory = tempfile::tempdir().unwrap(); - enable_uncertainty_resolution(directory.path()); - std::fs::create_dir_all(directory.path().join("src")).unwrap(); - std::fs::write( - directory.path().join("src/reference.rs"), - "// verification is always inserted\n", + let verbose = isolated_script( + &binary, + &[ + "review".to_string(), + "--repo".to_string(), + "acme/api".to_string(), + "--pr".to_string(), + "7".to_string(), + "--verbose".to_string(), + ], ) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("POSTIL_DEBUG") + .env_remove("NO_COLOR") + .env_remove("POSTIL_NO_PROGRESS") + .output() .unwrap(); - commit_uncertainty_fixture(directory.path()); - stage_uncertainty_review(directory.path()); - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--staged", "--output", "json"]) - .assert() - .success(); - - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert!(envelope["findings"].as_array().unwrap().is_empty()); - assert_eq!(envelope["counts"]["suppressed"], 1); - assert_eq!(envelope["suppressedFindings"].as_array().unwrap().len(), 1); - assert_eq!( - envelope["suppressedFindings"][0]["finding"]["body"], - original_body + assert!(verbose.status.success()); + let verbose = String::from_utf8_lossy(&verbose.stdout); + assert!( + verbose.contains("postil: github operation="), + "verbose interactive review omitted GitHub telemetry: {verbose:?}" ); - assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); - assert_model_usage_matches_aggregate(&envelope); + assert!(!verbose.contains("Reviewing changes"), "{verbose:?}"); } +#[cfg(unix)] #[tokio::test] -async fn uncertainty_resolution_confirmed_replaces_only_the_body() { +async fn no_progress_keeps_provider_telemetry_collapsed_until_verbose() { let server = MockServer::start().await; - let original_body = "`src/reference.rs` may omit `service_summary` before scheduling. Preserve the summary when scheduling the job."; - let revised_body = - "`src/reference.rs` omits `service_summary` when scheduling the notification job."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - &json!({ - "resolution": "confirmed", - "revisedBody": revised_body, - "evidence": "service_summary is omitted" - }) - .to_string(), - 1, - ) - .await; + mock_review(&server, json!([])).await; + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + + let run = |mode: &str| { + let mut command = isolated_script( + &binary, + &[ + "review".to_string(), + "--diff-file".to_string(), + diff.display().to_string(), + mode.to_string(), + ], + ); + command + .current_dir(dir.path()) + .env("TERM", "xterm") + .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_MODEL", "generator-model") + .env("MODEL_API_KEY", fixture_credential("provider")) + .env("POSTIL_DISABLE_SCORER", "1") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("POSTIL_DEBUG") + .env_remove("NO_COLOR") + .env_remove("POSTIL_NO_PROGRESS") + .output() + .unwrap() + }; - let directory = tempfile::tempdir().unwrap(); - enable_uncertainty_resolution(directory.path()); - std::fs::create_dir_all(directory.path().join("src")).unwrap(); - std::fs::write( - directory.path().join("src/reference.rs"), - "// service_summary is omitted\n", - ) - .unwrap(); - commit_uncertainty_fixture(directory.path()); - stage_uncertainty_review(directory.path()); - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--staged", "--output", "json"]) - .assert() - .success(); + let static_progress = run("--no-progress"); + assert!(static_progress.status.success()); + let static_progress = String::from_utf8_lossy(&static_progress.stdout); + assert!(static_progress.contains("postil: reviewing changes...")); + assert!(static_progress.contains("postil: review complete; no findings")); + assert!(!static_progress.contains("postil: llm attempt")); + assert!(!static_progress.contains("postil: queued source request")); + assert!(!static_progress.contains("\x1b[2K")); + + let verbose = run("--verbose"); + assert!(verbose.status.success()); + let verbose = String::from_utf8_lossy(&verbose.stdout); + assert!(verbose.contains("postil: llm attempt"), "{verbose:?}"); + assert!( + verbose.contains("postil: queued source request"), + "{verbose:?}" + ); + assert!(!verbose.contains("Reviewing changes"), "{verbose:?}"); +} - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - let finding = &envelope["findings"][0]; - assert_eq!(finding["body"], revised_body); - assert_eq!(finding["kind"], "uncertainty"); - assert_eq!(finding["severity"], "warn"); - assert_eq!(finding["confidence"], 0.9); - assert_eq!(finding["path"], "src/auth.rs"); - assert_eq!(finding["line"], 41); - assert_eq!(envelope["counts"]["suppressed"], 0); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); - assert_eq!(envelope["modelUsage"][0]["role"], "reviewGenerator"); - assert_eq!(envelope["modelUsage"][1]["role"], "reviewGenerator"); - assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); - assert_model_usage_matches_aggregate(&envelope); +#[cfg(unix)] +#[test] +fn bare_review_keeps_warnings_visible_and_marks_degraded_completion_in_a_pty() { + let dir = tempfile::tempdir().unwrap(); + assert!( + std::process::Command::new("git") + .args(["init", "--quiet"]) + .current_dir(dir.path()) + .status() + .unwrap() + .success() + ); + std::fs::write(dir.path().join("untracked.txt"), "review me\n").unwrap(); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + let output = isolated_script(&binary, &["review".to_string()]) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("NO_COLOR") + .output() + .unwrap(); + assert!(output.status.success()); + let rendered = String::from_utf8_lossy(&output.stdout); + assert!(rendered.contains("Reviewing changes")); + assert!(rendered.contains("\x1b[2K")); + assert!(rendered.contains("untracked files were not reviewed")); + assert!(rendered.contains("review complete; no findings; warnings were also reported")); } -#[tokio::test] -async fn uncertainty_resolution_malformed_response_preserves_original_finding() { - let server = MockServer::start().await; - let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution(&server, "generator-model", "{not valid JSON", 2).await; +#[cfg(unix)] +#[test] +fn bare_review_respects_no_color_in_a_pty() { + let dir = tempfile::tempdir().unwrap(); + assert!( + std::process::Command::new("git") + .args(["init", "--quiet"]) + .current_dir(dir.path()) + .status() + .unwrap() + .success() + ); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + let output = isolated_script(&binary, &["review".to_string()]) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env("NO_COLOR", "1") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .output() + .unwrap(); + assert!(output.status.success()); + let rendered = String::from_utf8_lossy(&output.stdout); + assert!(rendered.contains("reviewing changes")); + assert!( + !rendered.contains("\x1b["), + "NO_COLOR output contained terminal controls: {rendered:?}" + ); + assert!(rendered.contains("review complete")); +} - let directory = tempfile::tempdir().unwrap(); - enable_uncertainty_resolution(directory.path()); - std::fs::create_dir_all(directory.path().join("src")).unwrap(); - std::fs::write( - directory.path().join("src/reference.rs"), - "repository evidence\n", +#[cfg(unix)] +#[test] +fn interactive_early_errors_replace_progress_with_an_incomplete_result() { + let dir = tempfile::tempdir().unwrap(); + let binary = Command::cargo_bin("postil") + .unwrap() + .get_program() + .to_string_lossy() + .into_owned(); + let output = isolated_script( + &binary, + &[ + "review".to_string(), + "--output-file".to_string(), + dir.path().join("review.json").display().to_string(), + ], ) + .current_dir(dir.path()) + .env("TERM", "xterm") + .env_remove("CI") + .env_remove("POSTIL_HOSTED_MODE") + .env_remove("RUST_LOG") + .env_remove("NO_COLOR") + .output() .unwrap(); - commit_uncertainty_fixture(directory.path()); - stage_uncertainty_review(directory.path()); - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--staged", "--output", "json"]) - .assert() - .success(); + let rendered = String::from_utf8_lossy(&output.stdout); + assert!(rendered.contains("Reviewing changes"), "{rendered:?}"); + assert!( + rendered.contains("review incomplete; an operational error prevented completion"), + "{rendered:?}" + ); + assert!( + rendered.contains("--output-file requires --output or --output-json"), + "{rendered:?}" + ); +} - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["findings"][0]["kind"], "uncertainty"); - assert_eq!(envelope["counts"]["suppressed"], 0); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 4); - assert_model_usage_matches_aggregate(&envelope); +#[test] +fn models_and_config_explain_the_embedded_default() { + let dir = tempfile::tempdir().unwrap(); + let models = isolated_postil() + .env("MODEL_API_KEY", "models-secret-sentinel") + .env( + "POSTIL_API_BASE", + "https://models-secret-sentinel.invalid/v1", + ) + .args(["models"]) + .assert() + .success() + .get_output() + .stdout + .clone(); + let models = String::from_utf8(models).unwrap(); + assert!(models.contains("Postil model support (offline)")); + assert!(models.contains("No model setting is required")); + assert!(models.contains("Embedded local reviewer (Luna): openai/gpt-5.6-luna")); + assert!(models.contains("Reviewer source: embedded default")); + assert!(models.contains("Reviewer reasoning effort: low")); + assert!(models.contains("Local scorer: disabled")); + assert!(models.contains("Hosted scorer candidate: openai/gpt-5.6-luna")); + assert!(models.contains("Hosted scorer reasoning effort: none")); + assert!(models.contains("does not maintain a fixed local model-ID allowlist")); + assert!(models.contains("OpenAI-compatible endpoints accept any non-empty endpoint model ID")); + assert!(models.contains("OpenRouter commonly uses provider/model")); + assert!( + models.contains( + "Native Anthropic endpoints accept any non-empty Anthropic endpoint model ID" + ) + ); + assert!(models.contains("does not mean the model is hosted-qualified")); + assert!(models.contains("Hosted qualified model IDs: none (no embedded qualified profile)")); + assert!(models.contains("max|xhigh|high|medium|low|minimal|none")); + assert!(models.contains("postil doctor")); + assert!(models.contains("postil review --model provider/model")); + assert!(models.contains("model.apiFormat: anthropic")); + assert!(models.contains("model.name, model.reasoningEffort, and model.scorerReasoningEffort")); + assert!(!models.contains("models-secret-sentinel")); + let config = isolated_postil() + .current_dir(dir.path()) + .args(["config"]) + .assert() + .success() + .get_output() + .stdout + .clone(); + let config = String::from_utf8(config).unwrap(); + assert!(config.contains("model.source: embedded default")); + assert!(config.contains("model.name: openai/gpt-5.6-luna")); + assert!(config.contains("model.reasoningEffort: low")); + assert!(config.contains("model.reasoningEffort.source: embedded default")); + assert!(config.contains("model.scorer.enabled: false")); + assert!(config.contains("model.scorerReasoningEffort: none")); + assert!(config.contains("model.scorerReasoningEffort.source: embedded default")); } -#[tokio::test] -async fn uncertainty_resolution_defaults_on_and_fails_open_when_unresolved() { - let server = MockServer::start().await; - let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, - 1, - ) - .await; +#[test] +fn conflicting_local_sources_are_usage_errors() { + for arguments in [ + vec!["review", "--staged", "--base", "main"], + vec!["review", "--staged", "--diff-file", "change.diff"], + vec!["review", "--base", "main", "--diff-file", "change.diff"], + ] { + isolated_postil().args(arguments).assert().code(2); + } +} - let directory = tempfile::tempdir().unwrap(); - std::fs::create_dir_all(directory.path().join("src")).unwrap(); +#[test] +fn config_reports_separate_environment_reasoning_provenance() { + let dir = tempfile::tempdir().unwrap(); std::fs::write( - directory.path().join("src/reference.rs"), - "repository evidence\n", + dir.path().join(".postil.yaml"), + "model:\n reasoningEffort: medium\n scorerReasoningEffort: high\n", ) .unwrap(); - commit_uncertainty_fixture(directory.path()); - stage_uncertainty_review(directory.path()); let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--staged", "--output", "json"]) + .current_dir(dir.path()) + .env("REVIEW_REASONING_EFFORT", "xhigh") + .env("REVIEW_SCORER_REASONING_EFFORT", "minimal") + .args(["config"]) .assert() - .success(); - - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["counts"]["suppressed"], 0); + .success() + .get_output() + .stdout + .clone(); + let output = String::from_utf8(output).unwrap(); + assert!(output.contains("model.reasoningEffort: xhigh")); + assert!(output.contains("model.reasoningEffort.source: environment")); + assert!(output.contains("model.scorerReasoningEffort: minimal")); + assert!(output.contains("model.scorerReasoningEffort.source: environment")); } #[tokio::test] -async fn uncertainty_resolution_uses_diff_when_no_referenced_files_exist() { +async fn cli_reasoning_effort_overrides_environment_and_config_in_provider_request() { let server = MockServer::start().await; - let original_body = "The added call may pass the wrong value to the query executor."; - let revised_body = "The added call passes the request token directly to the query executor."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - &json!({ - "resolution": "confirmed", - "revisedBody": revised_body, - "evidence": "exec_query(&token);" - }) - .to_string(), - 1, + mock_review(&server, json!([])).await; + let dir = tempfile::tempdir().unwrap(); + std::fs::write( + dir.path().join(".postil.yaml"), + "model:\n reasoningEffort: medium\n", ) - .await; + .unwrap(); + let diff = write_diff(dir.path()); - let directory = tempfile::tempdir().unwrap(); - enable_uncertainty_resolution(directory.path()); - let diff = write_diff(directory.path()); - let output = postil() - .current_dir(directory.path()) + postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) + .env("REVIEW_REASONING_EFFORT", "high") + .args(["review", "--reasoning-effort", "xhigh", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) .assert() .success(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], revised_body); - assert_eq!(envelope["findings"][0]["severity"], "warn"); - assert_eq!(envelope["counts"]["suppressed"], 0); + let requests = server.received_requests().await.unwrap(); + let request: Value = requests[0].body_json().unwrap(); + assert_eq!(request["reasoning"], json!({"effort": "xhigh"})); } #[tokio::test] -async fn unresolved_error_uncertainty_is_retained_but_demoted_to_warn() { +async fn invalid_cli_reasoning_effort_fails_before_provider_access() { let server = MockServer::start().await; - let original_body = "The added call may pass the wrong value to the query executor."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding_with_severity(original_body, "error")]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, - 1, - ) - .await; + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); - let directory = tempfile::tempdir().unwrap(); - enable_uncertainty_resolution(directory.path()); - let diff = write_diff(directory.path()); let output = postil() - .current_dir(directory.path()) + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) + .args(["review", "--reasoning-effort", "turbo", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) .assert() - .success(); + .code(2) + .get_output() + .stderr + .clone(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["findings"][0]["severity"], "warn"); - assert_eq!(envelope["gate"]["failing"], false); + let stderr = String::from_utf8(output).unwrap(); + assert!(stderr.contains("invalid --reasoning-effort \"turbo\"")); + assert!(stderr.contains("max|xhigh|high|medium|low|minimal|none")); + assert!(server.received_requests().await.unwrap().is_empty()); } #[tokio::test] -async fn uncertainty_resolution_explicit_off_makes_no_resolution_call() { +async fn native_anthropic_minimal_effort_fails_before_provider_access() { let server = MockServer::start().await; - let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; - mock_review_model( - &server, - "generator-model", - json!([uncertainty_finding(original_body)]), - ) - .await; - mock_uncertainty_resolution( - &server, - "generator-model", - r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, - 0, - ) - .await; + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); - let directory = tempfile::tempdir().unwrap(); - std::fs::write( - directory.path().join(".postil.yaml"), - "review:\n uncertaintyResolution: false\n", - ) - .unwrap(); - std::fs::create_dir_all(directory.path().join("src")).unwrap(); - std::fs::write( - directory.path().join("src/reference.rs"), - "repository evidence\n", - ) - .unwrap(); - commit_uncertainty_fixture(directory.path()); - let diff = write_diff(directory.path()); let output = postil() - .current_dir(directory.path()) + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) + .env("POSTIL_API_FORMAT", "anthropic") + .args(["review", "--reasoning-effort", "minimal", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) .assert() - .success(); + .code(2) + .get_output() + .stderr + .clone(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["counts"]["suppressed"], 0); + let stderr = String::from_utf8(output).unwrap(); + assert!(stderr.contains("minimal is unsupported by the Anthropic request format")); + assert!(stderr.contains("max|xhigh|high|medium|low|none")); + assert!(server.received_requests().await.unwrap().is_empty()); } #[tokio::test] -async fn concise_findings_compresses_an_overlong_body_and_preserves_other_fields() { +async fn weak_human_escalation_remains_visible_without_blocking_gate() { let server = MockServer::start().await; - let original_body = - "The retry bypasses the idempotency guard and can duplicate the transaction. " - .repeat(9) - .trim_end() - .to_string(); - let compressed_body = "The retry bypasses the idempotency guard and can duplicate the transaction. Restore the guard before retrying the operation."; - let original_finding = concise_finding(&original_body); - mock_review_model( - &server, - "generator-model", - json!([original_finding.clone()]), - ) - .await; - mock_finding_compression( + mock_review( &server, - "generator-model", - &json!({"body": compressed_body}).to_string(), - 1, + json!([finding_at_with_kind(41, "error", "humanEscalation", 0.05)]), ) .await; - let directory = tempfile::tempdir().unwrap(); - let diff = write_diff(directory.path()); - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0\n").unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) .args(["--output", "json"]) .assert() - .success(); + .code(0); + let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - let finding = &envelope["findings"][0]; - assert_eq!(finding["body"], compressed_body); - let mut actual_other_fields = finding.as_object().unwrap().clone(); - actual_other_fields.remove("body"); - assert!(actual_other_fields.remove("id").is_some()); - let mut expected_other_fields = original_finding.as_object().unwrap().clone(); - expected_other_fields.remove("body"); - assert_eq!(actual_other_fields, expected_other_fields); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); - assert_model_usage_matches_aggregate(&envelope); + assert_eq!(env["findings"][0]["kind"], "humanEscalation"); + assert_eq!(env["findings"][0]["confidence"], 0.05); + assert_eq!(env["gate"]["failing"], false); } #[tokio::test] -async fn concise_findings_malformed_response_preserves_the_original_body() { +async fn human_escalation_at_floor_blocks_even_at_warn_severity() { let server = MockServer::start().await; - let original_body = - "The retry bypasses the idempotency guard and can duplicate the transaction. " - .repeat(9) - .trim_end() - .to_string(); - mock_review_model( + mock_review( &server, - "generator-model", - json!([concise_finding(&original_body)]), + json!([finding_at_with_kind(41, "warn", "humanEscalation", 0.30)]), ) .await; - mock_finding_compression(&server, "generator-model", "{not valid JSON", 1).await; - let directory = tempfile::tempdir().unwrap(); - let diff = write_diff(directory.path()); - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0\n").unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) .args(["--output", "json"]) .assert() - .success(); + .code(1); + let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); - assert_model_usage_matches_aggregate(&envelope); + assert_eq!(env["findings"][0]["severity"], "warn"); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn concise_findings_explicit_off_makes_no_compression_call() { +async fn scorer_lowers_confidence_and_stores_both_values() { let server = MockServer::start().await; - let original_body = - "The retry bypasses the idempotency guard and can duplicate the transaction. " - .repeat(9) - .trim_end() - .to_string(); - mock_review_model( - &server, - "generator-model", - json!([concise_finding(&original_body)]), - ) - .await; - mock_finding_compression(&server, "generator-model", r#"{"body":"unused"}"#, 0).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("generator-model")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), + ) + .mount(&server) + .await; + let mut scorer_response = scorer_content(json!([{ + "confidence": 0.7, + "kind": "risk", + "reason": "Impact depends on the query behavior shown here." + }])); + scorer_response["usage"]["completion_tokens_details"] = json!({"reasoning_tokens": 0}); + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .respond_with(ResponseTemplate::new(200).set_body_json(scorer_response)) + .mount(&server) + .await; - let directory = tempfile::tempdir().unwrap(); - std::fs::write( - directory.path().join(".postil.yaml"), - "review:\n conciseFindings: false\n", - ) - .unwrap(); - let diff = write_diff(directory.path()); - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") .args(["review", "--diff-file"]) .arg(&diff) .args(["--output", "json"]) .assert() - .success(); + .code(0); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - assert_model_usage_matches_aggregate(&envelope); + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + let finding = &env["findings"][0]; + assert_eq!(env["scorerModel"], "anthropic/claude-haiku-4.5"); + assert_eq!(env["scorerDisagreements"], 0); + assert_eq!(finding["confidence"], 0.7); + assert_eq!(finding["generatorConfidence"], 0.92); + assert_eq!(finding["scorerConfidence"], 0.7); + assert_eq!(finding["kind"], "risk"); + assert_eq!(finding["generatorKind"], "risk"); + assert_eq!(finding["scorerKind"], "risk"); + assert_eq!(env["usage"]["promptTokens"], 160); + assert_eq!(env["usage"]["completionTokens"], 70); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); + assert_eq!(env["modelUsage"][0]["costMicros"], 123); + assert_eq!(env["modelUsage"][1]["costMicros"], 45); + assert_eq!(env["modelUsage"][2]["costMicros"], 45); + assert_model_usage_matches_aggregate(&env); + assert!(stderr.contains("postil: attempting model: generator-model")); + assert!(stderr.contains("postil: model generator-model responded in")); + assert!(stderr.contains("postil: running scorer with anthropic/claude-haiku-4.5")); + assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 completed successfully in")); + assert!(stderr.contains("reasoning_tokens=0")); + + let requests = server.received_requests().await.unwrap(); + let scorer_request: Value = requests + .iter() + .map(|request| request.body_json::().unwrap()) + .find(|body| { + body["model"] == "anthropic/claude-haiku-4.5" + && body["messages"][0]["content"] + .as_str() + .is_some_and(|system| { + system.contains("Postil's independent second-model scorer") + }) + }) + .unwrap(); + assert_eq!(scorer_request["temperature"], 0.0); + assert_eq!(scorer_request["max_tokens"], 400); + let scorer_prompt_bytes = scorer_request["messages"] + .as_array() + .unwrap() + .iter() + .map(|message| message["content"].as_str().unwrap().len()) + .sum::(); + assert!( + scorer_prompt_bytes <= 17_000, + "qualification scorer prompt bound is too small: {scorer_prompt_bytes} bytes" + ); + let scorer_user = scorer_request["messages"][1]["content"].as_str().unwrap(); + assert!(!scorer_user.contains("0.92")); + assert!(!scorer_user.contains("\"kind\": \"risk\"")); + assert!(scorer_user.contains("diffHunk")); } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn concise_findings_short_body_makes_no_compression_call_by_default() { +async fn hidden_atomic_attribution_repairs_once_with_same_model_and_preserves_raw_evidence() { let server = MockServer::start().await; - let original_body = "The retry bypasses the idempotency guard."; - mock_review_model( - &server, - "generator-model", - json!([concise_finding(original_body)]), + let calls = Arc::new(AtomicUsize::new(0)); + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("provider/scorer")) + .respond_with(SequentialReviewResponder { + calls: calls.clone(), + responses: Arc::new(vec![ + attribution_text("{\"sameDefect\":\"yes\",\"reason\":\"Wrong type.\"}"), + attribution_text("{\"sameDefect\":true,\"reason\":\"Both describe a retry that bypasses idempotency.\"}"), + ]), + }) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let profile = dir.path().join("candidate.json"); + std::fs::write( + &profile, + json!({ + "benchmarkProviderIdentity": postil_cli::config::MANAGED_OPENROUTER_PROVIDER_IDENTITY, + "upstreamProviderIdentity": "test-provider", + "apiBase": postil_cli::config::MANAGED_OPENROUTER_API_BASE, + "apiFormat": "openai-compatible", + "generatorChain": ["openai/gpt-5-mini"], + "consensus": 1, + "scorerChain": ["provider/scorer"], + "modelPriceBounds": [ + {"model": "openai/gpt-5-mini", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000}, + {"model": "provider/scorer", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000} + ] + }) + .to_string(), ) - .await; - mock_finding_compression(&server, "generator-model", r#"{"body":"unused"}"#, 0).await; - - let directory = tempfile::tempdir().unwrap(); - let diff = write_diff(directory.path()); - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .unwrap(); + let input = dir.path().join("attribution.json"); + std::fs::write(&input, json!({ + "model": "provider/scorer", + "expectedProvider": "test-provider", + "target": { + "path": "src/payments.ts", "startLine": 41, "endLine": 41, + "contract": "A retry posts a second debit because the idempotency guard is bypassed." + }, + "candidate": { + "path": "src/payments.ts", "line": 41, "endLine": 41, + "severity": "error", "kind": "risk", + "title": "Retry duplicates the debit", + "body": "The retry skips idempotency and charges the payment again." + } + }).to_string()).unwrap(); + let out = postil() + .current_dir(dir.path()) + .env( + "POSTIL_API_BASE", + postil_cli::config::MANAGED_OPENROUTER_API_BASE, + ) + .env("CI", "true") + .env("GITHUB_API_URL", "http://127.0.0.1:9") + .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") + .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) + .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() .success(); - - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["body"], original_body); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - assert_model_usage_matches_aggregate(&envelope); + let output: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(output["sameDefect"], true); + assert_eq!(output["model"], "provider/scorer"); + assert_eq!(output["provider"], "test-provider"); + assert_eq!(output["responseIdentities"].as_array().unwrap().len(), 2); + assert_eq!(output["rawResponses"].as_array().unwrap().len(), 2); + assert_eq!(output["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(calls.load(Ordering::SeqCst), 2); + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + assert!(requests.iter().all(|request| { + let body: Value = serde_json::from_slice(&request.body).unwrap(); + body["model"] == "provider/scorer" + && body["temperature"] == 0.0 + && body["provider"]["order"] == json!(["test-provider"]) + && body["provider"]["allow_fallbacks"] == false + && body["provider"]["max_price"] == json!({"prompt": 0.435, "completion": 0.87}) + })); } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn scorer_error_fails_open_and_preserves_generator_values() { +async fn hidden_atomic_attribution_never_expands_an_empty_length_retry() { let server = MockServer::start().await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("generator-model")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), - ) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .and(body_string_contains("provider/scorer")) + .respond_with(SequentialReviewResponder { + calls: calls.clone(), + responses: Arc::new(vec![json!({ + "model": "provider/scorer", + "provider": "test-provider", + "choices": [{ + "finish_reason": "length", + "message": {"content": null, "reasoning": "budget exhausted"} + }], + "usage": { + "prompt_tokens": 30, + "completion_tokens": 180, + "cost": 0.000045 + } + })]), + }) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + let (profile, input) = write_atomic_attribution_inputs(dir.path()); + postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env( + "POSTIL_API_BASE", + postil_cli::config::MANAGED_OPENROUTER_API_BASE, + ) + .env("CI", "true") + .env("GITHUB_API_URL", "http://127.0.0.1:9") + .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") + .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) + .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() - .code(0); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - let finding = &env["findings"][0]; - assert_eq!(env["gate"]["failing"], false); - assert!(env.get("scorerModel").is_none()); - assert!( - env["scorerError"] - .as_str() - .unwrap() - .contains("scorer output invalid") - ); - assert!(env.get("scorerDisagreements").is_none()); - assert_eq!(env["modelUsage"][0]["model"], "generator-model"); - assert_eq!(env["modelUsage"][1]["model"], "anthropic/claude-haiku-4.5"); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); - assert_model_usage_matches_aggregate(&env); - assert_eq!(finding["confidence"], 0.92); - assert_eq!(finding["kind"], "risk"); - assert!(finding.get("generatorConfidence").is_none()); - assert!(finding.get("scorerConfidence").is_none()); - assert!(finding.get("generatorKind").is_none()); - assert!(finding.get("scorerKind").is_none()); - assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 failed after")); - assert!(stderr.contains("no fallback scorers remain")); - assert!(!stderr.contains("openai/gpt-5-mini")); + .failure() + .stderr( + predicates::str::contains( + "postil:atomic-attribution-terminal:v1:{\"category\":\"output-nonterminal-length\"", + ) + .and(predicates::str::contains("\"providerAttemptCount\":1")) + .and(predicates::str::contains( + "\"usageAccountingComplete\":true", + )), + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 1); + for request in requests { + let body: Value = serde_json::from_slice(&request.body).unwrap(); + assert_eq!(body["max_tokens"], 180); + assert!( + serde_json::to_vec(&body).unwrap().len() + <= postil_cli::attribution::MAX_PROVIDER_REQUEST_BYTES + ); + } } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn reasoning_only_scorer_length_response_is_nonterminal_invalid_output() { +async fn hidden_atomic_attribution_reports_terminal_length_response() { let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.92)]), - ) - .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("scorer-model")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "model": "provider/scorer", + "provider": "test-provider", "choices": [{ "finish_reason": "length", - "message": {"content": null, "reasoning": "internal reasoning only"} + "message": {"content": "{\"sameDefect\":true"} }], - "usage": { - "prompt_tokens": 30, - "completion_tokens": 400, - "completion_tokens_details": {"reasoning_tokens": 400}, - "cost": 0.000045 - } + "usage": {"prompt_tokens": 30, "completion_tokens": 180, "cost": 0.000045} }))) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + let (profile, input) = write_atomic_attribution_inputs(dir.path()); + postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "scorer-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env( + "POSTIL_API_BASE", + postil_cli::config::MANAGED_OPENROUTER_API_BASE, + ) + .env("CI", "true") + .env("GITHUB_API_URL", "http://127.0.0.1:9") + .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") + .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) + .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() - .code(0); + .failure() + .stderr(predicates::str::contains( + "postil:atomic-attribution-terminal:v1:{\"category\":\"output-nonterminal-length\"", + )); + assert_eq!(server.received_requests().await.unwrap().len(), 1); +} - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["path"], "src/auth.rs"); - let scorer_error = envelope["scorerError"].as_str().unwrap(); - assert!( - scorer_error.contains("model response was nonterminal (length)"), - "unexpected scorer error: {scorer_error}" - ); - assert!( - envelope["modelIncidents"] - .as_array() - .unwrap() - .iter() - .all( - |incident| incident["phase"] == "scorer" && incident["category"] == "invalidOutput" - ) - ); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(!stderr.contains("returned empty content")); - let requests = server.received_requests().await.unwrap(); - let scorer_max_tokens = requests - .iter() - .filter_map(|request| { - let body = request.body_json::().ok()?; - (body["model"] == "scorer-model" - && request_system_contains(request, "independent second-model scorer")) - .then(|| body["max_tokens"].as_u64().unwrap()) - }) - .collect::>(); - assert_eq!(scorer_max_tokens, vec![400]); +#[cfg(feature = "qualification-candidate")] +#[tokio::test] +async fn hidden_atomic_attribution_reports_terminal_provider_http_status() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "0")) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let (profile, input) = write_atomic_attribution_inputs(dir.path()); + postil() + .current_dir(dir.path()) + .env( + "POSTIL_API_BASE", + postil_cli::config::MANAGED_OPENROUTER_API_BASE, + ) + .env("CI", "true") + .env("GITHUB_API_URL", "http://127.0.0.1:9") + .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") + .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) + .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) + .assert() + .failure() + .stderr(predicates::str::contains( + "postil:atomic-attribution-terminal:v1:{\"category\":\"provider-http-429\"", + )); + assert_eq!(server.received_requests().await.unwrap().len(), 3); } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn scorer_provider_error_cannot_inject_stderr_lines() { +async fn hidden_atomic_attribution_rejects_oversized_repair_before_second_provider_call() { let server = MockServer::start().await; - mock_review_model( - &server, - "generator-model", - json!([finding_at(41, "warn", 0.92)]), - ) - .await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("anthropic/claude-haiku-4.5")) - .respond_with(ResponseTemplate::new(400).set_body_string("bad\n[stderr] forged\u{1b}[31m")) + .and(body_string_contains("provider/scorer")) + .respond_with(SequentialReviewResponder { + calls: calls.clone(), + responses: Arc::new(vec![attribution_text(&format!( + "{{\"sameDefect\":\"yes\",\"reason\":\"{}\"}}", + "x".repeat(4_000), + ))]), + }) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + let profile = dir.path().join("candidate.json"); + std::fs::write( + &profile, + json!({ + "benchmarkProviderIdentity": postil_cli::config::MANAGED_OPENROUTER_PROVIDER_IDENTITY, + "upstreamProviderIdentity": "test-provider", + "apiBase": postil_cli::config::MANAGED_OPENROUTER_API_BASE, + "apiFormat": "openai-compatible", + "generatorChain": ["openai/gpt-5-mini"], + "consensus": 1, + "scorerChain": ["provider/scorer"], + "modelPriceBounds": [ + {"model": "openai/gpt-5-mini", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000}, + {"model": "provider/scorer", "inputMicrosPerMillionTokens": 435000, "outputMicrosPerMillionTokens": 870000} + ] + }) + .to_string(), + ) + .unwrap(); + let input = dir.path().join("attribution.json"); + std::fs::write( + &input, + json!({ + "model": "provider/scorer", + "expectedProvider": "test-provider", + "target": { + "path": "src/payments.ts", "startLine": 41, "endLine": 41, + "contract": "A retry posts a second debit because the idempotency guard is bypassed." + }, + "candidate": { + "path": "src/payments.ts", "line": 41, "endLine": 41, + "severity": "error", "kind": "risk", + "title": "Retry duplicates the debit", + "body": "The retry skips idempotency and charges the payment again." + } + }) + .to_string(), + ) + .unwrap(); + postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "generator-model") - .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env( + "POSTIL_API_BASE", + postil_cli::config::MANAGED_OPENROUTER_API_BASE, + ) + .env("CI", "true") + .env("GITHUB_API_URL", "http://127.0.0.1:9") + .env("POSTIL_BENCH_REQUIRE_HOSTED_PROVIDER_PRIVACY", "1") + .env("POSTIL_QUALIFICATION_CANDIDATE_PROFILE", &profile) + .env("POSTIL_QUALIFICATION_CAPTURE_API_BASE", server.uri()) + .env("POSTIL_ALLOW_PRIVATE_API_BASE", "1") + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() - .code(0); - - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(!stderr.contains("\n[stderr] forged")); - assert!(!stderr.contains('\u{1b}')); - assert!(!stderr.contains("forged")); - assert!(!stderr.contains("bad")); - assert!(stderr.contains("status=400")); - assert!(stderr.contains("postil: scorer failed open after all scorer models failed")); + .failure() + .stderr( + predicates::str::contains("model provider request failed").and( + predicates::str::contains( + "postil:atomic-attribution-terminal:v1:{\"category\":\"provider-request-too-large\"", + ), + ), + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(server.received_requests().await.unwrap().len(), 1); } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn local_review_prints_yaml_output() { +async fn hidden_atomic_attribution_rejects_off_region_without_provider_call() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + let input = dir.path().join("attribution.json"); + std::fs::write(&input, json!({ + "model": "provider/scorer", + "expectedProvider": "test-provider", + "target": {"path": "src/payments.ts", "startLine": 41, "endLine": 41, "contract": "A retry posts a second debit."}, + "candidate": {"path": "src/payments.ts", "line": 42, "endLine": 42, "severity": "error", "kind": "risk", "title": "Other issue", "body": "A nearby line has another defect."} + }).to_string()).unwrap(); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "yaml"]) + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() - .success(); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let env: postil_cli::envelope::Envelope = yaml_serde::from_str(&stdout).unwrap(); - assert!(env.silent); - assert!(!env.gate.failing); + .failure() + .stderr(predicates::str::contains( + "candidate anchor inside the exact authored region", + )); + assert!(server.received_requests().await.unwrap().is_empty()); } -#[tokio::test] -async fn local_review_writes_csv_output_file_with_multiple_escaped_findings() { - let server = MockServer::start().await; - mock_review( - &server, - json!([ - finding_with_text( - 41, - "warn", - 0.88, - "Comma, quote \"and\" newline in title", - "First body has a comma, a \"quote\", and a newline\nsecond line." - ), - finding_with_text( - 42, - "info", - 0.77, - "Second finding", - "Body without CSV punctuation." - ) - ]), - ) - .await; +#[cfg(all(feature = "qualification-candidate", unix))] +#[test] +fn hidden_atomic_attribution_does_not_follow_input_symlinks() { + use std::os::unix::fs::symlink; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let output = dir.path().join("review.csv"); - let out = postil() + let target = dir.path().join("target.json"); + let link = dir.path().join("attribution.json"); + std::fs::write(&target, "{}").unwrap(); + symlink(&target, &link).unwrap(); + postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "csv", "--output-file"]) - .arg(&output) + .args(["atomic-attribution", "--input"]) + .arg(&link) .assert() - .success(); + .failure() + .stderr(predicates::str::contains( + "open atomic attribution input without following links", + )); +} - assert!(out.get_output().stdout.is_empty()); - let csv = std::fs::read_to_string(output).unwrap(); - let rows = parse_csv_rows(&csv); - assert_eq!(rows.len(), 2); - assert_eq!(rows[0]["version"], "1"); - assert_eq!(rows[0]["silent"], "false"); - assert_eq!( - rows[0]["summary"], - "Comma, quote \"and\" newline in title. Second finding." - ); - assert_eq!(rows[0]["path"], "src/auth.rs"); - assert_eq!(rows[0]["line"], "41"); - assert_eq!(rows[0]["endLine"], ""); - assert_eq!(rows[0]["severity"], "warn"); - assert_eq!(rows[0]["kind"], "risk"); - assert_eq!(rows[0]["confidence"], "0.88"); - assert_eq!(rows[0]["title"], "Comma, quote \"and\" newline in title"); - assert_eq!( - rows[0]["body"], - "First body has a comma, a \"quote\", and a newline\nsecond line." - ); - assert_eq!(rows[0]["gateFailOn"], "error"); - assert_eq!(rows[0]["gateFailing"], "false"); - assert_eq!(rows[0]["promptTokens"], "130"); - assert_eq!(rows[0]["completionTokens"], "60"); +#[cfg(feature = "qualification-candidate")] +#[test] +fn hidden_atomic_attribution_rejects_non_regular_input() { + let dir = tempfile::tempdir().unwrap(); + let input = dir.path().join("input-directory"); + std::fs::create_dir(&input).unwrap(); + postil() + .current_dir(dir.path()) + .args(["atomic-attribution", "--input"]) + .arg(&input) + .assert() + .failure() + .stderr(predicates::str::contains( + "atomic attribution input must be a regular file", + )); +} - assert_eq!(rows[1]["path"], "src/auth.rs"); - assert_eq!(rows[1]["line"], "42"); - assert_eq!(rows[1]["severity"], "info"); - assert_eq!(rows[1]["title"], "Second finding"); - assert_eq!(rows[1]["body"], "Body without CSV punctuation."); +#[cfg(feature = "qualification-candidate")] +#[test] +fn hidden_atomic_attribution_rejects_oversized_input_before_parsing() { + let dir = tempfile::tempdir().unwrap(); + let input = dir.path().join("attribution.json"); + std::fs::write(&input, vec![b' '; 4 * 1024 + 1]).unwrap(); + postil() + .current_dir(dir.path()) + .args(["atomic-attribution", "--input"]) + .arg(&input) + .assert() + .failure() + .stderr(predicates::str::contains( + "atomic attribution input exceeds 4096 bytes", + )); } +#[cfg(feature = "qualification-candidate")] #[tokio::test] -async fn local_review_writes_output_file_in_selected_format() { +async fn hidden_atomic_attribution_rejects_provider_substitution() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; - + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "model": "provider/scorer", + "provider": "substituted-provider", + "choices": [{"finish_reason": "stop", "message": {"content": "{\"sameDefect\":true,\"reason\":\"Same defect.\"}"}}], + "usage": {"prompt_tokens": 30, "completion_tokens": 10, "cost": 0.000045} + }))) + .mount(&server) + .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let output = dir.path().join("review.json"); - let out = postil() + let input = dir.path().join("attribution.json"); + std::fs::write( + &input, + json!({ + "model": "provider/scorer", + "expectedProvider": "test-provider", + "target": {"path": "src/payments.ts", "startLine": 41, "endLine": 41, "contract": "A retry posts a second debit."}, + "candidate": {"path": "src/payments.ts", "line": 41, "endLine": 41, "severity": "error", "kind": "risk", "title": "Duplicate debit", "body": "The retry posts another debit."} + }) + .to_string(), + ) + .unwrap(); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json", "--output-file"]) - .arg(&output) + .env("REVIEW_SCORER_MODEL", "provider/scorer") + .args(["atomic-attribution", "--input"]) + .arg(&input) .assert() - .success(); - - assert!(out.get_output().stdout.is_empty()); - let env: Value = serde_json::from_str(&std::fs::read_to_string(output).unwrap()).unwrap(); - assert_eq!(env["silent"], true); - assert_eq!(env["gate"]["failing"], false); + .failure() + .stderr( + predicates::str::contains("atomic attribution response identity does not match").and( + predicates::str::contains( + "postil:atomic-attribution-terminal:v1:{\"category\":\"response-identity-mismatch\"", + ), + ), + ); + assert_eq!(server.received_requests().await.unwrap().len(), 3); } #[tokio::test] -async fn deprecated_output_json_alias_prints_json_with_warning() { +async fn same_model_generator_and_scorer_emit_separate_balanced_usage_rows() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("same-model")) + .and(body_string_contains( + "Postil's independent second-model scorer", + )) + .respond_with( + ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ + "confidence": 0.8, + "kind": "risk", + "reason": "The changed line contains the reported unsafe data flow." + }]))), + ) + .with_priority(1) + .expect(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("same-model")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), + ) + .with_priority(2) + .expect(1) + .mount(&server) + .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "same-model") + .env("REVIEW_SCORER_MODEL", "same-model") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .success(); + .code(0); - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(env["silent"], true); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["usage"]["promptTokens"], 160); + assert_eq!(envelope["usage"]["completionTokens"], 70); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); assert!( - stderr.contains("warning: --output-json is deprecated; use --output json instead"), - "expected deprecation warning, got: {stderr}" + envelope["modelUsage"] + .as_array() + .unwrap() + .iter() + .all(|entry| entry["model"] == "same-model") ); -} - -#[test] -fn review_help_documents_machine_output_flags() { - let out = Command::cargo_bin("postil") - .unwrap() - .args(["review", "--help"]) - .assert() - .success(); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - assert!(stdout.contains("--output ")); - assert!(stdout.contains("--output-file ")); - assert!(stdout.contains("--output-json")); - assert!(stdout.contains("Deprecated in v0.2.1: use --output json")); -} - -#[test] -fn review_rejects_unknown_output_format() { - Command::cargo_bin("postil") - .unwrap() - .args(["review", "--output", "xml"]) - .assert() - .failure() - .stderr(predicates::str::contains( - "possible values: json, yaml, csv", - )); + assert_eq!(envelope["modelUsage"][0]["role"], "reviewGenerator"); + assert_eq!(envelope["modelUsage"][1]["role"], "findingScorer"); + assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); + assert_eq!(envelope["modelUsage"][0]["callOrdinal"], 1); + assert_eq!(envelope["modelUsage"][1]["callOrdinal"], 2); + assert_eq!(envelope["modelUsage"][2]["callOrdinal"], 3); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn full_remote_review_uses_an_immutable_merge_base_snapshot() { +async fn scorer_confidence_below_minimum_is_suppressed_and_nonblocking() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; + let finding = json!({ + "path": "src/orders.rs", + "line": 21, + "severity": "error", + "kind": "risk", + "confidence": 0.92, + "title": "Order update may race without a row lock", + "body": "Verify load_order_for_update issues FOR UPDATE before update_order writes the row.", + "evidence": "let row = load_order_for_update(id);" + }); Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .and(body_string_contains("generator-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([finding])))) .mount(&server) .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - }))) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .respond_with( + ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ + "confidence": 0.1, + "kind": "risk", + "reason": "Independent evidence does not support the generator claim." + }]))), + ) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + std::fs::write(dir.path().join(".postil.yaml"), "minConfidence: 0.6\n").unwrap(); + let diff = dir.path().join("change.diff"); + let diff_text = concat!( + "diff --git a/src/orders.rs b/src/orders.rs\n", + "--- a/src/orders.rs\n", + "+++ b/src/orders.rs\n", + "@@ -20,2 +20,3 @@ fn update_order_record() {\n", + " context line\n", + "+let row = load_order_for_update(id);\n", + " update_order(row);\n", + "diff --git a/tests/orders.rs b/tests/orders.rs\n", + "--- a/tests/orders.rs\n", + "+++ b/tests/orders.rs\n", + "@@ -60,2 +60,3 @@ fn load_order_locks_the_row() {\n", + " let query = load_order_for_update_query(7);\n", + "+assert!(query.contains(\"FOR UPDATE\"));\n", + " assert!(query.contains(\"WHERE id = $1\"));\n", + ); + let parsed = postil_cli::diff::parse(diff_text); + assert!(parsed.complete, "{parsed:#?}\n{diff_text}"); + std::fs::write(&diff, diff_text).unwrap(); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--repo", - "acme/api", - "--pr", - "7", - "--no-post", - "--output-json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() .code(0); - let requests = server.received_requests().await.unwrap(); - assert!(requests.iter().any(|request| { - request.url.path() == "/repos/acme/api/compare/bbbbbbbbbbbb...aaaaaaaaaaaa" - })); -} + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"].as_array().unwrap().len(), 0); + assert_eq!(envelope["silent"], true); + assert_eq!(envelope["gate"]["failing"], false); + assert_eq!(envelope["counts"]["error"], 0); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!(envelope["suppressedFindings"].as_array().unwrap().len(), 1); + assert_eq!( + envelope["suppressedFindings"][0]["reason"], + "belowConfidence" + ); + assert_eq!( + envelope["suppressedFindings"][0]["finding"]["path"], + "src/orders.rs" + ); + assert_eq!(envelope["scorerModel"], "anthropic/claude-haiku-4.5"); -#[tokio::test] -async fn insufficient_shared_context_reports_its_cause_without_provider_contact() { - let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "x".repeat(40_000), - "body": "", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) + let requests = server.received_requests().await.unwrap(); + let scorer_request: Value = requests + .iter() + .map(|request| request.body_json::().unwrap()) + .find(|body| { + body["model"] == "anthropic/claude-haiku-4.5" + && body["messages"][0]["content"] + .as_str() + .is_some_and(|system| { + system.contains("Postil's independent second-model scorer") + }) + }) + .unwrap(); + let scorer_user = scorer_request["messages"][1]["content"].as_str().unwrap(); + assert!(scorer_user.contains("relatedEvidence")); + assert!(scorer_user.contains("### tests/orders.rs")); + assert!(scorer_user.contains("FOR UPDATE")); +} + +#[tokio::test] +async fn malformed_scorer_reason_gets_one_same_model_schema_repair() { + let server = MockServer::start().await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.92)]), + ) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .and(body_string_contains("failed schema validation")) + .respond_with( + ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ + "confidence": 0.75, + "kind": "risk", + "reason": "This is a concrete defect." + }]))), + ) + .with_priority(1) + .expect(1) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(500)) - .expect(0) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .respond_with( + ResponseTemplate::new(200).set_body_json(scorer_content(json!([{ + "confidence": 0.75, + "kind": "risk", + "reason": "This reason is incomplete" + }]))), + ) + .with_priority(2) + .expect(1) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .env("REVIEW_MODEL", "example/unknown-context") - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--repo", - "acme/api", - "--pr", - "7", - "--no-post", - "--output", - "json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); + .code(0); let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let body = envelope["findings"][0]["body"].as_str().unwrap(); - assert!(body.contains("serialized shared review context")); - assert!(body.contains("conservative context limit")); - assert!(!body.contains("bounded review budget")); let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("review context budget is insufficient")); - let requests = server.received_requests().await.unwrap(); - assert!( - requests - .iter() - .all(|request| request.url.path() != "/chat/completions") - ); + assert_eq!(envelope["scorerModel"], "anthropic/claude-haiku-4.5"); + assert_eq!(envelope["findings"][0]["scorerKind"], "risk"); + assert_eq!(envelope["modelIncidents"][0]["phase"], "scorer"); + assert_eq!(envelope["modelIncidents"][0]["category"], "invalidOutput"); + assert_eq!(envelope["modelIncidents"][0]["recovered"], true); + assert_eq!(envelope["modelIncidents"][0]["recovery"], "repair"); + assert_eq!(envelope["usage"]["promptTokens"], 190); + assert_eq!(envelope["usage"]["completionTokens"], 80); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 4); + assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); + assert_eq!(envelope["modelUsage"][2]["phase"], "initial"); + assert_eq!(envelope["modelUsage"][3]["role"], "findingScorer"); + assert_eq!(envelope["modelUsage"][3]["phase"], "schemaRepair"); + assert_eq!(envelope["modelUsage"][3]["callOrdinal"], 4); + assert_model_usage_matches_aggregate(&envelope); + assert!(stderr.contains("requesting one schema repair")); } #[tokio::test] -async fn remote_dependabot_description_uses_bounded_provider_context() { - let server = MockServer::start().await; - let body = format!( - "Bumps example/action from 1 to 2.\n\nRelease notes\n\n{}\nDEPENDABOT_BODY_TAIL", - "x".repeat(128_000) - ); - Mock::given(method("GET")) - .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "bbbbbbbb"}, - "files": [] - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/files")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "filename": ".github/workflows/dependabot.yml", - "status": "modified", - "changes": 1 - }]))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path( - "/repos/acme/api/contents/.github/workflows/dependabot.yml", - )) - .respond_with(|request: &Request| { - let base = request - .url - .query_pairs() - .any(|(name, value)| name == "ref" && value.starts_with('b')); - let source = if base { - "uses: example/action@v1\n" - } else { - "uses: example/action@v2\n" - }; - ResponseTemplate::new(200).set_body_string(source) - }) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Bump example/action from 1 to 2", - "body": body, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) +async fn generic_provider_repairs_each_malformed_ordered_scorer_shape() { + let valid = json!([{ + "confidence": 0.75, + "kind": "risk", + "reason": "This is a concrete defect." + }]); + let byte_overflow = json!([{ + "confidence": 0.75, + "kind": "risk", + "reason": format!("{}。", "界".repeat(80)) + }]) + .to_string(); + let cases = [ + ( + "unknown-field", + r#"[{"index":0,"confidence":0.75,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), + ), + ( + "negative-confidence", + r#"[{"confidence":-1,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), + ), + ( + "high-confidence", + r#"[{"confidence":5,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), + ), + ( + "raw-nan", + r#"[{"confidence":NaN,"kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), + ), + ( + "string-nan", + r#"[{"confidence":"NaN","kind":"risk","reason":"This is a concrete defect."}]"#.to_string(), + ), + ("missing-entry", "[]".to_string()), + ( + "duplicate-entry", + r#"[{"confidence":0.75,"kind":"risk","reason":"This is a concrete defect."},{"confidence":0.75,"kind":"risk","reason":"This repeats the same input."}]"#.to_string(), + ), + ( + "edge-whitespace", + r#"[{"confidence":0.75,"kind":"risk","reason":" Leading whitespace is invalid."}]"#.to_string(), + ), + ( + "control-character", + r#"[{"confidence":0.75,"kind":"risk","reason":"A control\u0000character is invalid."}]"#.to_string(), + ), + ( + "missing-punctuation", + r#"[{"confidence":0.75,"kind":"risk","reason":"This reason is incomplete"}]"#.to_string(), + ), + ("byte-overflow", byte_overflow), + ]; + + for (label, malformed) in cases { + let server = MockServer::start().await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.92)]), + ) .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("scorer-model")) + .and(body_string_contains("failed schema validation")) + .respond_with(ResponseTemplate::new(200).set_body_json(scorer_content(valid.clone()))) + .with_priority(1) + .expect(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("scorer-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(scorer_text(&malformed))) + .with_priority(2) + .expect(1) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "scorer-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .code(0); + + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["scorerModel"], "scorer-model", "case {label}"); + assert_eq!( + envelope["findings"][0]["scorerConfidence"], 0.75, + "case {label}" + ); + assert_eq!( + envelope["modelIncidents"][0]["category"], "invalidOutput", + "case {label}" + ); + assert_eq!( + envelope["modelIncidents"][0]["recovery"], "repair", + "case {label}" + ); + assert_eq!( + envelope["modelUsage"].as_array().unwrap().len(), + 4, + "case {label}" + ); + } +} + +#[tokio::test] +async fn slow_explicit_scorer_times_out_without_unqualified_fallback() { + let server = MockServer::start().await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.92)]), + ) + .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(1) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(scorer_content(json!([]))), + ) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .env("REVIEW_MODEL", "z-ai/glm-5.2") - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--repo", - "acme/api", - "--pr", - "7", - "--no-post", - "--output", - "json", - ]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .success(); + .code(0); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert!(envelope["findings"].as_array().unwrap().is_empty()); - assert_eq!(envelope["gate"]["failing"], false); - let requests = server.received_requests().await.unwrap(); - let model_requests = requests + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert!(env.get("scorerModel").is_none()); + assert!(env["scorerError"].as_str().unwrap().contains("timed out")); + assert_eq!(env["usageAccountingComplete"], false); + assert_eq!(env["modelIncidents"][0]["phase"], "scorer"); + assert_eq!(env["modelIncidents"][0]["category"], "timeout"); + assert_eq!(env["modelIncidents"][0]["recovered"], false); + assert!(env["modelIncidents"][0].get("recovery").is_none()); + let models: Vec<_> = env["modelUsage"] + .as_array() + .unwrap() .iter() - .filter(|request| request.url.path() == "/chat/completions") - .collect::>(); - assert_eq!(model_requests.len(), 1); - let model_body = String::from_utf8_lossy(&model_requests[0].body); - let model_json: Value = serde_json::from_slice(&model_requests[0].body).unwrap(); - assert_eq!(model_json["model"], "z-ai/glm-5.2"); - assert!(model_body.contains("uses: example/action@v2")); - assert!(!model_body.contains("DEPENDABOT_BODY_TAIL")); - assert!(model_body.len() + 16_000 <= 128_000); + .map(|entry| entry["model"].as_str().unwrap()) + .collect(); + assert!(models.contains(&"generator-model")); + // The timed-out call is retained as an incomplete row with no invented + // token count. + assert!(models.contains(&"anthropic/claude-haiku-4.5")); + assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 timed out after")); + assert!(stderr.contains("no fallback scorers remain")); + assert!(!stderr.contains("openai/gpt-5-mini")); } #[tokio::test] -async fn github_large_lockfile_streams_past_legacy_response_limit_and_compacts() { +async fn scorer_kind_escalation_into_configured_blocking_kind_takes_effect() { let server = MockServer::start().await; - Mock::given(method("GET")) - .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "bbbbbbbb"}, - "files": [] - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/files")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "filename": "Cargo.lock", - "status": "modified", - "changes": 2 - }]))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/contents/Cargo.lock")) - .respond_with(GitHubLargeLockfileResponder) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "large lockfile", "body": "", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, - "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.9)]), + ) + .await; + mock_scorer_model( + &server, + "anthropic/claude-haiku-4.5", + json!([{ + "confidence": 0.88, + "kind": "guardrail", + "reason": "The finding violates the configured rule." + }]), + ) + .await; let dir = tempfile::tempdir().unwrap(); - postil() + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n blockOnKinds:\n - guardrail\n", + ) + .unwrap(); + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--repo", - "acme/api", - "--pr", - "7", - "--no-post", - "--output-json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .success(); + .code(1); - let requests = server.received_requests().await.unwrap(); - let model = requests - .iter() - .find(|request| request.url.path() == "/chat/completions") - .expect("model request"); - let body = String::from_utf8_lossy(&model.body); - assert!(body.contains("large-dependency@1.2.3")); - assert!(model.body.len() < 512 * 1024); + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(env["findings"][0]["kind"], "guardrail"); + assert_eq!(env["findings"][0]["generatorKind"], "risk"); + assert_eq!(env["findings"][0]["scorerKind"], "guardrail"); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn remote_setup_time_counts_against_total_llm_budget() { +async fn scorer_kind_deescalation_from_blocking_kind_is_ignored() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - let forge_delay = std::time::Duration::from_millis(1200); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(forge_delay) - .set_body_string(DIFF), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(forge_delay) - .set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - })), - ) - .mount(&server) - .await; + mock_review_model( + &server, + "generator-model", + json!([finding_at_with_kind(41, "warn", "guardrail", 0.9)]), + ) + .await; + mock_scorer_model( + &server, + "anthropic/claude-haiku-4.5", + json!([{ + "confidence": 0.88, + "kind": "risk", + "reason": "The finding is not a guardrail violation." + }]), + ) + .await; let dir = tempfile::tempdir().unwrap(); + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n blockOnKinds:\n - guardrail\n", + ) + .unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "1") - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--repo", - "acme/api", - "--pr", - "7", - "--no-post", - "--output-json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - let reqs = server.received_requests().await.unwrap(); - let model_calls = reqs - .iter() - .filter(|r| r.url.path() == "/chat/completions") - .count(); - assert_eq!(model_calls, 0); + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(env["findings"][0]["kind"], "guardrail"); + assert_eq!(env["findings"][0]["generatorKind"], "guardrail"); + assert_eq!(env["findings"][0]["scorerKind"], "risk"); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn sarif_is_written_for_local_review() { +async fn large_confidence_disagreement_escalates_to_uncertainty_with_default_gate() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), - ) - .mount(&server) - .await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 1.0)]), + ) + .await; + mock_scorer_model( + &server, + "anthropic/claude-haiku-4.5", + json!([{ + "confidence": 0.6, + "kind": "risk", + "reason": "The finding has weak supporting evidence." + }]), + ) + .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); - let sarif_path = dir.path().join("out.sarif"); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--sarif") - .arg(&sarif_path) + .args(["--output", "json"]) .assert() - .code(1); - let sarif: Value = - serde_json::from_str(&std::fs::read_to_string(&sarif_path).unwrap()).unwrap(); - assert_eq!(sarif["version"], "2.1.0"); - let result = &sarif["runs"][0]["results"][0]; - assert_eq!(result["ruleId"], "postil/risk"); - assert_eq!(result["level"], "error"); + .success(); + + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(env["findings"][0]["confidence"], 0.6); + assert_eq!(env["findings"][0]["kind"], "uncertainty"); + assert_eq!(env["findings"][0]["generatorKind"], "risk"); + assert_eq!(env["findings"][0]["scorerKind"], "risk"); + assert_eq!(env["scorerDisagreements"], 1); +} + +fn uncertainty_finding(body: &str) -> Value { + uncertainty_finding_with_severity(body, "warn") +} + +fn uncertainty_finding_with_severity(body: &str, severity: &str) -> Value { + json!({ + "path": "src/auth.rs", + "line": 41, + "severity": severity, + "kind": "uncertainty", + "confidence": 0.9, + "title": "Resolve the caller contract", + "body": body, + "evidence": "let token = format!(\"{}\", user_input);" + }) +} + +fn concise_finding(body: &str) -> Value { + json!({ + "path": "src/auth.rs", + "line": 41, + "severity": "warn", + "kind": "risk", + "confidence": 0.9, + "title": "Retry bypasses the idempotency guard", + "body": body, + "evidence": "let token = format!(\"{}\", user_input);" + }) +} + +fn enable_uncertainty_resolution(directory: &std::path::Path) { + std::fs::write( + directory.join(".postil.yaml"), + "review:\n uncertaintyResolution: true\n", + ) + .unwrap(); +} + +fn commit_uncertainty_fixture(directory: &std::path::Path) { + let run = |args: &[&str]| { + let output = std::process::Command::new("git") + .arg("-C") + .arg(directory) + .args(args) + .output() + .unwrap(); + assert!( + output.status.success(), + "git {} failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8(output.stdout).unwrap().trim().to_string() + }; + run(&["init", "--quiet"]); + run(&["add", "-A"]); + let tree = run(&["write-tree"]); + let output = std::process::Command::new("git") + .arg("-C") + .arg(directory) + .args(["commit-tree", &tree, "-m", "fixture"]) + .env("GIT_AUTHOR_NAME", "Fixture") + .env("GIT_AUTHOR_EMAIL", "fixture@example.invalid") + .env("GIT_COMMITTER_NAME", "Fixture") + .env("GIT_COMMITTER_EMAIL", "fixture@example.invalid") + .output() + .unwrap(); + assert!(output.status.success()); + let commit = String::from_utf8(output.stdout).unwrap().trim().to_string(); + run(&["update-ref", "HEAD", &commit]); +} + +fn stage_uncertainty_review(directory: &std::path::Path) { + std::fs::create_dir_all(directory.join("src")).unwrap(); + std::fs::write( + directory.join("src/auth.rs"), + format!( + "{}let token = format!(\"{{}}\", user_input);\nexec_query(&token);\n", + "\n".repeat(40) + ), + ) + .unwrap(); + let output = std::process::Command::new("git") + .arg("-C") + .arg(directory) + .args(["add", "src/auth.rs"]) + .output() + .unwrap(); + assert!(output.status.success()); +} + +#[tokio::test] +async fn uncertainty_resolution_refuted_with_verbatim_evidence_is_suppressed() { + let server = MockServer::start().await; + let original_body = "`src/reference.rs` may omit verification before insertion. Preserve the verification step."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + &json!({ + "resolution": "refuted", + "revisedBody": "", + "evidence": "verification is always inserted" + }) + .to_string(), + 1, + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + enable_uncertainty_resolution(directory.path()); + std::fs::create_dir_all(directory.path().join("src")).unwrap(); + std::fs::write( + directory.path().join("src/reference.rs"), + "// verification is always inserted\n", + ) + .unwrap(); + commit_uncertainty_fixture(directory.path()); + stage_uncertainty_review(directory.path()); + let output = postil() + .current_dir(directory.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--staged", "--output", "json"]) + .assert() + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert!(envelope["findings"].as_array().unwrap().is_empty()); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!(envelope["suppressedFindings"].as_array().unwrap().len(), 1); assert_eq!( - result["locations"][0]["physicalLocation"]["artifactLocation"]["uri"], - "src/auth.rs" + envelope["suppressedFindings"][0]["finding"]["body"], + original_body ); - assert_eq!(sarif["runs"][0]["properties"]["gateFailing"], true); + assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn advisory_on_error_lets_gate_stand_aside() { +async fn uncertainty_resolution_confirmed_replaces_only_the_body() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - // Non-retryable model error so the run fails fast; it becomes an - // operational fail-closed finding rather than a propagated error. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; + let original_body = "`src/reference.rs` may omit `service_summary` before scheduling. Preserve the summary when scheduling the job."; + let revised_body = + "`src/reference.rs` omits `service_summary` when scheduling the notification job."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + &json!({ + "resolution": "confirmed", + "revisedBody": revised_body, + "evidence": "service_summary is omitted" + }) + .to_string(), + 1, + ) + .await; - let dir = tempfile::tempdir().unwrap(); + let directory = tempfile::tempdir().unwrap(); + enable_uncertainty_resolution(directory.path()); + std::fs::create_dir_all(directory.path().join("src")).unwrap(); std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n onError: advisory\n", + directory.path().join("src/reference.rs"), + "// service_summary is omitted\n", ) .unwrap(); - postil() - .current_dir(dir.path()) + commit_uncertainty_fixture(directory.path()); + stage_uncertainty_review(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--staged", "--output", "json"]) .assert() - .code(0); // fail-open: an outage does not block the merge + .success(); - let reqs = server.received_requests().await.unwrap(); - let patches: Vec = reqs - .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .map(|r| r.body_json().unwrap()) - .collect(); - assert_eq!(patches.len(), 2); - // The gate stands aside (success), while the review check fails so the - // outage remains visible. - let mut conclusions: Vec<&str> = patches - .iter() - .map(|p| p["conclusion"].as_str().unwrap()) - .collect(); - conclusions.sort_unstable(); - assert_eq!(conclusions, vec!["failure", "success"]); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + let finding = &envelope["findings"][0]; + assert_eq!(finding["body"], revised_body); + assert_eq!(finding["kind"], "uncertainty"); + assert_eq!(finding["severity"], "warn"); + assert_eq!(finding["confidence"], 0.9); + assert_eq!(finding["path"], "src/auth.rs"); + assert_eq!(finding["line"], 41); + assert_eq!(envelope["counts"]["suppressed"], 0); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); + assert_eq!(envelope["modelUsage"][0]["role"], "reviewGenerator"); + assert_eq!(envelope["modelUsage"][1]["role"], "reviewGenerator"); + assert_eq!(envelope["modelUsage"][2]["role"], "findingScorer"); + assert_model_usage_matches_aggregate(&envelope); } -// Shared setup for the pre-review diff-fetch-failure tests: PR meta succeeds, -// the diff fetch (v3.diff Accept) fails, check-runs create/complete, and the -// error review comment posts. Parameterized by the .postil.yaml gate policy. -async fn diff_fetch_failure_server() -> MockServer { +#[tokio::test] +async fn uncertainty_resolution_malformed_response_preserves_original_finding() { let server = MockServer::start().await; - Mock::given(method("GET")) - .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "bbbbbbbb"}, - "files": [] - }))) - .mount(&server) - .await; - // The authoritative files fetch fails after meta already succeeded. - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/files")) - .respond_with( - ResponseTemplate::new(500) - .set_body_string("upstream down") - .set_delay(std::time::Duration::from_millis(25)), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - server + let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution(&server, "generator-model", "{not valid JSON", 2).await; + + let directory = tempfile::tempdir().unwrap(); + enable_uncertainty_resolution(directory.path()); + std::fs::create_dir_all(directory.path().join("src")).unwrap(); + std::fs::write( + directory.path().join("src/reference.rs"), + "repository evidence\n", + ) + .unwrap(); + commit_uncertainty_fixture(directory.path()); + stage_uncertainty_review(directory.path()); + let output = postil() + .current_dir(directory.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--staged", "--output", "json"]) + .assert() + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["findings"][0]["kind"], "uncertainty"); + assert_eq!(envelope["counts"]["suppressed"], 0); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 4); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn diff_fetch_failure_advisory_emits_envelope_and_exits_zero() { - // A pre-review diff fetch failure under gate.onError: advisory must not exit - // 2 with no output. The error envelope is emitted (machine output preserved) - // and the gate stands aside (exit 0). - let server = diff_fetch_failure_server().await; - let dir = tempfile::tempdir().unwrap(); +async fn uncertainty_resolution_defaults_on_and_fails_open_when_unresolved() { + let server = MockServer::start().await; + let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, + 1, + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(directory.path().join("src")).unwrap(); std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n onError: advisory\n", + directory.path().join("src/reference.rs"), + "repository evidence\n", ) .unwrap(); - let out = postil() - .current_dir(dir.path()) + commit_uncertainty_fixture(directory.path()); + stage_uncertainty_review(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--staged", "--output", "json"]) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - // The envelope survived: provider-class error, gate passing under advisory. - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - assert_eq!(env["gate"]["failing"], false); - assert_eq!(env["headSha"], "aaaaaaaaaaaa"); + .success(); - // The review check failed, while the gate stood aside with success. - let reqs = server.received_requests().await.unwrap(); - let conclusions: Vec = reqs - .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .map(|r| { - r.body_json::().unwrap()["conclusion"] - .as_str() - .unwrap() - .to_string() + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["counts"]["suppressed"], 0); +} + +#[tokio::test] +async fn uncertainty_resolution_uses_diff_when_no_referenced_files_exist() { + let server = MockServer::start().await; + let original_body = "The added call may pass the wrong value to the query executor."; + let revised_body = "The added call passes the request token directly to the query executor."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + &json!({ + "resolution": "confirmed", + "revisedBody": revised_body, + "evidence": "exec_query(&token);" }) - .collect(); - assert_eq!(conclusions, vec!["failure", "success"]); + .to_string(), + 1, + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + enable_uncertainty_resolution(directory.path()); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], revised_body); + assert_eq!(envelope["findings"][0]["severity"], "warn"); + assert_eq!(envelope["counts"]["suppressed"], 0); } #[tokio::test] -async fn diff_fetch_failure_block_emits_envelope_and_exits_one() { - // Same pre-review failure with the default (block) policy fails closed: - // exit 1 with the envelope emitted, not exit 2 with no output. - let server = diff_fetch_failure_server().await; - let dir = tempfile::tempdir().unwrap(); - let out = postil() - .current_dir(dir.path()) +async fn unresolved_error_uncertainty_is_retained_but_demoted_to_warn() { + let server = MockServer::start().await; + let original_body = "The added call may pass the wrong value to the query executor."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding_with_severity(original_body, "error")]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, + 1, + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + enable_uncertainty_resolution(directory.path()); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args(["review", "--repo", "acme/api", "--pr", "7", "--output-json"]) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - assert_eq!(env["gate"]["failing"], true); - assert!(env["durationMs"].as_u64().unwrap() >= 20); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["findings"][0]["severity"], "warn"); + assert_eq!(envelope["gate"]["failing"], false); } #[tokio::test] -async fn advisory_does_not_bypass_unusable_output() { +async fn uncertainty_resolution_explicit_off_makes_no_resolution_call() { let server = MockServer::start().await; - // Valid HTTP, garbage content twice (initial + repair): content-class - // failure. A malicious diff can induce this, so advisory must not bypass. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": "I cannot review this diff, sorry."}}] - }))) - .mount(&server) - .await; + let original_body = "`src/reference.rs` may omit required repository evidence. Restore the required value before merging."; + mock_review_model( + &server, + "generator-model", + json!([uncertainty_finding(original_body)]), + ) + .await; + mock_uncertainty_resolution( + &server, + "generator-model", + r#"{"resolution":"unresolved","revisedBody":"","evidence":""}"#, + 0, + ) + .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); + let directory = tempfile::tempdir().unwrap(); std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n onError: advisory\n", + directory.path().join(".postil.yaml"), + "review:\n uncertaintyResolution: false\n", ) .unwrap(); - let out = postil() - .current_dir(dir.path()) + std::fs::create_dir_all(directory.path().join("src")).unwrap(); + std::fs::write( + directory.path().join("src/reference.rs"), + "repository evidence\n", + ) + .unwrap(); + commit_uncertainty_fixture(directory.path()); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["gate"]["failing"], true); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["counts"]["suppressed"], 0); } #[tokio::test] -async fn advisory_does_not_bypass_missing_choices() { +async fn concise_findings_compresses_an_overlong_body_and_preserves_other_fields() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [], - "usage": {"prompt_tokens": 12, "completion_tokens": 0} - }))) - .expect(2) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - std::fs::write( - dir.path().join(".postil.yaml"), - "gate:\n onError: advisory\n", + let original_body = + "The retry bypasses the idempotency guard and can duplicate the transaction. " + .repeat(9) + .trim_end() + .to_string(); + let compressed_body = "The retry bypasses the idempotency guard and can duplicate the transaction. Restore the guard before retrying the operation."; + let original_finding = concise_finding(&original_body); + mock_review_model( + &server, + "generator-model", + json!([original_finding.clone()]), ) - .unwrap(); - let out = postil() - .current_dir(dir.path()) + .await; + mock_finding_compression( + &server, + "generator-model", + &json!({"body": compressed_body}).to_string(), + 1, + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(1); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); - assert_eq!(envelope["gate"]["failing"], true); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + let finding = &envelope["findings"][0]; + assert_eq!(finding["body"], compressed_body); + let mut actual_other_fields = finding.as_object().unwrap().clone(); + actual_other_fields.remove("body"); + assert!(actual_other_fields.remove("id").is_some()); + let mut expected_other_fields = original_finding.as_object().unwrap().clone(); + expected_other_fields.remove("body"); + assert_eq!(actual_other_fields, expected_other_fields); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn error_default_fails_closed_and_blocks() { +async fn concise_findings_malformed_response_preserves_the_original_body() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) - .mount(&server) - .await; + let original_body = + "The retry bypasses the idempotency guard and can duplicate the transaction. " + .repeat(9) + .trim_end() + .to_string(); + mock_review_model( + &server, + "generator-model", + json!([concise_finding(&original_body)]), + ) + .await; + mock_finding_compression(&server, "generator-model", "{not valid JSON", 1).await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - // No config: default gate.onError is block, so an unusable review fails closed. - let out = postil() - .current_dir(dir.path()) + let directory = tempfile::tempdir().unwrap(); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["gate"]["failing"], true); - // An HTTP-level model failure is provider-class, not unusable output. - assert_eq!(env["findings"][0]["path"], ".postil/provider"); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn clean_diff_is_silent_and_exits_zero() { +async fn concise_findings_explicit_off_makes_no_compression_call() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; + let original_body = + "The retry bypasses the idempotency guard and can duplicate the transaction. " + .repeat(9) + .trim_end() + .to_string(); + mock_review_model( + &server, + "generator-model", + json!([concise_finding(&original_body)]), + ) + .await; + mock_finding_compression(&server, "generator-model", r#"{"body":"unused"}"#, 0).await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) + let directory = tempfile::tempdir().unwrap(); + std::fs::write( + directory.path().join(".postil.yaml"), + "review:\n conciseFindings: false\n", + ) + .unwrap(); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["silent"], true); - assert_eq!(env["summary"], ""); - assert_eq!(env["gate"]["failing"], false); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn narrated_risk_without_findings_fails_closed() { - // The predecessor product's worst failure mode: risk prose in the summary, - // zero structured findings, green status. The model gets one corrective - // retry; if the contradiction persists the review fails closed. +async fn concise_findings_short_body_makes_no_compression_call_by_default() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) - .expect(2) // initial call + one semantic retry - .mount(&server) - .await; + let original_body = "The retry bypasses the idempotency guard."; + mock_review_model( + &server, + "generator-model", + json!([concise_finding(original_body)]), + ) + .await; + mock_finding_compression(&server, "generator-model", r#"{"body":"unused"}"#, 0).await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) + let directory = tempfile::tempdir().unwrap(); + let diff = write_diff(directory.path()); + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["silent"], false); - assert_eq!(env["gate"]["failing"], true); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!( - env["findings"][0]["title"], - "Model output could not be validated" - ); - assert_eq!(env["usage"]["promptTokens"], 200); - assert_eq!(env["usage"]["completionTokens"], 100); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); - assert_eq!(env["modelUsage"][0]["callOrdinal"], 1); - assert_eq!(env["modelUsage"][1]["callOrdinal"], 2); - assert_model_usage_matches_aggregate(&env); - // Internal model prose is not republished as a user-facing finding. - assert!( - !env["findings"][0]["body"] - .as_str() - .unwrap() - .contains("SQL injection risk in auth path.") - ); + .success(); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["body"], original_body); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + assert_model_usage_matches_aggregate(&envelope); } #[tokio::test] -async fn repeated_summary_only_output_falls_back_before_failing_the_review() { +async fn scorer_error_fails_open_and_preserves_generator_values() { let server = MockServer::start().await; - let descriptive = json!({ - "choices": [{"finish_reason": "stop", "message": {"content": json!({ - "summary": "Audit logging uses a dedicated sink and records secret use without values.", - "findings": [] - }).to_string()}}], - "usage": {"prompt_tokens": 100, "completion_tokens": 30} - }); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("summary-only-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(descriptive)) - .expect(2) + .and(body_string_contains("generator-model")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "warn", 0.92)]))), + ) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("qualified-fallback")) + .and(body_string_contains("anthropic/claude-haiku-4.5")) .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(1) .mount(&server) .await; @@ -8693,43 +8508,66 @@ async fn repeated_summary_only_output_falls_back_before_failing_the_review() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "summary-only-model") - .env("REVIEW_MODEL_CASCADE", "qualified-fallback") + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["silent"], true); + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + let finding = &env["findings"][0]; assert_eq!(env["gate"]["failing"], false); - assert_eq!(env["modelUsed"], "qualified-fallback"); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); + assert!(env.get("scorerModel").is_none()); assert!( - env["modelIncidents"] - .as_array() + env["scorerError"] + .as_str() .unwrap() - .iter() - .all(|incident| incident["recovered"] == true) + .contains("scorer output invalid") ); + assert!(env.get("scorerDisagreements").is_none()); + assert_eq!(env["modelUsage"][0]["model"], "generator-model"); + assert_eq!(env["modelUsage"][1]["model"], "anthropic/claude-haiku-4.5"); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); assert_model_usage_matches_aggregate(&env); + assert_eq!(finding["confidence"], 0.92); + assert_eq!(finding["kind"], "risk"); + assert!(finding.get("generatorConfidence").is_none()); + assert!(finding.get("scorerConfidence").is_none()); + assert!(finding.get("generatorKind").is_none()); + assert!(finding.get("scorerKind").is_none()); + assert!(stderr.contains("postil: scorer anthropic/claude-haiku-4.5 failed after")); + assert!(stderr.contains("no fallback scorers remain")); + assert!(!stderr.contains("openai/gpt-5-mini")); } #[tokio::test] -async fn schema_repair_contradiction_fails_without_a_third_model_call() { +async fn reasoning_only_scorer_length_response_is_nonterminal_invalid_output() { let server = MockServer::start().await; + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.92)]), + ) + .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_text("not json"))) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) - .expect(1) + .and(body_string_contains("scorer-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": [{ + "finish_reason": "length", + "message": {"content": null, "reasoning": "internal reasoning only"} + }], + "usage": { + "prompt_tokens": 30, + "completion_tokens": 400, + "completion_tokens_details": {"reasoning_tokens": 400}, + "cost": 0.000045 + } + }))) .mount(&server) .await; @@ -8738,75 +8576,58 @@ async fn schema_repair_contradiction_fails_without_a_third_model_call() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "test-model") - .env("REVIEW_MODEL_CASCADE", "test-model") + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "scorer-model") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["gate"]["failing"], true); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "schemaRepair"); - assert_model_usage_matches_aggregate(&env); -} - -#[tokio::test] -async fn narrated_risk_retry_recovers_structured_finding() { - // First response contradicts itself; the corrective retry returns a - // properly structured finding, which is used as the review result. - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), - ) - .mount(&server) - .await; + .code(0); - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") - .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], "src/auth.rs"); - assert_eq!(env["findings"][0]["line"], 41); - assert_eq!(env["counts"]["error"], 1); - assert_eq!(env["usage"]["promptTokens"], 230); - assert_eq!(env["usage"]["completionTokens"], 110); - assert_model_usage_matches_aggregate(&env); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["path"], "src/auth.rs"); + let scorer_error = envelope["scorerError"].as_str().unwrap(); + assert!( + scorer_error.contains("model response was nonterminal (length)"), + "unexpected scorer error: {scorer_error}" + ); + assert!( + envelope["modelIncidents"] + .as_array() + .unwrap() + .iter() + .all( + |incident| incident["phase"] == "scorer" && incident["category"] == "invalidOutput" + ) + ); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(!stderr.contains("returned empty content")); + let requests = server.received_requests().await.unwrap(); + let scorer_max_tokens = requests + .iter() + .filter_map(|request| { + let body = request.body_json::().ok()?; + (body["model"] == "scorer-model" + && request_system_contains(request, "independent second-model scorer")) + .then(|| body["max_tokens"].as_u64().unwrap()) + }) + .collect::>(); + assert_eq!(scorer_max_tokens, vec![400]); } #[tokio::test] -async fn low_confidence_only_finding_and_its_summary_are_suppressed_together() { +async fn scorer_provider_error_cannot_inject_stderr_lines() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([ - finding_at(41, "warn", 0.3) // grounded but below default minConfidence 0.6 - ])))) - // The model returns one (non-empty) raw finding, so the LLM-client - // semantic retry (which fires only on empty raw findings) does not run; - // the kept set empties later, in the filter. One call. - .expect(1) + mock_review_model( + &server, + "generator-model", + json!([finding_at(41, "warn", 0.92)]), + ) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("anthropic/claude-haiku-4.5")) + .respond_with(ResponseTemplate::new(400).set_body_string("bad\n[stderr] forged\u{1b}[31m")) .mount(&server) .await; @@ -8815,157 +8636,143 @@ async fn low_confidence_only_finding_and_its_summary_are_suppressed_together() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "generator-model") + .env("REVIEW_SCORER_MODEL", "anthropic/claude-haiku-4.5") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .success(); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["silent"], true); - assert_eq!(env["summary"], ""); - assert_eq!(env["gate"]["failing"], false); - assert!(env["findings"].as_array().unwrap().is_empty()); - assert_eq!(env["counts"]["suppressed"], 1); + .code(0); + + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(!stderr.contains("\n[stderr] forged")); + assert!(!stderr.contains('\u{1b}')); + assert!(!stderr.contains("forged")); + assert!(!stderr.contains("bad")); + assert!(stderr.contains("status=400")); + assert!(stderr.contains("postil: scorer failed open after all scorer models failed")); } #[tokio::test] -async fn misanchored_finding_does_not_turn_its_stale_summary_into_an_operational_error() { +async fn local_review_prints_yaml_output() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_content_with_summary( - "The terminal deny rule may not remain last.", - json!([terminal_deny_finding()]), - )), - ) - .expect(1) - .mount(&server) - .await; + mock_review(&server, json!([])).await; let dir = tempfile::tempdir().unwrap(); - let diff = write_terminal_deny_diff(dir.path()); - + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "yaml"]) .assert() .success(); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert!(envelope["findings"].as_array().unwrap().is_empty()); - assert_eq!(envelope["gate"]["failing"], false); - assert_eq!(envelope["counts"]["suppressed"], 1); - assert_eq!( - envelope["suppressedFindings"][0]["reason"], - "anchorMismatch" - ); - assert_ne!( - envelope["suppressedFindings"][0]["finding"]["path"], - ".postil/model-output" - ); + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let env: postil_cli::envelope::Envelope = yaml_serde::from_str(&stdout).unwrap(); + assert!(env.silent); + assert!(!env.gate.failing); } #[tokio::test] -async fn unstructured_summary_does_not_resurrect_a_rejected_finding() { +async fn local_review_writes_csv_output_file_with_multiple_escaped_findings() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_content_with_summary( - "SQL injection risk in an authentication query.", - json!([terminal_deny_finding()]), - )), - ) - .expect(1) - .mount(&server) - .await; + mock_review( + &server, + json!([ + finding_with_text( + 41, + "warn", + 0.88, + "Comma, quote \"and\" newline in title", + "First body has a comma, a \"quote\", and a newline\nsecond line." + ), + finding_with_text( + 42, + "info", + 0.77, + "Second finding", + "Body without CSV punctuation." + ) + ]), + ) + .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_terminal_deny_diff(dir.path()); + let diff = write_diff(dir.path()); + let output = dir.path().join("review.csv"); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "csv", "--output-file"]) + .arg(&output) .assert() .success(); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert!(envelope["findings"].as_array().unwrap().is_empty()); - assert_eq!(envelope["summary"], ""); - assert_eq!(envelope["gate"]["failing"], false); - assert_eq!(envelope["counts"]["suppressed"], 1); + assert!(out.get_output().stdout.is_empty()); + let csv = std::fs::read_to_string(output).unwrap(); + let rows = parse_csv_rows(&csv); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0]["version"], "1"); + assert_eq!(rows[0]["silent"], "false"); assert_eq!( - envelope["suppressedFindings"][0]["reason"], - "anchorMismatch" + rows[0]["summary"], + "Comma, quote \"and\" newline in title. Second finding." + ); + assert_eq!(rows[0]["path"], "src/auth.rs"); + assert_eq!(rows[0]["line"], "41"); + assert_eq!(rows[0]["endLine"], ""); + assert_eq!(rows[0]["severity"], "warn"); + assert_eq!(rows[0]["kind"], "risk"); + assert_eq!(rows[0]["confidence"], "0.88"); + assert_eq!(rows[0]["title"], "Comma, quote \"and\" newline in title"); + assert_eq!( + rows[0]["body"], + "First body has a comma, a \"quote\", and a newline\nsecond line." ); + assert_eq!(rows[0]["gateFailOn"], "error"); + assert_eq!(rows[0]["gateFailing"], "false"); + assert_eq!(rows[0]["promptTokens"], "130"); + assert_eq!(rows[0]["completionTokens"], "60"); + + assert_eq!(rows[1]["path"], "src/auth.rs"); + assert_eq!(rows[1]["line"], "42"); + assert_eq!(rows[1]["severity"], "info"); + assert_eq!(rows[1]["title"], "Second finding"); + assert_eq!(rows[1]["body"], "Body without CSV punctuation."); } #[tokio::test] -async fn mixed_kept_and_rejected_findings_derive_summary_from_the_kept_set() { +async fn local_review_writes_output_file_in_selected_format() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_content_with_summary( - "The terminal deny rule may not remain last.", - json!([finding_at(41, "warn", 0.9), terminal_deny_finding()]), - )), - ) - .expect(1) - .mount(&server) - .await; + mock_review(&server, json!([])).await; let dir = tempfile::tempdir().unwrap(); - let terminal_diff = std::fs::read_to_string(write_terminal_deny_diff(dir.path())).unwrap(); - let diff = dir.path().join("mixed.diff"); - std::fs::write(&diff, format!("{DIFF}{terminal_diff}")).unwrap(); + let diff = write_diff(dir.path()); + let output = dir.path().join("review.json"); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .args(["--output", "json", "--output-file"]) + .arg(&output) .assert() .success(); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"].as_array().unwrap().len(), 1); - assert_eq!( - envelope["findings"][0]["title"], - "Unsanitized input reaches query" - ); - assert_eq!(envelope["summary"], "Unsanitized input reaches query."); - assert!( - !envelope["summary"] - .as_str() - .unwrap() - .contains("terminal deny") - ); - assert_eq!(envelope["counts"]["suppressed"], 1); + assert!(out.get_output().stdout.is_empty()); + let env: Value = serde_json::from_str(&std::fs::read_to_string(output).unwrap()).unwrap(); + assert_eq!(env["silent"], true); + assert_eq!(env["gate"]["failing"], false); } #[tokio::test] -async fn ungrounded_output_fails_closed() { +async fn deprecated_output_json_alias_prints_json_with_warning() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(999, "error", 0.9)]))), - ) - .expect(2) - .mount(&server) - .await; + mock_review(&server, json!([])).await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); @@ -8976,740 +8783,728 @@ async fn ungrounded_output_fails_closed() { .arg(&diff) .arg("--output-json") .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["gate"]["failing"], true); - assert_eq!(env["usage"]["promptTokens"], 200); - assert_eq!(env["usage"]["completionTokens"], 100); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); - assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); - assert_eq!(env["modelIncidents"][0]["recovered"], false); - let body = env["findings"][0]["body"].as_str().unwrap(); - assert!(body.contains("validation categories: missingEvidenceAnchor=1")); - assert!(!body.contains("src/auth.rs:999")); + .success(); + + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("output validation failed categories=missingEvidenceAnchor=1")); - assert!(stderr.contains("semantic retry remained unusable categories=missingEvidenceAnchor=1")); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(env["silent"], true); + assert!( + stderr.contains("warning: --output-json is deprecated; use --output json instead"), + "expected deprecation warning, got: {stderr}" + ); +} + +#[test] +fn review_help_documents_machine_output_flags() { + let out = isolated_postil() + .args(["review", "--help"]) + .assert() + .success(); + + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + assert!(stdout.contains("--output ")); + assert!(stdout.contains("--output-file ")); + assert!(stdout.contains("--output-json")); + assert!(stdout.contains("Deprecated in v0.2.1: use --output json")); +} + +#[test] +fn review_rejects_unknown_output_format() { + isolated_postil() + .args(["review", "--output", "xml"]) + .assert() + .failure() + .stderr(predicates::str::contains( + "possible values: json, yaml, csv", + )); } #[tokio::test] -async fn all_ungrounded_output_retries_before_accepting_grounded_success() { +async fn full_remote_review_uses_an_immutable_merge_base_snapshot() { let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(SequentialReviewResponder { - calls: Arc::clone(&calls), - responses: Arc::new(vec![ - llm_content(json!([finding_at(999, "error", 0.9)])), - llm_content(json!([finding_at(41, "error", 0.9)])), - ]), - }) - .expect(2) + mount_github_complete_diff(&server, 7).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + }))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--repo", + "acme/api", + "--pr", + "7", + "--no-post", + "--output-json", + ]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + .code(0); - assert_eq!(calls.load(Ordering::SeqCst), 2); - assert_eq!(env["findings"][0]["path"], "src/auth.rs"); - assert_eq!(env["findings"][0]["line"], 41); - assert_eq!(env["counts"]["ungrounded"], 0); - assert_eq!(env["usage"]["promptTokens"], 230); - assert_eq!(env["usage"]["completionTokens"], 110); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); - assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); - assert_eq!(env["modelIncidents"][0]["recovered"], true); - assert_eq!(env["modelIncidents"][0]["recovery"], "repair"); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("returned unusable review content; requesting one semantic retry")); + let requests = server.received_requests().await.unwrap(); + assert!(requests.iter().any(|request| { + request.url.path() == "/repos/acme/api/compare/bbbbbbbbbbbb...aaaaaaaaaaaa" + })); } #[tokio::test] -async fn schema_repair_then_all_ungrounded_uses_no_second_correction() { +async fn insufficient_shared_context_reports_its_cause_without_provider_contact() { let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); + mount_github_complete_diff(&server, 7).await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "x".repeat(40_000), + "body": "", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 + }))) + .mount(&server) + .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(SequentialReviewResponder { - calls: Arc::clone(&calls), - responses: Arc::new(vec![ - llm_text("not review JSON"), - llm_content(json!([finding_at(999, "error", 0.9)])), - ]), - }) - .expect(2) + .respond_with(ResponseTemplate::new(500)) + .expect(0) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") + .env("POSTIL_DISABLE_SCORER", "1") + .env("REVIEW_MODEL", "example/unknown-context") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--repo", + "acme/api", + "--pr", + "7", + "--no-post", + "--output", + "json", + ]) .assert() .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(calls.load(Ordering::SeqCst), 2); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["usage"]["promptTokens"], 180); - assert_eq!(env["usage"]["completionTokens"], 80); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "schemaRepair"); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + let body = envelope["findings"][0]["body"].as_str().unwrap(); + assert!(body.contains("serialized shared review context")); + assert!(body.contains("conservative context limit")); + assert!(!body.contains("bounded review budget")); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("review context budget is insufficient")); + let requests = server.received_requests().await.unwrap(); + assert!( + requests + .iter() + .all(|request| request.url.path() != "/chat/completions") + ); } #[tokio::test] -async fn narration_retry_then_all_ungrounded_uses_no_second_correction() { +async fn remote_dependabot_description_uses_bounded_provider_context() { let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); + let body = format!( + "Bumps example/action from 1 to 2.\n\nRelease notes\n\n{}\nDEPENDABOT_BODY_TAIL", + "x".repeat(128_000) + ); + Mock::given(method("GET")) + .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "merge_base_commit": {"sha": "bbbbbbbb"}, + "files": [] + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/files")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ + "filename": ".github/workflows/dependabot.yml", + "status": "modified", + "changes": 1 + }]))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path( + "/repos/acme/api/contents/.github/workflows/dependabot.yml", + )) + .respond_with(|request: &Request| { + let base = request + .url + .query_pairs() + .any(|(name, value)| name == "ref" && value.starts_with('b')); + let source = if base { + "uses: example/action@v1\n" + } else { + "uses: example/action@v2\n" + }; + ResponseTemplate::new(200).set_body_string(source) + }) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "Bump example/action from 1 to 2", + "body": body, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 + }))) + .mount(&server) + .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(SequentialReviewResponder { - calls: Arc::clone(&calls), - responses: Arc::new(vec![ - llm_contradictory(), - llm_content(json!([finding_at(999, "error", 0.9)])), - ]), - }) - .expect(2) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(1) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") + .env("POSTIL_DISABLE_SCORER", "1") + .env("REVIEW_MODEL", "z-ai/glm-5.2") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--repo", + "acme/api", + "--pr", + "7", + "--no-post", + "--output", + "json", + ]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + .success(); - assert_eq!(calls.load(Ordering::SeqCst), 2); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["usage"]["promptTokens"], 200); - assert_eq!(env["usage"]["completionTokens"], 100); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(env["modelUsage"][0]["phase"], "initial"); - assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert!(envelope["findings"].as_array().unwrap().is_empty()); + assert_eq!(envelope["gate"]["failing"], false); + let requests = server.received_requests().await.unwrap(); + let model_requests = requests + .iter() + .filter(|request| request.url.path() == "/chat/completions") + .collect::>(); + assert_eq!(model_requests.len(), 1); + let model_body = String::from_utf8_lossy(&model_requests[0].body); + let model_json: Value = serde_json::from_slice(&model_requests[0].body).unwrap(); + assert_eq!(model_json["model"], "z-ai/glm-5.2"); + assert!(model_body.contains("uses: example/action@v2")); + assert!(!model_body.contains("DEPENDABOT_BODY_TAIL")); + assert!(model_body.len() + 16_000 <= 128_000); } #[tokio::test] -async fn all_ungrounded_primary_exhausts_one_retry_then_cascades_before_forge_writes() { +async fn github_large_lockfile_streams_past_legacy_response_limit_and_compacts() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("openai/gpt-5-mini")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(999, "error", 0.9)]))), - ) - .expect(2) + Mock::given(method("GET")) + .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "merge_base_commit": {"sha": "bbbbbbbb"}, + "files": [] + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/files")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ + "filename": "Cargo.lock", + "status": "modified", + "changes": 2 + }]))) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("z-ai/glm-5.2")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.9)]))), - ) - .expect(1) + Mock::given(method("GET")) + .and(path("/repos/acme/api/contents/Cargo.lock")) + .respond_with(GitHubLargeLockfileResponder) .mount(&server) .await; Mock::given(method("GET")) .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 + "title": "large lockfile", "body": "", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, + "changed_files": 1 }))) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .expect(2) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .expect(2) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .expect(1) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let out = postil() + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_DISABLE_SCORER", "1") .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "openai/gpt-5-mini") - .env("REVIEW_MODEL_CASCADE", "z-ai/glm-5.2") .args([ "review", - "--publish", "--repo", "acme/api", "--pr", "7", + "--no-post", "--output-json", ]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], "src/auth.rs"); - assert_eq!(env["modelUsed"], "z-ai/glm-5.2"); - assert_eq!(env["usage"]["promptTokens"], 330); - assert_eq!(env["usage"]["completionTokens"], 160); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); - assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); - assert_eq!(env["modelIncidents"][0]["recovered"], true); - assert_eq!(env["modelIncidents"][0]["recovery"], "fallback"); + .success(); let requests = server.received_requests().await.unwrap(); - let llm_positions = requests - .iter() - .enumerate() - .filter_map(|(index, request)| (request.url.path() == "/chat/completions").then_some(index)) - .collect::>(); - let first_forge_result_write = requests + let model = requests .iter() - .enumerate() - .find_map(|(index, request)| { - (request.method == wiremock::http::Method::PATCH - || (request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews")) - .then_some(index) - }) - .expect("forge result publication write"); - assert_eq!(llm_positions.len(), 4); - assert!( - llm_positions - .into_iter() - .all(|index| index < first_forge_result_write) - ); + .find(|request| request.url.path() == "/chat/completions") + .expect("model request"); + let body = String::from_utf8_lossy(&model.body); + assert!(body.contains("large-dependency@1.2.3")); + assert!(model.body.len() < 512 * 1024); } #[tokio::test] -async fn garbage_output_fails_closed_after_repair_attempt() { +async fn remote_setup_time_counts_against_total_llm_budget() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + let forge_delay = std::time::Duration::from_millis(1200); Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_text("I cannot review this diff, sorry.")), - ) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; - - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") - .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert_eq!(env["usage"]["promptTokens"], 160); - assert_eq!(env["usage"]["completionTokens"], 60); - // Initial call plus one schema-repair call for the explicit model. - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - let models: Vec = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect(); - assert_eq!(models, vec!["openai/gpt-5-mini", "openai/gpt-5-mini"]); -} - -#[tokio::test] -async fn cascade_falls_back_to_next_model() { - let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(500).set_body_json(json!({ - "error": {"metadata": {"error_type": "provider_unavailable"}}, - "usage": {"prompt_tokens": 12, "completion_tokens": 0} - }))) + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(forge_delay) + .set_body_string(DIFF), + ) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(forge_delay) + .set_body_json(json!({ + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + })), + ) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--output-json") + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "1") + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--repo", + "acme/api", + "--pr", + "7", + "--no-post", + "--output-json", + ]) .assert() - .code(0); + .code(1); let env: Value = serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["modelUsed"], "backup-model"); - assert_eq!(env["modelIncidents"][0]["phase"], "review"); - assert_eq!(env["modelIncidents"][0]["category"], "providerError"); - assert_eq!(env["modelIncidents"][0]["recovered"], true); - assert_eq!(env["modelIncidents"][0]["recovery"], "fallback"); - assert_eq!(env["usage"]["promptTokens"], 136); - assert_eq!(env["usage"]["completionTokens"], 50); - assert_eq!(env["modelUsage"][0]["model"], "primary-model"); - assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); - assert!( - env["modelUsage"].as_array().unwrap()[..3] - .iter() - .all(|entry| entry["model"] == "primary-model" && entry["promptTokens"] == 12) - ); - assert_model_usage_matches_aggregate(&env); - assert_eq!(env["usageAccountingComplete"], true); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("postil: attempting model: primary-model")); - assert!(stderr.contains("postil: model primary-model returned retryable HTTP 500")); - assert!(stderr.contains("retrying in ")); - assert!(stderr.contains("postil: model primary-model failed after")); - assert!(stderr.contains("falling back to next model")); - assert!(stderr.contains("postil: attempting model: backup-model")); - assert!(stderr.contains("postil: model backup-model responded in")); -} - -#[tokio::test] -async fn consensus_logs_each_model_outcome() { - let server = MockServer::start().await; - for model in ["consensus-a", "consensus-b"] { - mock_review_model(&server, model, json!([])).await; - } - - let dir = tempfile::tempdir().unwrap(); - std::fs::write(dir.path().join(".postil.yaml"), "model:\n consensus: 2\n").unwrap(); - let diff = write_diff(dir.path()); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "consensus-a") - .env("REVIEW_MODEL_CASCADE", "consensus-b") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .code(0); + assert_eq!(env["findings"][0]["path"], ".postil/provider"); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - for model in ["consensus-a", "consensus-b"] { - assert!(stderr.contains(&format!("postil: attempting consensus model: {model}"))); - assert!(stderr.contains(&format!("postil: consensus model {model} responded in"))); - } + let reqs = server.received_requests().await.unwrap(); + let model_calls = reqs + .iter() + .filter(|r| r.url.path() == "/chat/completions") + .count(); + assert_eq!(model_calls, 0); } #[tokio::test] -async fn slow_model_request_retries_same_model_then_succeeds() { +async fn sarif_is_written_for_local_review() { let server = MockServer::start().await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(llm_content(json!([]))), - ) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .and(path("/chat/completions")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), + ) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); - let out = postil() + let sarif_path = dir.path().join("out.sarif"); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) - .arg("--output-json") + .arg("--sarif") + .arg(&sarif_path) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["modelUsed"], "primary-model"); - assert_eq!(env["usageAccountingComplete"], false); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("postil: model primary-model hit a request timeout after")); - assert!(stderr.contains("timeout retry 1/1")); - assert!(stderr.contains("postil: model primary-model responded in")); - assert!(!stderr.contains("postil: attempting model: backup-model")); - - let requests = server.received_requests().await.unwrap(); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!(models, vec!["primary-model", "primary-model"]); + .code(1); + let sarif: Value = + serde_json::from_str(&std::fs::read_to_string(&sarif_path).unwrap()).unwrap(); + assert_eq!(sarif["version"], "2.1.0"); + let result = &sarif["runs"][0]["results"][0]; + assert_eq!(result["ruleId"], "postil/risk"); + assert_eq!(result["level"], "error"); + assert_eq!( + result["locations"][0]["physicalLocation"]["artifactLocation"]["uri"], + "src/auth.rs" + ); + assert_eq!(sarif["runs"][0]["properties"]["gateFailing"], true); } #[tokio::test] -async fn empty_success_response_retries_same_model_and_accumulates_usage() { +async fn advisory_on_error_lets_gate_stand_aside() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + // Non-retryable model error so the run fails fast; it becomes an + // operational fail-closed finding rather than a propagated error. Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "id": "empty-attempt", - "model": "primary-model", - "provider": "test-provider", - "choices": [], - "usage": {"prompt_tokens": 12, "completion_tokens": 0} + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 }))) - .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let out = postil() + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n onError: advisory\n", + ) + .unwrap(); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", + ]) .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "primary-model"); - assert_eq!(envelope["usage"]["promptTokens"], 112); - assert_eq!(envelope["usage"]["completionTokens"], 50); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - assert_eq!(envelope["modelUsage"][0]["promptTokens"], 12); - assert_eq!(envelope["modelUsage"][1]["promptTokens"], 100); - assert_eq!(envelope["modelUsage"][0]["attempt"], 1); - assert_eq!(envelope["modelUsage"][1]["attempt"], 2); - assert_eq!(envelope["usageAccountingComplete"], true); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("returned empty content")); - assert!(stderr.contains("phase=review")); - assert!(stderr.contains("provider=present")); - assert!(stderr.contains("usage=present")); - assert!(stderr.contains("response_id=present")); - assert!(!stderr.contains("empty-attempt")); + .code(0); // fail-open: an outage does not block the merge - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - assert!( - requests - .iter() - .all(|request| { request.body_json::().unwrap()["max_tokens"] == 8_000 }) - ); + let reqs = server.received_requests().await.unwrap(); + let patches: Vec = reqs + .iter() + .filter(|r| r.method == wiremock::http::Method::PATCH) + .map(|r| r.body_json().unwrap()) + .collect(); + assert_eq!(patches.len(), 2); + // The gate stands aside (success), while the review check fails so the + // outage remains visible. + let mut conclusions: Vec<&str> = patches + .iter() + .map(|p| p["conclusion"].as_str().unwrap()) + .collect(); + conclusions.sort_unstable(); + assert_eq!(conclusions, vec!["failure", "success"]); } -#[tokio::test] -async fn exhausted_reasoning_budget_expands_the_same_model_retry() { +// Shared setup for the pre-review diff-fetch-failure tests: PR meta succeeds, +// the diff fetch (v3.diff Accept) fails, check-runs create/complete, and the +// error review comment posts. Parameterized by the .postil.yaml gate policy. +async fn diff_fetch_failure_server() -> MockServer { let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "merge_base_commit": {"sha": "bbbbbbbb"}, + "files": [] + }))) + .mount(&server) + .await; + // The authoritative files fetch fails after meta already succeeded. + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/files")) + .respond_with( + ResponseTemplate::new(500) + .set_body_string("upstream down") + .set_delay(std::time::Duration::from_millis(25)), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + }))) + .mount(&server) + .await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(OutputBudgetResponder) - .expect(2) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) .mount(&server) .await; + server +} +#[tokio::test] +async fn diff_fetch_failure_advisory_emits_envelope_and_exits_zero() { + // A pre-review diff fetch failure under gate.onError: advisory must not exit + // 2 with no output. The error envelope is emitted (machine output preserved) + // and the gate stands aside (exit 0). + let server = diff_fetch_failure_server().await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n onError: advisory\n", + ) + .unwrap(); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", + ]) .assert() .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + // The envelope survived: provider-class error, gate passing under advisory. + assert_eq!(env["findings"][0]["path"], ".postil/provider"); + assert_eq!(env["gate"]["failing"], false); + assert_eq!(env["headSha"], "aaaaaaaaaaaa"); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "primary-model"); - assert_eq!(envelope["usage"]["promptTokens"], 30_845); - assert_eq!(envelope["usage"]["completionTokens"], 8_050); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("exhausted 8000 output tokens")); - assert!(stderr.contains("retrying the complete request with 16000 tokens")); - assert!(stderr.contains("reasoning_tokens=8000")); - - let requests = server.received_requests().await.unwrap(); - let max_tokens = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["max_tokens"] - .as_u64() + // The review check failed, while the gate stood aside with success. + let reqs = server.received_requests().await.unwrap(); + let conclusions: Vec = reqs + .iter() + .filter(|r| r.method == wiremock::http::Method::PATCH) + .map(|r| { + r.body_json::().unwrap()["conclusion"] + .as_str() .unwrap() + .to_string() }) - .collect::>(); - assert_eq!(max_tokens, vec![8_000, 16_000]); + .collect(); + assert_eq!(conclusions, vec!["failure", "success"]); } #[tokio::test] -async fn repeated_exhausted_reasoning_budget_does_not_consume_an_empty_retry() { +async fn diff_fetch_failure_block_emits_envelope_and_exits_one() { + // Same pre-review failure with the default (block) policy fails closed: + // exit 1 with the envelope emitted, not exit 2 with no output. + let server = diff_fetch_failure_server().await; + let dir = tempfile::tempdir().unwrap(); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args(["review", "--repo", "acme/api", "--pr", "7", "--output-json"]) + .assert() + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/provider"); + assert_eq!(env["gate"]["failing"], true); + assert!(env["durationMs"].as_u64().unwrap() >= 20); +} + +#[tokio::test] +async fn advisory_does_not_bypass_unusable_output() { let server = MockServer::start().await; + // Valid HTTP, garbage content twice (initial + repair): content-class + // failure. A malicious diff can induce this, so advisory must not bypass. Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{ - "finish_reason": "length", - "message": {"content": null, "reasoning": "budget exhausted"} - }], - "usage": { - "prompt_tokens": 4_194, - "completion_tokens": 8_000, - "completion_tokens_details": {"reasoning_tokens": 8_000} - } + "choices": [{"finish_reason": "stop", "message": {"content": "I cannot review this diff, sorry."}}] }))) - .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n onError: advisory\n", + ) + .unwrap(); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() .code(1); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("retrying the complete request with 16000 tokens")); - assert!(!stderr.contains("returned empty content")); - - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - let max_tokens = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["max_tokens"] - .as_u64() - .unwrap() - }) - .collect::>(); - assert_eq!(max_tokens, vec![8_000, 16_000]); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn openai_truncation_retries_the_complete_original_request() { +async fn advisory_does_not_bypass_missing_choices() { let server = MockServer::start().await; - let partial_title = "Partial output must never publish"; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(SequentialReviewResponder { - calls: Arc::new(AtomicUsize::new(0)), - responses: Arc::new(vec![ - json!({ - "choices": [{"finish_reason": "length", "message": {"content": json!({ - "summary": "Incomplete review.", - "findings": [{ - "path": "src/auth.rs", "line": 42, "severity": "error", - "kind": "risk", "confidence": 1.0, - "title": partial_title, "body": "This text is incomplete.", - "evidence": "exec_query(&token);" - }] - }).to_string()}}], - "usage": {"prompt_tokens": 100, "completion_tokens": 8000} - }), - llm_content(json!([])), - ]), - }) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": [], + "usage": {"prompt_tokens": 12, "completion_tokens": 0} + }))) .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); + std::fs::write( + dir.path().join(".postil.yaml"), + "gate:\n onError: advisory\n", + ) + .unwrap(); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .success(); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - assert!(!stdout.contains(partial_title)); - let envelope: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(envelope["silent"], true); - assert_eq!(envelope["usage"]["promptTokens"], 200); - assert_eq!(envelope["usage"]["completionTokens"], 8050); + .code(1); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); + assert_eq!(envelope["gate"]["failing"], true); assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - assert_model_usage_matches_aggregate(&envelope); - - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - let first: Value = requests[0].body_json().unwrap(); - let second: Value = requests[1].body_json().unwrap(); - assert_eq!(first["max_tokens"], 8_000); - assert_eq!(second["max_tokens"], 16_000); - assert_eq!(first["messages"], second["messages"]); - assert!( - !serde_json::to_string(&second) - .unwrap() - .contains(partial_title) - ); } #[tokio::test] -async fn repeated_openai_truncation_fails_closed_without_partial_text() { +async fn error_default_fails_closed_and_blocks() { let server = MockServer::start().await; - let partial_title = "Partial output must never publish"; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "length", "message": {"content": json!({ - "summary": "Incomplete review.", - "findings": [{ - "path": "src/auth.rs", "line": 42, "severity": "error", - "kind": "risk", "confidence": 1.0, - "title": partial_title, "body": "This text is incomplete.", - "evidence": "exec_query(&token);" - }] - }).to_string()}}], - "usage": {"prompt_tokens": 100, "completion_tokens": 8000} - }))) - .expect(2) + .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); + // No config: default gate.onError is block, so an unusable review fails closed. let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() .code(1); - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - assert!(!stdout.contains(partial_title)); - let envelope: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(envelope["gate"]["failing"], true); - assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); - assert_model_usage_matches_aggregate(&envelope); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - assert_eq!( - requests[0].body_json::().unwrap()["max_tokens"], - 8_000 - ); - assert_eq!( - requests[1].body_json::().unwrap()["max_tokens"], - 16_000 - ); - assert!(requests.iter().all(|request| { - !String::from_utf8_lossy(&request.body).contains("You repair malformed JSON") - })); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["gate"]["failing"], true); + // An HTTP-level model failure is provider-class, not unusable output. + assert_eq!(env["findings"][0]["path"], ".postil/provider"); } #[tokio::test] -async fn rate_limit_after_exhausted_output_uses_the_remaining_attempt_and_retry_after() { +async fn clean_diff_is_silent_and_exits_zero() { let server = MockServer::start().await; - let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(OutputThenRateLimitResponder { - calls: calls.clone(), - }) - .expect(3) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; @@ -9718,37 +9513,28 @@ async fn rate_limit_after_exhausted_output_uses_the_remaining_attempt_and_retry_ let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["silent"], true); - assert_eq!(calls.load(Ordering::SeqCst), 3); - assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("returned retryable HTTP 429")); - assert!(stderr.contains("retrying in 0ms")); - assert!(stderr.contains("attempt=3/3")); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["silent"], true); + assert_eq!(env["summary"], ""); + assert_eq!(env["gate"]["failing"], false); } #[tokio::test] -async fn empty_success_without_usage_marks_accounting_incomplete() { +async fn narrated_risk_without_findings_fails_closed() { + // The predecessor product's worst failure mode: risk prose in the summary, + // zero structured findings, green status. The model gets one corrective + // retry; if the contradiction persists the review fails closed. let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({"choices": []}))) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) + .expect(2) // initial call + one semantic retry .mount(&server) .await; @@ -9757,29 +9543,59 @@ async fn empty_success_without_usage_marks_accounting_incomplete() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .code(0); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["usageAccountingComplete"], false); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["silent"], false); + assert_eq!(env["gate"]["failing"], true); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!( + env["findings"][0]["title"], + "Model output could not be validated" + ); + assert_eq!(env["usage"]["promptTokens"], 200); + assert_eq!(env["usage"]["completionTokens"], 100); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); + assert_eq!(env["modelUsage"][0]["callOrdinal"], 1); + assert_eq!(env["modelUsage"][1]["callOrdinal"], 2); + assert_model_usage_matches_aggregate(&env); + // Internal model prose is not republished as a user-facing finding. + assert!( + !env["findings"][0]["body"] + .as_str() + .unwrap() + .contains("SQL injection risk in auth path.") + ); } #[tokio::test] -async fn malformed_success_without_usage_marks_accounting_incomplete_before_fallback() { +async fn repeated_summary_only_output_falls_back_before_failing_the_review() { let server = MockServer::start().await; + let descriptive = json!({ + "choices": [{"finish_reason": "stop", "message": {"content": json!({ + "summary": "Audit logging uses a dedicated sink and records secret use without values.", + "findings": [] + }).to_string()}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 30} + }); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_string("not-json")) + .and(body_string_contains("summary-only-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(descriptive)) + .expect(2) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) + .and(body_string_contains("qualified-fallback")) .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(1) .mount(&server) .await; @@ -9788,38 +9604,43 @@ async fn malformed_success_without_usage_marks_accounting_incomplete_before_fall let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") + .env("REVIEW_MODEL", "summary-only-model") + .env("REVIEW_MODEL_CASCADE", "qualified-fallback") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "backup-model"); - assert_eq!(envelope["usageAccountingComplete"], false); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("usage=missing")); - assert!(!stderr.contains("not-json")); + assert_eq!(env["silent"], true); + assert_eq!(env["gate"]["failing"], false); + assert_eq!(env["modelUsed"], "qualified-fallback"); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); + assert!( + env["modelIncidents"] + .as_array() + .unwrap() + .iter() + .all(|incident| incident["recovered"] == true) + ); + assert_model_usage_matches_aggregate(&env); } #[tokio::test] -async fn malformed_success_with_usage_records_the_attempt_exactly_once() { +async fn schema_repair_contradiction_fails_without_a_third_model_call() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": "invalid-shape", - "usage": {"prompt_tokens": 12, "completion_tokens": 3} - }))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_text("not json"))) + .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) + .expect(1) .mount(&server) .await; @@ -9828,123 +9649,75 @@ async fn malformed_success_with_usage_records_the_attempt_exactly_once() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") + .env("REVIEW_MODEL", "test-model") + .env("REVIEW_MODEL_CASCADE", "test-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "backup-model"); - assert_eq!(envelope["usage"]["promptTokens"], 112); - assert_eq!(envelope["usage"]["completionTokens"], 53); - assert_eq!(envelope["modelUsage"][0]["promptTokens"], 12); - assert_eq!(envelope["modelUsage"][0]["completionTokens"], 3); - assert_eq!(envelope["usageAccountingComplete"], true); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["gate"]["failing"], true); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "schemaRepair"); + assert_model_usage_matches_aggregate(&env); } #[tokio::test] -async fn timeout_after_exhausted_output_uses_remaining_attempt_then_preserves_cascade() { +async fn narrated_risk_retry_recovers_structured_finding() { + // First response contradicts itself; the corrective retry returns a + // properly structured finding, which is used as the review result. let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{ - "finish_reason": "length", - "message": {"content": null, "reasoning": "budget exhausted"} - }], - "usage": { - "prompt_tokens": 12, - "completion_tokens": 8_000, - "completion_tokens_details": {"reasoning_tokens": 8_000} - } - }))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_contradictory())) .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) .respond_with( ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(llm_content(json!([]))), + .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), ) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "backup-model"); - let requests = server.received_requests().await.unwrap(); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!( - models, - vec![ - "primary-model", - "primary-model", - "primary-model", - "backup-model" - ] - ); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], "src/auth.rs"); + assert_eq!(env["findings"][0]["line"], 41); + assert_eq!(env["counts"]["error"], 1); + assert_eq!(env["usage"]["promptTokens"], 230); + assert_eq!(env["usage"]["completionTokens"], 110); + assert_model_usage_matches_aggregate(&env); } #[tokio::test] -async fn empty_response_retry_http_failure_uses_the_third_primary_request_before_fallback() { +async fn low_confidence_only_finding_and_its_summary_are_suppressed_together() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [], - "usage": {"prompt_tokens": 12, "completion_tokens": 0} - }))) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(500).set_body_json(json!({ - "error": {"metadata": {"error_type": "provider_unavailable"}} - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([ + finding_at(41, "warn", 0.3) // grounded but below default minConfidence 0.6 + ])))) + // The model returns one (non-empty) raw finding, so the LLM-client + // semantic retry (which fires only on empty raw findings) does not run; + // the kept set empties later, in the filter. One call. + .expect(1) .mount(&server) .await; @@ -9953,254 +9726,201 @@ async fn empty_response_retry_http_failure_uses_the_third_primary_request_before let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "2") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .code(0); - - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["modelUsed"], "backup-model"); - let requests = server.received_requests().await.unwrap(); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!( - models, - vec![ - "primary-model", - "primary-model", - "primary-model", - "backup-model" - ] - ); + .success(); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["silent"], true); + assert_eq!(env["summary"], ""); + assert_eq!(env["gate"]["failing"], false); + assert!(env["findings"].as_array().unwrap().is_empty()); + assert_eq!(env["counts"]["suppressed"], 1); } -#[test] -fn connection_failure_after_exhausted_output_uses_the_remaining_third_request() { - use std::io::{Read, Write}; - - let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); - let address = listener.local_addr().unwrap(); - let server = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().unwrap(); - let mut request = [0_u8; 16_384]; - let _ = stream.read(&mut request).unwrap(); - let body = r#"{"choices":[{"finish_reason":"length","message":{"content":null,"reasoning":"budget exhausted"}}],"usage":{"prompt_tokens":12,"completion_tokens":8000,"completion_tokens_details":{"reasoning_tokens":8000}}}"#; - write!( - stream, - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", - body.len(), - body +#[tokio::test] +async fn misanchored_finding_does_not_turn_its_stale_summary_into_an_operational_error() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with( + ResponseTemplate::new(200).set_body_json(llm_content_with_summary( + "The terminal deny rule may not remain last.", + json!([terminal_deny_finding()]), + )), ) - .unwrap(); - stream.flush().unwrap(); - }); - - let dir = tempfile::tempdir().unwrap(); - std::fs::write( - dir.path().join(".postil.yaml"), - "model:\n name: primary-model\n cascade: []\n", - ) - .unwrap(); - let diff = write_diff(dir.path()); + .expect(1) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + let diff = write_terminal_deny_diff(dir.path()); + let out = postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", format!("http://{address}")) - .env("REVIEW_MODEL", "primary-model") - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--output", "json"]) + .arg("--output-json") .assert() - .code(1); - server.join().unwrap(); + .success(); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!( - stderr.contains("model=primary-model attempt=2/3"), - "unexpected log: {stderr}" + assert!(envelope["findings"].as_array().unwrap().is_empty()); + assert_eq!(envelope["gate"]["failing"], false); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!( + envelope["suppressedFindings"][0]["reason"], + "anchorMismatch" + ); + assert_ne!( + envelope["suppressedFindings"][0]["finding"]["path"], + ".postil/model-output" ); - assert!(stderr.contains("model=primary-model attempt=3/3")); - assert!(!stderr.contains("127.0.0.1")); } #[tokio::test] -async fn timeout_http_status_retries_same_model_then_succeeds() { +async fn unstructured_summary_does_not_resurrect_a_rejected_finding() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(408).set_body_string("request timed out")) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with( + ResponseTemplate::new(200).set_body_json(llm_content_with_summary( + "SQL injection risk in an authentication query.", + json!([terminal_deny_finding()]), + )), + ) + .expect(1) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); + let diff = write_terminal_deny_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) .arg("--output-json") .assert() - .code(0); - let envelope: Value = - serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); - assert_eq!(envelope["modelUsed"], "primary-model"); - assert_eq!(envelope["usageAccountingComplete"], false); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("returned timeout HTTP 408 Request Timeout")); - assert!(stderr.contains("timeout retry 1/1")); + .success(); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - assert!( - requests - .iter() - .all(|request| { request.body_json::().unwrap()["model"] == "primary-model" }) + assert!(envelope["findings"].as_array().unwrap().is_empty()); + assert_eq!(envelope["summary"], ""); + assert_eq!(envelope["gate"]["failing"], false); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!( + envelope["suppressedFindings"][0]["reason"], + "anchorMismatch" ); } #[tokio::test] -async fn exhausted_timeout_retry_falls_back_to_next_model() { +async fn mixed_kept_and_rejected_findings_derive_summary_from_the_kept_set() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) .respond_with( - ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(llm_content(json!([]))), + ResponseTemplate::new(200).set_body_json(llm_content_with_summary( + "The terminal deny rule may not remain last.", + json!([finding_at(41, "warn", 0.9), terminal_deny_finding()]), + )), ) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(1) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); + let terminal_diff = std::fs::read_to_string(write_terminal_deny_diff(dir.path())).unwrap(); + let diff = dir.path().join("mixed.diff"); + std::fs::write(&diff, format!("{DIFF}{terminal_diff}")).unwrap(); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") + .env("POSTIL_DISABLE_SCORER", "1") .args(["review", "--diff-file"]) .arg(&diff) .arg("--output-json") .assert() - .code(0); - let envelope: Value = - serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); - assert_eq!(envelope["modelUsed"], "backup-model"); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!(stderr.contains("model primary-model hit a request timeout after")); - assert!(stderr.contains("timeout retry 1/1")); - let attempts = envelope["modelUsage"] - .as_array() - .unwrap() - .iter() - .map(|entry| { - ( - entry["model"].as_str().unwrap(), - entry["attempt"].as_u64().unwrap(), - ) - }) - .collect::>(); - assert_eq!( - attempts, - vec![ - ("primary-model", 1), - ("primary-model", 2), - ("backup-model", 1) - ] - ); + .success(); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let requests = server.received_requests().await.unwrap(); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!(models.last().map(String::as_str), Some("backup-model")); - assert!(models.len() <= 3); + assert_eq!(envelope["findings"].as_array().unwrap().len(), 1); assert_eq!( - models - .iter() - .filter(|model| model.as_str() == "backup-model") - .count(), - 1 + envelope["findings"][0]["title"], + "Unsanitized input reaches query" ); + assert_eq!(envelope["summary"], "Unsanitized input reaches query."); assert!( - models[..models.len() - 1] - .iter() - .all(|model| model == "primary-model") + !envelope["summary"] + .as_str() + .unwrap() + .contains("terminal deny") ); + assert_eq!(envelope["counts"]["suppressed"], 1); } #[tokio::test] -async fn mixed_failures_share_the_existing_two_retry_cap() { +async fn ungrounded_output_fails_closed() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(500).set_body_string("upstream down")) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) .respond_with( ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(llm_content(json!([]))), + .set_body_json(llm_content(json!([finding_at(999, "error", 0.9)]))), ) - .up_to_n_times(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(500).set_body_string("upstream still down")) + .expect(2) .mount(&server) .await; + + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") + .assert() + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["gate"]["failing"], true); + assert_eq!(env["usage"]["promptTokens"], 200); + assert_eq!(env["usage"]["completionTokens"], 100); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); + assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); + assert_eq!(env["modelIncidents"][0]["recovered"], false); + let body = env["findings"][0]["body"].as_str().unwrap(); + assert!(body.contains("validation categories: missingEvidenceAnchor=1")); + assert!(!body.contains("src/auth.rs:999")); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("output validation failed categories=missingEvidenceAnchor=1")); + assert!(stderr.contains("semantic retry remained unusable categories=missingEvidenceAnchor=1")); +} + +#[tokio::test] +async fn all_ungrounded_output_retries_before_accepting_grounded_success() { + let server = MockServer::start().await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with(SequentialReviewResponder { + calls: Arc::clone(&calls), + responses: Arc::new(vec![ + llm_content(json!([finding_at(999, "error", 0.9)])), + llm_content(json!([finding_at(41, "error", 0.9)])), + ]), + }) + .expect(2) .mount(&server) .await; @@ -10209,57 +9929,44 @@ async fn mixed_failures_share_the_existing_two_retry_cap() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "15") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) .arg("--output-json") .assert() - .code(0); - let envelope: Value = - serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); - assert_eq!(envelope["modelUsed"], "backup-model"); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - let requests = server.received_requests().await.unwrap(); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!( - models, - vec![ - "primary-model", - "primary-model", - "primary-model", - "backup-model" - ] - ); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(env["findings"][0]["path"], "src/auth.rs"); + assert_eq!(env["findings"][0]["line"], 41); + assert_eq!(env["counts"]["ungrounded"], 0); + assert_eq!(env["usage"]["promptTokens"], 230); + assert_eq!(env["usage"]["completionTokens"], 110); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 3); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); + assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); + assert_eq!(env["modelIncidents"][0]["recovered"], true); + assert_eq!(env["modelIncidents"][0]["recovery"], "repair"); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("returned unusable review content; requesting one semantic retry")); } #[tokio::test] -async fn total_llm_timeout_caps_the_cascade() { +async fn schema_repair_then_all_ungrounded_uses_no_second_correction() { let server = MockServer::start().await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(1500)) - .set_body_json(llm_content(json!([]))), - ) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with(SequentialReviewResponder { + calls: Arc::clone(&calls), + responses: Arc::new(vec![ + llm_text("not review JSON"), + llm_content(json!([finding_at(999, "error", 0.9)])), + ]), + }) + .expect(2) .mount(&server) .await; @@ -10268,10 +9975,6 @@ async fn total_llm_timeout_caps_the_cascade() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "1") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) .arg("--output-json") @@ -10279,38 +9982,30 @@ async fn total_llm_timeout_caps_the_cascade() { .code(1); let env: Value = serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 1); - let model = requests[0].body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string(); - assert_eq!(model, "primary-model"); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["usage"]["promptTokens"], 180); + assert_eq!(env["usage"]["completionTokens"], 80); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "schemaRepair"); } #[tokio::test] -async fn shared_total_budget_can_expire_during_cascade_retry() { +async fn narration_retry_then_all_ungrounded_uses_no_second_correction() { let server = MockServer::start().await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("primary-model")) - .respond_with( - ResponseTemplate::new(400) - .set_delay(std::time::Duration::from_millis(400)) - .set_body_string("bad request"), - ) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(wiremock::matchers::body_string_contains("backup-model")) - .respond_with( - ResponseTemplate::new(200) - .set_delay(std::time::Duration::from_millis(2200)) - .set_body_json(llm_content(json!([]))), - ) + .respond_with(SequentialReviewResponder { + calls: Arc::clone(&calls), + responses: Arc::new(vec![ + llm_contradictory(), + llm_content(json!([finding_at(999, "error", 0.9)])), + ]), + }) + .expect(2) .mount(&server) .await; @@ -10319,10 +10014,6 @@ async fn shared_total_budget_can_expire_during_cascade_retry() { let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") - .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "2") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) .arg("--output-json") @@ -10330,69 +10021,64 @@ async fn shared_total_budget_can_expire_during_cascade_retry() { .code(1); let env: Value = serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - let requests = server.received_requests().await.unwrap(); - assert_eq!(requests.len(), 2); - let models = requests - .iter() - .map(|request| { - request.body_json::().unwrap()["model"] - .as_str() - .unwrap() - .to_string() - }) - .collect::>(); - assert_eq!(models, vec!["primary-model", "backup-model"]); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["usage"]["promptTokens"], 200); + assert_eq!(env["usage"]["completionTokens"], 100); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(env["modelUsage"][0]["phase"], "initial"); + assert_eq!(env["modelUsage"][1]["phase"], "semanticRetry"); } #[tokio::test] -async fn forge_post_failure_on_success_path_keeps_gate_derived_exit_code() { - // A completed review (gate failing on a real finding) whose forge comment - // post then fails (rate limit, transient 5xx) must not discard the - // already-computed envelope or exit 2: the exit code stays derived from - // the gate, the envelope is still emitted, and both check runs still - // complete. Only the review comment itself is lost, and only a stderr - // warning marks that. +async fn all_ungrounded_primary_exhausts_one_retry_then_cascades_before_forge_writes() { let server = MockServer::start().await; mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) + .and(body_string_contains("openai/gpt-5-mini")) .respond_with( ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), + .set_body_json(llm_content(json!([finding_at(999, "error", 0.9)]))), ) + .expect(2) .mount(&server) .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("z-ai/glm-5.2")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.9)]))), + ) + .expect(1) .mount(&server) .await; Mock::given(method("GET")) .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + "title": "t", "body": "b", "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; Mock::given(method("POST")) .and(path("/repos/acme/api/check-runs")) .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .expect(2) .mount(&server) .await; Mock::given(method("PATCH")) .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .expect(2) .mount(&server) .await; - // The review comment post itself fails, which is the fault this test is pinning. Mock::given(method("POST")) .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(500).set_body_string("secondary rate limit")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .expect(1) .mount(&server) .await; @@ -10402,6 +10088,8 @@ async fn forge_post_failure_on_success_path_keeps_gate_derived_exit_code() { .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") + .env("REVIEW_MODEL", "openai/gpt-5-mini") + .env("REVIEW_MODEL_CASCADE", "z-ai/glm-5.2") .args([ "review", "--publish", @@ -10412,1842 +10100,1667 @@ async fn forge_post_failure_on_success_path_keeps_gate_derived_exit_code() { "--output-json", ]) .assert() - .code(1); // gate-derived exit code, unaffected by the failed post - - let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); - let env: Value = serde_json::from_str(&stdout).unwrap(); - assert_eq!(env["gate"]["failing"], true); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); assert_eq!(env["findings"][0]["path"], "src/auth.rs"); + assert_eq!(env["modelUsed"], "z-ai/glm-5.2"); + assert_eq!(env["usage"]["promptTokens"], 330); + assert_eq!(env["usage"]["completionTokens"], 160); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); + assert_eq!(env["modelIncidents"][0]["category"], "invalidOutput"); + assert_eq!(env["modelIncidents"][0]["recovered"], true); + assert_eq!(env["modelIncidents"][0]["recovery"], "fallback"); - let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); - assert!( - stderr.contains("could not post review comment"), - "expected a warning about the failed post, got: {stderr}" - ); - - // Both check runs still completed despite the review-comment failure. - let reqs = server.received_requests().await.unwrap(); - let patches = reqs + let requests = server.received_requests().await.unwrap(); + let llm_positions = requests .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .count(); - assert_eq!(patches, 2); - let review_posts = reqs + .enumerate() + .filter_map(|(index, request)| (request.url.path() == "/chat/completions").then_some(index)) + .collect::>(); + let first_forge_result_write = requests .iter() - .filter(|request| { - request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews" + .enumerate() + .find_map(|(index, request)| { + (request.method == wiremock::http::Method::PATCH + || (request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews")) + .then_some(index) }) - .count(); - assert_eq!( - review_posts, 1, - "ambiguous review POST must not be replayed" + .expect("forge result publication write"); + assert_eq!(llm_positions.len(), 4); + assert!( + llm_positions + .into_iter() + .all(|index| index < first_forge_result_write) ); } #[tokio::test] -async fn hosted_review_post_failure_is_operational_after_the_envelope_is_persisted() { +async fn garbage_output_fails_closed_after_repair_attempt() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - mount_static_github_pr(&server).await; - mount_successful_hosted_check_patches(&server).await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(500).set_body_string("temporary review failure")) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) + .and(path("/chat/completions")) + .respond_with( + ResponseTemplate::new(200).set_body_json(llm_text("I cannot review this diff, sorry.")), + ) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), true); - let output = hosted_publish_command(dir.path(), &server).assert().code(2); - - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap_or_else(|e| { - panic!( - "hosted review did not emit its envelope: {e}; stderr: {}", - String::from_utf8_lossy(&output.get_output().stderr) - ) - }); - assert_eq!(envelope["modelUsed"], "none (disabled by config)"); - assert_eq!(envelope["gate"]["failing"], false); - let stderr = String::from_utf8_lossy(&output.get_output().stderr); - assert!(stderr.contains("required hosted review publication failed")); - - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .count(), - 2 - ); - assert_eq!( - requests - .iter() - .filter(|request| request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews") - .count(), - 3, - "each ambiguous failure must be reconciled before the bounded retry" - ); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") + .assert() + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert_eq!(env["usage"]["promptTokens"], 160); + assert_eq!(env["usage"]["completionTokens"], 60); + // Initial call plus one schema-repair call for the explicit model. + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + let models: Vec = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert_eq!(models, vec!["openai/gpt-5-mini", "openai/gpt-5-mini"]); } #[tokio::test] -async fn hosted_marker_reconciliation_accepts_an_ambiguous_review_post_without_a_duplicate() { +async fn cascade_falls_back_to_next_model() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - mount_static_github_pr(&server).await; - mount_successful_hosted_check_patches(&server).await; - let published_body = Arc::new(Mutex::new(None)); Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(AmbiguousReviewPostResponder { - published_body: published_body.clone(), - }) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(500).set_body_json(json!({ + "error": {"metadata": {"error_type": "provider_unavailable"}}, + "usage": {"prompt_tokens": 12, "completion_tokens": 0} + }))) .mount(&server) .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ReconciledReviewListResponder { published_body }) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), true); - hosted_publish_command(dir.path(), &server).assert().code(0); - - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") + .assert() + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["modelUsed"], "backup-model"); + assert_eq!(env["modelIncidents"][0]["phase"], "review"); + assert_eq!(env["modelIncidents"][0]["category"], "providerError"); + assert_eq!(env["modelIncidents"][0]["recovered"], true); + assert_eq!(env["modelIncidents"][0]["recovery"], "fallback"); + assert_eq!(env["usage"]["promptTokens"], 136); + assert_eq!(env["usage"]["completionTokens"], 50); + assert_eq!(env["modelUsage"][0]["model"], "primary-model"); + assert_eq!(env["modelUsage"].as_array().unwrap().len(), 4); + assert!( + env["modelUsage"].as_array().unwrap()[..3] .iter() - .filter(|request| request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews") - .count(), - 1, - "marker reconciliation must prevent a duplicate review" + .all(|entry| entry["model"] == "primary-model" && entry["promptTokens"] == 12) ); + assert_model_usage_matches_aggregate(&env); + assert_eq!(env["usageAccountingComplete"], true); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("postil: attempting model: primary-model")); + assert!(stderr.contains("postil: model primary-model returned retryable HTTP 500")); + assert!(stderr.contains("retrying in ")); + assert!(stderr.contains("postil: model primary-model failed after")); + assert!(stderr.contains("falling back to next model")); + assert!(stderr.contains("postil: attempting model: backup-model")); + assert!(stderr.contains("postil: model backup-model responded in")); } #[tokio::test] -async fn hosted_check_patch_failures_are_operational_and_do_not_hide_the_envelope() { +async fn consensus_logs_each_model_outcome() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - mount_static_github_pr(&server).await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) - .respond_with(ResponseTemplate::new(500).set_body_string("temporary check failure")) - .mount(&server) - .await; + for model in ["consensus-a", "consensus-b"] { + mock_review_model(&server, model, json!([])).await; + } let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), false); - let output = hosted_publish_command(dir.path(), &server).assert().code(2); + std::fs::write(dir.path().join(".postil.yaml"), "model:\n consensus: 2\n").unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "consensus-a") + .env("REVIEW_MODEL_CASCADE", "consensus-b") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .code(0); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap_or_else(|e| { - panic!( - "hosted review did not emit its envelope: {e}; stderr: {}", - String::from_utf8_lossy(&output.get_output().stderr) - ) - }); - assert_eq!(envelope["modelUsed"], "none (disabled by config)"); - let stderr = String::from_utf8_lossy(&output.get_output().stderr); - assert!(stderr.contains("required hosted check publication failed")); - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .count(), - 6, - "both required checks receive the complete bounded retry sequence" - ); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + for model in ["consensus-a", "consensus-b"] { + assert!(stderr.contains(&format!("postil: attempting consensus model: {model}"))); + assert!(stderr.contains(&format!("postil: consensus model {model} responded in"))); + } } #[tokio::test] -async fn hosted_freshness_lookup_failure_is_operational_before_any_write() { +async fn slow_model_request_retries_same_model_then_succeeds() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(GitHubFreshnessFailureResponder { - calls: Arc::new(AtomicUsize::new(0)), - }) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(llm_content(json!([]))), + ) + .up_to_n_times(1) .mount(&server) .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), false); - let output = hosted_publish_command(dir.path(), &server).assert().code(2); - let stderr = String::from_utf8_lossy(&output.get_output().stderr); - assert!(stderr.contains("snapshot freshness could not be verified")); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") + .assert() + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["modelUsed"], "primary-model"); + assert_eq!(env["usageAccountingComplete"], false); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("postil: model primary-model hit a request timeout after")); + assert!(stderr.contains("retry 1/13")); + assert!(stderr.contains("postil: model primary-model responded in")); + assert!(!stderr.contains("postil: attempting model: backup-model")); + + let requests = server.received_requests().await.unwrap(); + let models = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); + assert_eq!(models, vec!["primary-model", "primary-model"]); } #[tokio::test] -async fn hosted_stale_head_race_is_operational_and_suppresses_all_writes() { +async fn empty_success_response_retries_same_model_and_accumulates_usage() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(GitHubHeadRaceResponder { - calls: Arc::new(AtomicUsize::new(0)), - }) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "id": "empty-attempt", + "model": "primary-model", + "provider": "test-provider", + "choices": [], + "usage": {"prompt_tokens": 12, "completion_tokens": 0} + }))) + .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), false); - let output = hosted_publish_command(dir.path(), &server).assert().code(2); - let stderr = String::from_utf8_lossy(&output.get_output().stderr); - assert!(stderr.contains("pull request snapshot changed after review")); + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .code(0); + + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "primary-model"); + assert_eq!(envelope["usage"]["promptTokens"], 112); + assert_eq!(envelope["usage"]["completionTokens"], 50); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + assert_eq!(envelope["modelUsage"][0]["promptTokens"], 12); + assert_eq!(envelope["modelUsage"][1]["promptTokens"], 100); + assert_eq!(envelope["modelUsage"][0]["attempt"], 1); + assert_eq!(envelope["modelUsage"][1]["attempt"], 2); + assert_eq!(envelope["usageAccountingComplete"], true); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("returned empty content")); + assert!(stderr.contains("phase=review")); + assert!(stderr.contains("provider=present")); + assert!(stderr.contains("usage=present")); + assert!(stderr.contains("response_id=present")); + assert!(!stderr.contains("empty-attempt")); + + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + assert!( + requests + .iter() + .all(|request| { request.body_json::().unwrap()["max_tokens"] == 8_000 }) + ); } #[tokio::test] -async fn hosted_silent_review_requires_checks_but_not_a_pr_comment() { +async fn exhausted_reasoning_budget_expands_the_same_model_retry() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - mount_static_github_pr(&server).await; - mount_successful_hosted_check_patches(&server).await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(OutputBudgetResponder) + .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), false); - hosted_publish_command(dir.path(), &server).assert().code(0); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "primary-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .code(0); + + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "primary-model"); + assert_eq!(envelope["usage"]["promptTokens"], 30_845); + assert_eq!(envelope["usage"]["completionTokens"], 8_050); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("exhausted 8000 output tokens")); + assert!(stderr.contains("retrying the complete request with 16000 tokens")); + assert!(stderr.contains("reasoning_tokens=8000")); + + let requests = server.received_requests().await.unwrap(); + let max_tokens = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["max_tokens"] + .as_u64() + .unwrap() + }) + .collect::>(); + assert_eq!(max_tokens, vec![8_000, 16_000]); } #[tokio::test] -async fn hosted_silent_review_does_not_require_a_later_comment_freshness_check() { +async fn repeated_exhausted_reasoning_budget_does_not_consume_an_empty_retry() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - let calls = Arc::new(AtomicUsize::new(0)); - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(GitHubLateHeadRaceResponder { - calls: calls.clone(), - }) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api")) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "id": 42, - "full_name": "acme/api" + "choices": [{ + "finish_reason": "length", + "message": {"content": null, "reasoning": "budget exhausted"} + }], + "usage": { + "prompt_tokens": 4_194, + "completion_tokens": 8_000, + "completion_tokens_details": {"reasoning_tokens": 8_000} + } }))) - .mount(&server) - .await; - mount_successful_hosted_check_patches(&server).await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - disable_review_for_hosted_publication(dir.path(), false); - hosted_publish_command(dir.path(), &server).assert().code(0); - assert_eq!( - calls.load(Ordering::SeqCst), - 4, - "a no-comment result must stop after the required check delivery freshness reads" - ); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("REVIEW_MODEL", "primary-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) + .assert() + .code(1); + + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("retrying the complete request with 16000 tokens")); + assert!(!stderr.contains("returned empty content")); + + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + let max_tokens = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["max_tokens"] + .as_u64() + .unwrap() + }) + .collect::>(); + assert_eq!(max_tokens, vec![8_000, 16_000]); } #[tokio::test] -async fn github_unresolved_inline_line_retries_on_a_changed_line() { +async fn openai_truncation_retries_the_complete_original_request() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; + let partial_title = "Partial output must never publish"; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": "b", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with( - ResponseTemplate::new(422) - .set_body_string(r#"{"message":"Line could not be resolved"}"#), - ) - .up_to_n_times(1) - .with_priority(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) - .expect(1) - .mount(&server) - .await; - let (published_review, published_comments) = published_review_responders(); - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(published_review) - .with_priority(2) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/reviews/77/comments")) - .and(query_param("per_page", "100")) - .and(query_param("page", "1")) - .respond_with(published_comments) - .expect(1) - .mount(&server) - .await; - Mock::given(method("PUT")) - .and(path("/repos/acme/api/pulls/7/reviews/77")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .expect(1) + .respond_with(SequentialReviewResponder { + calls: Arc::new(AtomicUsize::new(0)), + responses: Arc::new(vec![ + json!({ + "choices": [{"finish_reason": "length", "message": {"content": json!({ + "summary": "Incomplete review.", + "findings": [{ + "path": "src/auth.rs", "line": 42, "severity": "error", + "kind": "risk", "confidence": 1.0, + "title": partial_title, "body": "This text is incomplete.", + "evidence": "exec_query(&token);" + }] + }).to_string()}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 8000} + }), + llm_content(json!([])), + ]), + }) + .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let receipt_name = "publication-receipt.json"; - let receipt_path = dir.path().join(receipt_name); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env( - "POSTIL_DETAILS_URL", - "https://postil.dev/orgs/acme/runs/run-1", - ) - .env("POSTIL_PUBLICATION_RECEIPT_PATH", receipt_name) - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "primary-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); + .success(); + + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + assert!(!stdout.contains(partial_title)); + let envelope: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(envelope["silent"], true); + assert_eq!(envelope["usage"]["promptTokens"], 200); + assert_eq!(envelope["usage"]["completionTokens"], 8050); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + assert_model_usage_matches_aggregate(&envelope); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("category=unresolved-line")); - assert!(stderr.contains("recovery=placement-ladder")); let requests = server.received_requests().await.unwrap(); - let review_bodies = requests - .iter() - .filter(|request| { - request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews" - }) - .map(|request| request.body_json::().unwrap()) - .collect::>(); - assert_eq!(review_bodies.len(), 2); - assert!(review_bodies[0].get("comments").is_some()); - let fallback_comments = review_bodies[1]["comments"].as_array().unwrap(); - assert_eq!(fallback_comments.len(), 1); - assert_eq!(fallback_comments[0]["path"], "src/auth.rs"); - assert_eq!(fallback_comments[0]["line"], 41); - assert_eq!(fallback_comments[0]["side"], "RIGHT"); - assert!(fallback_comments[0].get("start_line").is_none()); + assert_eq!(requests.len(), 2); + let first: Value = requests[0].body_json().unwrap(); + let second: Value = requests[1].body_json().unwrap(); + assert_eq!(first["max_tokens"], 8_000); + assert_eq!(second["max_tokens"], 16_000); + assert_eq!(first["messages"], second["messages"]); assert!( - !review_bodies[1]["body"] - .as_str() + !serde_json::to_string(&second) .unwrap() - .contains("inline placement unavailable") + .contains(partial_title) ); - let receipt: Value = serde_json::from_slice(&std::fs::read(&receipt_path).unwrap()).unwrap(); - assert_eq!(receipt["version"], 2); - assert_eq!(receipt["channel"], "reviewComments"); - assert_eq!(receipt["reviewId"], "77"); - assert_eq!(receipt["findings"][0]["initialOutcome"], "inline"); - assert_eq!(receipt["findings"][0]["commentId"], "500"); - assert!(!std::fs::read_dir(dir.path()).unwrap().any(|entry| { - let name = entry.unwrap().file_name(); - let name = name.to_string_lossy(); - name.starts_with(".publication-receipt.json.") && name.ends_with(".tmp") - })); - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt; - assert_eq!( - std::fs::metadata(&receipt_path) - .unwrap() - .permissions() - .mode() - & 0o777, - 0o600 - ); - } } #[tokio::test] -async fn hosted_path_completes_provided_check_run_ids_without_creating_new_ones() { +async fn repeated_openai_truncation_fails_closed_without_partial_text() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; + let partial_title = "Partial output must never publish"; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 + "choices": [{"finish_reason": "length", "message": {"content": json!({ + "summary": "Incomplete review.", + "findings": [{ + "path": "src/auth.rs", "line": 42, "severity": "error", + "kind": "risk", "confidence": 1.0, + "title": partial_title, "body": "This text is incomplete.", + "evidence": "exec_query(&token);" + }] + }).to_string()}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 8000} }))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .expect(2) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--sha", - "aaaaaaaa", - "--check-run-id", - "901", - "--gate-check-run-id", - "902", - "--output-json", - ]) + .env("REVIEW_MODEL", "primary-model") + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(0); + .code(1); - let reqs = server.received_requests().await.unwrap(); - // The worker owns check-run creation; the CLI must not create its own. - assert!( - !reqs.iter().any(|r| r.method == wiremock::http::Method::POST - && r.url.path() == "/repos/acme/api/check-runs") + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + assert!(!stdout.contains(partial_title)); + let envelope: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(envelope["gate"]["failing"], true); + assert_eq!(envelope["findings"][0]["path"], ".postil/model-output"); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 2); + assert_model_usage_matches_aggregate(&envelope); + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + assert_eq!( + requests[0].body_json::().unwrap()["max_tokens"], + 8_000 ); - // Both pre-created runs completed. - let patched: Vec<&str> = reqs - .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .map(|r| r.url.path()) - .collect(); - assert!(patched.contains(&"/repos/acme/api/check-runs/901")); - assert!(patched.contains(&"/repos/acme/api/check-runs/902")); + assert_eq!( + requests[1].body_json::().unwrap()["max_tokens"], + 16_000 + ); + assert!(requests.iter().all(|request| { + !String::from_utf8_lossy(&request.body).contains("You repair malformed JSON") + })); } #[tokio::test] -async fn hosted_path_rejects_a_changed_target_before_review_or_delivery() { +async fn rate_limit_after_exhausted_output_uses_the_remaining_attempt_and_retry_after() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, - "base": {"sha": "bbbbbbbb"}, - "changed_files": 1 - }))) - .mount(&server) - .await; + let calls = Arc::new(AtomicUsize::new(0)); Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(0) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({"id": 1}))) - .expect(0) + .and(body_string_contains("primary-model")) + .respond_with(OutputThenRateLimitResponder { + calls: calls.clone(), + }) + .expect(3) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--sha", - "aaaaaaaa", - "--base-sha", - "cccccccc", - ]) + .env("REVIEW_MODEL", "primary-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(2); + .code(0); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["silent"], true); + assert_eq!(calls.load(Ordering::SeqCst), 3); + assert_eq!(envelope["modelUsage"].as_array().unwrap().len(), 3); let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains( - "requested review target cccccccc is no longer the pull request target bbbbbbbb" - )); + assert!(stderr.contains("returned retryable HTTP 429")); + assert!(stderr.contains("retrying in 0ms")); + assert!(stderr.contains("attempt=3/3")); } #[tokio::test] -async fn github_flow_posts_review_and_completes_both_checks() { +async fn empty_success_without_usage_marks_accounting_incomplete() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - // LLM Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), - ) - .mount(&server) - .await; - // PR meta (default Accept) and diff (v3.diff Accept) on the same path. - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "body": "PR body", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - let (published_review, published_comments) = published_review_responders(); - Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(published_review) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/reviews/77/comments")) - .and(query_param("per_page", "100")) - .and(query_param("page", "1")) - .respond_with(published_comments) - .expect(2) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({"choices": []}))) + .up_to_n_times(1) .mount(&server) .await; - Mock::given(method("PUT")) - .and(path("/repos/acme/api/pulls/7/reviews/77")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env( - "POSTIL_DETAILS_URL", - "https://postil.dev/orgs/acme/runs/review-7", - ) - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "primary-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["headSha"], "aaaaaaaaaaaa"); - assert_eq!(env["baseSha"], "bbbbbbbb"); + .code(0); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["usageAccountingComplete"], false); +} - let reqs = server.received_requests().await.unwrap(); - // Two check-run creations. - let creates: Vec<_> = reqs - .iter() - .filter(|r| { - r.method == wiremock::http::Method::POST && r.url.path() == "/repos/acme/api/check-runs" - }) - .collect(); - assert_eq!(creates.len(), 2); - assert!(creates.iter().all(|request| { - request.body_json::().unwrap()["details_url"] - == "https://postil.dev/orgs/acme/runs/review-7" - })); - let names: Vec = creates - .iter() - .map(|r| { - r.body_json::().unwrap()["name"] - .as_str() - .unwrap() - .to_string() - }) - .collect(); - assert!(names.contains(&"postil/review".to_string())); - assert!(names.contains(&"postil/gate".to_string())); - // Both completed; gate concluded failure, advisory success. - let patches: Vec = reqs - .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .map(|r| r.body_json().unwrap()) - .collect(); - assert_eq!(patches.len(), 2); - assert!( - patches - .iter() - .all(|patch| { patch["details_url"] == "https://postil.dev/orgs/acme/runs/review-7" }) - ); - let conclusions: Vec<&str> = patches - .iter() - .map(|p| p["conclusion"].as_str().unwrap()) - .collect(); - assert!(conclusions.contains(&"success")); - assert!(conclusions.contains(&"failure")); - assert!( - patches - .iter() - .all(|patch| patch["output"].get("annotations").is_none()) - ); - let gate_patch = patches - .iter() - .find(|patch| patch["conclusion"] == "failure") - .unwrap(); - assert_eq!(gate_patch["output"]["title"], "Merge gate failed"); - assert_eq!( - gate_patch["output"]["summary"], - "Merge gate failed: 1 finding blocks under the configured policy (failOn: error).\n\n- `src/auth.rs:41` Unsanitized input reaches query\n" - ); - // Inline review posted with the finding at the cited line. - let review = reqs - .iter() - .find(|r| r.url.path() == "/repos/acme/api/pulls/7/reviews") - .expect("review posted"); - let body: Value = review.body_json().unwrap(); - assert_eq!(body["comments"][0]["path"], "src/auth.rs"); - assert_eq!(body["comments"][0]["line"], 41); - let initial_summary = body["body"].as_str().unwrap(); - assert!(!initial_summary.contains("posted inline")); - let update = reqs - .iter() - .find(|r| { - r.method == wiremock::http::Method::PUT - && r.url.path() == "/repos/acme/api/pulls/7/reviews/77" - }) - .expect("review summary updated"); - let update_body: Value = update.body_json().unwrap(); - let summary = update_body["body"].as_str().unwrap(); - assert!(summary.starts_with(&format!( - "{} **1 blocking finding open**\n", - postil_cli::forge::icon_md("error"), - ))); - assert!(summary.contains("1 finding posted inline")); - assert!(!summary.contains("Unsanitized input reaches query")); - assert!(!summary.contains("`src/auth.rs:41`")); - assert!(!summary.contains("Review metadata")); - assert!(!summary.contains("headsha")); - assert!(!summary.contains("Tokens")); - assert!( - summary.contains("[Review details](https://postil.dev/orgs/acme/runs/review-7)") - ); +#[tokio::test] +async fn malformed_success_without_usage_marks_accounting_incomplete_before_fallback() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_string("not-json")) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; - std::fs::write( - dir.path().join(".postil.yaml"), - "review:\n findingPresentation: checkAnnotations\n", - ) - .unwrap(); - let annotation_receipt_path = dir.path().join("annotation-receipt.json"); - let request_count_before_annotations = reqs.len(); - postil() + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env("POSTIL_PUBLICATION_RECEIPT_PATH", &annotation_receipt_path) - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); - let annotation_receipt: Value = - serde_json::from_slice(&std::fs::read(&annotation_receipt_path).unwrap()).unwrap(); - assert_eq!(annotation_receipt["version"], 2); - assert_eq!(annotation_receipt["channel"], "checkAnnotations"); - assert!(annotation_receipt.get("reviewId").is_none()); - assert_eq!( - annotation_receipt["findings"][0]["initialOutcome"], - "checkAnnotation" - ); - assert!(annotation_receipt["findings"][0].get("commentId").is_none()); - let annotation_requests = server.received_requests().await.unwrap(); - let annotation_requests = &annotation_requests[request_count_before_annotations..]; - assert!(!annotation_requests.iter().any(|request| { - request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/pulls/7/reviews" - })); - let annotation_patches = annotation_requests - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .map(|request| request.body_json::().unwrap()) - .collect::>(); - assert_eq!(annotation_patches.len(), 2); - let advisory_annotation = annotation_patches - .iter() - .find(|patch| patch["output"].get("annotations").is_some()) - .expect("advisory check annotation"); - assert_eq!( - advisory_annotation["output"]["annotations"][0]["path"], - "src/auth.rs" - ); - assert_eq!( - advisory_annotation["output"]["annotations"][0]["start_line"], - 41 - ); - assert_eq!( - annotation_patches - .iter() - .filter(|patch| patch["output"].get("annotations").is_some()) - .count(), - 1 - ); - std::fs::remove_file(dir.path().join(".postil.yaml")).unwrap(); - let request_count_before_deferred_gate = server.received_requests().await.unwrap().len(); + .code(0); + + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "backup-model"); + assert_eq!(envelope["usageAccountingComplete"], false); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("usage=missing")); + assert!(!stderr.contains("not-json")); +} + +#[tokio::test] +async fn malformed_success_with_usage_records_the_attempt_exactly_once() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": "invalid-shape", + "usage": {"prompt_tokens": 12, "completion_tokens": 3} + }))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--defer-gate-check", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["gate"]["failing"], true); + .code(0); - let reqs = server.received_requests().await.unwrap(); - let conclusions: Vec = reqs[request_count_before_deferred_gate..] - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .map(|request| { - request.body_json::().unwrap()["conclusion"] - .as_str() - .unwrap() - .to_string() - }) - .collect(); - assert_eq!(conclusions, vec!["success"]); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "backup-model"); + assert_eq!(envelope["usage"]["promptTokens"], 112); + assert_eq!(envelope["usage"]["completionTokens"], 53); + assert_eq!(envelope["modelUsage"][0]["promptTokens"], 12); + assert_eq!(envelope["modelUsage"][0]["completionTokens"], 3); + assert_eq!(envelope["usageAccountingComplete"], true); } #[tokio::test] -async fn github_push_after_acquisition_suppresses_all_stale_publication() { +async fn timeout_after_exhausted_output_uses_remaining_attempt_then_preserves_cascade() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(GitHubHeadRaceResponder { - calls: Arc::new(AtomicUsize::new(0)), - }) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "choices": [{ + "finish_reason": "length", + "message": {"content": null, "reasoning": "budget exhausted"} + }], + "usage": { + "prompt_tokens": 12, + "completion_tokens": 8_000, + "completion_tokens_details": {"reasoning_tokens": 8_000} + } + }))) + .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(llm_content(json!([]))), + ) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200)) - .expect(0) + .and(path("/chat/completions")) + .and(body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1) - .stderr(predicates::str::contains( - "publication skipped because the pull request snapshot changed", - )); -} + .code(0); -// An LLM response with a caller-provided summary and findings (used for -// content-policy scenarios where the finding is not the standard auth one). -fn llm_with_summary(summary: &str, findings: Value) -> Value { - let findings = explicit_repository_context(findings); - json!({ - "choices": [{"finish_reason": "stop", "message": {"content": json!({ - "summary": summary, - "findings": findings - }).to_string()}}], - "usage": {"prompt_tokens": 100, "completion_tokens": 50} - }) + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "backup-model"); + let requests = server.received_requests().await.unwrap(); + let models = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); + assert_eq!( + models, + vec![ + "primary-model", + "primary-model", + "primary-model", + "backup-model" + ] + ); } -// Shared GitHub remote setup for the content-policy PR-description tests. The PR -// body carries prose the model flags; content policy is active by default. -async fn content_policy_pr_server(llm: Value) -> MockServer { +#[tokio::test] +async fn empty_response_retry_http_failure_uses_the_third_primary_request_before_fallback() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) + .and(body_string_contains("primary-model")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", - "body": "This change updates review retention behavior.", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + "choices": [], + "usage": {"prompt_tokens": 12, "completion_tokens": 0} }))) + .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .and(path("/chat/completions")) + .and(body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(500).set_body_json(json!({ + "error": {"metadata": {"error_type": "provider_unavailable"}} + }))) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .and(path("/chat/completions")) + .and(body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; - server -} - -#[tokio::test] -async fn content_policy_pr_body_finding_survives_grounding() { - // A content-policy finding against the PR description grounds on the reserved - // `.postil/pr-description` path instead of being dropped as ungrounded (which - // would have spuriously fail-closed a run whose only finding was here). - let cp_finding = json!([{ - "path": ".postil/pr-description", "line": 2, "severity": "warn", - "kind": "contentPolicy", "confidence": 0.9, - "title": "Retention scope missing from PR description", - "body": "State the supported retention scope in the description.", - "evidence": "This change updates review retention behavior." - }]); - let server = content_policy_pr_server(llm_with_summary( - "PR description omits the supported retention scope.", - cp_finding, - )) - .await; let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "review", - "--publish", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "2") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(0); // warn severity: kept, but gate passes at default failOn=error - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - // The finding survived grounding: it is NOT a fail-closed model-output error. - assert_eq!(env["findings"][0]["path"], ".postil/pr-description"); - assert_eq!(env["findings"][0]["kind"], "contentPolicy"); - assert_eq!(env["counts"]["ungrounded"], 0); - assert_eq!(env["gate"]["failing"], false); - - // The model was shown the numbered PR-description block. - let reqs = server.received_requests().await.unwrap(); - let llm = reqs - .iter() - .find(|r| r.url.path() == "/chat/completions") - .unwrap(); - let sent: Value = llm.body_json().unwrap(); - let user_msg = sent["messages"][1]["content"].as_str().unwrap(); - assert!(user_msg.contains(".postil/pr-description")); - assert!(user_msg.contains(" 1 Add login")); - assert!(user_msg.contains(" 2 This change updates review retention behavior.")); + .code(0); - // The reserved-path finding has no real line, so its bounded detail appears - // in the PR summary instead of an inline comment. - let review = reqs + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["modelUsed"], "backup-model"); + let requests = server.received_requests().await.unwrap(); + let models = requests .iter() - .find(|r| r.url.path() == "/repos/acme/api/pulls/7/reviews") - .expect("review posted"); - let body: Value = review.body_json().unwrap(); + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); assert_eq!( - body["comments"].as_array().map(|a| a.len()).unwrap_or(0), - 0, - "reserved-path finding was posted as an inline comment" + models, + vec![ + "primary-model", + "primary-model", + "primary-model", + "backup-model" + ] ); - let summary = body["body"].as_str().unwrap(); - assert!(summary.contains(&format!( - "{} **1 advisory finding open**", - postil_cli::forge::icon_md("info") - ))); - assert!(summary.contains("1 finding in review details")); - assert!(summary.contains("Retention scope missing from PR description")); - assert!(summary.contains("in pull request description")); - assert!(summary.contains("State the supported retention scope in the description.")); } -#[tokio::test] -async fn content_policy_clean_run_does_not_fail_close() { - // With default content policy and no violations, the run stays clean: the numbered - // PR-description block must not induce a spurious ungrounded/fail-closed run. - let server = content_policy_pr_server(llm_with_summary("", json!([]))).await; +#[test] +fn connection_failure_after_exhausted_output_uses_the_remaining_third_request() { + use std::io::{Read, Write}; + + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let address = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut request = [0_u8; 16_384]; + let _ = stream.read(&mut request).unwrap(); + let body = r#"{"choices":[{"finish_reason":"length","message":{"content":null,"reasoning":"budget exhausted"}}],"usage":{"prompt_tokens":12,"completion_tokens":8000,"completion_tokens_details":{"reasoning_tokens":8000}}}"#; + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ) + .unwrap(); + stream.flush().unwrap(); + }); let dir = tempfile::tempdir().unwrap(); + std::fs::write( + dir.path().join(".postil.yaml"), + "model:\n name: primary-model\n cascade: []\n", + ) + .unwrap(); + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args(["review", "--repo", "acme/api", "--pr", "7", "--output-json"]) + .env("POSTIL_API_BASE", format!("http://{address}")) + .env("REVIEW_MODEL", "primary-model") + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["silent"], true); - assert_eq!(env["gate"]["failing"], false); - assert_eq!( - env["findings"].as_array().map(|a| a.len()).unwrap_or(0), - 0, - "a clean content-policy run produced a spurious finding" + .code(1); + server.join().unwrap(); + + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!( + stderr.contains("model=primary-model attempt=2/13"), + "unexpected log: {stderr}" ); + assert!(stderr.contains("model=primary-model attempt=3/13")); + assert!(!stderr.contains("127.0.0.1")); } #[tokio::test] -async fn github_clean_pr_stays_silent_but_completes_checks() { +async fn timeout_http_status_retries_same_model_then_succeeds() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(408).set_body_string("request timed out")) + .up_to_n_times(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/repos/acme/api/check-runs")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) - .mount(&server) - .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") .assert() .code(0); + let envelope: Value = + serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); + assert_eq!(envelope["modelUsed"], "primary-model"); + assert_eq!(envelope["usageAccountingComplete"], false); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("returned timeout HTTP 408 Request Timeout")); + assert!(stderr.contains("retry 1/13")); - let reqs = server.received_requests().await.unwrap(); - // Silence is a feature: no review comment posted on a clean PR. - assert!(!reqs.iter().any(|r| r.url.path().ends_with("/reviews"))); - // But both checks completed successfully. - let conclusions: Vec = reqs - .iter() - .filter(|r| r.method == wiremock::http::Method::PATCH) - .map(|r| { - r.body_json::().unwrap()["conclusion"] - .as_str() - .unwrap() - .to_string() - }) - .collect(); - assert_eq!(conclusions, vec!["success", "success"]); - let check_requests: Vec = reqs - .iter() - .filter(|request| { - request.url.path().starts_with("/repos/acme/api/check-runs") - && matches!( - request.method, - wiremock::http::Method::POST | wiremock::http::Method::PATCH - ) - }) - .map(|request| request.body_json().unwrap()) - .collect(); + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); assert!( - check_requests + requests .iter() - .all(|request| request.get("details_url").is_none()) - ); - let gate_patch = check_requests - .iter() - .find(|request| { - request["output"]["summary"] - .as_str() - .is_some_and(|summary| summary.starts_with("Merge gate passed:")) - }) - .expect("gate completion payload"); - assert_eq!(gate_patch["output"]["title"], "Merge gate passed"); - assert_eq!( - gate_patch["output"]["summary"], - "Merge gate passed: no findings block under the configured policy (failOn: error).\n" + .all(|request| { request.body_json::().unwrap()["model"] == "primary-model" }) ); +} - // The explicit onClean mode uses the same unified summary as finding-bearing - // reviews instead of falling back to the former one-line clean message. +#[tokio::test] +async fn exhausted_timeout_retry_falls_back_to_next_model() { + let server = MockServer::start().await; Mock::given(method("POST")) - .and(path("/repos/acme/api/pulls/7/reviews")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(llm_content(json!([]))), + ) .mount(&server) .await; - std::fs::write( - dir.path().join(".postil.yaml"), - "review:\n onClean: comment\n", - ) - .unwrap(); - postil() + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env( - "POSTIL_DETAILS_URL", - "https://postil.dev/orgs/acme/runs/clean-7", - ) - .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "10") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") .assert() .code(0); - let reqs = server.received_requests().await.unwrap(); - let clean_review = reqs + let envelope: Value = + serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); + assert_eq!(envelope["modelUsed"], "backup-model"); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!(stderr.contains("model primary-model hit a request timeout after")); + assert!(stderr.contains("retry 1/13")); + let attempts = envelope["modelUsage"] + .as_array() + .unwrap() .iter() - .rev() - .find(|request| request.url.path().ends_with("/reviews")) - .expect("onClean review posted"); - let clean_body: Value = clean_review.body_json().unwrap(); - let clean_summary = clean_body["body"].as_str().unwrap(); - assert!(clean_summary.starts_with("Postil reviewed this change")); - assert!(clean_summary.contains("Postil reviewed this change and found nothing")); - assert!(!clean_summary.contains("Review metadata")); + .map(|entry| { + ( + entry["model"].as_str().unwrap(), + entry["attempt"].as_u64().unwrap(), + ) + }) + .collect::>(); + assert_eq!( + attempts, + vec![ + ("primary-model", 1), + ("primary-model", 2), + ("primary-model", 3), + ("backup-model", 1) + ] + ); + + let requests = server.received_requests().await.unwrap(); + let models = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); + assert_eq!(models.last().map(String::as_str), Some("backup-model")); + assert!(models.len() <= 4); + assert_eq!( + models + .iter() + .filter(|model| model.as_str() == "backup-model") + .count(), + 1 + ); assert!( - clean_summary - .contains("[Review details](https://postil.dev/orgs/acme/runs/clean-7)") + models[..models.len() - 1] + .iter() + .all(|model| model == "primary-model") ); } #[tokio::test] -async fn incremental_review_resolves_and_carries_baseline_findings() { +async fn mixed_failures_share_the_existing_two_retry_cap() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(500).set_body_string("upstream down")) + .up_to_n_times(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(llm_content(json!([]))), + ) + .up_to_n_times(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with(ResponseTemplate::new(500).set_body_string("upstream still down")) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); // touches src/auth.rs:40-47 - - // Baseline: one finding inside the touched range (resolved), one elsewhere (carried). - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [ - {"path": "src/auth.rs", "line": 41, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "old auth bug", "body": "fixed now"}, - {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "still broken", "body": "not addressed"} - ], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 2, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,2], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - + let diff = write_diff(dir.path()); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "1") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "15") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--since-sha", "abc123", "--baseline"]) - .arg(&baseline_path) .arg("--output-json") .assert() - .code(1); // carried error finding keeps the gate failing - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["resolved"][0]["title"], "old auth bug"); - assert_eq!(env["findings"][0]["title"], "still broken"); - assert!( - env["findings"][0]["body"] - .as_str() - .unwrap() - .starts_with("[carried") + .code(0); + let envelope: Value = + serde_json::from_slice(&out.get_output().stdout).expect("review output should be JSON"); + assert_eq!(envelope["modelUsed"], "backup-model"); + + let requests = server.received_requests().await.unwrap(); + let models = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); + assert_eq!( + models, + vec![ + "primary-model", + "primary-model", + "primary-model", + "backup-model" + ] ); - assert_eq!(env["gate"]["failing"], true); - assert_eq!(env["sinceSha"], "abc123"); } #[tokio::test] -async fn incremental_unavailable_repository_receipt_carries_baseline_claim() { +async fn total_llm_timeout_caps_the_cascade() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(1500)) + .set_body_json(llm_content(json!([]))), + ) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Widget dependency is absent", - "body": "The repository does not contain widget version 2.0.", - "repositoryContext": {"claim": "absence", "resources": ["widget"], "versions": ["2.0"]} - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "1") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") .args(["review", "--diff-file"]) .arg(&diff) - .args(["--since-sha", "previous", "--baseline"]) - .arg(&baseline_path) - .args(["--output", "json"]) + .arg("--output-json") .assert() .code(1); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["resolved"], json!([])); - assert!( - envelope["findings"][0]["body"] - .as_str() - .unwrap() - .starts_with("[carried from previous review]") - ); - assert_eq!(envelope["repositorySearch"]["state"], "unavailable"); -} - -#[tokio::test] -async fn full_rereview_preserves_unresolved_baseline_for_unavailable_and_exhausted_receipts() { - for (name, repository, resources, state) in [ - ( - "unavailable", - false, - vec!["widget".to_string()], - "unavailable", - ), - ("exhausted", true, vec!["widget".to_string()], "exhausted"), - ] { - let server = MockServer::start().await; - mock_review(&server, json!([])).await; - let dir = tempfile::tempdir().unwrap(); - if repository { - initialize_staged_repository(dir.path()); - } - let diff = write_diff(dir.path()); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [ - { - "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Widget dependency is absent", - "body": "The repository does not contain the required widget dependency.", - "evidence": "exec_query(&token);", - "repositoryContext": {"claim": "absence", "resources": resources} - }, - { - "path": "src/other.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Unchanged widget dependency is absent", - "body": "The unchanged component requires the missing widget dependency.", - "evidence": "use widget::Client;", - "repositoryContext": {"claim": "absence", "resources": ["widget"]} - } - ], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 2, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,2], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join(format!("{name}-baseline.json")); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/provider"); - let mut command = postil(); - command - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .arg("review"); - if repository { - command.arg("--staged"); - } else { - command.arg("--diff-file").arg(&diff); - } - command - .arg("--baseline") - .arg(&baseline_path) - .args(["--output", "json"]); - let out = command.assert().code(1); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["repositorySearch"]["state"], state, "{name}"); - assert_eq!(envelope["resolved"], json!([]), "{name}"); - assert_eq!(envelope["counts"]["suppressed"], 0, "{name}"); - let titles = envelope["findings"] - .as_array() - .unwrap() - .iter() - .map(|finding| finding["title"].as_str().unwrap()) - .collect::>(); - assert_eq!( - titles, - std::collections::BTreeSet::from([ - "Unchanged widget dependency is absent", - "Widget dependency is absent", - ]), - "{name}" - ); - assert_eq!(envelope["gate"]["failing"], true, "{name}"); - } + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 1); + let model = requests[0].body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string(); + assert_eq!(model, "primary-model"); } #[tokio::test] -async fn full_rereview_resolves_false_absence_from_unchanged_repository_source() { +async fn shared_total_budget_can_expire_during_cascade_retry() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("single finding adjudicator")) - .respond_with(RepositoryEvidenceAdjudicator) - .with_priority(1) + .and(wiremock::matchers::body_string_contains("primary-model")) + .respond_with( + ResponseTemplate::new(400) + .set_delay(std::time::Duration::from_millis(400)) + .set_body_string("bad request"), + ) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .and(wiremock::matchers::body_string_contains("backup-model")) + .respond_with( + ResponseTemplate::new(200) + .set_delay(std::time::Duration::from_millis(2200)) + .set_body_json(llm_content(json!([]))), + ) .mount(&server) .await; - let directory = tempfile::tempdir().unwrap(); - initialize_staged_repository_with_unchanged_caller(directory.path()); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 1, "severity": "error", "kind": "risk", - "confidence": 0.95, "title": "Legacy API has no callers", - "body": "The repository has no caller for `legacy_api`; remove it or restore its caller.", - "evidence": "fn login() {}", - "repositoryContext": {"claim": "absence", "identifiers": ["legacy_api"]} - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = directory.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--staged", "--baseline"]) - .arg(&baseline_path) - .args(["--output", "json"]) + .env("POSTIL_LLM_REQUEST_TIMEOUT_SECS", "5") + .env("POSTIL_LLM_TOTAL_TIMEOUT_SECS", "2") + .env("REVIEW_MODEL", "primary-model") + .env("REVIEW_MODEL_CASCADE", "backup-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--output-json") .assert() - .code(0); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/provider"); - assert_eq!(envelope["findings"], json!([])); - assert_eq!( - envelope["resolved"][0]["title"], - "Legacy API has no callers" - ); - assert_eq!(envelope["repositorySearch"]["state"], "complete"); - assert_eq!(envelope["gate"]["failing"], false); - assert!(envelope["repositorySearch"].get("evidence").is_none()); - assert!(envelope.get("repositoryEvidence").is_none()); + let requests = server.received_requests().await.unwrap(); + assert_eq!(requests.len(), 2); + let models = requests + .iter() + .map(|request| { + request.body_json::().unwrap()["model"] + .as_str() + .unwrap() + .to_string() + }) + .collect::>(); + assert_eq!(models, vec!["primary-model", "backup-model"]); } #[tokio::test] -async fn same_hunk_groups_replacement_refutes_false_deletion_claim() { - const PATH: &str = "ansible/group_vars/atlas.yml"; - const OLD_LINE: &str = "groups: [atlas_packages_base]"; - const REPLACEMENT: &str = "groups: [atlas_packages_base, atlas_packages_test]"; - +async fn forge_post_failure_on_success_path_keeps_gate_derived_exit_code() { + // A completed review (gate failing on a real finding) whose forge comment + // post then fails (rate limit, transient 5xx) must not discard the + // already-computed envelope or exit 2: the exit code stays derived from + // the gate, the envelope is still emitted, and both check runs still + // complete. Only the review comment itself is lost, and only a stderr + // warning marks that. let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("single finding adjudicator")) - .respond_with(RefuteFromReceiptAdjudicator) - .with_priority(1) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + }))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + // The review comment post itself fails, which is the fault this test is pinning. + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(500).set_body_string("secondary rate limit")) .mount(&server) .await; - mock_review( - &server, - json!([{ - "path": PATH, - "line": 120, - "severity": "warn", - "kind": "uncertainty", - "confidence": 0.91, - "title": "The groups entry was deleted", - "body": "The `groups` entry was deleted without its replacement `atlas_packages_test`.", - "evidence": "policy: atlas", - "repositoryContext": { - "claim": "mismatch", - "resources": ["groups"], - "values": ["atlas_packages_test"], - "versions": [], - "paths": [], - "identifiers": [] - } - }]), - ) - .await; - - let directory = tempfile::tempdir().unwrap(); - let diff = directory.path().join("groups-replacement.diff"); - std::fs::write( - &diff, - format!( - "diff --git a/{PATH} b/{PATH}\n--- a/{PATH}\n+++ b/{PATH}\n@@ -120,4 +120,4 @@\n-policy: legacy\n+policy: atlas\n stable: true\n owner: atlas\n-{OLD_LINE}\n+{REPLACEMENT}\n" - ), - ) - .unwrap(); - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", + ]) .assert() - .success(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + .code(1); // gate-derived exit code, unaffected by the failed post - assert_eq!(envelope["findings"], json!([])); - assert_eq!(envelope["suppressedFindings"][0]["finding"]["path"], PATH); - assert_eq!( - envelope["suppressedFindings"][0]["finding"]["title"], - "The groups entry was deleted" + let stdout = String::from_utf8(out.get_output().stdout.clone()).unwrap(); + let env: Value = serde_json::from_str(&stdout).unwrap(); + assert_eq!(env["gate"]["failing"], true); + assert_eq!(env["findings"][0]["path"], "src/auth.rs"); + + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!( + stderr.contains("could not post review comment"), + "expected a warning about the failed post, got: {stderr}" ); - assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); - assert_eq!(envelope["counts"]["warn"], 0); - assert_eq!(envelope["counts"]["suppressed"], 1); - assert_eq!(envelope["gate"]["failing"], false); - let requests = server.received_requests().await.unwrap(); - let adjudication: Value = requests + // Both check runs still completed despite the review-comment failure. + let reqs = server.received_requests().await.unwrap(); + let patches = reqs .iter() - .find(|request| request_system_contains(request, "single finding adjudicator")) - .unwrap() - .body_json() - .unwrap(); - let payload: Value = serde_json::from_str( - adjudication["messages"].as_array().unwrap().last().unwrap()["content"] - .as_str() - .unwrap(), - ) - .unwrap(); + .filter(|r| r.method == wiremock::http::Method::PATCH) + .count(); + assert_eq!(patches, 2); + let review_posts = reqs + .iter() + .filter(|request| { + request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews" + }) + .count(); + assert_eq!( + review_posts, 1, + "ambiguous review POST must not be replayed" + ); +} + +#[tokio::test] +async fn hosted_review_post_failure_is_operational_after_the_envelope_is_persisted() { + let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + mount_static_github_pr(&server).await; + mount_successful_hosted_check_patches(&server).await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(500).set_body_string("temporary review failure")) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), true); + let output = hosted_publish_command(dir.path(), &server).assert().code(2); + + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap_or_else(|e| { + panic!( + "hosted review did not emit its envelope: {e}; stderr: {}", + String::from_utf8_lossy(&output.get_output().stderr) + ) + }); + assert_eq!(envelope["modelUsed"], "none (disabled by config)"); + assert_eq!(envelope["gate"]["failing"], false); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); + assert!(stderr.contains("required hosted review publication failed")); + + let requests = server.received_requests().await.unwrap(); assert_eq!( - payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidenceComplete"], - true + requests + .iter() + .filter(|request| request.method == wiremock::http::Method::PATCH) + .count(), + 2 ); assert_eq!( - payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["source"], - REPLACEMENT + requests + .iter() + .filter(|request| request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews") + .count(), + 3, + "each ambiguous failure must be reconciled before the bounded retry" ); } #[tokio::test] -async fn cross_file_package_existence_refutes_false_absence_claim() { - const CLAIM_PATH: &str = "ci/package-policy.yml"; - const PACKAGE_PATH: &str = "packages/atlas/manifest.yml"; - const REPLACEMENT: &str = "name: atlas_packages_base"; - +async fn hosted_marker_reconciliation_accepts_an_ambiguous_review_post_without_a_duplicate() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + mount_static_github_pr(&server).await; + mount_successful_hosted_check_patches(&server).await; + let published_body = Arc::new(Mutex::new(None)); Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("single finding adjudicator")) - .respond_with(RefuteFromReceiptAdjudicator) - .with_priority(1) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(AmbiguousReviewPostResponder { + published_body: published_body.clone(), + }) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ReconciledReviewListResponder { published_body }) .mount(&server) .await; - mock_review( - &server, - json!([{ - "path": CLAIM_PATH, - "line": 24, - "severity": "warn", - "kind": "uncertainty", - "confidence": 0.88, - "title": "The atlas package is absent", - "body": "The repository does not contain `atlas_packages_base`.", - "evidence": "package: atlas_packages_candidate", - "repositoryContext": { - "claim": "absence", - "resources": ["atlas package"], - "values": ["atlas_packages_base"], - "versions": [], - "paths": [], - "identifiers": [] - } - }]), - ) - .await; - - let directory = tempfile::tempdir().unwrap(); - let diff = directory.path().join("package-existence.diff"); - std::fs::write( - &diff, - format!( - "diff --git a/{CLAIM_PATH} b/{CLAIM_PATH}\n--- a/{CLAIM_PATH}\n+++ b/{CLAIM_PATH}\n@@ -24,1 +24,1 @@\n-package: atlas_packages_legacy\n+package: atlas_packages_candidate\ndiff --git a/{PACKAGE_PATH} b/{PACKAGE_PATH}\n--- a/{PACKAGE_PATH}\n+++ b/{PACKAGE_PATH}\n@@ -1,2 +1,2 @@\n-type: legacy package\n-name: atlas_packages_legacy\n+type: atlas package\n+{REPLACEMENT}\n" - ), - ) - .unwrap(); - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--output", "json"]) - .assert() - .success(); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), true); + hosted_publish_command(dir.path(), &server).assert().code(0); - assert_eq!(envelope["findings"], json!([])); - assert_eq!( - envelope["suppressedFindings"][0]["finding"]["path"], - CLAIM_PATH - ); + let requests = server.received_requests().await.unwrap(); assert_eq!( - envelope["suppressedFindings"][0]["finding"]["title"], - "The atlas package is absent" + requests + .iter() + .filter(|request| request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews") + .count(), + 1, + "marker reconciliation must prevent a duplicate review" ); - assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); - assert_eq!(envelope["counts"]["warn"], 0); - assert_eq!(envelope["counts"]["suppressed"], 1); - assert_eq!(envelope["gate"]["failing"], false); +} + +#[tokio::test] +async fn hosted_check_patch_failures_are_operational_and_do_not_hide_the_envelope() { + let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + mount_static_github_pr(&server).await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) + .respond_with(ResponseTemplate::new(500).set_body_string("temporary check failure")) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + let output = hosted_publish_command(dir.path(), &server).assert().code(2); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap_or_else(|e| { + panic!( + "hosted review did not emit its envelope: {e}; stderr: {}", + String::from_utf8_lossy(&output.get_output().stderr) + ) + }); + assert_eq!(envelope["modelUsed"], "none (disabled by config)"); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); + assert!(stderr.contains("required hosted check publication failed")); let requests = server.received_requests().await.unwrap(); - let adjudication: Value = requests - .iter() - .find(|request| request_system_contains(request, "single finding adjudicator")) - .unwrap() - .body_json() - .unwrap(); - let payload: Value = serde_json::from_str( - adjudication["messages"].as_array().unwrap().last().unwrap()["content"] - .as_str() - .unwrap(), - ) - .unwrap(); - assert_eq!( - payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidenceComplete"], - true - ); - assert_eq!( - payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["path"], - PACKAGE_PATH - ); assert_eq!( - payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["source"], - REPLACEMENT + requests + .iter() + .filter(|request| request.method == wiremock::http::Method::PATCH) + .count(), + 6, + "both required checks receive the complete bounded retry sequence" ); } #[tokio::test] -async fn fresh_unresolved_repository_claims_are_suppressed() { - for (name, repository, resources, state) in [ - ( - "unavailable", - false, - vec!["widget".to_string()], - "unavailable", - ), - ("exhausted", true, vec!["widget".to_string()], "exhausted"), - ] { - let server = MockServer::start().await; - mock_review( - &server, - json!([{ - "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", - "confidence": 0.99, "title": "Widget dependency is absent", - "body": "The repository does not contain the required widget dependency.", - "evidence": "exec_query(&token);", - "repositoryContext": {"claim": "absence", "resources": resources} - }]), - ) +async fn hosted_freshness_lookup_failure_is_operational_before_any_write() { + let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(GitHubFreshnessFailureResponder { + calls: Arc::new(AtomicUsize::new(0)), + }) + .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - if repository { - initialize_staged_repository(dir.path()); - } - let diff = write_diff(dir.path()); - let mut command = postil(); - command - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .arg("review"); - if repository { - command.arg("--staged"); - } else { - command.arg("--diff-file").arg(&diff); - } - command.args(["--output", "json"]); - let out = command.assert().code(0); - let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(envelope["repositorySearch"]["state"], state, "{name}"); - assert_eq!(envelope["counts"]["suppressed"], 1, "{name}"); - assert_eq!(envelope["findings"], json!([]), "{name}"); - assert_eq!(envelope["gate"]["failing"], false, "{name}"); - } + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) + .respond_with(ResponseTemplate::new(200)) + .expect(0) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200)) + .expect(0) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + let output = hosted_publish_command(dir.path(), &server).assert().code(2); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); + assert!(stderr.contains("snapshot freshness could not be verified")); +} + +#[tokio::test] +async fn hosted_stale_head_race_is_operational_and_suppresses_all_writes() { + let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(GitHubHeadRaceResponder { + calls: Arc::new(AtomicUsize::new(0)), + }) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) + .respond_with(ResponseTemplate::new(200)) + .expect(0) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200)) + .expect(0) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + let output = hosted_publish_command(dir.path(), &server).assert().code(2); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); + assert!(stderr.contains("pull request snapshot changed after review")); } #[tokio::test] -async fn full_rereview_rejects_exhausted_baseline_adjudication_capacity_before_provider_contact() { +async fn hosted_silent_review_requires_checks_but_not_a_pr_comment() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + mount_static_github_pr(&server).await; + mount_successful_hosted_check_patches(&server).await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200)) .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let findings = (0..20) - .map(|index| { - json!({ - "path": "src/auth.rs", "line": 42, "severity": "warn", "kind": "risk", - "confidence": 0.9, "title": format!("Open baseline finding {index}"), - "body": "The authorization query remains unsafe.", - "evidence": "exec_query(&token);" - }) - }) - .collect::>(); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, "findings": findings, - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 20, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,20], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join("capacity-baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--baseline") - .arg(&baseline_path) - .args(["--output", "json"]) - .assert() - .code(2); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!( - stderr.contains("exhausting its 20-candidate bound"), - "{stderr}" - ); - assert!(stderr.contains("no provider request was made"), "{stderr}"); - assert!(server.received_requests().await.unwrap().is_empty()); + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + hosted_publish_command(dir.path(), &server).assert().code(0); } #[tokio::test] -async fn oversized_adjudication_payload_preserves_findings_without_aborting_review() { +async fn hosted_silent_review_does_not_require_a_later_comment_freshness_check() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; + let calls = Arc::new(AtomicUsize::new(0)); + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(GitHubLateHeadRaceResponder { + calls: calls.clone(), + }) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "id": 42, + "full_name": "acme/api" + }))) + .mount(&server) + .await; + mount_successful_hosted_check_patches(&server).await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("single finding adjudicator")) - .respond_with(ResponseTemplate::new(500)) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200)) .expect(0) - .with_priority(1) .mount(&server) .await; - mock_review( - &server, - json!([{ - "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", - "confidence": 0.95, "title": "Fresh authorization failure", - "body": "The authorization query still accepts an untrusted token.", - "evidence": "exec_query(&token);" - }]), - ) - .await; - let directory = tempfile::tempdir().unwrap(); - let diff = write_diff(directory.path()); - let baseline_findings = (0..19) - .map(|index| { - json!({ - "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", - "confidence": 0.95, "title": format!("Prior authorization failure {index}"), - "body": format!("Historical finding {index}: {}.", "x".repeat(4_000)), - "evidence": "exec_query(&token);" - }) - }) - .collect::>(); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, "findings": baseline_findings, - "resolved": [], "counts": {"info": 0, "warn": 19, "error": 0, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,19], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = directory.path().join("oversized-baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - - let output = postil() - .current_dir(directory.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("POSTIL_DISABLE_SCORER", "1") - .args(["review", "--diff-file"]) - .arg(diff) - .arg("--baseline") - .arg(baseline_path) - .args(["--output", "json"]) - .assert() - .code(1); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - let stderr = String::from_utf8_lossy(&output.get_output().stderr); - assert_eq!(envelope["findings"].as_array().unwrap().len(), 21); + let dir = tempfile::tempdir().unwrap(); + disable_review_for_hosted_publication(dir.path(), false); + hosted_publish_command(dir.path(), &server).assert().code(0); assert_eq!( - envelope["findings"] - .as_array() - .unwrap() - .iter() - .filter(|finding| finding["path"] == ".postil/model-output") - .count(), - 1 - ); - assert_eq!(envelope["resolved"], json!([])); - assert_eq!(envelope["gate"]["failing"], true); - assert!( - stderr.contains("adjudication input exceeded its admitted bound"), - "{stderr}" + calls.load(Ordering::SeqCst), + 4, + "a no-comment result must stop after the required check delivery freshness reads" ); } #[tokio::test] -async fn scorer_cannot_suppress_an_unresolved_full_rereview_baseline() { +async fn github_unresolved_inline_line_retries_on_a_changed_line() { let server = MockServer::start().await; - mock_review(&server, json!([])).await; - let directory = tempfile::tempdir().unwrap(); - let diff = write_diff(directory.path()); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Authorization guard remains bypassed", - "body": "The authorization guard remains bypassed before query execution.", - "evidence": "exec_query(&token);" - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "model", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = directory.path().join("scorer-baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + mount_github_complete_diff(&server, 7).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.92)]))), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "t", "body": "b", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 + }))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with( + ResponseTemplate::new(422) + .set_body_string(r#"{"message":"Line could not be resolved"}"#), + ) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) + .expect(1) + .mount(&server) + .await; + let (published_review, published_comments) = published_review_responders(); + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(published_review) + .with_priority(2) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7/reviews/77/comments")) + .and(query_param("per_page", "100")) + .and(query_param("page", "1")) + .respond_with(published_comments) + .expect(1) + .mount(&server) + .await; + Mock::given(method("PUT")) + .and(path("/repos/acme/api/pulls/7/reviews/77")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .expect(1) + .mount(&server) + .await; - let output = postil() - .current_dir(directory.path()) + let dir = tempfile::tempdir().unwrap(); + let receipt_name = "publication-receipt.json"; + let receipt_path = dir.path().join(receipt_name); + let out = postil() + .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("REVIEW_SCORER_MODEL", "scorer-model") - .args(["review", "--diff-file"]) - .arg(&diff) - .arg("--baseline") - .arg(&baseline_path) - .args(["--output", "json"]) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env( + "POSTIL_DETAILS_URL", + "https://postil.dev/orgs/acme/runs/run-1", + ) + .env("POSTIL_PUBLICATION_RECEIPT_PATH", receipt_name) + .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", + ]) .assert() .code(1); - let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - assert_eq!( - envelope["findings"][0]["title"], - "Authorization guard remains bypassed" - ); - assert_eq!(envelope["findings"][0]["confidence"], 0.9); - assert_eq!(envelope["resolved"], json!([])); - assert_eq!(envelope["gate"]["failing"], true); + + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("category=unresolved-line")); + assert!(stderr.contains("recovery=placement-ladder")); let requests = server.received_requests().await.unwrap(); - assert!(requests.iter().all(|request| { - !String::from_utf8_lossy(&request.body).contains("independent second-model scorer") + let review_bodies = requests + .iter() + .filter(|request| { + request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews" + }) + .map(|request| request.body_json::().unwrap()) + .collect::>(); + assert_eq!(review_bodies.len(), 2); + assert!(review_bodies[0].get("comments").is_some()); + let fallback_comments = review_bodies[1]["comments"].as_array().unwrap(); + assert_eq!(fallback_comments.len(), 1); + assert_eq!(fallback_comments[0]["path"], "src/auth.rs"); + assert_eq!(fallback_comments[0]["line"], 41); + assert_eq!(fallback_comments[0]["side"], "RIGHT"); + assert!(fallback_comments[0].get("start_line").is_none()); + assert!( + !review_bodies[1]["body"] + .as_str() + .unwrap() + .contains("inline placement unavailable") + ); + let receipt: Value = serde_json::from_slice(&std::fs::read(&receipt_path).unwrap()).unwrap(); + assert_eq!(receipt["version"], 2); + assert_eq!(receipt["channel"], "reviewComments"); + assert_eq!(receipt["reviewId"], "77"); + assert_eq!(receipt["findings"][0]["initialOutcome"], "inline"); + assert_eq!(receipt["findings"][0]["commentId"], "500"); + assert!(!std::fs::read_dir(dir.path()).unwrap().any(|entry| { + let name = entry.unwrap().file_name(); + let name = name.to_string_lossy(); + name.starts_with(".publication-receipt.json.") && name.ends_with(".tmp") })); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&receipt_path) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o600 + ); + } } #[tokio::test] -async fn same_head_with_open_baseline_falls_back_to_full_review() { +async fn hosted_path_completes_provided_check_run_ids_without_creating_new_ones() { let server = MockServer::start().await; mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) .mount(&server) .await; Mock::given(method("GET")) @@ -12260,90 +11773,63 @@ async fn same_head_with_open_baseline_falls_back_to_full_review() { .mount(&server) .await; Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .and(path_regex(r"^/repos/acme/api/check-runs/(901|902)$")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .expect(2) .mount(&server) .await; - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [ - {"path": ".postil/change-metadata", "line": 1, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "old dependency risk", "body": "fixed now"} - ], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": "bbbbbbbb", "headSha": "aaaaaaaa", "sinceSha": null - }); let dir = tempfile::tempdir().unwrap(); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - - let out = postil() + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) - .args(["--sha", "aaaaaaaa", "--since-sha", "aaaaaaaa", "--baseline"]) - .arg(&baseline_path) .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--sha", + "aaaaaaaa", "--check-run-id", - "11", + "901", "--gate-check-run-id", - "12", + "902", "--output-json", ]) .assert() .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["resolved"], json!([])); - assert_eq!(env["findings"], json!([])); - assert_eq!( - env["suppressedFindings"][0]["finding"]["title"], - "old dependency risk" - ); - assert_eq!(env["suppressedFindings"][0]["reason"], "nonActionable"); - assert_ne!(env["modelUsed"], "none (empty diff)"); - assert_eq!(env["gate"]["failing"], false); - assert_eq!(env["sinceSha"], Value::Null); - let requests = server.received_requests().await.unwrap(); - assert!( - requests - .iter() - .any(|request| { request.url.path() == "/repos/acme/api/compare/bbbbbbbb...aaaaaaaa" }) - ); + let reqs = server.received_requests().await.unwrap(); + // The worker owns check-run creation; the CLI must not create its own. assert!( - !requests - .iter() - .any(|request| request.url.path().ends_with("/reviews")) + !reqs.iter().any(|r| r.method == wiremock::http::Method::POST + && r.url.path() == "/repos/acme/api/check-runs") ); - let llm_request = requests + // Both pre-created runs completed. + let patched: Vec<&str> = reqs .iter() - .find(|request| request.url.path() == "/chat/completions") - .unwrap(); - let body: Value = llm_request.body_json().unwrap(); - assert!( - !body["messages"][1]["content"] - .as_str() - .unwrap() - .contains("This is an INCREMENTAL review") - ); + .filter(|r| r.method == wiremock::http::Method::PATCH) + .map(|r| r.url.path()) + .collect(); + assert!(patched.contains(&"/repos/acme/api/check-runs/901")); + assert!(patched.contains(&"/repos/acme/api/check-runs/902")); } #[tokio::test] -async fn same_head_without_open_baseline_keeps_empty_diff_noop() { +async fn hosted_path_rejects_a_changed_target_before_review_or_delivery() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/bbbbbbbb...aaaaaaaa")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "bbbbbbbb"}, - "files": [] + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, + "base": {"sha": "bbbbbbbb"}, + "changed_files": 1 }))) .mount(&server) .await; @@ -12353,13 +11839,10 @@ async fn same_head_without_open_baseline_keeps_empty_diff_noop() { .expect(0) .mount(&server) .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({"id": 1}))) + .expect(0) .mount(&server) .await; @@ -12371,58 +11854,81 @@ async fn same_head_without_open_baseline_keeps_empty_diff_noop() { .env("GITHUB_TOKEN", "gh-test-token") .args([ "review", + "--publish", "--repo", "acme/api", "--pr", "7", "--sha", "aaaaaaaa", - "--since-sha", - "aaaaaaaa", - "--no-post", - "--output-json", + "--base-sha", + "cccccccc", ]) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["modelUsed"], "none (empty diff)"); - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests.len(), - 2, - "empty no-op fetched more than the immutable PR snapshot" - ); + .code(2); + + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains( + "requested review target cccccccc is no longer the pull request target bbbbbbbb" + )); } #[tokio::test] -async fn stale_incremental_baseline_falls_back_to_full_review() { +async fn github_flow_posts_review_and_completes_both_checks() { let server = MockServer::start().await; mount_github_complete_diff(&server, 7).await; - // A rebase or force-push leaves the recorded baseline off the head's - // ancestry, so the incremental compare cannot describe the change. + // LLM + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), + ) + .mount(&server) + .await; + // PR meta (default Accept) and diff (v3.diff Accept) on the same path. Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "dddddddd"}, - "files": [] + "title": "Add login", "body": "PR body", + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 }))) - .expect(1) .mount(&server) .await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(1) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + let (published_review, published_comments) = published_review_responders(); + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(published_review) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) + .and(path("/repos/acme/api/pulls/7/reviews/77/comments")) + .and(query_param("per_page", "100")) + .and(query_param("page", "1")) + .respond_with(published_comments) + .expect(2) + .mount(&server) + .await; + Mock::given(method("PUT")) + .and(path("/repos/acme/api/pulls/7/reviews/77")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) .mount(&server) .await; @@ -12432,106 +11938,182 @@ async fn stale_incremental_baseline_falls_back_to_full_review() { .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") + .env( + "POSTIL_DETAILS_URL", + "https://postil.dev/orgs/acme/runs/review-7", + ) .args([ "review", + "--publish", "--repo", "acme/api", "--pr", "7", - "--sha", - "aaaaaaaa", - "--since-sha", - "cccccccc", - "--no-post", "--output-json", ]) .assert() - .code(0); + .code(1); let env: Value = serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"], json!([])); + assert_eq!(env["headSha"], "aaaaaaaaaaaa"); + assert_eq!(env["baseSha"], "bbbbbbbb"); + + let reqs = server.received_requests().await.unwrap(); + // Two check-run creations. + let creates: Vec<_> = reqs + .iter() + .filter(|r| { + r.method == wiremock::http::Method::POST && r.url.path() == "/repos/acme/api/check-runs" + }) + .collect(); + assert_eq!(creates.len(), 2); + assert!(creates.iter().all(|request| { + request.body_json::().unwrap()["details_url"] + == "https://postil.dev/orgs/acme/runs/review-7" + })); + let names: Vec = creates + .iter() + .map(|r| { + r.body_json::().unwrap()["name"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert!(names.contains(&"postil/review".to_string())); + assert!(names.contains(&"postil/gate".to_string())); + // Both completed; gate concluded failure, advisory success. + let patches: Vec = reqs + .iter() + .filter(|r| r.method == wiremock::http::Method::PATCH) + .map(|r| r.body_json().unwrap()) + .collect(); + assert_eq!(patches.len(), 2); assert!( - !env["findings"] - .as_array() - .unwrap() + patches .iter() - .any(|finding| finding["path"] == ".postil/model-output"), - "stale baseline produced an operational failure instead of a review" - ); - assert_eq!(env["gate"]["failing"], false); - assert_ne!(env["modelUsed"], "none (empty diff)"); - assert_eq!( - env["sinceSha"], - Value::Null, - "a full review reported an incremental baseline it did not measure against" + .all(|patch| { patch["details_url"] == "https://postil.dev/orgs/acme/runs/review-7" }) ); - - let requests = server.received_requests().await.unwrap(); + let conclusions: Vec<&str> = patches + .iter() + .map(|p| p["conclusion"].as_str().unwrap()) + .collect(); + assert!(conclusions.contains(&"success")); + assert!(conclusions.contains(&"failure")); assert!( - requests + patches .iter() - .any(|request| request.url.path() == "/repos/acme/api/pulls/7/files"), - "fallback did not acquire the complete change" + .all(|patch| patch["output"].get("annotations").is_none()) ); - let llm_request = requests + let gate_patch = patches .iter() - .find(|request| request.url.path() == "/chat/completions") + .find(|patch| patch["conclusion"] == "failure") .unwrap(); - let body: Value = llm_request.body_json().unwrap(); + assert_eq!(gate_patch["output"]["title"], "Merge gate failed"); + assert_eq!( + gate_patch["output"]["summary"], + "Merge gate failed: 1 finding blocks under the configured policy (failOn: error).\n\n- `src/auth.rs:41` Unsanitized input reaches query\n" + ); + // Inline review posted with the finding at the cited line. + let review = reqs + .iter() + .find(|r| r.url.path() == "/repos/acme/api/pulls/7/reviews") + .expect("review posted"); + let body: Value = review.body_json().unwrap(); + assert_eq!(body["comments"][0]["path"], "src/auth.rs"); + assert_eq!(body["comments"][0]["line"], 41); + let initial_summary = body["body"].as_str().unwrap(); + assert!(!initial_summary.contains("posted inline")); + let update = reqs + .iter() + .find(|r| { + r.method == wiremock::http::Method::PUT + && r.url.path() == "/repos/acme/api/pulls/7/reviews/77" + }) + .expect("review summary updated"); + let update_body: Value = update.body_json().unwrap(); + let summary = update_body["body"].as_str().unwrap(); + assert!(summary.starts_with(&format!( + "{} **1 blocking finding open**\n", + postil_cli::forge::icon_md("error"), + ))); + assert!(summary.contains("1 finding posted inline")); + assert!(!summary.contains("Unsanitized input reaches query")); + assert!(!summary.contains("`src/auth.rs:41`")); + assert!(!summary.contains("Review metadata")); + assert!(!summary.contains("headsha")); + assert!(!summary.contains("Tokens")); assert!( - !body["messages"][1]["content"] - .as_str() - .unwrap() - .contains("This is an INCREMENTAL review"), - "fallback reviewed the change as incremental" + summary.contains("[Review details](https://postil.dev/orgs/acme/runs/review-7)") ); -} - -#[tokio::test] -async fn incremental_touched_carried_error_falls_back_to_full_review() { - let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "cccccccc"}, - "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] - }))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Prior authorization blocker", - "body": "The authorization path remains unsafe." - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null - }); - let dir = tempfile::tempdir().unwrap(); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + std::fs::write( + dir.path().join(".postil.yaml"), + "review:\n findingPresentation: checkAnnotations\n", + ) + .unwrap(); + let annotation_receipt_path = dir.path().join("annotation-receipt.json"); + let request_count_before_annotations = reqs.len(); + postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env("POSTIL_PUBLICATION_RECEIPT_PATH", &annotation_receipt_path) + .args([ + "review", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", + ]) + .assert() + .code(1); + let annotation_receipt: Value = + serde_json::from_slice(&std::fs::read(&annotation_receipt_path).unwrap()).unwrap(); + assert_eq!(annotation_receipt["version"], 2); + assert_eq!(annotation_receipt["channel"], "checkAnnotations"); + assert!(annotation_receipt.get("reviewId").is_none()); + assert_eq!( + annotation_receipt["findings"][0]["initialOutcome"], + "checkAnnotation" + ); + assert!(annotation_receipt["findings"][0].get("commentId").is_none()); + let annotation_requests = server.received_requests().await.unwrap(); + let annotation_requests = &annotation_requests[request_count_before_annotations..]; + assert!(!annotation_requests.iter().any(|request| { + request.method == wiremock::http::Method::POST + && request.url.path() == "/repos/acme/api/pulls/7/reviews" + })); + let annotation_patches = annotation_requests + .iter() + .filter(|request| request.method == wiremock::http::Method::PATCH) + .map(|request| request.body_json::().unwrap()) + .collect::>(); + assert_eq!(annotation_patches.len(), 2); + let advisory_annotation = annotation_patches + .iter() + .find(|patch| patch["output"].get("annotations").is_some()) + .expect("advisory check annotation"); + assert_eq!( + advisory_annotation["output"]["annotations"][0]["path"], + "src/auth.rs" + ); + assert_eq!( + advisory_annotation["output"]["annotations"][0]["start_line"], + 41 + ); + assert_eq!( + annotation_patches + .iter() + .filter(|patch| patch["output"].get("annotations").is_some()) + .count(), + 1 + ); + std::fs::remove_file(dir.path().join(".postil.yaml")).unwrap(); + let request_count_before_deferred_gate = server.received_requests().await.unwrap().len(); let out = postil() .current_dir(dir.path()) @@ -12540,205 +12122,167 @@ async fn incremental_touched_carried_error_falls_back_to_full_review() { .env("GITHUB_TOKEN", "gh-test-token") .args([ "review", + "--publish", + "--defer-gate-check", "--repo", "acme/api", "--pr", - "7", - "--sha", - "aaaaaaaa", - "--since-sha", - "cccccccc", - "--baseline", + "7", + "--output-json", ]) - .arg(&baseline_path) - .args(["--no-post", "--output-json"]) .assert() .code(1); - let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert_eq!(env["sinceSha"], Value::Null); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["gate"]["failing"], true); - let requests = server.received_requests().await.unwrap(); - assert!( - requests - .iter() - .any(|request| request.url.path() == "/repos/acme/api/pulls/7/files"), - "a touched carried Error must fetch the complete change" - ); - let source_request = requests + let reqs = server.received_requests().await.unwrap(); + let conclusions: Vec = reqs[request_count_before_deferred_gate..] .iter() - .find(|request| request.url.path() == "/chat/completions") - .unwrap(); - let body: Value = source_request.body_json().unwrap(); - assert!( - !body["messages"][1]["content"] - .as_str() - .unwrap() - .contains("This is an INCREMENTAL review"), - "full fallback must happen before model review and adjudication" - ); + .filter(|request| request.method == wiremock::http::Method::PATCH) + .map(|request| { + request.body_json::().unwrap()["conclusion"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert_eq!(conclusions, vec!["success"]); } #[tokio::test] -async fn local_incremental_diff_file_with_touched_carried_error_fails_closed_actionably() { +async fn github_push_after_acquisition_suppresses_all_stale_publication() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) + .respond_with(GitHubHeadRaceResponder { + calls: Arc::new(AtomicUsize::new(0)), + }) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200)) .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let diff = write_diff(dir.path()); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Prior authorization blocker", - "body": "The authorization path remains unsafe." - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - - let out = postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .args(["review", "--diff-file"]) - .arg(&diff) - .args(["--since-sha", "previous", "--baseline"]) - .arg(&baseline_path) - .args(["--output", "json"]) - .assert() - .code(1); - let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - let stderr = String::from_utf8_lossy(&out.get_output().stderr); - assert!(stderr.contains("cannot reconstruct the complete comparison")); - let incomplete = env["findings"] - .as_array() - .unwrap() - .iter() - .find(|finding| finding["path"] == ".postil/model-output") - .unwrap(); - assert_eq!(incomplete["title"], "Review incomplete"); - assert!( - incomplete["body"] - .as_str() - .unwrap() - .contains("complete comparison") - ); - let carried = env["findings"] - .as_array() - .unwrap() - .iter() - .find(|finding| finding["title"] == "Prior authorization blocker") - .expect("the prior blocker must remain in the next baseline"); - assert!( - carried["body"] - .as_str() - .unwrap() - .starts_with("[carried from previous review]") - ); - assert_eq!(env["resolved"], json!([])); - assert_eq!(env["gate"]["failing"], true); -} - -#[tokio::test] -async fn local_incremental_staged_review_with_touched_carried_error_fails_closed() { - let server = MockServer::start().await; Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200)) .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - initialize_staged_repository(dir.path()); - std::fs::write(dir.path().join(".postil.yaml"), "enabled: true\n").unwrap(); - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "Prior authorization blocker", - "body": "The authorization path remains unsafe." - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": null, "headSha": null, "sinceSha": null - }); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - let out = postil() + let dir = tempfile::tempdir().unwrap(); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") .args([ "review", - "--staged", - "--since-sha", - "previous", - "--baseline", + "--publish", + "--repo", + "acme/api", + "--pr", + "7", + "--output-json", ]) - .arg(&baseline_path) - .args(["--output", "json"]) .assert() - .code(1); - let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); - assert!( - env["findings"] - .as_array() - .unwrap() - .iter() - .any(|finding| finding["path"] == ".postil/model-output" - && finding["title"] == "Review incomplete") - ); - assert!(env["findings"].as_array().unwrap().iter().any(|finding| { - finding["title"] == "Prior authorization blocker" - && finding["body"] - .as_str() - .is_some_and(|body| body.starts_with("[carried from previous review]")) - })); - assert_eq!(env["resolved"], json!([])); - assert_eq!(env["gate"]["failing"], true); + .code(1) + .stderr(predicates::str::contains( + "publication skipped because the pull request snapshot changed", + )); } -#[tokio::test] -async fn incremental_forge_outage_still_fails_the_review() { +// An LLM response with a caller-provided summary and findings (used for +// content-policy scenarios where the finding is not the standard auth one). +fn llm_with_summary(summary: &str, findings: Value) -> Value { + let findings = explicit_repository_context(findings); + json!({ + "choices": [{"finish_reason": "stop", "message": {"content": json!({ + "summary": summary, + "findings": findings + }).to_string()}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 50} + }) +} + +// Shared GitHub remote setup for the content-policy PR-description tests. The PR +// body carries prose the model flags; content policy is active by default. +async fn content_policy_pr_server(llm: Value) -> MockServer { let server = MockServer::start().await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) - .respond_with(ResponseTemplate::new(503)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "bbbbbbbb"}, - "files": [] - }))) + mount_github_complete_diff(&server, 7).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm)) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7/files")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) - .expect(0) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) .mount(&server) .await; Mock::given(method("GET")) .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, + "title": "Add login", + "body": "This change updates review retention behavior.", "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 + "head": {"sha": "aaaaaaaaaaaa"}, "base": {"sha": "bbbbbbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + server +} + +#[tokio::test] +async fn content_policy_pr_body_finding_survives_grounding() { + // A content-policy finding against the PR description grounds on the reserved + // `.postil/pr-description` path instead of being dropped as ungrounded (which + // would have spuriously fail-closed a run whose only finding was here). + let cp_finding = json!([{ + "path": ".postil/pr-description", "line": 2, "severity": "warn", + "kind": "contentPolicy", "confidence": 0.9, + "title": "Retention scope missing from PR description", + "body": "State the supported retention scope in the description.", + "evidence": "This change updates review retention behavior." + }]); + let server = content_policy_pr_server(llm_with_summary( + "PR description omits the supported retention scope.", + cp_finding, + )) + .await; let dir = tempfile::tempdir().unwrap(); let out = postil() @@ -12748,27 +12292,86 @@ async fn incremental_forge_outage_still_fails_the_review() { .env("GITHUB_TOKEN", "gh-test-token") .args([ "review", + "--publish", "--repo", "acme/api", "--pr", "7", - "--sha", - "aaaaaaaa", - "--since-sha", - "cccccccc", - "--no-post", "--output-json", ]) .assert() - .code(1); + .code(0); // warn severity: kept, but gate passes at default failOn=error let env: Value = serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/provider"); - assert_eq!(env["gate"]["failing"], true); + // The finding survived grounding: it is NOT a fail-closed model-output error. + assert_eq!(env["findings"][0]["path"], ".postil/pr-description"); + assert_eq!(env["findings"][0]["kind"], "contentPolicy"); + assert_eq!(env["counts"]["ungrounded"], 0); + assert_eq!(env["gate"]["failing"], false); + + // The model was shown the numbered PR-description block. + let reqs = server.received_requests().await.unwrap(); + let llm = reqs + .iter() + .find(|r| r.url.path() == "/chat/completions") + .unwrap(); + let sent: Value = llm.body_json().unwrap(); + let user_msg = sent["messages"][1]["content"].as_str().unwrap(); + assert!(user_msg.contains(".postil/pr-description")); + assert!(user_msg.contains(" 1 Add login")); + assert!(user_msg.contains(" 2 This change updates review retention behavior.")); + + // The reserved-path finding has no real line, so its bounded detail appears + // in the PR summary instead of an inline comment. + let review = reqs + .iter() + .find(|r| r.url.path() == "/repos/acme/api/pulls/7/reviews") + .expect("review posted"); + let body: Value = review.body_json().unwrap(); + assert_eq!( + body["comments"].as_array().map(|a| a.len()).unwrap_or(0), + 0, + "reserved-path finding was posted as an inline comment" + ); + let summary = body["body"].as_str().unwrap(); + assert!(summary.contains(&format!( + "{} **1 advisory finding open**", + postil_cli::forge::icon_md("info") + ))); + assert!(summary.contains("1 finding in review details")); + assert!(summary.contains("Retention scope missing from PR description")); + assert!(summary.contains("in pull request description")); + assert!(summary.contains("State the supported retention scope in the description.")); } #[tokio::test] -async fn carried_only_incremental_run_updates_checks_without_posting_review() { +async fn content_policy_clean_run_does_not_fail_close() { + // With default content policy and no violations, the run stays clean: the numbered + // PR-description block must not induce a spurious ungrounded/fail-closed run. + let server = content_policy_pr_server(llm_with_summary("", json!([]))).await; + + let dir = tempfile::tempdir().unwrap(); + let out = postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .args(["review", "--repo", "acme/api", "--pr", "7", "--output-json"]) + .assert() + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["silent"], true); + assert_eq!(env["gate"]["failing"], false); + assert_eq!( + env["findings"].as_array().map(|a| a.len()).unwrap_or(0), + 0, + "a clean content-policy run produced a spurious finding" + ); +} + +#[tokio::test] +async fn github_clean_pr_stays_silent_but_completes_checks() { let server = MockServer::start().await; mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) @@ -12777,11 +12380,9 @@ async fn carried_only_incremental_run_updates_checks_without_posting_review() { .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "cccccccc"}, - "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] - }))) + .and(path("/repos/acme/api/pulls/7")) + .and(header("Accept", "application/vnd.github.v3.diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) .mount(&server) .await; Mock::given(method("GET")) @@ -12793,179 +12394,182 @@ async fn carried_only_incremental_run_updates_checks_without_posting_review() { }))) .mount(&server) .await; + Mock::given(method("POST")) + .and(path("/repos/acme/api/check-runs")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({"id": 11}))) + .mount(&server) + .await; Mock::given(method("PATCH")) .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) .mount(&server) .await; - let baseline = json!({ - "version": 1, "summary": "", "silent": false, - "findings": [ - {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "still broken", "body": "not addressed"} - ], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], - "gate": {"failOn": "error", "failing": true}, - "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null - }); let dir = tempfile::tempdir().unwrap(); - let baseline_path = dir.path().join("baseline.json"); - std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - - let out = postil() + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) - .args(["--sha", "aaaaaaaa", "--since-sha", "cccccccc", "--baseline"]) - .arg(&baseline_path) - .args([ - "--check-run-id", - "11", - "--gate-check-run-id", - "12", - "--output-json", - ]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert!( - env["findings"][0]["body"] - .as_str() - .unwrap() - .starts_with("[carried from previous review]") - ); - assert_eq!(env["gate"]["failing"], true); + .code(0); - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .count(), - 2 - ); + let reqs = server.received_requests().await.unwrap(); + // Silence is a feature: no review comment posted on a clean PR. + assert!(!reqs.iter().any(|r| r.url.path().ends_with("/reviews"))); + // But both checks completed successfully. + let conclusions: Vec = reqs + .iter() + .filter(|r| r.method == wiremock::http::Method::PATCH) + .map(|r| { + r.body_json::().unwrap()["conclusion"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert_eq!(conclusions, vec!["success", "success"]); + let check_requests: Vec = reqs + .iter() + .filter(|request| { + request.url.path().starts_with("/repos/acme/api/check-runs") + && matches!( + request.method, + wiremock::http::Method::POST | wiremock::http::Method::PATCH + ) + }) + .map(|request| request.body_json().unwrap()) + .collect(); assert!( - !requests + check_requests .iter() - .any(|request| request.url.path().ends_with("/reviews")) + .all(|request| request.get("details_url").is_none()) + ); + let gate_patch = check_requests + .iter() + .find(|request| { + request["output"]["summary"] + .as_str() + .is_some_and(|summary| summary.starts_with("Merge gate passed:")) + }) + .expect("gate completion payload"); + assert_eq!(gate_patch["output"]["title"], "Merge gate passed"); + assert_eq!( + gate_patch["output"]["summary"], + "Merge gate passed: no findings block under the configured policy (failOn: error).\n" ); -} -#[tokio::test] -async fn identical_fresh_finding_set_does_not_post_duplicate_review() { - let server = MockServer::start().await; - mount_github_complete_diff(&server, 7).await; + // The explicit onClean mode uses the same unified summary as finding-bearing + // reviews instead of falling back to the former one-line clean message. Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "merge_base_commit": {"sha": "cccccccc"}, - "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "t", "body": null, - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) + .and(path("/repos/acme/api/pulls/7/reviews")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) .mount(&server) .await; - Mock::given(method("PATCH")) - .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + std::fs::write( + dir.path().join(".postil.yaml"), + "review:\n onClean: comment\n", + ) + .unwrap(); + postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("GITHUB_API_URL", server.uri()) + .env("GITHUB_TOKEN", "gh-test-token") + .env( + "POSTIL_DETAILS_URL", + "https://postil.dev/orgs/acme/runs/clean-7", + ) + .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .assert() + .code(0); + let reqs = server.received_requests().await.unwrap(); + let clean_review = reqs + .iter() + .rev() + .find(|request| request.url.path().ends_with("/reviews")) + .expect("onClean review posted"); + let clean_body: Value = clean_review.body_json().unwrap(); + let clean_summary = clean_body["body"].as_str().unwrap(); + assert!(clean_summary.starts_with("Postil reviewed this change")); + assert!(clean_summary.contains("Postil reviewed this change and found nothing")); + assert!(!clean_summary.contains("Review metadata")); + assert!( + clean_summary + .contains("[Review details](https://postil.dev/orgs/acme/runs/clean-7)") + ); +} + +#[tokio::test] +async fn incremental_review_resolves_and_carries_baseline_findings() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; + let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); // touches src/auth.rs:40-47 + + // Baseline: one finding inside the touched range (resolved), one elsewhere (carried). let baseline = json!({ "version": 1, "summary": "", "silent": false, - "findings": [{ - "path": "src/auth.rs", "line": 41, "severity": "error", "kind": "risk", - "confidence": 0.95, "title": "Unsanitized input reaches query", - "body": "user_input flows into exec_query without sanitization." - }], - "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, - "confidenceBuckets": [0,0,0,0,1], + "findings": [ + {"path": "src/auth.rs", "line": 41, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "old auth bug", "body": "fixed now"}, + {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "still broken", "body": "not addressed"} + ], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 2, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,2], "gate": {"failOn": "error", "failing": true}, "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, - "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null + "baseSha": null, "headSha": null, "sinceSha": null }); - let dir = tempfile::tempdir().unwrap(); let baseline_path = dir.path().join("baseline.json"); std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) - .args(["--sha", "aaaaaaaa", "--since-sha", "cccccccc", "--baseline"]) + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--since-sha", "abc123", "--baseline"]) .arg(&baseline_path) - .args([ - "--check-run-id", - "11", - "--gate-check-run-id", - "12", - "--output-json", - ]) + .arg("--output-json") .assert() - .code(1); - - let requests = server.received_requests().await.unwrap(); - assert_eq!( - requests - .iter() - .filter(|request| request.method == wiremock::http::Method::PATCH) - .count(), - 2 - ); + .code(1); // carried error finding keeps the gate failing + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["resolved"][0]["title"], "old auth bug"); + assert_eq!(env["findings"][0]["title"], "still broken"); assert!( - !requests - .iter() - .any(|request| request.url.path().ends_with("/reviews")) + env["findings"][0]["body"] + .as_str() + .unwrap() + .starts_with("[carried") ); + assert_eq!(env["gate"]["failing"], true); + assert_eq!(env["sinceSha"], "abc123"); } #[tokio::test] -async fn disabled_review_does_not_carry_baseline_findings() { - // M2 regression: a repo with `enabled: false` plus a supplied baseline must - // not have baseline Errors reconciled and carried into the gate. With review - // off there is no fresh model run, so reconcile is skipped entirely and the - // gate stays clean. No LLM call should be made. +async fn incremental_unavailable_repository_receipt_carries_baseline_claim() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(0) // disabled: the model is never called - .mount(&server) - .await; + mock_review(&server, json!([])).await; let dir = tempfile::tempdir().unwrap(); let diff = write_diff(dir.path()); - std::fs::write(dir.path().join(".postil.yaml"), "enabled: false\n").unwrap(); - let baseline = json!({ "version": 1, "summary": "", "silent": false, - "findings": [ - {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", - "confidence": 0.9, "title": "still broken", "body": "not addressed"} - ], + "findings": [{ + "path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Widget dependency is absent", + "body": "The repository does not contain widget version 2.0.", + "repositoryContext": {"claim": "absence", "resources": ["widget"], "versions": ["2.0"]} + }], "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, "confidenceBuckets": [0,0,0,0,1], "gate": {"failOn": "error", "failing": true}, @@ -12980,585 +12584,691 @@ async fn disabled_review_does_not_carry_baseline_findings() { .env("POSTIL_API_BASE", server.uri()) .args(["review", "--diff-file"]) .arg(&diff) - .args(["--since-sha", "abc123", "--baseline"]) + .args(["--since-sha", "previous", "--baseline"]) .arg(&baseline_path) - .arg("--output-json") + .args(["--output", "json"]) .assert() - .code(0); // disabled + no carry => gate passes - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["gate"]["failing"], false); - assert_eq!( - env["findings"].as_array().map(|a| a.len()).unwrap_or(0), - 0, - "disabled review carried a baseline finding into the gate" + .code(1); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["resolved"], json!([])); + assert!( + envelope["findings"][0]["body"] + .as_str() + .unwrap() + .starts_with("[carried from previous review]") ); - assert_eq!(env["silent"], true); + assert_eq!(envelope["repositorySearch"]["state"], "unavailable"); } #[tokio::test] -async fn bitbucket_flow_posts_comment_and_sets_statuses() { +async fn full_rereview_preserves_unresolved_baseline_for_unavailable_and_exhausted_receipts() { + for (name, repository, resources, state) in [ + ( + "unavailable", + false, + vec!["widget".to_string()], + "unavailable", + ), + ("exhausted", true, vec!["widget".to_string()], "exhausted"), + ] { + let server = MockServer::start().await; + mock_review(&server, json!([])).await; + let dir = tempfile::tempdir().unwrap(); + if repository { + initialize_staged_repository(dir.path()); + } + let diff = write_diff(dir.path()); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [ + { + "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Widget dependency is absent", + "body": "The repository does not contain the required widget dependency.", + "evidence": "exec_query(&token);", + "repositoryContext": {"claim": "absence", "resources": resources} + }, + { + "path": "src/other.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Unchanged widget dependency is absent", + "body": "The unchanged component requires the missing widget dependency.", + "evidence": "use widget::Client;", + "repositoryContext": {"claim": "absence", "resources": ["widget"]} + } + ], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 2, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,2], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = dir.path().join(format!("{name}-baseline.json")); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let mut command = postil(); + command + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_DISABLE_SCORER", "1") + .arg("review"); + if repository { + command.arg("--staged"); + } else { + command.arg("--diff-file").arg(&diff); + } + command + .arg("--baseline") + .arg(&baseline_path) + .args(["--output", "json"]); + let out = command.assert().code(1); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["repositorySearch"]["state"], state, "{name}"); + assert_eq!(envelope["resolved"], json!([]), "{name}"); + assert_eq!(envelope["counts"]["suppressed"], 0, "{name}"); + let titles = envelope["findings"] + .as_array() + .unwrap() + .iter() + .map(|finding| finding["title"].as_str().unwrap()) + .collect::>(); + assert_eq!( + titles, + std::collections::BTreeSet::from([ + "Unchanged widget dependency is absent", + "Widget dependency is absent", + ]), + "{name}" + ); + assert_eq!(envelope["gate"]["failing"], true, "{name}"); + } +} + +#[tokio::test] +async fn full_rereview_resolves_false_absence_from_unchanged_repository_source() { let server = MockServer::start().await; - mount_bitbucket_complete_diff(&server).await; + mock_review(&server, json!([])).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), - ) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repositories/acme/api/pullrequests/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", - "summary": {"raw": "PR body"}, - "state": "OPEN", "source": {"commit": {"hash": "aaaaaaaa"}}, - "destination": {"commit": {"hash": "bbbbbbbb"}} - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repositories/acme/api/pullrequests/7/diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path_regex( - r"^/repositories/acme/api/commit/.+/statuses/build$", - )) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repositories/acme/api/pullrequests/7/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) + .and(body_string_contains("single finding adjudicator")) + .respond_with(RepositoryEvidenceAdjudicator) + .with_priority(1) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - let out = postil() - .current_dir(dir.path()) + let directory = tempfile::tempdir().unwrap(); + initialize_staged_repository_with_unchanged_caller(directory.path()); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 1, "severity": "error", "kind": "risk", + "confidence": 0.95, "title": "Legacy API has no callers", + "body": "The repository has no caller for `legacy_api`; remove it or restore its caller.", + "evidence": "fn login() {}", + "repositoryContext": {"claim": "absence", "identifiers": ["legacy_api"]} + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = directory.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("BITBUCKET_API_URL", server.uri()) - .env("BITBUCKET_TOKEN", "bb-test-token") - .env_remove("BITBUCKET_USER") - .args([ - "review", - "--publish", - "--forge", - "bitbucket", - "--repo", - "acme/api", - "--pr", - "7", - "--output-json", - ]) + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--staged", "--baseline"]) + .arg(&baseline_path) + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["headSha"], "aaaaaaaa"); + .code(0); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - let reqs = server.received_requests().await.unwrap(); - let statuses: Vec = reqs - .iter() - .filter(|r| r.url.path().ends_with("/statuses/build")) - .map(|r| r.body_json().unwrap()) - .collect(); - // Two on start (INPROGRESS) + two on complete. - assert_eq!(statuses.len(), 4); - let gate_final = statuses - .iter() - .rev() - .find(|s| s["key"] == "postil/gate") - .unwrap(); - assert_eq!(gate_final["state"], "FAILED"); - let review_final = statuses - .iter() - .rev() - .find(|s| s["key"] == "postil/review") - .unwrap(); - assert_eq!(review_final["state"], "SUCCESSFUL"); - // Inline comment anchored to the cited path/line. - let comment = reqs - .iter() - .find(|r| { - r.method == wiremock::http::Method::POST - && r.url.path().ends_with("/comments") - && r.body_json::() - .map(|b| b.get("inline").is_some()) - .unwrap_or(false) - }) - .expect("inline comment posted"); - let body: Value = comment.body_json().unwrap(); - assert_eq!(body["inline"]["path"], "src/auth.rs"); - assert_eq!(body["inline"]["to"], 41); + assert_eq!(envelope["findings"], json!([])); + assert_eq!( + envelope["resolved"][0]["title"], + "Legacy API has no callers" + ); + assert_eq!(envelope["repositorySearch"]["state"], "complete"); + assert_eq!(envelope["gate"]["failing"], false); + assert!(envelope["repositorySearch"].get("evidence").is_none()); + assert!(envelope.get("repositoryEvidence").is_none()); } #[tokio::test] -async fn bitbucket_incremental_is_disabled_without_verification_gate() { +async fn same_hunk_groups_replacement_refutes_false_deletion_claim() { + const PATH: &str = "ansible/group_vars/atlas.yml"; + const OLD_LINE: &str = "groups: [atlas_packages_base]"; + const REPLACEMENT: &str = "groups: [atlas_packages_base, atlas_packages_test]"; + let server = MockServer::start().await; - mount_bitbucket_merge_base(&server).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .expect(0) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repositories/acme/api/pullrequests/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", - "summary": {"raw": "PR body"}, - "state": "OPEN", "source": {"commit": {"hash": "aaaaaaaa"}}, - "destination": {"commit": {"hash": "bbbbbbbb"}} - }))) + .and(body_string_contains("single finding adjudicator")) + .respond_with(RefuteFromReceiptAdjudicator) + .with_priority(1) .mount(&server) .await; + mock_review( + &server, + json!([{ + "path": PATH, + "line": 120, + "severity": "warn", + "kind": "uncertainty", + "confidence": 0.91, + "title": "The groups entry was deleted", + "body": "The `groups` entry was deleted without its replacement `atlas_packages_test`.", + "evidence": "policy: atlas", + "repositoryContext": { + "claim": "mismatch", + "resources": ["groups"], + "values": ["atlas_packages_test"], + "versions": [], + "paths": [], + "identifiers": [] + } + }]), + ) + .await; - let dir = tempfile::tempdir().unwrap(); - let out = postil() - .current_dir(dir.path()) + let directory = tempfile::tempdir().unwrap(); + let diff = directory.path().join("groups-replacement.diff"); + std::fs::write( + &diff, + format!( + "diff --git a/{PATH} b/{PATH}\n--- a/{PATH}\n+++ b/{PATH}\n@@ -120,4 +120,4 @@\n-policy: legacy\n+policy: atlas\n stable: true\n owner: atlas\n-{OLD_LINE}\n+{REPLACEMENT}\n" + ), + ) + .unwrap(); + + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("BITBUCKET_API_URL", server.uri()) - .env("BITBUCKET_TOKEN", "bb-test-token") - .env_remove("BITBUCKET_USER") - .args([ - "review", - "--forge", - "bitbucket", - "--repo", - "acme/api", - "--pr", - "7", - "--sha", - "aaaaaaaa", - "--since-sha", - "cccccccc", - "--no-post", - "--output-json", - ]) + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["findings"][0]["path"], ".postil/model-output"); - assert!( - !env["findings"][0]["body"] - .as_str() - .unwrap() - .contains("POSTIL_ENABLE_BITBUCKET_INCREMENTAL") + .success(); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + + assert_eq!(envelope["findings"], json!([])); + assert_eq!(envelope["suppressedFindings"][0]["finding"]["path"], PATH); + assert_eq!( + envelope["suppressedFindings"][0]["finding"]["title"], + "The groups entry was deleted" ); + assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); + assert_eq!(envelope["counts"]["warn"], 0); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!(envelope["gate"]["failing"], false); - let reqs = server.received_requests().await.unwrap(); - assert!( - !reqs - .iter() - .any(|request| request.url.path().contains("/diff/")) + let requests = server.received_requests().await.unwrap(); + let adjudication: Value = requests + .iter() + .find(|request| request_system_contains(request, "single finding adjudicator")) + .unwrap() + .body_json() + .unwrap(); + let payload: Value = serde_json::from_str( + adjudication["messages"].as_array().unwrap().last().unwrap()["content"] + .as_str() + .unwrap(), + ) + .unwrap(); + assert_eq!( + payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidenceComplete"], + true + ); + assert_eq!( + payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["source"], + REPLACEMENT ); } #[tokio::test] -async fn bitbucket_incremental_fetches_documented_compare_when_enabled() { +async fn cross_file_package_existence_refutes_false_absence_claim() { + const CLAIM_PATH: &str = "ci/package-policy.yml"; + const PACKAGE_PATH: &str = "packages/atlas/manifest.yml"; + const REPLACEMENT: &str = "name: atlas_packages_base"; + let server = MockServer::start().await; - mount_bitbucket_complete_diff(&server).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repositories/acme/api/pullrequests/7")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", - "summary": {"raw": "PR body"}, - "state": "OPEN", "source": {"commit": {"hash": "aaaaaaaa"}}, - "destination": {"commit": {"hash": "bbbbbbbb"}} - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repositories/acme/api/diffstat/aaaaaaaa..cccccccc")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "values": [{ - "status": "modified", - "old": {"path": "src/auth.rs"}, - "new": {"path": "src/auth.rs"} - }] - }))) - .expect(1) + .and(body_string_contains("single finding adjudicator")) + .respond_with(RefuteFromReceiptAdjudicator) + .with_priority(1) .mount(&server) .await; + mock_review( + &server, + json!([{ + "path": CLAIM_PATH, + "line": 24, + "severity": "warn", + "kind": "uncertainty", + "confidence": 0.88, + "title": "The atlas package is absent", + "body": "The repository does not contain `atlas_packages_base`.", + "evidence": "package: atlas_packages_candidate", + "repositoryContext": { + "claim": "absence", + "resources": ["atlas package"], + "values": ["atlas_packages_base"], + "versions": [], + "paths": [], + "identifiers": [] + } + }]), + ) + .await; + + let directory = tempfile::tempdir().unwrap(); + let diff = directory.path().join("package-existence.diff"); + std::fs::write( + &diff, + format!( + "diff --git a/{CLAIM_PATH} b/{CLAIM_PATH}\n--- a/{CLAIM_PATH}\n+++ b/{CLAIM_PATH}\n@@ -24,1 +24,1 @@\n-package: atlas_packages_legacy\n+package: atlas_packages_candidate\ndiff --git a/{PACKAGE_PATH} b/{PACKAGE_PATH}\n--- a/{PACKAGE_PATH}\n+++ b/{PACKAGE_PATH}\n@@ -1,2 +1,2 @@\n-type: legacy package\n-name: atlas_packages_legacy\n+type: atlas package\n+{REPLACEMENT}\n" + ), + ) + .unwrap(); - let dir = tempfile::tempdir().unwrap(); - let out = postil() - .current_dir(dir.path()) + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("BITBUCKET_API_URL", server.uri()) - .env("BITBUCKET_TOKEN", "bb-test-token") - .env("POSTIL_ENABLE_BITBUCKET_INCREMENTAL", "1") - .env("REVIEW_MODEL", "test-model") - .env_remove("BITBUCKET_USER") - .args([ - "review", - "--forge", - "bitbucket", - "--repo", - "acme/api", - "--pr", - "7", - "--sha", - "aaaaaaaa", - "--since-sha", - "cccccccc", - "--no-post", - "--output-json", - ]) + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--output", "json"]) .assert() - .code(0); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - assert_eq!(env["sinceSha"], "cccccccc"); - assert_eq!(env["modelUsed"], "test-model"); + .success(); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); - let reqs = server.received_requests().await.unwrap(); - let llm_request = reqs + assert_eq!(envelope["findings"], json!([])); + assert_eq!( + envelope["suppressedFindings"][0]["finding"]["path"], + CLAIM_PATH + ); + assert_eq!( + envelope["suppressedFindings"][0]["finding"]["title"], + "The atlas package is absent" + ); + assert_eq!(envelope["suppressedFindings"][0]["reason"], "nonActionable"); + assert_eq!(envelope["counts"]["warn"], 0); + assert_eq!(envelope["counts"]["suppressed"], 1); + assert_eq!(envelope["gate"]["failing"], false); + + let requests = server.received_requests().await.unwrap(); + let adjudication: Value = requests .iter() - .find(|request| request.url.path() == "/chat/completions") + .find(|request| request_system_contains(request, "single finding adjudicator")) + .unwrap() + .body_json() .unwrap(); - let body: Value = llm_request.body_json().unwrap(); - assert!( - body["messages"][1]["content"] + let payload: Value = serde_json::from_str( + adjudication["messages"].as_array().unwrap().last().unwrap()["content"] .as_str() - .unwrap() - .contains("This is an INCREMENTAL review") + .unwrap(), + ) + .unwrap(); + assert_eq!( + payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidenceComplete"], + true + ); + assert_eq!( + payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["path"], + PACKAGE_PATH + ); + assert_eq!( + payload["diffCorpusReceipt"]["candidateCitations"][0]["refutationEvidence"]["source"], + REPLACEMENT ); } #[tokio::test] -async fn azure_flow_reconstructs_diff_and_posts_thread() { +async fn fresh_unresolved_repository_claims_are_suppressed() { + for (name, repository, resources, state) in [ + ( + "unavailable", + false, + vec!["widget".to_string()], + "unavailable", + ), + ("exhausted", true, vec!["widget".to_string()], "exhausted"), + ] { + let server = MockServer::start().await; + mock_review( + &server, + json!([{ + "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", + "confidence": 0.99, "title": "Widget dependency is absent", + "body": "The repository does not contain the required widget dependency.", + "evidence": "exec_query(&token);", + "repositoryContext": {"claim": "absence", "resources": resources} + }]), + ) + .await; + let dir = tempfile::tempdir().unwrap(); + if repository { + initialize_staged_repository(dir.path()); + } + let diff = write_diff(dir.path()); + let mut command = postil(); + command + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("POSTIL_DISABLE_SCORER", "1") + .arg("review"); + if repository { + command.arg("--staged"); + } else { + command.arg("--diff-file").arg(&diff); + } + command.args(["--output", "json"]); + let out = command.assert().code(0); + let envelope: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(envelope["repositorySearch"]["state"], state, "{name}"); + assert_eq!(envelope["counts"]["suppressed"], 1, "{name}"); + assert_eq!(envelope["findings"], json!([]), "{name}"); + assert_eq!(envelope["gate"]["failing"], false, "{name}"); + } +} + +#[tokio::test] +async fn full_rereview_rejects_exhausted_baseline_adjudication_capacity_before_provider_contact() { let server = MockServer::start().await; - mount_azure_merge_base(&server).await; - // Small file: line 2 changes; the model flags line 2. - let old_content = "fn login() {\n let token = sanitize(user_input);\n}\n"; - let new_content = "fn login() {\n let token = user_input;\n}\n"; - let az_finding = json!([{ - "path": "src/auth.rs", "line": 2, "severity": "error", "kind": "risk", - "confidence": 0.95, "title": "Unsanitized input reaches query", - "body": "user_input flows into the token without sanitization.", - "evidence": " let token = user_input;" - }]); Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(az_finding))) - .mount(&server) - .await; - let pr_path = "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7"; - Mock::given(method("GET")) - .and(path(pr_path)) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "description": "PR body", - "status": "active", "lastMergeSourceCommit": {"commitId": "HEAD"}, - "lastMergeTargetCommit": {"commitId": "BASE"} - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path( - "/myorg/myproj/_apis/git/repositories/myrepo/diffs/commits", - )) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "changes": [{"item": {"path": "/src/auth.rs", "isFolder": false}, "changeType": "edit"}], - "allChangesIncluded": true - }))) - .mount(&server) - .await; - let items_path = "/myorg/myproj/_apis/git/repositories/myrepo/items"; - Mock::given(method("GET")) - .and(path(items_path)) - .and(query_param("version", "BASE")) - .respond_with(ResponseTemplate::new(200).set_body_string(old_content)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path(items_path)) - .and(query_param("version", "HEAD")) - .respond_with(ResponseTemplate::new(200).set_body_string(new_content)) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path( - "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7/statuses", - )) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path( - "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7/threads", - )) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let findings = (0..20) + .map(|index| { + json!({ + "path": "src/auth.rs", "line": 42, "severity": "warn", "kind": "risk", + "confidence": 0.9, "title": format!("Open baseline finding {index}"), + "body": "The authorization query remains unsafe.", + "evidence": "exec_query(&token);" + }) + }) + .collect::>(); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, "findings": findings, + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 20, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,20], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = dir.path().join("capacity-baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("AZURE_DEVOPS_API_URL", server.uri()) - .env("AZURE_DEVOPS_TOKEN", "az-test-pat") - .args([ - "review", - "--publish", - "--forge", - "azure", - "--repo", - "myorg/myproj/myrepo", - "--pr", - "7", - "--output-json", - ]) - .assert() - .code(1); - let env: Value = - serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); - // The finding survived: reconstruction grounded line 2 in the rebuilt diff. - assert_eq!(env["headSha"], "HEAD"); - assert_eq!(env["findings"][0]["line"], 2); - assert_eq!(env["gate"]["failing"], true); - - let reqs = server.received_requests().await.unwrap(); - let thread = reqs - .iter() - .find(|r| r.url.path().ends_with("/threads")) - .expect("review thread posted"); - let body: Value = thread.body_json().unwrap(); - assert_eq!(body["threadContext"]["filePath"], "/src/auth.rs"); - assert_eq!(body["threadContext"]["rightFileStart"]["line"], 2); + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--baseline") + .arg(&baseline_path) + .args(["--output", "json"]) + .assert() + .code(2); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!( + stderr.contains("exhausting its 20-candidate bound"), + "{stderr}" + ); + assert!(stderr.contains("no provider request was made"), "{stderr}"); + assert!(server.received_requests().await.unwrap().is_empty()); } #[tokio::test] -async fn respond_to_pr_mention_posts_grounded_reply() { +async fn oversized_adjudication_payload_preserves_findings_without_aborting_review() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 5).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Line 41 interpolates `user_input` straight into the query; that is the \ - injection risk. Parameterize it.", - ))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "body": "PR body", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/issues/5/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) + .and(body_string_contains("single finding adjudicator")) + .respond_with(ResponseTemplate::new(500)) + .expect(0) + .with_priority(1) .mount(&server) .await; + mock_review( + &server, + json!([{ + "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", + "confidence": 0.95, "title": "Fresh authorization failure", + "body": "The authorization query still accepts an untrusted token.", + "evidence": "exec_query(&token);" + }]), + ) + .await; + let directory = tempfile::tempdir().unwrap(); + let diff = write_diff(directory.path()); + let baseline_findings = (0..19) + .map(|index| { + json!({ + "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", + "confidence": 0.95, "title": format!("Prior authorization failure {index}"), + "body": format!("Historical finding {index}: {}.", "x".repeat(4_000)), + "evidence": "exec_query(&token);" + }) + }) + .collect::>(); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, "findings": baseline_findings, + "resolved": [], "counts": {"info": 0, "warn": 19, "error": 0, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,19], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = directory.path().join("oversized-baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - let dir = tempfile::tempdir().unwrap(); - postil() - .current_dir(dir.path()) + let output = postil() + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "respond", - "--publish", - "--repo", - "acme/api", - "--pr", - "5", - "--comment", - "@postil is this safe?", - ]) + .env("POSTIL_DISABLE_SCORER", "1") + .args(["review", "--diff-file"]) + .arg(diff) + .arg("--baseline") + .arg(baseline_path) + .args(["--output", "json"]) .assert() - .success(); + .code(1); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + let stderr = String::from_utf8_lossy(&output.get_output().stderr); - let reqs = server.received_requests().await.unwrap(); - let comment = reqs - .iter() - .find(|r| r.url.path() == "/repos/acme/api/issues/5/comments") - .expect("reply posted"); - let body: Value = comment.body_json().unwrap(); - let text = body["body"].as_str().unwrap(); - assert!(text.contains("injection risk")); - assert!(!text.contains("Postil ·")); + assert_eq!(envelope["findings"].as_array().unwrap().len(), 21); + assert_eq!( + envelope["findings"] + .as_array() + .unwrap() + .iter() + .filter(|finding| finding["path"] == ".postil/model-output") + .count(), + 1 + ); + assert_eq!(envelope["resolved"], json!([])); + assert_eq!(envelope["gate"]["failing"], true); + assert!( + stderr.contains("adjudication input exceeded its admitted bound"), + "{stderr}" + ); } #[tokio::test] -async fn respond_without_publish_prints_locally_and_never_comments() { +async fn scorer_cannot_suppress_an_unresolved_full_rereview_baseline() { let server = MockServer::start().await; - mount_github_complete_diff(&server, 5).await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Line 41 passes untrusted input to the query. Parameterize it.", - ))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) - .and(header("Accept", "application/vnd.github.v3.diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "body": "PR body", - "state": "open", "merged": false, - "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 - }))) - .mount(&server) - .await; + mock_review(&server, json!([])).await; + let directory = tempfile::tempdir().unwrap(); + let diff = write_diff(directory.path()); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 42, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Authorization guard remains bypassed", + "body": "The authorization guard remains bypassed before query execution.", + "evidence": "exec_query(&token);" + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "model", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = directory.path().join("scorer-baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); - let dir = tempfile::tempdir().unwrap(); let output = postil() - .current_dir(dir.path()) - .env("CI", "true") - .env("GITHUB_ACTIONS", "true") + .current_dir(directory.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "respond", - "--repo", - "acme/api", - "--pr", - "5", - "--comment", - "@postil is this safe?", - ]) + .env("REVIEW_SCORER_MODEL", "scorer-model") + .args(["review", "--diff-file"]) + .arg(&diff) + .arg("--baseline") + .arg(&baseline_path) + .args(["--output", "json"]) .assert() - .success(); - - assert!(String::from_utf8_lossy(&output.get_output().stdout).contains("Parameterize it.")); + .code(1); + let envelope: Value = serde_json::from_slice(&output.get_output().stdout).unwrap(); + assert_eq!( + envelope["findings"][0]["title"], + "Authorization guard remains bypassed" + ); + assert_eq!(envelope["findings"][0]["confidence"], 0.9); + assert_eq!(envelope["resolved"], json!([])); + assert_eq!(envelope["gate"]["failing"], true); let requests = server.received_requests().await.unwrap(); - assert!(!requests.iter().any(|request| { - request.method == wiremock::http::Method::POST - && request.url.path() == "/repos/acme/api/issues/5/comments" + assert!(requests.iter().all(|request| { + !String::from_utf8_lossy(&request.body).contains("independent second-model scorer") })); } #[tokio::test] -async fn respond_truncation_retries_without_publishing_partial_text() { +async fn same_head_with_open_baseline_falls_back_to_full_review() { let server = MockServer::start().await; - let partial_answer = "Partial answer must never publish."; - mount_github_complete_diff(&server, 5).await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(SequentialReviewResponder { - calls: Arc::new(AtomicUsize::new(0)), - responses: Arc::new(vec![ - json!({ - "choices": [{"finish_reason": "length", "message": {"content": - respond_payload(partial_answer, None) - }}], - "usage": {"prompt_tokens": 40, "completion_tokens": 1024} - }), - respond_text("Parameterize `user_input` before executing the query."), - ]), - }) - .expect(2) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(1) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "body": "PR body", + "title": "t", "body": null, "state": "open", "merged": false, "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/issues/5/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) - .expect(1) + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .expect(2) .mount(&server) .await; + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [ + {"path": ".postil/change-metadata", "line": 1, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "old dependency risk", "body": "fixed now"} + ], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": "bbbbbbbb", "headSha": "aaaaaaaa", "sinceSha": null + }); let dir = tempfile::tempdir().unwrap(); - postil() + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "primary-model") + .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .args(["--sha", "aaaaaaaa", "--since-sha", "aaaaaaaa", "--baseline"]) + .arg(&baseline_path) .args([ - "respond", - "--publish", - "--repo", - "acme/api", - "--pr", - "5", - "--comment", - "@postil is this safe?", + "--check-run-id", + "11", + "--gate-check-run-id", + "12", + "--output-json", ]) .assert() - .success(); + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["resolved"], json!([])); + assert_eq!(env["findings"], json!([])); + assert_eq!( + env["suppressedFindings"][0]["finding"]["title"], + "old dependency risk" + ); + assert_eq!(env["suppressedFindings"][0]["reason"], "nonActionable"); + assert_ne!(env["modelUsed"], "none (empty diff)"); + assert_eq!(env["gate"]["failing"], false); + assert_eq!(env["sinceSha"], Value::Null); let requests = server.received_requests().await.unwrap(); - let model_requests = requests + assert!( + requests + .iter() + .any(|request| { request.url.path() == "/repos/acme/api/compare/bbbbbbbb...aaaaaaaa" }) + ); + assert!( + !requests + .iter() + .any(|request| request.url.path().ends_with("/reviews")) + ); + let llm_request = requests .iter() - .filter(|request| request.url.path() == "/chat/completions") - .collect::>(); - assert_eq!(model_requests.len(), 2); - let first: Value = model_requests[0].body_json().unwrap(); - let second: Value = model_requests[1].body_json().unwrap(); - assert_eq!(first["max_tokens"], 1024); - assert_eq!(second["max_tokens"], 2048); - assert_eq!(first["messages"], second["messages"]); + .find(|request| request.url.path() == "/chat/completions") + .unwrap(); + let body: Value = llm_request.body_json().unwrap(); assert!( - !serde_json::to_string(&second) + !body["messages"][1]["content"] + .as_str() .unwrap() - .contains(partial_answer) + .contains("This is an INCREMENTAL review") ); - let comment = requests - .iter() - .find(|request| request.url.path() == "/repos/acme/api/issues/5/comments") - .expect("reply posted"); - let body: Value = comment.body_json().unwrap(); - assert!(body["body"].as_str().unwrap().contains("Parameterize")); - assert!(!body["body"].as_str().unwrap().contains(partial_answer)); } #[tokio::test] -async fn repeated_respond_truncation_fails_without_publishing() { +async fn same_head_without_open_baseline_keeps_empty_diff_noop() { let server = MockServer::start().await; - let partial_answer = "Partial answer must never publish."; - mount_github_complete_diff(&server, 5).await; - Mock::given(method("POST")) - .and(path("/chat/completions")) + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/bbbbbbbb...aaaaaaaa")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "length", "message": {"content": - respond_payload(partial_answer, None) - }}], - "usage": {"prompt_tokens": 40, "completion_tokens": 1024} + "merge_base_commit": {"sha": "bbbbbbbb"}, + "files": [] }))) - .expect(2) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/pulls/5")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Add login", "body": "PR body", + "title": "t", "body": null, "state": "open", "merged": false, "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) @@ -13571,832 +13281,803 @@ async fn repeated_respond_truncation_fails_without_publishing() { .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "primary-model") .args([ - "respond", - "--publish", + "review", "--repo", "acme/api", "--pr", - "5", - "--comment", - "@postil is this safe?", + "7", + "--sha", + "aaaaaaaa", + "--since-sha", + "aaaaaaaa", + "--no-post", + "--output-json", ]) .assert() - .failure(); - - assert!(!String::from_utf8_lossy(&out.get_output().stdout).contains(partial_answer)); + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["modelUsed"], "none (empty diff)"); let requests = server.received_requests().await.unwrap(); assert_eq!( - requests - .iter() - .filter(|request| request.url.path() == "/chat/completions") - .count(), - 2 - ); - assert!( - requests - .iter() - .all(|request| request.url.path() != "/repos/acme/api/issues/5/comments") + requests.len(), + 2, + "empty no-op fetched more than the immutable PR snapshot" ); } #[tokio::test] -async fn respond_rejects_article_shape_and_preserves_usage_across_fallback() { - use std::os::unix::fs::PermissionsExt; - +async fn stale_incremental_baseline_falls_back_to_full_review() { let server = MockServer::start().await; - let slop = respond_article_slop(); - assert_eq!(slop.chars().count(), 7_186); - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("article-model")) + mount_github_complete_diff(&server, 7).await; + // A rebase or force-push leaves the recorded baseline off the head's + // ancestry, so the incremental compare cannot describe the change. + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": slop}}], - "usage": {"prompt_tokens": 30, "completion_tokens": 900} + "merge_base_commit": {"sha": "dddddddd"}, + "files": [] }))) .expect(1) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("compact-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": respond_payload( - "`src/queue.rs:18` retries forever. Add a terminal state before merge.", - None, - )}}], - "usage": {"prompt_tokens": 20, "completion_tokens": 15} - }))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .expect(1) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/9")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Queue retry", "body": "Review the retry behavior." + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let receipt_path = dir.path().join("respond-usage.json"); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "article-model") - .env("REVIEW_MODEL_CASCADE", "compact-model") - .env("POSTIL_USAGE_RECEIPT_PATH", &receipt_path) .args([ - "respond", + "review", "--repo", "acme/api", - "--issue", - "9", - "--comment", - "@postil review this", + "--pr", + "7", + "--sha", + "aaaaaaaa", + "--since-sha", + "cccccccc", "--no-post", + "--output-json", ]) .assert() - .success(); - - let stdout = String::from_utf8_lossy(&out.get_output().stdout); - assert!(stdout.contains("Add a terminal state")); - assert!(!stdout.contains("What this PR does")); - assert!(!stdout.contains("Postil ·")); - let receipt: Value = serde_json::from_slice(&std::fs::read(&receipt_path).unwrap()).unwrap(); - assert_eq!(receipt["promptTokens"], 50); - assert_eq!(receipt["completionTokens"], 915); - assert_eq!(receipt["models"][0]["model"], "article-model"); - assert_eq!(receipt["models"][1]["model"], "compact-model"); + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"], json!([])); + assert!( + !env["findings"] + .as_array() + .unwrap() + .iter() + .any(|finding| finding["path"] == ".postil/model-output"), + "stale baseline produced an operational failure instead of a review" + ); + assert_eq!(env["gate"]["failing"], false); + assert_ne!(env["modelUsed"], "none (empty diff)"); assert_eq!( - std::fs::metadata(&receipt_path) + env["sinceSha"], + Value::Null, + "a full review reported an incremental baseline it did not measure against" + ); + + let requests = server.received_requests().await.unwrap(); + assert!( + requests + .iter() + .any(|request| request.url.path() == "/repos/acme/api/pulls/7/files"), + "fallback did not acquire the complete change" + ); + let llm_request = requests + .iter() + .find(|request| request.url.path() == "/chat/completions") + .unwrap(); + let body: Value = llm_request.body_json().unwrap(); + assert!( + !body["messages"][1]["content"] + .as_str() .unwrap() - .permissions() - .mode() - & 0o777, - 0o600, + .contains("This is an INCREMENTAL review"), + "fallback reviewed the change as incremental" ); } #[tokio::test] -async fn respond_rejects_generated_mermaid_even_when_requested() { +async fn incremental_touched_carried_error_falls_back_to_full_review() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("diagram-model")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_text(&respond_payload( - "The request is queued before a worker handles it.", - Some("flowchart LR\n API --> Queue\n Queue --> Worker"), - ))), - ) + mount_github_complete_diff(&server, 7).await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "merge_base_commit": {"sha": "cccccccc"}, + "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] + }))) .expect(1) .mount(&server) .await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("compact-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "The API writes the job to the queue before a worker claims it.", - ))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .expect(1) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/10")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Queue flow", "body": "How does work reach a worker?" + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Prior authorization blocker", + "body": "The authorization path remains unsafe." + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null + }); let dir = tempfile::tempdir().unwrap(); + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "diagram-model") - .env("REVIEW_MODEL_CASCADE", "compact-model") .args([ - "respond", + "review", "--repo", "acme/api", - "--issue", - "10", - "--comment", - "@postil please include a Mermaid diagram of this flow", - "--no-post", + "--pr", + "7", + "--sha", + "aaaaaaaa", + "--since-sha", + "cccccccc", + "--baseline", ]) + .arg(&baseline_path) + .args(["--no-post", "--output-json"]) .assert() - .success(); + .code(1); + let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert_eq!(env["sinceSha"], Value::Null); - let stdout = String::from_utf8_lossy(&out.get_output().stdout); - assert!(stdout.contains("worker claims it")); - assert!(!stdout.contains("```mermaid")); - assert!(!stdout.contains("API --> Queue")); + let requests = server.received_requests().await.unwrap(); + assert!( + requests + .iter() + .any(|request| request.url.path() == "/repos/acme/api/pulls/7/files"), + "a touched carried Error must fetch the complete change" + ); + let source_request = requests + .iter() + .find(|request| request.url.path() == "/chat/completions") + .unwrap(); + let body: Value = source_request.body_json().unwrap(); + assert!( + !body["messages"][1]["content"] + .as_str() + .unwrap() + .contains("This is an INCREMENTAL review"), + "full fallback must happen before model review and adjudication" + ); } #[tokio::test] -async fn respond_rejects_unrequested_mermaid_and_uses_compact_fallback() { +async fn local_incremental_diff_file_with_touched_carried_error_fails_closed_actionably() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("diagram-model")) - .respond_with( - ResponseTemplate::new(200).set_body_json(llm_text(&respond_payload( - "The request enters a queue.", - Some("flowchart LR\n API --> Queue"), - ))), - ) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("compact-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "The API writes the job to the queue before a worker claims it.", - ))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/11")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Queue flow", "body": "Explain the worker handoff." - }))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); + let diff = write_diff(dir.path()); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Prior authorization blocker", + "body": "The authorization path remains unsafe." + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "diagram-model") - .env("REVIEW_MODEL_CASCADE", "compact-model") - .args([ - "respond", - "--repo", - "acme/api", - "--issue", - "11", - "--comment", - "@postil explain the worker handoff", - "--no-post", - ]) + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--since-sha", "previous", "--baseline"]) + .arg(&baseline_path) + .args(["--output", "json"]) .assert() - .success(); - - let stdout = String::from_utf8_lossy(&out.get_output().stdout); - assert!(stdout.contains("worker claims it")); - assert!(!stdout.contains("```mermaid")); + .code(1); + let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + let stderr = String::from_utf8_lossy(&out.get_output().stderr); + assert!(stderr.contains("cannot reconstruct the complete comparison")); + let incomplete = env["findings"] + .as_array() + .unwrap() + .iter() + .find(|finding| finding["path"] == ".postil/model-output") + .unwrap(); + assert_eq!(incomplete["title"], "Review incomplete"); + assert!( + incomplete["body"] + .as_str() + .unwrap() + .contains("complete comparison") + ); + let carried = env["findings"] + .as_array() + .unwrap() + .iter() + .find(|finding| finding["title"] == "Prior authorization blocker") + .expect("the prior blocker must remain in the next baseline"); + assert!( + carried["body"] + .as_str() + .unwrap() + .starts_with("[carried from previous review]") + ); + assert_eq!(env["resolved"], json!([])); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn respond_rejects_unsafe_reply_before_direct_forge_posting() { +async fn local_incremental_staged_review_with_touched_carried_error_fails_closed() { let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("unsafe-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Ask @maintainer to approve this.\n\n## Verdict\nLooks good.", - ))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("compact-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "`src/queue.rs:18` can retry forever. Add a terminal state before merge.", - ))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/12")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Queue retry", "body": "Review the retry behavior." - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/issues/12/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) - .expect(1) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) .mount(&server) .await; - let dir = tempfile::tempdir().unwrap(); - postil() + initialize_staged_repository(dir.path()); + std::fs::write(dir.path().join(".postil.yaml"), "enabled: true\n").unwrap(); + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "Prior authorization blocker", + "body": "The authorization path remains unsafe." + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "unsafe-model") - .env("REVIEW_MODEL_CASCADE", "compact-model") .args([ - "respond", - "--publish", - "--repo", - "acme/api", - "--issue", - "12", - "--comment", - "@postil review the retry behavior", + "review", + "--staged", + "--since-sha", + "previous", + "--baseline", ]) + .arg(&baseline_path) + .args(["--output", "json"]) .assert() - .success(); - - let requests = server.received_requests().await.unwrap(); - let post = requests - .iter() - .find(|request| request.url.path() == "/repos/acme/api/issues/12/comments") - .expect("validated fallback posted"); - let body: Value = post.body_json().unwrap(); - let reply = body["body"].as_str().unwrap(); - assert!(reply.contains("Add a terminal state")); - assert!(!reply.contains("@maintainer")); - assert!(!reply.contains("Verdict")); + .code(1); + let env: Value = serde_json::from_slice(&out.get_output().stdout).unwrap(); + assert!( + env["findings"] + .as_array() + .unwrap() + .iter() + .any(|finding| finding["path"] == ".postil/model-output" + && finding["title"] == "Review incomplete") + ); + assert!(env["findings"].as_array().unwrap().iter().any(|finding| { + finding["title"] == "Prior authorization blocker" + && finding["body"] + .as_str() + .is_some_and(|body| body.starts_with("[carried from previous review]")) + })); + assert_eq!(env["resolved"], json!([])); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn respond_writes_private_usage_receipt_across_model_fallback() { - use std::os::unix::fs::PermissionsExt; - +async fn incremental_forge_outage_still_fails_the_review() { let server = MockServer::start().await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [], - "usage": {"prompt_tokens": 10, "completion_tokens": 2} - }))) - .expect(2) + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) + .respond_with(ResponseTemplate::new(503)) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) + Mock::given(method("GET")) + .and(path_regex(r"^/repos/acme/api/compare/b+\.\.\.a+$")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": respond_payload("Use a bounded worker pool.", None)}}], - "usage": {"prompt_tokens": 20, "completion_tokens": 3, "cost": 0.00000049} + "merge_base_commit": {"sha": "bbbbbbbb"}, + "files": [] }))) - .expect(1) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/9")) + .and(path("/repos/acme/api/pulls/7/files")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([]))) + .expect(0) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Timeouts", "body": "Requests hang under load." + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - let receipt_path = dir.path().join("respond-usage.json"); - std::fs::write( - &receipt_path, - b"stale receipt from an interrupted attempt\n", - ) - .unwrap(); let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") - .env("POSTIL_USAGE_RECEIPT_PATH", &receipt_path) .args([ - "respond", + "review", "--repo", "acme/api", - "--issue", - "9", - "--comment", - "@postil how should this be bounded?", + "--pr", + "7", + "--sha", + "aaaaaaaa", + "--since-sha", + "cccccccc", "--no-post", + "--output-json", ]) .assert() - .success(); - - let stdout = String::from_utf8_lossy(&out.get_output().stdout); - assert!(stdout.contains("Use a bounded worker pool.")); - assert!(!stdout.contains("promptTokens")); - let receipt: Value = serde_json::from_slice(&std::fs::read(&receipt_path).unwrap()).unwrap(); - assert_eq!(receipt["version"], 2); - assert_eq!(receipt["models"][0]["role"], "mentionResponder"); - assert_eq!(receipt["models"][0]["phase"], "initial"); - assert!(receipt["models"][0]["callOrdinal"].is_number()); - assert!(receipt["models"][0]["attempt"].is_number()); - assert!(receipt["models"][0]["accountingComplete"].is_boolean()); - assert_eq!(receipt["operation"], "respond"); - assert_eq!(receipt["usageAccountingComplete"], true); - assert_eq!(receipt["promptTokens"], 40); - assert_eq!(receipt["completionTokens"], 7); - assert_eq!(receipt["models"][0]["model"], "primary-model"); - assert_eq!(receipt["models"][0]["promptTokens"], 10); - assert_eq!(receipt["models"][0]["completionTokens"], 2); - assert_eq!(receipt["models"][1]["model"], "primary-model"); - assert_eq!(receipt["models"][1]["promptTokens"], 10); - assert_eq!(receipt["models"][1]["completionTokens"], 2); - assert_eq!(receipt["models"][2]["model"], "backup-model"); - assert_eq!(receipt["models"][2]["promptTokens"], 20); - assert_eq!(receipt["models"][2]["completionTokens"], 3); - assert_eq!(receipt["models"][2]["costProviderDecimal"], "0.00000049"); - assert_eq!(receipt["models"][2]["costMicros"], 0); - assert_eq!(receipt["models"][2]["costSource"], "providerReported"); - assert_eq!( - std::fs::metadata(&receipt_path) - .unwrap() - .permissions() - .mode() - & 0o777, - 0o600, - ); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/provider"); + assert_eq!(env["gate"]["failing"], true); } #[tokio::test] -async fn respond_marks_receipt_incomplete_after_ambiguous_fallback() { +async fn carried_only_incremental_run_updates_checks_without_posting_review() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(503)) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("backup-model")) + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": respond_payload("Retry with a bounded backoff.", None)}}], - "usage": {"prompt_tokens": 20, "completion_tokens": 3} + "merge_base_commit": {"sha": "cccccccc"}, + "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] }))) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/10")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Retries", "body": "Requests fail intermittently." + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [ + {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "still broken", "body": "not addressed"} + ], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null + }); let dir = tempfile::tempdir().unwrap(); - let receipt_path = dir.path().join("respond-usage.json"); - postil() + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "backup-model") - .env("POSTIL_USAGE_RECEIPT_PATH", &receipt_path) + .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .args(["--sha", "aaaaaaaa", "--since-sha", "cccccccc", "--baseline"]) + .arg(&baseline_path) .args([ - "respond", - "--repo", - "acme/api", - "--issue", - "10", - "--comment", - "@postil how should this retry?", - "--no-post", + "--check-run-id", + "11", + "--gate-check-run-id", + "12", + "--output-json", ]) .assert() - .success(); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert!( + env["findings"][0]["body"] + .as_str() + .unwrap() + .starts_with("[carried from previous review]") + ); + assert_eq!(env["gate"]["failing"], true); - let receipt: Value = serde_json::from_slice(&std::fs::read(receipt_path).unwrap()).unwrap(); - assert_eq!(receipt["usageAccountingComplete"], false); - assert_eq!(receipt["models"].as_array().unwrap().len(), 4); - assert_eq!(receipt["models"][3]["model"], "backup-model"); + let requests = server.received_requests().await.unwrap(); + assert_eq!( + requests + .iter() + .filter(|request| request.method == wiremock::http::Method::PATCH) + .count(), + 2 + ); + assert!( + !requests + .iter() + .any(|request| request.url.path().ends_with("/reviews")) + ); } #[tokio::test] -async fn respond_marks_receipt_incomplete_after_internal_retry_succeeds() { +async fn identical_fresh_finding_set_does_not_post_duplicate_review() { let server = MockServer::start().await; + mount_github_complete_diff(&server, 7).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) - .respond_with(ResponseTemplate::new(408).set_body_string("request timed out")) - .up_to_n_times(1) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), + ) .mount(&server) .await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_string_contains("primary-model")) + Mock::given(method("GET")) + .and(path("/repos/acme/api/compare/cccccccc...aaaaaaaa")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "choices": [{"finish_reason": "stop", "message": {"content": respond_payload("Use a bounded retry.", None)}}], - "usage": {"prompt_tokens": 20, "completion_tokens": 3} + "merge_base_commit": {"sha": "cccccccc"}, + "files": [{"filename": "src/auth.rs", "status": "modified", "changes": 2}] }))) .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/11")) + .and(path("/repos/acme/api/pulls/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Retries", "body": "Requests fail intermittently." + "title": "t", "body": null, + "state": "open", "merged": false, + "head": {"sha": "aaaaaaaa"}, "base": {"sha": "bbbbbbbb"}, "changed_files": 1 }))) .mount(&server) .await; + Mock::given(method("PATCH")) + .and(path_regex(r"^/repos/acme/api/check-runs/\d+$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [{ + "path": "src/auth.rs", "line": 41, "severity": "error", "kind": "risk", + "confidence": 0.95, "title": "Unsanitized input reaches query", + "body": "user_input flows into exec_query without sanitization." + }], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": "bbbbbbbb", "headSha": "cccccccc", "sinceSha": null + }); let dir = tempfile::tempdir().unwrap(); - let receipt_path = dir.path().join("respond-usage.json"); + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("GITHUB_API_URL", server.uri()) .env("GITHUB_TOKEN", "gh-test-token") - .env("REVIEW_MODEL", "primary-model") - .env("REVIEW_MODEL_CASCADE", "") - .env("POSTIL_USAGE_RECEIPT_PATH", &receipt_path) + .args(["review", "--publish", "--repo", "acme/api", "--pr", "7"]) + .args(["--sha", "aaaaaaaa", "--since-sha", "cccccccc", "--baseline"]) + .arg(&baseline_path) .args([ - "respond", - "--repo", - "acme/api", - "--issue", + "--check-run-id", "11", - "--comment", - "@postil how should this retry?", - "--no-post", + "--gate-check-run-id", + "12", + "--output-json", ]) .assert() - .success(); + .code(1); - let receipt: Value = serde_json::from_slice(&std::fs::read(receipt_path).unwrap()).unwrap(); - assert_eq!(receipt["usageAccountingComplete"], false); - assert_eq!(receipt["models"].as_array().unwrap().len(), 2); - assert_eq!(receipt["models"][0]["model"], "primary-model"); + let requests = server.received_requests().await.unwrap(); assert_eq!( - server - .received_requests() - .await - .unwrap() + requests .iter() - .filter(|request| { request.url.path() == "/chat/completions" }) + .filter(|request| request.method == wiremock::http::Method::PATCH) .count(), 2 ); + assert!( + !requests + .iter() + .any(|request| request.url.path().ends_with("/reviews")) + ); } #[tokio::test] -async fn respond_to_issue_mention_uses_issue_body() { +async fn disabled_review_does_not_carry_baseline_findings() { + // M2 regression: a repo with `enabled: false` plus a supplied baseline must + // not have baseline Errors reconciled and carried into the gate. With review + // off there is no fresh model run, so reconcile is skipped entirely and the + // gate stays clean. No LLM call should be made. let server = MockServer::start().await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "This looks like a connection-pool exhaustion under load, not a logic bug.", - ))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/repos/acme/api/issues/9")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Timeouts under load", - "body": "Requests hang after ~200 concurrent users." - }))) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repos/acme/api/issues/9/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) // disabled: the model is never called .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let diff = write_diff(dir.path()); + std::fs::write(dir.path().join(".postil.yaml"), "enabled: false\n").unwrap(); + + let baseline = json!({ + "version": 1, "summary": "", "silent": false, + "findings": [ + {"path": "src/db.rs", "line": 10, "severity": "error", "kind": "risk", + "confidence": 0.9, "title": "still broken", "body": "not addressed"} + ], + "resolved": [], "counts": {"info": 0, "warn": 0, "error": 1, "suppressed": 0}, + "confidenceBuckets": [0,0,0,0,1], + "gate": {"failOn": "error", "failing": true}, + "modelUsed": "m", "usage": {"promptTokens": 0, "completionTokens": 0}, + "baseSha": null, "headSha": null, "sinceSha": null + }); + let baseline_path = dir.path().join("baseline.json"); + std::fs::write(&baseline_path, baseline.to_string()).unwrap(); + + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITHUB_API_URL", server.uri()) - .env("GITHUB_TOKEN", "gh-test-token") - .args([ - "respond", - "--publish", - "--repo", - "acme/api", - "--issue", - "9", - "--comment", - "@postil what do you think is happening?", - ]) + .args(["review", "--diff-file"]) + .arg(&diff) + .args(["--since-sha", "abc123", "--baseline"]) + .arg(&baseline_path) + .arg("--output-json") .assert() - .success(); - - let reqs = server.received_requests().await.unwrap(); - // The model was given the issue body as grounding. - let llm = reqs - .iter() - .find(|r| r.url.path() == "/chat/completions") - .unwrap(); - let sent: Value = llm.body_json().unwrap(); - let user_msg = sent["messages"][1]["content"].as_str().unwrap(); - assert!(user_msg.contains("200 concurrent users")); - assert!(reqs.iter().any(|r| { - r.method == wiremock::http::Method::POST - && r.url.path() == "/repos/acme/api/issues/9/comments" - })); + .code(0); // disabled + no carry => gate passes + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["gate"]["failing"], false); + assert_eq!( + env["findings"].as_array().map(|a| a.len()).unwrap_or(0), + 0, + "disabled review carried a baseline finding into the gate" + ); + assert_eq!(env["silent"], true); } #[tokio::test] -async fn respond_gitlab_mr_mention_posts_note() { +async fn bitbucket_flow_posts_comment_and_sets_statuses() { let server = MockServer::start().await; - mount_gitlab_source_files(&server).await; + mount_bitbucket_complete_diff(&server).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Line 41 interpolates `user_input` into the query; parameterize it.", - ))) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(llm_content(json!([finding_at(41, "error", 0.95)]))), + ) .mount(&server) .await; - // MR metadata (title/description + diff refs) for grounding. Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/5$")) + .and(path("/repositories/acme/api/pullrequests/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ "title": "Add login", - "description": "MR body", - "state": "opened", "diff_refs": {"base_sha": "b", "start_sha": "s", "head_sha": "h"} + "summary": {"raw": "PR body"}, + "state": "OPEN", "source": {"commit": {"hash": "aaaaaaaa"}}, + "destination": {"commit": {"hash": "bbbbbbbb"}} }))) .mount(&server) .await; Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/5/versions$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "state": "collected", "real_size": "1" - }]))) + .and(path("/repositories/acme/api/pullrequests/7/diff")) + .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) .mount(&server) .await; - // MR file diffs (paginated; one short page ends iteration). - Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/5/diffs$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "old_path": "src/auth.rs", "new_path": "src/auth.rs", - "diff": "@@ -40,6 +40,8 @@ fn login() {\n context line\n+let token = format!(\"{}\", user_input);\n+exec_query(&token);\n trailing context\n", - "new_file": false, "deleted_file": false - }]))) + Mock::given(method("POST")) + .and(path_regex( + r"^/repositories/acme/api/commit/.+/statuses/build$", + )) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) .mount(&server) .await; - // The reply note endpoint. Mock::given(method("POST")) - .and(path_regex(r"^/projects/.+/merge_requests/5/notes$")) + .and(path("/repositories/acme/api/pullrequests/7/comments")) .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITLAB_API_URL", server.uri()) - .env("GITLAB_TOKEN", fixture_credential("gitlab")) + .env("BITBUCKET_API_URL", server.uri()) + .env("BITBUCKET_TOKEN", "bb-test-token") + .env_remove("BITBUCKET_USER") .args([ - "respond", + "review", "--publish", "--forge", - "gitlab", + "bitbucket", "--repo", "acme/api", "--pr", - "5", - "--comment", - "@postil is this safe?", + "7", + "--output-json", ]) .assert() - .success(); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["headSha"], "aaaaaaaa"); let reqs = server.received_requests().await.unwrap(); - let note = reqs + let statuses: Vec = reqs .iter() - .find(|r| { - r.method == wiremock::http::Method::POST - && r.url.path().ends_with("/merge_requests/5/notes") - }) - .expect("reply note posted to the MR"); - let body: Value = note.body_json().unwrap(); - let text = body["body"].as_str().unwrap(); - assert!(text.contains("parameterize")); - assert!(!text.contains("Postil ·")); -} - -#[tokio::test] -async fn gitlab_diff_pagination_follows_authoritative_next_page_to_exhaustion() { - let server = MockServer::start().await; - mount_gitlab_source_files(&server).await; - Mock::given(method("POST")) - .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "The late-page change is included in the review context.", - ))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/6$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Paginated change", - "description": "", - "state": "opened", "diff_refs": {"base_sha": "b", "start_sha": "s", "head_sha": "h"} - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/6/versions$")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "state": "collected", "real_size": "101" - }]))) - .mount(&server) - .await; - let first_page: Vec = (0..100) - .map(|index| { - json!({ - "old_path": format!("src/early_{index}.rs"), - "new_path": format!("src/early_{index}.rs"), - "diff": "@@ -0,0 +1 @@\n+let early = true;\n", - "new_file": false, - "deleted_file": false, - "collapsed": false, - "too_large": false - }) - }) + .filter(|r| r.url.path().ends_with("/statuses/build")) + .map(|r| r.body_json().unwrap()) .collect(); - Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/6/diffs$")) - .and(query_param("per_page", "100")) - .and(query_param("page", "1")) - .respond_with( - ResponseTemplate::new(200) - .insert_header("x-next-page", "2") - .set_body_json(first_page), - ) - .expect(1) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/merge_requests/6/diffs$")) - .and(query_param("per_page", "100")) - .and(query_param("page", "2")) - .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ - "old_path": "src/late.rs", - "new_path": "src/late.rs", - "diff": "@@ -0,0 +1 @@\n+let AUTHORITATIVE_LAST_PAGE = true;\n", - "new_file": false, - "deleted_file": false, - "collapsed": false, - "too_large": false - }]))) - .expect(1) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path_regex(r"^/projects/.+/merge_requests/6/notes$")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) - .mount(&server) - .await; - - let dir = tempfile::tempdir().unwrap(); - postil() - .current_dir(dir.path()) - .env("POSTIL_API_BASE", server.uri()) - .env("GITLAB_API_URL", server.uri()) - .env("GITLAB_TOKEN", fixture_credential("gitlab")) - .args([ - "respond", - "--forge", - "gitlab", - "--repo", - "acme/api", - "--pr", - "6", - "--comment", - "@postil review this", - ]) - .assert() - .success(); - let requests = server.received_requests().await.unwrap(); - let model = requests + // Two on start (INPROGRESS) + two on complete. + assert_eq!(statuses.len(), 4); + let gate_final = statuses .iter() - .find(|request| request.url.path() == "/chat/completions") + .rev() + .find(|s| s["key"] == "postil/gate") .unwrap(); - let body: Value = model.body_json().unwrap(); - let model_context = body["messages"][1]["content"].as_str().unwrap(); - assert!( - model_context.contains("AUTHORITATIVE_LAST_PAGE"), - "final paginated evidence missing from bounded context: {model_context}" - ); + assert_eq!(gate_final["state"], "FAILED"); + let review_final = statuses + .iter() + .rev() + .find(|s| s["key"] == "postil/review") + .unwrap(); + assert_eq!(review_final["state"], "SUCCESSFUL"); + // Inline comment anchored to the cited path/line. + let comment = reqs + .iter() + .find(|r| { + r.method == wiremock::http::Method::POST + && r.url.path().ends_with("/comments") + && r.body_json::() + .map(|b| b.get("inline").is_some()) + .unwrap_or(false) + }) + .expect("inline comment posted"); + let body: Value = comment.body_json().unwrap(); + assert_eq!(body["inline"]["path"], "src/auth.rs"); + assert_eq!(body["inline"]["to"], 41); } #[tokio::test] -async fn respond_gitlab_issue_mention_uses_issue_body() { +async fn bitbucket_incremental_is_disabled_without_verification_gate() { let server = MockServer::start().await; + mount_bitbucket_merge_base(&server).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "This looks like connection-pool exhaustion under load, not a logic bug.", - ))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .expect(0) .mount(&server) .await; Mock::given(method("GET")) - .and(path_regex(r"^/projects/.+/issues/9$")) + .and(path("/repositories/acme/api/pullrequests/7")) .respond_with(ResponseTemplate::new(200).set_body_json(json!({ - "title": "Timeouts under load", - "description": "Requests hang after ~200 concurrent users." + "title": "Add login", + "summary": {"raw": "PR body"}, + "state": "OPEN", "source": {"commit": {"hash": "aaaaaaaa"}}, + "destination": {"commit": {"hash": "bbbbbbbb"}} }))) .mount(&server) .await; - Mock::given(method("POST")) - .and(path_regex(r"^/projects/.+/issues/9/notes$")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) - .mount(&server) - .await; let dir = tempfile::tempdir().unwrap(); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) - .env("GITLAB_API_URL", server.uri()) - .env("GITLAB_TOKEN", fixture_credential("gitlab")) + .env("BITBUCKET_API_URL", server.uri()) + .env("BITBUCKET_TOKEN", "bb-test-token") + .env_remove("BITBUCKET_USER") .args([ - "respond", - "--publish", + "review", "--forge", - "gitlab", + "bitbucket", "--repo", "acme/api", - "--issue", - "9", - "--comment", - "@postil what is happening?", + "--pr", + "7", + "--sha", + "aaaaaaaa", + "--since-sha", + "cccccccc", + "--no-post", + "--output-json", ]) .assert() - .success(); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["findings"][0]["path"], ".postil/model-output"); + assert!( + !env["findings"][0]["body"] + .as_str() + .unwrap() + .contains("POSTIL_ENABLE_BITBUCKET_INCREMENTAL") + ); let reqs = server.received_requests().await.unwrap(); - // The model was grounded on the issue body. - let llm = reqs - .iter() - .find(|r| r.url.path() == "/chat/completions") - .unwrap(); - let sent: Value = llm.body_json().unwrap(); - let user_msg = sent["messages"][1]["content"].as_str().unwrap(); - assert!(user_msg.contains("200 concurrent users")); - // The reply landed on the issue notes endpoint, not the MR one. - assert!(reqs.iter().any(|r| { - r.method == wiremock::http::Method::POST && r.url.path().ends_with("/issues/9/notes") - })); + assert!( + !reqs + .iter() + .any(|request| request.url.path().contains("/diff/")) + ); } #[tokio::test] -async fn respond_bitbucket_pr_mention_posts_comment() { +async fn bitbucket_incremental_fetches_documented_compare_when_enabled() { let server = MockServer::start().await; mount_bitbucket_complete_diff(&server).await; Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Line 41 interpolates `user_input`; that is the injection risk.", - ))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) .mount(&server) .await; Mock::given(method("GET")) @@ -14410,63 +14091,79 @@ async fn respond_bitbucket_pr_mention_posts_comment() { .mount(&server) .await; Mock::given(method("GET")) - .and(path("/repositories/acme/api/pullrequests/7/diff")) - .respond_with(ResponseTemplate::new(200).set_body_string(DIFF)) - .mount(&server) - .await; - Mock::given(method("POST")) - .and(path("/repositories/acme/api/pullrequests/7/comments")) - .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) + .and(path("/repositories/acme/api/diffstat/aaaaaaaa..cccccccc")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "values": [{ + "status": "modified", + "old": {"path": "src/auth.rs"}, + "new": {"path": "src/auth.rs"} + }] + }))) + .expect(1) .mount(&server) .await; let dir = tempfile::tempdir().unwrap(); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("BITBUCKET_API_URL", server.uri()) .env("BITBUCKET_TOKEN", "bb-test-token") + .env("POSTIL_ENABLE_BITBUCKET_INCREMENTAL", "1") + .env("REVIEW_MODEL", "test-model") .env_remove("BITBUCKET_USER") .args([ - "respond", - "--publish", + "review", "--forge", "bitbucket", "--repo", "acme/api", "--pr", "7", - "--comment", - "@postil is this safe?", + "--sha", + "aaaaaaaa", + "--since-sha", + "cccccccc", + "--no-post", + "--output-json", ]) .assert() - .success(); + .code(0); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + assert_eq!(env["sinceSha"], "cccccccc"); + assert_eq!(env["modelUsed"], "test-model"); let reqs = server.received_requests().await.unwrap(); - let comment = reqs + let llm_request = reqs .iter() - .find(|r| { - r.method == wiremock::http::Method::POST - && r.url.path() == "/repositories/acme/api/pullrequests/7/comments" - }) - .expect("reply comment posted to the PR"); - let body: Value = comment.body_json().unwrap(); - let text = body["content"]["raw"].as_str().unwrap(); - assert!(text.contains("injection risk")); - assert!(!text.contains("Postil ·")); + .find(|request| request.url.path() == "/chat/completions") + .unwrap(); + let body: Value = llm_request.body_json().unwrap(); + assert!( + body["messages"][1]["content"] + .as_str() + .unwrap() + .contains("This is an INCREMENTAL review") + ); } #[tokio::test] -async fn respond_azure_pr_mention_posts_thread() { +async fn azure_flow_reconstructs_diff_and_posts_thread() { let server = MockServer::start().await; mount_azure_merge_base(&server).await; + // Small file: line 2 changes; the model flags line 2. let old_content = "fn login() {\n let token = sanitize(user_input);\n}\n"; let new_content = "fn login() {\n let token = user_input;\n}\n"; + let az_finding = json!([{ + "path": "src/auth.rs", "line": 2, "severity": "error", "kind": "risk", + "confidence": 0.95, "title": "Unsanitized input reaches query", + "body": "user_input flows into the token without sanitization.", + "evidence": " let token = user_input;" + }]); Mock::given(method("POST")) .and(path("/chat/completions")) - .respond_with(ResponseTemplate::new(200).set_body_json(respond_text( - "Line 2 drops the sanitize() call; that is the risk.", - ))) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(az_finding))) .mount(&server) .await; let pr_path = "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7"; @@ -14502,6 +14199,13 @@ async fn respond_azure_pr_mention_posts_thread() { .respond_with(ResponseTemplate::new(200).set_body_string(new_content)) .mount(&server) .await; + Mock::given(method("POST")) + .and(path( + "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7/statuses", + )) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({}))) + .mount(&server) + .await; Mock::given(method("POST")) .and(path( "/myorg/myproj/_apis/git/repositories/myrepo/pullRequests/7/threads", @@ -14511,13 +14215,13 @@ async fn respond_azure_pr_mention_posts_thread() { .await; let dir = tempfile::tempdir().unwrap(); - postil() + let out = postil() .current_dir(dir.path()) .env("POSTIL_API_BASE", server.uri()) .env("AZURE_DEVOPS_API_URL", server.uri()) .env("AZURE_DEVOPS_TOKEN", "az-test-pat") .args([ - "respond", + "review", "--publish", "--forge", "azure", @@ -14525,49 +14229,144 @@ async fn respond_azure_pr_mention_posts_thread() { "myorg/myproj/myrepo", "--pr", "7", - "--comment", - "@postil is this safe?", + "--output-json", ]) .assert() - .success(); + .code(1); + let env: Value = + serde_json::from_str(&String::from_utf8(out.get_output().stdout.clone()).unwrap()).unwrap(); + // The finding survived: reconstruction grounded line 2 in the rebuilt diff. + assert_eq!(env["headSha"], "HEAD"); + assert_eq!(env["findings"][0]["line"], 2); + assert_eq!(env["gate"]["failing"], true); let reqs = server.received_requests().await.unwrap(); let thread = reqs .iter() - .find(|r| { - r.method == wiremock::http::Method::POST - && r.url.path().ends_with("/pullRequests/7/threads") - }) - .expect("reply thread posted to the PR"); + .find(|r| r.url.path().ends_with("/threads")) + .expect("review thread posted"); let body: Value = thread.body_json().unwrap(); - let text = body["comments"][0]["content"].as_str().unwrap(); - assert!(text.contains("risk")); - assert!(!text.contains("Postil ·")); + assert_eq!(body["threadContext"]["filePath"], "/src/auth.rs"); + assert_eq!(body["threadContext"]["rightFileStart"]["line"], 2); +} + +#[tokio::test] +async fn gitlab_diff_pagination_follows_authoritative_next_page_to_exhaustion() { + let server = MockServer::start().await; + mount_gitlab_source_files(&server).await; + Mock::given(method("POST")) + .and(path("/chat/completions")) + .respond_with(ResponseTemplate::new(200).set_body_json(llm_content(json!([])))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path_regex(r"^/projects/.+/merge_requests/6$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "title": "Paginated change", + "description": "", + "state": "opened", "diff_refs": {"base_sha": "b", "start_sha": "s", "head_sha": "h"} + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path_regex(r"^/projects/.+/merge_requests/6/versions$")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ + "state": "collected", "real_size": "101" + }]))) + .mount(&server) + .await; + let first_page: Vec = (0..100) + .map(|index| { + json!({ + "old_path": format!("src/early_{index}.rs"), + "new_path": format!("src/early_{index}.rs"), + "diff": "@@ -0,0 +1 @@\n+let early = true;\n", + "new_file": false, + "deleted_file": false, + "collapsed": false, + "too_large": false + }) + }) + .collect(); + Mock::given(method("GET")) + .and(path_regex(r"^/projects/.+/merge_requests/6/diffs$")) + .and(query_param("per_page", "100")) + .and(query_param("page", "1")) + .respond_with( + ResponseTemplate::new(200) + .insert_header("x-next-page", "2") + .set_body_json(first_page), + ) + .expect(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path_regex(r"^/projects/.+/merge_requests/6/diffs$")) + .and(query_param("per_page", "100")) + .and(query_param("page", "2")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!([{ + "old_path": "src/late.rs", + "new_path": "src/late.rs", + "diff": "@@ -0,0 +1 @@\n+let AUTHORITATIVE_LAST_PAGE = true;\n", + "new_file": false, + "deleted_file": false, + "collapsed": false, + "too_large": false + }]))) + .expect(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path_regex(r"^/projects/.+/merge_requests/6/notes$")) + .respond_with(ResponseTemplate::new(201).set_body_json(json!({}))) + .mount(&server) + .await; + + let dir = tempfile::tempdir().unwrap(); + postil() + .current_dir(dir.path()) + .env("POSTIL_API_BASE", server.uri()) + .env("GITLAB_API_URL", server.uri()) + .env("GITLAB_TOKEN", fixture_credential("gitlab")) + .args([ + "review", "--forge", "gitlab", "--repo", "acme/api", "--pr", "6", + ]) + .assert() + .success(); + let requests = server.received_requests().await.unwrap(); + let reviewed_last_page = requests + .iter() + .filter(|request| request.url.path() == "/chat/completions") + .any(|request| { + let body: Value = request.body_json().unwrap(); + body["messages"][1]["content"] + .as_str() + .is_some_and(|context| context.contains("AUTHORITATIVE_LAST_PAGE")) + }); + assert!( + reviewed_last_page, + "final paginated evidence was missing from every review batch" + ); } #[test] fn init_writes_starter_and_config_shows_provenance() { let dir = tempfile::tempdir().unwrap(); - Command::cargo_bin("postil") - .unwrap() + isolated_postil() .current_dir(dir.path()) .args(["init"]) .assert() .success(); assert!(dir.path().join(".postil.yaml").is_file()); // Second init refuses without --force. - Command::cargo_bin("postil") - .unwrap() + isolated_postil() .current_dir(dir.path()) .args(["init"]) .assert() .code(2); - let out = Command::cargo_bin("postil") - .unwrap() + let out = isolated_postil() .current_dir(dir.path()) - .env_remove("REVIEW_MODEL") - .env_remove("POSTIL_API_BASE") .args(["config"]) .assert() .success(); @@ -14579,8 +14378,7 @@ fn init_writes_starter_and_config_shows_provenance() { #[test] fn qualification_metadata_cli_emits_service_authority_fields() { - let output = Command::cargo_bin("postil") - .unwrap() + let output = isolated_postil() .args(["qualification-metadata"]) .assert() .success() @@ -14604,11 +14402,8 @@ fn coderabbit_config_is_translated() { "reviews:\n profile: chill\n path_filters:\n - \"!**/generated/**\"\n", ) .unwrap(); - let out = Command::cargo_bin("postil") - .unwrap() + let out = isolated_postil() .current_dir(dir.path()) - .env_remove("REVIEW_MODEL") - .env_remove("POSTIL_API_BASE") .args(["config"]) .assert() .success(); @@ -14645,8 +14440,7 @@ fn plan_replays_envelopes_deterministically() { ) .unwrap(); - let out = Command::cargo_bin("postil") - .unwrap() + let out = isolated_postil() .current_dir(dir.path()) .args(["plan", "--envelopes"]) .arg(&envelopes)