Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 59 additions & 53 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,7 @@ name: Release
on:
push:
tags: ["v*"]
workflow_dispatch:
inputs:
bench_live_override_reason:
description: >-
Publish even when the live benchmark misses its floor, recording this
reason on the run. Leave empty to require the benchmark to pass. The
baseline is never rewritten; an accepted miss stays visible as a miss.
required: false
default: ""
workflow_dispatch: {}

jobs:
validate-tag:
Expand Down Expand Up @@ -52,7 +44,7 @@ jobs:
bench-live:
needs: validate-tag
runs-on: ubuntu-latest
timeout-minutes: 90
timeout-minutes: 270
permissions:
contents: read
steps:
Expand All @@ -63,9 +55,8 @@ jobs:
with:
bun-version: 1.3.14

# A present-but-rejected key fails all 70 cases about three minutes later
# with nothing that names the credential. Probing it costs one request and
# turns that into an immediate, accurate failure.
# A present-but-rejected key fails every benchmark case without naming the
# credential. This bounded probe makes that failure immediate and explicit.
- name: Require a working OpenRouter credential
env:
COMPLETION_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
Expand Down Expand Up @@ -93,7 +84,7 @@ jobs:
if jq -e '.data.limit_remaining != null and .data.limit_remaining < 1' \
/tmp/openrouter-key.json >/dev/null; then
remaining="$(jq -r '.data.limit_remaining' /tmp/openrouter-key.json)"
echo "::error::OPENROUTER_API_KEY has ${remaining} credit remaining, below the cost of one benchmark run."
echo "::error::OPENROUTER_API_KEY has ${remaining} credit remaining, below the release benchmark reserve."
exit 1
fi
remaining="$(jq -r '.data.limit_remaining // "unlimited"' /tmp/openrouter-key.json)"
Expand Down Expand Up @@ -137,65 +128,80 @@ jobs:
if-no-files-found: warn
retention-days: 30

- name: Run diff-file live benchmark
- name: Run diff-file live benchmark sample 1
id: live-sample-1
continue-on-error: true
working-directory: bench
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
POSTIL_BIN: ${{ github.workspace }}/target/release/postil
run: >-
bun run bench:live -- --screen-profile ../provisional-models.json
--run-id "release-${{ github.ref_name }}-sample-1"
--json-out "${{ runner.temp }}/bench-live-report-1.json"

- name: Run diff-file live benchmark sample 2
id: live-sample-2
continue-on-error: true
working-directory: bench
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
POSTIL_BIN: ${{ github.workspace }}/target/release/postil
run: >-
bun run bench:live -- --screen-profile ../provisional-models.json
--run-id "release-${{ github.ref_name }}-sample-2"
--json-out "${{ runner.temp }}/bench-live-report-2.json"

- name: Run diff-file live benchmark sample 3
id: live-sample-3
continue-on-error: true
working-directory: bench
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
POSTIL_BIN: ${{ github.workspace }}/target/release/postil
run: >-
bun run bench:live -- --screen-profile ../provisional-models.json
--run-id "release-${{ github.ref_name }}"
--json-out "${{ runner.temp }}/bench-live-report.json"
--run-id "release-${{ github.ref_name }}-sample-3"
--json-out "${{ runner.temp }}/bench-live-report-3.json"

- name: Upload the diff-file live report
- name: Upload the diff-file live reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: bench-live-report
name: bench-live-reports
path: |
${{ runner.temp }}/bench-live-report.json
${{ runner.temp }}/bench-live-report.json.partial
${{ runner.temp }}/bench-live-report-1.json
${{ runner.temp }}/bench-live-report-2.json
${{ runner.temp }}/bench-live-report-3.json
if-no-files-found: warn
retention-days: 30

- name: Compare against the recorded baseline
id: compare-baseline
# Failure is decided by the next step so an operator-supplied reason can
# accept a miss. Recording the miss and rewriting the baseline are
# different acts, and only the first one happens here.
continue-on-error: true
if: always()
working-directory: bench
run: bun run bench:compare -- --result "${{ runner.temp }}/bench-live-report.json"

- name: Enforce the benchmark floor
env:
OVERRIDE_REASON: ${{ inputs.bench_live_override_reason }}
COMPARISON: ${{ steps.compare-baseline.outcome }}
SAMPLE_1_OUTCOME: ${{ steps.live-sample-1.outcome }}
SAMPLE_2_OUTCOME: ${{ steps.live-sample-2.outcome }}
SAMPLE_3_OUTCOME: ${{ steps.live-sample-3.outcome }}
run: |
set -euo pipefail
if [[ "${COMPARISON}" == "success" ]]; then
echo "The live benchmark met its floor."
exit 0
fi
if [[ -z "${OVERRIDE_REASON}" ]]; then
echo "::error::Release blocked by a benchmark regression. See the 'Compare against the recorded baseline' step above for the failing metric(s)."
{
echo "## Release blocked"
echo
echo "The live benchmark regressed past tolerance against \`bench/baseline.json\`."
echo "See the 'Compare against the recorded baseline' step log for the failing metric(s)."
} >> "$GITHUB_STEP_SUMMARY"
comparison=0
bun run bench:compare -- \
--binary "${{ github.workspace }}/target/release/postil" \
--screen-profile ../provisional-models.json \
--expected-run-id "release-${{ github.ref_name }}-sample-1" \
--expected-run-id "release-${{ github.ref_name }}-sample-2" \
--expected-run-id "release-${{ github.ref_name }}-sample-3" \
--result "${{ runner.temp }}/bench-live-report-1.json" \
--result "${{ runner.temp }}/bench-live-report-2.json" \
--result "${{ runner.temp }}/bench-live-report-3.json" || comparison=$?
if [[ "${SAMPLE_1_OUTCOME}" != "success" || \
"${SAMPLE_2_OUTCOME}" != "success" || \
"${SAMPLE_3_OUTCOME}" != "success" ]]; then
echo "::error::At least one live benchmark sample failed before comparison."
exit 1
fi
echo "::warning title=Benchmark floor missed and accepted::${OVERRIDE_REASON}"
{
echo "## Benchmark floor missed and accepted"
echo
echo "The live benchmark missed its floor and this release was published anyway."
echo
echo "**Reason:** ${OVERRIDE_REASON}"
echo
echo "\`bench/baseline.json\` is unchanged, so the next release faces the same floor."
} >> "$GITHUB_STEP_SUMMARY"
exit "${comparison}"

build:
needs: [validate-tag, bench-live]
Expand Down
29 changes: 17 additions & 12 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,12 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc
executable vendor hunks require direct source evidence. Low-risk hunks receive
semantic credit only from selected proof batches that retain the exact repository
path, hunk identity, every changed line, and a substantive added line in the final
model request. Missing
direct capacity, proof evidence, or complete receipt coverage fails before plan
registration or provider contact. When
model request. The route fills its fixed capacity in stable metadata, risk, and path
order. A hunk receives direct credit only when every direct batch is selected;
otherwise it receives eligible exact semantic credit or remains unreviewed. The
selected schedule still runs when receipt coverage is incomplete, preserves valid
findings from selected evidence, and adds an error-severity operational finding that
keeps review trust and the merge gate failed. When
`POSTIL_LARGE_REVIEW_PLAN_ENDPOINT` and `POSTIL_LARGE_REVIEW_PLAN_TOKEN` are set, the
CLI registers a versioned deterministic request plan with the authenticated loopback
endpoint before any provider call. A missing, rejected, or unreachable registration
Expand Down Expand Up @@ -252,10 +255,11 @@ additions (violations are `kind: contentPolicy`). Content policy is on by defaul
irreducible owner decision, not uncertainty about a concrete defect. Admin overrides
apply to that kind-only decision rather than ordinary risk findings.
9. Review resource or request-budget exhaustion cannot produce a clean verdict. An
incomplete deterministic receipt fails before durable plan registration or model
contact. Other preflight exhaustion emits the generic internal `Review incomplete`
operational finding without model contact. Both preserve the hosted global deadline
and keep full-review reconciliation untrustworthy.
incomplete deterministic receipt registers and runs only its immutable bounded
schedule, then emits an operational coverage finding and keeps full-review
reconciliation untrustworthy. Other preflight exhaustion emits the generic internal
`Review incomplete` operational finding without model contact. Both preserve the
hosted global deadline and cannot produce a clean gate verdict.
10. Bounded JSON metadata pages use a 32 MiB per-page cap and a 64 MiB aggregate
metadata cap. Source files and reconstructed diffs stream beyond that page limit
into one 512 MiB operation workspace shared by acquired source snapshots,
Expand All @@ -271,11 +275,12 @@ additions (violations are `kind: contentPolicy`). Content policy is on by defaul
12. Every completed review envelope records source-batch coverage when batching runs.
Deterministic large reviews also record a plan hash and direct, semantic, and
unreviewed hunk counts. Every normalized hunk has exactly one disposition; evidence
identifiers bind the exact hunk digest, and any unreviewed hunk rejects the plan
before registration or provider contact. Semantic coverage cannot resolve baseline
findings. Bounded reviews expose selected and total source-batch counts in compact
output. Planner fallback remains audit metadata and does not expose provider failure
details to a PR.
identifiers bind the exact hunk digest, and the plan hash binds the selected batch
order and payloads. Any unreviewed hunk forces failed review trust and a blocking
synthetic finding after the selected provider calls. Incomplete coverage cannot
resolve baseline findings. Bounded reviews expose selected and total source-batch
counts in compact output. Planner fallback remains audit metadata and does not expose
provider failure details to a PR.
13. Operational and provider virtual anchors expire after each run. Reviewable
PR-description and change-metadata anchors carry across unrelated incremental
reviews, and a same-head rerun with either anchor falls back to a full review.
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "postil-cli"
version = "0.9.2"
version = "0.9.3"
edition = "2024"
description = "Postil: a low-noise AI review gate. Silent on clean PRs, hard gate on real risk."
license = "Apache-2.0"
Expand Down
3 changes: 2 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@
semantic batches for low-risk hunks, a 24-request quality ceiling lowered by hosted
model and scorer fan-out, four-way concurrency, format-specific lockfile summaries,
oversized-line segmentation, a hosted ceiling that reserves every enabled
post-processing call, and fail-closed incomplete coverage.
post-processing call, and fail-closed partial review that preserves findings from the
selected schedule when normalized hunks remain unreviewed.
- `.coderabbit.yaml` translation for zero-cost migration.
- Model cascade + concurrent multi-model consensus over any OpenAI-compatible endpoint;
bounded retry with jittered backoff on transient provider errors.
Expand Down
Loading