[US-372] feat: coverage baseline auto commit-back — monotonic ratchet as a bot PR, opt-in and off by default - #405
Conversation
Decide AC5 (PR vs push) and the credential model against the POST-#234 regime, not today's unprotected main: - a pull_request run NEVER writes (uniform for forks; no head-SHA mutation, so a human approval is never invalidated) - the post-merge push proposes the raise as a bot PR from chore/coverage-baseline-ratchet — no branch-protection bypass is requested, which is the point of #234 - credential: repo-scoped COVERAGE_RATCHET_TOKEN (contents:write + pull-requests:write, no admin, no bypass list); GITHUB_TOKEN rejected because a PR it opens triggers no run and can never satisfy the required contexts - loop termination: marker predicate PLUS a floor(measured)-1pp fixpoint; [skip ci] rejected (it would skip the checks the ratchet PR must pass) - rejected options recorded with the reason each fails - Task: T-1 — Decide and record PR-vs-push behaviour + credential model Refs: #372
Gate logic in a tested module (ADL 2026-07-13); ci.yml stays a thin entrypoint that only forwards the measured numbers and exits 0. - ratchet writer: raises `^baseline.<type>=` values IN PLACE, never rewrites the surrounding markdown; only a value strictly above the committed one is a raise (equal/drop/unknown-type/malformed all hold) - concurrency: the config is re-read immediately before writing and every proposal re-checked, so a higher value from a concurrent run is dropped, never clobbered - skip predicate: flag-disabled (default) / not-base-push (every PR run) / automated-commit (marker or ratchet branch in the head commit) - git plan is data, so its non-negotiables are asserted by tests: one push, to the ratchet ref only, no `git add -A`, no branch switch, always restored to the checked-out SHA - push uses --force-with-lease AND first maps+fetches a remote-tracking ref for the ratchet branch: without it git rejects every later non-fast-forward push as `stale info` (verified), so the ratchet would have worked exactly once and then warned forever - refused write => named ::warning:: and exit 0; the guardrail's verdict is untouched (step runs after it) - token bound to the base-branch push event in ci.yml, so a PR run never has the credential in its environment - smoke scenario DEMONSTRATES loop termination as a real CLI sequence (marked commit, merge commit, and the fixpoint that needs no marker) - Tasks: T-2 ratchet writer, T-3 flag + CI wiring, T-4 loop termination, T-5 degradation on refused write Refs: #372
…doption (T-6) Surgical edits only — three files carry live hunks from parked PRs #389/#390, so each change is a single-line replacement with no reflow or reordering. - way-of-working.md: `Coverage baseline commit-back: disabled` appended to the existing Coverage guardrail bullet, with the credential requirement and why it stays off here until #234's protection lands and the secret exists - tier-aware-pipeline.md (both corpora): the nested opt-in, the push-not-PR trigger, the bot-PR landing, monotonicity, termination and the degrade-to-warning contract - coverage-baseline.md: what writes to this file and when, and that a human remains the only writer while the flag is disabled Two dead markdown links removed: both pointed at adr-018-pr-state-flow-required-checks.md, which exists only in parked PR #390. The local gate does not catch these — the broken-.md-link check is an e2e test, outside the local gate set — so they would have surfaced in CI or, worse, only once #390 merged. Refs #372 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verdict
PR: #405 · Author: rucka · Reviewer: independent reviewer (agent) · Date: 2026-07-31 · Story: US-372 · Type: feature Classification matrix — per dimension
Tier = max(assessed) = yellow, unchanged from refinement (no drift note). Cost = green. Review value is a floor (D17): confirmed, never lowered.
AssessmentsSecurity — Input validationVerdict: yellow — attacker-controlled commit message is correctly env-passed, but a PR-influenced coverage number is written unvalidated into Details
Security — Output handlingVerdict: yellow — refusal output is echoed verbatim into a Details
Security — AuthenticationVerdict: yellow — dedicated PAT via env, but the credential layering is ambiguous (finding 1). Details
Security — AuthorizationVerdict: green — least privilege in both the scope and the event dimension. Details
Security — Introduced vulnerabilitiesVerdict: yellow — 0 introduced red, 1 introduced low/hardening, 0 pre-existing. Details
No committed secrets (the CostVerdict: Details
Architecture (Coupling)Verdict: green — balanced; the workflow↔module contract is explicit and test-pinned. Details
DetailsFindings by severityCritical (must fix before merge)
Major (should fix before merge)
Minor (consider)
Questions
Positive feedback
Functionality & requirements (AC coverage)
Testing & quality gates
Adoption compliance
Tech debt
Documentation
Performance & deployment
|
… guarded I/O, anchored flags Code-side findings from PR #405's review. Work started by an implementer the watchdog killed; recovered from the worktree, verified, completed and refactored rather than redone. M1 credential layering — `extraHeader` is MULTI-valued and `actions/checkout` already persists the read-only GITHUB_TOKEN into the same key, so ADDING ours made git send two Authorization headers and left the winner to the server. The likely loser was ours: a refusal that AC6 turns into a warning on every push — a feature that never works and never says so. `gitAuthConfig` + `EXTRAHEADER_CONFIG_KEY` now override the single key instead of layering. M2 unguarded happy path — every file/exec call could throw and exit non-zero, contradicting AC6's own rule that persistence never changes the gate's verdict. `main` now funnels everything except an argument-parsing bug (a workflow-authoring error, which must be loud) into the same warning + exit 0. M3 concurrent clobber — the story's exceptional scenario was unimplemented. `applyRaises` re-checks every proposal against BOTH the text on disk and the config as committed on the open ratchet branch, so a higher value written by a concurrent run is never clobbered on either ref. Minor — the flag parse was an unanchored whole-document regex, first match wins, and the KB sentence this PR ships contains the literal flag string: an adopter quoting that sentence would have turned the flag ON. Now anchored to a list bullet. `readGuardrailFlag` makes the "nested under the guardrail" framing enforced rather than documentation-only. The loop guard no longer fires on a commit message that merely mentions the ratchet branch. `git config --add remote.fetch` no longer mutates the checkout's persisted config, so the plan's "leaves no trace" is now true; `restore` is `--mixed` + a checkout of the one edited path, not `--hard`, so it cannot revert unrelated tracked work before the E2E step. `--margin` and the measured values are validated instead of silently becoming NaN (which would have made the ratchet never raise, quietly). `ratchetGitPlan` split via `ratchetWriteCommands` — it had grown to 64 lines against a 50-line ceiling. Caught by the pre-push gate, not by the test run: vitest was green while lint was red, the same trap a PR fell into earlier today. 80 tests (was 70). `pnpm quality-gate` green. Refs #372 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r capability yet (M4) The shipped, adopter-facing KB documented the flag, the trigger, the bot-PR landing and the credential as if an adopter could enable it. They cannot: the logic lives in packages/knowledge-hub/src/tools/ (pair-internal — the shipped assets are coverage-gate.sh and tier-resolve.sh), and /pair-capability-setup-gates never asks about the nested flag, never records it and never emits the step. An adopter setting `Coverage baseline commit-back: enabled` got a SILENT NO-OP — the worst shape for an opt-in: config says on, docs say what it does, nothing happens, nothing complains. Both the guideline and the config example now state the pair-internal scope, in both corpora, and point at #409 for the adopter path. Deliberately NOT fixed here: the stale "automated commit-back is story #372" note in setup-gates/SKILL.md:199. It is one line, but that file is modified by BOTH parked PRs #389 and #390 — a three-way contention in a PR whose disjointness is a stated business rule. It belongs to #409, which has to edit that skill anyway to add the flag to the interview. Refs #372 · #409 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`$GITHUB_ENV` is the classic Actions environment-injection sink: a value carrying a newline defines arbitrary further variables for every later step in the job. The two measured percentages were written unvalidated. These come from a coverage report, so a stray value is a tooling bug rather than an attack — but the guard is one case statement and removes the sink outright. A non-numeric value is dropped with a `::warning::`, so the ratchet sees no measurement and proposes nothing: the conservative outcome, and it still cannot block the gate (AC6). Verified: a value of `20\nEVIL=1` is rejected and only the legitimate `85.04` reaches the file; `sh -n` clean on the snippet. Refs #372 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remediation report — PR #405 · story #372Closes the loop on the review above. 16 findings (0 Critical · 4 Major · 10 Minor · 2 Questions) → 0 open actionable findings, with two deliberate deferrals named below. Head at verification: Fixes landed in three commits: Major — all 4 resolved
Minor — 9 resolved, 1 deferred
Questions — both answered in the PR body
One thing the tests caught that the review had not
Verdict requested: da CHANGES-REQUESTED ad APPROVED, with the two named deferrals (#409 for the adopter path, the template line for the post-#390 rebase) carried to the merge gate as accepted, not hidden. |
…rs the repo's write scripts 11 findings (2 Major, 9 Minor), all resolved. Major: - Rebased onto current origin/main (twice — #389/#391, then #405/#408 landed mid-round) and cleared main's format baseline in its own commit, so `format:check` is green on the tree this actually lands on (AC3). The sync-version-in-docs.ts conflict with #391 keeps BOTH docstrings. - The advertised remedy was a trap: `pnpm format` fixes the dataset SKILL.md and cannot reach its generated .claude twin (not a workspace member), while skill-md-mirror asserts byte equality — so format:check-green became skills:conformance-red later in the SAME gate. Reproduced end-to-end. The two-step remedy is now stated in all three places that advertise it (DEVELOPMENT.md, the mdx twin, PRE_PUSH_REMEDY) and unit-tested; the structural fix is noted on #414. Minor: - Guard widened to the repo's real write scripts: `sync-version` (→ sync-version-in-docs.ts, writeFileSync across every .md/.mdx it walks; `--check` dry-run spared, bounded to the same command segment) and `test:perf` (→ benchmark-update-link.ts, no dry-run, banned outright). 6 tests, verified RED. The ADL + way-of-working now state that what is enforced is the explicit list, not the invariant in general. - `_reanchor-gitignore.awk`: `[!abc]` → `[^abc]` (ERE negates with ^, gitignore with !) — unfixed it matched a literal `!`, i.e. the inverse set, silently dropping the pattern. Pinned by a row in the table-driven smoke block (`apps/[!x]ebsite/build/`), verified RED. Known-approximation note extended to cover bracket expressions and the `[]abc]` leading-`]` case. - `_ignore-file.sh` "Effect:" now warns the EXIT trap REPLACES the caller's. - `format:check` aggregates instead of short-circuiting, so one run names both prettier AND markdownlint drift. Verified: the `&&` form hid the markdown violation entirely. - turbo.json `//` comment records the globalDependencies blast radius as a deliberate over-approximation, and why the per-task `inputs` option was passed over. - DEVELOPMENT.md and the mdx twin are now byte-identical modulo the link form (a `diff` of the two paragraphs is a one-line diff when in sync); command blocks mirrored both ways (`mdlint:check` added, `(prettier only)` added). The two remaining PR-description findings (a false "were removed" parenthetical and the stale #389/#391 merge-order section) are fixed in the PR body, not here. NOTE: `pnpm turbo test` is red on this base for TWO reasons that pre-date and have nothing to do with #394 — both reproduced on a pristine origin/main worktree: skill-md-mirror on process/bootstrap/quick-mode-defaults.md (#408 merged an unregenerated .claude mirror — the very coupling this round documents) and code-host-routing AC4 on assess-cost/SKILL.md:106 (#388 prose + #389 audit merge-order collision). main's own CI is failing.
…rs the repo's write scripts 11 findings (2 Major, 9 Minor), all resolved. Major: - Rebased onto current origin/main (twice — #389/#391, then #405/#408 landed mid-round) and cleared main's format baseline in its own commit, so `format:check` is green on the tree this actually lands on (AC3). The sync-version-in-docs.ts conflict with #391 keeps BOTH docstrings. - The advertised remedy was a trap: `pnpm format` fixes the dataset SKILL.md and cannot reach its generated .claude twin (not a workspace member), while skill-md-mirror asserts byte equality — so format:check-green became skills:conformance-red later in the SAME gate. Reproduced end-to-end. The two-step remedy is now stated in all three places that advertise it (DEVELOPMENT.md, the mdx twin, PRE_PUSH_REMEDY) and unit-tested; the structural fix is noted on #414. Minor: - Guard widened to the repo's real write scripts: `sync-version` (→ sync-version-in-docs.ts, writeFileSync across every .md/.mdx it walks; `--check` dry-run spared, bounded to the same command segment) and `test:perf` (→ benchmark-update-link.ts, no dry-run, banned outright). 6 tests, verified RED. The ADL + way-of-working now state that what is enforced is the explicit list, not the invariant in general. - `_reanchor-gitignore.awk`: `[!abc]` → `[^abc]` (ERE negates with ^, gitignore with !) — unfixed it matched a literal `!`, i.e. the inverse set, silently dropping the pattern. Pinned by a row in the table-driven smoke block (`apps/[!x]ebsite/build/`), verified RED. Known-approximation note extended to cover bracket expressions and the `[]abc]` leading-`]` case. - `_ignore-file.sh` "Effect:" now warns the EXIT trap REPLACES the caller's. - `format:check` aggregates instead of short-circuiting, so one run names both prettier AND markdownlint drift. Verified: the `&&` form hid the markdown violation entirely. - turbo.json `//` comment records the globalDependencies blast radius as a deliberate over-approximation, and why the per-task `inputs` option was passed over. - DEVELOPMENT.md and the mdx twin are now byte-identical modulo the link form (a `diff` of the two paragraphs is a one-line diff when in sync); command blocks mirrored both ways (`mdlint:check` added, `(prettier only)` added). The two remaining PR-description findings (a false "were removed" parenthetical and the stale #389/#391 merge-order section) are fixed in the PR body, not here. NOTE: `pnpm turbo test` is red on this base for TWO reasons that pre-date and have nothing to do with #394 — both reproduced on a pristine origin/main worktree: skill-md-mirror on process/bootstrap/quick-mode-defaults.md (#408 merged an unregenerated .claude mirror — the very coupling this round documents) and code-host-routing AC4 on assess-cost/SKILL.md:106 (#388 prose + #389 audit merge-order collision). main's own CI is failing.
…rs the repo's write scripts 11 findings (2 Major, 9 Minor), all resolved. Major: - Rebased onto current origin/main (twice — #389/#391, then #405/#408 landed mid-round) and cleared main's format baseline in its own commit, so `format:check` is green on the tree this actually lands on (AC3). The sync-version-in-docs.ts conflict with #391 keeps BOTH docstrings. - The advertised remedy was a trap: `pnpm format` fixes the dataset SKILL.md and cannot reach its generated .claude twin (not a workspace member), while skill-md-mirror asserts byte equality — so format:check-green became skills:conformance-red later in the SAME gate. Reproduced end-to-end. The two-step remedy is now stated in all three places that advertise it (DEVELOPMENT.md, the mdx twin, PRE_PUSH_REMEDY) and unit-tested; the structural fix is noted on #414. Minor: - Guard widened to the repo's real write scripts: `sync-version` (→ sync-version-in-docs.ts, writeFileSync across every .md/.mdx it walks; `--check` dry-run spared, bounded to the same command segment) and `test:perf` (→ benchmark-update-link.ts, no dry-run, banned outright). 6 tests, verified RED. The ADL + way-of-working now state that what is enforced is the explicit list, not the invariant in general. - `_reanchor-gitignore.awk`: `[!abc]` → `[^abc]` (ERE negates with ^, gitignore with !) — unfixed it matched a literal `!`, i.e. the inverse set, silently dropping the pattern. Pinned by a row in the table-driven smoke block (`apps/[!x]ebsite/build/`), verified RED. Known-approximation note extended to cover bracket expressions and the `[]abc]` leading-`]` case. - `_ignore-file.sh` "Effect:" now warns the EXIT trap REPLACES the caller's. - `format:check` aggregates instead of short-circuiting, so one run names both prettier AND markdownlint drift. Verified: the `&&` form hid the markdown violation entirely. - turbo.json `//` comment records the globalDependencies blast radius as a deliberate over-approximation, and why the per-task `inputs` option was passed over. - DEVELOPMENT.md and the mdx twin are now byte-identical modulo the link form (a `diff` of the two paragraphs is a one-line diff when in sync); command blocks mirrored both ways (`mdlint:check` added, `(prettier only)` added). The two remaining PR-description findings (a false "were removed" parenthetical and the stale #389/#391 merge-order section) are fixed in the PR body, not here. NOTE: `pnpm turbo test` is red on this base for TWO reasons that pre-date and have nothing to do with #394 — both reproduced on a pristine origin/main worktree: skill-md-mirror on process/bootstrap/quick-mode-defaults.md (#408 merged an unregenerated .claude mirror — the very coupling this round documents) and code-host-routing AC4 on assess-cost/SKILL.md:106 (#388 prose + #389 audit merge-order collision). main's own CI is failing.
PR Information
Story: #372 · Type: Feature · Priority: Medium (P2) · Assignee: @rucka
Classification:
risk:yellow·cost:greenSummary
What Changed
CI can now persist a raised coverage baseline back to
tech/coverage-baseline.md— opt-in, off by default, ratcheting only upward.coverage-baseline-ratchet.ts— all the decisions in one unit-tested module: the skip predicate, the monotonic per-type plan, the in-place value edit, the git/ghcommand plan and the refusal classification.ci.yml— a thin entrypoint (ADL 2026-07-13) that always exits 0. The guardrail step now also hands its measured numbers forward, so extraction is not duplicated.scripts/smoke-tests/scenarios/coverage-gate.shdrives the CLI end to end, which is how AC4's termination is demonstrated rather than asserted.2026-07-30-coverage-ratchet-pr-not-push.md.Why This Change
Raising a baseline relied on someone remembering to bump the file. The default stays human-committed for projects that want the baseline under review; this adds the automation as a nested opt-in.
Story Context
As a maintainer running the coverage guardrail I want CI to persist a raised baseline back, opt-in and ratcheting only upward So that improvements are locked in automatically.
AC1 default off · AC2 raise above baseline · AC3 never lowers · AC4 no CI loop · AC5 PR-vs-push decided explicitly · AC6 refused write degrades to a warning.
Changes Made
Implementation Details
T-1 was the real decision, and it was taken against the post-#234 regime rather than today's⚠️ dependency). PR #390, parked open, makes
main(the story'spair-reviewandpair-explicit-approvalrequired contexts and setsenforce_admins: true. Under that regime:mainis structurally impossible for CI — required checks are evaluated on pushes to a protected branch, and a workflow commit can never carrypair-explicit-approval, which is produced by a human approving a PR. The "simplest" option in the story body is not merely awkward after [US-234] feat: PR state flow (gate≠review) + pair review as required check #390: it is rejected by the host.head.sha— the exact commit both required contexts are pinned to. A raise would invalidate an approval already given, forcing re-approval on every coverage improvement. The automation would consume the very control PR state flow (gate≠review) + pair review as required check #234 introduces.So: the raise is proposed only by a push to the base branch, and lands as a bot pull request from
chore/coverage-baseline-ratchet— never a push to a protected branch, never a mutation of a PR under review. This is why the story lands working and stays working the day #390 merges.Credential: a dedicated repo-scoped
COVERAGE_RATCHET_TOKEN(contents: write+pull requests: write, no protection bypass). The default CI token is unusable on purpose — a PR it opens triggers no workflow run, so that PR could never satisfy required checks.Two security properties worth a reviewer's eye:
github.event.head_commit.messageis attacker-controlled text. It is read viaenvand never interpolated into the script body — the loop-guard marker check cannot become shell injection.pushonmain, so a pull-request run — where the diff under test can influence what the job executes — never has the credential in its environment. That is belt-and-braces over the module's ownnot-base-pushskip.AC4 termination is a real predicate, not luck: a run whose head commit carries the
[coverage-baseline-ratchet]marker is skipped, and the value written isfloor(measured) - 1pp, so an unchanged coverage proposes what is already committed and the plan comes out empty.Files Changed
coverage-baseline-ratchet.{ts,test.ts}, the ADL (+79)scripts/smoke-tests/scenarios/coverage-gate.sh(+188/−0) — an existing 195-line scenario from Coverage baseline + CI guardrail (tier-aware pipeline gate) #282, extended here, not a new file (corrected: review Minor).github/workflows/ci.yml(+34),knowledge-hub/package.json, and surgically:way-of-working.md,coverage-baseline.md,tier-aware-pipeline.md+coverage-config-example.md+coverage-gate.sh(both corpora)Testing
Test Coverage
60 unit tests, one
describeper contract: margin convention · config read (^baseline.<type>=<int>only) · flag read (default disabled) · skip predicate (AC1/AC4/AC5) · monotonic plan · in-place edit with surrounding markdown untouched · refusal classification (AC6) · the exact git/ghcommand sequence · rendered output · module constants that the CI step and the docs both depend on.Test Results
pnpm quality-gategreen — 22 + 12 turbo tasks successful.Quality Assurance
Review Areas
floor(measured) - 1ppthe right written value? It doubles as the termination guarantee, so changing the margin changes convergence.Notes for reviewers
Two dead markdown links were introduced during implementation and removed before this PR. Both pointed at
adr-018-pr-state-flow-required-checks.md, which exists only in parked PR #390. Worth flagging as a class of defect rather than an incident:pnpm quality-gatedoes not catch them — the broken-.md-link check is an e2e test, outside the local gate set. They would have surfaced in CI, or worse, only once #390 merged.Deliberate deferral — the framework template does not list the new flag.
dataset/.pair/adoption/tech/way-of-working.mddocumentsCoverage guardrail: disabled (default)at line 17, and PR #390's hunk on that file is-15,6— covering exactly that line. Adding theCoverage baseline commit-backline there would be a head-on conflict. The framework default is documented, intier-aware-pipeline.md(both corpora), and an absent flag reads asdisabled. The one-line template addition belongs to the cascade rebase after #390 merges.Expected rebase conflicts
Declared per the story's constraint, in likelihood order:
.pair/adoption/tech/way-of-working.md.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md(+ dataset twin)Open questions answered (review)
AC2 says the raised value is "written … and committed back" — under the bot-PR model, is it? Persistence is proposed, not applied: the ratchet opens a PR and a human merges it. That is a deliberate narrowing forced by the post-#234 regime, not an omission — a CI push to a protected base branch is rejected by the host, and writing to the PR branch would mutate the
head.shaboth required contexts are pinned to, invalidating an approval already given. The ADL records the reasoning. AC2's wording is stricter than what any design can deliver under branch protection; the behaviour here is the closest honest reading, and the divergence is called out rather than glossed.Has the enabled path ever executed? No. The smoke scenario runs
--dry-runplus a no-token refusal, and pair itself staysdisabled, sogit push+gh pr createwith a real PAT has never run anywhere. Everything about the enabled path is unit-verified at the plan level (the exact argv sequence is asserted) and unexercised end-to-end. First enablement should therefore be treated as a live test, on a throwaway branch, before anyone relies on it — and it cannot happen at all until #234's protection is applied andCOVERAGE_RATCHET_TOKENis provisioned. Stated here so the merge gate does not read "80 tests" as "the feature has run".Dependencies & Related Work
coverage-gate.sh.disabledhere until that protection is applied and the secret is provisioned.Closes #372
🤖 Generated with Claude Code