Skip to content

fix(asvs): survey progress counted a read-and-parked cell as never read - #150

Merged
wshallwshall merged 4 commits into
mainfrom
claude/asvs-security-handoff-0f696a
Aug 2, 2026
Merged

fix(asvs): survey progress counted a read-and-parked cell as never read#150
wshallwshall merged 4 commits into
mainfrom
claude/asvs-security-handoff-0f696a

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

ASVS-CURRENT.md — the CI-gated page that is the ASVS record under ADR 0156 — printed two different numbers for one quantity:

123 of 345 requirements have been read against the ASVS text (35.7%). 222 have not.

directly above its own table saying 221 unverified. 345 − 221 = 124.

The defect

Survey progress was computed from DECIDED_VERDICTS, which deliberately excludes needs-review. That exclusion is right for verdict counts — the cell was examined and then left open on purpose. But the survey line does not report what was decided, it reports what was read, and a needs-review cell has been read. The constant's own docstring says so: "the second was examined and left open on purpose."

The code used the right set for the wrong question.

The fix

EXAMINED_VERDICTS = DECIDED_VERDICTS | {"needs-review"}, used for progress only. Verdict counts keep DECIDED_VERDICTS, and so does the inherited warning — a parked cell is not an inherited verdict.

Why it survived every gate since the renderer was written

With zero needs-review cells the two sets are identical, so no test, no CI run and no reading of the output could distinguish them. It became observable only when the scorecard acquired its first one (11.4.4, in the V7/V11 baseline sweep) — a latent defect that needed new data, not new code, to surface. That is the shape this repo keeps finding: a control that passes because the case it gets wrong has never occurred.

Verification

  • Regression test asserts the two printed figures agree, since a reader comparing them is the only thing that ever noticed.
  • Proved red against HEAD before being trusted green. git stash is shared with the primary checkout and the worktree gate refuses it, so the renderer was reverted via git show HEAD: and restored afterwards.
  • ruff check + ruff format --check, mypy strict, 35/35 scorecard tests.

Data-only, no behaviour change outside the rendered progress line. The vault's mirrored copy of scripts/asvs/scorecard.py will be synced after merge — its gate now reds on drift between the two.

`ASVS-CURRENT.md` reported "123 of 345 requirements have been read ... 222 have not" directly above
its own table saying **221** unverified. Two numbers for one quantity, on the page that IS the record
under ADR 0156.

The renderer computed survey progress from `DECIDED_VERDICTS`, which deliberately excludes
`needs-review` -- correctly, for verdict COUNTS, since that cell was examined and then left open on
purpose. But the survey line does not report what was decided, it reports what was **read**, and a
`needs-review` cell has been read. The constant's own docstring says so: *"the second was examined
and left open on purpose."* The code used the right set for the wrong question.

Adds `EXAMINED_VERDICTS = DECIDED_VERDICTS | {"needs-review"}` and uses it for progress only. Verdict
counts, and the `inherited` warning about a decided verdict lacking a `last_verified` date, keep
`DECIDED_VERDICTS` -- a parked cell is not an inherited verdict.

**Why this survived every gate since the renderer was written.** With zero `needs-review` cells the
two sets are identical, so no test, no CI run and no reading of the output could tell them apart. It
became observable only when the scorecard acquired its first one (11.4.4, in the V7/V11 baseline
sweep) -- a latent defect that needed new *data*, not new code, to show itself. That is the shape
this repo keeps finding: a control that passes because the case it gets wrong has never occurred.

The regression test asserts the two printed figures agree, because a reader comparing them is the
only thing that ever noticed. Proved red against `HEAD` before being trusted green -- the renderer
was reverted with `git show HEAD:` and the file restored afterwards, since `git stash` is shared with
the primary checkout and the worktree gate refuses it.

ruff check + format, mypy strict, and all 35 scorecard tests pass.
@wshallwshall
wshallwshall enabled auto-merge (squash) August 2, 2026 15:54
@wshallwshall
wshallwshall merged commit d8afded into main Aug 2, 2026
32 checks passed
@wshallwshall
wshallwshall deleted the claude/asvs-security-handoff-0f696a branch August 2, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant