Skip to content

fix(ci): a failed fetch is not a finding — the audit withholds what it cannot see - #33

Merged
github-actions[bot] merged 1 commit into
masterfrom
ci/a-failed-fetch-is-not-a-finding
Aug 20, 2026
Merged

fix(ci): a failed fetch is not a finding — the audit withholds what it cannot see#33
github-actions[bot] merged 1 commit into
masterfrom
ci/a-failed-fetch-is-not-a-finding

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

What

Two defects in verify-floor-audit.sh itself, found by turning the audit's own discipline back on it:

1. A failed fetch was reported as a finding. Every remote read was gh api … 2>/dev/null, and the empty string a failure yields was read as a fact. One sweep produced four wrong verdicts: vitareba and aoz-housing (live Next.js apps) silently vanished from the report as "not a JS repo"; ai-forms and fleetcrown were accused of never running verify when their ci.yml does exactly that. The tell was arithmetic: two runs an hour apart inspected 24 and 22 repos with no repo created or deleted between them.

Fetches now go through a three-state gh_get: 0 = fetched, 2 = genuinely absent (404 — an answer, never retried), 1 = could not look (retried with backoff; callers withhold the verdict). An exhausted rate limit also fails fast — seconds of backoff cannot outlive an hour-long window.

2. ⊗ UNCALLED encoded its first example, not the property. It string-matched npm run verify, but the real contract is weaker: every gate verify composes runs in CI, unsoftened. aoz-housing satisfies it by fanning lint/typecheck/test into parallel jobs — the rule was asking a conforming repo to serialize the fleet-slowest pipeline to please a regex. That shape now reports as informational ≡ DECOMPOSED; botsmann's hand-copy-with---if-present is still caught.

Proof

  • gh_get lives in verify-predicates.sh so tests can stub gh — the original bug survived precisely because its failure path was unreachable without a real outage.
  • test-verify-predicates.sh: 29 passed, 0 failed, covering both rules in both directions and all three fetch states (incl. 404-not-retried and rate-limit fail-fast, each costing exactly one call).
  • Mutation proof: collapsing return 1 into return 2 (the original defect) turns exactly the two relevant cases red; baseline restored green.
  • Live sweep post-fix: 23 repos inspected, 0 below floor; the 3 fetches that failed mid-run were withheld under · unreadable instead of becoming verdicts — the fix demonstrating itself.

🤖 Generated with Claude Code

…at it cannot see

Two defects in the auditor itself, both found by turning its own
discipline back on it:

1. Every remote read was `gh api ... 2>/dev/null`, and the empty string
   a failure yields was read as a FACT. One sweep produced four wrong
   verdicts: vitareba and aoz-housing (live Next.js apps) vanished as
   "not a JS repo"; ai-forms and fleetcrown were accused of never
   running `verify` when their ci.yml does exactly that. Fetches now
   return three states — fetched / genuinely absent (404) / could not
   look — and every caller withholds its verdict on the third instead
   of charging the repo. A 404 is an answer and is not retried; neither
   is an exhausted rate limit (seconds of backoff cannot outlive an
   hour-long window). Transient failures retry with backoff.

2. `⊗ UNCALLED` string-matched `npm run verify`, but the property that
   matters is weaker: every gate `verify` composes runs in CI,
   unsoftened. aoz-housing satisfies it by fanning the gates into
   parallel jobs — the rule was asking a conforming repo to serialize
   the fleet-slowest pipeline to please a regex. Decomposed wiring now
   reports as informational `≡ DECOMPOSED`; botsmann's shape (hand-copy
   with --if-present on every step) is still caught.

The transport lives in verify-predicates.sh so tests can stub `gh`:
29 cases cover both rules in both directions and all three fetch
states, and a mutation run (collapse `return 1` into `return 2`)
turns exactly the two relevant cases red. Post-fix sweep: 23 repos
inspected, 0 below floor, 3 fetch failures withheld instead of
reported as findings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit 5bf16cb into master Aug 20, 2026
1 check passed
@github-actions
github-actions Bot deleted the ci/a-failed-fetch-is-not-a-finding branch August 20, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant