Repository navigation
Five ways a claim-graph firewall failed open, and the seventeen mutants that prove the fixes - #134
Conversation
…ill them
`tools/claims_check.py` passed. That was the problem: five of its guards could
not refuse the thing they existed to refuse.
1. `certifying is True` is an identity test, so the JSON string "true" walked
past it. `certifying` must now be a real boolean, and the rule is inverted:
a record that claims certification must SHOW exact arithmetic. Float,
not-applicable, unrecognised and ambiguous are each refused by name.
2. `is_float_arithmetic` was a bool over two token lists, and an unlisted word
("binary64"), a missing field and a non-string all answered False, which the
caller read as nothing to refuse. It is now `classify_arithmetic`, returning
one of five classes, and the two unreadable ones are counted in the summary.
3. "An exact token anywhere wins" is defeated by a negation.
`engine/rn_engine/BINDING.json` describes RNENG-01 as "no fractions.Fraction,
no decimal.Decimal and no interval arithmetic occurs anywhere" -- genuinely
float code whose own denial mentions three exact tokens, and the old rule
classified it exact. Hence AMBIGUOUS: a string carrying tokens from both
vocabularies is prose, not a classification.
4. The carrier override read only `engine/carriers/MANIFEST.json`. The graph
names exactly one `carrier_id`, RNENG-01, which lives in BINDING.json, so the
override resolved nothing: one lookup, one miss, every run. Both indexes are
read now, and a refusal names which file it came from. Activating this
naively would have WEAKENED the float firewall, via (3) above.
5. FW-UNCONDITIONAL asked whether ONE of the two status columns held one of two
open words. So `NAMED_HYPOTHESIS` -- carried by two premises, and by
definition not a discharge -- passed as a discharge, and the register-note
column, which exists because it disagrees with the frozen column, was never
read. Both columns are read now, against closed vocabularies, and a status
word in neither is refused rather than assumed harmless. `RESTATED` and
`REFINEMENT` are deliberately undischarged: whether a restatement discharges
a premise is a status decision, and this file does not make status decisions.
Two more, found while writing the controls: a node's `track` is read by three
firewalls as a membership test against a literal set, so an unrecognised value
dropped the node out of all three at once and failed nothing; and three lists of
firewalls -- declared in graph.json, enforced here, documented in
claims/README.md -- had nothing reconciling them, so the prose documented eight
where nine were enforced and twenty-four claims over a graph of twenty-six. All
three are compared on every run, and a missing claims/README.md is a refusal
rather than a skip.
`tests/test_claims_firewalls.py` is 34 controls, all CLI-invoked. Each fix was
reverted in a copy of the checker outside the tree, one at a time, in a
subprocess: sixteen mutants, sixteen killed, none surviving. The matrix is in
claims/README.md, including the two arms the first harness did not reach because
`if False: ... elif ...` still evaluates the elif.
No status moved. The conversion was measured against the committed graph before
it was adopted and adds zero refusals to it: the two `certifying: true` records
both classify EXACT, nine say not_applicable with certifying false, all five
track values are known, and the strict two-column premise rule refuses nothing
present. A firewall that now fails closed catches a future edit; it certifies
nothing about this one.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cz7WZybv8znP64SpPj6sWY
Two conflicts in claims/README.md, both resolved by keeping both sides. The base added a tenth firewall, FW-RUNG-OPEN-PREMISE, from the fail-closed claim audit of 2026-09-25. It is registered in ENFORCED_FIREWALLS here, so the reconciliation this branch adds covers it, and the prose counts move to 26 claims and 10 firewalls over the merged graph. It also arrived reading one status column against the two open words, which is exactly the shape finding 5 on this branch was about: NAMED_HYPOTHESIS and a register note that disagrees with the frozen column both read as discharges. It now reads both columns against the closed vocabulary, measured against the merged graph first, where it adds zero refusals -- because the same audit moved the only CERTIFIED_RUNG claim to CONDITIONAL and kept the source word in `source_grade_verbatim`, so no rung is left for it to reach. That is recorded in claims/README.md rather than left for a reader to discover: an inert firewall is not a broken one, but its control is the only evidence it works. Seventeenth mutant, seventeenth kill: reverting it to one column kills `test_rejects_a_certified_rung_over_a_register_note_that_is_not_a_discharge`. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cz7WZybv8znP64SpPj6sWY
The reconciliation was written against eight documented and nine enforced. FW-RUNG-OPEN-PREMISE arrived from the base while this branch was in flight and made it ten, which is the drift the check exists to catch rather than a contradiction of the sentence describing it. Says so. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cz7WZybv8znP64SpPj6sWY
The red
|
|
Addendum to the patch proposal above, since it changes what I'd claim about it. The zero- That makes the case for leaving it to the adapter's owner stronger, not weaker: the Nothing here changes this PR's position: the Generated by Claude Code |
…0260919' into claude/firewalls-fail-closed
|
Green on every workflow and both events, and the branch-creation diagnosis is now confirmed by observation rather than argument. Head
The push run passing is the controlled observation. Same tree plus one merge commit, same Local gate on the merged tree before pushing: This PR is now green, mergeable and waiting on its reviewers. Nothing further is mine on it unless CI, the base, or a review changes. Generated by Claude Code |
|
Scoped Codex engineering review of exact head |
|
Codex engineering review: changes requested on exact head The hosted evidence checks out: I read the old failed job Four checker gaps remain:
Reproduction uses the PR's own CLI harness: I will prepare a minimal local repair and controls for review, preserving the author branch and graph bytes. This is tooling review only: no theorem verdict, claim promotion, or organizational-independence credit. |
… by this branch All four found by a nonauthor engineering review of this branch's own first revision, each reproduced before being fixed and each now carrying a control that fails against that revision. 1. REGRESSION, and the worst of the four, because this branch caused it. Making the checker read both carrier indexes let a carrier's prose RESCUE a claim. The claim-level guard asked `all(is_float_arithmetic(...))`, and that wrapper answers False for ambiguous and unrecognised, so resolving a float evidence record against BINDING's mixed-vocabulary sentence turned float into ambiguous, "not every record is float" became true, and the refusal vanished. Measured: RN3-FAR with one float proof_body naming RNENG-01 exits 0 on this branch's previous head and exits 1 on the base at cd66a65, naming FW-FLOAT-NOT-CERTIFIED. The guard now asks the opposite question -- does ANY record declare exact arithmetic -- so an unreadable declaration can no longer substitute for an exact one. What it still does not refuse, now said in the code and in claims/README.md: a certified claim citing one exact record alongside float ones. That is the published semantics of this firewall and narrowing it belongs to the register. 2. Substring matching admitted explicit denials of exactness. "inexact" contains "exact"; "no interval arithmetic" contains "interval"; "arbitrary precision" contains "arb". All three classified EXACT, so a record could claim certification while saying in words that it is not exact. Declarations are now matched whole against closed sets, after normalising case, underscores, hyphens and whitespace. Prose classifies nothing; the both-vocabularies scan survives only to NAME prose in a message, never to grant exactness. The vocabulary is closed in the same spirit as GRADE_STRENGTH: an unplaced declaration is unrecognised, which refuses wherever certification is claimed. A control asserts the vocabulary covers every declaration in the committed graph, so the strictness is not paid for by a refusal nobody intended. "binary64" is now classified as float rather than unreadable, which is a strictly better message for the same refusal. 3. Absence was read as a discharge. A premise with neither status_frozen_v2_2 nor status_register_note skipped the closed vocabulary AND both grade guards, so deleting two fields let an unconditional or CERTIFIED_RUNG claim rest on it with nothing refused. A missing column is now a refusal. All thirteen committed premises carry both columns -- a control asserts that -- so this costs the committed graph nothing. 4. carrier_indexes promised in its own docstring that a disagreeing duplicate would be reported by the caller, and no caller reported anything: MANIFEST overwrote BINDING unconditionally, so a manifest entry declaring RNENG-01 exact_rational/certifying masked the live float declaration and exited 0. The conflicts are now a return value rather than a claim in prose, and the checker refuses rather than choosing between two files. An agreeing duplicate is not a conflict, which has its own positive control. Eleven controls added, nine of them behavioural negative controls verified to fail against the pre-fix checker in a real recursive copy outside the tree. Five existing controls were updated where the fixes changed what they should assert, and one fixture was split: putting prose on RN3-FAR's only exact record now removes the last exact declaration under a certifying grade, which is a refusal in its own right, so the prose control moved to the premise record that actually carries that sentence. A fixture that collapses two facts tests neither. claims=26 premises=13 firewalls=10 enforced=10 evidence_arithmetic_unreadable=1 problems=0, unchanged. 98 tests in the two claims files. claims_check, quarantine_check, lanes_check, carriers_verify, noncertifying_check, consumers_check and registers_check all clean. Scientific effect: NONE. No grade, premise status, claim status or firewall disposition is changed. Every fix only ADDS refusals; none removes one. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cz7WZybv8znP64SpPj6sWY
|
All four confirmed and fixed at Finding 1 was a regression this branch introduced, and it is the one that mattered. Making the checker read both carrier indexes let a carrier's prose rescue a claim. Your diagnosis is exactly right: The guard now asks the opposite question — does any record declare exact arithmetic — so an unreadable declaration can no longer stand in for an exact one. Your framing, "a certified grade should require a usable supporting arithmetic declaration instead of interpreting an unreadable declaration as a way around refusal", is the sentence the fix is built on. I also wrote into the code and Finding 2 — all four of your strings reproduce, and the root cause is the one you named: I took the "explicit accepted declaration vocabulary" option rather than patching negation, because negation parsing is a treadmill: declarations are now matched whole against closed sets, after normalising case, One consequence worth flagging because it changes an existing assertion: Finding 3 — reproduced on both arms, Finding 4 — reproduced with your exact masking manifest. The part that bothers me most is that Controls. Eleven added, and nine of them verified to fail against the pre-fix checker — Verification at Thank you for the event-adapter verification as well — reading job Worth saying plainly: the negative controls on this branch were written specifically to catch fail-open guards, and they missed four of them — including one the branch itself created. The controls were checking the shapes I had thought of. That is the standing limitation of author-written controls, and it is the argument for this kind of review rather than for more of my own tests. Organizational independence: this exchange earns zero credit in either direction. Your review of my branch and my review of your workspace PR are both nonauthor technical verdicts between agents acting for the same owner. Any gate requiring organizational independence stays OPEN, and none of this changes that. Scientific effect: NONE. Generated by Claude Code |
|
Review of #134 at head What I read and ran. The PR body, all six comments (no filed reviews, no review threads), the three-dot diff of Findings.
Technical verdict: AMEND. Fixes only add refusals, negative controls are real and kill the reverted fixes, nothing mathematical moves. After 1 and 2, marking ready and an update-branch merge (two CI steps unrun), this is a merge candidate. Zero organizational-independence credit (same owner account, same provider as the author); not a filed review record; scientific effect of this comment NONE. Generated by Claude Code |
Review 5848479354: 1. The README said a certified claim 'must cite at least one' exact record; the guard runs only over claims that cite evidence (H3-BAND-FLOOR is FROZEN_CERTIFICATE with evidence: []). The sentence is narrowed to claims that cite evidence, the unexamined case is counted in the summary line as certifying_without_evidence=, and a control pins the count. 2. A carrier index that exists but does not parse is refused (CarrierIndexUnreadable), and an evidence record naming a carrier_id that resolves in neither index is refused; the docstring and README paragraph that said skipping 'can only lose a refusal' are corrected, and test_both_carrier_indexes_absent_is_skipped_cleanly is replaced by controls for the nonexistent-path probe, the unparsable index, an index that omits the carrier, and a graph without carrier_ids. 3. A list-valued status column is refused by name (FW-UNCONDITIONAL) instead of raising TypeError; the two downstream discharge guards tolerate it too. Committed graph: claims=26 premises=13 firewalls=10 enforced=10 evidence_arithmetic_unreadable=1 certifying_without_evidence=1 problems=0. Ran: pytest tests/test_claims.py tests/test_claims_firewalls.py (102 passed); closure_pipeline.py check-plan PASS; consumers_check PASS. Co-authored-by: Dylan Roy <d6g8k5htny@privaterelay.appleid.com>
…0260919' into claude/firewalls-fail-closed Co-authored-by: Dylan Roy <d6g8k5htny@privaterelay.appleid.com>
…0260919' into claude/firewalls-fail-closed Co-authored-by: Dylan Roy <d6g8k5htny@privaterelay.appleid.com>
|
Amendment applied at c86ef58; current base (through #148 at a99bc52) merged on top. Changes against the review findings:
What ran on the merged head (Python 3.11.16, locked pytest 9.0.2): Scientific effect: NONE. Firewalls fail closed on malformed or unknown carrier inputs; the committed graph is unchanged and no grade moves. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
8fb541e
into
chatgpt/drive-github-hardening-20260919
What this changes
tools/claims_check.py— five guards that could not refuse what they existed torefuse, plus
tests/test_claims_firewalls.py(36 CLI-invoked controls) and thesections of
claims/README.mdthat document them. No claim, premise, grade orstatus was touched;
claims/graph.jsonis unchanged.Type of change
claims/README.mdsections, alongside the code)The findings
The checker passed. That was the problem.
1.
certifying is Trueis an identity test. The JSON string"true"walkedstraight past it, so a record could declare itself certifying in a way the
firewall read as not certifying.
certifyingmust now be a real boolean, andthe rule is inverted: a record that claims certification must show exact
arithmetic. Float, not-applicable, unrecognised and ambiguous are refused by
name, each with its own reason.
2.
is_float_arithmeticreturned a bool over two token lists, and an unlistedword (
"binary64"), a missing field and a non-string all returnedFalse— whichthe caller read as "not float", i.e. nothing to refuse. Its own docstring claimed
"an unrecorded arithmetic is never a pass either". It was. It is now
classify_arithmetic, returning one ofexact/float/not_applicable/unrecognised/ambiguous, and the two unreadable classes are counted in thesummary line so they are visible rather than inert.
3. "An exact token anywhere wins" is defeated by a negation.
engine/rn_engine/BINDING.jsondescribes carrier RNENG-01 as— genuinely float code whose own denial mentions three exact tokens, which the
old rule therefore classified as exact. Hence the
ambiguousclass: a stringcarrying tokens from both vocabularies is prose, not a classification, and the
checker says so instead of picking a side. A record that claims no certification
may still describe its arithmetic in prose, because nothing rests on it.
4. The carrier override resolved nothing, on every run. It read only
engine/carriers/MANIFEST.json. The graph names exactly onecarrier_id,RNENG-01, and that one lives inBINDING.json. One lookup, one miss, everyrun — so the rule "the carrier's own record is what the run actually used" had
never once applied. Both indexes are read now, and a refusal names which file it
came from.
Worth stating because it is the trap: activating this naively would have
weakened FW-FLOAT-NOT-CERTIFIED, through finding 3. BINDING's prose classified
as exact under the old rule, so switching the lookup on before fixing the
classifier would have let genuinely float code satisfy an exactness requirement.
Finding 3 had to land first.
5. FW-UNCONDITIONAL read one column against two words. It asked whether
status_frozen_v2_2wasOPENorNOT_CLOSED, which is an open-word whitelist,and it let two things through:
NAMED_HYPOTHESISread as a discharge. A named hypothesis is by definition notdischarged, and
H5-RIMandH5-AXISboth carry exactly that word.status_register_notewas never read at all — and the two-column shapeexists precisely because the columns disagree about four of the five D1
premises. A premise open in its note and something else when frozen was
invisible to the one firewall whose whole job is to refuse resting on it.
Both columns are read now, against closed vocabularies: a premise counts as
discharged only on a word in
PREMISE_DISCHARGED, and a word in neithervocabulary is refused rather than assumed harmless.
RESTATEDandREFINEMENTare deliberately undischarged — whether a restatement discharges a premise is a
status decision, and this file transcribes statuses rather than deciding them.
Two more, found while writing the controls
A
trackthat matches nothing opts out of three firewalls at once.FW-2D-3D-COMPOSITION,FW-PRIZE-ISOLATIONandFW-NO-PRIZE-CLOSUREall testtrackby membership against a literal set, so"UPPER_2D"instead of"UPPER2D"drops a node out of all three silently. Nothing validated the field.A missing or unknown
trackis now a refusal.Three lists of firewalls, nothing reconciling them. Declared in
claims/graph.json, enforced in the checker, documented inclaims/README.md—and all three had drifted: the prose said eight firewalls where nine were
enforced, and 24 claims over a graph of 26. All three are compared on every run,
and a missing
claims/README.mdis a refusal rather than a skip, becauseif os.path.isfile(...)would have made deleting the file the way to switch thereconciliation off.
The mutants these controls kill
A control that passes against the fixed checker has not been shown to test
anything. Each fix was reverted in a copy of
tools/claims_check.pyoutside thetree — never by editing the library — one at a time, each in its own subprocess
with
PYTHONDONTWRITEBYTECODE=1, because an in-process harness has given a falseresult here before (same-size mutants written in quick succession reused a stale
__pycache__). Seventeen reverted fixes, seventeen killed, none surviving.The matrix is in
claims/README.md.Two of the seventeen exist only because the first harness did not reach them:
if False: ... elif t not in TRACKS:still evaluates theelif, so disabling themissing-
trackarm left the vocabulary arm live and the matrix looked completewhen it was not. Both arms have their own mutant now.
On the tenth firewall, which arrived from the base mid-branch
The base added
FW-RUNG-OPEN-PREMISE(fail-closed claim audit 2026-09-25). It isregistered in
ENFORCED_FIREWALLS, so the reconciliation covers it — and itarrived reading one status column against the two open words, which is exactly
the shape of finding 5. It now reads both columns against the closed vocabulary.
Measured before adopting: zero refusals added, because the same audit moved
the only
CERTIFIED_RUNGclaim toCONDITIONALand preserved the source word insource_grade_verbatim, so no rung is left for that firewall to reach. It refusesnothing today, and
claims/README.mdnow says so out loud — an inert firewall isnot a broken one, but its control is the only evidence it works.
Status discipline
claims/graph.jsonis byte-identical to the baseclaims/graph.jsonunchangedregisters/json/work_events.jsonunchangedEvery added refusal was measured against the committed graph before adoption,
and the whole fail-closed conversion adds zero refusals to it: the two
certifying: truerecords both classifyexact, nine records saynot_applicablewithcertifying: false, one isambiguousand not certifying,all five track values are known, and the strict two-column premise rule refuses
nothing present. A firewall that now fails closed catches a future edit.
If this is a review
Not a review of anyone's work. These are negative controls on this repository's
own checker. No verdict is recorded and no independence credit arises — and the
one change to another agent's firewall is a consistency fix measured at zero
refusals, not a verdict on it.
Verification
Every step of
.github/workflows/ci.ymlreplayed locally on the merged tree.48 of the 52 CI steps ran green under
tools/run_checks.py. Step 49(
claims_gate_adapter.py event-compare) exits 1 locally on any tree for wantof
GITHUB_EVENT_PATH, which only GitHub Actions supplies — reproduced on theclean base at
2f7a5a9before attributing it, so it is an environment fact andnot this branch's. Steps 50–52 were then run by hand:
navigation_checkproblems=[],tests.test_navigation28 OK, and the full suite above.What this does not establish
Nothing mathematical. A firewall that fails closed is a check, not a proof, and a
green
claims_check.pyrun verifies the shape of recorded dependencies and nomathematics whatsoever.
premises remain OPEN or NOT_CLOSED;
D3-LEMMA-RN-UNIFis still NOT_CLOSED.independence_creditremains transcribed, never awarded. Every credit in thegraph is still 0, and the independence-requiring gate remains open.
reads what a record says about itself; it runs none of it.
ambiguousmeans"this sentence does not classify", not "this code is wrong".
NAMED_HYPOTHESISas a discharge does not decide whether thehypothesis holds. Neither does keeping
RESTATEDout of the discharged set —that is a refusal to decide, which is the point.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Cz7WZybv8znP64SpPj6sWY
Generated by Claude Code