test(library): pair every no-fire assertion with a positive control - #14
Merged
Conversation
A test that only asserts a rule stayed silent cannot tell "the rule declined to
fire" from "the rule can never fire". Both are green. This is the same defect
class as the vacuous prefetch test fixed in Unlimited-Context-LLM yesterday,
which asserted a wall-clock bound on a branch that never executed.
tests/test_examples.py already uses the right pattern — every quiet frame is
paired with a firing frame on the SAME graph, so the graph is proven live before
it is proven silent. tests/test_library.py did not, in four places:
- consecutive_loss_circuit (added in the previous commit)
- correlation_cluster_guard (added in the previous commit)
- atr_volatility_halt (pre-existing)
- cpi_impulse_pullback_short (pre-existing; the short arm was only ever
asserted silent, so the mutual-exclusion claim
rested entirely on the long arm)
Each now runs a crossing frame through the same graph and asserts the exact
intents and timestamps. All four fired, so none of the rules was broken — but
nothing in the suite could have told us that.
Two of the four were mine, written an hour earlier in the same session that
identified the pattern.
401 passed, 2 skipped — unchanged, because these are assertions added to
existing tests rather than new test functions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the vacuous-test sweep across the four repos. Tests only — no rule, IR, or runtime change.
The defect class
A test that only asserts a rule stayed silent cannot distinguish:
Both are green. This is the same shape as the vacuous prefetch test fixed in Unlimited-Context-LLM yesterday, which asserted a wall-clock bound on a branch that never executed — green for the wrong reason the entire time.
The repo already had the right pattern
tests/test_examples.pypairs every quiet frame with a firing frame on the same graph, so the graph is proven live before it is proven silent.tests/test_library.pydid not, in four places:consecutive_loss_circuitcorrelation_cluster_guardatr_volatility_haltcpi_impulse_pullback_shortThe CPI one is the most interesting: the short arm was only ever asserted silent. A short rule that could never fire would have passed unchanged, and the mutual-exclusion claim rested entirely on the long arm. It now gets a mirror hot print and asserts
SELL, SELLwhile the long arm goes quiet.Result
Each now runs a crossing frame through the same graph and asserts exact intents and timestamps. All four fired — so none of the rules was actually broken. Nothing in the suite could have told us that.
Two of the four were mine, written an hour earlier in the same session that identified the pattern. Worth recording rather than quietly fixing.
401 passed, 2 skipped— unchanged, because these are assertions added to existing tests rather than new test functions.Sweep status across the other repos
test_encode_and_recoverguards withassert hitsbefore asserting content;test_engine_beats_baseline_on_reachabilityis a proper ON/OFF head-to-head; the witness decay test assertsscore("doctrine") == 0.9before checking chatter faded. The one vacuous test there was fixed yesterday.statusand reason before checking nothing spawned, and one carries a positive control (absolute.status === "spawned") in the same body.AF-COLLECTOR-FLOW-UNSUPPORTED). No obvious cases found, but this was not verified by execution.