Clearance integrity: ten commits, and the census that counted a tool nobody called - #67
Merged
Merged
Conversation
…r eleven [M] 2026-08-31, MP-144's live aegis run: three distinct hard 400s each killed all six scenarios and produced NOTHING -- including one where the model HALLUCINATED a tool name (`verify_vendor` for `verify_vendor_bank`), which is itself the behaviour change Modelpin exists to catch. `report` already survived this; `check` -- the command CI runs -- wrapped the entire scenario loop in one `_guard_replay` and turned any ProviderError into exit 1 with no verdicts and no report. `check` now catches ProviderError PER SCENARIO: skip it, name it with the provider's own message, exclude it from every coverage number, and refuse the clean clearance. Deliberately not a retry -- MP-139's rule that a 400 is not retried still stands. - NotImplementedError stays a hard failure: an unimplemented adapter is a config error that would fail every scenario identically. - All scenarios rejected now exits EXIT_UNMEASURED with the provider's reason, instead of sending the user to `modelpin baseline` for a baseline they have. - A regression still outranks incomplete coverage: exit 1 wins over exit 3, because a real finding in the scenarios that DID run is the strongest true claim available. - `compared` excludes rejected ids, so the census and underpowered lists do not price coverage for a run that did not happen (the MP-138 trap, new cause). - Both renderers disclose them before any verdict bucket, and `_rejected_clearance` joins the underpowered/census pair so "safe to adopt" cannot render over a suite that is one scenario short. MP-28's hardening test moves 1 -> 3 with that reasoning recorded inline: its invariant is non-zero + no fabricated trace + no `unchanged` line, all still held.
[M] 2026-08-31 `check` wrote exactly one path, <store>/last-report.md, every run. The consequence is already in the repo: both dogfood write-ups in ops/launch/ had to be transcribed by hand, because the file they came from was overwritten before anyone could cite it. A tool whose north-star metric is false-positive RATE cannot compute one from artifacts that delete each other. `last-report.md` does not move -- action.yml:135 publishes that exact path, both CI workflows glob for it, the README documents it. The durable copy is written beside it at <store>/runs/check-<from>-to-<to>-<UTCstamp>.md: - byte-identical to the stable path (one render, two destinations), so a reader never has to know which one the Action posted; - written SECOND, so if only one write can succeed it is the one CI reads, and an archive failure is a warning that never costs the report; - same-second collisions disambiguate with a counter -- clobbering is the defect. README and the generated demo README both say where it lands.
[M] 2026-08-27 action.yml:22 listed `anthropic` in the provider input with no marker while providers/anthropic.py:17 is `raise NotImplementedError` -- the one place where following the documentation reaches a crash. [M] 2026-08-31, measured while fixing it, the row understated the surface: the unmarked advertisement was in FIVE places, not one -- action.yml, the `--provider` help on baseline/check/report (so `mp check --help` said it too), and the unknown-provider ValueError in providers/__init__, which is the first provider list a user reads after a typo. "The CLI itself is honest" was true of the runtime path (_preflight_or_fail, _unimplemented_msg) and false of the help text. Fixed with ONE source -- `providers.provider_help()` -- rather than five marked copies, because five hand-maintained copies is how the marker came to be missing from all of them at once (the MP-03 shape). `anthropic` is deliberately still NAMED: a user who types it deserves "coming, not yet implemented", not "unknown provider". The adapter itself stays unbuilt for the reason in MP-20's dated trigger -- no Anthropic key exists on this machine and the calibration suites are OpenAI-scored, so shipping an unvalidatable adapter would be worse than an honest stub. tests/test_advertised_providers.py ties the text to the CODE (`run` is read for a NotImplementedError body, not called) and is SYMMETRIC: a stub must be marked everywhere, and a working provider must not be disclaimed -- so the day the adapter lands, the stale caveat fails the build instead of outliving the stub.
…n judge meaning [M] 2026-08-31 build_judge rejected any non-OpenAI judge model. Honest -- it said so -- but the hole had teeth: cross-vendor is wedge item 3, and for a suite with no `tools` (a classifier, a summariser -- the long-tail wedge) the judge is the ONLY CI-failing channel that responds to a change in MEANING. Per _channel_census, such a run had no hard content channel at all; only a refusal could fail the build. Follows the ADAPTERS' pattern rather than inventing a second one: - the four OpenAI-compatible hosts REUSE OpenAIJudge with a different base_url and key env, exactly as build_openai_compatible_adapter reuses the OpenAI adapter; - GoogleJudge's SDK calls mirror GoogleAdapter line for line -- system_instruction and temperature in `config`, the answer off candidates[0].content.parts[*].text. Those shapes are already verified against the installed SDK, and this project has been bitten twice by writing provider calls from memory. Routing refuses to GUESS. [M] `openai/gpt-oss-120b` is an OpenRouter/Groq id whose first segment is a different vendor's name, so a `vendor/model` parse would misroute and spend the wrong key against the wrong host. Only `gpt-*` and `gemini-*` are inferred; anything else takes the new `judge_provider:` config key, or falls back to the replay provider -- which is printed on the console before any spend, never silent. A judge model equal to the model under test now draws a note: it is a legal, cheap choice, but it is not an independent reading. 23 offline tests, including one pinning that every judge asks the IDENTICAL question, because the semantic channel's calibration is measured against that exact wording and two rubrics would make cross-vendor verdicts incomparable. README and docs/fp-measurement.md now separate the two claims that used to be one: the judge RUNS on six hosts, and its false-positive rate has only ever been measured with an OpenAI judge. A judge that works is not a judge that is calibrated.
[M] 2026-08-31 refusal is the last of five verdict-bearing channels never observed on real material: 0 of 60 traces on the kavach dogfood, 0 of 30 on aegis. The zero says nothing about looks_like_refusal -- neither suite contains a request a competent model would decline, so the detector was never given the chance. A channel that has never fired is an assumption wearing a measurement's clothes, and for a suite with no tools and no judge, refusal is the ONLY channel that can fail a build. examples/refusal-suite/ -- five CAPABILITY/privacy declines (live market data, a URL fetch, a local file, sending mail with no tool, a private individual's details) plus one quiet anchor that must NOT read as a refusal. Capability boundaries over unsafe-content prompts, deliberately: harmless to ship in a public repo, consistent across vendors, and a model that GAINS the capability stops declining -- so the refusal rate moving is a real migration signal rather than a regex being tickled. The anchor is what makes the suite show the detector DISCRIMINATES rather than merely fires, which is the north-star framing. Role `fit`, not `score`: if the live run shows declines REFUSAL_MARKERS misses, the fix is widening those markers -- a threshold change, and ADR-0025 forbids scoring a rate on the set it was fitted on. The refusal FP rate belongs on the held-out examples/suite. Declared in examples/roles.json and pinned in EXPECTED_MEMBERS, the deliberate path that guard demands. NOT YET RUN LIVE. The suite README says so in those words: until 60 completions happen, "these scenarios decline" is an [A].
[M] 2026-08-31, found by MP-151's live run. `llama-3.3-70b-versatile` returns 404 model_not_found; Groq's live catalogue is 14 ids and it is not among them. It appeared in three places in README.md, including a copy-paste `modelpin check` command a new user would run. A product whose premise is "models get retired and your app finds out in production" shipped a quickstart that a model retirement had already broken. - the runnable ids become `qwen/qwen3.8-27b`, which is live today; - a line above the command says Groq rotates its catalogue and links the docs, so the next rotation costs a reader a lookup rather than a 404; - the MEASURED `gpt-4o-mini` vs `llama-3.3-70b-versatile` 8/8 result is KEPT and annotated with the retirement, not deleted. It is a true statement about a run that happened; erasing a measurement because its subject was retired would be the opposite of this project's evidence discipline. The watcher half is deliberately NOT closed and stays on the row: data/models.json and watcher/registry.py track the labs' schedules, and nothing tracks the OpenAI-compatible hosts' catalogues, which rotate faster. This will recur. Also updates examples/refusal-suite/ with the live results (75 of 90 refuse_* traces detected, 0 of 15 on the anchor) and scrubs its links into ops/, which is a separate private repo and would be a broken link in the public tree.
…te -> 3.0.0 MP-142 proved by differential -- not by grep -- that expected_tool_calls and output_schema were consulted by NOTHING: five trace configurations, including baseline SATISFIES while candidate VIOLATES, returned byte-identical verdict, confidence and explanation with the field set and with assertions=None. This deletes them, which is a public suite version bump, not a cleanup. examples/report-suite (role public, ADR-0009): 2.0.0 -> 3.0.0, sha256:ffd99774f681 -> sha256:5cba1dc8b691. No scenario's MEANING changed and no verdict moves; compute_suite_hash covers the VALIDATED model, so removing a field moves it anyway -- and a published artifact whose fingerprint changes without its version changing is an unannounced change. [M] Half of MP-147's own prediction did not reproduce, and the test file records it rather than adopting the new numbers quietly: the row predicted sha256:eed334061b5e / sha256:5482ccd734fd, computed with the fields deleted from the model but the dead keys LEFT in the JSON files. Reconstructing exactly that reproduces eed334061b5e for report-suite to the digit, and still does not reproduce 5482ccd734fd for examples/suite (it gives 22a03cce1e6c). The files were cleaned too, so the pinned hashes are the measured ones. Deleting the fields ALONE would have made things worse for users: pydantic ignores unknown keys, so a stale expected_tool_calls would go from documented-but-inert to invisible -- MP-142's defect pushed into a file we do not own. So the advisory moved with the fields: it now reads the RAW json keys instead of model fields, and therefore also catches a typo like `must_containn`, which no version has ever checked. The generalised guard replaces the two specific ones: every field on Assertion must appear in violates_text_assertions, so the NEXT write-only field fails on the commit that adds it instead of surviving to be found by an audit. Nothing was lost. expected_tool_calls was redundant with a channel that works -- the tool-trajectory diff measures tool use distributionally over N runs, which is the job the static list only appeared to do. docs/fp-measurement.md's numbers are unaffected: verdicts are byte-identical, and it cites no suite hash. Also updates the superseded published report's note about what HEAD now serves, and the README's test count (702 passing + 3 xfail = 705 collected).
…at was backwards The gates in CLAUDE.md's routing table, run over this session's work. Everything below was found by them, not by me, and each was reproduced before it was fixed. claims review, and this is the one that mattered: - THE REFUSAL SUITE'S STATED RATIONALE DESCRIBED A SIGNAL THE ENGINE DOES NOT HAVE. examples/refusal-suite/ said "a model that GAINS the capability stops declining -- exactly the migration signal Modelpin exists to catch". [M] Reproduced on the engine: baseline 5/5 refused -> candidate 0/5 returns `unchanged`, confidence 1.00, refusal_delta -1.0. The channel is ONE-SIDED by design (ADR-0002) and cannot see a drop; on a suite with no tools and no assertions that direction is visible only to a configured judge. A reader running the reproduction in that README on a browsing-enabled model would have got `unchanged` at confidence 1.00 and concluded the tool was broken. The one-sidedness is a deliberate FP protection and is NOT changed; the claim is. - The same error made my `allam-2-7b` inference backwards: I read `unchanged` as proof it had also refused, "because a 5/5 -> 0/5 flip could not produce `unchanged` at N=5". It produces exactly `unchanged`. Those 25 traces are [A]. - And the denominator was wrong: 75 of 90 counted refusals against a total that INCLUDED the 15 anchor traces reported separately. Measured is 50 of 50. - The public suite README now opens with the fact that its numbers are NOT reproducible from this repository -- the raw traces were never published, unlike the Drift Map's. - "Every check run leaves an artifact" -> a run that produces a report does. Two paths exit before one exists. - "five places, now generated from one source" -> four render from provider_help(); action.yml cannot call Python and is test-pinned instead. - The differential proved `expected_tool_calls`; `output_schema` had zero readers. - examples/suite's fingerprint moved too (44cbde8e3b74 -> 3edf6b1ae19a) and was announced nowhere public. The CHANGELOG entry stated the principle and applied it to one of the two shipped suites. - README said no FP rate has been measured on "any of them" where "them" included OpenAI -- contradicting the same sentence's first clause. - The README's own free-run example had the candidate judging itself, the exact configuration `mp check` warns about and that demoted the self-judge calibration. - `mp init`'s scaffolded modelpin.yaml still named the retired llama id -- written verbatim into every new user's config, 56 lines from the entry declaring it dead. - The banned-word guard covered only report-suite, so the new public suite landed outside it. It now parametrises over every directory roles.json declares. provider-SDK review: - `_is_reasoning_model` missed the whole gpt-5 family. [S] Azure's reasoning-models page lists temperature/top_p/max_tokens as unsupported there, and a live 400 reads "Only the default (1) value is supported". [M] Our own data/models.json ships gpt-5.5 as ACTIVE and points gpt-5.2 at it -- so the registry recommended a judge model that would 400 on every call, and preflight() makes no network request, so it would have failed only after every replay was paid for. Also fixed the namespaced-id hole: `openai/gpt-5.6` and `openai/o3` are real host ids a bare prefix can never match -- precisely the hosts MP-143 just opened the judge to. - A non-OpenAI judge error named the wrong vendor: `_explain_api_error`'s `label` defaults to "OpenAI", so a rejected Groq key read "read your key from GROQ_API_KEY - check that variable holds a current OpenAI key". MP-136's exact defect, reintroduced hours earlier on the path MP-143 created. - OpenRouter ROUTES: [S] its docs say an upstream that does not support a parameter still receives the request and ignores it. `temperature: 0` returns HTTP 200 and is silently dropped, so the judge quietly stops being deterministic with no error to catch. `require_parameters` now rides OpenRouter judge requests only. first-run review: - The MP-147 advisory answered every unknown key with "this version checks must_contain / must_not_contain". Right for a typo, WRONG for `expected_tool_calls`: there is no text assertion for "expect this tool call", so the reader either guesses the field is safe to delete or tries to shoehorn a tool check into a string match. Removed keys now get the remedy true for them. - `Unknown provider: x (try: ... run.))` -- provider_help() ends in its own parenthetical and the caller wrapped it in another. - `check --help` documented no exit codes, which is where someone staring at a nonzero exit in a CI log looks. MP-148 made 3 far more reachable.
… test A visitor who sees a co-author trailer or an internal review-agent handle concludes the project is machine-generated and leaves without reading it. For a product whose entire pitch is measurement rigour, that read is fatal. The rule existed as a setting since 2026-08-29 and it still recurred: [M] handles had reached 22 places in tracked source -- cli.py, judge.py, providers/, report/, nine test files and CHANGELOG.md -- because a preference does not fail a build. All 22 now name the gate by what it CHECKS: FP review, claims review, provider-SDK review, first-run review, packaging review. tests/test_no_tool_attribution.py makes it enforceable, and three of its own bugs are worth recording because each would have made it a guard that reports "clean": - [M] `text=True` without an explicit encoding decodes with the Windows ANSI codepage. This repo's own history holds a byte cp1252 cannot map, so the reader thread died, stdout came back None, and returncode was still 0. - [M] A `%H%x00%B%x00` format split on two NULs mis-framed the stream -- git writes a newline between records, so each body bled into the next and the guard's first run named a commit that was clean. - [M] git writes the key as `Co-authored-by:` while the tool that adds it writes `Co-Authored-By:`. An exact-case needle catches one of the two. And the count it ratchets was wrong all day: [M] not 38 trailered commits but 42, confirmed by two independent methods. The 38 came from `grep -c` over a format containing NULs, which grep read as binary and under-counted. The related landmine claiming "38 SHA refs in 21 threads" is wrong twice over -- [M] it is 21 SHAs across 13 PR bodies. Scoped at a pinned commit, not a date: [M] a date boundary cannot work in a squash-merge repo, because GitHub replays a branch's message onto a commit stamped with the MERGE time -- b5c5e40 is authored and committed 2026-08-31 and carries a trailer from before the setting. Whether to rewrite that history is the maintainer's call and stays out of the test. A counter-test guards the other direction: Modelpin WATCHES model releases, so Anthropic model ids in the registry, the provider, and the detector's regex are the product working and must never be scrubbed by over-applying this rule.
… declared An UNUSED `tools:` key bought a false green. `[M]` On byte-identical fixtures, a scenario declaring one tool that no run on either side ever called turned `could not measure ... NOT cleared on content` into `no behavioral change ... looks safe to adopt` -- over a baseline of "FRAUD DETECTED: block this transaction" against a candidate of "Looks fine, approve it. asdf qwerty zzzz garbage." The census read the suite and the config; it never read the traces. A declared tool that nothing calls sits exactly where refusal already sits: it can only fire if the candidate STARTS calling tools, so a confident wrong answer never touches it. `hard_content_channels` had always excluded refusal for that reason, and this applies the same rule one case over. Worse, the disclosure's own remedy was the exploit -- it said "add `tools`", and following that advice was what bought the green. `[M]` Not hypothetical: on a six-scenario suite run against `openai/gpt-oss-120b`, all six scenarios declared `tools` and only four ever called one. The census now requires BOTH a declaration and a recorded call. The FP review blocked the first cut, which read the trace alone and was not monotone: a scenario with no `tools` key whose traces carried `tool_calls` went from "NOT cleared" to "looks safe to adopt" -- MP-159's own defect arriving through the other door, in the commit that fixed it. The conjunction makes the new blind set a superset of the old one for every input, so this change can only withhold a clearance, never grant one. That path is now pinned by a test; nothing had pinned it. No verdict, confidence or exit code moves -- the census governs disclosure only. The candidate traces had to be captured in the replay loop to make any of this possible: `cand` was a loop local and `DiffResult` carries none. This closes the declaration-shaped instance, not the class. A tool called identically on both sides still clears a run whose answer inverted, because the trajectory channel reads what the model does, not what it says -- pinned as a strict xfail rather than left to a tracker (MP-165). Review also corrected: the remedy named scenario ids as though they were tool names and wrapped mid-claim at 80 columns; the new string was reachable by no cp1252 test, the gap whose shape once shipped as a console crash; and the README's collected-test count had been left at its pre-change value. `[M]` /verify READY -- 8 steps, 723 passed + 4 xfailed, 94% coverage, ruff + black clean, mypy 5 = baseline.
`[M] 2026-09-02` `test_both_paths_are_printed` asserted a raw substring against console output. Rich wraps at the console width and will break a path mid-token, so whether `last-report.md` survives intact depends only on how long the temp directory happens to be. It passed on Windows and FAILED on both Linux CI jobs -- CI's `/tmp/pytest-of-runner/...` split it as `last-repor` + `t.md`, the local `C:/Users/.../Temp/...` broke one character earlier. This has been red on CI and has blocked PR #67 and #68 from merging. The product was never wrong; the test was measuring where the line happened to break. Compared with all whitespace stripped now -- the same correction three assertions in `test_error_messages_survive_rich.py` needed today, for the same reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten commits from three sessions. The theme is clearance integrity: several of these exist because a review gate proved the tool was calling runs clean that it had no business calling clean.
[M]/verifyREADY — 8 steps, 723 passed + 4 xfailed, 94% of 2041 statements,ruff+blackclean over 87 files,mypy5 = documented baseline.modelpin/diff/untouched, so ADR-0030's freeze holds.Breaking
feat(mp-147)!—Assertion.expected_tool_callsandAssertion.output_schemaare removed. They were recorded and read by nothing;expected_tool_callswas proved inert by differential over five trace configurations, not by grep. The public suite moves to 3.0.0 (sha256:ffd99774f681→sha256:5cba1dc8b691), becausecompute_suite_hashhashes the validated model.The clearance fixes
fix(mp-159)— the channel census counts a tool that was called, not one a scenario merely declared.[M]On byte-identical fixtures, a scenario declaring one tool no run ever called turnedcould not measure … NOT cleared on contentintono behavioral change … looks safe to adopt, over a baseline of "FRAUD DETECTED: block this transaction" against "Looks fine, approve it." The disclosure's own remedy — "addtools" — was the exploit.[M]Not hypothetical: on a six-scenario suite run againstopenai/gpt-oss-120b, all six declaredtoolsand only four ever called one.The census requires both a declaration and a recorded call. The FP review blocked the first cut, which read the trace alone and was not monotone: a scenario with no
toolskey whose traces carriedtool_callswent fromNOT clearedtolooks safe to adopt— the same defect through the other door. The conjunction makes the new blind set a superset of the old for every input, so it can only withhold a clearance, never grant one.No verdict, confidence or exit code moves; the census governs disclosure only. This closes the declaration-shaped instance, not the class — a tool called identically on both sides still clears an inverted answer, pinned as a strict
xfail.fix(mp-148)— one scenario a provider rejects no longer deletes the other eleven.[M]Validated in production hours after it was written: two scenarios hard-400'd and the run survived, named them, and still found the regression.feat(mp-150)— a run's report survives the next run. Everycheckused to delete the previous one's evidence.Cross-vendor and first-run
feat(mp-143)— the semantic judge runs cross-vendor. For a suite that declares notools, the judge is the only CI-failing channel that responds to a change in meaning, so a Groq- or Gemini-only user was running an engine that could not catch a wrong-but-confident answer at all. The judge's false-positive rate has only ever been measured with an OpenAI judge; running everywhere is not the same as being calibrated everywhere, anddocs/fp-measurement.mdnow separates those two claims.feat(mp-151)— a suite that actually asks a model to decline. Refusal was the last channel never observed on real material.fix(mp-152)— the README's Groq quickstart named a model Groq had retired, in the product about model retirement.fix(mp-128)— stop advertising a provider that raisesNotImplementedError.Housekeeping
fix— what three mandated review gates found, including a published claim that was backwards.chore— no tool attribution in anything a visitor reads, enforced by a test.Known-open, deliberately
MP-160is the remaining P0 on this axis: a scenario with no baseline still vanishes from the PR comment and never touches the exit code. Until it lands, a green result still has one way to be uninformative.