MP-178: a dated public proof page, and the refusal channel's first real verdict

[samarthputhraya]

Closes MP-178. Publishes docs/reports/modelpin-voicerag-abstention-1.md — a live run dated 2026-09-05, so the only Modelpin proof a stranger can open is no longer the Drift Map's 2026-06-24.

Labelled a SELF-dogfood. Both apps measured are the maintainer's own public repos. Under ADR-0031 the maintainer is not a non-maintainer, so this moves metric A by exactly zero. It is evidence the engine works, not evidence anyone uses it.

The result

App	Migration	Verdict
VoiceRAG (calibrated abstention)	gpt-oss-20b → gpt-oss-120b	1 regression, 7 unchanged
aegis (real tool calls)	gpt-oss-120b → gpt-oss-20b	3 regressions, 3 unchanged

VoiceRAG's contract says decline with the sentinel token INSUFFICIENT_CONTEXT. [M] On the same unsafe question, gpt-oss-20b emitted it 5/5; gpt-oss-120b emitted I'm sorry, but I can't help with that. 5/5. Both models declined — only one declined in the language the app parses. VoiceRAG's parser would take the second as an uncited answer and its grounding guard would reject it as ungrounded: the right outcome for entirely the wrong reason.

refusal_delta +1.0, confidence 1.00, reproduced across two independent check runs. This is the first time the refusal channel has moved a verdict on real material. MP-151 predicted it would need "a paid tier, or a real user's migration" — it needed neither, only an app whose decline is not English.

What the run says about our own measurement — published on the same page

• [M] 15 of 15 genuine abstentions recorded refused=False. REFUSAL_MARKERS are nine first-person English phrases; a sentinel token matches none and never will.
• [M] A text assertion caps at changed_minor (diff/__init__.py:454) and cannot fail a build alone. So an app whose decline is a token has no channel that can fail it on abstention drift — unless the migration happens to move toward an English refusal, which is the direction this one moved. Our exit 1 came from the refusal channel, not the assertion.
• [M] Twelve statements across seven files claimed refusal "never fired — 0 of 30 on aegis". That run's own committed artifact stores refused: true on 5 of its 30 traces. All three PUBLIC sites are corrected in this PR; the private sweep is MP-187.
• [M] One near-miss disclosed against ourselves: bec_urgent_pressure returned unchanged while 5/5 candidate runs escalated against 2/5 baseline.
• [M] On legitimate_payment the engine's representative trajectory (2/5) is not the modal one (3/5), so the report and the verifier disagree there. Said out loud rather than left to be found.

Invariants held

• Engine untouched — nothing under diff/, no threshold, no constant. ADR-0030 freeze holds.
• ADR-0025 — voicerag-suite is declared score. An adversarial review found six real assertion defects after it was scored; they are filed as MP-186, not applied, because re-cutting an assertion to change a result is exactly the fitting that rule forbids.
• ADR-0009 — measurement/opinion throughout. The 120b model's English refusal is arguably better safety behaviour; the report says so, and frames the break against the app's written contract, never model quality.
• ADR-0008 — end-user's own keys. [M] The judge (gpt-4o-mini) is in neither compared pair, closing the weakness the Drift Map disclosed about itself.

Verification

python docs/reports/data/verify_voicerag_report.py — no API key, no network. Recomputes every number in both Result sections from the 140 committed traces.

Guard: tests/test_voicerag_report_claims.py, 13 tests derived from the artifacts. [M] 9 of 9 mutants caught — including one that survived a presence-only check of the ADR-0031 label and is now pinned by position.

[M] Gate: 766 passed + 4 xfailed, ruff clean, black clean (91 files), mypy 5 errors = baseline. 254 Groq replays on the free tier.

Also filed: MP-185 (the RUNBOOK records the GROQ key polarity backwards, and the loader it prescribes now selects the dead key — worked around here with a probe-and-select loader), MP-188 (a public and a private file disagree on an anchor denominator).