Repository navigation
mr-verifier: Jev reference column beside the local ceiling (v0.42.0) - #67
Merged
Merged
Conversation
…sion model The verifier's external reference column needs a caller for TypeSafe's Jev as served through OpenRouter's decisions path: a System One model that takes a state plus a map of typed questions and answers them all in one parallel pass with probabilities. This is that caller, and nothing more. - The served model is a package CONSTANT, never a field, flag or environment variable. The operator's standing rule on this credential is that it funds this one model, so the code offers no path through which another id could reach the wire. A test decodes the request body and asserts the constant. - The credential is passed in by the caller (loaded from the orchestrator's credential file) and never read from the environment — the free lanes' rule, because the operator's shell may carry keys for other businesses. It rides in the Authorization header only, is never serialized into the body, and is redacted out of every error string (pinned by a test). - Retry set is the documented transient one (rate limit and the three upstream faults), three attempts total, doubling backoff that a cancelled context cuts short. The permanent statuses — malformed question, unauthorized, credits exhausted, oversized state — are surfaced once with the vendor's own reason in the message, because each is a fact to act on rather than a fault to wait out. An undecodable or answer-less 200 is a contract break, also not retried. - 16 MiB read cap and refused redirects: a proxy's error page must not become a memory event, and a redirecting endpoint must never hand the bearer to another host. - Answer carries Confidence because the wire does, with a comment that it is not max(probabilities), that its formula is unpublished, and that a gate belongs on a probability instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A decisive accuracy measured on 40 snippets is uninterpretable on its own: the local ceiling needs something to be a ceiling BELOW. This adds a second column that reads the same corpus with the same question through an external, calibrated decision model, and prints the two ceilings side by side. - Off by default and it spends: about $0.0002 for the committed corpus, read out of the responses rather than estimated and printed with the dated build that produced it, because a moved build invalidates every threshold fitted on the old one. - Fail open, twice over. A missing credential file warns and skips the column; a missing offload binary warns and skips the local one; either way the run exits 0 and prints whatever it could measure. The two failures are independent, so an unreachable local binary no longer cancels the reference column (tests pin both directions, plus the unchanged disarmed behaviour). - The confidence carried into the selective-risk curves is the PROBABILITY OF THE CHOSEN OPTION, never the response's own confidence field: that field is not max(probabilities), its formula is unpublished, and a curve needs a score whose meaning is fixed. A test fixture answers 0.4 confidence on an 0.83 probability so severing this is red. - Three options, not two: a choice picks the nearest option unless declining is on the list, so "cannot tell from this snippet alone" is offered and maps to a defer — a non-answer that never becomes a false pass. An option we never offered, or none at all, is an error rather than a quiet defer. - main is now a run(cfg, io.Writer) int so the fail-open paths can be tested without a real binary, a real credential or a real network; the endpoint override is a struct field with no flag behind it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- README: a paragraph under the eval substrate saying what the column is (an external calibrated second reading of the same corpus, printed beside the local ceiling), that it is off by default, what a run costs (about $0.0002 for the committed 40-snippet corpus, measured 2026-09-18), where the credential comes from, that a missing credential file is a warning and not a failure, and that the route and the model are fixed constants with no flag behind them. - CHANGELOG: the 0.42.0 section, including the two judgment calls worth re-reading later — why the score fed to the curves is the probability of the chosen option rather than the response's own confidence field, and why the question offers a third option instead of forcing a verdict. - VERSION and mr-orchestrate's version var move together with the changelog's top entry; B11 checks all three. - B12 raised 30043 → 30534, the measured figure, with the breakdown per file and the reason in the note. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First live run of both columns, 40 snippets, one question each. The reference column's seed is committed beside the local one, with a real probability on every record and the dated build on every row. The run corrected its own planning figure. The backlog carried $0.0002 as the cost of a 40-snippet run; the measured run cost $0.000701, because $0.0000175 was a PER-CALL smoke measurement that had been written down as a run total. The flag's help text, the package comment, the README, the changelog and the budget note now all quote the measured run — the estimate is named as the estimate it was. What the two columns said: identical headline (decisive 40/40, accuracy 0.900, coverage 1.000, effective ceiling 0.900), and only two of the four misses shared. Two verifiers landing on the same number while failing on different snippets is precisely what one column cannot tell you, and it is the argument for the column existing. The curve gap (AUGRC 0.0319 → 0.0081) is a ranking claim and the weaker reading of the two: 19 distinct confidence levels against the local column's 40, at n=40, on scores that are not the same quantity. B12 reconciled to the final measured figure: 30537, +494. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t a decoded 200 is metered Review findings on the reference column: the verdict switch normalised the vendor's choice but the probability lookup trusted the raw string, so a choice echoed in another casing than its own probabilities map would have read as confidence 0 with no error. probabilityOf() looks it up by the same normalised key (+12 LOC, B12 raised to 30549). The mapping test also pins that a decoded 200 naming no usable option is still metered as a call — money was spent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Add an off-by-default external reference column to
mr-verifierso the CC1 ceiling re-measurement that the offload-harness denominator fix forces can be read against a calibrated reference that did not move. TypeSafe's Jev decision model, reached through one client package, on the same 40-snippet corpus and the same gate question.What changed
internal/jevclient: a one-endpoint, one-model client. The served model and the endpoint are package constants — no flag, field or env var behind them; a test decodes the outgoing body and asserts the constant reached the wire. Credential passed in by the caller fromfreelane.LoadToken(statepaths.StateDir(), "openrouter"), never read from the environment, Authorization header only, redacted out of every error. Retries 429/502/524/529 ×3 with doubling backoff; 400/401/402/413 surface once; an undecodable or answer-less 200 is a contract break. 16 MiB read cap, redirects refused.cmd/mr-verifier:-jev-column(default false) and-jev-out. One three-option choice per snippet; yes→Pass, no→Fail, unsure→Defer, HTTP/parse error→Errored;Record.Confidence= the chosen option's probability looked up by its normalised key (not the response'sconfidencefield). Fails open: no credential → warning, local column still runs, exit 0. Prints a second ceiling block and a side-by-side table.mainrefactored torun(cfg, io.Writer) intso the fail-open paths are testable with no binary, credential or network.## [0.42.0],VERSIONand the orchestrator version var (B11),docs/complexity-budget.jsonraised 30043 → 30549 with the measured breakdown (B12),testdata/verifier-seed-jev.jsonl(40 records from the live run).How tested
gofmt clean on LF copies of every touched file;
go vet ./...clean;go test ./...53 packages ok including the canaries. New suites: 11 client tests (httptest only) and 7 verifier tests (6 mappings, metering of a decoded 200, three fail-open paths, disarmed run never says "jev"). One live run: both columns 0.900 / 1.000 / 0.900, AUGRC 0.0319 (local) vs 0.0081 (Jev), 40/40 answered, $0.000701 — the record with the miss sets and caveats is in the private docs repo. Independent sonnet review applied (normalised-key lookup; metering pinned).Risk
Low — off by default; no lane, router or free-lane code touched; nothing operator-specific in the repo.
🤖 Generated with Claude Code