Repository navigation
Fix/v0.1.31 - #7
Merged
Merged
Conversation
pin_demos pinned unconditionally when the pinned set's held-out identity
differed from the round's ("no comparable score"), so a regressing round
could reach the production judge by changing the subset it was scored on.
With nothing to compare, refuse and leave the pinned demos in force;
force pins deliberately and the reason says so. The CLI's step_pin does
not double the --force-pin hint when the reason already names force, and
the reviewer card reads out a refusal rather than implying a win.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…PLIT-HYGIENE-1) With `split` omitted, the optimizer built its corpus with build_calib_rows, which ordered every labelled workspace case by id and strided every third into `test` — ignoring the split each case was imported with. On a workspace loaded from a labelled dataset that let the dataset's own test rows train the demos and decide the pin. build_calib_rows now refuses a corpus whose labelled cases carry split tags, naming the tags and counts; the optimize route refuses the same request with a 422 before any paid call or job record; and the reviewer card no longer offers a stride toggle, so a split-tagged corpus always calibrates on a named split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
POST /v1/cases/grade had no confirm field at all, so the shell's cost dialog was the only thing between a click and a paid cohort — and the job-resume button bypassed it, posting straight through while the resumed job kept the original round's paid flags. The route now refuses a paid grade, new or resumed, without confirm: true (422, before any case is graded and before a job record exists). The $0 replay path spends nothing and stays confirm-free. The shell's Resume dispatches the grade-cohort bridge so it opens the same cost dialog, which names the job and the cases left to grade; the CLI carries --confirm-cost through to the service. Breaking change, recorded in the changelog. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…OLLER-1) Three job-plane defects from the review: - The resume branch flipped the record to running and dropped its unverdicted rows BEFORE the duplicate check, so a resume refused by the 409 left the job stuck reading "running" with nothing driving it and no resume offered. Every refusal now precedes the mutation. - followJob started a poller per caller, so the tab that began a job and the mount-time restore polled it twice, each re-entering beginBatch/endBatch; a second follow now joins the first, and stopFollowing ends the previous agent's pollers on a switch so its job never lands in the new agent's chrome. JudgeEditor shares one poller per calibration across mounted cards. - The cost dialog waited on the finished job with Cancel disabled — minutes at cohort size. It closes when the service accepts the job (202) and the status-bar chip reports the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…TE-KEY-1) step_optimize spawned the optimizer with a bare os.environ copy: no per-role binding, no model pin, and no key from the .provider_env the app writes. A key connected in the UI graded fine and then failed at calibrate, which fell back to the role's default deployment — after the human had already paid for the grade. judge_env_for builds the same per-role env a grade runs with: the persisted binding first (provider, model, endpoint, api version), the --model pin as the fallback, and the key resolved into LITHRIM_LLM_API_KEY_<ROLE>. The secret is read, never persisted to the binding store and never printed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hit live during the reproduction run: the Azure key began returning 401 mid-cohort, every judge call failed, each case recorded a WARN vote with no served model, and the job ran to completion reporting a scorecard built from nothing. An auth/config failure is the same for every case, so after three consecutive cases whose every judge call failed that way the job fails with the provider's own message and the graded rows are kept for a resume. A provider refusal (a content filter) is case-specific: it stays a miss and never aborts. The judge's error text now rides the row's votes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…OMPOSE-ENV-1) The BFF reads LITHRIM_ALLOWED_ORIGINS (serving the UI to a LAN address or an iPad) and the two subprocess timeouts, and DEPLOY.md tells a reader to raise the optimize one for a pilot-scale calibration — but neither compose file passed any of them through, so setting them in the .env beside the compose file did nothing and the only way in was an override file. All three pass through now, and DEPLOY.md says so and documents the origins knob beside the UI image's baked BFF URL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…COMPARE-SPLIT-1) - A round name resolved to any round with that name, so a before round on the calibration split could sit beside an after round on the test split: two measurements on different cases, read as a swing. It now resolves within the same split, and the shell asks for a comparison only for the after round (a replay or a calibration round asked too). - The export picker offered every training format; an importer with no training prompt module cannot write the chat one, so the human met a 422. It reads training_formats from the importer and disables the rest, naming the importer. - EXPORT-MODULE-1 (hardening): the chat export loads only the module the importer manifest declares — the route used to import whatever path the request named. - Copy: the calibrate blurb and header say the dev slice, not the "fixed test split"; the journey app resolves VITE_BFF_URL like the rest of the shell instead of hardcoding localhost; --training-tokens says what it does and that it never leaves the machine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- POLICY_HOLDOUT_HYGIENE, IMPORTERS and the CLI loop header still said the optimizer measures on the test split and that the calibration corpus carries test rows; both stopped being true with the dev slice. - The RAGTruth citation was Wu et al. in three places and Niu et al. in three others; it is Wu everywhere now. - The reproduction pages: the UI page carries the curl for the dataset and for the reviewer definition at the tag (no clone to copy from) and its acceptance sentence claims the loop runs, not a published table; the CLI page says azure/<your deployment> with the OpenAI form beside it and states the held-out rule once. DEPLOY fetches compose from the tag, names the release it pins and the sample image tags that exist. The README says the stack pulls, because the root compose has no build block. - INGEST-LIMIT-1 (hardening): both ingest routes refuse a blob over LITHRIM_INGEST_MAX_MB (default 64) with the two sizes and the knob, instead of dying inside the decoder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version, the compose image pins, and the changelog for the review patch. The grade route's confirm is called out as the breaking change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…FER-1)
GET /v1/judges/{role} returned the static _KNOWN_VALIDATORS tuple whatever
the workspace's pack was, so the reviewer editor's "Attach fact-checks"
row offered dosage_grounding — a clinical floor whose tool class and
executor ship with the healthcare pack — on the neutral _core pack, and a
reviewer could reference a check nothing could execute.
The offer now resolves per request the way _grounding_contract_types does
(OFFER and GATE agree): a name is offered when the active pack registers
an executor for it or core implements its verification tool. On _core that
is structural_jute, kb_rag, jute_gen and record_rag; in_row drops too,
since nothing implements it. An undiscoverable pack falls back to the core
registries rather than failing a $0 read.
Found by the review of v0.1.30 (reported as a UI chip; it was server-side).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A short description.
Why
The problem or motivation.
Checklist
ruff check .is cleanpytest -qis green without a model key (credential-free)make demostill runs