diff --git a/CHANGELOG.md b/CHANGELOG.md index 279a1f8e..1d0611f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,62 @@ date-based pre-1.0 versions. ## [Unreleased] +## [0.1.31] — 2026-09-15 + +The three critical findings from the v0.1.30 review, plus the job, provider and docs +follow-ups behind them. + +### Changed +- **Breaking (API):** `POST /v1/cases/grade` refuses a PAID grade without `confirm: true` + (422), the way `POST /v1/judges/{role}/optimize` always has. This includes `resume`, which + runs on the original job's paid flags — the shell's Resume button billed with no cost dialog + before. The $0 replay path (neither `live` nor `in_process`) spends nothing and needs no + confirm. The shell sends the confirm from its in-DOM dialog and routes Resume through it; + `lithrim grade` / `regrade` send it with `--confirm-cost`. + +### Fixed +- A calibration round whose held-out set is not the one the pinned demos were scored on is now + refused instead of pinned. The "no comparable score" branch pinned unconditionally, so a + regressing round could reach the production judge by changing the subset it was scored on; + `force` (`--force-pin`) still pins and the reason says so. +- Calibrating a corpus that carries its own split tags no longer strides across them. The + stride ordered every labelled case by id and took every third as held-out, ignoring the split + each case was imported with, so a labelled dataset's test rows could train the demos and + decide the pin. The engine refuses such a corpus by name, the optimize route answers 422 + before any paid call, and the reviewer card no longer offers the stride. +- A resume refused by the duplicate-job 409 no longer leaves the job stuck reading `running`: + the record is touched only after every refusal. +- A grade whose judge calls all fail the same auth/config way (401, 403, a missing deployment) + stops after three such cases and fails with the provider's own message, keeping the graded + rows for a resume, instead of grading the whole cohort into empty votes. A provider refusal + (a content filter) stays a miss and never aborts. The judge's error text rides the row. +- `lithrim calibrate` runs on the provider binding and the key connected in the app, like a + grade; it used to fall back to the role's default deployment and fail after the grade was + already paid for. +- One poller per job in the shell: a second follower joins the first instead of doubling the + polls and clobbering the progress chip, an agent switch stops the previous agent's pollers, + and the reviewer card shares one poller per calibration across mounted cards. The cost dialog + now closes when the service accepts the job instead of waiting out the whole run with Cancel + disabled. +- A round name compares only within its own split, so a calibration round never sits beside a + test round as a before/after. The shell asks for a comparison only for the after round. +- The reviewer editor's fact-check list offers only the checks the ACTIVE pack can run: the + static list offered `dosage_grounding` (a clinical floor that ships with the healthcare pack) + on the neutral `_core` pack, so a reviewer could reference a check nothing would execute. +- The export format picker offers only the formats the importer declares (a dataset with no + training prompt module cannot write the chat one), instead of sending the human into a 422. +- Both compose files pass `LITHRIM_ALLOWED_ORIGINS`, `LITHRIM_OPTIMIZE_TIMEOUT_S` and + `LITHRIM_GRADE_TIMEOUT_S` through, so the documented knobs work from a compose `.env`. +- Docs: the stale "held out on the test split" lines, the RAGTruth citation (Wu et al. + everywhere), the Azure deployment model string with the OpenAI form beside it, the curls the + no-clone reproduction needs, the release the compose file pins, and the README's "pulls". + +### Security +- The chat export loads only the training prompt module the importer manifest declares; the + route used to import whatever path the request named. +- Both ingest routes refuse a blob over `LITHRIM_INGEST_MAX_MB` (default 64) with the two sizes + and the knob, instead of reading an unbounded paste into memory. + ## [0.1.30] — 2026-09-11 Calibrate as a background job, so a pilot-scale calibration runs from the shell. @@ -38,8 +94,8 @@ Held-out hygiene for the pin gate, and the training export from the shell. - A pinned score is compared only on the same held-out set. The score sidecar records the held-out identity (split, size, digest of the case ids) and the manifest names the split; a set pinned before this change (scored on the test cut) or a changed dev slice is not - comparable, so the next round pins with that reason stated instead of being accepted or - refused on the difference between two sets. + comparable, so the round is refused with that reason rather than pinned (re-scoring the + pinned set would be a paid round); `force` pins it deliberately and says so. - The chat (azure-chat) export no longer crashes on a structured prose source (every RAGTruth QA row); a row the training prompt still cannot fill is refused by name. diff --git a/README.md b/README.md index 2f17e875..29bae349 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ You provide the key; Lithrim provides the harness. No accounts, no hosted infere **Run it in containers — `docker compose up`.** No local Python/Node toolchain needed; a stranger gets the whole stack in two commands: ```bash -docker compose up # builds + starts BFF (:8787), UI (:5180), and the JUTE mapper (:3031) +docker compose up # pulls + starts BFF (:8787), UI (:5180), and the JUTE mapper (:3031) ``` That's three services: the **BFF** (the API, `:8787`), the **UI** (`:5180`), and the bundled **JUTE mapper** (`:3031`, used only for ingesting arbitrary JSON; see [`docs/JUTE_MAPPER_ADDON.md`](docs/JUTE_MAPPER_ADDON.md)). If you never paste arbitrary JSON, `docker compose up bff ui` runs core-only and skips the mapper. diff --git a/apps/bff/app.py b/apps/bff/app.py index e6e6c0a3..d3ac03ce 100644 --- a/apps/bff/app.py +++ b/apps/bff/app.py @@ -179,6 +179,42 @@ def _reviewer_label(role: str) -> str: ) +def _core_verification_tool_names() -> frozenset[str]: + """The verification tool names core IMPLEMENTS (one class per name in + ``verification/tools.py``). A pack-local tool (the clinical ``dosage_grounding``) is absent + here and reaches the offer only through its pack's executor registry.""" + import inspect + + from lithrim_bench.verification import tools as _tools + + return frozenset( + name + for _, cls in inspect.getmembers(_tools, inspect.isclass) + if isinstance(name := getattr(cls, "name", None), str) and name + ) + + +def _available_validators(pack: str | None = None) -> list[str]: + """VALIDATOR-OFFER-1: the fact-checks the reviewer editor may offer for ``pack`` — the known + names this pack can actually RUN: an executor it registers, or a verification tool core + implements. The static tuple offered every name whatever the pack, so the neutral ``_core`` + offered the clinical ``dosage_grounding`` and a reviewer could reference a check nothing + could execute. Resolved per request like ``_grounding_contract_types``, so OFFER and GATE + agree; an undiscoverable pack falls back to the core-only offer rather than failing a read.""" + from lithrim_bench.harness import grounding as _grounding + + runnable = set(_core_verification_tool_names()) + for candidate in (pack, "_core"): # an undiscoverable pack falls back to the core registries + try: + runnable |= set(_grounding.suppress_executors(candidate)) | set( + _grounding.floor_executors(candidate) + ) + break + except Exception: # noqa: BLE001 — a missing pack must never fail a $0 config read + continue + return [name for name in _KNOWN_VALIDATORS if name in runnable] + + def get_config_db() -> Path: """The SQLite config plane the BFF resolves agents from — scoped to the ACTIVE workspace (switching the workspace switches agents/judges/flags/audit). Override in tests.""" @@ -1178,6 +1214,22 @@ def _grade_via_subprocess(*, agent_name, config_db, ontology_path, collections_d raise HTTPException(status_code=500, detail="grade subprocess emitted no __GRADE_JSON__ record") +def _split_tags_in_corpus(ws=None) -> dict[str, int]: + """SPLIT-HYGIENE-1: {tag: n} over the workspace's LABELLED cases that carry a ``split``. + Empty when the corpus has no splits of its own (the stride path's only honest input).""" + tags: dict[str, int] = {} + corpus: list[dict] = [] + with contextlib.suppress(Exception): # unreadable corpus → no tags; the engine still refuses + corpus = _read_ingested_corpus(ws) + for case in corpus: + if not isinstance(case.get("expected_safety_flags"), list): + continue + tag = str(case.get("split") or "") + if tag: + tags[tag] = tags.get(tag, 0) + 1 + return tags + + def _optimize_via_subprocess( *, role, ws, collections_db, out_dir, limit, case_ids=None, force_pin=False, split=None ) -> dict: @@ -2552,6 +2604,33 @@ def _jsonl_rows(path: Path): } +def _refuse_a_second_grade(agent: str, *, exclude: str | None) -> None: + """409 when a grade job is already running for ``agent`` (``exclude`` is the job being + resumed). JOB-STATE-1: callable before any mutation, so a refusal leaves no half-state.""" + running = _running_job_for(agent, exclude=exclude) + if running is not None: + raise HTTPException( + status_code=409, + detail=f"a grade job is already running for {agent}: {running['job_id']}", + ) + + +def _require_grade_confirm(confirmed: bool, paid: bool, *, resumed: bool) -> None: + """GRADE-CONFIRM-1: refuse a paid grade that never confirmed its cost (422), the way the + optimize route does. A resumed job runs on the ORIGINAL job's paid flags, so its own request + must confirm them again — the resume button billed silently before this.""" + if not paid or confirmed: + return + raise HTTPException( + status_code=422, + detail=( + ("resuming this job makes PAID judge calls (it keeps the flags the first round ran " + "with). " if resumed else "grading makes PAID judge calls, one per case. ") + + "Resend with confirm=true only after an explicit cost check." + ), + ) + + class GradeCasesRequest(BaseModel): # NARR-LOOP: batch-grade the ingested corpus (the "evaluate all of them → report" loop). # case_ids None → ALL ingested cases. live/in_process are the SAME paid knobs as run-eval @@ -2571,6 +2650,41 @@ class GradeCasesRequest(BaseModel): # calibration split). Selectors, never paid knobs. round: str | None = None split: str | None = None + # GRADE-CONFIRM-1: a PAID grade (live or in_process, new or resumed) crosses this boundary + # only with an explicit confirm, the way optimize does. The $0 replay path spends nothing + # and stays confirm-free. A resume inherits the original job's paid flags, so it needs one. + confirm: bool = False + + +# INGEST-LIMIT-1: the front door reads the whole blob into memory before the mapper sees it, so +# it states a ceiling instead of dying deep inside the decoder on someone's whole trace export. +_INGEST_MAX_MB_DEFAULT = 64 + + +def _ingest_max_bytes() -> int: + """The ingest ceiling in bytes (LITHRIM_INGEST_MAX_MB, default 64). A bad value reads as the + default rather than refusing every upload.""" + raw = os.environ.get("LITHRIM_INGEST_MAX_MB") + try: + mb = int(raw) if raw else _INGEST_MAX_MB_DEFAULT + except ValueError: + mb = _INGEST_MAX_MB_DEFAULT + return max(1, mb) * 1024 * 1024 + + +def _refuse_oversized_ingest(raw: str) -> None: + """422 naming what arrived, the ceiling, and the knob that raises it.""" + limit = _ingest_max_bytes() + size = len(raw.encode("utf-8", "ignore")) + if size > limit: + raise HTTPException( + status_code=422, + detail=( + f"this upload is {size / (1024 * 1024):.1f} MB and the ingest limit is " + f"{limit // (1024 * 1024)} MB. Load a smaller cut, or raise " + "LITHRIM_INGEST_MAX_MB on the service." + ), + ) class IngestPreviewRequest(BaseModel): @@ -2855,6 +2969,9 @@ def _grade_row( "served_model": v.get("served_model"), "latency_ms": v.get("latency_ms"), "usage": v.get("usage"), + # JUDGE-ERROR-1 / GRADE-ABORT-1: the failure text rides the row, so a dead + # run is recognisable (and a reader can see WHY a vote is missing). + "errors": v.get("errors") or [], "scores_raw": v.get("scores_raw")} for v in (rec.get("council") or {}).get("votes", []) ], @@ -3013,10 +3130,46 @@ def _run_optimize_job(job: dict, *, out_dir: Path, kwargs: dict) -> None: _save_job(out_dir, job) +# GRADE-ABORT-1: an auth/config failure is the SAME for every case — the key, the deployment or +# the endpoint is wrong — so a cohort that keeps hitting one is billing for nothing. These are the +# markers; a provider REFUSAL (a content filter) is deliberately absent: it is case-specific, stays +# a miss on the row, and never aborts the run. +_AUTH_ABORT_MARKERS = ( + "401", "403", "authenticationerror", "invalid api key", "invalid_api_key", + "incorrect api key", "invalid subscription key", "access denied", "permissiondenied", + "permission denied", "deploymentnotfound", "unauthorized", +) +_AUTH_ABORT_AFTER = 3 # consecutive cases whose every judge call failed this way + + +def _auth_failure_reason(row: dict) -> str | None: + """The auth/config message behind a row whose judge calls ALL failed, else None. A row with + some judge answering is not a dead run; a refusal message matches nothing here.""" + votes = row.get("votes") or [] + messages: list[str] = [] + if row.get("error"): + messages.append(str(row["error"])) + failed = 0 + for vote in votes: + errors = vote.get("errors") or [] + if errors: + failed += 1 + messages.extend(str(e) for e in errors) + if votes and failed != len(votes): + return None # a judge answered — the run is not dead + for message in messages: + low = message.lower() + if any(marker in low for marker in _AUTH_ABORT_MARKERS): + return message + return None + + def _run_grade_job(job: dict, *, db_path, out_dir, workdir, collections_db, code_families) -> None: """The job body: grade every target without a verdict, saving the record after each case so a restart can resume; then the cohort report. Any failure lands on the record.""" r = job["request"] + consecutive_auth_failures = 0 + last_auth_reason: str | None = None try: done_rows = {row["case_id"]: row for row in job["rows"] if row.get("verdict")} for cid in job["targets"]: @@ -3031,6 +3184,20 @@ def _run_grade_job(job: dict, *, db_path, out_dir, workdir, collections_db, code job["rows"] = [done_rows[c] for c in job["targets"] if c in done_rows] job["done"] = sum(1 for x in job["rows"] if x.get("verdict") or x.get("error")) _save_job(out_dir, job) + # GRADE-ABORT-1: every judge call failing the same auth/config way, case after case, + # is a dead run — stop and name it instead of grading the cohort into empty votes. + reason = _auth_failure_reason(row) + if reason is None: + consecutive_auth_failures = 0 + else: + consecutive_auth_failures += 1 + last_auth_reason = reason + if consecutive_auth_failures >= _AUTH_ABORT_AFTER: + raise RuntimeError( + f"stopped after {consecutive_auth_failures} cases in a row whose every " + f"judge call failed the same way: {last_auth_reason}. Nothing further was " + "graded; fix the provider setup and resume this job." + ) job["result"] = _cohort_report( [done_rows[c] for c in job["targets"] if c in done_rows], job["targets"], agent=job["agent"], live=r["live"], in_process=r["in_process"], db_path=db_path, @@ -3152,6 +3319,11 @@ def _job_scorecard( and j.get("job_id") != job.get("job_id") and j.get("agent") == job.get("agent") and j.get("round") == compare + # COMPARE-SPLIT-1: a round name resolves within the SAME split — a + # before round on the calibration split and one on the test split are + # measurements on different cases, and pairing them reads as a swing + # that never happened. + and j.get("split") == job.get("split") and j.get("status") == "done" ): candidates.append(j) @@ -3170,6 +3342,28 @@ def _job_scorecard( return card +def _resolve_prompt_module(requested: str | None, vocab) -> str: + """EXPORT-MODULE-1: the chat row's user turn comes from the module the IMPORTER MANIFEST + declares. The route used to import whatever path the request named, so a caller could make + the service exec an arbitrary .py; the manifest is the contract, and a request may only + repeat it.""" + declared = getattr(vocab, "training_prompt_module", None) + if not declared: + raise HTTPException( + status_code=422, + detail=f"format chat needs a prompt module; importer {vocab.id!r} declares none", + ) + if requested and requested != declared: + raise HTTPException( + status_code=422, + detail=( + f"prompt_module {requested!r} is not the module importer {vocab.id!r} declares " + f"({declared!r}); the manifest is the contract for what the export may load" + ), + ) + return declared + + class ExportRequest(BaseModel): # UI-JOURNEY-1 (B8): the Export verb — the graded corpus of one grade job as labeled rows # (``lithrim_bench.cli.export``): the job's split, a label-basis tier per row, the dataset's @@ -3291,14 +3485,7 @@ def export_endpoint( gold[cid] = row fill_prompt = None if req.format == "chat": - # FT-FROM-SHELL-1: the chat row's user turn comes from the importer manifest's training - # prompt module unless the caller names one; neither is a 422. - prompt_module = req.prompt_module or vocab.training_prompt_module - if not prompt_module: - raise HTTPException( - status_code=422, - detail=f"format chat needs a prompt module; importer {vocab.id!r} declares none", - ) + prompt_module = _resolve_prompt_module(req.prompt_module, vocab) try: fill_prompt = _export._load_fill_prompt(prompt_module) except (OSError, SystemExit, AttributeError) as exc: @@ -3453,6 +3640,12 @@ def grade_cases_endpoint( ) if job.get("status") == "running": raise HTTPException(status_code=409, detail=f"job {req.resume} is still running") + paid = bool((job.get("request") or {}).get("live") or (job.get("request") or {}).get("in_process")) + _require_grade_confirm(req.confirm, paid, resumed=True) + # JOB-STATE-1: every refusal comes BEFORE the record is touched. Flipping it to running + # and dropping its unverdicted rows first left a job refused by the 409 below stuck + # reading "running" with nothing driving it, and the UI then offered no resume at all. + _refuse_a_second_grade(req.agent, exclude=req.resume) targets = list(job["targets"]) job["rows"] = [row for row in job.get("rows") or [] if row.get("verdict")] job["status"], job["error"], job["result"] = "running", None, None @@ -3472,6 +3665,7 @@ def grade_cases_endpoint( ), ) job = None + _require_grade_confirm(req.confirm, live or in_process, resumed=False) if not (req.background or req.resume): rows = [ _grade_row( @@ -3485,12 +3679,8 @@ def grade_cases_endpoint( rows, targets, agent=req.agent, live=live, in_process=in_process, db_path=db_path, workdir=workdir, code_families=code_families, ) - running = _running_job_for(req.agent, exclude=req.resume) - if running is not None: - raise HTTPException( - status_code=409, - detail=f"a grade job is already running for {req.agent}: {running['job_id']}", - ) + if not req.resume: # a resume was already checked, before it touched the record + _refuse_a_second_grade(req.agent, exclude=None) if job is None: job = { "job_id": f"job-{uuid.uuid4().hex[:12]}", @@ -4318,7 +4508,9 @@ def _judge_summary(role: str, jc, ontology, bindings: dict | None = None) -> dic "assigned_flags": assigned, "validator_refs": (list(jc.validator_refs) if jc else []), "available_flags": available, - "available_validators": list(_KNOWN_VALIDATORS), + "available_validators": _available_validators( + workspace.get_active_workspace().pack + ), "questions": questions, "authored": jc is not None, # Per-reviewer sampling config (independent-axes model). ``k`` falls back to the per-role @@ -4686,6 +4878,21 @@ def optimize_judge_endpoint( "an explicit cost check." ), ) + if not req.split: + # SPLIT-HYGIENE-1: without a split the optimizer STRIDES over every labelled case and + # ignores the tag each was imported with, so a labelled dataset's held-out rows would + # train the demos and decide the pin. Refuse here: before the paid call, before a job. + tags = _split_tags_in_corpus(ws) + if tags: + named = ", ".join(f"{k}: {n}" for k, n in sorted(tags.items())) + raise HTTPException( + status_code=422, + detail=( + f"this workspace's cases carry their own split tags ({named}). Name the " + "split to calibrate on, so the demos never train on rows the dataset held " + "out; calibrating without one would stride across every split." + ), + ) resolved_out = out_dir if out_dir is not None else (REPO_ROOT / "out" / "bff" / "optimize") kwargs = dict( role=role, ws=ws, collections_db=collections_db, out_dir=resolved_out, limit=req.limit, @@ -8898,6 +9105,7 @@ def ingest_preview_endpoint( template, apply it, and return the extracted cases + the template for the human to validate. Pins NOTHING and writes NO corpus — the human approves at /commit. A bad blob / non-converging transform is a calm 422 (the front door surfaces the reason), never a bare 500.""" + _refuse_oversized_ingest(req.raw) actor = _resolve_actor(x_actor, default_actor) agent = _resolve_chat_agent(req.agent, db_path) ctx = _build_tool_context(agent, db_path, out_dir, workdir, collections_db, actor, x_actor) @@ -8922,6 +9130,7 @@ def ingest_commit_endpoint( ) -> dict: """CE-INGEST-FRONTDOOR-1: pin the human-APPROVED template + upsert the corpus (no LM gen). The decode is deterministic, so this reproduces exactly the cases shown in /preview.""" + _refuse_oversized_ingest(req.raw) actor = _resolve_actor(x_actor, default_actor) agent = _resolve_chat_agent(req.agent, db_path) ctx = _build_tool_context(agent, db_path, out_dir, workdir, collections_db, actor, x_actor) diff --git a/apps/shell/src/app.cohort.test.jsx b/apps/shell/src/app.cohort.test.jsx index ec3409c9..88ecc3e8 100644 --- a/apps/shell/src/app.cohort.test.jsx +++ b/apps/shell/src/app.cohort.test.jsx @@ -51,6 +51,32 @@ describe("COHORT-SUBSET-1: the ⌘K 'Grade all cases' cohort trigger", () => { const sent = JSON.parse(String(init?.body || "{}")); expect(sent.in_process).toBe(true); expect(sent.case_ids).toBeUndefined(); // "Grade all" carries NO subset — the whole cohort + expect(sent.confirm).toBe(true); // GRADE-CONFIRM-1: the dialog's answer reaches the service + unmount(); + }); + + // JOB-POLLER-1: the dialog used to sit there, with Cancel disabled, for the WHOLE run — minutes + // at cohort size, with no way out — because it waited on the finished job rather than on the + // service accepting it. A 202 hands over to the status-bar chip. + it("closes as soon as the job is accepted and leaves the chip to report it", async () => { + let polls = 0; + vi.stubGlobal( + "fetch", + vi.fn((url) => { + const u = String(url); + let body = {}; + if (u.includes("/v1/cases/grade")) body = { job_id: "job-c1", status: "running", done: 0, total: 9 }; + else if (u.includes("/v1/jobs/job-c1")) body = { job_id: "job-c1", status: ++polls > 3 ? "done" : "running", done: polls, total: 9, result: { matrix: [], summary: {}, scorecard: { cases: [] } } }; + else if (u.includes("/v1/jobs")) body = { jobs: [] }; + return Promise.resolve({ ok: true, json: async () => body }); + }), + ); + const { unmount } = render( {}} />); + openPalette(); + fireEvent.click(await screen.findByText(/Grade all cases/i)); + fireEvent.click(await screen.findByRole("button", { name: /Grade all cases \(paid\)/i })); + await waitFor(() => expect(screen.queryByRole("dialog")).toBeNull()); + expect(await screen.findByTestId("grade-progress")).toBeInTheDocument(); // the chip took over unmount(); }); }); diff --git a/apps/shell/src/app.jsx b/apps/shell/src/app.jsx index b7c1ebb9..3d3a6a89 100644 --- a/apps/shell/src/app.jsx +++ b/apps/shell/src/app.jsx @@ -371,7 +371,11 @@ function StatusBar({ activeWs, activeAgent = "ws0_default" }) { grade {prog.interrupted.job_id} stopped at {prog.interrupted.done}/{prog.interrupted.total} (the service restarted) + onClick={() => { + // GRADE-CONFIRM-1: resuming bills (the job keeps the round's paid flags), so it goes + // through the SAME cost dialog as a fresh grade — this button never spends by itself. + try { window.dispatchEvent(new CustomEvent("lithrim:grade-cohort", { detail: { resume: prog.interrupted } })); } catch {} + }}>Resume )} {meta && {meta.workspace} · {meta.pack}} diff --git a/apps/shell/src/bff.js b/apps/shell/src/bff.js index 38ccbc32..dfdb37aa 100644 --- a/apps/shell/src/bff.js +++ b/apps/shell/src/bff.js @@ -90,11 +90,13 @@ export const runEvalPack = ({ pack_id, agents = ["ws0_default"], live = false }) /* POST /v1/cases/grade — RUN-ALL-1: grade the whole ingested cohort and return {matrix, summary, scorecard}. case_ids null → ALL cases. live/in_process are the SAME paid knobs as run-eval; a paid cohort grade is the human's cost-confirmed call (never an agent tool). The `scorecard` field - is the case_id-attributed consolidated report the inline ScorecardCard renders. */ -export const gradeCases = ({ agent = "ws0_default", live = false, in_process = false, case_ids = null, background = false, resume = null, round = null, split = null } = {}) => + is the case_id-attributed consolidated report the inline ScorecardCard renders. + GRADE-CONFIRM-1: `confirm` carries that in-DOM confirm to the service, which refuses a paid + grade (including a resume, which inherits the first round's paid flags) without it. */ +export const gradeCases = ({ agent = "ws0_default", live = false, in_process = false, case_ids = null, background = false, resume = null, round = null, split = null, confirm = false } = {}) => call("/v1/cases/grade", { method: "POST", - body: { agent, live, in_process, ...(case_ids ? { case_ids } : {}), ...(background ? { background } : {}), ...(resume ? { resume } : {}), ...(round ? { round } : {}), ...(split ? { split } : {}) }, + body: { agent, live, in_process, ...(case_ids ? { case_ids } : {}), ...(background ? { background } : {}), ...(resume ? { resume } : {}), ...(round ? { round } : {}), ...(split ? { split } : {}), ...(confirm ? { confirm: true } : {}) }, }); /* GET /v1/council/rules?agent= — UI-JOURNEY-1 (B10): how the council decides (the frozen rules in diff --git a/apps/shell/src/genui/ImportCard.test.jsx b/apps/shell/src/genui/ImportCard.test.jsx index 66f03ee3..a9026583 100644 --- a/apps/shell/src/genui/ImportCard.test.jsx +++ b/apps/shell/src/genui/ImportCard.test.jsx @@ -3,7 +3,7 @@ import { describe, it, expect, vi, beforeEach } from "vitest"; import { render, screen, fireEvent, waitFor } from "@testing-library/react"; import ImportCard from "./ImportCard.jsx"; -const IMPORTERS = [{ id: "ragtruth_vocabulary", dataset: "ragtruth", adapter: true, files: ["response.jsonl", "source_info.jsonl"], citation: "Niu et al.", license: "MIT", verdict_rule: { ragtruth: "a response is hallucinated iff it carries at least one annotated span" } }]; +const IMPORTERS = [{ id: "ragtruth_vocabulary", dataset: "ragtruth", adapter: true, files: ["response.jsonl", "source_info.jsonl"], citation: "Wu et al.", license: "MIT", verdict_rule: { ragtruth: "a response is hallucinated iff it carries at least one annotated span" } }]; let calls; beforeEach(() => { @@ -28,7 +28,7 @@ describe("ImportCard", () => { it("lists the pack's importers with their provenance and one file input per declared file", async () => { render(); await screen.findByTestId("import-importer"); - expect(screen.getByTestId("import-provenance").textContent).toMatch(/Niu et al.*MIT.*annotated span/); + expect(screen.getByTestId("import-provenance").textContent).toMatch(/Wu et al.*MIT.*annotated span/); expect(screen.getByTestId("import-cases").querySelectorAll('input[type="file"]')).toHaveLength(2); expect(screen.getByTestId("import-load")).toBeDisabled(); // no files yet }); diff --git a/apps/shell/src/genui/JudgeEditor.jsx b/apps/shell/src/genui/JudgeEditor.jsx index 67221df4..1b4667ca 100644 --- a/apps/shell/src/genui/JudgeEditor.jsx +++ b/apps/shell/src/genui/JudgeEditor.jsx @@ -35,7 +35,9 @@ const OPT_POLL_MS = 3000; // OPTIMIZE-JOB-1: a calibration takes minutes; poll g const fmt = (x) => (typeof x === "number" ? x.toFixed(2) : "—"); /* The HONEST held-out Δ render (D-G, inline — NOT the calibration_chart reliability - diagram). Shows baseline→optimized precision/recall/graded on the FIXED test split. + diagram). Shows baseline→optimized precision/recall/graded on the held-out slice the round + was scored on: the dev slice carved from the calibration split (HOLDOUT-DEV-1) when the + corpus carries splits, else the strided test split. A win renders the lift; a ≤0 Δ renders EXPLICITLY as a loss (R1 — never hidden, never spun; the accept-gate is never loosened to manufacture a win). */ function OptimizeDelta({ result }) { @@ -44,10 +46,13 @@ function OptimizeDelta({ result }) { // PIN-GATE-2: the server decides whether the compiled demos reach the production judge (only a // set that does not regress the pinned one). `pin` is absent on a pre-gate server: say nothing // about binding rather than the old (false) "binding is the next step" line. + const forced = /force/i.test(pin?.reason || ""); const pinLine = pin ? pin.pinned - ? `Pinned: the production judge now grades with these demos${/force/i.test(pin.reason || "") ? " (pinned by force over a lower held-out score)" : pin.comparable === false ? " (the pinned set's score came from a different held-out set, so there was nothing comparable to beat)" : ""}.` - : "Not pinned: the previously pinned demos (or none) stay in force." + ? `Pinned: the production judge now grades with these demos${forced ? (pin.comparable === false ? " (pinned by force, with no comparable held-out score to beat)" : " (pinned by force over a lower held-out score)") : ""}.` + : pin.comparable === false + ? "Not pinned: this round was scored on a different held-out set from the pinned demos, so there was nothing to compare it against. The pinned demos (or none) stay in force." + : "Not pinned: the previously pinned demos (or none) stay in force." : null; const rows = [ { k: "graded", label: "Graded (hard-accept)" }, @@ -62,7 +67,7 @@ function OptimizeDelta({ result }) { className="flex flex-col gap-2 rounded-[var(--radius-sm)] border border-border bg-secondary px-3 py-2.5" >
- Held-out Δ (fixed test split) + Held-out Δ ({corpus_source && corpus_source.dev != null ? "dev slice of the calibration split" : "held-out split"}) n_train {n_train ?? "—"} · n_heldout {n_heldout ?? "—"} · {compile_config.n_demos_bootstrapped ?? 0} demos @@ -117,6 +122,42 @@ function OptimizeDelta({ result }) { ); } +// JOB-POLLER-1: the calibration pollers in flight, keyed by job id, shared across every mounted +// reviewer card so one job is polled once no matter how many cards are on screen. +const optimizePollers = new Map(); + +// Poll one calibration job to its end; a dropped poll is retried with a growing wait, more than +// five in a row is the error. Resolves with the final record, or {status: "error"} shape. +async function pollOptimizeJob(jobId) { + let failures = 0; + for (;;) { + let job; + try { + job = await getJob(jobId); + failures = 0; + } catch (e) { + failures += 1; + if (failures > 5) return { status: "poll-failed", error: e }; + await new Promise((r) => setTimeout(r, Math.min(OPT_POLL_MS * failures, 10000))); + continue; + } + if (job.status !== "running") return job; + await new Promise((r) => setTimeout(r, OPT_POLL_MS)); + } +} + +// Write one poller's final record into a card's own state (each mounted card renders it). +function applyOptimizeJob(jobId, job, setOpt) { + if (job.status === "poll-failed") { setOpt({ state: "error", result: null, error: friendlyError(job.error), job: jobId }); return; } + if (job.status === "done") { setOpt({ state: "done", result: job.result, error: null, job: jobId }); return; } + setOpt({ + state: "error", result: null, job: jobId, + error: job.status === "interrupted" + ? "The calibration stopped when the service restarted; nothing was pinned. Run it again." + : (job.error || `calibration ${job.status}`), + }); +} + export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default", onResult }) { const [status, setStatus] = useState("loading"); // loading | ready | error const [error, setError] = useState(null); @@ -148,28 +189,26 @@ export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default" const [opt, setOpt] = useState({ state: "idle", result: null, error: null }); // idle|running|done|error // OPTIMIZE-JOB-1: a calibration runs as a background job; poll it (dropped polls are retried // with a growing wait, so a reconnect does not lose it) and render its result block. + // JOB-POLLER-1: ONE poller per job id across every mounted card — the reviewer card can be on + // screen twice (the chat's card and the editor), and each mount started its own poller for the + // same calibration, doubling the polls and racing two writers onto the same result. const followOptimize = async (jobId) => { setOpt({ state: "running", result: null, error: null, job: jobId }); - let failures = 0; - for (;;) { - let job; - try { - job = await getJob(jobId); - failures = 0; - } catch (e) { - failures += 1; - if (failures > 5) { setOpt({ state: "error", result: null, error: friendlyError(e), job: jobId }); return; } - await new Promise((r) => setTimeout(r, Math.min(OPT_POLL_MS * failures, 10000))); - continue; - } - if (job.status === "done") { setOpt({ state: "done", result: job.result, error: null, job: jobId }); return; } - if (job.status !== "running") { - setOpt({ state: "error", result: null, job: jobId, - error: job.status === "interrupted" ? "The calibration stopped when the service restarted; nothing was pinned. Run it again." : (job.error || `calibration ${job.status}`) }); - return; - } - await new Promise((r) => setTimeout(r, OPT_POLL_MS)); + const shared = optimizePollers.get(jobId); + if (shared) { + const job = await shared; + applyOptimizeJob(jobId, job, setOpt); + return; + } + const run = pollOptimizeJob(jobId); + optimizePollers.set(jobId, run); + let job; + try { + job = await run; + } finally { + optimizePollers.delete(jobId); } + applyOptimizeJob(jobId, job, setOpt); }; // On mount: a calibration of this reviewer that is running (or just finished) is found again. @@ -195,8 +234,10 @@ export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default" const splitCounts = { calibration: cases.filter((c) => c.split === "calibration").length, test: cases.filter((c) => c.split === "test").length }; // HOLDOUT-DEV-1: the calibration split alone is enough (a dev slice is carved from it); the // test split is never part of the calibration corpus. + // SPLIT-HYGIENE-1: when the corpus carries its own splits this is not a choice — the service + // refuses a calibration that strides across them (it would train on held-out rows), so the + // split is always sent and the subset picker yields to it. const splitAvailable = splitCounts.calibration >= 2; - const [useSplit, setUseSplit] = useState(true); useEffect(() => { let live = true; @@ -314,7 +355,7 @@ export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default" try { // optimize-on-subset: scope to the chosen cases ONLY when a subset is picked — an empty // selection sends no case_ids, keeping today's whole-workspace optimize byte-identical. - const onSplit = splitAvailable && useSplit; + const onSplit = splitAvailable; const result = await optimizeJudge(role, { confirm: true, background: true, @@ -520,9 +561,10 @@ export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default"

- Compile few-shot demos from the by-construction calibration split, then measure the - honest held-out Δ on the fixed test split. Did the edit move the number? — win or loss, - shown straight. + Compile few-shot demos from part of the calibration split, then measure the honest + held-out Δ on the rest of it — a source-disjoint dev slice the demos never saw. The + test split stays untouched. Did the edit move the number? — win or loss, shown + straight.

{judge && (
@@ -532,16 +574,13 @@ export default function JudgeEditor({ role = "risk_judge", agent = "ws0_default"
)} {splitAvailable && ( - +

+ Calibrating on the calibration split ({splitCounts.calibration} cases): 70% trains the demos and a source-disjoint 30% dev slice decides the pin. The test split ({splitCounts.test} cases) stays untouched until you grade again. These cases carry their own splits, so calibration never strides across them. +

)} {/* optimize-on-subset: scope the calibration to a CHOSEN case set. No selection = the whole workspace (today's behaviour). A $0 selector — the paid confirm is below. */} - {cases.length > 0 && !(splitAvailable && useSplit) && ( + {cases.length > 0 && !splitAvailable && (