Sync main from GitForge: coverage lifts, dep pins, CI triage docs, PR #11 integration

benchmarks/decision-model/JEVBENCH.md

@@ -184,6 +184,48 @@ JevBench RESULTS; jev.page):
 | **Jev-Style-0.8B v3** | **64.1 % (148/231)** | **self-run, official harness — our bridge** |
 | open-jev-zefan-2b | 64.5 % | board |
 | Laya | 58.4 % | official |
+| autotrust/JEV-27B | 88.70 (family-macro) | published (autotrust announcement) — the B2 contamination-audit target |
+| LiquidAI/d1-2B | JevBench v1.4.1 77.9 | vendor card (their split/protocol; our public-231 replay below) |
+
+7. **autotrust/JEV-27B — the contamination-audit arm (queue B2,
+   2026-10-28).** autotrust's own serving path, not a port:
+   `serve_decide.py` (vLLM OpenAI server + `POST /v1/decide` from
+   their HF repo) with their `adapter_vllm/` LoRA and shipped
+   `calibration.json`; the runner (`run_jevbench.py --arm jev27
+   --jev27-endpoint`) posts each task verbatim — kind from the item
+   type, state passthrough, instructions + criteria folded into the
+   single question string the wire carries — with `thinking: "off"`
+   so the row is a pure System-1 audit. Probabilities native as-is
+   (never renormalized), noul false/true mapped to no/yes, failures
+   score incorrect (serial no-retry). Host: Modal H100 burst
+   (~$3-5 of the authorized budget; 27B bf16 + decision head does not
+   fit the T5500 V100 pair, and the head has no llama.cpp path).
+   The audit question: autotrust published **88.70 family-macro** on
+   this split; their model card records training on
+   `SargeDev/jev-distill-corpus-v3` — if their number does not
+   reproduce through their own serving code on our lane, the §14
+   dominance table needs a footnote; if it does, the row is real and
+   the chase target stands.
+
+8. **LiquidAI/d1-2B — the commercial decision-model survey arm
+   (2026-10-28).** A 1.92B multimodal decision model (LFM3-VL-2B
+   backbone: 13 gated short convs + 5 SWA + 6 global attention
+   layers; readout head, no generation) through llama.cpp v0.8.0's
+   native `/v1/systemone` wire (`run_jevbench.py --arm d1
+   --d1-endpoint`, local CPU Q8_0): one question per request, noul
+   criteria folded into instructions, choice criteria as the
+   criteria dict (label itself when absent — the choice_candidates
+   rule), score levels positional, probabilities native as-is
+   (the server applies the model-file temperatures — d1 ships its
+   own calibration, the same discipline as our D7/D9), noul p(true)
+   mapped to no/yes, failures score incorrect (serial no-retry,
+   same posture as jev27). Independent of the Jev distill lineage —
+   Liquid's own MOPD multi-teacher recipe — so a clean external
+   reference for what a purpose-trained 2B decision model scores
+   outside that corpus. **License: lfm1.0** — revenue-capped ($10M)
+   commercial use, derivative works inherit the cap, not
+   OSI-approved: weights are a benchmark reference and an
+   extraction teacher only, never a redistributable arm.
 
 ## Comparability caveats (stated up front)
 
@@ -235,6 +277,7 @@ milliseconds. Acceptance criteria:
 | engine + 4B rung, fusion-v2 gate (2026-10-05) | 0.5584 | 0.5622 | 0.335 | 0.734 | 1.2 ms / 0.73 s | 231/231 | native |
 | engine + 4B rung, proofs-only posture (2026-10-05) | **0.6883** | 0.6622 | 0.171 | 0.431 | 0.58 s / 14.0 s | 231/231 | native |
 | engine + 4B rung, proofs-only posture (2026-10-06 D35 rerun) | 0.6840 | 0.6593 | 0.170 | 0.436 | 0.82 s / 18.7 s | 231/231 | native |
+| d1-2B (LiquidAI Q8_0, llama.cpp v0.8.0 `/v1/systemone`, 2026-10-28) | 0.7489 | 0.7616 | 0.065 | 0.279 | 5.9 s / 57.2 s | replay 91.7% (33/36 noul pairs; ordinal/choice labels stable) | native |
 
 **The third-party 2B GGUF rows (task #125, 2026-10-08).** Two
 independently trained 2B decision models, each under its authors' own

benchmarks/decision-model/modal/run_b2_modal.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""B2 contamination-audit probe: autotrust/JEV-27B on JevBench public 231.
+
+Runs entirely inside a Modal H100 function: vLLM serves JEV-27B through
+autotrust's own serve_decide.py; the local runner (run_jevbench.py --arm
+jev27) replays the 231 public items against it in-process over
+localhost. Weights persist in a Modal Volume so only the first run pays
+the ~54 GB download.
+
+Usage:
+  modal run run_b2_modal.py            # full: server + replay
+"""
+import logging
+import os
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+import modal
+
+LOG = logging.getLogger("b2")
+logging.basicConfig(level=logging.INFO,
+                    format="%(asctime)s %(levelname)s %(message)s")
+
+APP_NAME = "b2-jev27-contamination-audit"
+MODEL_DIR = "/models/JEV-27B"
+REPO_ROOT = Path("/oc")
+WORK_DIR = Path(__file__).resolve().parent
+
+app = modal.App(APP_NAME)
+volume = modal.Volume.from_name("jev27-weights", create_if_missing=True)
+
+JEVBENCH_REF = Path("/nas/Temp/work/oc-model-eval/jevbench-ref")
+
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .pip_install("vllm==0.11.0", "huggingface_hub[hf_transfer]", "requests")
+    .add_local_dir(str(WORK_DIR.parent.parent.parent),
+                   str(REPO_ROOT / "OpenCodifier"))
+    .add_local_dir(str(JEVBENCH_REF), str(REPO_ROOT / "jevbench-ref"))
+    .pip_install(str(REPO_ROOT / "jevbench-ref"))
+)
+
+
+def wait_ready(url: str, deadline_s: float = 1800.0) -> None:
+    t0 = time.time()
+    while time.time() - t0 < deadline_s:
+        try:
+            with urllib.request.urlopen(f"{url}/v1/decide/info", timeout=5) as r:
+                LOG.info("ready: %s", r.read()[:200])
+                return
+        except Exception:
+            time.sleep(10)
+    raise SystemExit("vLLM /v1/decide did not become ready")
+
+
+@app.function(
+    gpu="H100",
+    timeout=60 * 60 * 3,
+    volumes={"/models": volume},
+    secrets=[modal.Secret.from_name("huggingface")],
+)
+def burst() -> str:
+    env = dict(os.environ)
+    env.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
+    if not (Path(MODEL_DIR) / "config.json").exists():
+        LOG.info("downloading autotrust/JEV-27B (~54 GB) ...")
+        subprocess.run(
+            ["hf", "download", "autotrust/JEV-27B", "--local-dir", MODEL_DIR],
+            check=True, env=env,
+        )
+        volume.commit()
+
+    serve_cmd = [
+        sys.executable, f"{MODEL_DIR}/serve_decide.py",
+        "--model", MODEL_DIR,
+        "--served-model-name", "autotrust/JEV-27B",
+        "--enable-lora", "--max-lora-rank", "32",
+        "--lora-modules", f"jev-decision={MODEL_DIR}/adapter_vllm",
+        "--logprobs-mode", "processed_logprobs",
+        "--max-model-len", "32768",
+        "--port", "8000",
+    ]
+    server = subprocess.Popen(serve_cmd, stdout=sys.stdout, stderr=sys.stderr)
+    try:
+        wait_ready("http://127.0.0.1:8000")
+        tasks = ",".join(
+            str(REPO_ROOT / "jevbench-ref" / "datasets" / "public" / f)
+            for f in ("easy.jsonl", "hard.jsonl", "original.jsonl"))
+        out = "/runs/jev27"
+        replay_cmd = [
+            sys.executable,
+            str(REPO_ROOT / "OpenCodifier" / "benchmarks" / "decision-model"
+                / "runner" / "run_jevbench.py"),
+            "--arm", "jev27",
+            "--jev27-endpoint", "http://127.0.0.1:8000",
+            "--jev27-model", "autotrust/JEV-27B",
+            "--tasks", tasks,
+            "--out-dir", out,
+            "--timeout", "600",
+        ]
+        subprocess.run(replay_cmd, check=True)
+        summary = Path(out) / "summary.json"
+        LOG.info("SUMMARY: %s", summary.read_text() if summary.exists()
+                 else "(summary missing)")
+        return str(summary) if summary.exists() else out
+    finally:
+        server.terminate()
+        server.wait(timeout=30)
+
+
+@app.local_entrypoint()
+def main() -> None:
+    result = burst.remote()
+    LOG.info("result: %s", result)

benchmarks/decision-model/results/CALIBRATION.md

@@ -230,3 +230,60 @@ are evidence at any n. The 231-item JevBench arm clears the floor; the
 still carry the pair on disk, which nothing in-repo loads through the
 engine's loader — the next compute-box refit regenerates them into the
 floor-honest shape (analysis-only aurc), changing no T or ECE value.
+
+## Per-kind vs global-T on the E1-E board run (2026-10-29)
+
+`perkind_holdout_fit.py --probs` (T3.2 pattern) on the E1-E 2B JevBench
+public-231 run (`runs/jevbench/e1e-slot-231`, accuracy 0.6840,
+T-invariant) against the pinned jevbench-ref split. Fits are in-sample
+on the board — a sizing probe for the per-kind D15 artifact, not a
+shipped calibration; disclosure per the T3.2 rule.
+
+| fit | board ECE | auto@5% |
+|---|---|---|
+| T=1 (raw) | 0.2067 | 23/231 (0.100) |
+| global T=3.362 (shipped holdout fit) | 0.1305 | 9/231 (0.039) |
+| per-kind (bool 20.0*, choice 2.7974, score 4.1458) | **0.0678** | **48/231 (0.208)** |
+
+Per-kind temperature halves the global fit's ECE and doubles the
+auto-accept coverage — the d1-2B / MOPD pattern (per-question-type
+calibration) transfers to our arm through the D25 ladder seam without
+any training. Notes:
+
+- \* bool T hit the fit grid's 20.0 cap; the optimum is at or beyond
+  the cap — bool confidences want near-uniformization (ECE 0.359 →
+  0.069 on 74 rows). Grid should extend before shipping.
+- score kind has only 18 board rows; its per-kind T (4.15) is
+  weakly identified and still leaves ECE 0.152 on that slice —
+  fitting belongs on the holdout/suite split, where score rows are
+  denser.
+- Next step when a compute window opens: holdout-side per-kind fit
+  (train-side rows, not the board) → exact D15 artifact for E1-E →
+  board ECE reported out-of-sample. Expected to land between the
+  in-sample 0.068 and the global 0.131.
+
+### Out-of-sample correction (2026-10-29, same day)
+
+The predicted out-of-sample landing zone was wrong — tested with the
+T5500-recovered E1-E suite rows (holdout-120 + rel200-280):
+
+- Suite-pooled choice fit **T=6.144** (holdout alone fits 4.763,
+  rel200 alone 6.740). Applied to the board: **ECE 0.1511 — worse
+  than the shipped global 3.362 (0.1305)**. The earlier "expected
+  between 0.068 and the global 0.131" is retracted.
+- Transfer is asymmetric and domain-bound: holdout-fitted T helps
+  rel200 (0.262 → 0.149) but rel200-fitted T hurts holdout
+  (0.179 → 0.212); 2-fold CV on the suite union shows the direction
+  is real on-suite (0.226→0.145, 0.256→0.148) but the gain does not
+  survive the suite→board shift.
+- Verdict: the per-kind board 0.0678 is **mostly in-sample fitting**.
+  What survives: (a) per-kind *shape* — bool/score kinds genuinely
+  want different temperatures than choice, so the D25 per-kind seam
+  stays the right mechanism; (b) temperature is fit-per-surface, not
+  global — an artifact fitted on one surface must not be trusted on
+  another (the D15 shipping gate's ECE-check rule, now confirmed
+  cross-domain). An honest per-kind artifact for the board needs
+  board-like fit rows that are not the board itself — i.e. a held-out
+  slice of the JevBench-style distribution, which we do not have;
+  the cross_validate path on the board itself is the only available
+  estimator and must be labeled as such.