diff --git a/docs-site/src/content/docs/experiments/proposed.md b/docs-site/src/content/docs/experiments/proposed.md index e6efd23..284dffc 100644 --- a/docs-site/src/content/docs/experiments/proposed.md +++ b/docs-site/src/content/docs/experiments/proposed.md @@ -3,8 +3,6 @@ title: "Proposed Experiments Register (Hypotheses & Designs)" description: "Register of experiments we intend to run, each with a motivation, a falsifiable pre-registered hypothesis, a design, metrics, decision criteria, and dependencies. Seeded with the open questions from Invariant Decision Calibration (IDC)." --- -# Proposed Experiments Register - The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. This page records experiments we **intend to run**, written down *before* any results exist, so that the hypothesis, the metric, and the pass/fail line can't quietly move after the data comes in. **Lifecycle:** `Proposed` → `Ready` (dependencies met, design reviewed) → `Running` → `Done`. When an entry starts running, give it the next free `EXP-XX` number, add it to the ledger, and link it back here. Keep the `PROP-XX` entry, but mark it `Done → EXP-XX` and note any change to the design along with the reason for it. @@ -28,7 +26,7 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. | ID | Title | Question it answers | Depends on | Priority | Status | | :--- | :--- | :--- | :--- | :---: | :--- | -| [`PROP-00`](#prop-00-expose-mirror-tvd-on-all-four-surfaces-enabler) | Expose Mirror TVD on all four surfaces (*enabler*) | Can production use the mirror signal at all? | — | **P0** | Proposed | +| [`PROP-00`](#prop-00-expose-mirror-tvd-on-all-four-surfaces-enabler) | Expose Mirror TVD on all four surfaces (*enabler*) | Can production use the mirror signal at all? | — | **P0** | Closed: not worth it after [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) | | [`PROP-10`](#prop-10-run-to-run-and-revision-noise-floor) | Run-to-run & revision noise floor | How big a difference is real? | — | **P0** | Done → [EXP-14](/dgem/experiments/exp-14-idc-rerun/) (noise floor: ±1–2 items, Brier ±0.02) | | [`PROP-01`](#prop-01-held-out-temperature-scaling) | Held-out temperature scaling | Does the ECE gain from $T^*$ survive out of sample? | — | **P0** | Done → [EXP-14](/dgem/experiments/exp-14-idc-rerun/) (met on 231 JevBench items, not on 50) | | [`PROP-02`](#prop-02-end-to-end-idc-cascade) | End-to-end IDC cascade | Does IDC + a mirror-aware gate beat the entropy-only cascade? | `PROP-00`, `PROP-01` | **P0** | Partly → EXP-14 (entropy gates); mirror gate moves to `PROP-13` | @@ -41,10 +39,11 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. | [`PROP-09`](#prop-09-base-rate-label-shift-adaptation) | Base-rate (label-shift) adaptation | Can unlabeled target traffic correct for different class frequencies? | — | P2 | Proposed | | [`PROP-11`](#prop-11-letter-collision-in-the-mirror) | Letter collision in the mirror | Is the mirror's damage caused by shared letters rather than by a second slot? | — | **P0** | Done → [EXP-15](/dgem/experiments/exp-15-letter-collision/) (supported) | | [`PROP-12`](#prop-12-separate-pass-mirror) | Separate-pass mirror (replaces `PROP-03`) | Does a mirror read in its own pass give a clean order signal? | `PROP-11` | P1 | Done → [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) (supported, small effect) | -| [`PROP-13`](#prop-13-mirror-aware-cascade) | Mirror-aware cascade (unblocks `PROP-02`) | Does an uncoupled mirror improve the hand-off gate? | `PROP-12` | P1 | **Next** | +| [`PROP-13`](#prop-13-mirror-aware-cascade) | Mirror-aware cascade (unblocks `PROP-02`) | Does an uncoupled mirror improve the hand-off gate? | `PROP-12` | P1 | Closed: not worth it after [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) | | [`PROP-14`](#prop-14-in-context-vs-blank-question-prior) | In-context vs blank-question prior | Does a prior estimated from real items fix null-prior's overcorrection? | — | P1 | Proposed | | [`PROP-15`](#prop-15-correction-strength-by-question-type) | Correction strength by question type (extends `PROP-06`) | Do yes/no and lettered choices need different correction strengths? | `PROP-14` | P2 | Proposed | | [`PROP-16`](#prop-16-slot-names-are-part-of-the-prompt) | Slot names are part of the prompt | How much do slot ids change answers? | — | P1 | Done → [EXP-16](/dgem/experiments/exp-16-slot-names/) (single slot: no; second slot: yes) | +| [`PROP-17`](#prop-17-calibrated-agent-context-pre-compiler-internal-pilot) | Calibrated agent context pre-compiler (internal pilot) | Can one multi-slot pass decide which context blocks an agent turn needs, dropping little that matters? | — | P2 | Running (internal pilot) | --- @@ -52,6 +51,11 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. ### `PROP-00`: Expose Mirror TVD on all four surfaces (*enabler*) +* **Closed (2026-09-28), not run.** EXP-15–EXP-17 showed the mirror signal adds little over hesitation: a + separate-pass mirror improves cross-validated error detection by at most 0.016 AUROC at twice the cost, and the + same-canvas mirror costs accuracy unless labels are digits. Exposing it on every surface (PROP-00) or gating the + cascade on it (PROP-13) is not worth the complexity; hesitation remains the production signal. Reopen only if a + cheaper, uncoupled order signal appears. * **Motivation:** `--dual-mirror` computes Mirror TVD/JSD in `pkg/permutation.PostProcessDecisionResponse`, but `cmd/decide.go` discards the return values. `dgem serve`, MCP, and the Studio don't offer `dual_mirror` / `null_prior_debias` at all, which violates the four-surface parity rule in `AGENTS.md`. A mirror-aware gate (`PROP-02`) can't run in production until this is fixed. * **Deliverable (not a hypothesis):** * Add `diagnostics.mirror_tvd` / `mirror_jsd` per slot to the `dgem decide` output. @@ -189,6 +193,11 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. ### `PROP-13`: Mirror-aware cascade +* **Closed (2026-09-28), not run.** EXP-15–EXP-17 showed the mirror signal adds little over hesitation: a + separate-pass mirror improves cross-validated error detection by at most 0.016 AUROC at twice the cost, and the + same-canvas mirror costs accuracy unless labels are digits. Exposing it on every surface (PROP-00) or gating the + cascade on it (PROP-13) is not worth the complexity; hesitation remains the production signal. Reopen only if a + cheaper, uncoupled order signal appears. * **Motivation:** Unblocks `PROP-02` once an uncoupled mirror exists. * **Hypothesis (H13):** A gate of hesitation ≥ τ **or** disagreement ≥ τ′ beats entropy alone at the same hand-off rate. * **Design:** Offline, using `PROP-12` readings and the existing Gemini-on-every-item receipt; choose τ, τ′ on half the items and report on the other half. @@ -220,6 +229,25 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. * **Cost:** about 1,200 requests. * **Pre-registered design (2026-09-26, before data):** JevBench 231, Vertex G4, one session. Single slot: 3 baselines with id `decision` (noise band = min–max), then ids `q1`, `x7k2q` (random), `mirror`, `check`. Two slots (`--dual-mirror --mirror-mode copy`, so no letter collision): suffixes `__rev`, `__mirror_rev`, `_b`. An id "matters" if its correct count is more than 3 items outside the baseline band, or (two-slot) more than 3 items away from `copy`+`__rev`. Secondary: per-item agreement with the baseline majority answer. About 2,300 requests (revised up from 1,200). +### `PROP-17`: Calibrated agent context pre-compiler (internal pilot) + +* **Motivation:** Agents re-send large context windows (tool schemas, history, retrieved documents) every turn. A + single `dgem` pass could grade every candidate block at once and use hesitation to downgrade uncertain blocks to a + one-line stub instead of dropping them. The pilot's design and data are internal for now; this entry holds the + number and the decision rule. +* **Hypothesis (H17):** On labelled agent turns, one pass keeps at least 95% of the blocks the gold labels mark as + required (`full` or `summary`), removes at least 50% of candidate tokens, and flags planted prompt-injection + blocks with recall ≥ 0.9; hesitation-driven "stub instead of drop" reduces required-block misses compared with + taking the most likely action. +* **Design:** ~40 labelled turns across several domains, 8 candidate blocks each (tool schemas, history turns, + retrieved documents, notes), gold action per block from one annotator with a second model labelling independently + to measure agreement; production serving image, `samples: 1`, neutral slot ids (EXP-16). +* **Metrics:** required-block recall (primary), token savings, quarantine precision/recall, rescue rate + (required blocks kept as stubs only because of hesitation), agreement with the gold labels. +* **Decision:** If H17 holds, build a larger labelled set from real agent traces and consider publishing the + templates; if recall < 95%, test a two-stage variant (coarse keep/drop, then action) before continuing. +* **Cost / Dependencies:** ~200 requests plus labelling; none. + --- ## 4. How to Add an Entry