Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 32 additions & 4 deletions docs-site/src/content/docs/experiments/proposed.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ title: "Proposed Experiments Register (Hypotheses & Designs)"
description: "Register of experiments we intend to run, each with a motivation, a falsifiable pre-registered hypothesis, a design, metrics, decision criteria, and dependencies. Seeded with the open questions from Invariant Decision Calibration (IDC)."
---

# Proposed Experiments Register

The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**. This page records experiments we **intend to run**, written down *before* any results exist, so that the hypothesis, the metric, and the pass/fail line can't quietly move after the data comes in.

**Lifecycle:** `Proposed` → `Ready` (dependencies met, design reviewed) → `Running` → `Done`. When an entry starts running, give it the next free `EXP-XX` number, add it to the ledger, and link it back here. Keep the `PROP-XX` entry, but mark it `Done → EXP-XX` and note any change to the design along with the reason for it.
Expand All @@ -28,7 +26,7 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**.

| ID | Title | Question it answers | Depends on | Priority | Status |
| :--- | :--- | :--- | :--- | :---: | :--- |
| [`PROP-00`](#prop-00-expose-mirror-tvd-on-all-four-surfaces-enabler) | Expose Mirror TVD on all four surfaces (*enabler*) | Can production use the mirror signal at all? | — | **P0** | Proposed |
| [`PROP-00`](#prop-00-expose-mirror-tvd-on-all-four-surfaces-enabler) | Expose Mirror TVD on all four surfaces (*enabler*) | Can production use the mirror signal at all? | — | **P0** | Closed: not worth it after [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) |
| [`PROP-10`](#prop-10-run-to-run-and-revision-noise-floor) | Run-to-run & revision noise floor | How big a difference is real? | — | **P0** | Done → [EXP-14](/dgem/experiments/exp-14-idc-rerun/) (noise floor: ±1–2 items, Brier ±0.02) |
| [`PROP-01`](#prop-01-held-out-temperature-scaling) | Held-out temperature scaling | Does the ECE gain from $T^*$ survive out of sample? | — | **P0** | Done → [EXP-14](/dgem/experiments/exp-14-idc-rerun/) (met on 231 JevBench items, not on 50) |
| [`PROP-02`](#prop-02-end-to-end-idc-cascade) | End-to-end IDC cascade | Does IDC + a mirror-aware gate beat the entropy-only cascade? | `PROP-00`, `PROP-01` | **P0** | Partly → EXP-14 (entropy gates); mirror gate moves to `PROP-13` |
Expand All @@ -41,17 +39,23 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**.
| [`PROP-09`](#prop-09-base-rate-label-shift-adaptation) | Base-rate (label-shift) adaptation | Can unlabeled target traffic correct for different class frequencies? | — | P2 | Proposed |
| [`PROP-11`](#prop-11-letter-collision-in-the-mirror) | Letter collision in the mirror | Is the mirror's damage caused by shared letters rather than by a second slot? | — | **P0** | Done → [EXP-15](/dgem/experiments/exp-15-letter-collision/) (supported) |
| [`PROP-12`](#prop-12-separate-pass-mirror) | Separate-pass mirror (replaces `PROP-03`) | Does a mirror read in its own pass give a clean order signal? | `PROP-11` | P1 | Done → [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) (supported, small effect) |
| [`PROP-13`](#prop-13-mirror-aware-cascade) | Mirror-aware cascade (unblocks `PROP-02`) | Does an uncoupled mirror improve the hand-off gate? | `PROP-12` | P1 | **Next** |
| [`PROP-13`](#prop-13-mirror-aware-cascade) | Mirror-aware cascade (unblocks `PROP-02`) | Does an uncoupled mirror improve the hand-off gate? | `PROP-12` | P1 | Closed: not worth it after [EXP-17](/dgem/experiments/exp-17-separate-pass-mirror/) |
| [`PROP-14`](#prop-14-in-context-vs-blank-question-prior) | In-context vs blank-question prior | Does a prior estimated from real items fix null-prior's overcorrection? | — | P1 | Proposed |
| [`PROP-15`](#prop-15-correction-strength-by-question-type) | Correction strength by question type (extends `PROP-06`) | Do yes/no and lettered choices need different correction strengths? | `PROP-14` | P2 | Proposed |
| [`PROP-16`](#prop-16-slot-names-are-part-of-the-prompt) | Slot names are part of the prompt | How much do slot ids change answers? | — | P1 | Done → [EXP-16](/dgem/experiments/exp-16-slot-names/) (single slot: no; second slot: yes) |
| [`PROP-17`](#prop-17-calibrated-agent-context-pre-compiler-internal-pilot) | Calibrated agent context pre-compiler (internal pilot) | Can one multi-slot pass decide which context blocks an agent turn needs, dropping little that matters? | — | P2 | Running (internal pilot) |

---

## 3. Entries

### `PROP-00`: Expose Mirror TVD on all four surfaces (*enabler*)

* **Closed (2026-09-28), not run.** EXP-15–EXP-17 showed the mirror signal adds little over hesitation: a
separate-pass mirror improves cross-validated error detection by at most 0.016 AUROC at twice the cost, and the
same-canvas mirror costs accuracy unless labels are digits. Exposing it on every surface (PROP-00) or gating the
cascade on it (PROP-13) is not worth the complexity; hesitation remains the production signal. Reopen only if a
cheaper, uncoupled order signal appears.
* **Motivation:** `--dual-mirror` computes Mirror TVD/JSD in `pkg/permutation.PostProcessDecisionResponse`, but `cmd/decide.go` discards the return values. `dgem serve`, MCP, and the Studio don't offer `dual_mirror` / `null_prior_debias` at all, which violates the four-surface parity rule in `AGENTS.md`. A mirror-aware gate (`PROP-02`) can't run in production until this is fixed.
* **Deliverable (not a hypothesis):**
* Add `diagnostics.mirror_tvd` / `mirror_jsd` per slot to the `dgem decide` output.
Expand Down Expand Up @@ -189,6 +193,11 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**.

### `PROP-13`: Mirror-aware cascade

* **Closed (2026-09-28), not run.** EXP-15–EXP-17 showed the mirror signal adds little over hesitation: a
separate-pass mirror improves cross-validated error detection by at most 0.016 AUROC at twice the cost, and the
same-canvas mirror costs accuracy unless labels are digits. Exposing it on every surface (PROP-00) or gating the
cascade on it (PROP-13) is not worth the complexity; hesitation remains the production signal. Reopen only if a
cheaper, uncoupled order signal appears.
* **Motivation:** Unblocks `PROP-02` once an uncoupled mirror exists.
* **Hypothesis (H13):** A gate of hesitation ≥ τ **or** disagreement ≥ τ′ beats entropy alone at the same hand-off rate.
* **Design:** Offline, using `PROP-12` readings and the existing Gemini-on-every-item receipt; choose τ, τ′ on half the items and report on the other half.
Expand Down Expand Up @@ -220,6 +229,25 @@ The [Experiment Ledger](/dgem/experiments/) records experiments we have **run**.
* **Cost:** about 1,200 requests.
* **Pre-registered design (2026-09-26, before data):** JevBench 231, Vertex G4, one session. Single slot: 3 baselines with id `decision` (noise band = min–max), then ids `q1`, `x7k2q` (random), `mirror`, `check`. Two slots (`--dual-mirror --mirror-mode copy`, so no letter collision): suffixes `__rev`, `__mirror_rev`, `_b`. An id "matters" if its correct count is more than 3 items outside the baseline band, or (two-slot) more than 3 items away from `copy`+`__rev`. Secondary: per-item agreement with the baseline majority answer. About 2,300 requests (revised up from 1,200).

### `PROP-17`: Calibrated agent context pre-compiler (internal pilot)

* **Motivation:** Agents re-send large context windows (tool schemas, history, retrieved documents) every turn. A
single `dgem` pass could grade every candidate block at once and use hesitation to downgrade uncertain blocks to a
one-line stub instead of dropping them. The pilot's design and data are internal for now; this entry holds the
number and the decision rule.
* **Hypothesis (H17):** On labelled agent turns, one pass keeps at least 95% of the blocks the gold labels mark as
required (`full` or `summary`), removes at least 50% of candidate tokens, and flags planted prompt-injection
blocks with recall ≥ 0.9; hesitation-driven "stub instead of drop" reduces required-block misses compared with
taking the most likely action.
* **Design:** ~40 labelled turns across several domains, 8 candidate blocks each (tool schemas, history turns,
retrieved documents, notes), gold action per block from one annotator with a second model labelling independently
to measure agreement; production serving image, `samples: 1`, neutral slot ids (EXP-16).
* **Metrics:** required-block recall (primary), token savings, quarantine precision/recall, rescue rate
(required blocks kept as stubs only because of hesitation), agreement with the gold labels.
* **Decision:** If H17 holds, build a larger labelled set from real agent traces and consider publishing the
templates; if recall < 95%, test a two-stage variant (coarse keep/drop, then action) before continuing.
* **Cost / Dependencies:** ~200 requests plus labelling; none.

---

## 4. How to Add an Entry
Expand Down
Loading