Add Stage 8 holdout and emission ablation package

HANDOFF_README.md

@@ -0,0 +1,152 @@
+# QVDF-E Handoff Package — 2026-07-25
+
+For: Mohammad (QVDFE repo owner)
+From: Simon's working session (Claude-assisted; all numbers machine-generated and reproducible)
+
+This package contains everything produced in the empirical-section redesign
+round: two bug fixes, the new Stage 8/8b holdout + emission-ablation
+experiments, the redesigned LaTeX empirical section (v12 draft) with 8
+figures and 6 tables, and all machine-readable results. Files are laid out
+**repo-relative** so you can copy them straight into your working trees.
+
+---
+
+## 1. What goes where
+
+### `qvdfe_changes/` → drop into `Mmdabb/QVDFE` repo root
+
+| File | Status | What it does |
+|---|---|---|
+| `experiments/codes/qvdfe_cbi/src/outliers.py` | **REPLACES existing** | Two-tier outlier screen (see §2) |
+| `experiments/codes/qvdfe_cbi/src/holdout_ablation.py` | new | Stage 8 core: site selection, blocked 70/30 date split, QVDF refit, B0–B3 ablation, paired bootstrap |
+| `experiments/codes/qvdfe_cbi/src/ablation_figures.py` | new | Figures A–E |
+| `experiments/codes/qvdfe_cbi/run_holdout_ablation.py` | new | Stage 8 driver (~10 s, post-processes frozen Stage 0–7 outputs) |
+| `experiments/codes/qvdfe_cbi/run_emission_supplement.py` | new | Stage 8b: Figure F + Table 5 |
+
+Run order (from repo root, after the main pipeline outputs exist):
+```bash
+python experiments/codes/qvdfe_cbi/run_holdout_ablation.py
+python experiments/codes/qvdfe_cbi/run_emission_supplement.py
+```
+Outputs land in `experiments/output/qvdfe_cbi/stage8_holdout_ablation/`.
+
+### `cbi_plus_changes/` → drop into `asu-trans-ai-lab/cbi_plus` repo root
+
+| File | Status | What it does |
+|---|---|---|
+| `cbi_pipeline/stage2_episodes.py` | REPLACES existing | Fixed `discharge_window()` (see §2) |
+| `cbi_pipeline/stage2b_measured_diagnostics.py` | REPLACES existing | Two-tier outlier screen + hard range rules |
+
+`tests/test_cbi_plus_fixes.py` verifies both fixes on synthetic episodes
+(all passing). Point `PKG_ROOT` at your cbi_plus checkout and run with
+plain `python`.
+
+### `paper/` → the v12 empirical section
+
+- `section_empirical_v12.tex` — drop-in replacement for
+  `\section{Empirical evaluation...}` (`sec:numerical`) of
+  `main_v11_community_narrative.tex`. Notation matches the v11 notation
+  table exactly. Self-contained numbers (no `\input` of generated tables).
+- `figures/` — the 8 PDFs it includes (`fig_A`–`fig_F` + the two Pigou
+  figures). Copy into the paper's `figures/` directory.
+- Needs: `enumerate` label style already used by the S1–S7 lists in v11.
+
+### `results/` — machine-readable evidence behind every number
+
+- `site_selection.csv` — the ex-ante rule applied to all candidate cells
+- `structural_validation_metrics.csv` — Table 3 source (P, μ_e, v_q: QVDF
+  vs Vickrey vs fixed-speed, held-out)
+- `emission_ablation_metrics.csv` — Table 4 source (B0–B3 + reconstructed
+  diagnostic, NMAE/MAPE/bias/R² + paired-bootstrap ΔNMAE CIs)
+- `pooled_cell_registry.csv` — the 9 blocked-holdout cells and their
+  train-block QVDF parameters
+- `evidence_ladder.csv`, `stage8_manifest.json`
+- `tables/table1..5_*.{csv,tex}` — auto-generated table files
+
+---
+
+## 2. The two bug fixes (please review these diffs first)
+
+**(a) Discharge-window tagging** (cbi_plus `stage2_episodes.py`,
+`discharge_window`). Old logic kept bins with `Δq ≤ 0` and spanned a
+non-contiguous mask. Queue dissipation moves *up* the congested FD branch
+— flow **rises** toward the discharge rate during recovery — so `Δq ≤ 0`
+tags the demand-starvation tail, not discharge, and the returned span
+contained bins that failed the rule's own criterion. New logic mirrors
+your validated `identify_discharge_window` in QVDFE
+(`cbi_state_transition.py`): below-v_c bins after t2, accepted only with
+positive-Δq share ≥ 0.60 (or non-negative net Δq) plus speed recovery;
+speed-only data keeps the relaxed path.
+
+**(b) Outlier screen** (both repos). Two defects in opposite directions:
+- *Too strict:* three 3-MAD Huber screens + MAD z-scores were unioned
+  flag-by-flag → legitimate heavy-congestion days excluded. Relational
+  ("soft") flags now need **two agreeing votes**.
+- *Out-of-range leak:* every rule was a `>` comparison, so NaN/inf passed
+  as False → "clean"; no absolute range checks existed. New "hard" rules
+  (non-finite state, P > period window, D/C > 8 h, speed outside
+  [0, 90] mph, non-positive μ) exclude outright.
+- Log-axis Huber fits no longer clamp non-positives to 1e-6.
+
+**Measured consequence on the frozen outputs:** 95 of 1,827 evaluated
+episodes have z > 8 (impossible demand accumulation) and are removed by
+the hard screen; the physically-admissible count drops **403 → 364**
+(strict 402 → 363). ~10% of the old admissible set was range leakage.
+Corridor split after fix: I-10 EB 151/151 (physical/strict), I-405 SB
+212/212, I-17 NB 1/0.
+
+---
+
+## 3. Experiment design and headline results
+
+Design (per the empirical-section redesign plan):
+- **Sites** (rule fixed before any emission error was examined; measured
+  volume, ≥12 clean episodes — the 27-weekday window caps cells at 18–19,
+  so the planned 20–30 floor is infeasible — stable FD, ≥50% strict
+  coverage, top CBI): **I-10 EB `pems::718412` PM** (CBI 3.89) and
+  **I-405 SB `pems::767053` AM** (CBI 4.36), both 18 episodes.
+- **Blocked holdout:** first 70% of dates calibrate, last 30% validate
+  (13/5 per site); pooled = 9 measured-flow cells, 41 validation episodes
+  (19 strict). No random episode splits.
+- **Ablation:** B0 mean-speed, B1 delay-only, B2 fixed queued speed, B3
+  FD-consistent v_e(z), vs observed-speed 5-min MOVES totals; paired
+  bootstrap (2,000) on ΔNMAE.
+
+Headlines:
+1. QVDF halves Vickrey's held-out duration error (10.5 vs 20.6% NMAE at
+   I-10; 15.7 vs 34.5% at I-405); μ_e(z) beats μ_e=C^h by 2–4×.
+2. **n hits the 1.0 bound at both sites**: the data identify a stable
+   retention *level* (μ_e/C^h = 1/f_d ≈ 0.72), not a retention *slope*.
+   Framed honestly in v12; n>1 needs multi-site pooling or longer windows.
+3. Delay-only B1 fails by +42% to +245%; every queued-state model removes
+   60–90% of that error (all CIs exclude 0). **B3 beats B2 decisively at
+   the I-405 merge** (NOx 29 vs 77% NMAE); B0 mean-speed is an honest,
+   strong total-mass baseline but unusable as the assignment primitive.
+4. Two-branch decomposition (Table 5): queue branch = 25–38% of totals for
+   CO2/CO/HC with 5–8% cubic-Γ error; **NOx has negative exact Γ (median
+   −1.2 g/h) and 38% cubic-Γ error** → v12 states the pollutant scope:
+   CO2/CO/HC support accounting + assignment; NOx needs trajectory rates.
+5. v_e(z) is biased low ~2× vs observed mean queued speed — the dominant
+   B3 error source; k_j = 220 sensitivity is the top follow-up.
+6. Pigou subsection (from your `QVDFE_Pigou_Additional_Results` package)
+   is integrated after the admissibility figure: z_EUE=13.303,
+   z_ESO=7.368 at λ=0.4, price of anarchy 1.099, ordering stable in λ.
+
+---
+
+## 4. Open items (suggested order)
+
+1. Full pipeline re-run (stages 2b–7, ~1.5 h) with the fixed outlier
+   screen, then regenerate Stage 8/8b → ladder counts in Fig A / Table 1
+   will shift slightly.
+2. k_j sensitivity (160–220 veh/mi/lane) and/or per-site ω calibration to
+   attack the v_e low bias.
+3. Multi-site pooled (shared-n) fit to test n > 1 identifiability.
+4. `DETECTOR = "deepest_queue"` robustness pass of Stage 8 (one constant).
+5. Wire `section_empirical_v12.tex` into the master tex and rebuild;
+   decide what of v11's `sec:numerical` moves to the appendix.
+6. PRs: when ready, apply `qvdfe_changes/` on a branch of `Mmdabb/QVDFE`
+   and `cbi_plus_changes/` on a branch of `asu-trans-ai-lab/cbi_plus`
+   (deferred by Simon's request — nothing has been pushed anywhere).
+
+See `NUMERICAL_EXPERIMENTS_PROGRESS.md` (included) for the full ledger.

NUMERICAL_EXPERIMENTS_PROGRESS.md

@@ -0,0 +1,158 @@
+# QVDF-E Numerical Experiments — Progress Log
+
+Status ledger for the redesigned empirical section (v12).
+Last update: **2026-07-25, round 2** (Claude-assisted session; all results machine-generated).
+
+## 0. Round-2 additions (emission supplement + Pigou)
+
+| Item | Status | Where |
+|---|---|---|
+| Stage 8b: Figure F (held-out obs-vs-model emission scatter, 4 pollutants) | DONE | `experiments/codes/qvdfe_cbi/run_emission_supplement.py` → `stage8_holdout_ablation/figures/fig_F_heldout_emission_scatter.*` |
+| Stage 8b: Table 5 (two-branch decomposition of B3 on strict held-out set) | DONE | `stage8_holdout_ablation/tables/table5_two_branch_decomposition.{csv,tex}` |
+| Pigou two-route subsection integrated into v12 section | DONE | `section_empirical_v12.tex` §"Two-route Pigou assignment illustration" (adapted from `QVDFE_Pigou_Additional_Results/paper/pigou_section_insert.tex`; figures copied into package `figures/`) |
+
+**Table 5 headline (strict held-out, n=19):** queue-exposure branch carries
+25% (CO2) / 38% (CO) / 37% (HC) of episode totals with Γ>0 and cubic-Γ
+errors of only 5–8%; **NOx has a NEGATIVE exact Γ (median −1.2 g/h)** and a
+38% cubic-Γ error — added queue exposure nominally *reduces* NOx under
+zero-acceleration running-exhaust rates. Figure F visualizes the practical
+consequence (B2 collapses to a flat total for a NOx episode cluster
+spanning ~an order of magnitude observed). v12 now states the pollutant
+scope explicitly: accounting + assignment defensible for CO2/CO/HC; NOx
+needs trajectory-resolved rates.
+
+**Pigou headline:** on the calibrated I-10 prototype (α=0.0049, β=1.648,
+Γ≈const over z∈[5.2,14]): z_TUE=14.000, z_TSO=7.754, z_EUE=13.303,
+z_ESO=7.368 at λ=0.4; emission-weighted price of anarchy 1.099 (exact-rate
+sensitivity: 1.086); ordering z_ESO<z_EUE≤z_TUE stable in λ. Verifies that
+emissions rescale the z^β coefficient without introducing α_e/β_e, and
+links to Fig E's admissible-domain statement.
+
+---
+
+## 1. What this iteration delivered
+
+| Item | Status | Where |
+|---|---|---|
+| Fix cbi_plus discharge-window tagging | DONE | `cbi_plus-master/cbi_plus-master/cbi_pipeline/stage2_episodes.py` (`discharge_window`) |
+| Fix cbi_plus outlier screen (two-tier + range rules) | DONE | `cbi_plus-master/cbi_plus-master/cbi_pipeline/stage2b_measured_diagnostics.py` |
+| Harmonize QVDF-E outlier screen with the same fix | DONE | `QVDFE-main/QVDFE-main/experiments/codes/qvdfe_cbi/src/outliers.py` |
+| Stage 8: blocked holdout + B0–B3 ablation + site selection | DONE | `experiments/codes/qvdfe_cbi/src/holdout_ablation.py`, `run_holdout_ablation.py` |
+| Figures A–E + Tables 1–4 (CSV + .tex) | DONE | `experiments/output/qvdfe_cbi/stage8_holdout_ablation/{figures,tables}` |
+| New LaTeX empirical section (v12 draft) | DONE | `QVDFE_paper_v11_community_narrative/QVDFE_paper_v11_package/section_empirical_v12.tex` (+ 5 figure PDFs copied to `figures/`) |
+| Unit tests for the two cbi_plus fixes | DONE (all pass) | scratchpad `test_cbi_plus_fixes.py` (synthetic episodes) |
+
+## 2. The two bug fixes (student's report — both confirmed real)
+
+**Discharge-window tagging (cbi_plus).** The old rule kept bins with
+`Δq ≤ 0` inside (t2, t3] and then spanned a non-contiguous mask. Queue
+dissipation moves the state *up* the congested FD branch, so flow *rises*
+toward the discharge rate during recovery; `Δq ≤ 0` tags the
+demand-starvation tail instead, and the span included bins failing the
+rule's own criterion. New rule (mirrors the validated QVDF-E
+`identify_discharge_window`): window = finite-speed bins below v_c after
+t2; accepted only with positive-Δq share ≥ 0.60 (or non-negative net Δq)
+plus speed recovery; speed-only data keeps the relaxed path.
+
+**Outlier screen (cbi_plus + QVDF-E).** Two defects in opposite
+directions, exactly as the student said:
+- *Too strict:* three independent 3-MAD Huber screens + MAD z-scores were
+  unioned one-by-one → legitimate heavy-congestion days excluded. Now
+  relational ("soft") flags need **two agreeing votes**.
+- *Out-of-range leak:* every rule was a `>` comparison, so NaN/inf values
+  compared False and sailed through as clean; there were no absolute range
+  checks at all. Now "hard" rules (non-finite state, P > period window,
+  D/C > 8 h, speed outside [0, 90] mph, non-positive μ) exclude outright.
+- Also: log-axis Huber fits no longer clamp non-positives to 1e-6 (was
+  dragging the robust fit toward log(1e-6) = −13.8).
+
+**Measured consequence:** the hard-range rescreen removes **95 of 1,827**
+evaluated episodes (all with z > 8, impossible demand accumulation), and
+the "physically admissible" set drops **403 → 364** (strict 402 → 363).
+~10% of the old admissible set was out-of-range leakage. Corridor split
+after fix: I-10 EB 151 physical/151 strict, I-405 SB 212/212, I-17 NB 1/0.
+
+## 3. Experiment design (per the redesign plan)
+
+- **Frozen substrate:** Stage 0–7 outputs of the prior full run (unchanged
+  on disk); Stage 8 is pure post-processing (~10 s runtime).
+- **Site rule (fixed ex ante):** measured volume; ≥ 12 clean episodes per
+  sensor-period cell (the 27-weekday window caps cells at 18–19, so the
+  planned 20–30 floor is infeasible — documented deviation); stable FD;
+  ≥ 50% strict coverage; highest CBI. Bound-free full-sample QVDF fit was
+  demoted from gate to tiebreaker (it excluded *every* high-CBI cell);
+  the site refit's own bound status is reported in Table 2.
+  - Selected: **I-10 EB `pems::718412` PM** (CBI 3.89, 18 eps, strict 1.00)
+  - Selected: **I-405 SB `pems::767053` AM** (CBI 4.36, 18 eps, strict 1.00)
+- **Blocked holdout:** first 70% of dates calibrate, last 30% validate
+  (13 train / 5 validation episodes per site); pooled run = 9
+  measured-flow cells → 41 validation episodes (19 strict). No random
+  episode splits (temporal leakage).
+- **Ablation:** B0 mean-speed, B1 delay-only, B2 fixed queued speed
+  (train-median), B3 FD-consistent v_e(z); benchmark = observed-speed
+  5-min MOVES totals; paired bootstrap (2,000) CIs on ΔNMAE vs B1;
+  reconstructed-speed MOVES carried as diagnostic.
+
+## 4. Headline results
+
+**Structural (held-out, Table 3):**
+- QVDF halves Vickrey's duration error: NMAE 10.5% vs 20.6% (I-10),
+  15.7% vs 34.5% (I-405); Vickrey bias is negative (episodes outlast the
+  fixed-capacity prediction).
+- μ_e(z) beats μ_e = C^h: 10.0% vs 28.9% (I-10), 27.2% vs 63.4% (I-405).
+- **n hits the 1.0 lower bound at both sites**: the data identify a stable
+  capacity-retention *level* (μ_e/C^h = 1/f_d ≈ 0.72, f_d ≈ 1.38), not a
+  retention *slope* (n > 1) at single-site scale. Honest framing in v12:
+  level supported decisively, slope not identifiable in 27 dates.
+- FD v_e(z) is biased low ~2× vs observed mean queued speed (−51%/−64%)
+  — the known within-link-storage/state tension, now quantified OOS.
+
+**Emission ablation (held-out, Table 4 / Fig D):**
+- B1 delay-only fails by construction: +42% to +245% bias (prices queued
+  hours at the reference-speed rate). Every queued-state model removes
+  60–90% of that error; all bootstrap CIs exclude zero.
+- **At the I-405 merge site B3 beats B2** (CO2 21.5 vs 28.5, NOx 29.0 vs
+  76.9, CO 10.9 vs 25.2 NMAE%); HC tie. Pooled strict: B3 wins CO, B2 wins
+  NOx, CO2/HC tie. Pollutant pattern matches the plan's prediction
+  (CO/CO2 support the reduction; NOx exposes rate-curve limits).
+- B0 mean-speed is a strong total-mass baseline (10–17% NMAE) — reported
+  honestly; it is not usable as the assignment primitive (needs the full
+  rate curve in-loop, no Γ·w̄ decomposition).
+- B3's residual is almost pure positive bias ← v_e biased low. Fixing the
+  congested-branch speed mapping is the highest-leverage refinement.
+
+**Admissibility (Fig E):** over both sites' calibrated z ranges all
+physical/rate/monotone-g conditions hold → separable emission-weighted
+assignment is admissible *on the identified domain*.
+
+## 5. How to reproduce
+
+```bash
+cd C:/source_codes/1_new_source_code/CBI_plus_QVDF_E/QVDFE-main/QVDFE-main
+python experiments/codes/qvdfe_cbi/run_holdout_ablation.py
+```
+Outputs → `experiments/output/qvdfe_cbi/stage8_holdout_ablation/`
+(`site_selection.csv`, `structural_validation_metrics.csv`,
+`emission_ablation_metrics.csv`, `pooled_cell_registry.csv`,
+`evidence_ladder.csv`, per-site state CSVs, `tables/`, `figures/`,
+`stage8_manifest.json`).
+
+## 6. Open items (next session)
+
+1. **Full pipeline re-run with the fixed outlier screen** (stages 2b–7,
+   ~1.5 h; Stage 5 reconstruction is the bottleneck). The two-tier screen
+   will slightly *grow* the clean set (soft flags now need 2 votes) while
+   hard rules remove the z > 8 rows at the source; episode counts in the
+   ladder will shift and Table 1/Fig A should be regenerated afterwards.
+2. **n > 1 identifiability**: pool sensor-period cells within a corridor
+   (mixed-effects or shared-n fit) or extend the observation window; the
+   single-site 27-date sample cannot separate retention slope from level.
+3. **v_e bias**: revisit k_j = 220 veh/mi/lane (sensitivity: k_j 160–220)
+   and/or calibrate ω per site; this is the dominant error source in B3.
+4. **DQ-detector robustness pass**: rerun Stage 8 with
+   `DETECTOR = "deepest_queue"` and report deltas (script constant).
+5. Decide whether v12 section replaces or augments v11 `sec:numerical`;
+   wire `section_empirical_v12.tex` into the master file and rebuild the
+   PDF (needs the enumerate label style already used by S1–S7 lists).
+6. Optional: per-episode appendix tables (exact-vs-cubic) unchanged from
+   v11 — move to appendix per the plan's §9.