Skip to content

Commit 95cd4f9

Browse files
igerberclaude
andcommitted
synthetic-control: address CI codex R2 — poor-fit warning fires on flat pre-path (P2)
The poor-fit warning was gated by `pre_sd > 0`, so a FLAT treated pre-period path (SD == 0) with nonzero pre-RMSPE never warned even though the synthetic clearly fails to reproduce a constant series. Change the gate to the literal REGISTRY contract (warn when pre_rmspe > pre_sd), including the SD == 0 case, with a scale-aware absolute floor (1e-8 * max(|Z1|, 1)) so a near-perfect flat fit (RMSPE ~ roundoff) does not spuriously warn. REGISTRY poor-fit Note updated to document the flat-path behavior (slightly broader than SyntheticDiD's SD>0-gated form). Regression: test_poor_fit_warning_flat_treated_pre_path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 536aeb5 commit 95cd4f9

3 files changed

Lines changed: 27 additions & 3 deletions

File tree

diff_diff/synthetic_control.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -424,9 +424,14 @@ def fit(
424424
pre_rmspe = float(np.sqrt(np.mean(pre_gaps**2)))
425425
att = float(np.mean(post_gaps))
426426

427-
# Poor-fit warning (REGISTRY contract; mirrors synthetic_did.py).
427+
# Poor-fit warning (REGISTRY contract: warn when pre-RMSPE exceeds the SD of
428+
# the treated unit's pre-period outcomes). This includes a FLAT treated pre-path
429+
# (pre_sd == 0): any non-trivial RMSPE then means the synthetic cannot reproduce
430+
# a constant series. A scale-aware absolute floor (`_fit_tol`) guards against a
431+
# spurious warning on a near-perfect flat fit (RMSPE ~ roundoff).
428432
pre_sd = float(np.std(Z1, ddof=1)) if Z1.size > 1 else 0.0
429-
if pre_sd > 0 and pre_rmspe > pre_sd:
433+
_fit_tol = 1e-8 * max(float(np.max(np.abs(Z1))) if Z1.size else 0.0, 1.0)
434+
if pre_rmspe > pre_sd and pre_rmspe > _fit_tol:
430435
warnings.warn(
431436
f"Pre-treatment fit is poor: RMSPE ({pre_rmspe:.4f}) exceeds the "
432437
f"standard deviation of treated pre-treatment outcomes ({pre_sd:.4f}). "

docs/methodology/REGISTRY.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1979,7 +1979,7 @@ Classic synthetic control (donor/unit weights only) for a single treated unit, d
19791979
- **Note:** The standardization divisor `divisor = sqrt(apply(cbind(X0,X1), 1, var))` (per-predictor SD over donors+treated, ddof=1) and the inner/outer optimizer are **not specified in ADH 2010** (which defers these numerics to Abadie & Gardeazabal 2003 App. B / the `Synth` software). The divisor is pinned from the R `Synth::synth` source; `solution.v` lives in this scaled predictor space, so the deterministic R-parity test feeds `custom_v` in the same scaled space.
19801980
- **Note:** The outer objective minimizes the pre-period outcome MSPE over **all** pre periods, whereas R `Synth` uses a `time.optimize.ssr` window (1960–1969 in the Basque example). The nested `V` therefore differs from R by an efficiency-only choice (the paper notes inferential validity holds for *any* `V`), so end-to-end nested parity is a tolerance band, not equality.
19811981
- **Note:** `V` is parametrized on the unit simplex via a softmax of an unconstrained vector (trace-normalization is identification-fixing, not a constraint loss); the multistart Nelder-Mead + derivative-free Powell polish approximates R's best-of-`optimx` behavior over the non-smooth outer objective.
1982-
- **Note:** The 1×SD poor-fit threshold is a defensive implementation choice matching the `SyntheticDiD` convention; ADH 2010 gives only the qualitative guidance "do not use SCM when the fit is poor" (no numeric cutoff).
1982+
- **Note:** The 1×SD poor-fit threshold is a defensive implementation choice in the spirit of the `SyntheticDiD` convention; ADH 2010 gives only the qualitative guidance "do not use SCM when the fit is poor" (no numeric cutoff). The warning fires whenever pre-period RMSPE exceeds the SD of the treated unit's pre-period outcomes — **including a flat treated pre-path** (`SD = 0`) with non-trivial RMSPE (a scale-aware roundoff floor suppresses the warning on a near-perfect flat fit). This is slightly broader than `SyntheticDiD`'s `SD > 0`-gated form, matching the literal RMSPE-exceeds-SD contract above.
19831983
- **Deviation from R:** `standardize="none"` disables predictor standardization entirely; R `Synth` always scales by the predictor SD. Provided for diagnostics; changes the geometry of the `V` objective.
19841984
- **Note:** predictor rows support only **equal-weight** linear combinations of pre-period values — `mean` (`k_s = 1/T0`), `sum` (`k_s = 1`), and per-period outcome lags (identity, a single `k_s = 1`). ADH (2010) §2.3 defines the general form `Ȳ_i^{K_m} = Σ_s k_s Y_is` with *arbitrary* weights `k_s`; this release does NOT accept user-supplied non-uniform `K_m` weight vectors (and `median` and other non-linear aggregations are intentionally excluded). The supported set still spans the standard `Synth::dataprep` `predictors.op` + `special.predictors` usage; arbitrary-weight `K_m` is a deferred extension.
19851985
- **Deviation from R:** predictor/outcome **aggregation fails closed on any non-finite (NaN/inf) cell**, whereas R `Synth::dataprep` hardwires `na.rm=TRUE` (aggregating over the observed cells of a partially-missing window). The fail-closed contract is deliberate: na-dropping silently aggregates different period subsets across units, yielding incomparable predictors with no warning. The analyst must restrict `predictor_window` / `special_predictors` / `pre_period_outcomes` periods (and the outcome panel) to where each variable is observed; both partially- and fully-missing windows raise `ValueError`. Only the row *ordering* matches `dataprep`, not the missing-data handling.

tests/test_methodology_synthetic_control.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -416,6 +416,25 @@ def test_poor_fit_warning():
416416
synthetic_control(df, "y", "treated", "unit", "year", seed=0)
417417

418418

419+
def test_poor_fit_warning_flat_treated_pre_path():
420+
# Flat treated pre-path (SD == 0) that donors near 10 cannot reproduce: RMSPE > 0
421+
# must still warn (the former `pre_sd > 0` gate suppressed this case).
422+
rng = np.random.default_rng(2)
423+
years = list(range(2000, 2010))
424+
T0 = 7
425+
rows = []
426+
for j in range(4):
427+
for yr in years:
428+
rows.append({"unit": f"d{j}", "year": yr, "y": 10 + rng.normal(0, 0.1), "treated": 0})
429+
for i, yr in enumerate(years):
430+
rows.append(
431+
{"unit": "treated", "year": yr, "y": (5.0 if i < T0 else 8.0), "treated": int(i >= T0)}
432+
)
433+
df = pd.DataFrame(rows)
434+
with pytest.warns(UserWarning, match="Pre-treatment fit is poor"):
435+
synthetic_control(df, "y", "treated", "unit", "year", seed=0)
436+
437+
419438
# ---------------------------------------------------------------------------
420439
# Validation 7: duplicate predictor labels rejected
421440
# ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)