A statistically defensible shortlist of process signals associated with lot failure in the UCI SECOM semiconductor dataset — not a yield predictor.
Read REPORT.md for the findings, including what cannot be concluded.
Headline: of 590 signals, a naive screen returns 86 "significant" ones. After FDR control, stability selection, and re-deriving the shortlist inside chronological training folds only, one signal (S59) survives every test, with S21 and S129 supported and S129 limited to the later half of the record.
| Figure | What it shows | The number to take away | Script |
|---|---|---|---|
| S59 | S59 by pass/fail | P(fail > pass) = 0.692, Mann-Whitney p = 5.7e-11 | plot_signals.py |
| S21 | S21 by pass/fail | 0.603, p = 4.5e-04; effect lives in the tails, medians nearly coincide | plot_signals.py |
| S129 | S129 by pass/fail | 0.619, Welch t p = 6.8e-08 — but absent from the first 60% of the record | plot_signals.py |
| S103 | A rejected candidate | 0.668, p = 9.6e-09, yet selected in 0 of 4 training folds — redundant given S59 | plot_signals.py |
| S348 | A rejected candidate | shift holds at only 17 of 19 vigintiles; 0 of 4 folds | plot_signals.py |
| Fail rate over time | Label non-stationarity | 14.8% → 2.0% across the record, χ² p = 5.7e-10 | plot_failrate.py |
| Missingness over time | Why missingness is MAR-on-time | 58% vs 25% blockiness; Spearman(missing, time) = −0.417 | plot_missingness.py |
| p-values vs null | Why 86 hits is not 86 findings | null mean 24.1, and noise reached 84 once in 2000 draws | plot_pvalues.py |
| Stability frequencies | Selection frequency vs noise | noise ceiling 0.89; only S59 (0.98) clears it; 159 of 446 never selected | plot_stability.py |
| Split gap | Temporal vs random AUC | selection leakage +0.135; model A's rolling range 0.466–0.715 | plot_splitgap.py |
| precision@20 | What 20 inspections buy | 7 of 26 failures with the shortlist, 0 of 26 with all 590 | plot_precision.py |
Each effect is a distributional shift, not a separation. A randomly chosen failing lot exceeds a randomly chosen passing lot 69% of the time on S59, 62% on S129, 60% on S21 — useful for ranking lots to inspect, nowhere near a decision rule. Fail quantiles exceed pass quantiles at all 19 vigintiles for all three.
The fail rate swings 14.8% → 2.0% across the record (χ² p = 5.7e-10). That non-stationarity is why the temporal split is mandatory, why missingness had to be tested conditional on time, and why S129's association exists only in the later half.
28 signals sit in long instrument outages (median 58% of a column's gaps in one run) and are excluded rather than imputed; the other 562 scatter. Conditioned on time, none of the 538 partially-missing columns relate to the label — which is what licenses imputation.
A naive screen returns 86 signals at p<0.05. A true null with the same correlation structure averages 24 and reached 84 once in 2000 draws. Nearly all the evidence in 474 tests sits in one histogram bin, and most of that bin is still noise.
Ranked against what noise reaches at the same rank, the observed curve leaves the null band for only ~8 signals. Noise hit 0.89 at rank 1, so S59 (0.98) is the only signal above anything a permuted null produced. S64 reproduces in 4/4 folds and is still not a finding — univariately null, a suppressor.
Random splits train on future lots and flatter every model. The rolling-origin bars matter as much as the gaps: model A ranges 0.466–0.715 across four cuts. Model C's +0.135 is selection leakage, not split leakage.
Inspecting the top 20 of 471 lots catches 7 of 26 failures with the shortlist and zero with all 590 signals. 6 of those 7 hits are in the top 10. The bootstrap CI is [15%, 60%], so the panel ranks nothing.
S103 is the trap: the second-strongest picture in the set, yet L1 never selects it once S59 is in the model — a distribution plot cannot show redundancy. S348 fails on its own face, with the shift holding at only 17 of 19 vigintiles.
mkdir -p data && cd data
curl -sSfL -O https://archive.ics.uci.edu/static/public/179/secom.zip && unzip -o secom.zip && cd ..
python3 phase0.py # data pathology; decides whether imputation is legitimate
python3 phase0b.py # label non-stationarity + time-conditioned missingness
python3 phase0c.py # exact stratified permutation test (replaces an invalid CMH)
python3 phase1.py # univariate screen + empirical null (writes the .npy phase2 needs)
python3 phase2.py # BH / BY / Bonferroni + permutation FDR
python3 phase3.py # stability selection, full record
python3 phase3_folds.py # stability selection inside each training fold
python3 phase3_folds_calib.py # same, penalty calibrated to equal sparsity
python3 phase4.py # temporal vs random split, precision@20
python3 phase4b.py # selection-leakage measurement + bootstrap CIs
python3 phase5.py # final shortlist table
python3 plot_signals.py # figures/{s59,s21,s129,s103,s348}_distribution.png
python3 plot_failrate.py # figures/failrate_over_time.png
python3 plot_missingness.py # figures/missingness_over_time.png
python3 plot_pvalues.py # figures/pvalue_null.png (needs phase1.py's .npy)
python3 plot_splitgap.py # figures/split_gap.png (needs phase4.py's csv)
python3 plot_stability.py # figures/stability_frequencies.png
python3 plot_precision.py # figures/precision_at_20.pngRequires numpy, scipy, pandas, scikit-learn, matplotlib. Phases 3–4b take a few minutes each (permutation nulls and per-fold refits).
- No imputation precedes the Phase 0 missingness analysis; missingness is established as MAR-conditional-on-time before any value is filled, and every imputation is fit inside a training fold or resample.
- Per-signal test choice (Welch t vs Mann-Whitney) is driven by distribution shape.
- Null calibration is empirical wherever correlation between signals could distort it — 112 of 590 columns are near-duplicates at |r| > 0.99.
- The temporal split is mandatory for any headline number; random splits are reported only to show the gap.
- Selection leakage is measured directly (+0.146 AUC) by re-running the entire pipeline inside each training fold.
Code, report and figures: MIT.
The SECOM dataset is not redistributed here — data/ is gitignored and fetched from
UCI at run time, so it stays under its own terms rather than this licence.
Data: UCI SECOM (McCann & Johnston, 2008).