Semiconductor Yield Prediction: Predicting Wafer Fails Before Final Test

A BA-led machine learning project: from business problem to evaluated model.

The Business Problem

In semiconductor fabrication, a wafer that fails final test after running through hundreds of process steps represents wasted materials, tool time, and cycle time, cost that could have been avoided if the failure had been flagged earlier in the line.

This project uses real in-line sensor data from a semiconductor manufacturing process (590 sensors per wafer) to predict pass/fail outcomes before final test, so engineering teams can intervene earlier.

Dataset: UCI SECOM Dataset — 1,567 wafer records, 590 anonymised sensor features, binary pass/fail label. Heavily imbalanced (104 fails vs 1,463 passes) with real missing data — representative of genuine fab data quality, not a cleaned-up teaching dataset.

Why This Project

This isn't a model-accuracy exercise. It's a demonstration of the full path from ambiguous business need to a working, defensible solution, the same path a Business Analyst would drive on a real data/AI initiative:

• Defining the problem and success criteria before touching data
• Documenting data quality issues and the decisions made to handle them
• Choosing evaluation metrics that reflect actual business cost, not textbook accuracy
• Iterating against a stated requirement when the first model falls short, rather than reporting only the result that looks best
• Quantifying the benefit case, not just reporting a model score

Key Result

A logistic regression baseline fell short of the project's own recall requirement (≥0.50). This was diagnosed as likely due to non-linear sensor interactions, tested with a Random Forest comparison model, which met the requirement (recall 0.54) and more than tripled the net £ benefit on the test set. See docs/business-case.md for the full model comparison.

Project Structure

semicon-yield-prediction/
├── docs/
│   ├── business-case.md          # Problem, stakeholders, cost of the status quo, model outcome
│   ├── requirements.md           # Functional & non-functional requirements + acceptance status
│   └── data-quality-log.md       # Assumptions, missing data strategy, decisions
├── data/
│   ├── raw/                      # SECOM source files (not committed — see .gitignore)
│   └── processed/                # Cleaned dataset
├── notebooks/                    # Exploration and pipeline run notebooks
├── src/
│   ├── data_loader.py            # Load, quality-report, and clean the raw data
│   ├── train.py                  # Logistic regression baseline + business-translated evaluation
│   └── train_gbm.py              # Random Forest comparison model
├── app/                          # Streamlit demo app (in progress)
├── reports/                      # Evaluation summaries (auto-generated by src/ scripts)
└── README.md

Approach

1. Business case — quantify the cost of late-stage yield escapes and define what "useful" looks like for this model
2. Requirements — agree success criteria upfront (recall on fails prioritised over overall accuracy, given the cost asymmetry of a missed fail vs a false alarm)
3. Data quality assessment — handle 590 sensors with missing values, near-zero-variance features, and multicollinearity; every decision documented
4. Modelling — imbalanced classification, evaluated on recall, precision, and ROC-AUC rather than accuracy; threshold optimised for £ net benefit, not the default 0.5 cutoff
5. Model iteration — when the baseline missed the requirement, a second model was tested against identical conditions to test a specific hypothesis about why
6. Evaluation in business terms — model performance translated into cost avoided vs cost of false alarms, extrapolated to a hypothetical monthly fab volume

Running the Pipeline

pip install -r requirements.txt

1. Download the dataset: https://archive.ics.uci.edu/static/public/179/secom.zip — place secom.data and secom_labels.data in data/raw/
2. python src/data_loader.py — cleans data, prints data quality report, writes data/processed/secom_cleaned.csv
3. python src/train.py — trains logistic regression baseline, writes reports/evaluation_summary.md
4. python src/train_gbm.py — trains Random Forest comparison model, writes reports/evaluation_summary_gbm.md

Author

Teo — Founder, Nanosystems Advisory. Background in compound semiconductor physics (SiO2/Ta2O5/HfO2/Ga2O3) and DFT/TCAD modelling, applying BA discipline to technical and data-driven projects.