Antifraud Detector

An end-to-end fraud detection project built as a portfolio piece against this job requirement:

Практический опыт построения ML-моделей для fraud detection или смежных задач (credit scoring, anomaly detection, churn prediction)

It covers the full lifecycle: EDA → leakage-safe modeling → cost-based evaluation → interpretability → a self code-review → adversarial/robustness testing ("pentest" of the model) — with every claim backed by a script you can re-run, not just prose.

Start here if you're short on time: reports/evaluation.md (does it work?) → reports/robustness_report.md (does it break?) → reports/code_review.md (what did a review catch?).

Data

Credit Card Fraud Detection (ULB Machine Learning Group / Worldline): 284,807 European card transactions over 2 days, 492 (0.17%) fraudulent. Features V1-V28 are PCA components of undisclosed raw features (confidentiality); Time and Amount are raw. scripts/download_data.py fetches it via the Kaggle API if credentials are present, otherwise falls back to a public mirror and validates the result against the dataset's known shape before accepting it (see the script for why that fallback is explicitly not good enough for a real pipeline).

Why this problem is harder than a generic classification task

• ~0.17% positive class. Accuracy is meaningless; the headline metric throughout is PR-AUC / recall-at-a-cost-optimal-threshold, not ROC-AUC.
• Deployment is against the future, not a random holdout. All splits are chronological (src/data_prep.py), not random — see reports/eda.md for why a random split would leak and inflate offline metrics.
• The decision threshold is a business decision, not 0.5. It's chosen by minimizing an explicit (documented, simplified) cost function on the validation fold only, in src/evaluate.py.
• A static classifier is not the whole system. The robustness report quantifies where this model alone falls short and what it would take operationally to close those gaps.

Repository layout

src/
  data_prep.py    # chronological split, dedup, leakage-safe preprocessing pipeline
  eda.py          # class balance / amount / timing analysis -> reports/eda.md
  train.py        # baseline LogisticRegression + XGBoost w/ small val-selected grid
  evaluate.py     # PR-AUC/ROC-AUC, cost-optimal threshold, confusion matrix, calibration
  explain.py      # SHAP global + local explanations
  inference.py    # FraudScorer: validated, production-shaped scoring entrypoint
scripts/
  download_data.py      # fetch + validate the dataset
  adversarial_tests.py  # black-box evasion / corruption / drift / determinism tests
tests/            # pytest: preprocessing invariants + input-validation edge cases
reports/          # every artifact below is generated by the scripts above, not hand-written
  eda.md, evaluation.md, interpretability.md, code_review.md, robustness_report.md
  *.json            # raw numbers behind each .md report
  figures/          # plots referenced by the reports
models/           # trained artifacts (preprocessor, baseline, XGBoost) — gitignored, regenerate via the commands below

Reproduce

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python scripts/download_data.py     # data/raw/creditcard.csv
python -m src.eda                   # reports/eda.md + figures
python -m src.data_prep             # data/processed/{train,val,test}.parquet, models/preprocessor.joblib
python -m src.train                 # models/{baseline_logreg,xgb_fraud_model}.joblib
python -m src.evaluate              # reports/evaluation.{md,json} + figures
python -m src.explain               # reports/interpretability.md
python -m pytest -q                 # 17 tests
python -m scripts.adversarial_tests # reports/robustness_report.md backing data

(libomp is required for XGBoost on macOS: brew install libomp if import xgboost fails with a libomp.dylib load error.)

Results (test set, threshold chosen on validation only)

Model	PR-AUC	ROC-AUC	Precision	Recall	Cost reduction vs. never flagging
Logistic regression (baseline)	0.754	0.984	0.979	0.635	56.0%
XGBoost (main)	0.770	0.986	0.753	0.743	61.5%

Full numbers, confusion matrices, and the cost model's assumptions are in reports/evaluation.md. Note the baseline and main model are close on PR-AUC — with only ~50-60 fraud cases per fold, that gap is within noise, and this is stated plainly rather than oversold; the cost-based comparison is the more decision-relevant number.

Robustness / "pentest" findings (see the full report for methodology and mitigations)

• Black-box evasion: 20% of caught frauds evadable within a 2,000-query budget via small, realistic (<1 std) feature changes.
• Operational robustness is the bigger risk: recall falls from 74% to 4% if half the input features silently go stale — a pipeline bug does more damage than a deliberate attacker.
• Temporal drift: PR-AUC falls from 0.81 to 0.70 across the test window alone, arguing for scheduled retraining and live monitoring rather than trusting one offline evaluation indefinitely.
• Full writeup, numbers, and concrete mitigations: reports/robustness_report.md.

What a self code-review caught

Lint fixes, an inaccurate docstring corrected against measured data instead of assumption, and — importantly — the tradeoffs that were kept on purpose with the reasoning written down (e.g. the validation fold doing double duty for both hyperparameter selection and threshold calibration). Full list: reports/code_review.md.

Honest limitations of this project

• V1-V28 being anonymized PCA components means SHAP explanations are useful for model debugging but not for analyst-facing reason codes — a real deployment needs the underlying feature dictionary.
• The cost model (FRICTION_COST = 5.0 per false positive, false negative = lost transaction amount) is a stated simplification, not a calibrated business cost matrix.
• No live scoring service was built, so the robustness testing targets the model, not a deployed API — see "Out of scope" in reports/robustness_report.md for what a service-level pentest would add.
• 48 hours of data is enough to demonstrate the methodology, not enough to fully validate long-horizon drift claims — the temporal-stability finding should be read as directional.