🧾 ExplainOps — Explainability as a Pipeline Stage

Your pipeline stores every prediction. It stores nothing about why.

MLOps solved model deployment. But when a regulator, an auditor, or a customer asks "why was this decision made?", most production systems have no record — the explanation was never an artifact in the first place. ExplainOps makes it one: versioned, partitioned, queryable, and generated as a normal pipeline stage.

The problem

Model predictions land in a warehouse table. Explanations, if they exist at all, live in a notebook a data scientist ran once. So:

• There is no history — you cannot answer "was support_tickets always the top driver, or did that change last Tuesday?"
• There is no lineage — the explanation is not tied to a run, a model version, or a date
• There is nothing to query — no SQL engine can read a notebook cell

Under the EU AI Act's Article 86 (right to explanation of individual decisions), "we can regenerate it if asked" is a fragile answer. Explanations need to be stored like any other pipeline output.

The idea

[Model + batch data] → [ExplainerStage] → [attributions DataFrame]
                                            ├── Parquet store (partitioned by model / run_date)
                                            └── summary JSON (for monitoring & alerting)

Two components, both plain Python:

from explainops import ExplainerStage, ExplanationStore

stage = ExplainerStage(model, feature_names)
attributions = stage.explain_batch(X_batch)      # per-feature attribution + std + rank

store = ExplanationStore("explanation_store")
store.write(attributions, model_name="churn_gbm")   # versioned Parquet + summary JSON
history = store.read_all("churn_gbm")               # every past run, as one DataFrame

The store lays files out as model=<name>/run_date=<date>/attributions_<run_id>.parquet, so DuckDB, Athena, or Spark can query explanation history directly — no bespoke API needed:

SELECT run_date, feature, attribution
FROM 'explanation_store/model=churn_gbm/run_date=*/attributions_*.parquet'
WHERE rank = 1
ORDER BY run_date;

That one query answers "did the model's top driver change over time?" — which is the foundation for attribution-drift monitoring.

Quick start

pip install -r requirements.txt
python demo.py                  # trains a churn model, explains a batch, stores and reads it back
python -m pytest tests/ -v

Components

File	What it does
src/explainops/explainer.py	ExplainerStage — permutation-importance attributions for any sklearn-compatible model (dependency-light by design)
src/explainops/store.py	ExplanationStore — versioned Parquet writer/reader, partitioned by model and run date, plus summary JSON
demo.py	End-to-end run: train → explain → store → read history
tests/test_core.py	Attribution shape and ordering guarantees

Roadmap

• Phase 1 — permutation-importance explainer + versioned Parquet store + demo
• Phase 2 — SHAP and Integrated Gradients backends behind the same interface
• Phase 3 — Airflow operator and Dagster asset, so explanations become a scheduled DAG step
• Phase 4 — drift hooks: alert when the top-ranked feature or attribution distribution shifts

Why this exists

I'm a Data Engineer at IBM and a PhD scholar in Trustworthy & Explainable AI — this project sits exactly at that intersection. My core thesis: you can't explain a model if you can't trace your data. Explainability starts in the pipeline, not the model.

📰 I write about this weekly in Explainable Pipelines · 💼 LinkedIn · 🐦 X · 🌐 Portfolio

License

MIT