Freamon enables data scientists to automatically reconstruct and query the intermediate data from ML pipelines to reduce the level of expertise and manual effort required to debug this data.
This repository contains a prototypical implementation for our abstract on "Reconstructing and Querying ML Pipeline Intermediates" to be presented at CIDR'23.
We provide notebooks to showcase provenance-based data debugging for a complex ML pipeline that learns to classify product reviews:
- The sklearn/pandas example notebook shows how debug the data of the ML pipeline implemented with sklearn and pandas.
- The pyspark/sparkml example notebook shows how debug the data of the ML pipeline implemented with pyspark and sparkml.