Crawler Workflow

1. Read Data

   • Extract raw data from the source.

2. Classify the Data

   • Identify and categorize data types.

3. Detect PHI (Protected Health Information)

   • Apply rules or ML models to flag sensitive data.

4. Infer Schema

   • Automatically determine the structure and format of the data.

5. Compute Extended Statistics

   • Generate descriptive metrics (e.g., mean, standard deviation, distribution).

6. Detect Outliers

   • Identify anomalies and unusual patterns in the data.

7. Save to a JSON

   • Store processed results in a structured JSON format.