Autonomous Data Analysis Agent

An autonomous, multi-step AI agent that profiles arbitrary CSV datasets, surfaces data quality issues, makes decision-branching choices about how to handle each issue, and produces a recruiter-ready executive summary via an LLM.

Overview

Data quality work is repetitive but high-stakes. This project demonstrates how a small agent can take a raw CSV, reason about it across several discrete stages, and hand a human-readable report back to a stakeholder. It is designed as a reference implementation for agentic workflows: deterministic where determinism matters (profiling, rules) and generative where natural-language framing matters (summary).

Features

• End-to-end CLI run: one command from CSV to Markdown report
• Deterministic data profiler (row/column stats, null percentages, duplicate counts)
• Issue detector covering missing values, duplicate rows, IQR-based numeric outliers, and mixed-type columns
• Transparent rule-based decision engine: every issue is mapped to FLAG, SKIP, or ESCALATE with a written rationale
• LLM-backed executive summary using LangChain + OpenAI chat completions
• Timestamped Markdown report with profile, issues table, and appendix
• Graceful degradation when no API key is configured
• Unit tests for every deterministic component

Architecture

flowchart LR
    A[CSV File] --> B[DataProfiler]
    B --> C[IssueDetector]
    C --> D{DecisionEngine}
    D -->|low severity| E[SKIP]
    D -->|medium severity| F[FLAG]
    D -->|high severity| G[ESCALATE]
    E --> H[LLMSummarizer]
    F --> H
    G --> H
    H --> I[ReportWriter]
    I --> J[reports/timestamp_report.md]

Loading

The pipeline is one-directional. Each stage consumes the previous stage's output and emits a structured artifact. This makes the agent easy to test, easy to debug, and easy to extend with new detectors or decision rules.

Quickstart

git clone <repo-url>
cd 01-autonomous-data-analysis-agent
python -m venv .venv
.venv\Scripts\activate    # Windows
# source .venv/bin/activate  # macOS / Linux
pip install -r requirements.txt
cp .env.example .env       # then edit .env and add your OpenAI key
python -m src.agent --csv data/sample_transactions.csv

The report is written to reports/<timestamp>_report.md and a short summary is printed to the console.

Configuration

Variable	Required	Default	Purpose
OPENAI_API_KEY	No*	-	Enables LLM-generated executive summary
OPENAI_MODEL	No	gpt-4o-mini	Override the chat completion model

*If OPENAI_API_KEY is not set, the agent still runs and writes a full report; the summary section displays a placeholder.

Sample Output

# Data Analysis Report — 2026-05-28 09:14:02

## Executive Summary
The dataset contains 30 transactions across seven fields. Two issues require
attention: the notes column is missing in roughly forty percent of rows, and
one transaction amount appears to be an outlier worth reviewing. Duplicate
records were detected but represent a small fraction of the file.

Recommended next steps: confirm the outlier with the source system and
decide whether the notes field is optional or should be backfilled.

## Dataset Profile
- Rows: 30
- Columns: 7
- Duplicate rows: 2

## Issues & Decisions
| ID | Category   | Severity | Column   | Action    | Rationale                         |
|----|------------|----------|----------|-----------|-----------------------------------|
| 1  | missing    | medium   | notes    | FLAG      | medium severity is flagged        |
| 2  | duplicate  | high     | -        | ESCALATE  | high severity escalates           |
| 3  | outlier    | low      | amount   | SKIP      | low severity is skipped           |
| 4  | type       | high     | currency | ESCALATE  | high severity escalates           |

Decision Logic

Condition	Action	Meaning
severity == high	ESCALATE	Surface as a blocker for human review
severity == medium	FLAG	Note the issue in the report and continue
severity == low	SKIP	Acknowledge but do not surface in the summary

The mapping is constructor-injectable, so a downstream caller can tighten or relax thresholds without modifying the engine.

Limitations

• The agent operates on a single CSV per run; it does not join across sources.
• IQR-based outlier detection assumes roughly unimodal distributions.
• The LLM summary is non-deterministic. Reports should not be treated as audit artifacts.
• Schema drift detection is structural only; semantic drift (a column changing meaning) is out of scope.

Project Structure

01-autonomous-data-analysis-agent/
├── README.md
├── LICENSE
├── .gitignore
├── .env.example
├── requirements.txt
├── data/
│   └── sample_transactions.csv
├── docs/
│   ├── USAGE.md
│   └── ARCHITECTURE.md
├── src/
│   ├── __init__.py
│   ├── agent.py
│   ├── profiler.py
│   ├── detector.py
│   ├── decider.py
│   ├── summarizer.py
│   └── reporter.py
└── tests/
    ├── __init__.py
    ├── test_profiler.py
    ├── test_detector.py
    └── test_decider.py

License

Released under the MIT License. See LICENSE for full text.