CFPB Complaint Analyzer

This repository contains streaming tools for working with the CFPB complaints dataset without loading the full raw file into memory. The main workflow in the repo is a visualization pipeline for complaints related to credit reporting and student loans, plus a couple of utility scripts for converting and sampling the dataset.

What This Repo Contains

The repo is organized into a small package under src/ plus thin script wrappers under scripts/.

Core Package

• src/cfpb_analyzer/analysis.py Streams data/complaints.jsonl, normalizes Product and Sub-product, filters to credit-reporting and student-loan complaints, and computes aggregate counts in one pass.
• src/cfpb_analyzer/plotting.py Renders plots from the aggregated results using matplotlib and seaborn.
• src/cfpb_analyzer/reporting.py Generates the Markdown summary with high-level findings and caveats.
• src/cfpb_analyzer/cleaning.py Filters, deduplicates, renames Issue/Sub-issue values, and applies a date cutoff. Two execution paths share the same streaming loop: a true in-place rewrite (no extra disk) and a .tmp+atomic-rename rewrite (used when renames may grow row length).
• src/cfpb_analyzer/cli.py Orchestrates the end-to-end analysis run for the visualization pipeline.

Scripts

• scripts/visualize_complaints.py Thin wrapper around the package CLI. This is the main entrypoint for the visualization workflow.
• scripts/complaints_csv_to_jsonl.py Converts the original CFPB CSV file into JSONL in a streaming way.
• scripts/clean_complaints_jsonl.py Filters, dedupes, renames, and date-trims the JSONL. Defaults to dropping pre-2018 complaints and rows whose Product/Sub-product aren't credit reporting or student loan.
• scripts/view_random_jsonl.py Builds a sidecar line-offset index and lets you inspect random JSONL entries without loading the full file into memory.

Tests

• tests/test_analysis.py Unit tests for normalization, categorization, and aggregation behavior.
• tests/test_cli.py Smoke test for the visualization CLI and expected output files.
• tests/test_clean_complaints_jsonl.py Covers the cleaning pipeline: filter, dedup, rename, date filter, dry-run, and the in-place vs. .tmp+rename code paths.

Data Files

By default the repo expects data in data/:

• data/complaints.csv Source CSV, if you start from the raw CFPB export.
• data/complaints.jsonl Streaming-friendly newline-delimited JSON version of the dataset.

Large data files and generated artifacts are ignored by Git in this repo.

Environment Setup

This project uses uv for dependency management.

1. Sync the environment

uv sync

That creates or updates .venv/ with the project dependencies.

2. Run commands through the environment

Use uv run to execute commands inside the project environment:

uv run python --version

Common Workflows

Convert CSV to JSONL

If you only have the CSV:

uv run python scripts/complaints_csv_to_jsonl.py

With explicit paths:

uv run python scripts/complaints_csv_to_jsonl.py \
  data/complaints.csv \
  data/complaints.jsonl

Useful flags:

• --limit: convert only the first N rows
• --progress-every: print progress every N rows

Clean the JSONL

Optional preprocessing step that shrinks the dataset and standardizes labels before the visualization run:

uv run python scripts/clean_complaints_jsonl.py \
  --issue-map data/issue_map.json --dry-run

Always preview with --dry-run first — it scans without modifying anything and prints the projected counts. When the numbers look right, drop --dry-run to apply the changes (you'll be prompted to type yes, or pass --yes).

What the cleaner does:

1. Filter — keeps only rows whose Product or Sub-product (after strip().casefold()) contains credit reporting or student loan.
2. Date filter — drops complaints whose Date received is before 2018 (or missing/malformed). Override with --min-year YYYY; --min-year 0 disables the filter.
3. Deduplicate — drops later occurrences of the same Complaint ID.
4. Rename — applies the Issue and Sub-issue mappings from the --issue-map JSON file.

Issue-map file format:

{
  "Issue": { "old label": "new label", "...": "..." },
  "Sub-issue": { "old label": "new label", "...": "..." }
}

Useful flags:

• --dry-run: scan and report only; never writes.
• --out-path NEW.jsonl: write the cleaned output to a new file instead of rewriting in place. Preserves the original.
• --min-year YYYY: change or disable the date cutoff (default 2018; 0 disables).
• --limit N: stop after N scanned rows; only allowed with --dry-run or --out-path.
• --yes: skip the destructive-write confirmation prompt.

Two execution paths chosen automatically:

• In-place when the rename map is empty — uses a trailing-write-pointer rewrite (rb+, seek/write/truncate). Zero extra disk needed during the run. A crash mid-run leaves the file corrupt; recover by re-running complaints_csv_to_jsonl.py against the CSV.
• .tmp+atomic-rename when the rename map is non-empty — needs ~equal extra disk during the run, but is crash-safe (a partial .tmp is unlinked, the original survives).

View Random Records From the JSONL

To inspect random complaints from a large JSONL file:

uv run python scripts/view_random_jsonl.py

Examples:

uv run python scripts/view_random_jsonl.py --count 5
uv run python scripts/view_random_jsonl.py --interactive
uv run python scripts/view_random_jsonl.py --raw

The first run creates a sidecar index file (*.idx) so later random lookups are fast.

Generate Visualization Artifacts

This is the main workflow:

uv run python scripts/visualize_complaints.py

By default it:

• reads data/complaints.jsonl
• filters to complaints where Product or Sub-product contains credit reporting or student loan
• writes plots and a generated summary to artifacts/credit_student_analysis/

Useful options:

uv run python scripts/visualize_complaints.py data/complaints.jsonl \
  --output-dir artifacts/credit_student_analysis \
  --top-n 10 \
  --progress-every 500000

Optional smoke-test mode:

uv run python scripts/visualize_complaints.py \
  --limit 10000 \
  --output-dir /tmp/cfpb-smoke

What the Visualization Pipeline Produces

The visualization run writes:

• summary.md
• category_share.png
• us_state_complaints_map.png
• credit_reporting_monthly_trends.png
• student_loan_monthly_trends.png
• credit_reporting_yearly_trends.png
• student_loan_yearly_trends.png
• credit_reporting_top_issues.png
• student_loan_top_issues.png
• credit_reporting_top_companies.png
• student_loan_top_companies.png
• credit_reporting_top_states.png
• student_loan_top_states.png
• student_loan_subproducts.png
• credit_reporting_response_outcomes.png
• student_loan_response_outcomes.png

The summary file includes:

• total matched complaint volume
• date range of usable complaints
• category dominance
• an overall US state complaint map with state acronyms
• issue concentration
• company concentration
• state concentration
• year-over-year and month-over-month spikes
• student-loan sub-product mix
• malformed JSON and date caveats

How the Analysis Works

The visualization pipeline is intentionally streaming and aggregation-first:

• It never loads the full raw jsonl file into memory.
• It scans the file once and updates counters as it goes.
• It keeps only compact aggregate results in memory for plotting.
• Plotting happens after the scan is complete.

Normalization rules:

• Product and Sub-product are matched with strip() and casefold()
• rows are included if either field contains credit reporting or student loan
• blank categorical fields are normalized to Unknown
• malformed JSON rows are skipped and counted
• missing or malformed Date received values are skipped for time-series charts and counted in the summary

Running Tests

Run the test suite with:

uv run pytest

The tests cover:

• category normalization and filtering
• aggregation correctness
• malformed-row handling
• end-to-end CLI artifact generation

Notes

• The plotting code uses a seaborn theme and palette for consistent output.
• State counts are raw complaint totals, not population-adjusted rates.
• The generated charts are static PNGs rather than an interactive dashboard.
• For the full dataset, expect the visualization run to take time because it scans the raw jsonl file directly on each run.