SEC Aggregator

I built this machine learning dashboard to help automate qualitative stock analysis, inspired by the fundamental approaches of investors like Peter Lynch and Warren Buffett. Instead of manually reading through hundreds of pages of SEC filings, this tool extracts, models, and visualizes the underlying narratives and risks from unstructured financial reports.

The project is designed to be completely local-first. It runs entirely on your own hardware, meaning zero cloud costs, no API subscriptions, and full data privacy.

✨ Features

• 100% Local Processing: All data processing and machine learning inference happen on your machine.
• SEC EDGAR API Integration: Production-grade data fetching that handles rate limits, caching, and connection pooling.
• Keyword-Optimized Parsing: A custom HTML parser extracts targeted sections of the filings based on specific keywords prior to ML ingestion. This keeps the NLP pipeline efficient and allows complex modeling to run smoothly on consumer hardware.
• Topic Modeling: Integrates BERTopic to automatically discover hidden narrative clusters and themes across thousands of filings.
• Interactive Dashboard: A modular, widget-based UI to explore the statistical data and narrative visualizations.
• Background Jobs: A queue system for processing multi-ticker, multi-year analyses without blocking the user interface.

🚀 Quickstart

Because this application relies on local machine learning models, GPU access is highly recommended. Using Docker is the most straightforward way to manage the necessary databases, background workers, and system dependencies.

Option 1: Docker (Recommended)

Prerequisites:

1. Docker installed.
2. NVIDIA Container Toolkit installed (to allow Docker to access your local GPU).
3. A .env file in the root directory containing your SEC API credentials (the SEC requires a User-Agent string, typically structured as Company Name email@domain.com).

Build and Run:

First, build the Docker image:

docker build -t sec-aggregator .

Then, run the container. The -v flag mounts a local data directory so your downloaded filings and database state persist between restarts:

docker run -d \
  --name sec-app \
  --gpus all \
  --env-file .env \
  -v $(pwd)/data:/app/data \
  -p 8000:8000 \
  sec-aggregator

If you have already created the container once, you do not need to run the full docker run command again. Instead, simply start the existing container:

docker start sec-app

Once running, the application will be available at:

• API / Dashboard: http://localhost:8000
• Swagger Documentation: http://localhost:8000/docs

Option 2: Local Installation

If you prefer to run the application outside of Docker, you can use uv. Please note that you will need to manually manage PyTorch GPU dependencies.

# Sync dependencies
uv sync

# Start the application
python -m src.run

(Make sure your .env file is properly configured in the root directory before starting).

📐 Architecture

The application is structured as a pipeline that fetches raw filings, chunks and filters the text, embeds the data using local Hugging Face models, and serves the resulting analytics to the frontend.

%%{init: {
  'themeVariables': {
    'clusterBkg': '#ffffff',
    'clusterBorder': '#cccccc',
    'fontFamily': 'arial'
  }
}}%%
graph TB
    classDef frontend fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#1967d2,rx:6px,ry:6px
    classDef backend fill:#e6f4ea,stroke:#1e8e3e,stroke-width:2px,color:#0d652d,rx:6px,ry:6px
    classDef pipeline fill:#fef7e0,stroke:#f29900,stroke-width:2px,color:#b06000,rx:6px,ry:6px
    classDef storage fill:#fce8e6,stroke:#d93025,stroke-width:2px,color:#a50e0e,rx:6px,ry:6px
    classDef external fill:#f1f3f4,stroke:#80868b,stroke-width:1px,color:#3c4043,rx:6px,ry:6px,stroke-dasharray: 4 4

    User((Analyst))

    subgraph External ["External Services"]
        direction LR
        SEC[SEC EDGAR API<br/>Data Source]:::external
        ModelHub[Model Registry<br/>LLMs & Embeddings]:::external
    end

    subgraph AppTier ["Application Tier"]
        UI[<b>Analytics Dashboard</b><br/>Web Interface]:::frontend
        AppServer[<b>Application Server</b><br/>API, State & UI Rendering]:::backend
        Orchestrator[<b>Task Orchestrator</b><br/>Background Job <br/>Management]:::backend
    end

    subgraph PipelineTier ["Data Processing & ML"]
        direction LR
        Ingest[<b>Ingestion Engine</b><br/>Fetch & Structuring]:::pipeline
        NLP[<b>Semantic Engine</b><br/>Parsing, Chunking, <br/>Embedding]:::pipeline
        ML[<b>Analytics Engine</b><br/>Topic Modeling, <br/>Clustering]:::pipeline

        Ingest --> NLP --> ML
    end

    subgraph DataTier ["Data & Storage Tier"]
        direction LR
        DocStore[(<b>Document & State Store</b><br/>Raw, Parsed, & Analytics Data)]:::storage
        VectorDB[(<b>Vector Database</b><br/>Semantic Search Index)]:::storage
    end

    User <-->|Interacts| UI
    UI <-->|API/View Requests| AppServer

    AppServer -->|Delegates Jobs| Orchestrator
    Orchestrator -->|Triggers Pipeline| Ingest
    AppServer -.->|Reads Data for UI| DataTier

    SEC -->|Raw Filings| Ingest
    ModelHub -.->|Weights/Models| NLP
    ModelHub -.->|Weights/Models| ML

    Ingest -->|Stores Raw & Parsed| DocStore
    NLP -->|Upserts Vectors| VectorDB
    ML -->|Stores Analytics| DocStore

    VectorDB -.->|Query Context| ML

Loading

🛠️ Development & Tests

The project uses pytest for testing. You can run the test suite using the following commands:

Run all tests:

python -m pytest -v

Run specific test directories or files:

python -m pytest tests/data/ -v
python -m pytest tests/data/test_extract_year.py -v

Run a specific test function:

python -m pytest tests/data/test_paragraph_chunker.py::test_max_words_split -v

Run tests with standard output (useful for debugging):

python -m pytest -v -s