EDGAR is the SEC's Electronic Data Gathering, Analysis, and Retrieval system. SupplyLens pulls large-cap US issuers' filings from EDGAR, normalizes them through a Bronze→Silver→Gold medallion architecture, and serves a research API on top — including vector search over narrative filings, insider-transaction rollups, and per-company filing timelines.
- ADR (Architecture Decision Record) and tooling
- Data Sources
- What makes this project interesting
- Overview of the Project in Stages
- Infrastructure Setup (Terraform)
- Data Quality & Observability
- Data Modeling with dbt
- Project Structure
- Virtual Environments
- Project Cost
- Potential improvements to make this project closer to "production grade"
- Testing
For day-to-day operations (running flows, backfilling, adding companies, querying the API), see docs/ARCHITECTURE.md.
The project is intentionally biased toward open, self-hostable primitives so the full system can run on a laptop with no vendor lock-in. Each layer of the stack is swappable.
| # | Decision | Why |
|---|---|---|
| 1 | Apache Iceberg as the table format for Silver + Gold | Schema evolution, hidden partitioning, time-travel, and Athena compatibility — without paying for a managed warehouse |
| 2 | PyIceberg (not Spark) to write Silver | Single-node, no JVM, runs comfortably in a small Docker container |
| 3 | dbt-athena-community to build Gold marts | SQL-only transformation layer; the same ref() graph that data teams already know |
| 4 | Qdrant Cloud for vector search | Free tier covers 16 companies, cosine-similarity search in <50 ms, REST + gRPC clients |
| 5 | Local model runner (Docker Model Runner + nomic-embed-text-v2-moe + Qwen 2.5 0.5B) for embeddings and topic labels | Zero API cost, no data leaves the host, GGUF quantisation runs on a Mac CPU. OpenAI API was rejected (cost + data egress); local sentence-transformers in venv was rejected (slower startup, harder GPU story) |
| 6 | Prefect 3 self-hosted (Docker server + runner) | Pure-Python, no DAG file, cron schedules live in flow code, UI is useful without being required. Airflow was rejected (too heavy on a laptop; Astro CLI adds another tool); |
| 7 | No graph database (Neo4j was originally in the design, removed in v2) | dbt marts + Athena SQL answer 90% of the analytical questions; a graph would be a curiosity, not a necessity. Neo4j was the original choice — removed because the operational overhead wasn't proportional to the value at 16-company scale |
| 8 | Soda 3 for data quality checks | Declarative YAML, runs against Athena via soda-core-athena, no separate server. Great Expectations was rejected (heavier, Python-only); "dbt tests only" was rejected (no row-level aggregates) |
| 9 | Terraform for all AWS infra | State file in the repo, one terraform apply provisions S3 / Glue / Athena / IAM |
| 10 | uv as the Python package manager | Lock file, fast resolver, PEP 723 inline metadata when needed |
- Language: Python 3.12 (required by pyiceberg 0.11 and the Prefect 3 image)
- Container: Docker Compose, two images (API on python:3.12-slim, runner on the same base with
uv) - Orchestration: Prefect 3.1 with
flow.serve()cron deployments - Storage: S3 + Glue Data Catalog + Athena (all provisioned by Terraform)
- Table format: Apache Iceberg (parquet data + JSON metadata)
- Transformation: dbt-core 1.8 + dbt-athena-community
- Vector search: Qdrant Cloud (free tier)
- Embeddings: nomic-embed-text-v2-moe-GGUF via Docker Model Runner (
/v1/embeddings) - Topic labels: Qwen 2.5 0.5B via the same Model Runner (cold path, training time only)
- Data quality: Soda 3.5.6 + soda-core-athena
- IaC: Terraform 1.8+ with AWS provider 5.x
- Dev tooling: ruff (lint + format), mypy (strict), pytest + pytest-asyncio, jupyterlab for ad-hoc analysis
EDGAR is the canonical source for US public-company filings. SupplyLens pulls 5 form types per company:
| Form | What it is | Key contents | How we use it |
|---|---|---|---|
| 10-K | Annual report (fiscal year, audited) | Full-year financial statements, business overview, risk factors, MD&A, properties, legal proceedings, executive comp summary | Chunked → embedded → searchable; the canonical source for company strategy and risk language |
| 10-Q | Quarterly report (Q1, Q2, Q3; unaudited) | QTD + YTD financial statements, MD&A for the quarter, updated risk factors (if material), default disclosures | Same as 10-K; quarterly cadence makes topic drift over time visible |
| 8-K | Material event report (within 4 business days of a triggering event) | One of ~35 item codes — 1.01 material agreements, 2.02 earnings releases, 2.05 restructuring, 5.02 executive departures/appointments, 5.07 shareholder vote results, 7.01 Reg FD, 8.01 other | Two paths: (a) narrative sections chunked + embedded, (b) item codes + summaries written to silver.material_events |
| DEF 14A | Proxy statement (annual, sent before the shareholder meeting) | Executive compensation tables (say-on-pay, golden parachutes, equity grants), director bios and independence, shareholder proposals, beneficial ownership, governance structure | Embedded for search; useful for "what did shareholders vote on" and "what is the CEO's total comp" |
| Form 4 | Insider transaction (within 2 business days, by an officer/director/10%+ owner) | Who (insider name, title, relationship to the company), what (shares, price per share, transaction code: P=Buy, S=Sell, A=Grant, F=Tax-withholding, etc.), when (transaction date), how (direct vs indirect ownership) | Structured XML, not chunked; parsed into silver.insider_transactions (one row per buy/sell/grant) |
Watchlist (16 US domestic issuers):
- 10 US semiconductor issuers: NVDA, AMD, INTC, QCOM, AVGO, MU, AMAT, LRCX, KLAC, TXN
- 6 US mega-cap issuers: AAPL, MSFT, GOOGL, AMZN, META, TSLA
Rate limits: SEC requires polite use — 10 requests/sec maximum, with a descriptive User-Agent header identifying the project + contact email. The pipeline defaults to 0.11 s sleep between requests (~9 req/s) to leave headroom.
Default scope: 1 calendar year (2025-01-01 → 2025-12-31). The pipeline is date-agnostic (not hardcoded) — widening the window is a one-line change in flows/watchlist.yaml.
- Foreign private issuers (e.g. TSMC CIK 0001046179) — file 20-F (annual) + 6-K (interim) instead of 10-K/10-Q/8-K, and only Form 4 for US-resident insiders. The current parsers target the SEC's canonical XML/HTML for domestic forms. Foreign-issuer support is a known extension point.
- Form 4 filing-agent XML variants — some filers (QCOM, TXN, KLAC, META) submit Form 4 through filing agents (Toppan Merrill, Donnelley) whose XML schema has subtle differences. The current parser handles the canonical schema and logs a warning + skips the variant. ~85-95% of Form 4s per company are still captured for the affected issuers.
- Full-text search of PDFs — every 10-K is also filed as PDF. We index the HTML primary document only.
- Pure-Iceberg medallion without a warehouse. Most "modern data stack" projects assume Snowflake/BigQuery/Databricks. This one uses S3 + Glue + Athena as a serverless warehouse, with PyIceberg doing the writes. The same
MERGE INTO-style upserts a managed warehouse gives you, on raw S3. - No API cost for embeddings. A 0.5B-param GGUF model running in Docker on a MacBook Air produces 24k embeddings in ~40 minutes. Zero per-token charges, no data egress.
- Local-first orchestration. A single
docker compose upbrings up the entire pipeline (server + runner + API + embedding model). No control plane outside the host. - Real filings, real companies. This is not a synthetic demo — the dataset is 1,643 actual 2025 SEC filings, 24,556 chunks, 24,556 Qdrant vectors. You can point the API at "AAPL Q3 2025 risk factors" and get back real 10-Q text.
- Open contracts between layers. The Pydantic models in
src/supply_chain/models.pyare the single source of truth for the data shape — the sameChunkobject is validated at parse time, written to Iceberg as parquet, and serialized to JSON for the API.
| Stage | Flow | Schedule (UTC) | Input | Output | Duration @ 16 cos. |
|---|---|---|---|---|---|
| 1. Ingest Bronze | ingest_bronze |
0 6 * * * |
EDGAR submissions JSON + filing index | S3 bronze/{cik}/{form}/{accession}.{ext} |
~25 min |
| 2. Parse Silver | parse_silver |
0 9 * * * |
S3 Bronze | Iceberg silver.{filings, chunks, insider_transactions, material_events} |
~40 min |
| 3. Embed Chunks | embed_chunks |
30 10 * * * |
Iceberg silver.chunks |
Qdrant filing_chunks collection (24,556 vectors) |
~40 min |
| 4. Soda Quality | check_silver_quality |
0 11 * * * |
Iceberg Silver | 10 pass/fail checks | ~30 s |
| 5. Source Freshness | check_source_freshness |
0 12 * * * |
silver.ingest_date vs dbt/.last_dbt_run |
Pass/warn/error per source | ~20 s |
| 6. dbt build | (run manually or via cron) | on-demand | Iceberg Silver | Iceberg Gold marts in supply_chain_gold |
~3 min |
| 7. API | (always running) | — | Athena + Qdrant | JSON | <500 ms per request |
The five Prefect flows are registered and served by flows/serve.py, the long-running entry point of the prefect-runner container.
All AWS resources are declared in infra/main.tf and tracked in infra/terraform.tfstate (local backend — fine for a single-developer project; switch to S3 + DynamoDB lock for team use).
| Resource | Name | Purpose |
|---|---|---|
| S3 bucket | supplylens-data-lake |
Holds Bronze, Silver, Gold data + Athena query results |
| Glue Catalog DB | supplylens_catalog |
Iceberg table metadata (PyIceberg writes here) |
| Athena workgroup | supplylens-workgroup |
Query execution context; output to s3://.../athena/results/ |
| IAM role | supplylens-athena-glue-role |
Service role for Athena + Glue with least-privilege S3 + Glue + Athena permissions |
- Versioning: enabled (rollback / accidental-delete recovery)
- Server-side encryption: AES256 (SSE-S3)
- Public access: fully blocked
- Lifecycle:
- 0 → 90 days:
STANDARD - 90 → 365 days:
STANDARD_IA - 365+ days:
GLACIER - Noncurrent versions expire after 30 days
- 0 → 90 days:
enforce_workgroup_configuration = true— clients cannot override per-querypublish_cloudwatch_metrics_enabled = true— query duration, data scanned, etc. visible in CloudWatch- Query result encryption:
SSE_S3 - Output bucket:
s3://supplylens-data-lake/athena-results/
supplylens-athena-glue-role is assumable by athena.amazonaws.com and glue.amazonaws.com only. It grants:
- S3:
Get/Put/Delete/List/GetBucketLocationon the data-lake bucket and all prefixes - Glue:
CreateDatabase/Table,GetDatabase/Table,UpdateTable,DeleteTable,CreatePartition+ batch variants, all on thesupplylens_catalogdatabase - Athena:
StartQueryExecution,GetQueryExecution,GetQueryResults,StopQueryExecution, prepared-statement management, all on*(Athena does not support resource-level auth on most of these)
cd infra/
terraform init
terraform plan
terraform applyResource names are all derived from the project_name variable (default supplylens) so the same module can be re-deployed for a different project by changing one variable.
Three layers of quality checks run on top of the data:
flows/check_silver_quality runs once a day at 11:00 UTC. It uses soda-core-athena to execute the checks in soda/checks/silver.yml against the four Silver Iceberg tables. Connection config lives in soda/conf/configuration.yml.
10 checks across 4 tables:
| Table | Checks |
|---|---|
filings |
row_count > 0, no null cik, no duplicate accession_number within a CIK |
chunks |
row_count > 0, no null cik |
insider_transactions |
row_count > 0, no null cik, no duplicate txn_id |
material_events |
row_count > 0, no null cik |
The "no duplicate accession_number" check is scoped to within a CIK because parse-silver uses truncate_by_cik before appending — duplicates across CIKs are expected; duplicates within one CIK are a parser bug.
dbt build runs 28 tests on the 6 mart models. These include not_null, unique, and relationship tests (e.g. every fct_filing_timeline.cik exists in dim_companies.cik).
dbt_project.yml defines:
vars:
freshness_warn_after_hours: 24
freshness_error_after_hours: 168The check_source_freshness flow wraps dbt source freshness and runs at 12:00 UTC. If a Silver source has no new ingest_date in 24h, it's a warn; in 168h, it's an error. This is the alarm that something in the pipeline stalled.
A successful dbt build writes the current UTC timestamp to dbt/.last_dbt_run. The check_source_freshness flow reads this file and warns if it's more than 24 h old — the Gold layer going stale is independent of the Silver layer going stale.
- Prefect UI at
http://localhost:4200— flow run history, per-task duration, logs, retry status - CloudWatch — Athena query metrics, S3 request metrics (when configured to publish)
- Athena query history — every SQL run is in the workgroup's history
- Iceberg time-travel —
SELECT * FROM table FOR SYSTEM_TIME AS OF '2026-01-15'to inspect prior snapshots
The dbt project lives in dbt/. It builds the Gold layer on top of the Silver Iceberg tables.
dbt/
├── dbt_project.yml
├── profiles.yml # connection to Athena + S3 (gitignored)
├── packages.yml # dbt_utils, etc.
├── models/
│ ├── staging/
│ │ ├── _sources.yml # 4 Silver sources + freshness config
│ │ ├── _staging.yml # 4 stg_* models + column tests
│ │ ├── stg_filings.sql
│ │ ├── stg_chunks.sql
│ │ ├── stg_insider_transactions.sql
│ │ └── stg_material_events.sql
│ └── marts/
│ ├── _marts.yml
│ ├── dim_companies.sql
│ ├── fct_filing_timeline.sql
│ ├── fct_insider_alerts.sql
│ ├── fct_insider_summary.sql
│ ├── fct_search_index.sql
│ └── fct_topic_prevalence.sql
└── .last_dbt_run # written by scripts/run_dbt.sh
One-to-one with Silver tables. Materialized as views (cheap, no storage). All rename snake_case columns, pass through the ingest_date audit column from Silver, and apply a single standard set of tests (not_null on cik, unique on natural keys).
The Gold layer follows Kimball dim_/fct_ naming, but the six marts aren't six variations on the same pattern — each one is shaped by how the API actually consumes it, which means the "grain" question has a different answer for nearly every table.
Grain and fact type
| Mart | Grain | Type |
|---|---|---|
dim_companies |
1 row per CIK | Dimension, no SCD (see below) |
fct_filing_timeline |
(cik, filing_year, form_type) |
Aggregate fact (count of filings) |
fct_insider_summary |
(cik, txn_year, txn_quarter, transaction_class, insider_tier) |
Aggregate fact |
fct_insider_alerts |
1 row per qualifying large transaction | Precomputed, filtered transaction fact |
fct_search_index |
1 row per chunk | Atomic fact, dual-storage source of truth |
fct_topic_prevalence |
(cik, topic_id) |
Contract-first placeholder |
Why dim_companies has no SCD, and why that's a deliberate call, not an oversight
At 16 fixed watchlist companies, company reference data changes rarely enough that history tracking
wasn't worth the complexity — the same reasoning applied to the static dimensions in the other
projects in this portfolio (e.g. Wattstream's DIM_ASSETS). SCD Type 2 solves a real problem, but
only when attribute changes are frequent enough, or consequential enough, to need point-in-time
correctness. Sixteen large-cap issuers with stable names and tickers don't clear that bar.
Aggregate facts vs. one precomputed exception-based fact
fct_filing_timeline and fct_insider_summary are standard aggregate facts — counts and
summaries rolled up to a coarser grain than the raw source data. fct_insider_alerts looks similar
(sparse, filtered) but is architecturally different: it bakes a business rule directly into the
table at build time — c-suite/officer sales ≥ $250k, plus a rolling 30-day window sum — so the
API serves alerts without recomputing the rule on every request. The trade-off is explicit: the
threshold lives in a WHERE clause in the model, not in application code, which makes it easy to
tune but means changing the alert threshold requires a dbt build, not a config change.
Grain enforcement via tests, not a single surrogate key
Rather than relying on one surrogate primary key to imply correctness, every mart's grain is
enforced directly with dbt_utils.unique_combination_of_columns tests against its declared grain
columns. The reasoning was blunt and practical: marts are queried directly by the API, so "a NULL
PK here means a 500 in production" — testing the grain explicitly catches a broken assumption
before it becomes a runtime error, rather than trusting a key that could be technically unique but
built on the wrong columns.
Deriving classifications once, with one deliberate exception to "staging is 1:1"
Business-logic mappings (SEC Form 4 transaction codes → 6 analyst categories, insider title →
tier, 8-K item codes → semantic labels) are resolved once in staging so no mart re-derives them.
The one exception: stg_material_events explodes the 8-K items array via cross join unnest,
changing the grain from 1 row per filing to 1 row per item — a deliberate break from the "staging
is a clean 1:1 passthrough" rule elsewhere in the project, made because unnesting an array is
fundamentally a cleaning operation, not business logic, and doing it early means every downstream
consumer sees one row per item without repeating the unnest.
Dual-storage consistency: Iceberg as source of truth, Qdrant as a served copy
fct_search_index is the analytical source of truth for filing chunks, while Qdrant holds the
HNSW-indexed copy actually used for vector search. Rather than treating these as two independent
stores, a qdrant_synced_at column on the Iceberg table tracks sync state explicitly — so a
mismatch between "what's in the lakehouse" and "what's searchable" is a queryable fact, not a
silent inconsistency.
Contract-first placeholder: designing the schema before the data exists
fct_topic_prevalence is intentionally empty (WHERE 1 = 0) but fully typed to its final schema.
This let the /timeline API be built and tested against a stable contract before the topic-modeling
flow that actually populates it existed — a genuinely useful pattern when a downstream consumer
and an upstream pipeline are built out of sequence.
Platform workarounds, documented rather than hidden
Two Athena/Trino limitations shaped the models directly: Athena Iceberg has no timestamptz, so
every mart casts to a plain UTC timestamp; and Trino can't mix GROUP BY with first_value(), so
dim_companies resolves the canonical company name in a separate CTE from the aggregation CTE
rather than fighting the query engine. Both are the kind of constraint that only shows up once you're
actually running SQL against the target engine, not from reading documentation.
Three of the six marts are partitioned by bucket(cik, 16) to align with the CIK-scoped predicates on all four API endpoints:
| Mart | Partition spec |
|---|---|
fct_filing_timeline |
bucket(cik, 16) |
fct_insider_alerts |
bucket(cik, 16) |
fct_insider_summary |
bucket(cik, 16) |
dim_companies |
none (16 rows — overhead exceeds the benefit) |
fct_search_index |
none (served by Qdrant for /search) |
fct_topic_prevalence |
none (placeholder; revisit on Day 5) |
The partitioning is declared inline on each mart's {{ config(...) }} block (not at the project level). Iceberg implements bucketing as hidden partitions, so the cik column is still queryable in the result set while the underlying file layout is bucket-sorted. With 16 companies the bucket count of 16 gives 6.25× headroom under the 100-partition soft cap and matches every CIK-scoped predicate the API issues. The first dbt build --select marts after this change triggers a clean rebuild of the three affected marts; subsequent runs are unaffected.
# from inside the prefect-runner container
docker compose exec prefect-runner bash -lc '
source /opt/prefect/cli_venv/bin/activate &&
cd /opt/prefect/dbt &&
dbt deps &&
dbt source freshness &&
dbt build --select tag:gold
'Or use the wrapper, which writes .last_dbt_run on success:
docker compose exec prefect-runner /opt/prefect/scripts/run_dbt.sh build --select martsThe dbt-athena-community adapter limitation: it writes Iceberg tables to Glue in a way that PyIceberg's GlueCatalog.load_table() cannot subsequently read (NoSuchPropertyException). The API works around this by querying the same marts via Athena instead of PyIceberg. This is the only place in the stack where the choice of write path and read path diverges.
edgar-supply-graph/
├── README.md ← you are here
├── CLAUDE.md ← project guidance for Claude Code
├── Dockerfile ← API image (python:3.12-slim)
├── prefect.Dockerfile ← runner image (python:3.12-slim + uv + cli_venv)
├── docker-compose.yaml ← 3 services: prefect-server, prefect-runner, api
├── pyproject.toml ← uv-managed Python deps
├── uv.lock
│
├── .env.example ← template; copy to .env (gitignored)
│
├── infra/ ← Terraform: S3 + Glue + Athena + IAM
│ ├── main.tf
│ ├── outputs.tf
│ └── terraform.tfstate ← local backend
│
├── flows/ ← Prefect flows (the "dags" of this project)
│ ├── serve.py ← single long-running process; registers all flows
│ ├── watchlist.yaml ← 16 CIKs + form types + date range
│ ├── ingest_bronze.py
│ ├── parse_silver.py
│ ├── embed_chunks.py
│ ├── check_silver_quality.py
│ ├── check_source_freshness.py
│ └── _lib/ ← shared utilities (iceberg, edgar client, parsers, s3, etc.)
│
├── src/supply_chain/ ← orchestrator-agnostic Python code
│ ├── config.py ← pydantic-settings: AWS, EDGAR, Qdrant, Prefect
│ ├── models.py ← Pydantic data contracts (Filing, Chunk, InsiderTransaction, ...)
│ └── api/
│ ├── main.py ← FastAPI app (4 JSON endpoints + /ui + /static)
│ └── static/ ← DaisyUI single-page UI (index.html, app.js)
│
├── dbt/ ← dbt-athena project (Gold layer)
│ ├── dbt_project.yml
│ ├── profiles.yml
│ ├── packages.yml
│ ├── models/
│ │ ├── staging/ ← 4 stg_* views
│ │ └── marts/ ← 6 fct_*/dim_* Iceberg tables
│ └── .last_dbt_run
│
├── soda/ ← Soda 3 checks for Silver
│ ├── conf/configuration.yml ← Athena connection
│ └── checks/silver.yml ← 10 checks
│
├── scripts/
│ └── run_dbt.sh ← dbt wrapper; writes .last_dbt_run
│
├── tests/ ← code-layer test suite (~35 tests, runs in <3s)
│ ├── conftest.py ← shared fixtures (mock settings, athena, qdrant, openai)
│ ├── test_models.py ← Pydantic validation
│ ├── test_config.py ← Settings + lru_cache
│ ├── test_watchlist.py ← YAML loader
│ ├── test_parsers.py ← Form 4 / 8-K / dispatcher
│ ├── test_chunker.py ← chunk_html_bytes
│ ├── test_api.py ← FastAPI integration with mocked clients
│ └── fixtures/ ← sample 8-K HTML + Form 4 XML
│
└── data/ ← (empty in repo) host-mounted volume if needed
| Image | Base | Purpose | Size |
|---|---|---|---|
Dockerfile (API) |
python:3.12-slim |
Runs FastAPI/uvicorn on port 8000 | ~600 MB |
prefect.Dockerfile (runner) |
python:3.12-slim |
Runs Prefect flows; two Python venvs (main + cli_venv for dbt/Soda) | ~1.2 GB |
Both images have a models: embedding block (Docker Model Runner) injected at compose time so the embedding model is served from the host's GPU/Metal without copying the weights into the container.
The project manages two Python venvs in production (inside the runner container) and one locally (on the host).
uv is the only Python tool the project uses outside containers.
# from the repo root
uv sync # install all deps into .venv from uv.lock
uv run python script.py # run a script in the venv
uv add package # add a runtime dep (updates pyproject.toml + uv.lock)
uv add --dev package # add a dev-only dep
uvx tool # run a CLI tool without installing it (e.g. `uvx ruff`)CLAUDE.md is explicit: always use uv add for runtime deps, uv add --dev for dev tools, and uv run for scripts. The .venv is activated automatically by uv run; the manual source .venv/bin/activate is only needed if you want a plain shell with the venv active.
The runner image needs both the Prefect-time deps and the dbt/Soda-time deps. The latter pull in pyathena, which still imports distutils (removed from Python 3.12 stdlib). Rather than try to keep one venv working, prefect.Dockerfile creates:
| venv | Path | Contents | Used by |
|---|---|---|---|
| main | /opt/prefect/.venv |
All pyproject.toml deps (Prefect, pyiceberg, boto3, fastapi, etc.) |
All flows; mounted from the image |
| cli_venv | /opt/prefect/cli_venv |
dbt-core, dbt-athena-community, soda-core-athena, setuptools |
dbt and soda invocations only |
Activate the CLI venv in shell snippets:
docker compose exec prefect-runner bash -lc '
source /opt/prefect/cli_venv/bin/activate &&
dbt --version
'The scripts/run_dbt.sh wrapper does this activation for you.
The whole stack fits in a free-tier budget for the 16-company, 1-year demo corpus. Real cost numbers from the last full run (2026-06-08):
| Service | What it does | Monthly cost |
|---|---|---|
| S3 storage | Bronze + Silver + Gold + Athena results | <$0.50 (≈30 GB total, mostly Bronze + Iceberg metadata) |
| Athena queries | dbt build + Soda scan + API queries | <$1.00 (≈100 GB scanned/mo, $5/TB) |
| Qdrant Cloud | 24k vectors, 768-dim | $0.00 (free tier: 1 GB) |
| Docker Model Runner | Local GGUF inference | $0.00 (runs on MacBook CPU) |
| Prefect | Self-hosted, single process | $0.00 |
| Total | < $2 / month |
What changes at scale (50+ companies, 3+ years, daily refresh):
- S3: ~$5-10/mo (Bronze dominates; lifecycle moves old Bronze to GLACIER at 90d)
- Athena: ~$10-20/mo (more scans, but Gold marts are small and cached)
- Qdrant Cloud: free tier exceeded around 100k vectors → ~$25/mo for the 1 GB+ plan
- Egress to Qdrant: ~$0 if Qdrant is hosted in the same region (currently
us-east4-0GCP, separate fromus-east-1AWS — paid egress is a known inefficiency; move Qdrant tous-east-1AWS for the prod build)
Honest list, ordered by impact:
- Postgres for Prefect metadata. SQLite works for one user on one laptop. The first day a second developer runs
prefect server starton their own machine, you'll have two divergent metadata DBs. SetPREFECT_API_DATABASE_CONNECTION_URLto a Postgres URL on a shared RDS instance. - Pre-commit hooks.
ruff check+ruff format+mypyon commit. Skipped today because the dev loop is single-user; would catch the issues that took the longest to find (CI cost: trivial). - CI. GitHub Actions:
ruff check,mypy,pytest,dbt buildagainst a dev AWS account. The pipeline is reproducible fromterraform apply+uv sync+docker compose up, so a CI environment is one workflow file away. - Idempotent
ingest_bronze. The flow currently downloads every filing in the date range on every run. Add anIf-None-Match(S3 ETag) check to skip already-downloaded files. Cuts Bronze sync time from 25 min to <1 min on a re-run. - Iceberg compaction job. PyIceberg writes produce many small Parquet files over time. A daily
rewrite_position_delete_files+rewrite_data_filescompaction keeps query latency predictable. Currently a "rewrite the world once a quarter" manual job.
- Tests, tests, tests. See Testing — no test suite today. The dbt + Soda coverage is real but doesn't cover parser edge cases (the Form 4 filing-agent variants, the 8-K with no items, the 10-Q with no MD&A).
- dbt exposures. Wire the API endpoints as dbt
exposuressodbt run --select +exposure:api_companiesknows which marts to rebuild. Currently marts are built by tag. - Lineage beyond dbt. OpenLineage + Marquez would close the gap between the Prefect run (which inserts into Silver) and the dbt run (which reads from Silver). Out of scope until something breaks that only lineage can explain.
- Foreign private issuers. TSMC, ASML, SAP all file 20-F + 6-K. Adding 6-K narrative parser + 20-F form-type mapping unlocks the rest of the global semis universe.
- Form 4 filing-agent XML variants. The 5 affected issuers (QCOM, TXN, KLAC, META partial, TSLA partial) lose 5-15% of insider transactions. A small variant-aware parser would close the gap.
- Pre-2024 data. The watchlist is 1 year. Widening to 3-5 years unlocks insider-pattern trend analysis, 8-K cadence shifts, and risk-factor evolution studies.
- Year/quarter partitioning across all three layers. Bronze S3 keys are flat (
bronze/{cik}/{form}/{accession}.ext), Iceberg Silver writes are not partitioned byfiling_year, and Gold dbt marts have nopartitioned_byconfig — so "show me NVDA's 2024 filings" is a list-everything-in-bronze/0001045810/operation. Closing the gap means changingFiling.bronze_key, addingpartition_byto the Iceberg table writes, andpartitioned_byto the dbt project — a meaningful refactor that also breaks theraw_s3_uricolumn ondim_companies. Out of scope until data contracts and SLAs are formalized.
- Athena query cost guardrails. Set
bytes_scanned_cutoff_per_queryon the workgroup (10 GB) to prevent a bad SQL from bankrupting the demo. Currently nothing limits a single query's blast radius. - PagerDuty / Slack alerts on Soda failure. The flow is registered with Prefect, so the alert is "Prefect flow run failed" → wire that to a channel.
- (DaisyUI web UI shipped — see
docs/ARCHITECTURE.md"Web UI".)
- AWS credentials via IAM role, not access keys. Long-term access keys in
.envare fine for a laptop demo; production needs an OIDC trust between the runner's host (or ECS task role) and AWS. - Qdrant payload encryption. Free tier doesn't support it. Upgrade to a paid plan for the production build.
The project has a focused code-layer test suite in tests/ (44 tests, runs in <3 seconds):
| File | Coverage |
|---|---|
test_models.py |
Pydantic validation: Filing.bronze_key, accession_clean, Chunk.token_estimate, InsiderTransaction rejects unknown TxnCode, MaterialEvent rejects non-8-K/DEF 14A, TopicAssignment probability bounds, TopicMetadata.created_at is UTC |
test_config.py |
Settings() reads env vars, rejects missing required fields at runtime, get_settings() is @lru_cached |
test_watchlist.py |
load_watchlist zero-pads CIKs, per-company overrides, defaults are required, _expand_form_types alias map |
test_parsers.py |
Form 4 XML → InsiderTransaction, malformed XML → warns accumulator, 8-K item code extraction, _build_title for director/officer/other combinations, _first_text default-on-missing, parse_filing dispatcher |
test_chunker.py |
chunk_html_bytes produces non-empty chunks, chunk_id is deterministic, token_estimate == len(text) // 4, _extract_section falls back to empty string |
test_api.py |
/health returns 200; /companies calls start_query_execution with dim_companies SQL; /companies/{cik}/filings includes cik = '{cik}' in SQL; ?year= adds filing_year filter; /companies/{cik}/insider-summary does the same on fct_insider_summary; POST /search calls OpenAI embeddings + Qdrant query_points; /ui returns HTML with all required DOM hooks; /static/app.js is served |
Run the suite:
just test # pytest only
just check # ruff + mypy + pytestOut of scope for the test suite: dbt models (covered by dbt's own framework + 28 dbt tests), E2E against live AWS/Qdrant (would need moto + a live Qdrant mock), flows/_lib/edgar_client.py and flows/_lib/iceberg.py (heavy I/O construction in __init__; downstream dbt tests cover the behavior).