Skip to content

INGEST.md

Latest commit

 

History

History
85 lines (61 loc) · 2.04 KB

INGEST.md

File metadata and controls

85 lines (61 loc) · 2.04 KB
Raw

How to ingest (2025 star schema)

Kimball star layout written by the ingest script:

data/parquet/f1/
  dims/
    drivers.parquet
    constructors.parquet
    circuits.parquet
    races.parquet
  facts/
    season=2025/
      results.parquet
      laps.parquet
      pit_stops.parquet
      qualifying.parquet

Prerequisites

cd /Users/kareem/Documents/Projects/Personal/Pitwall
npm install

Run 2025 ingest

npm run ingest -- --seasons 2025

Then validate:

npm run partition

Jolpica rate limits

Unauthenticated API (docs):

  • Burst: 4 requests / second
  • Sustained: 500 requests / hour

This project defaults to 8 seconds between live requests so a full season (~350–450 calls including lap pages) stays under the hourly cap.

Expect ~50–70 minutes for a cold 2025 ingest (laps are the expensive part).

If you get HTTP 429

  1. Stop the script (Ctrl+C) if it is stuck retrying.
  2. Wait ~30–60 minutes for the hourly window to recover.
  3. Re-run the same command:
npm run ingest -- --seasons 2025

Successful responses are cached under data/cache/jolpica/, so re-runs skip already downloaded pages and only fetch what is missing.

Optional env vars

Variable Default Meaning
JOLPICA_MIN_INTERVAL_MS 8000 Delay between live HTTP calls
JOLPICA_NO_CACHE=1 off Force refetch everything
JOLPICA_CACHE_DIR data/cache/jolpica Cache location
DATA_DIR data/parquet/f1 Lake root

Faster (risks 429):

JOLPICA_MIN_INTERVAL_MS=3000 npm run ingest -- --seasons 2025

What the script does

  1. Fetch season calendar, drivers, constructors
  2. Per round: results, qualifying, pit stops, paginated laps
  3. Stage in DuckDB, validate grain uniqueness
  4. Merge/upsert dims Parquet
  5. Replace facts/season=YYYY/ Parquet (idempotent)

Natural keys only (driver_id, constructor_id, circuit_id, season+round).