diff --git a/builders/scripts/AGENTS.md b/builders/scripts/AGENTS.md new file mode 100644 index 00000000..457e9a0a --- /dev/null +++ b/builders/scripts/AGENTS.md @@ -0,0 +1,107 @@ +# Datasets and builders (`builders/scripts`) + +Datasets are timeseries of `(timestamp, data)` pairs, uniquely identified by a `(name, version)` +pair where version is a SemVer (e.g. `1.1.12`). A dataset may depend on other datasets (e.g. a +5-day moving average depends on daily pricing). Datasets with no dependencies are root datasets. + +Builders are stateless Python scripts. Each dataset has exactly one builder that builds only that +dataset. The server dynamically imports and runs builders in isolated subprocesses (see +`builders/server/core/runtime/AGENTS.md`). Builder scripts are written by internal users; their +code is trusted, but they are not trusted to never crash, so each runs in its own subprocess. + +## Directory layout + +``` +builders/scripts/// + builder.py builder script (required) + config.toml dataset config (required) + requirements.txt python dependencies (optional) + .env environment variables (optional, gitignored) + .env.template documents required env vars (optional) + .venv/ per-builder venv (auto-created, gitignored) +``` + +Other Python files may live alongside and are imported via the module system. + +## `builder.py` + +```python +from datetime import datetime +from typing import Any + +def build(dependencies: dict[str, dict[datetime, list[dict]]], timestamp: datetime) -> list[dict[str, Any]]: + return [{"ticker": "AAPL", "price": 123}] +``` + +- `timestamp`: a `datetime.datetime` with microsecond precision. +- `dependencies`: maps each dependency's **name** (not name+version) to a dict keyed by + timestamp, each value a list of data dicts. Without lookback the dict holds only the current + timestamp. With lookback it holds all timestamps in the window `[T - lookback + step, T]` + (N points inclusive). Versions are resolved by the server, so builders never reference them. +- Return value: a list of dicts, one row per dict. Single-row datasets return a length-1 list. + +## `config.toml` + +```toml +name = "dataset name" +version = "0.0.1" +builder = "builder.py" # optional +calendar = "NYSE" # valid timestamps for this dataset (required) +granularity = "1d" # smallest time step, e.g. "1s", "1m", "1d" +start-date = "2020-01-01" # earliest date data can be built for, YYYY-MM-DD (required) +env-vars = true # optional, default false; load .env into the subprocess + +[schema] +ticker = "str" +price = "int" + +[dependencies] +dependency-a = "0.0.2" # simple: version only, no lookback +dependency-b = { version = "0.0.1", lookback = "5d" } # with lookback window +``` + +The `[schema]` section drives runtime validation: after a builder returns, each dict is checked +for all declared keys with matching types before insert. + +## Start date + +`start-date` defines the earliest buildable date. At build time, if `end` is before it a +`ValueError` is raised; if `start` is before it, `start` is clamped with a warning. The field is +required and validated at config load. + +## Granularity + +Each dataset declares a `granularity`. A dataset may only depend on another whose granularity is +finer or equal to its own (e.g. a daily dataset may depend on per-second or daily data, but an +hourly dataset cannot depend on a daily one). Enforced at build time. + +## Lookback dependencies + +Some datasets need a window of historical data from a dependency (e.g. a moving average needs the +last 5 days of closes). + +- Format: simple `dep = "0.1.0"` (no lookback), or `dep = {version = "0.1.0", lookback = "5d"}`. +- Duration units: `d` (days), `h` (hours), `m` (minutes), `s` (seconds); must be positive. +- Semantics: `lookback = "5d"` means an inclusive window of 5 points, `[T - 4d, T]`. After + parsing, deps are normalized to `{"version": str, "lookback_subtract": timedelta | None}`, + where `lookback_subtract` is `amount - 1` units (e.g. `timedelta(days=4)` for "5d"). +- Build-range expansion: when building a dependency recursively, the server expands its start via + `dep_start = start - lookback_subtract` so history covers the inclusive window. +- Edge case: near a dependency's `start-date` the window may return fewer points; builders must + handle short windows gracefully. + +## Environment variables + +Set `env-vars = true` to have the server load a `.env` from the builder directory into the +subprocess. The `.env` is gitignored and parsed only inside the subprocess, so secrets never +reach the parent process. Commit a `.env.template` to document required variables (the server does +not read it). See `builders/server/core/runtime/AGENTS.md` for the runtime behavior. + +## Mock builders + +For testing and development: +- `mock-ohlc/0.1.0`: root, single-row OHLC for AAPL per timestamp. +- `mock-daily-close/0.1.0`: depends on mock-ohlc, extracts the close (single row). +- `mock-multi-ohlc/0.1.0`: root, multi-row OHLC for AAPL, MSFT, GOOG per timestamp. +- `mock-multi-close/0.1.0`: depends on mock-multi-ohlc, extracts close per ticker (multi-row). +- `mock-moving-avg/0.1.0`: depends on mock-daily-close with `lookback = "5d"`, 5-day moving avg. diff --git a/builders/server/AGENTS.md b/builders/server/AGENTS.md new file mode 100644 index 00000000..ef695fd2 --- /dev/null +++ b/builders/server/AGENTS.md @@ -0,0 +1,93 @@ +# Backend map (`builders/server`) + +A single Python FastAPI service serves public API traffic and runs builds. It listens on +port 3000. `main.py` is the uvicorn entrypoint (`main:app`): it creates the app, mounts the +routers from `core/api`, and runs a `lifespan` handler that loads api keys, calls +`load_all_configs(SCRIPTS_DIR)` + `setup_builder_venvs(SCRIPTS_DIR)`, and opens the DB pool. + +Detailed docs live co-located with the code they describe (this file plus the per-package +`AGENTS.md` files below). Read the one nearest to what you are changing. Per-function +mechanics live in module and function docstrings, which are the source of truth for "what a +function does". These files carry the cross-cutting "why". + +## Layer layout + +Dependencies flow strictly downward: `api -> service -> db/runtime`. No layer imports upward. + +``` +builders/server/ + main.py entrypoint: FastAPI app, router mounts, lifespan, request-id middleware + log_config.py central structlog configuration + core/ + api/ endpoint handlers (APIRouter). See core/api/AGENTS.md + auth/ api-key hashing + verify_api_key dependency + mint CLI. See core/auth/AGENTS.md + service/ build orchestration, scheduling, execution, stores. See core/service/AGENTS.md + runtime/ config loading, subprocess isolation, schema validation, venvs. See core/runtime/AGENTS.md + db/ connection pool, dataset queries, migrations. See core/db/AGENTS.md + calendars/ calendar ABC + registry + definitions. See core/calendars/AGENTS.md + utils/ shared helpers (retry.py, semver.py) + workers/ subprocess_worker.py: stdlib-only builder worker (see core/runtime/AGENTS.md) + benchmarks/ end-to-end build benchmarks (see "Benchmarking" below) + tests/ mirrors the layer structure +``` + +Authoring datasets and builder scripts is documented in `builders/scripts/AGENTS.md`. +Containers, Caddy, and infra live in `infra/AGENTS.md`. + +**Scripts directory resolution**: `SCRIPTS_DIR` in `core/runtime/config.py` defaults to a path +relative to the source file (`../scripts` from the server package root). This works in Docker +where scripts are volume-mounted at `/app/scripts`. For local dev the scripts live at +`builders/scripts` (a sibling of `builders/server`), so the `SCRIPTS_DIR` env var overrides the +default. `just backend-dev` sets this automatically. + +## Logging + +The server uses `structlog`. Configuration lives in `log_config.py`, called once at import +time in `main.py`. + +**Processor pipeline**: `merge_contextvars` -> `add_log_level` -> `TimeStamper(iso)` -> renderer. +The renderer is `ConsoleRenderer` by default, or `JSONRenderer` when `LOG_FORMAT=json` is set. + +**stdlib integration**: stdlib `logging` is routed through structlog via `ProcessorFormatter`, +so uvicorn logs flow through the same pipeline. + +**Request context**: a FastAPI middleware in `main.py` clears contextvars per request and binds +a unique `request_id`. The build and data endpoints also bind `dataset_name` and `version`. + +**What is logged**: +- `main.py`: api key count at startup (info) +- `core/auth`: matched `team` label bound to the request context on each authenticated request +- `core/api/routes.py`: build and data-fetch failures (exception) +- `core/service/scheduler.py`: start-date clamping (warning) +- `core/service/orchestrator.py`: build plan summary (info), level start/complete (info) +- `core/service/worker.py`: skipped builds (info), build progress (info), insert counts (info) +- `core/db/connection.py`: new connections (debug) +- `core/db/datasets.py`: query execution (debug), rows inserted (info) +- `core/runtime/runner.py`: subprocess start/complete (info), stderr (warning), timeouts/crashes (error) +- `core/runtime/registry.py`: config loaded during startup preload (debug) +- `core/runtime/loader.py`: builder script imported (debug) +- `core/runtime/venv_management.py`: venv creation progress (info), failures (exception) + +## CI + +Backend checks run in `.github/workflows/backend-ci.yml`. The workflow triggers on +`builders/**`, `pyproject.toml`, `uv.lock`, and `.github/workflows/backend-ci.yml`, keeping +backend CI scoped to builder and shared Python dependency changes. + +## Benchmarking + +End-to-end build benchmarks live under `builders/server/benchmarks/`. They measure wall-clock +time for a full build request through the server (HTTP handler -> dependency resolution -> +subprocess spawns -> DB insert) using a testcontainer postgres so the real database is not +touched. + +- `just bench`: `pytest-benchmark` over a 90-day `mock-ohlc` build, 3 rounds, timing table. +- `just bench-profile [DAYS]`: wraps the standalone `benchmarks/bench_build.py` with + `py-spy record --subprocesses`, producing `bench-flamegraph.svg`. The `--subprocesses` flag + captures the builder subprocess spawns, which dominate build time for simple datasets. + +The benchmark test uses `benchmark.pedantic(rounds=3, warmup_rounds=0)`: explicit rounds because +each round is expensive, no warmup because every round must do real work (the DB is truncated +between rounds by the `clean_db` fixture). The standalone `__main__` script does the same +testcontainer setup without pytest so it can be wrapped directly by `py-spy`; module patching is +done via direct attribute assignment rather than `monkeypatch`. diff --git a/builders/server/core/api/AGENTS.md b/builders/server/core/api/AGENTS.md new file mode 100644 index 00000000..47928db1 --- /dev/null +++ b/builders/server/core/api/AGENTS.md @@ -0,0 +1,112 @@ +# API layer (`core/api`) + +Endpoint handlers using `APIRouter`. `routes.py` defines two routers, both mounted in +`main.py` under the `/api/v1` prefix: + +- `public_router`: carries the unauthenticated `GET /status` (the docker healthcheck hits + `/api/v1/status`, so it must stay open). +- `router`: everything else, mounted with `dependencies=[Depends(verify_api_key)]`, so every + other endpoint requires a valid api key. See `core/auth/AGENTS.md`. + +Handlers are thin: parse and validate input, delegate to `core/service`, shape the response. +Business logic lives in the service layer, not here. + +## Endpoints + +``` +GET /status +GET /datasets +POST /build/{dataset_name}/{dataset_version}?start=&end=&dry-run= +GET /data/{dataset_name}/{dataset_version}?start=&end=&build-data= +``` + +### `GET /datasets` + +Returns all datasets pre-loaded into the runtime config registry at startup, annotated with +whether each has any data in the DB. Delegates to `core.service.catalog.list_datasets`. + +```json +{ + "datasets": [ + {"name": "mock-ohlc", "version": "0.1.0", "has_data": true}, + {"name": "faang-daily-close", "version": "0.1.0", "has_data": false} + ] +} +``` + +- `has_data`: `true` when at least one row exists in the DB for that `(name, version)` pair. +- sorted alphabetically by name, then by version. + +| Status | Meaning | +|--------|---------| +| `200` | success | +| `500` | unexpected failure (filesystem error, DB error) | + +### `GET /data` + +Fetches dataset data for a time range. By default it builds missing data before returning. +`build-data=false` opts out for fast read-only access. Delegates to `core.service.builder.get_data`. + +Query params: `start`, `end` (required timestamps); `build-data` (optional, default `true`). + +```json +{ + "dataset_name": "mock-ohlc", + "dataset_version": "0.1.0", + "total_timestamps": 3, + "returned_timestamps": 3, + "rows": [ + {"timestamp": "2024-01-02T00:00:00", "data": [ + {"ticker": "AAPL", "open": 100, "high": 150, "low": 90, "close": 130} + ]} + ] +} +``` + +Each entry in `rows` holds all data dicts for that timestamp (multiple rows can share a +timestamp). Rows are sorted by timestamp; empty list when nothing matches. +- `total_timestamps`: valid calendar timestamps in the requested range (via `generate_timestamps()`). +- `returned_timestamps`: distinct timestamps actually returned from the DB. + +| Status | Meaning | +|--------|---------| +| `200` | Data complete, or `build-data=true` (default) was used | +| `206` | `build-data=false` and `returned_timestamps < total_timestamps` (incomplete) | +| `400` | Malformed input (invalid version or timestamp) | +| `401` | Missing or invalid API key | +| `422` | `build-data=true` but no valid calendar timestamps in range | +| `500` | Unexpected failure (config not found, DB error) | + +### `POST /build` + +Builds missing data for the range and writes it to the DB. Delegates to +`core.service.builder.build_dataset`. See `core/service/AGENTS.md` for the build pipeline. + +`dry-run` (optional, default `false`): when `true`, the whole build runs against a +request-private in-memory store and nothing is written to the DB. A real build returns +`{"status": "ok"}`; a dry run returns the produced rows: + +```json +{ + "dataset_name": "mock-ohlc", + "dataset_version": "0.1.0", + "dry_run": true, + "rows": [ + {"timestamp": "2024-01-02T00:00:00", "data": [ + {"ticker": "AAPL", "open": 100, "high": 150, "low": 90, "close": 130} + ]} + ] +} +``` + +`rows` uses the same shape as `GET /data`. Builder and validation failures surface the same +400/422/500 semantics as a real build. + +| Status | Meaning | +|--------|---------| +| `400` | Malformed input (invalid version format or unparseable timestamp) | +| `401` | Missing or invalid API key | +| `422` | Valid input but no valid calendar timestamps in range (e.g. weekday-only dataset over a weekend) | +| `500` | Unexpected failure (config not found, builder crash, DB error) | + +The service is the only public-facing security boundary (auth, rate limiting, input validation). diff --git a/builders/server/core/auth/AGENTS.md b/builders/server/core/auth/AGENTS.md new file mode 100644 index 00000000..eed8fc08 --- /dev/null +++ b/builders/server/core/auth/AGENTS.md @@ -0,0 +1,63 @@ +# Auth (`core/auth`) + +API-key authentication. Every endpoint except `GET /status` requires a valid api key sent as +a bearer token: + +``` +Authorization: Bearer +``` + +`/status` is intentionally unauthenticated so the docker healthcheck (which hits +`/api/v1/status`) keeps working. Enforcement is wired once in `main.py`: `public_router` +(carrying `/status`) mounts open; `router` (everything else) mounts with +`dependencies=[Depends(verify_api_key)]`. + +## Key storage + +Valid keys live in the `API_KEYS` env var as comma-separated `label:sha256hex` pairs, e.g. +`default:ab12...,team-a:cd34...`. Only the sha256 hash of each key is stored, so a leaked +config or log never reveals a working key. The `label` identifies the caller (a team) and is +bound to the structlog context as `team` on every authenticated request. The env format is +already a `label -> hash` map, so extending from one shared key to a key-per-team is just +another entry, with no format change. + +## Verification + +The `verify_api_key` dependency hashes the presented token and looks it up in the key map. +Comparing the hash (not the raw key) is timing-safe: the compared value is already a sha256 of +the secret, so timing cannot recover the key. + +## Fail-closed startup + +The `lifespan` handler in `main.py` loads `API_KEYS` (via `load_key_map`) and raises if it is +empty, so the service refuses to start unauthenticated on a public network. There is no +"auth disabled" flag. + +## Key generation + +Mint a key and its env line with: + +``` +cd builders/server && uv run python -m core.auth generate