docs: add co-located backend AGENTS.md context files

builders/scripts/AGENTS.md

@@ -0,0 +1,107 @@
+# Datasets and builders (`builders/scripts`)
+
+Datasets are timeseries of `(timestamp, data)` pairs, uniquely identified by a `(name, version)`
+pair where version is a SemVer (e.g. `1.1.12`). A dataset may depend on other datasets (e.g. a
+5-day moving average depends on daily pricing). Datasets with no dependencies are root datasets.
+
+Builders are stateless Python scripts. Each dataset has exactly one builder that builds only that
+dataset. The server dynamically imports and runs builders in isolated subprocesses (see
+`builders/server/core/runtime/AGENTS.md`). Builder scripts are written by internal users; their
+code is trusted, but they are not trusted to never crash, so each runs in its own subprocess.
+
+## Directory layout
+
+```
+builders/scripts/<dataset_name>/<version>/
+  builder.py            builder script (required)
+  config.toml           dataset config (required)
+  requirements.txt      python dependencies (optional)
+  .env                  environment variables (optional, gitignored)
+  .env.template         documents required env vars (optional)
+  .venv/                per-builder venv (auto-created, gitignored)
+```
+
+Other Python files may live alongside and are imported via the module system.
+
+## `builder.py`
+
+```python
+from datetime import datetime
+from typing import Any
+
+def build(dependencies: dict[str, dict[datetime, list[dict]]], timestamp: datetime) -> list[dict[str, Any]]:
+    return [{"ticker": "AAPL", "price": 123}]
+```
+
+- `timestamp`: a `datetime.datetime` with microsecond precision.
+- `dependencies`: maps each dependency's **name** (not name+version) to a dict keyed by
+  timestamp, each value a list of data dicts. Without lookback the dict holds only the current
+  timestamp. With lookback it holds all timestamps in the window `[T - lookback + step, T]`
+  (N points inclusive). Versions are resolved by the server, so builders never reference them.
+- Return value: a list of dicts, one row per dict. Single-row datasets return a length-1 list.
+
+## `config.toml`
+
+```toml
+name = "dataset name"
+version = "0.0.1"
+builder = "builder.py"        # optional
+calendar = "NYSE"             # valid timestamps for this dataset (required)
+granularity = "1d"            # smallest time step, e.g. "1s", "1m", "1d"
+start-date = "2020-01-01"     # earliest date data can be built for, YYYY-MM-DD (required)
+env-vars = true               # optional, default false; load .env into the subprocess
+
+[schema]
+ticker = "str"
+price = "int"
+
+[dependencies]
+dependency-a = "0.0.2"                                    # simple: version only, no lookback
+dependency-b = { version = "0.0.1", lookback = "5d" }    # with lookback window
+```
+
+The `[schema]` section drives runtime validation: after a builder returns, each dict is checked
+for all declared keys with matching types before insert.
+
+## Start date
+
+`start-date` defines the earliest buildable date. At build time, if `end` is before it a
+`ValueError` is raised; if `start` is before it, `start` is clamped with a warning. The field is
+required and validated at config load.
+
+## Granularity
+
+Each dataset declares a `granularity`. A dataset may only depend on another whose granularity is
+finer or equal to its own (e.g. a daily dataset may depend on per-second or daily data, but an
+hourly dataset cannot depend on a daily one). Enforced at build time.
+
+## Lookback dependencies
+
+Some datasets need a window of historical data from a dependency (e.g. a moving average needs the
+last 5 days of closes).
+
+- Format: simple `dep = "0.1.0"` (no lookback), or `dep = {version = "0.1.0", lookback = "5d"}`.
+- Duration units: `d` (days), `h` (hours), `m` (minutes), `s` (seconds); must be positive.
+- Semantics: `lookback = "5d"` means an inclusive window of 5 points, `[T - 4d, T]`. After
+  parsing, deps are normalized to `{"version": str, "lookback_subtract": timedelta | None}`,
+  where `lookback_subtract` is `amount - 1` units (e.g. `timedelta(days=4)` for "5d").
+- Build-range expansion: when building a dependency recursively, the server expands its start via
+  `dep_start = start - lookback_subtract` so history covers the inclusive window.
+- Edge case: near a dependency's `start-date` the window may return fewer points; builders must
+  handle short windows gracefully.
+
+## Environment variables
+
+Set `env-vars = true` to have the server load a `.env` from the builder directory into the
+subprocess. The `.env` is gitignored and parsed only inside the subprocess, so secrets never
+reach the parent process. Commit a `.env.template` to document required variables (the server does
+not read it). See `builders/server/core/runtime/AGENTS.md` for the runtime behavior.
+
+## Mock builders
+
+For testing and development:
+- `mock-ohlc/0.1.0`: root, single-row OHLC for AAPL per timestamp.
+- `mock-daily-close/0.1.0`: depends on mock-ohlc, extracts the close (single row).
+- `mock-multi-ohlc/0.1.0`: root, multi-row OHLC for AAPL, MSFT, GOOG per timestamp.
+- `mock-multi-close/0.1.0`: depends on mock-multi-ohlc, extracts close per ticker (multi-row).
+- `mock-moving-avg/0.1.0`: depends on mock-daily-close with `lookback = "5d"`, 5-day moving avg.

builders/server/AGENTS.md

@@ -0,0 +1,93 @@
+# Backend map (`builders/server`)
+
+A single Python FastAPI service serves public API traffic and runs builds. It listens on
+port 3000. `main.py` is the uvicorn entrypoint (`main:app`): it creates the app, mounts the
+routers from `core/api`, and runs a `lifespan` handler that loads api keys, calls
+`load_all_configs(SCRIPTS_DIR)` + `setup_builder_venvs(SCRIPTS_DIR)`, and opens the DB pool.
+
+Detailed docs live co-located with the code they describe (this file plus the per-package
+`AGENTS.md` files below). Read the one nearest to what you are changing. Per-function
+mechanics live in module and function docstrings, which are the source of truth for "what a
+function does". These files carry the cross-cutting "why".
+
+## Layer layout
+
+Dependencies flow strictly downward: `api -> service -> db/runtime`. No layer imports upward.
+
+```
+builders/server/
+  main.py            entrypoint: FastAPI app, router mounts, lifespan, request-id middleware
+  log_config.py      central structlog configuration
+  core/
+    api/             endpoint handlers (APIRouter). See core/api/AGENTS.md
+    auth/            api-key hashing + verify_api_key dependency + mint CLI. See core/auth/AGENTS.md
+    service/         build orchestration, scheduling, execution, stores. See core/service/AGENTS.md
+    runtime/         config loading, subprocess isolation, schema validation, venvs. See core/runtime/AGENTS.md
+    db/              connection pool, dataset queries, migrations. See core/db/AGENTS.md
+    calendars/       calendar ABC + registry + definitions. See core/calendars/AGENTS.md
+    utils/           shared helpers (retry.py, semver.py)
+  workers/           subprocess_worker.py: stdlib-only builder worker (see core/runtime/AGENTS.md)
+  benchmarks/        end-to-end build benchmarks (see "Benchmarking" below)
+  tests/             mirrors the layer structure
+```
+
+Authoring datasets and builder scripts is documented in `builders/scripts/AGENTS.md`.
+Containers, Caddy, and infra live in `infra/AGENTS.md`.
+
+**Scripts directory resolution**: `SCRIPTS_DIR` in `core/runtime/config.py` defaults to a path
+relative to the source file (`../scripts` from the server package root). This works in Docker
+where scripts are volume-mounted at `/app/scripts`. For local dev the scripts live at
+`builders/scripts` (a sibling of `builders/server`), so the `SCRIPTS_DIR` env var overrides the
+default. `just backend-dev` sets this automatically.
+
+## Logging
+
+The server uses `structlog`. Configuration lives in `log_config.py`, called once at import
+time in `main.py`.
+
+**Processor pipeline**: `merge_contextvars` -> `add_log_level` -> `TimeStamper(iso)` -> renderer.
+The renderer is `ConsoleRenderer` by default, or `JSONRenderer` when `LOG_FORMAT=json` is set.
+
+**stdlib integration**: stdlib `logging` is routed through structlog via `ProcessorFormatter`,
+so uvicorn logs flow through the same pipeline.
+
+**Request context**: a FastAPI middleware in `main.py` clears contextvars per request and binds
+a unique `request_id`. The build and data endpoints also bind `dataset_name` and `version`.
+
+**What is logged**:
+- `main.py`: api key count at startup (info)
+- `core/auth`: matched `team` label bound to the request context on each authenticated request
+- `core/api/routes.py`: build and data-fetch failures (exception)
+- `core/service/scheduler.py`: start-date clamping (warning)
+- `core/service/orchestrator.py`: build plan summary (info), level start/complete (info)
+- `core/service/worker.py`: skipped builds (info), build progress (info), insert counts (info)
+- `core/db/connection.py`: new connections (debug)
+- `core/db/datasets.py`: query execution (debug), rows inserted (info)
+- `core/runtime/runner.py`: subprocess start/complete (info), stderr (warning), timeouts/crashes (error)
+- `core/runtime/registry.py`: config loaded during startup preload (debug)
+- `core/runtime/loader.py`: builder script imported (debug)
+- `core/runtime/venv_management.py`: venv creation progress (info), failures (exception)
+
+## CI
+
+Backend checks run in `.github/workflows/backend-ci.yml`. The workflow triggers on
+`builders/**`, `pyproject.toml`, `uv.lock`, and `.github/workflows/backend-ci.yml`, keeping
+backend CI scoped to builder and shared Python dependency changes.
+
+## Benchmarking
+
+End-to-end build benchmarks live under `builders/server/benchmarks/`. They measure wall-clock
+time for a full build request through the server (HTTP handler -> dependency resolution ->
+subprocess spawns -> DB insert) using a testcontainer postgres so the real database is not
+touched.
+
+- `just bench`: `pytest-benchmark` over a 90-day `mock-ohlc` build, 3 rounds, timing table.
+- `just bench-profile [DAYS]`: wraps the standalone `benchmarks/bench_build.py` with
+  `py-spy record --subprocesses`, producing `bench-flamegraph.svg`. The `--subprocesses` flag
+  captures the builder subprocess spawns, which dominate build time for simple datasets.
+
+The benchmark test uses `benchmark.pedantic(rounds=3, warmup_rounds=0)`: explicit rounds because
+each round is expensive, no warmup because every round must do real work (the DB is truncated
+between rounds by the `clean_db` fixture). The standalone `__main__` script does the same
+testcontainer setup without pytest so it can be wrapped directly by `py-spy`; module patching is
+done via direct attribute assignment rather than `monkeypatch`.

builders/server/core/api/AGENTS.md

@@ -0,0 +1,112 @@
+# API layer (`core/api`)
+
+Endpoint handlers using `APIRouter`. `routes.py` defines two routers, both mounted in
+`main.py` under the `/api/v1` prefix:
+
+- `public_router`: carries the unauthenticated `GET /status` (the docker healthcheck hits
+  `/api/v1/status`, so it must stay open).
+- `router`: everything else, mounted with `dependencies=[Depends(verify_api_key)]`, so every
+  other endpoint requires a valid api key. See `core/auth/AGENTS.md`.
+
+Handlers are thin: parse and validate input, delegate to `core/service`, shape the response.
+Business logic lives in the service layer, not here.
+
+## Endpoints
+
+```
+GET  /status
+GET  /datasets
+POST /build/{dataset_name}/{dataset_version}?start=<ts>&end=<ts>&dry-run=<bool>
+GET  /data/{dataset_name}/{dataset_version}?start=<ts>&end=<ts>&build-data=<bool>
+```
+
+### `GET /datasets`
+
+Returns all datasets pre-loaded into the runtime config registry at startup, annotated with
+whether each has any data in the DB. Delegates to `core.service.catalog.list_datasets`.
+
+```json
+{
+  "datasets": [
+    {"name": "mock-ohlc", "version": "0.1.0", "has_data": true},
+    {"name": "faang-daily-close", "version": "0.1.0", "has_data": false}
+  ]
+}
+```
+
+- `has_data`: `true` when at least one row exists in the DB for that `(name, version)` pair.
+- sorted alphabetically by name, then by version.
+
+| Status | Meaning |
+|--------|---------|
+| `200` | success |
+| `500` | unexpected failure (filesystem error, DB error) |
+
+### `GET /data`
+
+Fetches dataset data for a time range. By default it builds missing data before returning.
+`build-data=false` opts out for fast read-only access. Delegates to `core.service.builder.get_data`.
+
+Query params: `start`, `end` (required timestamps); `build-data` (optional, default `true`).
+
+```json
+{
+  "dataset_name": "mock-ohlc",
+  "dataset_version": "0.1.0",
+  "total_timestamps": 3,
+  "returned_timestamps": 3,
+  "rows": [
+    {"timestamp": "2024-01-02T00:00:00", "data": [
+      {"ticker": "AAPL", "open": 100, "high": 150, "low": 90, "close": 130}
+    ]}
+  ]
+}
+```
+
+Each entry in `rows` holds all data dicts for that timestamp (multiple rows can share a
+timestamp). Rows are sorted by timestamp; empty list when nothing matches.
+- `total_timestamps`: valid calendar timestamps in the requested range (via `generate_timestamps()`).
+- `returned_timestamps`: distinct timestamps actually returned from the DB.
+
+| Status | Meaning |
+|--------|---------|
+| `200` | Data complete, or `build-data=true` (default) was used |
+| `206` | `build-data=false` and `returned_timestamps < total_timestamps` (incomplete) |
+| `400` | Malformed input (invalid version or timestamp) |
+| `401` | Missing or invalid API key |
+| `422` | `build-data=true` but no valid calendar timestamps in range |
+| `500` | Unexpected failure (config not found, DB error) |
+
+### `POST /build`
+
+Builds missing data for the range and writes it to the DB. Delegates to
+`core.service.builder.build_dataset`. See `core/service/AGENTS.md` for the build pipeline.
+
+`dry-run` (optional, default `false`): when `true`, the whole build runs against a
+request-private in-memory store and nothing is written to the DB. A real build returns
+`{"status": "ok"}`; a dry run returns the produced rows:
+
+```json
+{
+  "dataset_name": "mock-ohlc",
+  "dataset_version": "0.1.0",
+  "dry_run": true,
+  "rows": [
+    {"timestamp": "2024-01-02T00:00:00", "data": [
+      {"ticker": "AAPL", "open": 100, "high": 150, "low": 90, "close": 130}
+    ]}
+  ]
+}
+```
+
+`rows` uses the same shape as `GET /data`. Builder and validation failures surface the same
+400/422/500 semantics as a real build.
+
+| Status | Meaning |
+|--------|---------|
+| `400` | Malformed input (invalid version format or unparseable timestamp) |
+| `401` | Missing or invalid API key |
+| `422` | Valid input but no valid calendar timestamps in range (e.g. weekday-only dataset over a weekend) |
+| `500` | Unexpected failure (config not found, builder crash, DB error) |
+
+The service is the only public-facing security boundary (auth, rate limiting, input validation).

builders/server/core/auth/AGENTS.md

@@ -0,0 +1,63 @@
+# Auth (`core/auth`)
+
+API-key authentication. Every endpoint except `GET /status` requires a valid api key sent as
+a bearer token:
+
+```
+Authorization: Bearer <raw-key>
+```
+
+`/status` is intentionally unauthenticated so the docker healthcheck (which hits
+`/api/v1/status`) keeps working. Enforcement is wired once in `main.py`: `public_router`
+(carrying `/status`) mounts open; `router` (everything else) mounts with
+`dependencies=[Depends(verify_api_key)]`.
+
+## Key storage
+
+Valid keys live in the `API_KEYS` env var as comma-separated `label:sha256hex` pairs, e.g.
+`default:ab12...,team-a:cd34...`. Only the sha256 hash of each key is stored, so a leaked
+config or log never reveals a working key. The `label` identifies the caller (a team) and is
+bound to the structlog context as `team` on every authenticated request. The env format is
+already a `label -> hash` map, so extending from one shared key to a key-per-team is just
+another entry, with no format change.
+
+## Verification
+
+The `verify_api_key` dependency hashes the presented token and looks it up in the key map.
+Comparing the hash (not the raw key) is timing-safe: the compared value is already a sha256 of
+the secret, so timing cannot recover the key.
+
+## Fail-closed startup
+
+The `lifespan` handler in `main.py` loads `API_KEYS` (via `load_key_map`) and raises if it is
+empty, so the service refuses to start unauthenticated on a public network. There is no
+"auth disabled" flag.
+
+## Key generation
+
+Mint a key and its env line with:
+
+```
+cd builders/server && uv run python -m core.auth generate <label>
+```
+
+This prints a `dsk_`-prefixed raw key (hand to the client) and the `label:hash` line (add to
+`API_KEYS`). Rotation is a superset operation: add a new key alongside the old, roll clients
+over, then remove the old entry.
+
+## Scaling assumption
+
+Keys are loaded from process env into an in-memory (cached) map, consistent with the
+single-uvicorn-worker model used elsewhere in the service.
+
+## Clients
+
+The Python SDK sends the header automatically when given a key: pass
+`DatastreamClient(api_key=...)` (or module-level `get_data(..., api_key=...)`), set it globally
+via `configure(api_key=...)`, or export `DATASTREAM_API_KEY`. The browser frontend does not yet
+send a key, so its requests currently return `401`; browser auth is handled in a later change
+(see `SPEC-frontend.md`).
+
+| Status | Meaning |
+|--------|---------|
+| `401` | Missing or invalid API key (on any endpoint except `/status`) |