docs: document dataset proposal flow in specs

dev-docs/SPEC-backend.md

@@ -28,6 +28,10 @@ GET /data/{dataset_name}/{dataset_version}?start=<timestamp>&end=<timestamp>&bui
 DELETE /data/{dataset_name}/{dataset_version}?start=<timestamp>&end=<timestamp>
 ```
 
+```
+POST /datasets
+```
+
 ### API authentication
 
 Every endpoint except `GET /status` requires a valid API key sent as a bearer token:
@@ -179,6 +183,38 @@ Each entry in `rows` contains all data dicts for that timestamp (matching the DB
 | `404` | Dataset exists but has no rows in the requested range |
 | `500` | Unexpected failure (DB error) |
 
+### Dataset proposals endpoint
+
+`POST /datasets` submits a new dataset for review. **The server never writes to its own scripts directory**: it validates the submission, generates the dataset files, and opens a GitHub pull request adding `builders/scripts/<name>/<version>/` to the repo. The dataset goes live only after the PR is reviewed, merged, and the server restarts. This keeps code review as the trust gate — a builder script is code that executes on the server, and the "internal users write trusted builders" assumption (see "Build behavior") is enforced by human review, not by the API key alone.
+
+**Request body** (JSON): `name`, `version`, `calendar`, `granularity`, `start_date`, `schema` (field → type map), `builder_script`, proposer identity (`author_name`, `team`, `discord_user`, `description` — all required, surfaced in the PR body), and optional `dependencies` (list of `{name, version, lookback?}`), `env_vars`, `requirements_txt`, `env_template`.
+
+**Validation** (`core/service/proposals.py`), in order:
+1. name must match `^[a-z0-9][a-z0-9_-]*$` (it becomes a directory and branch name); version must parse as SemVer; proposer fields must be non-empty
+2. `(name, version)` must not already exist in the config registry
+3. the canonical `config.toml` is generated, then **those exact bytes** are re-parsed and run through the same `validate_config` used at startup — the committed config is byte-identical to what was validated
+4. registry cross-checks against live configs: dependencies exist, granularity is not finer than any dependency's, start-date is not before any dependency's (a new dataset is a leaf nothing references, so cycles are impossible)
+5. builder script: AST-parsed, must define a top-level `build(dependencies, timestamp)` with exactly two positional args
+6. the script is then run through **ruff autofix + format** server-side (same rule selection as repo CI — kept in sync manually with the root `pyproject.toml`); unfixable violations reject the proposal with ruff's output. Proposal PRs therefore land pre-linted and cannot fail the CI lint gate
+
+**PR shape**: branch `add-dataset/<name>-<version>` off `main`, one commit containing `config.toml`, `builder.py`, and (when provided) `requirements.txt` / `.env.template`. **The real `.env` is never committed** — for `env-vars = true` datasets the PR body carries a checklist item to place it on the server manually before the first build. Title: `feat: add dataset <name>/<version>`. Reviewers are auto-requested (best-effort: the PR author is excluded, and a failed batch falls back to per-reviewer requests).
+
+**Configuration** (env vars, documented in `infra/.env.template`):
+- `GITHUB_TOKEN` (required for this endpoint only): PAT with Contents + Pull requests read/write on the repo. Without it the endpoint returns 502; everything else works.
+- `GITHUB_REPO` (default `Wat-Street/datastream`), `GITHUB_API_URL` (default `https://api.github.com`; overridable for testing against a fake), `GITHUB_REVIEWERS` (default `Blackgaurd,Scr4tch587`).
+
+**Response** (200): `{"dataset_name", "dataset_version", "pr_url", "branch"}`.
+
+| Status | Meaning |
+|--------|---------|
+| `400` | Invalid submission (validation or lint failure; `detail` is safe to show in the UI) |
+| `401` | Missing or invalid API key |
+| `409` | Dataset already registered, or a proposal branch for it is already open |
+| `422` | Malformed request body (missing/mistyped fields) |
+| `502` | GitHub unreachable or `GITHUB_TOKEN` missing/invalid |
+
+The requesting API key's team label is logged and included in the PR body alongside the proposer fields. A CI test (`tests/core/runtime/test_real_scripts.py`) validates every real config in `builders/scripts/` with the startup validation path, so green CI on a proposal PR means the dataset cannot break server boot.
+
 ### Build behavior
 
 - On a `POST /build` request, it builds missing data for the requested range and writes it to the database.

dev-docs/SPEC-frontend.md

@@ -18,9 +18,10 @@ This is a purely internal tool. The goal is functional clarity, not aesthetics.
 
 ### Navigation
 
-State-based navigation in `App.tsx`. Two views:
+State-based navigation in `App.tsx`. Three views:
 - `'list'`: dataset catalog (landing page)
 - `'detail'`: dataset data viewer
+- `'create'`: dataset proposal form (lazy-loaded via `React.lazy` — its CodeMirror chunk never loads during normal browsing)
 
 No routing library. A single discriminated-union `view` state controls which component renders. This also means GitHub Pages needs no SPA 404 fallback.
 
@@ -48,11 +49,12 @@ frontend/src/
     json-view.tsx                # recursive jsx syntax highlighter (no innerHTML)
     json-modal.tsx               # row details in a shadcn Dialog
     api-key-dialog.tsx           # zod + react-hook-form key entry dialog
+    create-dataset.tsx           # proposal form (lazy-loaded): schema/dep editors, CodeMirror
 ```
 
 ### Dependencies
 
-Runtime: React 19, TanStack Query (fetch state, caching, retries, central error handling), react-hook-form + zod + `@hookform/resolvers` (forms/validation), shadcn/ui's underlying packages (`radix-ui`, `lucide-react`, `sonner`, `class-variance-authority`, `clsx`, `tailwind-merge`).
+Runtime: React 19, TanStack Query (fetch state, caching, retries, central error handling), react-hook-form + zod + `@hookform/resolvers` (forms/validation), shadcn/ui's underlying packages (`radix-ui`, `lucide-react`, `sonner`, `class-variance-authority`, `clsx`, `tailwind-merge`), and `@uiw/react-codemirror` + `@codemirror/lang-python` (builder-script editor, loaded only by the create view).
 
 ### Tooling and quality gates
 
@@ -77,8 +79,9 @@ All API calls go through `lib/api.ts` and use the `/api/v1` prefix.
 - **Base URL**: `VITE_API_BASE_URL` (baked in at build time, set for Pages builds) or relative `/api` in dev, where Vite proxies to localhost:3000
 - `fetchDatasets()` -- `GET /api/v1/datasets`
 - `fetchData(name, version, start, end)` -- `GET /api/v1/data/{name}/{version}?start=...&end=...&build-data=false`
+- `proposeDataset(payload)` -- `POST /api/v1/datasets` (the only write; the backend opens a dataset-proposal PR, see "Dataset proposals endpoint" in `SPEC-backend.md`)
 
-The frontend is read-only and never triggers builds (`build-data=false` always). Both 200 and 206 responses are treated as valid (206 indicates partial/incomplete data). Failures throw `ApiError` (carries the HTTP status) so the query layer can react per-status.
+Browsing never triggers builds (`build-data=false` always). Both 200 and 206 responses are treated as valid (206 indicates partial/incomplete data). Failures throw `ApiError` carrying the HTTP status **and the backend's `detail` message** when present, so server-side validation errors render verbatim in forms. 401s from both queries and mutations funnel through one handler (`QueryCache` + `MutationCache` `onError`).
 
 ### API-key auth
 
@@ -128,6 +131,19 @@ The backend requires `Authorization: Bearer <dsk_...>` on everything except `/st
 - Highlighting is done by `json-view.tsx`, a recursive JSX renderer (colors keys, strings, numbers, booleans, null) — no HTML strings, no `dangerouslySetInnerHTML`
 - Closes on: Escape key, backdrop click, or close button (built into the Dialog)
 
-## Planned (not yet implemented)
+### Create dataset (proposal form)
 
-- **Create-dataset UI**: a form for name/version/calendar/granularity/start-date/schema/dependencies plus a builder-script editor (CodeMirror). Blocked on approval of the corresponding backend endpoint, which changes the builder trust model (script upload = code execution gated by API key instead of git review).
+`create-dataset.tsx` — reachable via the "new dataset" header button — submits a dataset proposal to `POST /datasets`, which opens a GitHub PR (creation is **never direct**; review remains the gate for code that runs on the server).
+
+Form sections:
+
+- **Proposer** (all required): name, Wat Street team, Discord user, and a description of what the dataset is for — surfaced verbatim in the PR body for reviewers
+- **Basics**: name, version, calendar + granularity selects (choices hardcoded as frontend constants; the server rejects unknowns), start date
+- **Schema**: dynamic key/type rows (`str`/`int`/`float`/`bool`), minimum one
+- **Dependencies**: picker populated from `GET /datasets`, optional lookback string (`5d`-style)
+- **Builder script**: CodeMirror editor (Python highlighting, dark theme) pre-filled with a `build(dependencies, timestamp)` template
+- **Extras**: optional `requirements.txt`; an env-vars checkbox reveals a `.env.template` field (the real `.env` never goes in the PR and must be placed on the server manually — the success screen and PR checklist both say so)
+
+Validation is two-layered: zod mirrors the server's format rules (name regex, semver, `YYYY-MM-DD`, lookback format) for instant feedback, and any server-side 400/409 (`ApiError.message` carries the backend `detail`, e.g. dependency granularity violations) renders in an inline error box. The server also lint-fixes the builder script with ruff before committing, so submitted scripts may differ cosmetically from what lands in the PR.
+
+On success the form is replaced by a panel linking the opened PR, with a "create another" reset.