Skip to content

feat: dataset proposal service and endpoint#384

Open
Scr4tch587 wants to merge 3 commits into
kai/github-clientfrom
kai/dataset-proposals
Open

feat: dataset proposal service and endpoint#384
Scr4tch587 wants to merge 3 commits into
kai/github-clientfrom
kai/dataset-proposals

Conversation

@Scr4tch587

Copy link
Copy Markdown
Collaborator

What changed

`POST /api/v1/datasets` — submit a dataset proposal; the server validates it and opens a GitHub PR adding `builders/scripts///` (config.toml, builder.py, optional requirements.txt / .env.template). Nothing is written to the server's own scripts dir; the dataset goes live only after review + merge + restart.

Validation before any PR opens (`core/service/proposals.py`):

  • generates the canonical `config.toml`, then re-parses those exact bytes and runs the same `validate_config` the server runs at startup, plus registry cross-checks (deps exist, granularity ordering, start-date ordering; a new dataset is a leaf, so cycles are impossible)
  • builder script: AST check for a top-level `build(dependencies, timestamp)`, then ruff autofix + format server-side (same rule set as repo CI) so proposal PRs land pre-linted; unfixable violations reject with ruff's output
  • proposer identity is required (name, Wat Street team, discord user, description) and surfaced in the PR body along with the API key's team label

PR shape: branch `add-dataset/-`, single commit, title `feat: add dataset /`, body with field table + review checklist (including the manual-`.env` step for `env-vars` datasets — secrets are never committed, only `.env.template`). Reviewers auto-requested from `GITHUB_REVIEWERS` (default `Blackgaurd,Scr4tch587`).

Error mapping: 400 invalid submission (message safe to show in the form) · 409 dataset exists or proposal already open · 502 GitHub unreachable/misconfigured.

Deps/config: `ruff` added as a server runtime dependency (for the lint step); `GITHUB_TOKEN` / `GITHUB_REPO` / `GITHUB_REVIEWERS` documented in `infra/.env.template`.

Why

Lead-approved design: dataset creation must go through code review, not direct server writes — uploading a builder script is uploading code that executes on the server. This endpoint automates everything up to the human gate.

Benefit

A submission that passes this endpoint cannot fail CI or break server startup: the committed config is byte-identical to what was validated, the script is pre-linted, and the CI gate (#380) re-verifies on the PR. E2E-tested against a fake GitHub server and the real API.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant