Store dataset generation settings and reuse them

[duncankmckinnon]

Follow-up to #9.

Why

Generation steering was used for one call and then discarded. Dataset persists only name, description, columns, and label_schema — so nothing could show how a dataset's rows came to be, and repeating an ask meant reconstructing the instructions, column notes, and label mix by hand.

Where the settings live

A separate DatasetGeneration table, not columns on Dataset. That's load-bearing: init_db is a bare create_all, which creates missing tables but never missing columns. New fields on Dataset would leave every existing SQLite database raising no such column on any dataset query; a new table is created on next start and existing data is untouched. test_init_db_adds_the_generation_table_to_an_existing_database drops the table and re-runs init_db to hold that property.

It also keeps provenance out of the dataset's own metadata — an uploaded dataset simply has no row, which is why the read returns null rather than an empty object.

Both generate routes record what they were asked for. set_generation replaces rather than accumulates: the settings describe the most recent ask, which is what a form repopulates from. source_version_id is provenance only — nothing reads through it to derive shape, since the version may since have changed or been deleted.

Three consumers

Read the settings. GET /api/datasets/{id}/generation returns them, or null for a dataset that was never generated. A collapsed panel on the dataset detail page renders them — the mix converted back to percentages, and the source version as an id rather than a link that could 404.

Top up an existing dataset. POST /api/datasets/{id}/generate-rows appends rows and falls back to the stored settings, so repeating an ask needs only {"count": 10}. Anything you do pass overrides the stored value and becomes the new stored ask, so the next top-up repeats what actually ran.

Shape is deliberately not overridable — the dataset's own columns and label space always apply, which is what keeps new rows compatible with the existing rows and with any evaluator already running against them. A dataset with an empty label schema gets no suggested labels and rejects label_guidance, matching the existing rule on /generate-from-version. Deleting a dataset drops its generation record, which would otherwise be unreachable.

The modal prefills from the stored settings, and prunes any note whose column has since been removed by an edit — sending it would fail the server's unknown-column check.

Duplicate a dataset. An action on the datasets list opens the generate form seeded from an existing dataset's shape and settings, so you can change one thing and produce a fresh dataset.

Shared plumbing

labelMix.fromProportions inverts toProportions so a stored mix repopulates the percent editor; both prefill paths use it.

DatasetGenerateForm takes an optional initial, read once at mount, with callers remounting via key to change it. That keeps the fields plain state rather than props that would have to be synced back on every keystroke.

Verification

• Python: 583 passed, 1 skipped (was 563); ruff check and ruff format --check clean
• Web: 247 tests passed (was 219); tsc --noEmit and production build clean

New coverage: the store round-trip and replace-not-accumulate behaviour, cascade on delete, the create_all migration property, settings stored by both generate routes, null for an uploaded dataset, fallback and override precedence on top-up, index continuation, shape being fixed by the dataset, and the prefill in both forms.

Note that web/dist is gitignored, so reviewing the UI from a checkout needs cd web && npm run build first.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!