Add portable eval-package export and import for evaluators and datasets - #12
Merged
Conversation
Add the five export-modal class names the new export UI needs (export-format, export-layout, export-file, export-file-name, export-file-actions) to CONTRACT_CLASSES in styles.test.ts, under a new "Export modal." group. These are static literals, so they belong in CONTRACT_CLASSES, not INTERPOLATED_CLASSES. The contract-table assertion now fails because styles.css does not yet define these classes; the CSS lands in a later wave. The orphan, token, and favicon assertions continue to pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover the new export endpoints per the authoritative contract, ahead of implementation: - evaluators.exportFiles: code -> export.py (no split param); json -> export.json with split=false (bundled) / split=true (split) - datasets.exportFiles: code -> export.py; json -> export.json with version_id included when supplied and omitted entirely when absent - both unwrap to the inner `files` map and surface non-OK as rejections These fail today because the methods do not exist yet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Define the five export-modal classes the styles contract already promises: .export-format, .export-layout, .export-file, .export-file-name, and .export-file-actions, reusing existing tokens. The components that consume them arrive in a later wave; the contract test passes now that these are defined. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ExportFilesResponse, ExportFormat, and ExportLayout types and
exportFiles methods to the evaluators and datasets clients, unwrapping
the {files} envelope. JSON exports thread split (and version_id for
datasets); the code form sends no query params.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add edge cases asserting the "code" form ignores layout/versionId even when supplied, so neither `split` nor `version_id` can leak into the URL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover the capability registry refactor before implementation exists: - CapabilityEntry is a frozen dataclass of (module, class_name) - CAPABILITY_REGISTRY holds exactly the five harness capabilities, each entry's class_name equals its key, and every entry imports successfully - CodeMode resolves to the deep runtime module path (the reconciliation) - VALID_CAPABILITIES equals the registry key set; models re-exports it - CapabilitySpec validation is driven by the registry - consolidation guard: a registry addition is visible to export rendering and factory with no other edit, keeping the triplication from returning - SCALAR_TYPES[ft].__name__ yields str/int/float/bool and excludes enum These fail today (valcore.capabilities and models.SCALAR_TYPES do not exist). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collapse three parallel capability lists (models.VALID_CAPABILITIES, factory._CAPABILITY_MODULES, export._CAPABILITY_IMPORTS) and two FieldType maps (factory._SCALAR_TYPES, export._BASE_TYPES) into one home each, with no behavior change. - New src/valcore/capabilities.py holds CAPABILITY_REGISTRY (the five harness capabilities) and VALID_CAPABILITIES, importing nothing from valcore so models can depend on it without a cycle. CodeMode is reconciled to the deep runtime module path (pydantic_ai_harness.code_mode) that factory imports. - CapabilityEntry carries both the canonical runtime module and an optional render_module. A single module cannot be both the deep canonical path (which test_capabilities pins and factory imports at run time) and the shorter re-export path that render_script has always emitted; render_module keeps render_script byte-identical while the registry stays the single source. - models.py re-exports VALID_CAPABILITIES and gains SCALAR_TYPES beside FieldType; factory reads both; export derives type names from SCALAR_TYPES[ft].__name__ and reads the registry for capability imports. Existing test_export/test_factory/test_models pass unmodified; render_script output verified byte-identical against HEAD for all five capabilities. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… import order Add a regression test asserting all five capabilities render their original import line (test_export only pinned CodeMode), guarding the render_module fallback that keeps render_script byte-identical. Also apply ruff's import ordering fix to the delivered test file so `ruff check .` passes on a cold cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
parse.test.ts pins the new ParsedFile.kind discriminator and package disambiguation: CSV/JSONL keep their current parsing (kind "csv"/"jsonl"), a pydantic_evals dataset and a bundled valcore package both parse to kind "package" with columns/rows derived from cases[].inputs, and a malformed or unrecognized .json body falls through to the JSONL parser without throwing. DatasetUpload.test.tsx pins the UI: a detected package shows a detection line, previews its rows, hides the label-column control, and posts without a label_column field; a CSV keeps its preview table and label-column control. Queries by role, label, and text only. Tests fail until parse.ts/DatasetUpload.tsx implement the feature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite ExportModal tests for the widened subject prop and Code/JSON format picker: evaluator Code fetches the script, JSON fetches the file package (bundled/split), dataset Code goes through datasets.exportFiles, JSON bodies pretty-print with a raw fallback, per-file Copy/Download, the error banner, and the stale-response guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Disambiguate .json uploads into eval-package / pydantic_evals dataset vs JSONL by shape, add kind to ParsedFile, and hide the label-column control (and omit label_column) for detected packages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover the reverse output_schema map for every FieldType, field-order round trips, unsupported-construct rejection, capability translation both ways, row/case mapping, scalar-input wrapping, column inference with the 50-case limit, label-schema resolution across all branches, and version_to_spec reloadability. Tests fail because src/valcore/spec.py does not exist yet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen ExportModal to a discriminated subject prop serving both evaluators and datasets. Format radios (Code/JSON, Code default) with a JSON-only Bundled/Split layout Select. Evaluator Code keeps the legacy single-block script with footer Copy; every other combination renders per-file blocks with Copy and Download. JSON bodies are pretty-printed with a raw fallback, fetches refetch on format/layout change, and the cancelled-flag guard discards stale responses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…block Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add src/valcore/spec.py, the single place either foreign model (pydantic_ai AgentSpec, pydantic_evals Dataset) is constructed or read. Covers the lossless output_schema reverse map, capability translation both ways, valcore-block metadata, and row<->case mapping with column inference and label-schema resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend DatasetDetail's suite for the not-yet-built header Export button: the action renders alongside Edit/Delete, clicking it opens the real ExportModal (fetching this dataset's Code export for d1 in the default bundled layout with no version id), and closing unmounts the modal the way EvaluatorDetail gates its own. Fails until the button is wired up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an Export action to the dataset detail header, gated on local `exporting` state the way EvaluatorDetail gates its own modal. It opens ExportModal with a dataset subject (no version id — the page exports the dataset alone), defaulting to the pydantic_evals.Dataset code module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add failing tests for render_tool_sources, render_dataset_module, and render_judge_module in export.py: - render_tool_sources returns the import + verbatim source blocks for named tools, and empty blocks for an empty list. - render_dataset_module builds a pydantic_evals.Dataset (no valcore import), omits expected_output for unlabelled rows, and repr()s values for round-trip. - render_judge_module emits a ValcoreJudge whose evaluate reads the package with stdlib json, builds the agent via Agent.from_spec with an explicit output_type (guarded: output is a real model, not a StructuredDict), scores to the categorical label under TestModel, imports neither valcore nor yaml, and omits/includes tools= and custom_capability_types= per the version. - Pin render_script output determinism as a regression guard; existing render_script tests remain unmodified. Tests fail at import (functions not implemented yet) by design. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add render_tool_sources, render_dataset_module, and render_judge_module to export.py, reusing the existing _Imports/_render_tools and _render_output_model helpers so the code artifacts never drift. render_script output is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…trip Add tests/test_config_io.py pinning the eval-package JSON contract before config_io exists: bundled/split serialization, the four-way from_text format detection, native loadability of split halves, bundled-vs-bare-Dataset guard, round-trip reconstruction of version and dataset fields, and every documented rejection including ExceptionGroup unwrapping to a flat ContractError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add src/valcore/config_io.py with EvalPackage and ValcoreMeta. EvalPackage carries the optional agent (AgentSpec), dataset (pydantic_evals.Dataset), and valcore halves, and: - from_version / from_dataset build each half, routing every foreign-model mapping through spec.py - to_text emits a uniform name-to-content mapping, bundled (one file) or split (two), injecting the ValcoreJudge evaluator reference pointing at whichever file the agent lives in, or [] when there is no agent - from_text resolves the four formats (bundled envelope, bare dataset, bare agent + optional valcore block, else ContractError) and rejects unknown kind/version, non-object roots, unparseable JSON, and foreign evaluator names - merge recombines a split pair, rejecting two of the same half - to_version_fields / to_dataset_fields decode back to valcore field values JSON only; foreign models are dumped with model_dump(mode="json", by_alias=True) so output is indistinguishable from library-produced files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ValcoreJudge stub was built via type(), which skips the @DataClass decoration pydantic-evals requires for custom evaluators, so the with-evaluator load test errored. Decorate it directly and add coverage for merge conflicts on the dataset/valcore halves, a bare-string foreign evaluator, and the missing-section guards on to_version_fields / to_dataset_fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add failing TDD tests for the CLI 'formats' work: --format code|json, --dataset, and --split on export, plus a new import command. Guards the locked byte-identical code export and asserts a full export->import round trip, invalid-agent rejection, and .py-path refusal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mplementation
Extend the evaluator and dataset API suites with failing tests for the four new
export endpoints and package-aware upload:
- evaluators: GET .../export.json (bundled + split) yields a files map with the
agent JSON and valcore_judge.py; GET .../export.py yields the standalone
script; the legacy .../export keeps its {"source": ...} shape (regression).
- datasets: GET .../export.json is dataset-only without version_id and gains the
agent section plus companion module with it; split yields two JSON files;
GET .../export.py yields the dataset module.
- upload: an exported .json package round-trips into a matching dataset; a JSONL
body with a .json name still parses as JSONL; a label_column form field is
rejected; an explicit label_schema wins over the package's; a malformed .json
body is a client error, not a 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend `valcore export` with `--format json`, `--dataset`, and `--split`, and add a new `valcore import` command. Code export for a lone evaluator is unchanged and byte-identical to render_script. JSON export assembles an EvalPackage (merging evaluator and dataset when both are named), writes the companion valcore_judge.py beside a config file, and notes its omission on stderr when writing to stdout. Import validates the reconstructed version before any persistence so a bad agent creates nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… export Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add code and config export siblings beside the existing evaluator export,
add dataset export in both forms, and teach the upload endpoint to
disambiguate an eval-package .json body from a JSONL stream.
Evaluators gain GET .../export.py (standalone script) and .../export.json
(agent JSON + valcore_judge.py, bundled or --split). The legacy
.../export keeps its {"source": ...} shape untouched. Datasets gain
GET /{id}/export.py (pydantic_evals.Dataset module) and /{id}/export.json,
which merges in a version when version_id is given and only then rides a
companion module.
Upload resolves a .json body by whole-body json.loads: a recognized
EvalPackage imports its dataset half (rejecting label_column, honoring an
explicit label_schema over the package's); anything else falls through to
the JSONL parser unchanged. Route modules go through config_io and export
and never touch pydantic_evals directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…yte-identity Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add full type hints to _export_code / _export_json (Dataset, DatasetRow, Evaluator, EvaluatorVersion), matching every other signature in main.py. - Thread the Evaluator through to _export_json so a stdout stem slugifies the evaluator name, per spec, rather than the version name. - Drop the redundant update_evaluator call in import_; create_version already sets the new version as active. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a section covering the evaluator/dataset code-vs-JSON matrix, the eval-package format, the export/import commands, how a consumer runs a package via custom_evaluator_types=[ValcoreJudge], and the three honest limitations of the bare foreign formats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
valcore could previously export an evaluator only as a standalone Python
script and could not export datasets at all. This PR gives both evaluators and
datasets a JSON config form and a Python code form, driven from a single
canonical translation layer, and adds import for the JSON form. The existing
script export stays byte-identical and no new dependencies are added.
Changes
capabilities.py, a shared capability registry and scalar-type map;factory.pynow reads it instead of holding its own copy.spec.py, the canonical translation betweenEvaluatorVersionandpydantic-ai
AgentSpec(includingoutput_fields<->output_schema).config_io.pyto assemble, serialize, detect, and round-tripeval-package JSON (bundled and split forms).
export.pywith JSON export, dataset/judge/tool source renderers, andsplit-package output, alongside the retained byte-identical script renderer.
cli/main.pywith JSON and dataset-code export, animportcommand,and a no-clobber guard on named output.
api/routes/evaluators.pyandapi/routes/datasets.py, plus eval-package upload;models.pygainssupporting fields.
exportFilesclient methods and types, a two-level format picker inExportModal, eval-package detection/preview inDatasetUpload, a Datasetheader Export action, and the accompanying CSS style-contract entries.
README.md.Test plan
valcore export <evaluator>with no new flags emits the samePython script byte-for-byte as before.
--split, verifying two files are produced in the split case.reconstructed; confirm importing a
.pyfile raisesContractErrorandthat a non-
ValcoreJudgeevaluator entry also raisesContractError.uv run pytest -q, and lint withuv run ruff check .anduv run ruff format ..cd web && npm ci && npx vitest run, covering theexport modal, dataset upload preview, and client methods.
cd web && npx tsc --noEmit.Select, and thatswitching JSON -> Code restores the script block.
🤖 Generated by workbench pr_writer