Derive the model catalog from pydantic-ai and seed new evaluator versions - #16
Merged
Merged
Conversation
Two changes to evaluator version authoring. Open the model catalog. MODEL_CATALOG was five hand-maintained strings while the Gateway serves far more, so it is now derived: model_catalog() filters pydantic-ai's known_model_names() through the existing GATEWAY_ROUTES, yielding 160 models that track whatever version is pinned. Filtering through GATEWAY_ROUTES rather than a bare "gateway/" prefix means every suggestion is a string validate_model_string accepts, so the two cannot drift. The catalog is type-ahead only, not a whitelist. The model field becomes an input + datalist instead of a select, so any well-formed gateway/<route>:<name> is enterable and the Gateway stays the authority on what exists. validate_model_string was already shape-only and is unchanged. /api/config now also returns default_model. The SPA defaulted new versions to models[0], which was claude-sonnet-5 by luck of the short list; against a sorted 160-entry catalog it would silently become an arbitrary Bedrock model. pydantic_ai.models is imported inside model_catalog() rather than at module scope: it costs ~235ms, every CLI invocation imports valcore.settings, and only the API config endpoint needs the catalog. Seed new versions from the one on screen. "New version" opened a blank editor, losing all the prior text. It now prefills from the version being viewed, mirroring the server's copy_version, including carrying version_name over verbatim. It remains a create, not an edit-in-place. Incidentally, the frozen model field is now readOnly rather than disabled, matching its siblings; select has no readOnly attribute, which is the only reason it differed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related changes to evaluator version authoring.
Open the model catalog
MODEL_CATALOGwas five hand-maintained strings while the Pydantic AI Gateway serves far more. It is now derived:model_catalog()filters pydantic-ai's publicknown_model_names()through the existingGATEWAY_ROUTES, yielding 160 models that track whatever version is pinned.gateway/openaigateway/bedrockgateway/anthropicgateway/google-cloudgateway/googlegateway/groqFiltering through
GATEWAY_ROUTESrather than a baregateway/prefix means every suggestion is a stringvalidate_model_stringaccepts, so the two cannot drift.The catalog is type-ahead only, not a whitelist. The model field is now an
<input>+<datalist>instead of a<select>, so any well-formedgateway/<route>:<name>is enterable and the Gateway stays the authority on what actually exists.validate_model_stringwas already shape-only and is unchanged. This matters because the library's list reflects what the pinned pydantic-ai knows, not what a given account is entitled to — it can be both too wide and too narrow.Seed new versions from the one on screen
"New version" opened a blank editor, discarding all the prior text. It now prefills from the version being viewed, mirroring the server's existing
copy_version(the frozen "save as new version" path), including carryingversion_nameover verbatim — the server does not constrain names to be unique. It remains a create, not an edit-in-place.Notes for review
/api/configgaineddefault_model. The SPA defaulted new versions tomodels[0], which wasclaude-sonnet-5only by luck of the short list. Against a sorted 160-entry catalog that silently becomes an arbitrary Bedrock model. A test pinsdefault_model != models[0]so this cannot regress.pydantic_ai.modelsis imported insidemodel_catalog(), not at module scope. It costs ~235ms to import, every CLI invocation loadsvalcore.settings, and only the API config endpoint needs the catalog. Module-level it tookimport valcore.settingsfrom 136ms to 373ms.readOnlyrather thandisabled, matching its siblings in the same section.<select>has noreadOnlyattribute, which is the only reason it differed; as an input a frozen version's model string stays selectable for copying.Testing
951 Python tests pass (+7), 595 web tests pass (+5),
tscclean.New coverage: catalog derivation and route completeness; every catalog entry passes
validate_model_string; a well-formed name absent from the catalog is accepted while malformed ones are still rejected;default_modelis distinct frommodels[0]; the editor prefills from a seed; the seed is a deep copy so editing the draft cannot mutate the source version; and a blank fallback when an evaluator has no versions yet.Note this repo has no ESLint config, so no JS linting ran —
ruffandpytest(the pre-commit hooks) are clean.🤖 Generated with Claude Code