[BENCH-839] Drive the LLM roster from the models table - #607
Merged
Conversation
coval-cale
force-pushed
the
cale/bench-827-llm-proxy-per-provider
branch
from
September 8, 2026 23:32
b99710c to
1ea8a7e
Compare
sync-llm, the LLM fetch specs, and the proxy routes now read LLM rows from benchmarks_v2.models instead of a code registry. collected is the switch: sync reconciles the Coval schedule's enabled flag from it, the fetch skips uncollected rows, and the proxy refuses their turns. Client construction stays in code as a per-provider factory map, and startup logs any collected LLM model that has no client.
coval-cale
force-pushed
the
cale/bench-827-llm-proxy-per-provider
branch
from
September 8, 2026 23:36
1ea8a7e to
4e01863
Compare
ben-coval
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #605, which made the LLM proxy, Coval agent, sync, and fetch provider-agnostic but left the provider list as a code constant. This replaces that constant with the LLM rows in
benchmarks_v2.models, so thecollectedflag STT and TTS already use becomes the on/off switch for an LLM provider.What changes
enabledflag fromcollected. It skips the fetch for uncollected rows./llm/{provider}against the roster on each request. No row orcollectedfalse is a 404. Row present but no client is a 503.llm/benchmark.py. Startup logsllm_client_missingfor any collected LLM row without a factory, which addresses the registry/client drift finding from Proxy and sync every LLM provider through one definition #605.from_settingsfactory. Its HTTP behavior is unchanged.What it enables
collectedto false on its row through the admin models endpoint. The next llm-sync (5:00 AM Pacific) disables the Coval schedule and stops fetching, and the proxy refuses its turns. Flipping back re-enables.coval_llm_<slug>_agent_idsetting, and credentials.Not changed
Database schema, config, the persona defaults from BENCH-825, and Phonely's live Coval agent. Deploys as a no-op for Phonely.
Tests pin the schedule disable/unchanged/enable transitions, the CLI skipping the fetch for a paused row, the proxy 404 for an uncollected row, and roster-built fetch specs.
Linear: BENCH-839