Skip to content

[BENCH-839] Drive the LLM roster from the models table - #607

Merged
coval-cale merged 1 commit into
mainfrom
cale/bench-827-llm-proxy-per-provider
Sep 8, 2026
Merged

[BENCH-839] Drive the LLM roster from the models table#607
coval-cale merged 1 commit into
mainfrom
cale/bench-827-llm-proxy-per-provider

Conversation

@coval-cale

@coval-cale coval-cale commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Follows #605, which made the LLM proxy, Coval agent, sync, and fetch provider-agnostic but left the provider list as a code constant. This replaces that constant with the LLM rows in benchmarks_v2.models, so the collected flag STT and TTS already use becomes the on/off switch for an LLM provider.

What changes

  • sync-llm reads LLM rows from the DB, reconciles one Coval agent, run template, and schedule per row, and sets the schedule's enabled flag from collected. It skips the fetch for uncollected rows.
  • LLM fetch builds its specs from collected rows only, and requires the dental test set up front.
  • Proxy resolves /llm/{provider} against the roster on each request. No row or collected false is a 404. Row present but no client is a 503.
  • Clients stay in code as a per-provider factory map in llm/benchmark.py. Startup logs llm_client_missing for any collected LLM row without a factory, which addresses the registry/client drift finding from Proxy and sync every LLM provider through one definition #605.
  • Phonely's client gains a from_settings factory. Its HTTP behavior is unchanged.

What it enables

  • Pausing Phonely is flipping collected to false on its row through the admin models endpoint. The next llm-sync (5:00 AM Pacific) disables the Coval schedule and stops fetching, and the proxy refuses its turns. Flipping back re-enables.
  • A second LLM is a models row, a client class, one factory-map line, a coval_llm_<slug>_agent_id setting, and credentials.

Not changed

Database schema, config, the persona defaults from BENCH-825, and Phonely's live Coval agent. Deploys as a no-op for Phonely.

Tests pin the schedule disable/unchanged/enable transitions, the CLI skipping the fetch for a paused row, the proxy 404 for an uncollected row, and roster-built fetch specs.

Linear: BENCH-839

@coval-cale
coval-cale requested a review from a team as a code owner September 8, 2026 23:30
@coval-cale
coval-cale force-pushed the cale/bench-827-llm-proxy-per-provider branch from b99710c to 1ea8a7e Compare September 8, 2026 23:32
sync-llm, the LLM fetch specs, and the proxy routes now read LLM rows
from benchmarks_v2.models instead of a code registry. collected is the
switch: sync reconciles the Coval schedule's enabled flag from it, the
fetch skips uncollected rows, and the proxy refuses their turns. Client
construction stays in code as a per-provider factory map, and startup
logs any collected LLM model that has no client.
@coval-cale
coval-cale force-pushed the cale/bench-827-llm-proxy-per-provider branch from 1ea8a7e to 4e01863 Compare September 8, 2026 23:36
@coval-cale coval-cale changed the title Put the LLM info in the db [BENCH-839] Drive the LLM roster from the models table Sep 8, 2026
@coval-cale
coval-cale added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit e6f1796 Sep 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants