Skip to content

Registry maintenance: weekly models.dev diff report and monthly reachability probe #170

Description

@cunninghamcard-bit

Part of #166 (track A follow-up; RFC-0033 §4.1 items 4–5).

provider_registry.json is the hand-maintained source of truth (251 rows) and must stay that way: it carries manual fixes (94b5000, b24e59e) that any regeneration would clobber. What is missing is a way to notice when upstream reality moves. A one-off comparison against models.dev on 2026-09-04: 97 rows exist on both sides, 27 of those disagree on base_url, and 116 models.dev providers are absent from aimux.

Reference: pi-ai regenerates model data from models.dev at build time and fails the build when the data is stale (scripts/generate-models.ts, scripts/check-model-data.ts). aimux cannot regenerate, so the equivalent is a diff report plus a reachability probe.

Deliverables

  1. scripts/sync_registry.py --report — fetch https://models.dev/api.json (213 providers, 187 with base URL, all with env names); optionally read a local reference/litellm checkout (absorbs extract_litellm_bases.py / scan_litellm_urls.py, which are then deleted). Output three tables: providers missing from aimux, base_url disagreements, env_var disagreements. Rows with a "note" field are treated as intentional deviations and not reported.
  2. scripts/probe_registry.py — for each row, unauthenticated GET {base_url}/models; 401/403 counts as alive; DNS failure or 5xx is reported. Never edits the JSON.
  3. A scheduled GitHub Action: weekly sync report, monthly probe, posting to a fixed tracking issue (or opening a PR with the proposed row edits). Non-blocking.
  4. Add the optional note and status ("unreachable") fields to the row schema and to the Rust validation tests in provider.rs.

Acceptance

  • Running the report locally reproduces the 27 / 116 numbers above (until the triage issue lands).
  • CI on pull requests is unchanged; the schedule never blocks merges.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions