Provider data pipeline: registry tiers, generated model catalogue, maintenance jobs, RFC-0033 - #176
Closed
cunninghamcard-bit wants to merge 13 commits into
Closed
cunninghamcard-bit wants to merge 13 commits into
cunninghamcard-bit wants to merge 13 commits into
Conversation
Planning document for the code reduction effort tracked in arcships#166 and arcships#167: measured baseline, tracks A-E, design-level changes (flat cross-language error model after OpenDAL, executor pattern for single-modality providers after LiteLLM, transport-level replay), defect list, docs reorganization, dependency order and ledger.
RFC-0033 L1: the registry is a hand-maintained identity table, but nothing in it said how much a row could be trusted. Each of the 251 rows now carries a `tier` — verified (a cassette replays it), listed (an upstream catalogue knows the name), unverified (a lead from vendor docs) — plus optional `note` (why this row deviates from upstream; exempts it from the sync report) and `status` (only "unreachable"). Initial split: 19 verified / 124 listed / 108 unverified. `listed` matches models.dev and anya2a ids through catalogue.rs's normalize_provider_name, since both upstreams use hyphens where the registry uses underscores. `tier` defaults to Unverified on deserialize so RFC-0020 external entries keep loading; registry_entries_are_valid re-reads the raw JSON to insist every built-in row states its tier explicitly.
The generated provider table now has a Tier column with verified rows first and a legend defining the three tiers. `--check` mirrors the gen_provider_names.py UX so CI can catch a stale docs/api/providers.md instead of trusting people to rerun the generator.
RFC-0033 L2: model knowledge (context windows, pricing, modalities) is upstream data, not something to hand-maintain. scripts/gen_models.py normalises models.dev (primary) and anya2a (fallback for providers models.dev lacks) into one flat per-model schema and writes one file per registry provider: 137 providers / 4871 models, 2.7M on disk. Hand corrections go in data/models.overrides.json (partial field patches or "_drop": true), applied after normalisation, so a fix survives the next refresh. data/models/.manifest.json records counts and a sha256 over the provider files; `--check` recomputes it offline in ~0.15s and also rejects a file missing its `_generated` header or naming a provider the registry does not have.
…d-hoc scanners
Two maintenance jobs the registry needs on a cadence, neither of which may
edit the JSON — a human decides what to take from upstream.
sync_registry.py --report diffs the registry against models.dev, anya2a and
(with --litellm) litellm's api_base constants, printing three Markdown
tables: ids missing from the registry, base_url disagreements, env_var
disagreements. A row with a `note` is a deliberate deviation and is exempt
from the last two. On the snapshots: 150 missing rows / 84 base_url / 19
env_var disagreements.
probe_registry.py does an unauthenticated GET {base_url}/models over a thread
pool and reports only what fails: 5xx as server-error, DNS/connect/timeout as
dead, anything that answers (401/403/404 included) as alive, templated URLs
skipped.
extract_litellm_bases.py and scan_litellm_urls.py are deleted — sync_registry
does the same api_base scan behind --litellm, with the result cross-checked
against the registry instead of dumped as raw JSON.
The generator rule (do-not-edit header + a CI --check) now covers all three provider generators: gen_providers_doc.py --check and gen_models.py --check join gen_provider_names.py --check in the contract-tests job and in scripts/local-ci.sh, so a stale providers.md or model manifest fails the PR rather than rotting. registry-maintenance.yml runs the jobs that need a clock rather than a PR: weekly upstream diff and monthly reachability probe comment on arcships#170, weekly gen_models.py opens or updates bot/refresh-model-data. It never writes to provider_registry.json.
docs/contributing/adding-a-provider.md states the layer table (L0 protocol / L1 identity / L2 model data / L3 derived artifacts, each with its owner and its gate), the generator rule, the tier definitions, and one checklist per case: OpenAI-compatible vendor, new wire protocol, single-modality vendor (RFC-0033 §8.2 executor pattern rather than another hand-written polling loop). Linked from README's docs table; reference.md's Providers section now names gen_providers_doc.py --check and gen_models.py.
models.dev answers 403 to urllib's default User-Agent, so the live (non --offline) path of gen_models.py and sync_registry.py failed outright — the weekly Action would never have worked. Model data refreshed from the live fetch that verified it (llmgateway gained a model since the snapshot).
…line; exclude data/ from the crate
Contributor
Author
|
Closing: opened prematurely, the author wants to review this branch locally first. Branch kept. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One branch for the whole provider data pipeline so it can be reviewed as a system. Companion to #168/#169 (which delete the one-shot generators); tracks #170–#173 and #166.
The model (RFC-0033 §4.1, included in this PR)
aimux-providers/src/<protocol>/provider_registry.json, one row per provider, now withtier/note/statusaimux-providers/data/models/<provider>.json+.manifest.json, from models.dev (primary) and anya2a (secondary), merged with hand-writtenmodels.overrides.jsonscripts/gen_models.py; weekly refresh PRgen_models.py --check(hash, no network)ProviderName×8,docs/api/providers.md, ts-rs types--checkpi-ai is the reference (
packages/ai:src/api,src/providers/<id>.ts,scripts/generate-models.ts+check-model-data.ts). Two deliberate differences: L1 is data rather than TypeScript factories (251 providers vs 39), and L2 is committed and refreshed by a scheduled PR rather than hydrated at build time, so the crate and the seven binding CIs build offline.Tiers — making "251 providers" honest
verifiedtests/cassettes/<name>/listedunverifiedstatus: "unreachable"is reserved for the monthly probe; nothing is deleted by machines. Rows with anoteare treated as intentional deviations by the diff report.What is in the diff
tier;RegistryEntrygainstier/note/statuswith serde defaults so RFC-0020 external entries keep loading;registry_entries_are_validinsists built-in rows state their tier.docs/api/providers.mdshows a Tier column and legend.scripts/gen_models.py(fetch /--offline/--check), 137 provider files, 4,872 models, 2.7 MB,.manifest.jsonwith sources, counts and sha256.data/is excluded from the published crate until a Rust consumer reads it (runtime wiring into RFC-0027's catalogue is a follow-up).scripts/sync_registry.py --report(absorbsextract_litellm_bases.pyandscan_litellm_urls.py, which are removed) andscripts/probe_registry.py;.github/workflows/registry-maintenance.ymlruns the weekly diff → comment on Registry maintenance: weekly models.dev diff report and monthly reachability probe #170, the weekly model-data refresh → PR onbot/refresh-model-data, the monthly probe → comment on Registry maintenance: weekly models.dev diff report and monthly reachability probe #170. models.dev rejects urllib's default User-Agent, so both fetchers send one.gen_providers_doc.py --checkandgen_models.py --checkadded to CI next togen_provider_names.py --check, and toscripts/local-ci.sh.docs/contributing/adding-a-provider.md(four layers, three checklists, the generator rule), linked from README; RFC-0033 (code convergence plan) added with §4.1 rewritten to this design.Verification
cargo test -p aimux-providers --lib provider::23 passed;--test conformance_test53 passed; clippy and fmt clean; all three--checkgenerators green;sync_registry.py --reportproduces identical output from the snapshot and live (150 missing / 84 base_url / 19 env_var disagreements — the backlog for #171);probe_registry.pysmoke-tested on five rows.Line count is dominated by the generated
data/models/*.json(≈122k of the 122.6k insertions).