feat(ci): ask the vendor, daily, whether our model ids still exist - #43
Merged
Conversation
AOZ reported "KI-Assistent nicht konfiguriert. Bitte GROQ_API_KEY setzen" on a
deployment whose key was valid. Groq had retired the whole llama-3.x family and
the pinned id answered 404. Production had been failing exactly as long as the
demo and nobody knew, because "not configured" is the only thing the app can
say.
Six pins across five repos were dead the same morning — botsmann, evig, kivvi,
orangecat, truthseeker — three of them serving live traffic. None of it was
visible from inside a repo: every gate was green, because the code is correct
and the vendor changed underneath it. No lint rule, type check or unit test can
see this, which is exactly the gap the central audits exist to cover.
So this asks the only authority that knows. One GET /models per vendor, ZERO
tokens, which is what makes it schedulable — and "somebody is supposed to
remember" is the mechanism that already failed for eight days in ai-ration's
own consumer.
It calls ai-ration's checkCatalog rather than growing a second vendor query
here; the package takes the chain as an argument, so it generalised to
arbitrary ids with no change. That makes dotfiles its second adopter, which is
the medicine this file prescribes to everyone else.
Three findings survive as tests because the first live runs produced them:
- xAI's `grok-3-mini` was filed under Groq — one letter apart, and xAI's URL
sat 6 lines from the pin while Groq's sat 18. Attribution now follows the
marker ABOVE the pin, since that is the branch that owns it, and vendors we
cannot query still own their ids so they report UNCHECKED instead of being
absorbed by the nearest vendor we can.
- fleetcrown's computed `${link.provider.id}/${link.model}` was reported
retired. An interpolated id is the absence of a pin.
- botsmann's pin lives in lib/constants.ts, which no AI-shaped filename
filter opens. Candidates are now two-tiered and ranked, and truncation is
REPORTED — a bounded sweep that stays quiet about what it skipped reads
exactly like one that found nothing.
An unreadable catalogue reports UNCHECKED, never GONE. Treating "I could not
look" as "nothing is there" would mark every pin retired and invent a
fleet-wide outage somebody then acts on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap this closes
AOZ reported "KI-Assistent nicht konfiguriert. Bitte GROQ_API_KEY setzen" on a deployment whose key was valid. Groq had retired the whole llama-3.x family; the pinned id answered
404 model_not_found. Production had been failing exactly as long as the demo and nobody knew, because "not configured" is the only thing the app can say.It was never an AOZ bug. Six pins across five repos were dead the same morning, three of them serving live traffic:
lib/constants.ts:54llama-3.1-8b-instantapps/web/lib/ai/call-provider.ts:77llama-3.1-8b-instantsrc/lib/ai/providers.ts:23llama-3.3-70b-versatilesrc/services/cat/platform-llm.ts:21llama-3.3-70b-versatilesrc/lib/llm.ts:5llama-3.3-70b-versatile.env.example:116llama3.2No gate in any of those repos could have caught it. Lint, types and tests were all green — the code is correct and the vendor changed underneath it. That is precisely the gap
verify-floor-audit.shandui-defect-audit.mjsalready exist to cover, so this follows the same doctrine: one central script, remote default branches, self-tested, reporting into a job summary.Why daily, and why it's free
One
GET /modelsper vendor. Zero tokens. That is what makes it schedulable — and "somebody is supposed to remember" is the mechanism that already failed for eight days inai-ration's own consumer. A retirement is decided outside this fleet and lands without warning, so unlike the weekly UI sweep it needs a daily cadence.It reuses
ai-rationrather than growing a second vendor querycheckCatalogalready owns this and already carries the scars. It takes the chain as an argument, so it generalised to arbitrary discovered ids with no change to the package. That makes dotfiles its second adopter — the medicine SHARED.md prescribes to everyone else.Three findings became tests, because the first live runs produced them
grok-3-miniwas filed under Groq. One letter apart; xAI's URL sat 6 lines from the pin, Groq's 18. Attribution now follows the marker above the pin — the branch that actually owns it — and vendors we cannot query still own their ids, so they reportUNCHECKEDinstead of being absorbed by the nearest vendor we can.${link.provider.id}/${link.model}was reported retired. An interpolated id is the absence of a pin.lib/constants.ts, which no AI-shaped filename filter opens. Candidates are now two-tiered and ranked, and truncation is reported — a bounded sweep that stays quiet about what it skipped reads exactly like one that found nothing.An unreadable catalogue reports
UNCHECKED, neverGONE. Treating "I could not look" as "nothing is there" would mark every pin retired and invent a fleet-wide outage somebody then acts on. That guard is pinned by a test.Verification
29 self-test assertions, no network/key/checkout, wired into
ci.ymland into the workflow's ownpull_requesttrigger. Live run across 34 repos on their default branches, full coverage with no truncation, findings above reproduced exactly.Defaults to
--warn-onlyon the schedule: this reports six real findings today, and a check that goes red on day one gets muted by day three.