Skip to content

feat(ci): ask the vendor, daily, whether our model ids still exist - #43

Merged
github-actions[bot] merged 1 commit into
masterfrom
worktree-model-pin-audit
Aug 26, 2026
Merged

feat(ci): ask the vendor, daily, whether our model ids still exist#43
github-actions[bot] merged 1 commit into
masterfrom
worktree-model-pin-audit

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

The gap this closes

AOZ reported "KI-Assistent nicht konfiguriert. Bitte GROQ_API_KEY setzen" on a deployment whose key was valid. Groq had retired the whole llama-3.x family; the pinned id answered 404 model_not_found. Production had been failing exactly as long as the demo and nobody knew, because "not configured" is the only thing the app can say.

It was never an AOZ bug. Six pins across five repos were dead the same morning, three of them serving live traffic:

Repo File Pin
botsmann lib/constants.ts:54 llama-3.1-8b-instant
kivvi apps/web/lib/ai/call-provider.ts:77 llama-3.1-8b-instant
evig src/lib/ai/providers.ts:23 llama-3.3-70b-versatile
orangecat src/services/cat/platform-llm.ts:21 llama-3.3-70b-versatile
truthseeker src/lib/llm.ts:5 llama-3.3-70b-versatile
evig .env.example:116 llama3.2

No gate in any of those repos could have caught it. Lint, types and tests were all green — the code is correct and the vendor changed underneath it. That is precisely the gap verify-floor-audit.sh and ui-defect-audit.mjs already exist to cover, so this follows the same doctrine: one central script, remote default branches, self-tested, reporting into a job summary.

Why daily, and why it's free

One GET /models per vendor. Zero tokens. That is what makes it schedulable — and "somebody is supposed to remember" is the mechanism that already failed for eight days in ai-ration's own consumer. A retirement is decided outside this fleet and lands without warning, so unlike the weekly UI sweep it needs a daily cadence.

It reuses ai-ration rather than growing a second vendor query

checkCatalog already owns this and already carries the scars. It takes the chain as an argument, so it generalised to arbitrary discovered ids with no change to the package. That makes dotfiles its second adopter — the medicine SHARED.md prescribes to everyone else.

Three findings became tests, because the first live runs produced them

  • xAI's grok-3-mini was filed under Groq. One letter apart; xAI's URL sat 6 lines from the pin, Groq's 18. Attribution now follows the marker above the pin — the branch that actually owns it — and vendors we cannot query still own their ids, so they report UNCHECKED instead of being absorbed by the nearest vendor we can.
  • fleetcrown's computed ${link.provider.id}/${link.model} was reported retired. An interpolated id is the absence of a pin.
  • botsmann's pin lives in lib/constants.ts, which no AI-shaped filename filter opens. Candidates are now two-tiered and ranked, and truncation is reported — a bounded sweep that stays quiet about what it skipped reads exactly like one that found nothing.

An unreadable catalogue reports UNCHECKED, never GONE. Treating "I could not look" as "nothing is there" would mark every pin retired and invent a fleet-wide outage somebody then acts on. That guard is pinned by a test.

Verification

29 self-test assertions, no network/key/checkout, wired into ci.yml and into the workflow's own pull_request trigger. Live run across 34 repos on their default branches, full coverage with no truncation, findings above reproduced exactly.

Defaults to --warn-only on the schedule: this reports six real findings today, and a check that goes red on day one gets muted by day three.

AOZ reported "KI-Assistent nicht konfiguriert. Bitte GROQ_API_KEY setzen" on a
deployment whose key was valid. Groq had retired the whole llama-3.x family and
the pinned id answered 404. Production had been failing exactly as long as the
demo and nobody knew, because "not configured" is the only thing the app can
say.

Six pins across five repos were dead the same morning — botsmann, evig, kivvi,
orangecat, truthseeker — three of them serving live traffic. None of it was
visible from inside a repo: every gate was green, because the code is correct
and the vendor changed underneath it. No lint rule, type check or unit test can
see this, which is exactly the gap the central audits exist to cover.

So this asks the only authority that knows. One GET /models per vendor, ZERO
tokens, which is what makes it schedulable — and "somebody is supposed to
remember" is the mechanism that already failed for eight days in ai-ration's
own consumer.

It calls ai-ration's checkCatalog rather than growing a second vendor query
here; the package takes the chain as an argument, so it generalised to
arbitrary ids with no change. That makes dotfiles its second adopter, which is
the medicine this file prescribes to everyone else.

Three findings survive as tests because the first live runs produced them:

  - xAI's `grok-3-mini` was filed under Groq — one letter apart, and xAI's URL
    sat 6 lines from the pin while Groq's sat 18. Attribution now follows the
    marker ABOVE the pin, since that is the branch that owns it, and vendors we
    cannot query still own their ids so they report UNCHECKED instead of being
    absorbed by the nearest vendor we can.
  - fleetcrown's computed `${link.provider.id}/${link.model}` was reported
    retired. An interpolated id is the absence of a pin.
  - botsmann's pin lives in lib/constants.ts, which no AI-shaped filename
    filter opens. Candidates are now two-tiered and ranked, and truncation is
    REPORTED — a bounded sweep that stays quiet about what it skipped reads
    exactly like one that found nothing.

An unreadable catalogue reports UNCHECKED, never GONE. Treating "I could not
look" as "nothing is there" would mark every pin retired and invent a
fleet-wide outage somebody then acts on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit 01aec5e into master Aug 26, 2026
2 checks passed
@github-actions
github-actions Bot deleted the worktree-model-pin-audit branch August 26, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant