fix: four of the nine default pins were already retired - #4
Merged
Conversation
Checked `freeChain()` against the live catalogues on 2026-08-25. Four of its
nine ids no longer exist:
groq llama-3.3-70b-versatile GONE
groq llama-3.1-8b-instant GONE <- whole vendor dead
openrouter openai/gpt-oss-20b:free GONE <- was FIRST in the list
openrouter nvidia/nemotron-3-nano-30b... GONE
The chain exists because a single pinned free model is a scheduled outage. That
reasoning has a hole this package fell into: the chain is ITSELF a list of pins,
so it rots too — and a chain whose first vendor is entirely dead is a slower
version of the failure it was built to prevent. Every caller paid two Groq 404s
and then a third on the preferred OpenRouter fallback before reaching a model
that could answer.
Consumers inherited it. FleetCrown used the same Groq id for direct, unchained
calls and was silently broken for eight days: its nightly frontier digest fell
back to a canned headline, and activity digests, calendar extraction, proposal
extraction and hosted-runner analysis all failed one at a time, quietly enough
that nothing looked broken.
Replacements probed, not guessed — the package's own rule. `openai/gpt-oss-120b`
and `openai/gpt-oss-20b` each returned a correct native tool_call with correct
arguments. The two retired OpenRouter ids are removed; the other five were
confirmed present in the 419-model catalogue.
── catalog: the check that can actually run on a timer ──────────────────────
New module, exported as `checkCatalog` / `hasRot` / `deadProviders` /
`catalogReport`. One GET /models per provider and ZERO tokens, which is the
entire point: a tool-call probe costs real money and therefore can never be
scheduled, so the only existing defence was a command someone had to remember
to type. Existence is cheap enough to check nightly. `npm run check:catalog`
runs it against this package's own defaults and exits 1 on confirmed rot.
It lives here rather than in each app because the check is the same everywhere,
and the first app to need it had already written its own, slightly differently.
Three states, not two. `live: null` means the catalogue could not be READ — no
key, network failure, non-200, or a body that parses but lists nothing. That is
not an empty catalogue. Reporting it as rot invents an outage; reporting it as
a pass hides one. `hasRot` counts only CONFIRMED absences, so an offline or
keyless run cannot fail a pipeline for something it never saw.
That distinction earned itself immediately: Node's global fetch is intermittently
broken on the laptop this was written on, and the checker correctly reported
7 UNCHECKED rather than "every model retired" — which would have sent me
"fixing" ids that were live.
`deadProviders` is separate from `hasRot` on purpose: a chain missing some
models is degraded, a chain missing an entire vendor is back to being a single
point of failure, and those deserve different words.
── modelCost is provider-dependent, and was silently wrong about it ─────────
The existing `paidModelsIn` guard caught the new Groq pins and called them PAID.
It was not a false alarm to suppress — it was the guard finding a real hole in
its own rule.
`modelCost` decides from the id: a slashed id without `:free` is paid. That is
right for OpenRouter, where `openai/gpt-oss-20b` genuinely bills. It is wrong
for Groq, where the same string is just that vendor's name for a model whose
cost is the account's tier. The rule was only ever safe because direct vendors
used bare ids (`llama-3.1-8b-instant` → "unknown"); Groq now ships
vendor-prefixed ids, so the shape stopped identifying the vendor.
So `Provider` gains `routed?: boolean` (true for OpenRouter), and `modelCostAt`
judges an id AT its provider. At a direct vendor the verdict is "unknown" —
never "free", which would reopen the hole this module was written to close.
`modelCost(id)` is unchanged for existing callers, with a docstring saying what
it assumes.
A false "paid" here is not harmless: it pressures someone into "fixing" a
working free model, and a guard that cries wolf gets disabled — taking the three
real cases it does catch with it. Pinned by a test that the routed vendor's
`anthropic/claude-sonnet-5` and `google/gemini-2.0-flash-001` are still flagged.
verify: 44/44 pass. Live catalogue check: all 7 pins present, no rot. Mutation:
restoring the retired pins turns it red (exit 1, deadProviders ["groq"]).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Consumers pin this package by git TAG (FleetCrown: github:maonakamoto/ai-ration#v0.2.0), so the retired-model fix only reaches them once a new tag exists and their pin moves. Bumping here keeps package.json and the tag in agreement — the previous release left an installed copy reporting 0.1.0 under a #v0.2.0 pin, which makes 'which code is actually running' unanswerable from the tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions Bot
pushed a commit
to bitbaum/fleetcrown
that referenced
this pull request
Aug 25, 2026
…ad (#352) Companion to #350. That PR fixed FleetCrown's own direct Groq calls; this one fixes the shared chain underneath Loki. `ai-ration` is the package that exists BECAUSE a single pinned free model is a scheduled outage. It had the same disease: checked against the live catalogues, four of its nine default ids were gone. groq llama-3.3-70b-versatile GONE groq llama-3.1-8b-instant GONE <- whole vendor dead openrouter openai/gpt-oss-20b:free GONE <- was FIRST in the list openrouter nvidia/nemotron-3-nano-30b-... GONE So Loki's chain burned two Groq 404s and then a third on its preferred OpenRouter fallback before reaching a model that could answer. Not an outage — OpenRouter's remaining links carried it — but the fastest vendor and the whole 100k/day Groq budget were unreachable. Checked the direction that costs money: every OpenRouter link carries `:free`, so nothing had silently moved to paid. Upstream (bitbaum/ai-kit#4, tag v0.2.1) replaces the dead pins with `openai/gpt-oss-120b` / `openai/gpt-oss-20b` — probed with a real tool call, not guessed — and adds `checkCatalog()`: a zero-token existence check, three-state (present / retired / could-not-read), cheap enough to run on a schedule. The existing `probe:models` costs real tokens and so could never be scheduled, which is why five rots in a row went unnoticed. ── one test changed, and it was pinning a dead model ──────────────────────── `scripts/test/agent-name-lookup.ts` asserted the chain still contains `llama-3.1-8b-instant` — "a model verified to drive the loop". That model is one of the four the vendor retired. The guard was correct in intent and anchored to a single rotting id, so it failed because its one EXAMPLE died, not because the property broke. A rot-detector anchored to one pinned id reproduces the exact failure it is watching for. It now asserts the property against a set of models actually probed for tool-calling, with the probe date and an instruction not to add a model you have not run. Mutation-proven: emptying the verified set turns it red (exit 1). Verified end-to-end rather than assumed: the installed copy reports 0.2.1, resolves to 07c492c, and its dist/chain.js carries the live ids. The lockfile diff is exactly one dependency — no incidental churn. Co-authored-by: t <t@t.t> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: four of the nine default pins were already retired
Checked
freeChain()against the live catalogues on 2026-08-25. Four of itsnine ids no longer exist:
The chain exists because a single pinned free model is a scheduled outage. That
reasoning has a hole this package fell into: the chain is ITSELF a list of pins,
so it rots too — and a chain whose first vendor is entirely dead is a slower
version of the failure it was built to prevent. Every caller paid two Groq 404s
and then a third on the preferred OpenRouter fallback before reaching a model
that could answer.
Consumers inherited it. FleetCrown used the same Groq id for direct, unchained
calls and was silently broken for eight days: its nightly frontier digest fell
back to a canned headline, and activity digests, calendar extraction, proposal
extraction and hosted-runner analysis all failed one at a time, quietly enough
that nothing looked broken.
Replacements probed, not guessed — the package's own rule.
openai/gpt-oss-120band
openai/gpt-oss-20beach returned a correct native tool_call with correctarguments. The two retired OpenRouter ids are removed; the other five were
confirmed present in the 419-model catalogue.
── catalog: the check that can actually run on a timer ──────────────────────
New module, exported as
checkCatalog/hasRot/deadProviders/catalogReport. One GET /models per provider and ZERO tokens, which is theentire point: a tool-call probe costs real money and therefore can never be
scheduled, so the only existing defence was a command someone had to remember
to type. Existence is cheap enough to check nightly.
npm run check:catalogruns it against this package's own defaults and exits 1 on confirmed rot.
It lives here rather than in each app because the check is the same everywhere,
and the first app to need it had already written its own, slightly differently.
Three states, not two.
live: nullmeans the catalogue could not be READ — nokey, network failure, non-200, or a body that parses but lists nothing. That is
not an empty catalogue. Reporting it as rot invents an outage; reporting it as
a pass hides one.
hasRotcounts only CONFIRMED absences, so an offline orkeyless run cannot fail a pipeline for something it never saw.
That distinction earned itself immediately: Node's global fetch is intermittently
broken on the laptop this was written on, and the checker correctly reported
7 UNCHECKED rather than "every model retired" — which would have sent me
"fixing" ids that were live.
deadProvidersis separate fromhasRoton purpose: a chain missing somemodels is degraded, a chain missing an entire vendor is back to being a single
point of failure, and those deserve different words.
── modelCost is provider-dependent, and was silently wrong about it ─────────
The existing
paidModelsInguard caught the new Groq pins and called them PAID.It was not a false alarm to suppress — it was the guard finding a real hole in
its own rule.
modelCostdecides from the id: a slashed id without:freeis paid. That isright for OpenRouter, where
openai/gpt-oss-20bgenuinely bills. It is wrongfor Groq, where the same string is just that vendor's name for a model whose
cost is the account's tier. The rule was only ever safe because direct vendors
used bare ids (
llama-3.1-8b-instant→ "unknown"); Groq now shipsvendor-prefixed ids, so the shape stopped identifying the vendor.
So
Providergainsrouted?: boolean(true for OpenRouter), andmodelCostAtjudges an id AT its provider. At a direct vendor the verdict is "unknown" —
never "free", which would reopen the hole this module was written to close.
modelCost(id)is unchanged for existing callers, with a docstring saying whatit assumes.
A false "paid" here is not harmless: it pressures someone into "fixing" a
working free model, and a guard that cries wolf gets disabled — taking the three
real cases it does catch with it. Pinned by a test that the routed vendor's
anthropic/claude-sonnet-5andgoogle/gemini-2.0-flash-001are still flagged.verify: 44/44 pass. Live catalogue check: all 7 pins present, no rot. Mutation:
restoring the retired pins turns it red (exit 1, deadProviders ["groq"]).
Co-Authored-By: Claude Opus 5 noreply@anthropic.com