Skip to content

fix: four of the nine default pins were already retired - #4

Merged
catomean merged 2 commits into
mainfrom
fix/catalog-rot
Aug 25, 2026
Merged

fix: four of the nine default pins were already retired#4
catomean merged 2 commits into
mainfrom
fix/catalog-rot

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

fix: four of the nine default pins were already retired

Checked freeChain() against the live catalogues on 2026-08-25. Four of its
nine ids no longer exist:

groq       llama-3.3-70b-versatile        GONE
groq       llama-3.1-8b-instant           GONE   <- whole vendor dead
openrouter openai/gpt-oss-20b:free        GONE   <- was FIRST in the list
openrouter nvidia/nemotron-3-nano-30b...  GONE

The chain exists because a single pinned free model is a scheduled outage. That
reasoning has a hole this package fell into: the chain is ITSELF a list of pins,
so it rots too — and a chain whose first vendor is entirely dead is a slower
version of the failure it was built to prevent. Every caller paid two Groq 404s
and then a third on the preferred OpenRouter fallback before reaching a model
that could answer.

Consumers inherited it. FleetCrown used the same Groq id for direct, unchained
calls and was silently broken for eight days: its nightly frontier digest fell
back to a canned headline, and activity digests, calendar extraction, proposal
extraction and hosted-runner analysis all failed one at a time, quietly enough
that nothing looked broken.

Replacements probed, not guessed — the package's own rule. openai/gpt-oss-120b
and openai/gpt-oss-20b each returned a correct native tool_call with correct
arguments. The two retired OpenRouter ids are removed; the other five were
confirmed present in the 419-model catalogue.

── catalog: the check that can actually run on a timer ──────────────────────

New module, exported as checkCatalog / hasRot / deadProviders /
catalogReport. One GET /models per provider and ZERO tokens, which is the
entire point: a tool-call probe costs real money and therefore can never be
scheduled, so the only existing defence was a command someone had to remember
to type. Existence is cheap enough to check nightly. npm run check:catalog
runs it against this package's own defaults and exits 1 on confirmed rot.

It lives here rather than in each app because the check is the same everywhere,
and the first app to need it had already written its own, slightly differently.

Three states, not two. live: null means the catalogue could not be READ — no
key, network failure, non-200, or a body that parses but lists nothing. That is
not an empty catalogue. Reporting it as rot invents an outage; reporting it as
a pass hides one. hasRot counts only CONFIRMED absences, so an offline or
keyless run cannot fail a pipeline for something it never saw.

That distinction earned itself immediately: Node's global fetch is intermittently
broken on the laptop this was written on, and the checker correctly reported
7 UNCHECKED rather than "every model retired" — which would have sent me
"fixing" ids that were live.

deadProviders is separate from hasRot on purpose: a chain missing some
models is degraded, a chain missing an entire vendor is back to being a single
point of failure, and those deserve different words.

── modelCost is provider-dependent, and was silently wrong about it ─────────

The existing paidModelsIn guard caught the new Groq pins and called them PAID.
It was not a false alarm to suppress — it was the guard finding a real hole in
its own rule.

modelCost decides from the id: a slashed id without :free is paid. That is
right for OpenRouter, where openai/gpt-oss-20b genuinely bills. It is wrong
for Groq, where the same string is just that vendor's name for a model whose
cost is the account's tier. The rule was only ever safe because direct vendors
used bare ids (llama-3.1-8b-instant → "unknown"); Groq now ships
vendor-prefixed ids, so the shape stopped identifying the vendor.

So Provider gains routed?: boolean (true for OpenRouter), and modelCostAt
judges an id AT its provider. At a direct vendor the verdict is "unknown" —
never "free", which would reopen the hole this module was written to close.
modelCost(id) is unchanged for existing callers, with a docstring saying what
it assumes.

A false "paid" here is not harmless: it pressures someone into "fixing" a
working free model, and a guard that cries wolf gets disabled — taking the three
real cases it does catch with it. Pinned by a test that the routed vendor's
anthropic/claude-sonnet-5 and google/gemini-2.0-flash-001 are still flagged.

verify: 44/44 pass. Live catalogue check: all 7 pins present, no rot. Mutation:
restoring the retired pins turns it red (exit 1, deadProviders ["groq"]).

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

catomean and others added 2 commits August 25, 2026 15:07
Checked `freeChain()` against the live catalogues on 2026-08-25. Four of its
nine ids no longer exist:

    groq       llama-3.3-70b-versatile        GONE
    groq       llama-3.1-8b-instant           GONE   <- whole vendor dead
    openrouter openai/gpt-oss-20b:free        GONE   <- was FIRST in the list
    openrouter nvidia/nemotron-3-nano-30b...  GONE

The chain exists because a single pinned free model is a scheduled outage. That
reasoning has a hole this package fell into: the chain is ITSELF a list of pins,
so it rots too — and a chain whose first vendor is entirely dead is a slower
version of the failure it was built to prevent. Every caller paid two Groq 404s
and then a third on the preferred OpenRouter fallback before reaching a model
that could answer.

Consumers inherited it. FleetCrown used the same Groq id for direct, unchained
calls and was silently broken for eight days: its nightly frontier digest fell
back to a canned headline, and activity digests, calendar extraction, proposal
extraction and hosted-runner analysis all failed one at a time, quietly enough
that nothing looked broken.

Replacements probed, not guessed — the package's own rule. `openai/gpt-oss-120b`
and `openai/gpt-oss-20b` each returned a correct native tool_call with correct
arguments. The two retired OpenRouter ids are removed; the other five were
confirmed present in the 419-model catalogue.

── catalog: the check that can actually run on a timer ──────────────────────

New module, exported as `checkCatalog` / `hasRot` / `deadProviders` /
`catalogReport`. One GET /models per provider and ZERO tokens, which is the
entire point: a tool-call probe costs real money and therefore can never be
scheduled, so the only existing defence was a command someone had to remember
to type. Existence is cheap enough to check nightly. `npm run check:catalog`
runs it against this package's own defaults and exits 1 on confirmed rot.

It lives here rather than in each app because the check is the same everywhere,
and the first app to need it had already written its own, slightly differently.

Three states, not two. `live: null` means the catalogue could not be READ — no
key, network failure, non-200, or a body that parses but lists nothing. That is
not an empty catalogue. Reporting it as rot invents an outage; reporting it as
a pass hides one. `hasRot` counts only CONFIRMED absences, so an offline or
keyless run cannot fail a pipeline for something it never saw.

That distinction earned itself immediately: Node's global fetch is intermittently
broken on the laptop this was written on, and the checker correctly reported
7 UNCHECKED rather than "every model retired" — which would have sent me
"fixing" ids that were live.

`deadProviders` is separate from `hasRot` on purpose: a chain missing some
models is degraded, a chain missing an entire vendor is back to being a single
point of failure, and those deserve different words.

── modelCost is provider-dependent, and was silently wrong about it ─────────

The existing `paidModelsIn` guard caught the new Groq pins and called them PAID.
It was not a false alarm to suppress — it was the guard finding a real hole in
its own rule.

`modelCost` decides from the id: a slashed id without `:free` is paid. That is
right for OpenRouter, where `openai/gpt-oss-20b` genuinely bills. It is wrong
for Groq, where the same string is just that vendor's name for a model whose
cost is the account's tier. The rule was only ever safe because direct vendors
used bare ids (`llama-3.1-8b-instant` → "unknown"); Groq now ships
vendor-prefixed ids, so the shape stopped identifying the vendor.

So `Provider` gains `routed?: boolean` (true for OpenRouter), and `modelCostAt`
judges an id AT its provider. At a direct vendor the verdict is "unknown" —
never "free", which would reopen the hole this module was written to close.
`modelCost(id)` is unchanged for existing callers, with a docstring saying what
it assumes.

A false "paid" here is not harmless: it pressures someone into "fixing" a
working free model, and a guard that cries wolf gets disabled — taking the three
real cases it does catch with it. Pinned by a test that the routed vendor's
`anthropic/claude-sonnet-5` and `google/gemini-2.0-flash-001` are still flagged.

verify: 44/44 pass. Live catalogue check: all 7 pins present, no rot. Mutation:
restoring the retired pins turns it red (exit 1, deadProviders ["groq"]).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Consumers pin this package by git TAG (FleetCrown:
github:maonakamoto/ai-ration#v0.2.0), so the retired-model fix only reaches
them once a new tag exists and their pin moves. Bumping here keeps
package.json and the tag in agreement — the previous release left an installed
copy reporting 0.1.0 under a #v0.2.0 pin, which makes 'which code is actually
running' unanswerable from the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@catomean
catomean merged commit 07c492c into main Aug 25, 2026
1 check passed
@catomean
catomean deleted the fix/catalog-rot branch August 25, 2026 13:11
github-actions Bot pushed a commit to bitbaum/fleetcrown that referenced this pull request Aug 25, 2026
…ad (#352)

Companion to #350. That PR fixed FleetCrown's own direct Groq calls; this one
fixes the shared chain underneath Loki.

`ai-ration` is the package that exists BECAUSE a single pinned free model is a
scheduled outage. It had the same disease: checked against the live catalogues,
four of its nine default ids were gone.

    groq       llama-3.3-70b-versatile         GONE
    groq       llama-3.1-8b-instant            GONE   <- whole vendor dead
    openrouter openai/gpt-oss-20b:free         GONE   <- was FIRST in the list
    openrouter nvidia/nemotron-3-nano-30b-...  GONE

So Loki's chain burned two Groq 404s and then a third on its preferred
OpenRouter fallback before reaching a model that could answer. Not an outage —
OpenRouter's remaining links carried it — but the fastest vendor and the whole
100k/day Groq budget were unreachable. Checked the direction that costs money:
every OpenRouter link carries `:free`, so nothing had silently moved to paid.

Upstream (bitbaum/ai-kit#4, tag v0.2.1) replaces the dead pins with
`openai/gpt-oss-120b` / `openai/gpt-oss-20b` — probed with a real tool call, not
guessed — and adds `checkCatalog()`: a zero-token existence check, three-state
(present / retired / could-not-read), cheap enough to run on a schedule. The
existing `probe:models` costs real tokens and so could never be scheduled, which
is why five rots in a row went unnoticed.

── one test changed, and it was pinning a dead model ────────────────────────

`scripts/test/agent-name-lookup.ts` asserted the chain still contains
`llama-3.1-8b-instant` — "a model verified to drive the loop". That model is one
of the four the vendor retired. The guard was correct in intent and anchored to
a single rotting id, so it failed because its one EXAMPLE died, not because the
property broke. A rot-detector anchored to one pinned id reproduces the exact
failure it is watching for.

It now asserts the property against a set of models actually probed for
tool-calling, with the probe date and an instruction not to add a model you have
not run. Mutation-proven: emptying the verified set turns it red (exit 1).

Verified end-to-end rather than assumed: the installed copy reports 0.2.1,
resolves to 07c492c, and its dist/chain.js carries the live ids. The lockfile
diff is exactly one dependency — no incidental churn.

Co-authored-by: t <t@t.t>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant