Skip to content

fix(ci): read model maps, and stop reporting things that were never pins - #48

Merged
github-actions[bot] merged 1 commit into
masterfrom
fix/audit-model-maps
Aug 27, 2026
Merged

fix(ci): read model maps, and stop reporting things that were never pins#48
github-actions[bot] merged 1 commit into
masterfrom
fix/audit-model-maps

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

fix(ci): read model maps, and stop reporting things that were never pins

Repairing six repos today was a live test of this audit, and it failed in both
directions at once: it missed retired ids in shapes it could not parse, and
reported ids that were never pins. Six faults, each found by acting on the
previous report and looking at what it pointed to.

MISSED — ids the audit could not see

  1. Model MAPS. The walker only opened on [, and two repos keep their ids in
    object literals — on opposite sides of the colon. Hirnli:
    { '8b': 'llama-3.1-8b-instant' }, ids as values. OrangeCat:
    { 'llama-3.3-70b-versatile': { contextWindow: ... } }, ids as keys,
    including DEFAULT_GROQ_MODEL — the baseline every free non-BYOK user gets.
    Both registries were entirely retired and both read as clean.

  2. \bmodels?\b does not match GROQ_MODELS. The underscore before it is a
    word character, so there is no boundary there — and almost every map in this
    fleet is named exactly that way. This was why (1) still found nothing after
    the walker was taught about braces.

  3. modelId is not model. Kivvi's fallback is
    const FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }
    — singular, so no collection walker covers it, and the single-id pattern was
    anchored on the bare word. Found only by re-running the live sweep after
    fix (4) and noticing a real finding had disappeared.

REPORTED — things that were never pins

  1. for (const model of models) { opened a region over an entire loop body,
    so the request headers inside were read as ids and a Content-Type: application/json was reported as a retired Groq model. The opener now
    requires a declaration ([:=] right after the token) and a PLURAL token —
    function supportsReasoningEffort(model: string): boolean { is a
    declaration by the first rule alone, and its startsWith("qwen/") prefixes
    were being reported too.

  2. The audit was reading ids out of COMMENTS. A model list is exactly where
    someone documents the id they just replaced, usually in backticks — so
    Kivvi's // replaces meta-llama/llama-3.2-3b-instruct:free, retired was
    reported as a live pin in the very commit that removed it. Comments are
    stripped from a region before extraction. An audit reporting on prose rather
    than code is the failure it exists to catch elsewhere.

  3. Two misattributions, both "a true statement about the wrong vendor":

    • gpt-4o-mini under an openai: key was called a retired OPENROUTER
      model. It is not an OpenRouter id at all; there it would be
      openai/gpt-4o-mini. Nearest-marker-above found OpenRouter's base URL in
      the block before. A provider-keyed record now names its own rows via
      keyMarker, anchored to a line starting with the key and a colon —
      necessary, because bare openai appears inside api.groq.com/openai/v1.
    • OLLAMA_MODEL=llama3.2 was called a retired Groq model. Ollama is now a
      listed non-queryable vendor, so its tags are attributed and reported
      unchecked. What the operator pulled locally is not a question Groq can
      answer.

Also: a size alias is not a model id. Reading maps turned Hirnli's
{ '70b': ..., '8b': ... } keys into findings — 70b has a digit and no space,
so it looked like an id, and 8b escaped only by being two characters long. A
vendor id always carries a separator: a slash, a hyphen or a dot. Nothing in
either live catalogue is a single unseparated word.

Net effect on the live sweep, across the same 34 repos:

before this session 12 retired, 5 repos, several of them wrong
after coverage (#47) 16 retired, and hirnli appeared for the first time
now 2 retired, both real, both in Kivvi

The intervening repairs account for most of the drop; this commit accounts for
the false ones. Every remaining line is a genuine retired id in a genuine call
path.

Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture
is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and
provider record, Kivvi's typed array and fallback, the loop body, the comment.

🤖 Generated with Claude Code

Repairing six repos today was a live test of this audit, and it failed in both
directions at once: it missed retired ids in shapes it could not parse, and
reported ids that were never pins. Six faults, each found by acting on the
previous report and looking at what it pointed to.

MISSED — ids the audit could not see

1. Model MAPS. The walker only opened on `[`, and two repos keep their ids in
   object literals — on opposite sides of the colon. Hirnli:
   `{ '8b': 'llama-3.1-8b-instant' }`, ids as values. OrangeCat:
   `{ 'llama-3.3-70b-versatile': { contextWindow: ... } }`, ids as keys,
   including `DEFAULT_GROQ_MODEL` — the baseline every free non-BYOK user gets.
   Both registries were entirely retired and both read as clean.

2. `\bmodels?\b` does not match `GROQ_MODELS`. The underscore before it is a
   word character, so there is no boundary there — and almost every map in this
   fleet is named exactly that way. This was why (1) still found nothing after
   the walker was taught about braces.

3. `modelId` is not `model`. Kivvi's fallback is
   `const FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }`
   — singular, so no collection walker covers it, and the single-id pattern was
   anchored on the bare word. Found only by re-running the live sweep after
   fix (4) and noticing a real finding had disappeared.

REPORTED — things that were never pins

4. `for (const model of models) {` opened a region over an entire loop body,
   so the request headers inside were read as ids and a `Content-Type:
   application/json` was reported as a retired Groq model. The opener now
   requires a declaration (`[:=]` right after the token) and a PLURAL token —
   `function supportsReasoningEffort(model: string): boolean {` is a
   declaration by the first rule alone, and its `startsWith("qwen/")` prefixes
   were being reported too.

5. The audit was reading ids out of COMMENTS. A model list is exactly where
   someone documents the id they just replaced, usually in backticks — so
   Kivvi's `// replaces `meta-llama/llama-3.2-3b-instruct:free`, retired` was
   reported as a live pin in the very commit that removed it. Comments are
   stripped from a region before extraction. An audit reporting on prose rather
   than code is the failure it exists to catch elsewhere.

6. Two misattributions, both "a true statement about the wrong vendor":

   - `gpt-4o-mini` under an `openai:` key was called a retired OPENROUTER
     model. It is not an OpenRouter id at all; there it would be
     `openai/gpt-4o-mini`. Nearest-marker-above found OpenRouter's base URL in
     the block before. A provider-keyed record now names its own rows via
     `keyMarker`, anchored to a line starting with the key and a colon —
     necessary, because bare `openai` appears inside `api.groq.com/openai/v1`.
   - `OLLAMA_MODEL=llama3.2` was called a retired Groq model. Ollama is now a
     listed non-queryable vendor, so its tags are attributed and reported
     unchecked. What the operator pulled locally is not a question Groq can
     answer.

Also: a size alias is not a model id. Reading maps turned Hirnli's
`{ '70b': ..., '8b': ... }` keys into findings — `70b` has a digit and no space,
so it looked like an id, and `8b` escaped only by being two characters long. A
vendor id always carries a separator: a slash, a hyphen or a dot. Nothing in
either live catalogue is a single unseparated word.

Net effect on the live sweep, across the same 34 repos:

  before this session   12 retired, 5 repos, several of them wrong
  after coverage (#47)  16 retired, and hirnli appeared for the first time
  now                    2 retired, both real, both in Kivvi

The intervening repairs account for most of the drop; this commit accounts for
the false ones. Every remaining line is a genuine retired id in a genuine call
path.

Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture
is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and
provider record, Kivvi's typed array and fallback, the loop body, the comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit c45719f into master Aug 27, 2026
2 checks passed
@github-actions
github-actions Bot deleted the fix/audit-model-maps branch August 27, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant