fix(ci): read model maps, and stop reporting things that were never pins - #48
Merged
Merged
Conversation
Repairing six repos today was a live test of this audit, and it failed in both
directions at once: it missed retired ids in shapes it could not parse, and
reported ids that were never pins. Six faults, each found by acting on the
previous report and looking at what it pointed to.
MISSED — ids the audit could not see
1. Model MAPS. The walker only opened on `[`, and two repos keep their ids in
object literals — on opposite sides of the colon. Hirnli:
`{ '8b': 'llama-3.1-8b-instant' }`, ids as values. OrangeCat:
`{ 'llama-3.3-70b-versatile': { contextWindow: ... } }`, ids as keys,
including `DEFAULT_GROQ_MODEL` — the baseline every free non-BYOK user gets.
Both registries were entirely retired and both read as clean.
2. `\bmodels?\b` does not match `GROQ_MODELS`. The underscore before it is a
word character, so there is no boundary there — and almost every map in this
fleet is named exactly that way. This was why (1) still found nothing after
the walker was taught about braces.
3. `modelId` is not `model`. Kivvi's fallback is
`const FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }`
— singular, so no collection walker covers it, and the single-id pattern was
anchored on the bare word. Found only by re-running the live sweep after
fix (4) and noticing a real finding had disappeared.
REPORTED — things that were never pins
4. `for (const model of models) {` opened a region over an entire loop body,
so the request headers inside were read as ids and a `Content-Type:
application/json` was reported as a retired Groq model. The opener now
requires a declaration (`[:=]` right after the token) and a PLURAL token —
`function supportsReasoningEffort(model: string): boolean {` is a
declaration by the first rule alone, and its `startsWith("qwen/")` prefixes
were being reported too.
5. The audit was reading ids out of COMMENTS. A model list is exactly where
someone documents the id they just replaced, usually in backticks — so
Kivvi's `// replaces `meta-llama/llama-3.2-3b-instruct:free`, retired` was
reported as a live pin in the very commit that removed it. Comments are
stripped from a region before extraction. An audit reporting on prose rather
than code is the failure it exists to catch elsewhere.
6. Two misattributions, both "a true statement about the wrong vendor":
- `gpt-4o-mini` under an `openai:` key was called a retired OPENROUTER
model. It is not an OpenRouter id at all; there it would be
`openai/gpt-4o-mini`. Nearest-marker-above found OpenRouter's base URL in
the block before. A provider-keyed record now names its own rows via
`keyMarker`, anchored to a line starting with the key and a colon —
necessary, because bare `openai` appears inside `api.groq.com/openai/v1`.
- `OLLAMA_MODEL=llama3.2` was called a retired Groq model. Ollama is now a
listed non-queryable vendor, so its tags are attributed and reported
unchecked. What the operator pulled locally is not a question Groq can
answer.
Also: a size alias is not a model id. Reading maps turned Hirnli's
`{ '70b': ..., '8b': ... }` keys into findings — `70b` has a digit and no space,
so it looked like an id, and `8b` escaped only by being two characters long. A
vendor id always carries a separator: a slash, a hyphen or a dot. Nothing in
either live catalogue is a single unseparated word.
Net effect on the live sweep, across the same 34 repos:
before this session 12 retired, 5 repos, several of them wrong
after coverage (#47) 16 retired, and hirnli appeared for the first time
now 2 retired, both real, both in Kivvi
The intervening repairs account for most of the drop; this commit accounts for
the false ones. Every remaining line is a genuine retired id in a genuine call
path.
Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture
is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and
provider record, Kivvi's typed array and fallback, the loop body, the comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(ci): read model maps, and stop reporting things that were never pins
Repairing six repos today was a live test of this audit, and it failed in both
directions at once: it missed retired ids in shapes it could not parse, and
reported ids that were never pins. Six faults, each found by acting on the
previous report and looking at what it pointed to.
MISSED — ids the audit could not see
Model MAPS. The walker only opened on
[, and two repos keep their ids inobject literals — on opposite sides of the colon. Hirnli:
{ '8b': 'llama-3.1-8b-instant' }, ids as values. OrangeCat:{ 'llama-3.3-70b-versatile': { contextWindow: ... } }, ids as keys,including
DEFAULT_GROQ_MODEL— the baseline every free non-BYOK user gets.Both registries were entirely retired and both read as clean.
\bmodels?\bdoes not matchGROQ_MODELS. The underscore before it is aword character, so there is no boundary there — and almost every map in this
fleet is named exactly that way. This was why (1) still found nothing after
the walker was taught about braces.
modelIdis notmodel. Kivvi's fallback isconst FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }— singular, so no collection walker covers it, and the single-id pattern was
anchored on the bare word. Found only by re-running the live sweep after
fix (4) and noticing a real finding had disappeared.
REPORTED — things that were never pins
for (const model of models) {opened a region over an entire loop body,so the request headers inside were read as ids and a
Content-Type: application/jsonwas reported as a retired Groq model. The opener nowrequires a declaration (
[:=]right after the token) and a PLURAL token —function supportsReasoningEffort(model: string): boolean {is adeclaration by the first rule alone, and its
startsWith("qwen/")prefixeswere being reported too.
The audit was reading ids out of COMMENTS. A model list is exactly where
someone documents the id they just replaced, usually in backticks — so
Kivvi's
// replacesmeta-llama/llama-3.2-3b-instruct:free, retiredwasreported as a live pin in the very commit that removed it. Comments are
stripped from a region before extraction. An audit reporting on prose rather
than code is the failure it exists to catch elsewhere.
Two misattributions, both "a true statement about the wrong vendor":
gpt-4o-miniunder anopenai:key was called a retired OPENROUTERmodel. It is not an OpenRouter id at all; there it would be
openai/gpt-4o-mini. Nearest-marker-above found OpenRouter's base URL inthe block before. A provider-keyed record now names its own rows via
keyMarker, anchored to a line starting with the key and a colon —necessary, because bare
openaiappears insideapi.groq.com/openai/v1.OLLAMA_MODEL=llama3.2was called a retired Groq model. Ollama is now alisted non-queryable vendor, so its tags are attributed and reported
unchecked. What the operator pulled locally is not a question Groq can
answer.
Also: a size alias is not a model id. Reading maps turned Hirnli's
{ '70b': ..., '8b': ... }keys into findings —70bhas a digit and no space,so it looked like an id, and
8bescaped only by being two characters long. Avendor id always carries a separator: a slash, a hyphen or a dot. Nothing in
either live catalogue is a single unseparated word.
Net effect on the live sweep, across the same 34 repos:
before this session 12 retired, 5 repos, several of them wrong
after coverage (#47) 16 retired, and hirnli appeared for the first time
now 2 retired, both real, both in Kivvi
The intervening repairs account for most of the drop; this commit accounts for
the false ones. Every remaining line is a genuine retired id in a genuine call
path.
Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture
is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and
provider record, Kivvi's typed array and fallback, the loop body, the comment.
🤖 Generated with Claude Code