fix(ci): the audit was reporting a fraction and calling it a total - #47
Merged
Conversation
Kivvi has seven retired model ids. The audit reported two.
Understating a repo is worse than skipping one. A skipped repo is visibly
absent; an understated repo prints a number that reads like an answer, and the
five pins it left out were in `packages/ai` — Kivvi's actual provider chain.
Three independent causes, each found by chasing the previous one.
1. The AI word had to appear in the FILENAME. Kivvi keeps its provider clients
side by side:
packages/ai/src/providers/anthropic.ts ← scanned
packages/ai/src/providers/groq.ts ← never opened
packages/ai/src/providers/openrouter.ts ← never opened
packages/ai/src/providers/index.ts ← never opened
Two files in one directory, one seen and one not, decided entirely by which
vendor names someone had typed into a regex. A directory called `ai/` counted
for nothing. Path matching now looks at every segment below the source root,
and the vendor list is no longer four names long.
Matching is by TOKEN, never substring: `ai` is inside `mail`, `chain`,
`domain`, `detail` and `maintenance`, and substring matching would have
traded one blind spot for a flood that the file cap then drops real
candidates to make room for. Tested in both directions.
2. Opening the files was not enough — nothing came out of them. `extractPins`
matched `models` followed directly by `[`, and Kivvi writes
models: AIModel[] = [
where a TypeScript type annotation sits in between. The first `[` on that
line belongs to `AIModel[]` and closes immediately, so the pattern read an
empty array: a silent nothing, the worst output an audit can produce. The
same regex also capped the array at 400 characters, losing later entries of
any richly described list. Replaced with a bracket walker that has neither
limit and reports each id's own line — which also improves attribution, since
that is measured in lines from the pin.
3. The coverage ledger stopped being a caveat and became a finding: OrangeCat
opened 90 of 133 candidates with 4 likely-AI files dropped, FleetCrown 90 of
109 with 15 dropped. The two largest repos in the fleet, both partly blind.
Cap raised 90 -> 160, which clears both with headroom. The ledger stays, so
the next repo to outgrow it says so.
What the fleet actually looks like, before -> after: 12 retired pins -> 16, and
the composition changed more than the count. Newly visible:
- hirnli — 2 dead pins, a repo that had never appeared in a report at all
- orangecat:src/lib/ai/form-prefill-service.ts:23 — genuinely broken, and it
is the repo this audit cleared as a false positive earlier today. It was
right that the pricing-table string was not a pin, and wrong that the repo
was fine.
- kivvi packages/ai — 5 more, including a `google/gemini-2.0-flash-001` that
no previous run had seen
- evig src/lib/hirn/providers — 2 more
Every one of them was retired before this commit. None of them was new.
Self-test: 57 -> 76 checks, still no network, no key, no checkout. The fixtures
are the real paths and the real declaration, so this specific miss cannot come
back quietly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 27, 2026
…ins (#48) Repairing six repos today was a live test of this audit, and it failed in both directions at once: it missed retired ids in shapes it could not parse, and reported ids that were never pins. Six faults, each found by acting on the previous report and looking at what it pointed to. MISSED — ids the audit could not see 1. Model MAPS. The walker only opened on `[`, and two repos keep their ids in object literals — on opposite sides of the colon. Hirnli: `{ '8b': 'llama-3.1-8b-instant' }`, ids as values. OrangeCat: `{ 'llama-3.3-70b-versatile': { contextWindow: ... } }`, ids as keys, including `DEFAULT_GROQ_MODEL` — the baseline every free non-BYOK user gets. Both registries were entirely retired and both read as clean. 2. `\bmodels?\b` does not match `GROQ_MODELS`. The underscore before it is a word character, so there is no boundary there — and almost every map in this fleet is named exactly that way. This was why (1) still found nothing after the walker was taught about braces. 3. `modelId` is not `model`. Kivvi's fallback is `const FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }` — singular, so no collection walker covers it, and the single-id pattern was anchored on the bare word. Found only by re-running the live sweep after fix (4) and noticing a real finding had disappeared. REPORTED — things that were never pins 4. `for (const model of models) {` opened a region over an entire loop body, so the request headers inside were read as ids and a `Content-Type: application/json` was reported as a retired Groq model. The opener now requires a declaration (`[:=]` right after the token) and a PLURAL token — `function supportsReasoningEffort(model: string): boolean {` is a declaration by the first rule alone, and its `startsWith("qwen/")` prefixes were being reported too. 5. The audit was reading ids out of COMMENTS. A model list is exactly where someone documents the id they just replaced, usually in backticks — so Kivvi's `// replaces `meta-llama/llama-3.2-3b-instruct:free`, retired` was reported as a live pin in the very commit that removed it. Comments are stripped from a region before extraction. An audit reporting on prose rather than code is the failure it exists to catch elsewhere. 6. Two misattributions, both "a true statement about the wrong vendor": - `gpt-4o-mini` under an `openai:` key was called a retired OPENROUTER model. It is not an OpenRouter id at all; there it would be `openai/gpt-4o-mini`. Nearest-marker-above found OpenRouter's base URL in the block before. A provider-keyed record now names its own rows via `keyMarker`, anchored to a line starting with the key and a colon — necessary, because bare `openai` appears inside `api.groq.com/openai/v1`. - `OLLAMA_MODEL=llama3.2` was called a retired Groq model. Ollama is now a listed non-queryable vendor, so its tags are attributed and reported unchecked. What the operator pulled locally is not a question Groq can answer. Also: a size alias is not a model id. Reading maps turned Hirnli's `{ '70b': ..., '8b': ... }` keys into findings — `70b` has a digit and no space, so it looked like an id, and `8b` escaped only by being two characters long. A vendor id always carries a separator: a slash, a hyphen or a dot. Nothing in either live catalogue is a single unseparated word. Net effect on the live sweep, across the same 34 repos: before this session 12 retired, 5 repos, several of them wrong after coverage (#47) 16 retired, and hirnli appeared for the first time now 2 retired, both real, both in Kivvi The intervening repairs account for most of the drop; this commit accounts for the false ones. Every remaining line is a genuine retired id in a genuine call path. Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and provider record, Kivvi's typed array and fallback, the loop body, the comment. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(ci): the audit was reporting a fraction and calling it a total
Kivvi has seven retired model ids. The audit reported two.
Understating a repo is worse than skipping one. A skipped repo is visibly
absent; an understated repo prints a number that reads like an answer, and the
five pins it left out were in
packages/ai— Kivvi's actual provider chain.Three independent causes, each found by chasing the previous one.
The AI word had to appear in the FILENAME. Kivvi keeps its provider clients
side by side:
Two files in one directory, one seen and one not, decided entirely by which
vendor names someone had typed into a regex. A directory called
ai/countedfor nothing. Path matching now looks at every segment below the source root,
and the vendor list is no longer four names long.
Matching is by TOKEN, never substring:
aiis insidemail,chain,domain,detailandmaintenance, and substring matching would havetraded one blind spot for a flood that the file cap then drops real
candidates to make room for. Tested in both directions.
Opening the files was not enough — nothing came out of them.
extractPinsmatched
modelsfollowed directly by[, and Kivvi writeswhere a TypeScript type annotation sits in between. The first
[on thatline belongs to
AIModel[]and closes immediately, so the pattern read anempty array: a silent nothing, the worst output an audit can produce. The
same regex also capped the array at 400 characters, losing later entries of
any richly described list. Replaced with a bracket walker that has neither
limit and reports each id's own line — which also improves attribution, since
that is measured in lines from the pin.
The coverage ledger stopped being a caveat and became a finding: OrangeCat
opened 90 of 133 candidates with 4 likely-AI files dropped, FleetCrown 90 of
109 with 15 dropped. The two largest repos in the fleet, both partly blind.
Cap raised 90 -> 160, which clears both with headroom. The ledger stays, so
the next repo to outgrow it says so.
What the fleet actually looks like, before -> after: 12 retired pins -> 16, and
the composition changed more than the count. Newly visible:
is the repo this audit cleared as a false positive earlier today. It was
right that the pricing-table string was not a pin, and wrong that the repo
was fine.
google/gemini-2.0-flash-001thatno previous run had seen
Every one of them was retired before this commit. None of them was new.
Self-test: 57 -> 76 checks, still no network, no key, no checkout. The fixtures
are the real paths and the real declaration, so this specific miss cannot come
back quietly.
🤖 Generated with Claude Code