Skip to content

fix(ci): the audit was reporting a fraction and calling it a total - #47

Merged
github-actions[bot] merged 1 commit into
masterfrom
fix/audit-coverage
Aug 26, 2026
Merged

fix(ci): the audit was reporting a fraction and calling it a total#47
github-actions[bot] merged 1 commit into
masterfrom
fix/audit-coverage

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

fix(ci): the audit was reporting a fraction and calling it a total

Kivvi has seven retired model ids. The audit reported two.

Understating a repo is worse than skipping one. A skipped repo is visibly
absent; an understated repo prints a number that reads like an answer, and the
five pins it left out were in packages/ai — Kivvi's actual provider chain.

Three independent causes, each found by chasing the previous one.

  1. The AI word had to appear in the FILENAME. Kivvi keeps its provider clients
    side by side:

    packages/ai/src/providers/anthropic.ts   ← scanned
    packages/ai/src/providers/groq.ts        ← never opened
    packages/ai/src/providers/openrouter.ts  ← never opened
    packages/ai/src/providers/index.ts       ← never opened
    

    Two files in one directory, one seen and one not, decided entirely by which
    vendor names someone had typed into a regex. A directory called ai/ counted
    for nothing. Path matching now looks at every segment below the source root,
    and the vendor list is no longer four names long.

    Matching is by TOKEN, never substring: ai is inside mail, chain,
    domain, detail and maintenance, and substring matching would have
    traded one blind spot for a flood that the file cap then drops real
    candidates to make room for. Tested in both directions.

  2. Opening the files was not enough — nothing came out of them. extractPins
    matched models followed directly by [, and Kivvi writes

    models: AIModel[] = [
    

    where a TypeScript type annotation sits in between. The first [ on that
    line belongs to AIModel[] and closes immediately, so the pattern read an
    empty array: a silent nothing, the worst output an audit can produce. The
    same regex also capped the array at 400 characters, losing later entries of
    any richly described list. Replaced with a bracket walker that has neither
    limit and reports each id's own line — which also improves attribution, since
    that is measured in lines from the pin.

  3. The coverage ledger stopped being a caveat and became a finding: OrangeCat
    opened 90 of 133 candidates with 4 likely-AI files dropped, FleetCrown 90 of
    109 with 15 dropped. The two largest repos in the fleet, both partly blind.
    Cap raised 90 -> 160, which clears both with headroom. The ledger stays, so
    the next repo to outgrow it says so.

What the fleet actually looks like, before -> after: 12 retired pins -> 16, and
the composition changed more than the count. Newly visible:

  • hirnli — 2 dead pins, a repo that had never appeared in a report at all
  • orangecat:src/lib/ai/form-prefill-service.ts:23 — genuinely broken, and it
    is the repo this audit cleared as a false positive earlier today. It was
    right that the pricing-table string was not a pin, and wrong that the repo
    was fine.
  • kivvi packages/ai — 5 more, including a google/gemini-2.0-flash-001 that
    no previous run had seen
  • evig src/lib/hirn/providers — 2 more

Every one of them was retired before this commit. None of them was new.

Self-test: 57 -> 76 checks, still no network, no key, no checkout. The fixtures
are the real paths and the real declaration, so this specific miss cannot come
back quietly.

🤖 Generated with Claude Code

Kivvi has seven retired model ids. The audit reported two.

Understating a repo is worse than skipping one. A skipped repo is visibly
absent; an understated repo prints a number that reads like an answer, and the
five pins it left out were in `packages/ai` — Kivvi's actual provider chain.

Three independent causes, each found by chasing the previous one.

1. The AI word had to appear in the FILENAME. Kivvi keeps its provider clients
   side by side:

       packages/ai/src/providers/anthropic.ts   ← scanned
       packages/ai/src/providers/groq.ts        ← never opened
       packages/ai/src/providers/openrouter.ts  ← never opened
       packages/ai/src/providers/index.ts       ← never opened

   Two files in one directory, one seen and one not, decided entirely by which
   vendor names someone had typed into a regex. A directory called `ai/` counted
   for nothing. Path matching now looks at every segment below the source root,
   and the vendor list is no longer four names long.

   Matching is by TOKEN, never substring: `ai` is inside `mail`, `chain`,
   `domain`, `detail` and `maintenance`, and substring matching would have
   traded one blind spot for a flood that the file cap then drops real
   candidates to make room for. Tested in both directions.

2. Opening the files was not enough — nothing came out of them. `extractPins`
   matched `models` followed directly by `[`, and Kivvi writes

       models: AIModel[] = [

   where a TypeScript type annotation sits in between. The first `[` on that
   line belongs to `AIModel[]` and closes immediately, so the pattern read an
   empty array: a silent nothing, the worst output an audit can produce. The
   same regex also capped the array at 400 characters, losing later entries of
   any richly described list. Replaced with a bracket walker that has neither
   limit and reports each id's own line — which also improves attribution, since
   that is measured in lines from the pin.

3. The coverage ledger stopped being a caveat and became a finding: OrangeCat
   opened 90 of 133 candidates with 4 likely-AI files dropped, FleetCrown 90 of
   109 with 15 dropped. The two largest repos in the fleet, both partly blind.
   Cap raised 90 -> 160, which clears both with headroom. The ledger stays, so
   the next repo to outgrow it says so.

What the fleet actually looks like, before -> after: 12 retired pins -> 16, and
the composition changed more than the count. Newly visible:

  - hirnli — 2 dead pins, a repo that had never appeared in a report at all
  - orangecat:src/lib/ai/form-prefill-service.ts:23 — genuinely broken, and it
    is the repo this audit cleared as a false positive earlier today. It was
    right that the pricing-table string was not a pin, and wrong that the repo
    was fine.
  - kivvi packages/ai — 5 more, including a `google/gemini-2.0-flash-001` that
    no previous run had seen
  - evig src/lib/hirn/providers — 2 more

Every one of them was retired before this commit. None of them was new.

Self-test: 57 -> 76 checks, still no network, no key, no checkout. The fixtures
are the real paths and the real declaration, so this specific miss cannot come
back quietly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit 776afc3 into master Aug 26, 2026
2 checks passed
@github-actions
github-actions Bot deleted the fix/audit-coverage branch August 26, 2026 22:21
github-actions Bot pushed a commit that referenced this pull request Aug 27, 2026
…ins (#48)

Repairing six repos today was a live test of this audit, and it failed in both
directions at once: it missed retired ids in shapes it could not parse, and
reported ids that were never pins. Six faults, each found by acting on the
previous report and looking at what it pointed to.

MISSED — ids the audit could not see

1. Model MAPS. The walker only opened on `[`, and two repos keep their ids in
   object literals — on opposite sides of the colon. Hirnli:
   `{ '8b': 'llama-3.1-8b-instant' }`, ids as values. OrangeCat:
   `{ 'llama-3.3-70b-versatile': { contextWindow: ... } }`, ids as keys,
   including `DEFAULT_GROQ_MODEL` — the baseline every free non-BYOK user gets.
   Both registries were entirely retired and both read as clean.

2. `\bmodels?\b` does not match `GROQ_MODELS`. The underscore before it is a
   word character, so there is no boundary there — and almost every map in this
   fleet is named exactly that way. This was why (1) still found nothing after
   the walker was taught about braces.

3. `modelId` is not `model`. Kivvi's fallback is
   `const FALLBACK_MODEL: ModelSelection = { providerId: "groq", modelId: "..." }`
   — singular, so no collection walker covers it, and the single-id pattern was
   anchored on the bare word. Found only by re-running the live sweep after
   fix (4) and noticing a real finding had disappeared.

REPORTED — things that were never pins

4. `for (const model of models) {` opened a region over an entire loop body,
   so the request headers inside were read as ids and a `Content-Type:
   application/json` was reported as a retired Groq model. The opener now
   requires a declaration (`[:=]` right after the token) and a PLURAL token —
   `function supportsReasoningEffort(model: string): boolean {` is a
   declaration by the first rule alone, and its `startsWith("qwen/")` prefixes
   were being reported too.

5. The audit was reading ids out of COMMENTS. A model list is exactly where
   someone documents the id they just replaced, usually in backticks — so
   Kivvi's `// replaces `meta-llama/llama-3.2-3b-instruct:free`, retired` was
   reported as a live pin in the very commit that removed it. Comments are
   stripped from a region before extraction. An audit reporting on prose rather
   than code is the failure it exists to catch elsewhere.

6. Two misattributions, both "a true statement about the wrong vendor":

   - `gpt-4o-mini` under an `openai:` key was called a retired OPENROUTER
     model. It is not an OpenRouter id at all; there it would be
     `openai/gpt-4o-mini`. Nearest-marker-above found OpenRouter's base URL in
     the block before. A provider-keyed record now names its own rows via
     `keyMarker`, anchored to a line starting with the key and a colon —
     necessary, because bare `openai` appears inside `api.groq.com/openai/v1`.
   - `OLLAMA_MODEL=llama3.2` was called a retired Groq model. Ollama is now a
     listed non-queryable vendor, so its tags are attributed and reported
     unchecked. What the operator pulled locally is not a question Groq can
     answer.

Also: a size alias is not a model id. Reading maps turned Hirnli's
`{ '70b': ..., '8b': ... }` keys into findings — `70b` has a digit and no space,
so it looked like an id, and `8b` escaped only by being two characters long. A
vendor id always carries a separator: a slash, a hyphen or a dot. Nothing in
either live catalogue is a single unseparated word.

Net effect on the live sweep, across the same 34 repos:

  before this session   12 retired, 5 repos, several of them wrong
  after coverage (#47)  16 retired, and hirnli appeared for the first time
  now                    2 retired, both real, both in Kivvi

The intervening repairs account for most of the drop; this commit accounts for
the false ones. Every remaining line is a genuine retired id in a genuine call
path.

Self-test: 68 -> 102 checks, still no network, no key, no checkout. Every fixture
is the real code that fooled it — Hirnli's alias map, OrangeCat's registry and
provider record, Kivvi's typed array and fallback, the loop body, the comment.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant