feat(ci): ask the vendor, daily, whether our model ids still exist

.github/workflows/ci.yml

@@ -106,6 +106,16 @@ jobs:
       - name: The auto-merge sweep behaves
         run: bash scripts/ci/test-auto-merge-sweep.sh
 
+      # This detector reports on repos nobody is watching, so its own failure
+      # mode is silence: a clean report from a broken audit is worse than no
+      # audit, because it prints a ✓. Both sides are pinned — the real AOZ
+      # regression is still caught, corrected code stays quiet — plus the two
+      # false positives the first live run produced (xAI's `grok-3-mini` filed
+      # under Groq, and a computed `${...}` id read as a pin). No network, no
+      # key, no checkout, so it runs here as well as in the daily sweep.
+      - name: The model-pin audit still detects, and still stays quiet
+        run: node scripts/ci/test-model-pin-audit.mjs
+
       # Drift guard. The fleet is on v7; templates handing out v4 is exactly
       # how this repo fell behind the repos it governs.
       - name: No stale action versions

.github/workflows/model-pins.yml

@@ -0,0 +1,95 @@
+# Is any model id this fleet pins still served by its vendor?
+#
+# On 2026-08-26 the AOZ assistant reported "KI-Assistent nicht konfiguriert" on
+# a deployment whose key was valid. Groq had retired the whole llama-3.x family.
+# Production had been failing exactly as long as the demo, and nobody knew,
+# because "not configured" is the only thing the app can say.
+#
+# Six pins across five repos were dead the same morning. None of it was
+# detectable from inside a repo: every gate was green, because the code is
+# correct and the vendor changed underneath it.
+#
+# Daily, not weekly. A retirement is decided by someone outside this fleet and
+# lands without warning — unlike the UI defects next door, which arrive with a
+# design change. The check costs ZERO tokens (one GET /models per vendor), so
+# the only argument against running it often is noise, and a silent day is one
+# line in a job summary.
+name: Model pins
+
+on:
+  schedule:
+    # 07:10 UTC, before the working day — a retirement found at 09:00 is a
+    # morning's work; one found by a user is an outage of unknown age.
+    - cron: '10 7 * * *'
+  workflow_dispatch:
+    inputs:
+      strict:
+        description: 'Fail the run when pins are retired'
+        type: boolean
+        default: false
+  pull_request:
+    paths:
+      - 'scripts/ci/model-pin-audit.mjs'
+      - 'scripts/ci/test-model-pin-audit.mjs'
+      - '.github/workflows/model-pins.yml'
+
+permissions:
+  contents: read
+
+jobs:
+  audit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 22
+
+      # The detector's own test needs no network, no key and no checkout, so it
+      # runs on pull_request too — where the sweep below cannot, because a fork
+      # has no secrets and an unreadable catalogue is deliberately not a pass.
+      - name: Self-test the detector
+        run: node scripts/ci/test-model-pin-audit.mjs
+
+      # dotfiles has no package.json on purpose. Install ai-ration into a
+      # scratch dir and point the audit at it — the same shape ui-defects.yml
+      # uses for playwright. The vendor query lives in ai-ration precisely so
+      # this repo does not grow a second copy of it.
+      - name: Install ai-ration
+        if: github.event_name != 'pull_request'
+        run: |
+          mkdir -p "$RUNNER_TEMP/air" && cd "$RUNNER_TEMP/air"
+          npm init -y >/dev/null
+          npm i --no-audit --no-fund github:maonakamoto/ai-ration#v0.2.1 >/dev/null
+
+      - name: Audit the fleet's pins
+        if: github.event_name != 'pull_request'
+        env:
+          AI_RATION_FROM: ${{ runner.temp }}/air
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          # Absent keys are handled, not fatal: the vendor is reported UNCHECKED
+          # rather than clean. "I could not look" is not "nothing is wrong".
+          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+        run: |
+          set -uo pipefail
+          flag=--warn-only
+          if [ "${{ inputs.strict }}" = "true" ]; then flag=""; fi
+
+          # Into the job summary, not just the log. A finding nobody scrolls to
+          # is the same as no finding — which is how eight days passed.
+          {
+            echo '## Model pins'
+            echo
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+          set +e
+          node scripts/ci/model-pin-audit.mjs $flag 2>&1 | tee -a "$GITHUB_STEP_SUMMARY"
+          status=${PIPESTATUS[0]}
+          set -e
+
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          exit "$status"

SHARED.md

@@ -28,7 +28,9 @@ the inventory underneath it is **generated**, and the number it produces is a
 | [`limitkit`](https://github.com/maonakamoto/limitkit) | `npm i github:maonakamoto/limitkit#v0.1.0` | the fleet's **12 hand-rolled rate limiters** (this file's own "next extraction" row). Sliding/fixed windows over an injectable two-method `Store`; **bounded** memory default (the unbounded-Map leak is impossible by construction); standard `X-RateLimit-*` + `Retry-After` headers — what orangecat's ADR-0002 specified seven months before anything enforced it; `clientIp()`. Refusals count nothing, so a hammered key recovers. Ships no middleware and **no limit values** — how many attempts a route allows is app semantics, asserted locally. |
 
 **Adopted:** `ai-forms` — fleetcrown, evig, aoz-housing, surf-your-life.
-`ai-ration` — fleetcrown. `threadkit` — **nobody yet**.
+`ai-ration` — fleetcrown, **and this repo** (`model-pin-audit.mjs` calls
+`checkCatalog`; the audit needed exactly the vendor query the package owns, so
+writing a second one here would have been this file's own sin). `threadkit` — **nobody yet**.
 `limitkit` — fleetcrown (proving consumer; its old limiter had the unbounded
 Map). **Next adopter should be orangecat** — it closes ADR-0002 by making its
 Upstash client a 12-line `Store` adapter and deleting three of its four
@@ -73,6 +75,7 @@ fleet-wide checker.
 | Script | Answers |
 |---|---|
 | `scripts/ci/verify-floor-audit.sh` | does every repo's `verify` actually run lint + typecheck + test? |
+| `scripts/ci/model-pin-audit.mjs` | is any model id the fleet pins no longer served by its vendor? Zero tokens — one `GET /models` per vendor — so it runs DAILY. Uses `ai-ration`'s `checkCatalog` rather than a second vendor query. Self-tested by `scripts/ci/test-model-pin-audit.mjs`, which pins both sides and both real defects the first run produced: xAI's `grok-3-mini` misfiled under Groq, and a computed `${...}` id reported as a pin. |
 | `scripts/ci/ui-defect-audit.mjs` | do any live sites ship an interactive label below its WCAG AA floor, or a stack whose rows start at different x? Renders each site; no repo checkout involved. Self-tested by `scripts/ci/test-ui-defect-audit.mjs`, which pins BOTH sides — the real defect is still caught, correct markup stays silent. |
 
 Both report into a weekly workflow's job summary rather than only a log.
@@ -86,7 +89,7 @@ Ranked by (copies × how identical the logic is). Counts from
 |---|---|---|
 | `auto-merge-sweep.sh` | ~~22~~ **6** | **EXTRACTED 2026-08-16/20.** Sixteen repos call the canonical as a reusable workflow, each verified to actually *run* it (a sweep that fails to start looks exactly like one with nothing to do). The six remaining are deliberate: dotfiles is the canonical home and runs it directly; ai-forms, datacat, petvity, solon had dirty working trees owned by other sessions when swept — convert when clear. The two repos that had ever *tested* their copies (evig, orangecat) had that coverage ported into the canonical suite **before** deletion: 17 cases, mutation-proven. |
 | rate limiting | **14 → adopting** | **Extracted 2026-08-20 as [`limitkit`](https://github.com/maonakamoto/limitkit)** (see registry above). fleetcrown converted as the proving consumer; 13 files remain across 8 repos, orangecat first in line (its ADR-0002 becomes a 12-line `Store` adapter + three deletions). The ratchet holds the count until each adoption lands. |
-| AI provider client | **16** | evig 7, orangecat 5. `ai-ration` already owns the hard part (chain, 429, budget); these are the callers. |
+| AI provider client | **16** | evig 7, orangecat 5. `ai-ration` already owns the hard part (chain, 429, budget); these are the callers. **Priced 2026-08-26:** Groq retired the llama-3.x family and six pins across botsmann, evig, kivvi, orangecat and truthseeker died at once, with three deployed apps failing live and AOZ reporting a missing key it already had. Every one of those repos hand-rolls this layer; fleetcrown, which adopted the package, was unaffected. |
 | logger | **10** | sbb-lost-found alone has 4. |
 | health route | **8** | Identical shape in 8 repos; a 20-line contract. |
 | ~~`@ai-native-cms/core`~~ | — | **Withdrawn — measurement error.** `maonakamoto/revampit` *redirects* to `maonakamoto/evig` (renamed in the pivot); the "two repos" with byte-identical trees were two clones of ONE repo. Nothing to extract. Two directories are not two repos: check `git remote -v` before reporting cross-repo duplication. |