diff --git a/.github/workflows/feed-refresh.yml b/.github/workflows/feed-refresh.yml index 0c065e3..c86850b 100644 --- a/.github/workflows/feed-refresh.yml +++ b/.github/workflows/feed-refresh.yml @@ -1,5 +1,5 @@ # Weekly feed refresh - regenerates feeds/ from the live provider pages and the -# AWS Bedrock and Google Vertex AI lifecycle pages, updates the README freshness +# AWS Bedrock, Google Vertex AI, and Azure Foundry lifecycle pages, updates the README freshness # line, and opens a PR with the semantic diff for human review. Never pushes to # main directly. PR reuse is ownership-checked and protected by an exact lease. # diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..06386d2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,45 @@ +# Agent instructions + +Read [CONTRIBUTING.md](CONTRIBUTING.md) first. Its layout table, test rules, +conventions, list of docs that tests check, and design decisions all apply to +agents. This file adds what an agent needs on top of that. + +## Before you finish a change + +1. Run `npm test`. It must pass offline. +2. For refresh changes, run `node refresh/refresh.mjs --check` against the live + sources and read the semantic diff. If you regenerated feeds, confirm + `--check` then exits 0. +3. For scanner or `--changed` changes, run `scripts/e2e-changed.mjs` as shown + in CONTRIBUTING.md. +4. Read your own diff for stacked `//` comment blocks, em-dashes, and tests that + assert live feed values. These are the mistakes agents make most often here. + +## Feeds + +- Never edit `feeds/*.json` by hand. +- If you have no network access, prove a parser change against fixtures and + leave the committed feeds untouched. Say in your report that the feeds need a + live regeneration. Never commit feeds generated with `--fixtures`. + +## Commits + +- No `Co-Authored-By` or other model-attribution trailers. Put + "Implemented by ; reviewed by " in the commit body instead. +- Rationale goes in the commit body, not in code comments. +- Don't commit unless you were asked to. + +## Releases and publishing + +These steps are usually done by the maintainer, but agents are sometimes asked +to run them. + +- A merge that changes only `feeds/` and the README freshness line releases a + patch to npm automatically. +- Any other commit since the last tag makes the next release a manual dispatch + of `npm-release.yml` with an explicit `release_version`. Choose the version + from the diff of the published `files` in `package.json`, not from the whole + repository diff. Changes to `docs/` count as code commits here. +- The Pages site publishes only feeds that match a feed-refresh receipt. After + merging feeds regenerated outside the refresh workflow, run + `gh workflow run feed-refresh.yml --ref main` so a matching receipt exists. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 10dcad8..dbf9ec8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,44 +1,39 @@ -# Contributing to model-eol +# Contributing -Thanks for your interest in model-eol - a machine-readable deprecation feed -format for AI models, plus the tooling that turns it into a Dependabot for -models. This guide covers the contributions the project most needs and the -conventions it holds to. +## Feed corrections -By contributing, you agree that your contributions are licensed under the -project's [MIT License](LICENSE). +If a feed has a wrong date, is missing a model, or doesn't carry a retirement +you know about, that's the most useful thing you can report. -## The most valuable contribution: feed corrections +Don't edit `feeds/*.json` by hand. The files are generated by `refresh/` from +publisher and distributor pages, and the next weekly refresh would overwrite +your change. Instead: -Found a wrong date, a missing model, or a retirement we don't carry? That is -exactly the signal this project runs on. One rule matters: - -**Feeds are generated - never hand-edit `feeds/*.json`.** Every entry comes -from `refresh/` parsing a provider's live page or endpoint. A hand-edited date -would be silently overwritten by the next weekly refresh. - -Instead: - -- **Wrong or missing data, source exists** - open an issue with the model ID, - the channel (publisher or distributor), the correct value, and the URL that - states it. If the parser should have caught it, that URL is the failing case. -- **Parser gap** - fix it in `refresh/providers.mjs` or +- **The data is wrong or missing and a source says so:** open a + [feed correction issue](https://github.com/thossullivan/model-eol/issues/new?template=feed-correction.md) + with the model ID, the channel (the publisher's own API or a distributor), the + correct value, and the URL that states it. If the parser should have picked + it up, that URL is the failing test case. +- **A parser missed something:** fix it in `refresh/providers.mjs` or `refresh/distributors.mjs`, add a fixture under `refresh/test/fixture/`, and - regenerate with `node refresh/refresh.mjs`. The semantic diff it prints is - your PR description. -- **New publisher or distributor** - the highest-value PR there is. Copy the - shape of an existing parser: strict table identity, model-ID grammar - validation, refusal over guessing. A parser that cannot be certain must - throw, not produce plausible data. + regenerate from the live pages with `node refresh/refresh.mjs`. Put the + semantic diff it prints in the PR description. Don't commit feeds generated + with `--fixtures`: the fixture's test models and aliases end up in the feed + as if they were real. +- **A new publisher or distributor:** start from the closest existing parser. + [`refresh/README.md`](refresh/README.md) explains the rules parsers follow. + The main one is that a parser that isn't sure must throw. If your change + makes a parser accept more input, add a test showing it still refuses input + that would attach a row to the wrong model. -If you work at a model provider: you can make the scraping unnecessary by -serving your own feed at `/.well-known/model-eol.json`. `SPEC.md` is small on -purpose. Open an issue - that conversation is the whole point of the project. +If you work at a model provider, the most useful change is to publish your own +feed at `/.well-known/model-eol.json`. [`SPEC.md`](SPEC.md) is short. Please +open an issue to discuss it. -## Development setup +## Development -Node 22+, nothing else. There are no dependencies to install - not for the -tool, not for development. +You need Node 22 or later and nothing else. There are no dependencies to +install, for the tool or for development. ```bash git clone https://github.com/thossullivan/model-eol @@ -46,46 +41,128 @@ cd model-eol npm test ``` -## Quality gates +| Path | What lives there | +|---|---| +| `check.mjs` | CLI entry point. It only dispatches; the logic is in `lib/`. | +| `lib/` | Scanner, lifecycle dates, config, reports, plan and apply, validation, verify. | +| `bot/` | `model-eol-bot`: plan, evaluate, and publish against the GitHub API. | +| `refresh/` | Feed generation. See [`refresh/README.md`](refresh/README.md). | +| `feeds/` | Generated feeds. | +| `schema/` | Public JSON schemas. | +| `scripts/` | Release, Pages, changelog, and test helpers. | +| `docs/` | Reference docs linked from the README. | + +`npm test` runs every suite: checker, refresh, bot, Action, schemas, the +example eval harness, the packed npm package, the changelog, and the public +site. It runs offline with no API keys, and CI runs the same command. For +changes to the scanner or `--changed`, also run the end-to-end harness against +a few real repositories: + +```bash +node scripts/e2e-changed.mjs --checker ./check.mjs --repo ../some-repo --repo ../another-repo +``` + +It replays recent commits through `check --changed` and compares each result +with a full scan of the same commit. -`npm test` composes the checker, refresh, bot, Action, schema validation, -example eval-harness, packed-consumer, changelog, and public-site suites - it -must pass offline, with no API keys. CI runs exactly this. Every behavior change -needs a regression test that fails on the old code. +### Tests -Two testing conventions that bite newcomers: +Every behavior change needs a test that fails on the old code. Two things +catch people out: -- **Content-stable assertions.** Feeds regenerate weekly, so never assert - against a committed feed's specific values (counts, dates, the `generated` - timestamp). Build a small fixture feed inside the test instead. -- **No network in tests.** Parsers are tested against recorded fixtures; - fetch paths take an injectable `fetchImpl`. +- **Committed feed values change every week.** The refresh workflow runs + `npm test` against freshly regenerated feeds, so a test that checks a real + count, date, or `generated` timestamp breaks the weekly refresh. Build a + small feed inside the test. A test that runs `refresh --check` against the + committed feeds should accept exit code 0 or 3. +- **Tests don't use the network.** Parsers are tested against recorded pages, + and fetch functions take an injectable `fetchImpl`. + +### Docs that tests check + +Some docs are read by tests or rewritten by scripts: + +- `README.md` has `` markers. The refresh workflow + rewrites the text between them. Keep the markers. +- `README.md` must contain exactly one Action snippet using + `actions/checkout@v7` and `actions/setup-node@v7` with `node-version: 22` + and `package-manager-cache: false`, plus the feed-refresh badge. +- Fragments of the local evaluation script in `docs/BOT.md` are checked by + `bot/test/package.mjs`. Change the script and the test together. +- `docs/RECEIPTS.md` must document `model-eol verify`. + +User-facing basics go in `README.md`, reference material in `docs/`, the feed +format in `SPEC.md`, and parser behavior in `refresh/README.md`. ## Conventions -- **Zero runtime dependencies** - Node builtins only. This is a feature, not - an accident; PRs adding a dependency will be declined. -- **Fail loudly.** Malformed input is refused with a named reason, never - coerced into plausible data. When in doubt, throw. -- Comments are single terse lines stating what the code cannot - no stacked - comment blocks. Rationale goes in the commit body. -- Use " - " rather than em-dashes in prose. -- Spec changes (`SPEC.md`, `schema/`) start as an issue, not a PR. Settled - design decisions live in `docs/CONTEXT.md` - read it first; it is the - record of what has already been decided and why. +- **No dependencies,** runtime or development. The test suite fails if one is + added. Everything the tool runs is in this repository, which keeps it easy to + audit. +- **Refuse bad input.** Malformed input fails with an error that names the + problem. It is never replaced with a default or a guess. +- **Comments are one short line** and say what the code can't. Put the + reasoning in the commit message. +- **No em-dashes in prose.** Use a comma, a colon, parentheses, or a new + sentence. +- **Write for a public repository.** Commits, issues, and docs contain no local + paths, private repository names, or personal data. +- Changes to `SPEC.md` or `schema/` start as an issue, not a PR. + +## Design decisions + +These are settled. To reopen one, open an issue that explains what has changed. +Decisions about the feed format itself are recorded in [`SPEC.md`](SPEC.md). + +The checker reports only strings that appear in a feed. It doesn't try to +recognize anything that merely looks like a model name, because false alarms in +CI get a check switched off faster than misses do. Unknown model-like strings +are listed as candidates and never fail or create work. + +Cloud and gateway references are hints until something resolves them. Static +analysis can see that code calls Bedrock or OpenRouter but usually can't see +which model sits behind a deployment name, so those references stay in the +inventory and don't fail `--scope direct`. The bot opens an issue for each such +provider instead, so a person can check what is deployed. + +Patching is strict. `plan` patches only references it is confident are direct +API calls, and only when the feed's `replacement` is itself safe. Cloud and +gateway references, prose replacements, and dates that came from +`publisher-fallback` become issues for a person. + +A distribution entry without a date means nothing is scheduled on that channel. +It doesn't inherit the publisher's date; only a missing entry falls back, and +the result is labeled `publisher-fallback`. An earlier, more cautious reading +told Azure users their models were gone five months early. + +`refresh --check` exits 3 only when the generated files change materially. +Rows the parser can't confirm show up in the diff but don't count. Otherwise a +weekly run without API keys would open the same empty PR every Monday, which is +what the first live run did. + +model-eol reports when a model stops answering and runs your eval command, but +it doesn't evaluate models itself. The reasons are in +[docs/DESIGN_EVAL_BOUNDARY.md](docs/DESIGN_EVAL_BOUNDARY.md). The eval command +runs only in a job without write access; see +[docs/BOT.md](docs/BOT.md#security-model). ## AI-assisted contributions -Much of this codebase was written by AI under human direction and adversarial -review - the commit history says which. Contributions are welcome however you -author them, under the same terms: you own what you submit, you have verified -it yourself, and it arrives with tests. "The model wrote it" explains nothing -in review; the diff stands or falls on its own. +Much of this codebase was written by AI models directed by the maintainer, with +adversarial review. When a model wrote or reviewed a change, the commit message +body says so, for example "Implemented by ; reviewed by ". +Commits don't use `Co-Authored-By` trailers for models. + +Contributions are welcome however you write them, on the same terms: you own +what you submit, you've checked it yourself, and it comes with tests. Coding +agents should also read [AGENTS.md](AGENTS.md). ## Security -If you find a vulnerability - especially in the bot's trust boundaries (feed -ingestion, plan application, the eval hook) - please use GitHub's private -vulnerability reporting rather than a public issue. Two full adversarial -review cycles are dispositioned in the issue tracker (`adversarial-review` -label); reading them first will tell you what has already been considered. +Please report vulnerabilities privately; see [SECURITY.md](SECURITY.md). Past +adversarial reviews are tracked in issues with the `adversarial-review` label, +and reading them first shows what has already been considered. + +## License + +Contributions are licensed under the project's [MIT License](LICENSE). diff --git a/README.md b/README.md index 31f31e5..f6ed143 100644 --- a/README.md +++ b/README.md @@ -9,124 +9,99 @@ ![node](https://img.shields.io/badge/node-%3E%3D22-brightgreen) ![license](https://img.shields.io/badge/license-MIT-blue) -**A machine-readable deprecation feed format for AI models, plus the reference -tooling that turns it into a Dependabot for models.** - -![model-eol demo: retired models, distributor clocks, worst-case date](docs/img/demo.gif) - -## The problem - -On July 23, 2026, OpenAI shut down a scheduled wave of model snapshots. One of my -research workflows failed that morning with a model-not-found error. The -retirement had been documented for months. I had not seen it, because a model ID -was the one dependency my toolchain could not track. - -Other dependencies do not work this way. When an npm package is deprecated, the -warning prints at install, Dependabot opens a PR, and an advisory can fail CI. A -model's retirement date usually lives in an HTML docs page and an email. - -Ordinary software already solved this layer: endoflife.date for end-of-life -dates, OSV for vulnerability feeds. This project is the model version. The -schema comes first and the tooling second, so existing trackers can share one -format instead of each scraping the same pages. - -## What's here - -- **`SPEC.md`** - the feed format (model-eol/0.1): id, aliases, announced, shutdown, - replacement, `distributions` for per-channel lifecycles, and publisher `policy` - floors. It is small enough that a provider could serve it at - `/.well-known/model-eol.json` in an afternoon. -- **`feeds/`** - Amazon (4 entries), Anthropic (36 entries), Cohere (12 entries), Google (101 entries), Mistral (43 entries) and OpenAI (204 entries), generated from the providers' live deprecation pages plus the AWS Bedrock, Google Vertex AI, and Azure Foundry lifecycle pages, feed data generated 2026-10-19. Every - dated entry carries a source URL. Anthropic's "not sooner than" dates for - active models are included as `tentative` planning floors. -- **`check.mjs`** - zero-dependency CLI: CI gate, PR diff gate, inventory, CycloneDX - ML-BOM export, retirement schedule, alerts and badges, migration plan/apply, - and public document validation. -- **`refresh/`** - regenerates the feeds from provider pages, models endpoints, and - distributor lifecycle pages, with a semantic diff for human review. -- **`bot/`** - the Dependabot part: a scheduled GitHub workflow that maintains one - migration PR or issue per retiring model. It ships as the `model-eol-bot` - binary in the same npm package. -- **Public contract** - versioned schemas, hosted feeds, refresh health, a - publication manifest, and an Atom changelog at - [`thossullivan.github.io/model-eol`](https://thossullivan.github.io/model-eol/). - Publication requires a refresh receipt for the exact feeds on `main`, followed - by an exact-byte check of every deployed asset. - -## Try it - -One command, no install, no API keys: +model-eol finds AI model IDs in your code and tells you which ones are retired +or about to be. It runs as a CLI, a GitHub Action, and a scheduled bot that +opens migration PRs. The retirement dates come from feeds in an open format +([`SPEC.md`](SPEC.md)) that any tool can read. + +![Terminal recording: model-eol reports retired models, the same repo under Amazon Bedrock's dates, and the earliest retirement date](docs/img/demo.gif) + +## Why this exists + +On July 23, 2026, OpenAI shut down a batch of model snapshots, and one of my +research workflows started failing with model-not-found errors. The shutdown +had been announced three months earlier. I missed it because nothing in my +toolchain knew which models I used. + +Package dependencies don't have this problem. npm prints a warning when a +package is deprecated, and Dependabot opens a PR. A model's retirement date sits +on a provider's docs page and in an email to whoever owns the account. + +Ordinary software has shared data for this: endoflife.date for end-of-life +dates and OSV for vulnerabilities. model-eol is an attempt at the same thing for +models. The feed format came first, and the tooling is a reference consumer of +it. + +## Quick start + +It runs with `npx`, needs Node 22 or later, and needs no API key or account: ```sh npx model-eol path/to/your/repo --days 90 ``` -That is the whole CI gate. If you prefer not to use npm, clone and run. The -result is identical, because there are no dependencies to install either way: +Output looks like this: -```sh -git clone https://github.com/thossullivan/model-eol && cd model-eol -node check.mjs path/to/your/repo --days 90 ``` +✗ app.py:2 o3-deep-research RETIRED 2026-07-23 (92 days ago) -> gpt-5.6-sol [still answers via azure-ai-foundry until 2026-11-19] +✗ app.py:3 claude-opus-4-1-20250805 RETIRED 2026-08-05 (79 days ago) -> claude-opus-4-8 [still answers via aws-bedrock until 2027-01-08] +· app.py:4 gpt-5.6-sol no retirement scheduled -
-Every command - PR gate, distributor clocks, SBOM, badges, migration plans - -```sh -# PR gate: report only findings THIS change adds or makes newly actionable -node check.mjs check . --changed origin/main --days 90 -# the same repo judged by a distributor's clock instead of the publisher's -node check.mjs path/to/your/repo --via aws-bedrock -# inventory, and a CycloneDX 1.6 ML-BOM for your SBOM pipeline -node check.mjs inventory path/to/your/repo -node check.mjs inventory path/to/your/repo --format cyclonedx > model-bom.json -# the primary CI gate is also a schema-valid public report -node check.mjs check path/to/your/repo --json > model-eol-check.json -node check.mjs validate model-eol-check.json -# retirement schedule with the repo-level worst case -node check.mjs schedule path/to/your/repo -# GitHub Actions annotations, Markdown, or a shields.io badge -node check.mjs alert path/to/your/repo --format github -node check.mjs alert path/to/your/repo --format badge > model-eol-badge.json -# migration plan (only high-confidence direct API refs are patchable) and safe apply -node check.mjs plan path/to/your/repo --days 90 > plan.json -node check.mjs apply --plan plan.json --dry-run -# validate provider feeds, repository policy, or model-eol machine reports -node check.mjs validate feeds/openai.json .model-eol.json plan.json +model-eol: 2 finding(s) at or past the 90-day threshold ``` -
+`✗` is retired, `!` retires within the `--days` threshold, `·` doesn't retire +within the threshold, and `?` means a retirement is announced but has no date +yet. A model can retire on one channel (the publisher's own API, +or a distributor such as Amazon Bedrock, Azure AI Foundry, or Vertex AI) and +keep answering on another, so the line says where and until when. -Sample output against the test fixture (2026-08-01): +The exit code is 0 when the repo is clean, 1 when any finding is retired or +inside the threshold, and 2 for usage, feed, or scan errors, so you can use the +command directly as a CI step. A clean run that can't write its output exits 3. +When something fails, `plan` and `apply` make the safe edits for you: + +```sh +npx model-eol plan path/to/your/repo --days 90 > plan.json +npx model-eol apply --plan plan.json --dry-run ``` -✗ app.py:2 o3-deep-research RETIRED 2026-07-23 (9 days ago) -> gpt-5.6-sol -! app.py:3 claude-opus-4-1-20250805 RETIRES 2026-08-05 (4 days) -> claude-opus-4-8 -· app.py:4 gpt-5.6-sol no retirement scheduled -earliest risk: 2026-07-23 (o3-deep-research-2025-06-26) + +There is nothing to install, so a clone runs the same way: + +```sh +git clone https://github.com/thossullivan/model-eol && cd model-eol +node check.mjs path/to/your/repo --days 90 ``` -The command exits 1 when any finding is at or past the threshold, so it works -in CI as-is. The badge JSON works with the -[shields endpoint](https://shields.io/endpoint) from any URL your CI can reach. +Feed coverage: Amazon (4 entries), Anthropic (36 entries), Cohere (12 entries), Google (101 entries), Mistral (43 entries) and OpenAI (204 entries), generated from the providers' live deprecation pages plus the AWS Bedrock, Google Vertex AI, and Azure Foundry lifecycle pages, feed data generated 2026-10-19. + +## Commands -**No API keys, no accounts.** Scanning is static analysis and the feeds are data -files. The checker, PR gate, schedule, SBOM export, and badge run with no -credentials at all. The bot needs a GitHub token, and a fine-grained PAT if you -want its PRs to trigger checks. Provider API keys are optional in exactly two -places: your own eval command, and the models-endpoint coverage in the feed -refresh. Refresh accepts `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` -(or `GOOGLE_API_KEY`), `MISTRAL_API_KEY`, and `COHERE_API_KEY`. +| Command | What it does | +|---|---| +| `check` (default) | Fails on retired and retiring models. `--changed origin/main` reports only what a change adds. | +| `inventory` | Lists every model reference. `--format cyclonedx` writes a CycloneDX 1.6 ML-BOM with one component per model and channel. | +| `schedule` | Lists retirement dates and the earliest one in the repo. | +| `alert` | Writes GitHub annotations (`--format github`), Markdown, or a [shields.io endpoint](https://shields.io/endpoint) badge. | +| `plan` / `apply` | Builds a migration plan and applies it. `apply --dry-run` previews the edits. | +| `validate` | Checks a feed, a `.model-eol.json`, or any model-eol report against its schema. | +| `verify` | Checks a receipt's claims against files you supply. See [docs/RECEIPTS.md](docs/RECEIPTS.md). | + +The flags you'll use most are `--json`, `--days N`, `--scope all|direct`, and +`--via DISTRIBUTOR`. `npx model-eol --help` lists which command takes which. -Both command-line binaries ship in one zero-dependency npm package. The test -suite packs that package, installs it into an empty directory with the network -disabled, and runs both binaries. The suite also fails if any runtime, -development, peer, or optional npm dependency is added. +`plan` proposes only plain string swaps. The reference must be a direct API +call the scanner is sure about, and the feed must name a replacement that is +itself safe. Other retired or retiring references are left for manual review, +and the bot files those as issues. -## Use it as a GitHub Action +Feed updates ship as patch releases, so `npx model-eol` and the `@v0` Action +tag pick up new dates with each release (see [docs/FEEDS.md](docs/FEEDS.md) for +when releases happen). Pinning an exact version, such as `npx model-eol@0.10.2` +or `thossullivan/model-eol@v0.10.2`, also pins the feeds to that release. -The same gate as a composite action. No npm install, feeds bundled, pinned to -the `v0` line: +## GitHub Action ```yaml - uses: actions/checkout@v7 @@ -144,571 +119,188 @@ the `v0` line: scope: direct ``` -This is the PR gate. It reports the findings *this* change adds or makes newly -actionable, and fails when one is retired or retiring within the threshold: a -model on an added line, or an unchanged reference the change makes bad (a -`.model-eol.json` edit, nearby SDK or cloud code, a rename, an in-repo feeds -change, an untracked file). References already bad at the base ref stay out of -the way. Add `via: aws-bedrock` (or `azure-ai-foundry`) to judge by a -distributor's clock. - -A copy-paste workflow with both gates, the PR diff and a weekly full-repository -check, is in -[`examples/workflows/model-eol.yml`](examples/workflows/model-eol.yml). The -Action also exposes `inventory`, `schedule`, `alert`, `plan`, and `validate`. -Use `format: cyclonedx` for an ML-BOM. Use `output-file` to keep any report as -a workflow artifact while the same output still prints in the job log. - -CycloneDX exports contain one machine-learning component per canonical model -and lifecycle channel. A direct publisher reference (`publisher-direct`), an -Azure reference, and a Bedrock reference to the same model therefore keep their -own status and shutdown date, under deterministic channel-qualified `bom-ref` -values. Occurrences are sorted and attached to the component whose clock judged -them. - -## One repository policy - -The CLI, the Action, and the bot all read a strict `.model-eol.json` at the -root of the target Git repository. CLI flags and Action inputs override -configured values. `--config FILE` selects an explicit policy when discovery -is ambiguous: +This is the PR gate. It fails when the change adds a reference to a retired or +retiring model. It also fails when the change makes an existing reference bad, +for example by editing `.model-eol.json`, renaming a file, or adding SDK code +next to an old model ID. References that were already bad on the base branch +don't fail the PR; the weekly full scan in the example workflow catches those. + +[`examples/workflows/model-eol.yml`](examples/workflows/model-eol.yml) has both +jobs ready to copy. The Action also runs `inventory`, `schedule`, `alert`, +`plan`, and `validate`. Set `via: aws-bedrock` to use a distributor's dates, +`format: cyclonedx` for an ML-BOM, and `output-file` to save a report as a +workflow artifact. + +## Configuration + +The CLI, the Action, and the bot all read `.model-eol.json` from the root of the +repository: ```json { "days": 90, "scope": "direct", - "via": null, - "ignore": { - "models": ["o3-deep-research"], - "paths": ["test/fixtures/**", "vendor"] - }, + "ignore": { "paths": ["test/fixtures/**", "data/catalog"] }, "waivers": [ { "model": "gpt-4", "paths": ["services/legacy/**"], - "via": "aws-bedrock", "reason": "Vendor contract pins this model until the Q1 migration.", "owner": "@platform-team", "expires": "2026-12-31" } - ], - "issues": { "enabled": true }, - "eval": { "command": "node scripts/model-eol-eval.mjs" } + ] } ``` -Unknown keys and invalid values are errors. A canonical model ID or any alias -in `ignore.models` suppresses the whole alias family. `ignore.paths` takes -repo-relative `*`, `**`, and `?` globs and excludes matching files before scan -coverage is counted. - -With no config, the CLI keeps its historical `scope: all`. Once a config exists, -omitted values use the bot defaults: `days: 90`, `scope: direct`, and the -publisher clock. The portable structure is defined by -[`schema/model-eol.bot-config.schema.json`](schema/model-eol.bot-config.schema.json). -`model-eol validate` also enforces the runtime rules that Draft-07 cannot -express. - -### Ignores and waivers +`ignore` drops files and models that aren't live dependencies, such as fixtures +and vendored catalogs. A waiver keeps the finding in every report, with its +owner and expiry, but stops it failing CI until the expiry date. Monorepos can +set different thresholds and channels per path. Unknown keys are errors. -Ignores and waivers answer different questions. +Adding a config file changes one default. Without one, the CLI also fails on +cloud and gateway references (`scope: all`). With one, any key you leave out +takes the bot's defaults: 90 days, `scope: direct`, and the publisher's dates. -`ignore` hides inventory noise: fixtures, caches, and vendored catalogs that are -not live dependencies. Ignored references do not appear in reports. - -A **waiver** records a deliberate exception on a live dependency: which model, -why, who owns it, and the date it expires. A waiver never hides its finding. -The reference stays in `check`, `inventory`, `schedule`, `alert`, `plan`, and -the CycloneDX export, with its reason, owner, expiry, and active state attached. -The `waiver` field appears on a finding only when a waiver matched it: - -``` -✗ app.py:2 o3-deep-research RETIRED 2026-07-23 (42 days ago) -> gpt-5.6-sol [waived until 2026-12-31 by @research-platform: Research pipeline pinned until the Q4 rerun.] -``` +[docs/CONFIGURATION.md](docs/CONFIGURATION.md) covers every key, monorepo +overrides and routes, and what the scanner reads and skips. -While a waiver is active, that finding is left out of the exit-1 total, the -badge counts, and the alert errors, and the bot opens no migration work for it. -On the expiry date the finding is actionable again, and the report names the -waiver that lapsed and who owned it. Expiry uses UTC dates, and the expiry date -itself counts as expired. +## How retirement dates work -The waiver fields work like this: +### One model, several dates -- `model` matches the whole alias family. -- `paths` takes the same repo-relative globs as `ignore.paths`. -- `via` narrows the waiver to one clock: a distributor id, or `publisher` for - the direct publisher clock. Omit it to waive every clock. -- The first active matching waiver wins. -- A policy holds at most 500 waivers across the root and its overrides. +A model can retire on different dates depending on where you call it. +`claude-opus-4-1` retired on Anthropic's API on August 5, 2026, but Amazon +Bedrock serves it until January 8, 2027. `o3-deep-research` left OpenAI's API on +July 23 and stays on Azure AI Foundry until November 19. -A malformed waiver, such as a missing owner, a relative duration, or an unknown -key, is a config error like any other. An expired waiver is not an error. It is -simply no longer active. The CycloneDX export records each component's waiver -state as active, partial, or expired. +The feed records each channel's date under `distributions`. Pass +`--via aws-bedrock`, `--via azure-ai-foundry`, or `--via vertex-ai` to check the +repo against the channel it actually calls. If the feed has no entry for that +channel, the checker uses the publisher's date and labels the line +`publisher-fallback`. -### Coverage limits +The later date buys time for the migration, and it is also the last chance to +record the old model's outputs if your eval compares against them. When a +retired or retiring model still answers on another channel, `check` adds +`[still answers via CHANNEL until DATE]`, and the bot writes a "Capture window" +section in its PRs and issues. [docs/DESIGN_EVAL_BOUNDARY.md](docs/DESIGN_EVAL_BOUNDARY.md) +explains why model-eol reports this window but never runs the capture. -For content-heavy repositories, start with `inventory`. Then exclude caches, -generated catalogs, archived fixtures, and documentation that are not live -model dependencies. The published-repository tests verify both outcomes: an -unconfigured scan reports the evidence, and an explicit path policy produces a -clean check and an empty bot decision table without changing repository -content. +### Tentative dates and notice policies -Scans do not recurse into tracked Git submodules. A detected gitlink is -reported as incomplete coverage, so `check` and `plan` fail unless the path is -ignored or `--allow-incomplete` is set. To inspect a submodule, run model-eol -against its checkout as a separate target with its own policy. - -Untracked nested Git repositories work the same way. The parent reports -`nested-repository-skipped` and never recurses into the nested checkout. Ignore -an intentional nested repository, allow incomplete coverage, or scan it -separately. - -### Monorepos - -Mixed-provider monorepos keep the top-level values as defaults, then apply path -policies and a lifecycle channel per reference: - -```json -{ - "days": 90, - "scope": "direct", - "overrides": [ - { "paths": ["services/**"], "scope": "all" }, - { "paths": ["services/bedrock/**"], "days": 180, "via": "aws-bedrock" } - ], - "routes": [ - { "paths": ["services/vertex/**"], "via": "vertex-ai" }, - { - "paths": ["services/bedrock/**"], - "match": "chat-prod", - "model": "claude-3-7-sonnet-20250219", - "via": "aws-bedrock" - } - ] -} -``` - -Matching overrides apply in order. Later scalar values win. Ignore and waiver -lists add up, so an override can carry path-scoped `waivers` with the same shape -as the root. The last matching route wins. An exact `match`/`model` pair -resolves a deployment alias without making cloud or gateway references -patchable. Explicit CLI flags and Action inputs still take precedence. - -JSON inventory and plan output record each reference's effective threshold, -scope, requested channel, and matching rule indexes. A Bedrock service and a -direct publisher call in the same repository therefore never share one global -clock by accident. - -## One model, several retirement dates - -The same model retires on different dates on different channels. A checker that -ignores this is wrong in both directions. o3-deep-research retired at OpenAI on -July 23 but stays available on Azure AI Foundry until December 26. -claude-opus-4-1 retires at Anthropic on August 5 but stays on Amazon Bedrock -until January 8, 2027. - -`distributions` in the spec carries these per-channel dates. `--via ` -judges your repo by the channel you call. The distributor refresh keeps the -Bedrock, Vertex, and Azure Foundry dates current from their lifecycle pages. -Bedrock combines its [legacy table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html) -with [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html). -Card floors become tentative distribution dates without announcements. -Under `--via aws-bedrock`, these floors remain `scheduled` and display "not sooner than". -A distributor's later date gives you more time for the same migration. It is not a reason to -skip it. - -The later date is also the last day you can still call the old model, when the -feed gives it as an exact date. A tentative or earliest floor is "at least until", -and the old model may answer past it. When a retired or retiring model still -answers on another channel, `check` appends -`[still answers via aws-bedrock until 2026-10-14]` to the line, and the bot adds -a `## Capture window` section to its PRs and issues. If your eval compares the -replacement's outputs against the old model's, capture that baseline before the -window closes. model-eol reports the window from the feeds and never runs the -capture; see [docs/DESIGN_EVAL_BOUNDARY.md](docs/DESIGN_EVAL_BOUNDARY.md) for -where that line sits and why. - -## Policy floors - -A feed entry with no `announced` and no `shutdown` means "nothing scheduled as -of `generated`". That is a positive statement, and it is one that scraping HTML -cannot make on its own. - -Publishers with a stated minimum notice period carry a `policy` floor. Anthropic -states at least 60 days. The schedule uses that floor to compute a -no-earlier-than planning date from the feed refresh. This is stated policy, not -a contract. OpenAI publishes no formal notice period, so its feed makes no -forward claim. - -Anthropic also lists a tentative retirement date, "not sooner than", for every -active model. The feed carries those as `shutdown` with -`date_precision: "tentative"` and no `announced` date, because nothing has been -announced. The checker treats a tentative date as a floor, not a schedule. The -finding is `scheduled`, never `retiring` or `retired`. `safe_until` is the later -of the floor and the policy date. A tentative floor cannot fail CI, and the bot -opens no work for it. It does appear in every schedule, so planning can start -before the announcement: +Anthropic lists a "not sooner than" retirement date for every active model. +The feed records these with `date_precision: "tentative"` and no `announced` +date. The checker shows them in schedules so you can plan ahead, but a tentative +date never fails CI and the bot opens nothing for it: ``` -· app.py:1 claude-sonnet-4-5-20250929 scheduled - tentative, not announced: not sooner than 2026-09-29, guaranteed until 2026-10-17 by anthropic policy +· not sooner than 2026-10-15 app.py:1 claude-haiku-4-5-20251001 scheduled - tentative, not announced: not sooner than 2026-10-15, guaranteed until 2026-12-17 by anthropic policy [model-reference] [safe until 2026-12-17] ``` -When Anthropic announces the retirement, the announcement wins. The entry -becomes an ordinary exact date with an `announced` date. +The "guaranteed until" part comes from the publisher's stated minimum notice +period, which the feed carries as `policy`. Anthropic states 60 days, so the +date is the feed's `generated` date plus 60 days. Anthropic can change that +policy, and the line reports it as stated. OpenAI states no fixed notice +period, so its feed has no `policy`. -`earliest` is different. Google's "earliest possible" shutdown dates are -announced deprecations, and the checker treats them as the scheduled date. +Google's "earliest possible" shutdown dates are real announcements. The feed +marks them `earliest`, and the checker uses them as the shutdown date, so they +can fail CI. -A repository that uses many active Claude models gets one floor line per -reference. To read only announced retirements, filter on `date_precision`: +To list only announced retirements: ```sh -node check.mjs schedule path/to/your/repo --json \ +npx model-eol schedule path/to/your/repo --json \ | jq '.items |= map(select(.date_precision != "tentative"))' ``` -## Direct-first inventory +### Direct calls versus clouds and gateways -Direct API usage, meaning OpenAI, Anthropic, Gemini, or similar SDK calls with -real model IDs, is scanned and checked immediately. Cloud providers and -gateways such as Azure Foundry, Bedrock, Vertex, OpenRouter, LiteLLM, and -Portkey often hide the model behind a deployment name or a routing alias. Those -references stay in the inventory as resolver targets, because static analysis -cannot know which model is deployed behind them: +When code calls OpenAI, Anthropic, Gemini, or another publisher SDK with a real +model ID, the scanner can check it. Azure AI Foundry, Bedrock, Vertex AI, +OpenRouter, LiteLLM, and Portkey often hide the model behind a deployment name +or a route alias, and static analysis can't see what is deployed behind +`chat-prod`. Those references appear in `inventory` and `schedule` as `?` +lines: ``` -? src/ai.ts:12 azure-ai-foundry Azure Foundry/OpenAI deployment reference -? src/ai.ts:30 openrouter OpenRouter gateway reference +? src/ai.ts:12 azure-ai-foundry Azure Foundry/OpenAI deployment reference (needs provider/gateway resolver) +? src/ai.ts:30 openrouter OpenRouter gateway reference (needs provider/gateway resolver) ``` -Use `--scope direct` to fail CI only on what static analysis can prove. -Model-like strings that are absent from the feeds are reported as non-failing -**candidates**, so feed gaps stay visible without false alarms. - -The scanner covers code and configuration files. The list in -[`CODE_EXT`](lib/scanner.mjs) is the source of truth: Python, -TypeScript/JavaScript, JSON/YAML/TOML, shell, Ruby, Go, Java, C#, Rust, C/C++, -PHP, Swift, Kotlin, Terraform, SQL, Scala, and Objective-C, including `.baml` -sources. Publisher-compatible BAML provider/model pairs can be plan targets. -Generated BAML clients and sources marked `DO NOT EDIT` or `@generated` are -skipped. Model-eol's own JSON reports are skipped on reruns. Third-party -CycloneDX files are scanned unless they carry explicit model-eol generator -provenance. Terraform files (`.tf`, `.tfvars`) are scanned, and `.terraform` -directories are excluded. +`--scope direct` fails CI only on references the scanner can prove. The bot +still opens one issue for each cloud or gateway provider it finds, so someone +checks which model is behind it. If you know, a `routes` entry in the config +maps an alias to a model. A routed reference still counts as cloud or gateway +usage, so it fails CI only where `scope` is `all`. Strings that look like model +IDs but aren't in any feed are listed as candidates and never fail. ## The bot -Copy `bot.yml.example` to `.github/workflows/model-eol-bot.yml` for a weekly -run. It maintains exactly one labelled migration PR per retiring model, with -the full feed context in the body and never auto-merged, and one issue per -finding that needs a human. - -The workflow resolves `model-eol@0` from npm once and uses that exact version -across its plan, read-only evaluation, and write-token publication jobs. -Consumer repositories do not copy `check.mjs` or the `bot/` source directory. - -Dismissals are respected. A reintroduced retired model creates fresh work. -Bot-owned PRs and issues close when their finding disappears or is waived. When -a waiver expires, the bot treats the finding as new rather than inheriting the -earlier dismissal. +The bot is a scheduled workflow that keeps one PR open per retiring model, with +the dates, sources, and replacement in the PR body. It never merges. Findings it +can't patch safely, and cloud or gateway references it can't resolve, become +issues. Closing a PR without merging dismisses that +retirement, and the bot won't reopen it unless the shutdown date or the +replacement changes. -In a repository without a configured eval, preview everything with no GitHub -calls and no project dependency: +To set it up, copy [`bot.yml.example`](bot.yml.example) to +`.github/workflows/model-eol-bot.yml`. To see what it would do first, run this +in your repository: ```sh npx --yes --package=model-eol@0 model-eol-bot --dry-run --target-dir . --repo OWNER/REPO ``` -The workflow honors configured `days`, `scope`, and `via` values. Without a -config it uses the bot defaults of 90 days and direct references. `eval.command` -also comes from the config, unless the `MODEL_EOL_EVAL_COMMAND` repository -variable is set as an explicit override. - -Each patchable model is applied in an isolated checkout and evaluated on its -own, with its own old and new IDs, timeout, bounded report, and explicitly -forwarded environment names. The result manifest is bound to its content and -pinned to the evaluated Git commit. A passing migration can proceed while a -failing one stays blocked, and nothing is published against a newer, -unevaluated default branch. - -### Prove the swap works in your repository - -Model behavior is specific to your repository, so model-eol never guesses which -test is good enough to authorize a migration. Check a harness into the -repository and name it in `.model-eol.json`. A non-empty -`MODEL_EOL_EVAL_COMMAND` repository variable overrides that command. Otherwise -the config is authoritative. With neither configured, the result manifest -records that evaluation was not configured. - -Start by copying [`examples/model-eol-eval.mjs`](examples/model-eol-eval.mjs) to -`scripts/model-eol-eval.mjs`. Replace its `VERIFY` command with the smallest -deterministic test that proves the behavior your application relies on. A good -harness checks response or tool-call shape, structured-output parsing, required -capabilities, and a small set of golden tasks. A passing build is supporting -evidence, but it does not prove model behavior by itself. - -The starter calls a dedicated `npm run eval:model-eol` script, so adopting it -requires an explicit repository test. That prevents an unrelated unit suite from -being treated as an LLM-behavior eval by accident. Point `eval:model-eol` at the -behavior test, not back at the harness. - -The harness runs after the proposed migration, in an isolated checkout, and -receives: - -- `MODEL_EOL_OLD_ID` and `MODEL_EOL_NEW_ID` for the current migration. -- `MODEL_EOL_PLAN`, a one-model plan containing the exact changed references. -- `MODEL_EOL_REPORT`, the required path for a bounded, publishable Markdown receipt. -- `MODEL_EOL_VIA`, the distributor channel the plan was built for, or empty for the publisher's direct API. Use it to map the publisher's replacement ID to your client's ID on that channel. -- `MODEL_EOL_EVAL_MODE`, always `evaluate` today. The value `capture` is reserved for a future run on the unpatched checkout against the old ID. The bot does not run it, and the hook refuses any other value. - -Exit zero plus a regular report file means pass. Any other exit, a timeout, a -missing report, a tracked workspace change, or checkout drift fails that -migration. Test stdout and stderr are not published. Put only non-secret -evidence in the report. Forward only the credentials the eval code needs, -through `eval.pass_env`. That allowlist controls normal subprocess environment -forwarding. It is not an OS sandbox and does not isolate code running as the -same user. The security boundary is that the evaluation job has no repository -write token, and the publication job never runs repository-owned eval code. - -Before enabling the scheduled bot, commit the harness and policy on a clean -branch and run the same isolated evaluator locally: - -```bash -set -euo pipefail -repo="$(pwd -P)" -version="$(npm view model-eol@0 version --json | node -e 'let s="";process.stdin.on("data",c=>s+=c).on("end",()=>{const v=JSON.parse(s);const r=Array.isArray(v)?v.at(-1):v;if(typeof r!=="string")process.exit(1);process.stdout.write(r)})')" -tmp="$(mktemp -d)" -trap 'rm -rf -- "$tmp"' EXIT -cd "$tmp" -if [ -f "$repo/.model-eol.json" ]; then - MODEL_EOL_UAT_REPO="$repo" npx --yes --package="model-eol@$version" \ - -c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan .' > "$tmp/plan.json" -else - MODEL_EOL_UAT_REPO="$repo" npx --yes --package="model-eol@$version" \ - -c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan . --days 90 --scope direct' > "$tmp/plan.json" -fi -npx --yes --package="model-eol@$version" model-eol-bot evaluate \ - --target-dir "$repo" \ - --plan-file "$tmp/plan.json" \ - --output-file "$tmp/eval.json" -node -e 'const r=require(process.argv[1]); console.log(r.results)' "$tmp/eval.json" -MODEL_EOL_EVAL_RESULTS_FILE="$tmp/eval.json" \ - npx --yes --package="model-eol@$version" model-eol-bot \ - --dry-run --target-dir "$repo" --repo OWNER/REPO -``` - -Each planned model is patched and evaluated on its own. The cleanup trap -removes the temporary files when the shell exits. Copy `eval.json` elsewhere -after the dry run if you want to keep it as evidence. Then let -`bot.yml.example` run the same contract, with any provider keys you opt in -available only to its read-only evaluate job, never to its publish job. - -Inline `model-eol-bot --eval` and the older unbound report and status artifacts -are refused. A configured publication must consume the commit-bound manifest -from `model-eol-bot evaluate`. Missing results fail before any GitHub API call. - -On its first actionable run, the bot creates the `model-eol` label. If -repository policy prevents label creation or assignment, the bot stops rather -than publish work whose ownership cannot be verified. Existing work is trusted -only when the label, metadata, deterministic branch, repository, base, and Git -lease all agree. For HTTPS remotes, including private repositories, the bot -passes `GITHUB_TOKEN` to temporary Git processes as a host-scoped authorization -header and never puts it in a remote URL. - -Two operational notes. PRs created with `GITHUB_TOKEN` do not trigger -`pull_request` workflows; store a fine-grained PAT or GitHub App token as the -optional `MODEL_EOL_BOT_TOKEN` secret when checks must run. And the workflow -splits privileges: only the provider secrets you add for the eval live in the -read-only evaluation job, and write tokens live only in the reconciliation job. - -## Keeping the feeds honest - -```sh -node refresh/refresh.mjs --check # semantic diff vs live pages; exit 3 = PR-worthy -node refresh/refresh.mjs --distributor aws-bedrock,vertex-ai,azure-ai-foundry # distributor lifecycle clocks -node scripts/feed-changelog.mjs # local rendering of the hosted Atom feed -``` - -A parse failure stops the run. The refresh never writes a guessed feed. - -Anthropic reads model IDs and aliases from overview pages listed in `https://platform.claude.com/docs/llms.txt`. -It reads at most 100 unique pages with 30-second deadlines and 8 MiB response limits. -Only the deprecations page supplies Anthropic dates. -Offline checks use `anthropic-llms.txt` and `anthropic-model-pages/.md`; missing pages stop refresh. -Alias attachments and moves emit notices and appear in semantic diffs. - -AWS changed its Bedrock lifecycle policy on 2026-09-07. -The legacy table covers models launched before that date; newer models publish lifecycle dates on individual cards. -The Bedrock refresh reads both sources and keeps legacy-table records when IDs overlap. -Conflicting exact dates appear under "Source conflicts"; the legacy table wins. -When two card floors differ, the later day-precision date applies. -Cards without lifecycle fields or day-precision dates appear under "Model cards without lifecycle fields" and emit notices. -The crawl reads at most 400 unique cards sequentially, with 30-second deadlines and 8 MiB response limits. -Offline checks use `bedrock-lifecycle.html`, `bedrock-model-cards.html`, and `bedrock-model-cards/` fixtures. -Missing card fixtures stop the refresh. - -This runs automatically. A weekly workflow -(`.github/workflows/feed-refresh.yml`, Mondays 05:23 UTC) checks the live -sources and opens a PR with the semantic diff, plus an updated README freshness -line, whenever something material changed. If an earlier refresh PR is still -open and no human has touched it, the run updates that PR instead of opening -another one. - -The first live runs found real errors. They caught the hand-compiled feeds -drifting from Anthropic's recommendations within one week, and corrected a -hand-compiled Bedrock date that was three months wrong. - -Every successful live refresh produces a byte-exact receipt. The Pages workflow -publishes the [public contract](https://thossullivan.github.io/model-eol/) only -when that receipt matches the exact feeds on `main`. A no-change check advances -`last_checked` immediately. A material-change run waits for its feed PR to -merge. The feed's `generated` date changes only when source data changes. The -hosted contract contains canonical Draft-07 schemas, feeds with SHA-256 -receipts, and the Atom changelog. - -The refresh also removes dates. On August 3, Google removed the October 16 -earliest shutdown dates it had listed for the Gemini 2.5 Pro, Flash, and -Flash-Lite GA models. The pipeline opened -[a PR with the semantic retraction](https://github.com/thossullivan/model-eol/pull/60), -and downstream checks stopped warning about dates Google no longer listed. A -lifecycle tracker has to retract stale alarms as well as add new ones. - -## Receipts - -model-eol writes three receipt kinds. Each one is a public Draft-07 schema, -and `model-eol validate receipt.json` selects the schema from the receipt's -`schema` value. - -| Kind | `schema` value | Binds | -|---|---|---| -| [Eval results manifest](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.eval.schema.json) | `model-eol.eval/0.1` | One pass, fail, or timeout result per migration group to the evaluated base commit, the plan digest, the eval settings digest, and each group's feed digest. Written by `model-eol-bot evaluate`. | -| [Feed-refresh receipt](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.feed-refresh-receipt.schema.json) | `model-eol/feed-refresh-receipt@0.1` | The refresh run URL, the checked commit, and the check time to the SHA-256 of every `feeds/*.json` file. A pending receipt also names the open feed PR. | -| [npm release result](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.npm-release-result.schema.json) | `model-eol/npm-release-result@0.2` | The release workflow's source commit to either no release, or one exact version, its tag, the release commit, and the registry tarball's sha512 integrity. | - -Additive changes keep a receipt's version. Any change that makes an old -receipt invalid, or changes what a digest covers, is a new version. Validators -reject unknown versions. A few rules, such as feed sort order and `tag` equal -to `v` plus `version`, cannot be written in Draft-07. Each schema's -`description` lists them, and `model-eol validate` enforces them in code. - -`model-eol validate` checks a receipt's structure only. `model-eol verify` -checks what a receipt claims against inputs you supply: - -```bash -# release result against the exact tarball you hold (no registry lookup) -npm pack model-eol@0.8.4 -model-eol verify npm-release-result.json --package model-eol-0.8.4.tgz \ - --commit RELEASE_SHA --source-commit SOURCE_SHA --version 0.8.4 - -# refresh receipt against a checkout's feeds directory -model-eol verify feed-refresh-receipt.json --feeds-dir ./feeds \ - --commit REFRESH_SHA --run-url https://github.com/OWNER/REPO/actions/runs/ID - -# eval manifest against the plan, config, and feeds the evaluator used -model-eol verify eval-results.json --commit BASE_SHA --plan plan.json \ - --config .model-eol.json --feeds-dir ./feeds -``` - -The receipt kind comes from its `schema` value. verify validates the receipt's -structure first, then reports each claim. It never reads the current -directory, git state, or the network; every input is an explicit flag. - -| Flag | Checks | -|---|---| -| `--feeds-dir DIR` | Refresh: the SHA-256 of every `DIR/*.json`, compared with the receipt's `feeds` list as a set. Changed, missing, and extra files are each a mismatch. `DIR` is the feeds directory itself; a repository root (a directory holding `feeds/`) is refused. Eval: with `--plan` and `--config`, recomputes each result's `feed_digest` (`--plan` and `--config` alone already fix the expected result set). | -| `--commit SHA` | Refresh `refresh_commit`, release `release_commit`, or eval `base_sha`. | -| `--run-url URL` | Refresh `refresh_run`, after the same HTTPS URL normalization the receipt writer applies. | -| `--package FILE.tgz` | Release `registry_integrity`: the sha512 SRI of the tarball bytes. | -| `--source-commit SHA` | Release `source_commit`. | -| `--version X.Y.Z` | Release `version`; `tag` is bound to it as `vX.Y.Z`. | -| `--plan FILE` | Eval `plan_digest`, from a validated `model-eol.plan/0.1` document. | -| `--config FILE` | Eval `eval_config_digest` and `configured`, with the bot's settings resolution. `MODEL_EOL_EVAL_COMMAND` in the environment overrides `eval.command` as it does for the bot, and the output says so. For a repository without `.model-eol.json`, pass a file containing `{}`. | -| `--require-all` | Exit 1 when the result is `partial`. | -| `--json` | Emit a [`model-eol/verification@0.1`](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.verification.schema.json) document. | - -SHA inputs are 40 to 64 hex characters; uppercase is lowercased. A flag that -does not apply to the receipt's kind is a usage error. - -Each claim is `verified`, `mismatch` (with the receipt's value and the -computed one), `not-checked` (with the flag that would check it), or -`not-applicable` (a `released: false` result makes no release claims, so only -`source_commit` applies). Supplying `--commit`, `--version`, or `--package` for a -`released: false` result adds a `released` claim that mismatches, because those -inputs assert a release the receipt says did not happen. For a refresh receipt, -the report also shows its `state`, `checked_at`, and `pending_pr` as information; -a pending receipt verifies by the same rules as a clean one. For a release -result the report shows `released`; a release gate must pass `--package`, -`--version`, `--commit`, and `--require-all`, so that a no-release receipt -mismatches instead of verifying through `--source-commit` alone. - -For a configured eval manifest, `--plan` and `--config` fix the set of result -identities the evaluator had to produce: a planned identity with no result, a -result for an unplanned identity, and a duplicated identity are each a -mismatch, and `--feeds-dir` additionally checks each result's `feed_digest`. -Without `--plan` and `--config`, a `results` claim is reported as not checked, -so stripping results from a manifest never verifies. Text output escapes -control characters in every printed value. The overall result is `verified` only when every -applicable claim is verified. It is `partial` when nothing mismatched but at -least one claim was not checked: the receipt is consistent with what you -supplied, not proven. It is `mismatch` when any claim mismatched. - -Eval outcomes are never claims. `verify` does not check `status`, `exit_code`, -`report`, or `generated`, because an evaluation cannot be reproduced offline. A -`verified` eval manifest is bound to the supplied commit, plan, config, and -feeds. It does not mean the evals passed. A gate that needs passing evals must -read the results itself. - -| Exit | Meaning | -|---|---| -| 0 | No mismatch: `verified`, or `partial` without `--require-all`. | -| 1 | At least one claim mismatched, or `partial` with `--require-all`. | -| 2 | Usage error, unreadable file, structurally invalid receipt, or unsupported receipt schema or version. No claim is reported. | -| 3 | Output write failure when the result would otherwise exit 0. | - -## The actual ask - -The goal is the format, not another checker. Providers already publish -deprecation *pages* because customers asked for them. A `retirement_date` field -in the models endpoint, or this feed at a well-known URL, would let every -existing tracker read the same data. endoflife.date now tracks Claude with -nearly this field set, so the convergence has started. If you maintain one of -the community trackers, agreeing on a shared schema, this one or a better one, -is the most useful thing any of us can do here. - -## Status - -- Feeds refresh automatically. The weekly feed-refresh workflow is live in this - repo. Parse failures fail the run, and material changes become a reviewed PR. - CI runs the full suite on every push and PR. The copy-ready bot workflow ships - as `bot.yml.example` and uses the published package, not local copies of the - tooling. -- The [public contract](https://thossullivan.github.io/model-eol/) publishes - canonical schemas, feeds, refresh health, a publication manifest, and Atom. - Each deployment is bound to an exact feed-refresh receipt and verified against - every live asset. `model-eol validate` checks feeds, repository policy, check - reports, inventories, schedules, alerts, plans, and receipts against the same - zero-dependency runtime contracts. -- Current-model entries, and therefore policy-floor horizons, populate only when - the refresh runs with provider API keys for the models endpoints. Anthropic's - tentative dates come from its public deprecations page and need no key. -- The checker matches known IDs only. It does not discover models that are - absent from the feeds. This is deliberate: a CI gate needs precision more than - discovery. -- Fetchers: OpenAI, Anthropic, Google, Mistral, Cohere, aws-bedrock, vertex-ai, - and azure-ai-foundry are live. xAI publishes no lifecycle index and its models - API carries no deprecation fields, so xAI is not ingested. -- On npm as [`model-eol`](https://www.npmjs.com/package/model-eol). Material - feed changes republish automatically as patch versions, with trusted - publishing and provenance. Code releases require an explicit stable version. - `npx model-eol` therefore always checks against current dates. -- Not yet: feed signing, and authenticated account-level gateway or cloud - resolvers. +A model swap can change behavior, so the bot can run a test command of yours on +each migration before it opens the PR. That command runs in a job with no write +access, and the job that writes to GitHub never runs your code. +[docs/BOT.md](docs/BOT.md) covers the eval hook, tokens, and the security model. + +## Where the data comes from + +The publisher feeds are generated from each publisher's deprecation page and +model listings, plus the Amazon Bedrock, Vertex AI, and Azure AI Foundry +lifecycle pages, and each dated entry links to the page it came from. The +files are not edited by hand. (`amazon.json` is the exception: its entries are +carried forward, and only their Bedrock dates are refreshed.) + +A weekly workflow regenerates the feeds and opens a PR whenever something +material changes, with a semantic diff of dates that moved, models that were +added, and replacements that changed. Withdrawn dates are removed too +([example](https://github.com/thossullivan/model-eol/pull/60)). + +The feeds, the schemas, and an Atom changelog are published at +[thossullivan.github.io/model-eol](https://thossullivan.github.io/model-eol/). +[docs/FEEDS.md](docs/FEEDS.md) describes the refresh and publication pipeline. + +## For providers and tracker maintainers + +What I'd most like is for providers to publish this data themselves: a +`retirement_date` field in the models endpoint, or a feed at +`/.well-known/model-eol.json`. [`SPEC.md`](SPEC.md) is short enough to implement +in an afternoon. endoflife.date already tracks Claude with nearly the same +fields. If you maintain a model tracker, I'd rather we agree on one schema, this +one or a better one, than keep scraping the same pages separately. + +## Limits + +- The checker only knows the models in the feeds. A model missing from every + feed can't fail CI until a feed adds it. +- model-eol doesn't query cloud accounts or gateways to find out which model is + deployed; that is out of scope + ([#79](https://github.com/thossullivan/model-eol/issues/79)). Those + references stay in the inventory, and `routes` can map known aliases. +- xAI publishes no lifecycle index and its models API has no deprecation fields, + so there is no xAI feed. +- New current models (entries with nothing scheduled) come from the OpenAI, + Google, Mistral, and Cohere models endpoints, so the refresh adds them only + when it has API keys. Without keys it keeps the entries already in the feeds. +- Feeds are not signed yet. ## Contributing -The most valuable contribution is a feed correction with a source URL. The one -rule is that `feeds/*.json` are generated, never hand-edited. See -[CONTRIBUTING.md](CONTRIBUTING.md) for that workflow, the zero-dependency rule, -and what a new publisher parser needs. +The most useful contribution is a feed correction with a source URL. Feed files +are generated, so a correction usually means fixing a parser. +[CONTRIBUTING.md](CONTRIBUTING.md) has the details. diff --git a/SECURITY.md b/SECURITY.md index 2af7a0a..570c289 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,33 +2,36 @@ ## Reporting -Report vulnerabilities privately via -[GitHub private vulnerability reporting](https://github.com/thossullivan/model-eol/security/advisories/new) - -please don't open a public issue for a security report. Expect a first -response within a week. +Please report vulnerabilities through +[GitHub private vulnerability reporting](https://github.com/thossullivan/model-eol/security/advisories/new), +not a public issue. You should get a first response within a week. ## Supported versions -The latest 0.x release only. There is no backport branch; fixes ship as the -next patch on npm and move the `v0` action tag. +Only the latest 0.x release. There are no backport branches. Fixes ship as the +next patch on npm and move the `v0` Action tag. ## Scope -The surface that matters, given zero runtime dependencies: - -- The scanner runs against untrusted repository content, so path handling and - output escaping (GitHub annotations, badge JSON) are in scope. -- Feed integrity: `feeds/*.json` are generated from provider pages, and a - poisoned feed changes CI verdicts downstream. Feed signing is on the - roadmap, not shipped. -- The bot workflow's privilege split: the plan job is read-only, the evaluate - job receives only provider secrets explicitly needed by trusted eval code and - has no write token, and the publish job receives the write token but never - executes repository-owned eval code. `eval.pass_env` controls normal - subprocess forwarding; it is not an OS sandbox or same-user secret-isolation - boundary. Anything that bridges the evaluate/publish privilege split is a - finding. - -The checker itself needs no credentials: scanning is static analysis, so a -report that assumes an API key inside `check.mjs` is out of scope by -construction. +model-eol has no runtime dependencies, so the attack surface is its own code: + +- **The scanner reads untrusted repository content.** Path handling and output + escaping are in scope: text output and stderr (no Actions workflow commands), + JSON printed to logs, Markdown alerts, GitHub annotations, badge JSON, and the + bot's PR and issue bodies (no live mentions, references, or links from + repository, config, or feed text). +- **Feed integrity.** The feeds are generated from provider pages, and a + poisoned feed changes CI results for everyone downstream. Feeds are not + signed yet. +- **The bot's privilege split.** The plan job is read-only. The evaluate job + runs repository-owned eval code, gets only the provider secrets you add, and + has no write token. The publish job has the write token and never runs + repository-owned eval code. Anything that bridges the evaluate/publish + privilege split is a finding. `eval.pass_env` filters the environment for + ordinary child processes; it is not an OS sandbox or a secret boundary + between processes running as the same user. + See [docs/BOT.md](docs/BOT.md#security-model). + +`check.mjs` needs no credentials and makes no API calls, because scanning is +static analysis. Reports that assume a credential inside the checker are out of +scope. diff --git a/SPEC.md b/SPEC.md index 8987697..59a1139 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,14 +1,23 @@ -# model-eol - a machine-readable model deprecation feed (public draft v0.1) +# model-eol feed format, version 0.1 -> Status: public draft, updated 2026-08-18. The problem: model retirement dates live in HTML docs -> and emails, so every deprecation checker is a scraper with a private registry -> format. Ordinary software solved this layer - endoflife.date for EOL data, OSV for -> vulnerability feeds. This is the model version: small enough that a provider could -> ship it in an afternoon, useful enough that scrapers converge on it in the meantime. +Status: public draft. -## The format +Model retirement dates are published as HTML pages and emails, so every tool +that tracks them scrapes those pages into its own private format. This document +defines a small JSON format for the same data. A publisher could serve it +directly; until then, aggregators can publish it on a publisher's behalf. -A feed is a JSON document: metadata + a list of model entries. +## Terms + +- **Publisher**: the company that makes a model and serves it on its own API, + such as OpenAI or Anthropic. +- **Distributor**: a cloud platform that serves another publisher's models, + such as Amazon Bedrock, Azure AI Foundry, or Vertex AI. +- **Channel**: either of those, as a place a model can be called. The + publisher's own API is one channel, and each distributor is another. A + distributor channel is named by its `via` value. + +## Example ```json { @@ -25,8 +34,8 @@ A feed is a JSON document: metadata + a list of model entries. "replacement": "gpt-5.6-sol", "notes": "Deep-research post-trained agent; replacement is a general model + tools, not a structural successor.", "distributions": [ - { "via": "azure-ai-foundry", "shutdown": "2026-12-26", - "source": "https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-retirements" } + { "via": "azure-ai-foundry", "shutdown": "2026-11-19", + "source": "https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule" } ] }, { @@ -36,69 +45,139 @@ A feed is a JSON document: metadata + a list of model entries. } ``` -### Fields +A feed is a JSON object with metadata and a `models` array. The JSON Schema is +[`schema/model-eol.schema.json`](schema/model-eol.schema.json). + +## Feed fields + +| Field | Required | Meaning | +|---|---|---| +| `spec` | yes | `"model-eol/0.1"`. | +| `publisher` | yes | The publisher this feed describes, for example `openai`. | +| `generated` | yes | When the feed was generated, as an ISO 8601 timestamp. | +| `source` | no | URL for the feed's claims. Entries can override it with their own `source`. | +| `note` | no | Free text about the feed as a whole. | +| `policy` | no | The publisher's stated notice period. See [Notice policy](#notice-policy). | +| `models` | yes | The model entries. | + +## Model fields + +| Field | Required | Meaning | +|---|---|---| +| `id` | yes | The exact model identifier sent over the wire. A dated snapshot is preferred. | +| `aliases` | no | Other strings that resolve to this model, such as undated aliases. | +| `announced` | no | The date the deprecation was announced. Absent means not deprecated. | +| `shutdown` | no | The date calls stop working on the publisher's own API. Absent means none scheduled. | +| `date_precision` | no | How to read `shutdown`. See below. | +| `replacement` | no | One exact model ID that is a safe drop-in string substitution. See below. | +| `replacement_options` | no | An ordered list of exact model IDs when there is more than one choice, or when the choice isn't in this feed. | +| `replacement_note` | no | Free-text migration guidance. | +| `notes` | no | Context for people: structural differences, migration pitfalls. | +| `distributions` | no | Lifecycles on other channels (Azure, Bedrock, Vertex). See below. | +| `source` | no | URL for this entry's claims, when it differs from the feed's `source`. | + +Model dates (`announced`, `shutdown`) are ISO 8601 calendar dates in UTC, +because publishers announce retirements by day. Only the feed's `generated` is a +full timestamp. + +Every entry that states a date must have a `source` URL, its own or the feed's. +The reference validator rejects a dated entry with neither. + +The reference validator accepts a model ID in `id`, `aliases`, `replacement`, or +`replacement_options` when it is at most 256 characters, starts with a letter or +digit, and contains only letters, digits, `.`, `_`, and `-`. + +### No dates means nothing scheduled + +An entry with neither `announced` nor `shutdown` states that no retirement was +scheduled as of `generated`. A scraper can't make that statement, because a +model missing from a deprecations page might simply be missing from the page. A +publisher serving its own feed can. + +### `date_precision` + +- Absent or `"exact"`: `shutdown` is the stated day. +- `"earliest"`: a scheduled shutdown that can't move earlier, but may happen + later. The reference checker uses it as the shutdown date, so it can fail CI. +- `"tentative"`: the publisher lists a not-before date without announcing a + deprecation. Tentative entries carry no `announced`. Consumers must not treat + their dates as scheduled retirements. The reference checker reports them as + `scheduled` and never fails on them. + +### Replacements + +`replacement` is one exact model ID that resolves to an `id` or alias in the +same feed and is safe as a plain string substitution. It is the only field the +reference `plan` and `apply` patch from. If no such ID exists, it is absent. + +`replacement_options` is an ordered list of exact model IDs, in the order the +publisher gives them. Each option must pass the model ID grammar, but it need +not resolve in the feed, since the target may not be listed yet. + +`replacement_note` carries guidance that isn't an ID, such as a required +parameter (`reasoning.mode: pro`) or a pointer to another platform. + +Replacement options and notes are for issues only. The reference planner never +patches from them; it patches from `replacement` alone. + +OpenAI changed the recommended replacement for its deep-research models during +their notice period. Consumers should read `replacement` again on each run +rather than cache it. + +### `distributions` + +The same model weights can retire on different dates on different channels. +OpenAI shut off `o3-deep-research` on 2026-07-23, while Azure AI Foundry +kept it for months longer. A checker that ignores channels tells an Azure user +their model is gone too early, or tells an OpenAI user they're fine because +Azure still serves it. + +Each distribution entry has: -| Field | Req | Meaning | +| Field | Required | Meaning | |---|---|---| -| `id` | yes | The exact model identifier as sent over the wire (dated snapshot preferred) | -| `aliases` | no | Other strings that resolve to this model (undated aliases, marketing names) | -| `announced` | no | Date the deprecation was announced (absent = not deprecated) | -| `shutdown` | no | Date calls stop working on the publisher's own API (absent = none scheduled) | -| `replacement` | no | A single exact model ID that resolves against an id or alias in this feed and is safe as a drop-in string substitution. This is the only field plan/apply may patch from | -| `replacement_options` | no | An ordered array of exact model IDs for multiple or unresolved choices. These are issue-only and need not resolve in this feed | -| `replacement_note` | no | Free-text guidance such as parameter requirements or provider platform alternatives | -| `notes` | no | Human context: structural differences, migration gotchas | -| `distributions` | no | Per-distributor lifecycles - the same weights on different clocks (Azure, Bedrock, Vertex). Each entry: `via`, optional `announced`/`shutdown`/`status`/`date_precision`, `source` | -| `date_precision` | no | Qualifies `shutdown`: absent or `"exact"` means the stated day; `"earliest"` is a scheduled date that cannot move earlier; `"tentative"` means the publisher lists a not-before retirement date without announcing deprecation | - -Tentative entries carry no `announced`. Consumers must not treat their dates as scheduled retirements. The reference checker reports them as `scheduled` and never fails on them. - -Dates are ISO 8601, UTC. A model with no `announced` and no `shutdown` is an -affirmative statement of "no retirement scheduled as of `generated`" - the absence -is data, which is exactly what scraping HTML can never give you. - -`distributions[].status`, when present, is one of `active`, `legacy`, -`extended-access`, or `retired`. `legacy` means no new consumers but existing -usage works. `extended-access` means the distributor has announced or entered a -distinct extended-access portion of that lifecycle; it does not imply availability -after `shutdown`. On Amazon Bedrock, public extended access is a potentially -higher-priced portion of the Legacy period for active users and ends at EOL, so -`shutdown` remains the Bedrock EOL date. Bedrock still calls the encompassing -state Legacy; `extended-access` is this feed's finer-grained channel signal. -Private arrangements after EOL are not represented by this status. `retired` -means the distributor explicitly reports the model as no longer generally -available even when it does not publish an exact shutdown date. - -Bedrock model-card floors are tentative distribution dates (`date_precision: "tentative"`) without `announced`. - -Azure maps GA/Preview/Legacy to `active`, Deprecated to `legacy`, and Retired to `retired`; Azure Legacy permits new deployments. -Azure OpenAI uses a resolving bare identity for single-row models only when a dated snapshot candidate is absent. +| `via` | yes | The channel, for example `aws-bedrock`, `azure-ai-foundry`, or `vertex-ai`. | +| `announced` | no | As for the model. | +| `shutdown` | no | The date the channel stops serving the model. | +| `date_precision` | no | As for the model. | +| `status` | no | `active`, `legacy`, `extended-access`, or `retired`. | +| `source` | no | URL for the channel's claim. | A model may contain at most one distribution for each `via`. Duplicate channel -records are ambiguous and fail runtime semantic validation. Draft-07 can enforce -whole-object uniqueness, but cannot express uniqueness keyed by the `via` field, -so `model-eol validate` and feed loading enforce this constraint. +records are ambiguous and fail validation. JSON Schema can't express uniqueness +by one field, so `model-eol validate` and feed loading enforce it in code. -### Structured replacement contract +How the reference checker reads distribution entries: an entry without +`shutdown` means nothing is scheduled on that channel and never inherits the +publisher's date, unless its `status` is `retired` (reported as retired) or it +has `announced` without a date (reported as `watch`). When a model has no entry +for the requested channel, the checker uses the publisher's date and labels the +result `publisher-fallback`. -`replacement` is a single exact model ID that resolves against an id or alias in -the same feed and is safe as a drop-in string substitution. This is the ONLY -field plan/apply may patch from. Otherwise absent. +`status` values: -`replacement_options` is an optional array of exact model IDs. Each option must -pass the identifier grammar, but feed resolution is NOT required since a target -may not be carried yet. Options are ordered as the provider lists them. +- `active`: generally available. +- `legacy`: no new customers, but existing use keeps working. +- `extended-access`: the distributor has announced or entered a separate + extended-access part of the lifecycle. It does not mean the model is + available after `shutdown`. +- `retired`: the distributor says the model is no longer generally available, + even if it published no exact date. -`replacement_note` is optional free-text guidance. It can carry parameter -requirements such as `reasoning.mode: pro` or prose alternatives such as Google's -platform pointers. +Notes on current distributors: -Replacement options and notes are issue-only. The planner never auto-patches -from options; it patches `replacement` exclusively. +- On Amazon Bedrock, public extended access is a possibly higher-priced part of + the Legacy period and ends at end of life, so `shutdown` stays the Bedrock + end-of-life date. Bedrock still calls the whole state Legacy; + `extended-access` is this feed's finer-grained signal. Private arrangements + after end of life are not represented. "EOL no sooner than" dates from Bedrock + model cards are recorded as `tentative`. +- Azure's GA, Preview, and Legacy map to `active`, Deprecated to `legacy`, and + Retired to `retired`. Azure's Legacy still allows new deployments. -### Policy floors +## Notice policy -A feed MAY include a publisher's stated minimum deprecation notice period: +A feed MAY state the publisher's minimum notice period: ```json "policy": { @@ -107,59 +186,55 @@ A feed MAY include a publisher's stated minimum deprecation notice period: } ``` -For an entry with neither `announced` nor `shutdown`, a policy floor means the -entry cannot become unavailable before `generated` + `min_notice_days` days, -provided the provider honors its stated policy. This is a stated policy, not a -contract or a guarantee that the provider will remain available. - -The `policy` field is optional. Its absence means no forward claim can be made, -intentionally. OpenAI publishes no formal notice floor, so the OpenAI feed carries -no `policy` field. - -### Why `distributions` is in the core and not an extension - -The July 2026 evidence: OpenAI shut off `o3-deep-research` on July 23; Azure lists it -retiring December 26. Anthropic retired Claude 3.7 Sonnet on February 19; Bedrock -kept it to April 28. The same BOM entry has different sourcing terms per channel, and -a checker that doesn't know that will tell an Azure shop their model is dead five -months early - or tell an OpenAI-direct shop they're fine because Azure says so. - -## Serving it - -- **Providers** (the ask): serve the feed at `/.well-known/model-eol.json`, or add - `announced`/`shutdown`/`replacement` to the existing models endpoint. Either kills - the scraping industry overnight. -- **Aggregators** (the interim): community datasets re-publish provider pages in this - format, with `source` on every entry so claims are auditable. Multiple feeds - compose - a checker loads provider feeds where they exist and community feeds where - they don't. -- **Consumers**: anything that can read JSON - a CI step, a dashboard, a Dependabot - clone that opens migration PRs when `shutdown - today < threshold`. - -Repository tools that export CycloneDX SHOULD represent a canonical model once -per lifecycle channel, with a deterministic channel-qualified `bom-ref`. This -keeps direct publisher and distributor clocks distinct and binds each source -occurrence to the lifecycle decision that evaluated it. - -**Compatibility within 0.1.** The `model-eol/0.1` line evolves additively: new -optional fields and new enumeration values (such as `date_precision: -"tentative"`, added in 2026-09) may appear in feeds and reports without a -version change. A consumer that validates strictly against a snapshot of these -schemas should expect to refresh that snapshot when it upgrades, and a consumer -that reads defensively should ignore fields and values it does not recognise -rather than reject the document. Removing a field or changing the meaning of an -existing value is a new spec line. - -## Non-goals (v0.1) - -Pricing, capabilities, context windows (models.dev and friends already do this); -behavior-change tracking (that's an eval suite's job, not a manifest's); signing and -provenance (worth doing the moment anyone depends on this; premature today). - -## Prior art, gratefully - -endoflife.date (the shape of the solution), OSV (proof that a shared schema turns N -scrapers into an ecosystem), CycloneDX ML-BOM / SPDX AI profile (where an `eol` field -belongs once feeds exist), and the current scrapers - llm-model-deprecation, -llmstatus.ai, llm-info, ai-model-watch-data, modelradar-data - which are the demand -signal this spec is trying to give a common target. +For an entry with neither `announced` nor `shutdown`, this means the model won't +become unavailable before `generated` plus `min_notice_days`, as long as the +publisher follows its own policy. The policy is the publisher's statement and +can change; it is not a contractual guarantee. + +Leaving `policy` out means no forward claim can be made. OpenAI publishes no +fixed notice period, so the OpenAI feed has no `policy`. + +## Serving a feed + +A publisher can serve its feed at `/.well-known/model-eol.json`, or add +`announced`, `shutdown`, and `replacement` to its models endpoint. + +Until publishers do, aggregators can publish feeds on their behalf, with a +`source` on every claim so it can be audited. A consumer can load a publisher's +own feed where one exists and a community feed where it doesn't. + +Tools that export CycloneDX SHOULD write one component per model per channel, +with a deterministic `bom-ref` that includes the channel. That keeps the +publisher's dates and each distributor's dates separate, and ties each +occurrence in the code to the date it was checked against. + +## Compatibility + +The `model-eol/0.1` line changes only by addition. New optional fields and new +enum values (such as `date_precision: "tentative"`, added in September 2026) +can appear without a version change. A consumer that validates strictly against +a copy of the schemas should refresh that copy when it upgrades. A consumer +that reads loosely should ignore fields and values it doesn't recognize. +Removing a field or changing what an existing value means requires a new +version. + +## Out of scope for 0.1 + +- Pricing, capabilities, and context windows. models.dev and similar projects + already cover these. +- Behavior changes between model versions. See + [docs/DESIGN_EVAL_BOUNDARY.md](docs/DESIGN_EVAL_BOUNDARY.md). +- Signing and provenance. These become worth doing once people depend on the + feeds, which is premature today. + +## Prior art + +- [endoflife.date](https://endoflife.date/): end-of-life data for software + products, and the model for this project's shape. +- [OSV](https://osv.dev/): a shared vulnerability schema that let many + scrapers become one ecosystem. +- CycloneDX ML-BOM and the SPDX AI profile, which are where an end-of-life + field belongs once feeds exist. +- Existing deprecation trackers: llm-model-deprecation, llmstatus.ai, + llm-info, ai-model-watch-data, and modelradar-data. They show the demand, and + this format is meant to give them a common target. diff --git a/bot/test/package.mjs b/bot/test/package.mjs index 70cc644..ddf5b86 100644 --- a/bot/test/package.mjs +++ b/bot/test/package.mjs @@ -60,15 +60,15 @@ try { assert(source.match(/node-version: 22/g)?.length === count, `${file} pins every composite Action job to Node 22`) assert(source.match(/package-manager-cache: false/g)?.length === count, `${file} disables package-manager caching for every composite Action job`) } - const readme = fs.readFileSync(path.join(root, 'README.md'), 'utf8') + const botDoc = fs.readFileSync(path.join(root, 'docs', 'BOT.md'), 'utf8') assert( - readme.includes('repo="$(pwd -P)"') - && readme.includes('cd "$tmp"') - && readme.match(/MODEL_EOL_UAT_REPO="\$repo"/g)?.length === 2 - && readme.includes(`-c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan .'`) - && readme.includes(`-c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan . --days 90 --scope direct'`) - && readme.match(/--target-dir "\$repo"/g)?.length >= 2, - 'README resolves published bins externally while planning from the consumer repository for clone-portable paths', + botDoc.includes('repo="$(pwd -P)"') + && botDoc.includes('cd "$tmp"') + && botDoc.match(/MODEL_EOL_UAT_REPO="\$repo"/g)?.length === 2 + && botDoc.includes(`-c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan .'`) + && botDoc.includes(`-c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan . --days 90 --scope direct'`) + && botDoc.match(/--target-dir "\$repo"/g)?.length >= 2, + 'docs/BOT.md resolves published bins externally while planning from the consumer repository for clone-portable paths', ) assert(workflow.match(/node-version: 22/g)?.length === 3, 'consumer workflow runs every model-eol job on supported Node 22') assert(workflow.match(/package-manager-cache: false/g)?.length === 3, 'consumer workflow disables automatic package-manager caching in every job') diff --git a/docs/BOT.md b/docs/BOT.md new file mode 100644 index 0000000..4dcde47 --- /dev/null +++ b/docs/BOT.md @@ -0,0 +1,272 @@ +# The bot + +`model-eol-bot` is a scheduled GitHub workflow that opens PRs and issues for +findings. For each retiring model it keeps one migration PR open. Findings it +can't patch are grouped into issues by model, shutdown date, and channel. It +never merges anything. + +The bot ships in the same npm package as the CLI. Consumer repositories don't +copy any model-eol source; the workflow installs the published package. + +## Setup + +1. Copy [`bot.yml.example`](../bot.yml.example) to + `.github/workflows/model-eol-bot.yml`. It runs every Monday and can also be + started by hand. Each of its three jobs already declares the permissions it + needs. +2. Optionally add `.model-eol.json` (see [CONFIGURATION.md](CONFIGURATION.md)). + The bot reads the same policy as the CLI. Without a config it uses 90 days + and `scope: direct`. +3. Optionally add an eval command (see [below](#the-eval-hook)). +4. Recommended: store a fine-grained PAT or a GitHub App token as the + `MODEL_EOL_BOT_TOKEN` secret, with read and write access to contents, pull + requests, and issues (the same access the publish job asks for). PRs opened + with the default `GITHUB_TOKEN` don't trigger `pull_request` workflows, so + your CI won't run on them. The PR body says so when that applies. + +To preview what the bot would do, with no GitHub calls: + +```sh +npx --yes --package=model-eol@0 model-eol-bot --dry-run --target-dir . --repo OWNER/REPO +``` + +That command works only when no eval command is configured. With an eval +configured, use the [local evaluation run](#run-the-evaluation-locally) instead. + +The bot exits 0 when everything it planned was published or skipped, 1 when any +migration is blocked (a failed eval, a lease or label problem, a conflict, or a +report-only run), and 2 on errors. + +The bot uses the feeds bundled in the npm package it runs, and the example +workflow relies on that. `model-eol-bot --feeds-url URL[,URL...]` downloads +feeds instead, but the publish job checks the plan and the eval results against +the feeds it loads. All three jobs must therefore use the same feed content: the +plan job would also need those feeds, passed to `model-eol plan` with +`--feeds DIR`. The example workflow sets +`concurrency: model-eol-bot` so two scheduled runs never overlap. If you start +the bot some other way as well, make sure those runs can't overlap either. + +## What it opens + +The bot runs `plan` and sorts findings into two kinds. + +A finding becomes a **migration PR** only when all of these hold: + +- The reference is a direct API call and the scanner is confident about it. +- The model is retired, or retires within the threshold. +- The feed names a `replacement` from the same publisher, that replacement is + in the loaded feeds, and it isn't retiring itself. +- The date comes from the channel's own entry, not from `publisher-fallback`. + +Other retired or retiring findings become **issues**: generic model +references, references whose date came from `publisher-fallback`, and models +whose feed entry offers only `replacement_options` or a `replacement_note`. + +Cloud and gateway references also become issues, one per provider, whether or +not a retirement is known, because someone has to check which model is behind +the deployment name. A `routes` entry maps an alias on the line where it +appears, but the provider's other references stay, and so does the issue. +Ignore the path or set `issues.enabled: false` to stop them. + +Waived findings, findings outside the threshold, and candidates (strings that +look like model IDs but aren't in any feed) never create work. + +There is one PR per model and channel. Its branch name is deterministic: +`model-eol///-`, with the `` part left +out for the publisher's own API. The PR body lists the announced and shutdown +dates, days remaining, sources, feed notes, the channel, and the date the +replacement was read from the feed. Replacements can change during a notice +period, so that date matters. When the model still answers on another channel, +the body also has a "Capture window" section with that channel and date, and it +says so when the feed gives no window. + +## Keeping track of its own work + +The bot keeps no state outside GitHub. Each PR and issue it owns carries the +`model-eol` label and a JSON metadata block in an HTML comment on the first line +of the body. The block holds the model, shutdown date, replacement, channel, +feed digest, and the base and head commits. + +The bot trusts an existing PR or issue only when the label, the metadata, the +repository, the base branch, the branch name, and the recorded head commit all +agree. A hand-written comment that looks like the metadata block can't make the +bot skip a finding. Before it pushes to or skips an existing branch, the bot +fetches the branch again and checks it against the recorded lease: the head +commit it last pushed and its committer email, +`model-eol[bot]@users.noreply.github.com`. Anyone with push access can forge +both, so the lease guards against stale runs and accidental edits, not against +a collaborator acting in bad faith. + +On its first run with something to open, the bot creates the `model-eol` label. +If repository policy stops it from creating or applying the label, the bot stops +and exits non-zero. Unlabeled work would let anyone forge ownership later. + +Lifecycle rules: + +- **Dismissal.** Closing a bot PR or issue without merging means "don't reopen + this". The bot stays quiet until the shutdown date or the replacement changes. +- **Cleanup.** When a finding disappears, gets ignored or waived, or the feed + retracts its date, the bot comments on its PR or issue and closes it. +- **Reintroduction.** A model that was migrated and then added back gets new + work. A merged migration doesn't suppress it. +- **Expired waivers.** Work the bot closed because of a waiver comes back as new + when the waiver expires. A PR or issue that a person closed before the waiver + stays dismissed. + +## The eval hook + +A model swap can pass your unit tests and still change behavior, so the bot can +run a command you choose against each proposed migration. With an eval command +configured, only migrations that pass get a PR. Your command decides what counts as good enough; model-eol has +no scorer or judge model of its own. +[DESIGN_EVAL_BOUNDARY.md](DESIGN_EVAL_BOUNDARY.md) explains why. + +### Configure it + +```json +{ + "eval": { + "command": "node scripts/model-eol-eval.mjs", + "timeout_ms": 600000, + "max_report_bytes": 65536, + "pass_env": ["OPENAI_API_KEY"] + } +} +``` + +A non-empty `MODEL_EOL_EVAL_COMMAND` repository variable overrides +`eval.command`. With neither set, the result manifest records that no eval was +configured, and PRs open without an eval result. + +`timeout_ms` defaults to 600000 (10 minutes) and may be 1000 to 3600000. +`max_report_bytes` defaults to 65536 and may be up to 8388608. + +Your command always gets `PATH`, `HOME`, `TMPDIR`, `LANG`, `USER`, and `SHELL`, +plus the variables below. `pass_env` lists any others it may see. Names that +look like GitHub, Actions, SSH, or AWS secrets, or that contain `TOKEN`, +`SECRET`, `PASSWORD`, or `CREDENTIAL`, are refused. Any provider key you list +must also be added as a secret to the evaluate job in the workflow. + +### What the command receives + +The bot applies each migration in its own temporary checkout and runs the +command there with: + +- `MODEL_EOL_OLD_ID` and `MODEL_EOL_NEW_ID`: the model being replaced and its + replacement. +- `MODEL_EOL_PLAN`: a one-model plan with the exact references that changed. +- `MODEL_EOL_REPORT`: the path where the command must write a Markdown report. +- `MODEL_EOL_VIA`: the channel the plan was built for, or empty for the + publisher's API. The feed's replacement is the publisher's model ID; on + Bedrock or Azure your client may use a different string, and this tells you + which mapping to apply. +- `MODEL_EOL_EVAL_MODE`: always `evaluate`. The value `capture` is reserved for + a possible future run against the old model. The bot doesn't run it, and any + other value is refused. + +A migration passes when the command exits 0 and has written a regular file at +`MODEL_EOL_REPORT`. It fails on any other exit code, a timeout, a missing +report, a change to tracked files, a new untracked file that isn't gitignored, +a change to the `MODEL_EOL_PLAN` file, or a change to the checkout's commit. +The report is embedded in the PR body, so keep secrets out of it. The command's +stdout and stderr are never published. + +A failed migration gets no PR. If an earlier PR for it is open, the bot closes +it, and it opens an issue with the failure when issues are enabled. Other +migrations in the same run still go ahead. + +### Write a harness + +Copy [`examples/model-eol-eval.mjs`](../examples/model-eol-eval.mjs) to +`scripts/model-eol-eval.mjs`. It calls `npm run eval:model-eol`, so you need to +add that script to `package.json` and point it at a test that exercises model +behavior. The separate script name is deliberate: it stops an unrelated unit +suite from passing as an LLM eval by accident. Don't point `eval:model-eol` back +at the harness, or it will call itself. + +A useful test checks the response or tool-call shape, structured-output +parsing, any capability you depend on, and a few golden tasks. A green build is +supporting evidence, but it says little about how the new model behaves. + +### Run the evaluation locally + +Before you turn on the scheduled workflow, commit the harness and the policy on +a clean branch, then run the same evaluator locally. From the repository root, +the script below: + +1. Finds the newest published `0.x` version and uses it for every step, as the + workflow does. +2. Builds a plan with your `.model-eol.json`, or with the bot's defaults if you + have none. +3. Runs `model-eol-bot evaluate`, which applies each migration in a temporary + checkout and runs your eval command. +4. Prints the results, then runs the bot in dry-run mode with those results so + you can see which PRs and issues it would open. + +```bash +set -euo pipefail +repo="$(pwd -P)" +version="$(npm view model-eol@0 version --json | node -e 'let s="";process.stdin.on("data",c=>s+=c).on("end",()=>{const v=JSON.parse(s);const r=Array.isArray(v)?v.at(-1):v;if(typeof r!=="string")process.exit(1);process.stdout.write(r)})')" +tmp="$(mktemp -d)" +trap 'rm -rf -- "$tmp"' EXIT +cd "$tmp" +if [ -f "$repo/.model-eol.json" ]; then + MODEL_EOL_UAT_REPO="$repo" npx --yes --package="model-eol@$version" \ + -c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan .' > "$tmp/plan.json" +else + MODEL_EOL_UAT_REPO="$repo" npx --yes --package="model-eol@$version" \ + -c 'cd "$MODEL_EOL_UAT_REPO" && model-eol plan . --days 90 --scope direct' > "$tmp/plan.json" +fi +npx --yes --package="model-eol@$version" model-eol-bot evaluate \ + --target-dir "$repo" \ + --plan-file "$tmp/plan.json" \ + --output-file "$tmp/eval.json" +node -e 'const r=require(process.argv[1]); console.log(r.results)' "$tmp/eval.json" +MODEL_EOL_EVAL_RESULTS_FILE="$tmp/eval.json" \ + npx --yes --package="model-eol@$version" model-eol-bot \ + --dry-run --target-dir "$repo" --repo OWNER/REPO +``` + +The script runs `npx` from a temporary directory, so a local package that +happens to be named `model-eol` can't shadow the published one. The temporary +files are deleted when the shell exits; copy `eval.json` somewhere else first +if you want to keep it. + +## Security model + +The workflow has three jobs with different privileges: + +| Job | Can write to the repo | Sees provider keys | Runs your eval code | +|---|---|---|---| +| plan | no | no | no | +| evaluate | no | only those you add | yes | +| publish | yes | no | never | + +The evaluate job writes a result manifest with one pass, fail, or timeout +record per migration. The manifest is bound to the evaluated commit, the plan, +the eval settings, and the feeds by digest. The publish job rebuilds the plan, +checks the manifest against it, and refuses to publish if the default branch +has moved past the evaluated commit. It publishes the migrations that passed +and marks the rest as blocked. If an eval is configured and the manifest is +missing, it fails before any GitHub API call. + +`pass_env` filters the environment that normal child processes get. It is not +an OS sandbox. Your eval code and its dependencies run as the same user as the +rest of the evaluate job and are trusted there. The privilege boundary is +between jobs: the job running your code has no write token, and the job holding +the write token never runs your code. Reports are treated as untrusted text: they +are size-limited and fenced when embedded in a PR. + +For HTTPS remotes, including private repositories, the bot passes +`GITHUB_TOKEN` to Git as a host-scoped authorization header. It never puts the +token in a remote URL. + +To report a security problem, see [SECURITY.md](../SECURITY.md). + +## Upgrading from inline `--eval` + +Older setups ran the eval inline with `model-eol-bot --eval` and wrote unbound +report and status files. Both are now refused, because separating environment +variables isn't a privilege boundary when evaluation and publication share a +user and a write token. Move to the plan, evaluate, and publish jobs in +`bot.yml.example`. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md new file mode 100644 index 0000000..1a84c19 --- /dev/null +++ b/docs/CONFIGURATION.md @@ -0,0 +1,205 @@ +# Configuration + +The CLI, the GitHub Action, and the bot read one policy file, `.model-eol.json`, +from the root of the target Git repository. If you pass targets in several +repositories that each have their own `.model-eol.json`, the CLI stops and asks +you to choose one with `--config FILE`. + +Command-line flags and Action inputs override the file. Unknown keys and +invalid values are errors, not warnings. The schema is +[`schema/model-eol.bot-config.schema.json`](../schema/model-eol.bot-config.schema.json), +and `model-eol validate .model-eol.json` also checks the rules that JSON Schema +can't express. + +Adding a config file changes the CLI's defaults. Without one, the CLI fails on +cloud and gateway references too (`scope: all`). With one, any key you leave out +takes the bot's defaults: 90 days, `scope: direct`, and the publisher's dates. + +## Keys + +```json +{ + "days": 90, + "scope": "direct", + "via": null, + "ignore": { + "models": ["o3-deep-research"], + "paths": ["test/fixtures/**", "data/catalog"] + }, + "waivers": [], + "overrides": [], + "routes": [], + "issues": { "enabled": true }, + "eval": { "command": "node scripts/model-eol-eval.mjs" }, + "feeds": { "allow_vendored_fallback": false } +} +``` + +| Key | Meaning | +|---|---| +| `days` | Threshold. A model retiring within this many days is a finding. | +| `scope` | `direct` fails only on references the scanner can prove. `all` also fails on cloud and gateway references. | +| `via` | Channel whose dates apply, for example `aws-bedrock`, `azure-ai-foundry`, or `vertex-ai`. `null` means the publisher's own API. The gateways `openrouter`, `litellm`, `portkey`, and `ai-gateway`, and any channel named in a feed, are also accepted. | +| `ignore` | Models and paths to leave out of every report. See below. | +| `waivers` | Accepted exceptions that stay visible but stop failing. See below. | +| `overrides`, `routes` | Per-path settings for monorepos. See below. | +| `issues.enabled` | Whether the bot opens issues for findings it can't patch and for unresolved cloud and gateway references. | +| `eval` | The bot's per-migration test command. See [BOT.md](BOT.md#the-eval-hook). | +| `feeds.allow_vendored_fallback` | When the bot is given remote feeds with `model-eol-bot --feeds-url URL[,URL...]` and the download fails, continue with the bundled feeds in report-only mode instead of stopping. | + +## Ignores + +`ignore` is for references that aren't live dependencies: test fixtures, +caches, generated catalogs, vendored code. Ignored references don't appear in +any report. + +- `ignore.models` takes canonical IDs or aliases. Either one ignores the whole + family, so `o3-deep-research` also covers `o3-deep-research-2025-06-26`. +- `ignore.paths` takes repository-relative globs with `*`, `**`, and `?`, or a + plain directory path such as `data/catalog`, which covers everything under it. + Matching files are excluded before scan coverage is counted. Outside a Git + repository, an ignored directory is still walked, so an unreadable directory + inside it still reports incomplete coverage. + +Some directories are always skipped, so they don't need an ignore: +`node_modules`, `vendor`, `dist`, `build`, `target`, `coverage`, `logs`, `tmp`, +`.git`, `.venv`, `venv`, `__pycache__`, `.next`, `.astro`, `.worktrees`, +`.terraform`, `.tox`, and the `.mypy_cache`, `.ruff_cache`, and `.pytest_cache` +directories. + +## Waivers + +A waiver records a decision to keep using a model on purpose. Unlike an ignore, +it never hides the finding. + +```json +{ + "model": "o3-deep-research", + "paths": ["research/**"], + "reason": "Research pipeline pinned until the Q4 rerun.", + "owner": "@research-platform", + "expires": "2026-12-31" +} +``` + +The finding still appears in `check`, `inventory`, `schedule`, `alert`, `plan`, +and the CycloneDX export, with the waiver attached: + +``` +✗ research/app.py:2 o3-deep-research RETIRED 2026-07-23 (92 days ago) -> gpt-5.6-sol [still answers via azure-ai-foundry until 2026-11-19] [waived until 2026-12-31 by @research-platform: Research pipeline pinned until the Q4 rerun.] +``` + +While the waiver is active, the finding doesn't count toward the exit code, +badge counts, or alert errors, and the bot opens no work for it. On the expiry +date the finding fails again, and the report names the waiver that lapsed and +its owner. Expiry is compared in UTC, and a waiver is already expired on its +`expires` date. + +- `model` covers the whole alias family, like `ignore.models`. +- `paths` works like `ignore.paths`, including plain directory paths. +- `via` limits the waiver to one channel. Use `publisher` for the publisher's + own API, and `publisher-fallback` for findings that fell back to the + publisher's date under `--via` (a `publisher` waiver doesn't cover those). + Leave `via` out to waive every channel. +- When several waivers match, the first active one applies. +- A policy can hold at most 500 waivers, counting the root and all overrides. + +`reason`, `owner`, and an absolute `expires` date are required. A missing owner +or a relative date such as `90d` is a config error. An expired waiver is not an +error; it just stops applying. The CycloneDX export records each component's +waiver state in the `model-eol:waiver_active` property as `true`, `partial`, or +`false`. + +When a waiver expires, the bot opens new work for findings it had closed +because of the waiver. See [BOT.md](BOT.md#keeping-track-of-its-own-work). + +## Monorepos + +Top-level values are defaults. `overrides` change settings by path, and +`routes` choose a channel per reference: + +```json +{ + "days": 90, + "scope": "direct", + "overrides": [ + { "paths": ["services/**"], "scope": "all" }, + { "paths": ["services/bedrock/**"], "days": 180, "via": "aws-bedrock" } + ], + "routes": [ + { "paths": ["services/vertex/**"], "via": "vertex-ai" }, + { + "paths": ["services/bedrock/**"], + "match": "chat-prod", + "model": "claude-3-7-sonnet-20250219", + "via": "aws-bedrock" + } + ] +} +``` + +- `paths` in overrides and routes are globs only. A plain directory path such + as `services/bedrock` matches nothing there; write `services/bedrock/**`. +- Every matching override applies, in order. For single values like `days`, + the last match wins. +- `ignore` and `waivers` inside overrides add to the root lists rather than + replacing them. +- For routes, the last match wins. +- A route with `match` and `model` tells the scanner that a deployment alias + (here `chat-prod`) serves a known model, so the reference is reported with + that model's dates. A routed reference still counts as cloud or gateway + usage: it fails CI only where `scope` is `all` (in the example, through the + `services/**` override), and the bot never patches it, because the string in + the code is the alias, not the model ID. +- Command-line flags and Action inputs still override everything here. + +JSON output from `inventory` and `plan` records the threshold, scope, channel, +and matching override and route indexes for each reference, so you can see +which rule applied. + +## What the scanner reads + +The scanner reads source and config files. [`CODE_EXT` in +`lib/scanner.mjs`](../lib/scanner.mjs) is the exact list. It includes Python, +TypeScript and JavaScript, JSON, YAML, TOML, shell, Ruby, Go, Java, C#, Rust, +C and C++, PHP, Swift, Kotlin, Scala, Objective-C, SQL, Terraform (`.tf`, +`.tfvars`), BAML, and `.env` files. + +It skips: + +- Markdown, text, and reStructuredText files, unless you pass `--include-docs`. +- Generated files: those whose first 8 KiB contain a `Code generated ... DO NOT + EDIT` or `AUTO-GENERATED FILE ... DO NOT EDIT` header, a line starting with + `@generated`, or the BAML generator banner. +- model-eol's own JSON reports, so a rerun doesn't find the previous run's + output. Third-party CycloneDX files are scanned unless their metadata names + model-eol as the generator. +- The always-skipped directories listed under [Ignores](#ignores). + +A BAML client whose `provider` is a publisher's own API, such as +`provider openai`, counts as a direct API call, so `plan` can patch its model +ID. + +## Incomplete coverage + +When the scanner can't read part of the target, it reports incomplete +coverage, and `check` and `plan` exit 2. The causes are: + +| Reason | What happened | +|---|---| +| `submodule-skipped` | A Git submodule. The scanner doesn't enter it. | +| `nested-repository-skipped` | A nested Git repository that isn't a submodule. | +| `file-too-large` | A file over 2 MiB. | +| `invalid-utf8` | A file that isn't valid UTF-8. | +| `symlink-skipped` | A symbolic link. Links are never followed. | +| `unreadable-file`, `unreadable-path`, `unreadable-directory` | A permissions or I/O error. | +| `file-count-cap` | More than 100,000 files. | +| `git-listing-failure` | `git ls-files` failed. | + +For each one, ignore the path, pass `--allow-incomplete`, or (for submodules +and nested repositories) scan that checkout as its own target with its own +policy. + +For a repository with a lot of content, start with `inventory` to see what the +scanner finds. Then add `ignore.paths` for caches, catalogs, archived fixtures, +and docs that aren't live dependencies. diff --git a/docs/CONTEXT.md b/docs/CONTEXT.md deleted file mode 100644 index 92f49ca..0000000 --- a/docs/CONTEXT.md +++ /dev/null @@ -1,226 +0,0 @@ -# Context - read this first if you're picking the project up cold - -*Written 2026-07-25, the week the problem bit; updated 2026-08-18 after the -public release and field UAT. Everything in this file is public-safe.* - -## Origin - -On July 23, 2026, OpenAI shut down a scheduled wave of models, including -`o3-deep-research` - a model the author had a tuned research workflow on. The -companion essay ("The Model BOM," forthcoming on tomsullivan.dev) argues that the -model line in a software bill of materials is categorically different from every -other line: it can change behavior without changing version, it can vanish on the -provider's schedule regardless of pinning, and **nothing in your toolchain warns you -about either** - no npm-style install warning, no Dependabot PR, no advisory in CI. -The deprecation notice is an email and an HTML page; the first machine-visible -signal is the production 404. - -This repo is the constructive half: a draft **feed format** (SPEC.md) that makes -model retirement dates machine-readable, seed **feeds** for OpenAI and Anthropic, -and a zero-dependency **reference CLI** (check.mjs) that can fail CI when a pinned -model is retired or retiring, emit a repo inventory, or build a deprecation -schedule. - -## Positioning - why format-first - -The niche has tools already (see LANDSCAPE below). All of them are scrapers with -private registry formats. Building a sixth scraper adds nothing; the missing layer -is the shared schema - the endoflife.date / OSV move. The endgame is providers -serving the feed themselves (`/.well-known/model-eol.json`, or a `retirement_date` -field in the models endpoint). Until then, community feeds in a common format let -N tools share data instead of N scrapers each rotting separately. - -The one capability here that no existing tool has: **per-distributor lifecycles** -(`distributions` in the spec). The same model retires on different dates per channel -- o3-deep-research died at OpenAI on 2026-07-23 but lives on Azure AI Foundry until -2026-12-26; Claude 3.7 Sonnet retired at Anthropic 2026-02-19 but on Bedrock -2026-04-28. A checker ignorant of channels gives wrong answers in both directions. -`check.mjs --via azure-ai-foundry` demonstrates it working. - -The product direction is **direct-first inventory**, not "CI catches everything." -Direct API usage can be scanned and checked from code. Cloud providers and gateways -often hide the exact model behind deployment names or aliases, so the scanner records -Azure/Bedrock/Vertex/OpenRouter/LiteLLM/etc. as resolver targets unless it can see a -real model ID. Model-like strings absent from the loaded feeds are emitted as -non-failing candidates so feed gaps are visible. See `docs/PRODUCT_PLAN.md`. - -## LANDSCAPE (as of 2026-07-25) - -| Tool | What it is | Gap | -|---|---|---| -| llm-model-deprecation (PyPI + GH Action, techdevsynergy) | directory scanner, weekly-updated registry | private registry format, no distributor clocks | -| llmstatus.ai | hosted tracker + CLI, refreshes 6h from models.dev/OpenRouter/provider APIs | own format; hosted dependency | -| llm-info (npm) | model metadata incl. deprecated mappings | metadata lib, not a CI gate | -| Khavel/ai-model-watch-data | open daily dataset (prices, context, deprecation dates) | own format | -| alexanderkatsovych/modelradar-data | open dataset, ~108 models with retirement dates/successors | own format | -| models.dev | open model-metadata registry several tools build on | capabilities-focused; not an EOL feed | -| endoflife.date | machine-readable EOL for hundreds of software products | started tracking Anthropic Claude (July 2026, `/api/anthropic-claude.json` with eol + recommendedReplacement) - one provider, no distributor clocks, no announced/shutdown split. Strongest convergence target; contribute OpenAI/Google there | - -All small, all recent, all evidence of demand. The play is convergence, not -competition - see the outreach issue. - -## Design decisions already made (and why) - -1. **Match known IDs only, never discover.** The checker flags strings that exist in - loaded feeds; it doesn't try to regex "anything model-shaped." Precision over - recall for a CI gate - a false positive in CI erodes trust faster than a miss. -2. **`distributions` in the spec core, not an extension.** See above; it's the - load-bearing novelty. -3. **Absence is data.** A feed entry with no `announced`/`shutdown` is an - affirmative "nothing scheduled as of `generated`" - a scraper can never say that; - a provider-served feed can. This is the strongest argument for provider adoption. -4. **`replacement` is explicitly a snapshot, not a constant.** OpenAI's recommended - replacement for the deep-research models changed *during* the notice window - (gpt-5.6-sol didn't GA until 2026-07-09, two weeks before the shutoff). The spec - says so; tools should re-read, not cache forever. -5. **Zero-dependency Node for the reference checker.** The tool that guards against - dependency rot should not itself be a dependency tree. Same reason the validator - (PR #1) hand-rolls structural checks instead of pulling ajv. -6. **Dates are UTC ISO, day precision.** Providers announce days, not instants. -7. **Cloud/gateway references are inventory hints until resolved.** Static code can - identify that a repo uses Azure Foundry, Bedrock, Vertex, OpenRouter, LiteLLM, or - Portkey, but it usually cannot prove the deployed model behind an alias. Resolver - integrations should enrich inventory output later. -8. **Alerting starts stateless.** `check.mjs alert` emits GitHub Actions annotations, - Markdown, badge JSON, or JSON and exits nonzero on retired/retiring findings. The - bot adapter (bot/) is the stateful consumer: it reads `plan --json` over a - subprocess boundary and keeps all its state in GitHub PR/issue metadata blocks. -9. **Patch gating is strict.** Only high-confidence direct-api findings with a - feed-resolvable, non-retiring replacement are auto-patchable; generic - model-references, cloud/gateway hints, and publisher-fallback clocks are - issue-only; candidates never generate work items at all. -10. **`--via` semantics.** A distribution entry without `shutdown` means "no - retirement scheduled on that channel" - it never inherits the publisher clock. - Only a missing distribution entry falls back, labelled `publisher-fallback`, - which is never patchable. (The original conservative-fallback read told Azure - shops their models were dead five months early.) -11. **Informational is not changed.** refresh `--check` exits 3 only when generated - files materially differ; unconfirmed entries and no-publisher-feed rows render - in the diff but do not trip it (a keyless weekly run would otherwise open a - no-op PR forever - the first live workflow dispatch proved it). -12. **Policy floors are stated policy, not contracts** - the wording in SPEC.md and - the schedule output says exactly that, and feeds without a stated floor make no - forward claim. -13. **The eval hook crosses a trust boundary.** Repository-owned code runs only in - the read-only evaluate job, with explicit normal environment forwarding, - bounded runtime/report size, and reports treated as untrusted content. The - forwarding allowlist is not an OS sandbox; the hard privilege boundary is no - repository write token, and publication never executes the eval command. -14. **No model-attribution trailers on commits.** Implementation is a mix of Claude - and Codex delegation; per-commit honesty lives in commit bodies ("Implemented by - Codex (gpt-5.6-luna); reviewed by Claude"), never in Co-Authored-By trailers. - History was rewritten 2026-08-01 to enforce this. -15. **Structured replacement fields are settled.** `replacement` is a single exact - model ID that resolves against an id or alias in the same feed and is safe as a - drop-in string substitution. This is the ONLY field plan/apply may patch from. - Otherwise absent. `replacement_options` is an optional ordered array of exact - model IDs that must pass the identifier grammar, but feed resolution is NOT - required since a target may not be carried yet. `replacement_note` is optional - free-text guidance for parameter requirements or prose alternatives. Options and - notes are issue-only; the planner never auto-patches from options and patches - `replacement` exclusively. - -## Publish gates (do not flip public before) - -1. The companion essay is live (this repo is its artifact; sequence matters). -2. ~~Feeds re-verified against provider pages on publish day~~ DONE and automated: - feeds regenerated from live sources 2026-08-01; the weekly feed-refresh - workflow keeps them verified. -3. ~~Name decision~~ RESOLVED 2026-08-01: keep "model-eol". npm name verified free, - no GitHub collisions, and endoflife.date entering LLM tracking strengthens the - lineage argument. package.json is publish-ready; npm publish is the maintainer's call. -4. Sweep this repo for anything non-public-safe (should be nothing; keep it that way - - write every commit as if the repo were already public). - -## State as of 2026-08-18 - -The direct-first product is shipped on npm and as a moving `v0` Action: bounded -scanner, strict repository policy, mixed-repository overrides/routes, lifecycle -reports, CycloneDX, safe plan/apply, and a stateless GitHub bot with trusted -ownership, isolated per-migration evals, and stale-work reconciliation. Provider -refresh covers OpenAI, Anthropic, Google, Mistral, and Cohere; distributor refresh covers Bedrock, -Vertex, and Azure Foundry. Weekly refresh, failure receipts, semantic PRs, trusted npm publishing, -and exact-package consumer tests are live. - -Bedrock refresh follows the 2026-09-07 policy change. -It combines the [legacy table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html) -with [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html). -The legacy table wins overlaps; card floors become tentative distribution dates without announcements. -Refresh reports exact-date conflicts, unmatched publisher IDs, and cards without usable lifecycle dates. -Legacy precedence also applies when different Bedrock IDs resolve to one publisher model through aliases. -Conflicting card records still stop refresh after alias resolution. -Each card must contain exactly one Model ID table. -Any first-row cell containing the exact text `Model ID` identifies that table, including `td` cells. -Card parsing removes comments before extraction and rejects lifecycle labels outside their expected paragraphs. -The label guard counts whole labels regardless of case or colon presence. -A standalone non-N/A Model EOL date supplies lifecycle data. -Index links resolve within the index directory after query and fragment removal. -Invalid card links stop refresh. -Distributor table expansion caps each span at 64 and each table at 10,000 cells. - -Mistral rejects lifecycle headers containing data cells or unexpected labels. -Mistral checks rows containing `th` cells for header drift. -Without `th` cells or a pending header, it checks only the first row. -Cohere collects identities and aliases from every section before merging records in ascending heading-date order. - -Anthropic reads model identities from overview pages listed in `https://platform.claude.com/docs/llms.txt`. -It imports only `Claude API` and `Claude API alias` rows. -The index allows at most 100 unique pages; missing fixtures stop refresh. -Model pages supply no dates; the deprecations page remains the only Anthropic date source. -Refresh reports each attached or moved alias through notices and semantic diffs. - -Azure binds publisher feeds through the Azure OpenAI, Anthropic, Mistral AI, and Cohere section headings. -OpenAI dates first resolve exact snapshot IDs or aliases. -Azure OpenAI uses a resolving bare identity for single-row models only when a dated snapshot candidate is absent. -Azure counts Model cells before deduplication and ignores Replacement cells. -Four-digit versions always require exact snapshot IDs or aliases. -OpenAI versions `1`, `2`, `001`, and dashes use bare IDs. -Mistral and Cohere integer versions and dashes use bare IDs. -Cohere binding strips one leading `Cohere-` or `cohere-` prefix. -Binding preserves the remaining case. -Missing candidates remain unconfirmed. -Matching hosting rows collapse. -Conflicting Mistral or Cohere rows stop refresh. -Lifecycle header drift stops refresh. -Fine-tuning and unrelated tables remain excluded. -Refresh PR bodies include source conflicts even when the corresponding check reports no material changes. - -The staged public-contract milestone gives every public schema a canonical ID, exposes -`model-eol validate`, and adds a Pages publisher for hosted feeds, Atom, and -`last_checked` health. The host becomes authoritative only after Pages is enabled -and the exact-byte deployment UAT passes. The same milestone treats distributor -status changes as material, preserves structured replacement guidance in -machine-readable reports, -and closes the remaining symlink false-clean edge. Node 22 is the supported floor; -runtime and development dependencies remain zero. - -Field-test receipts include a hosted plan/evaluate/publish run that created a -one-line migration PR, owned-repository sweeps, a packed offline consumer, and an -independent published `0.4.1` run against a content-heavy repository. The latter -correctly found eight raw retired references, then went clean with an explicit -policy excluding cached data, draft fixtures, and a reference catalog. That is -the intended operating model: inventory first, then auditable repository policy. - -Expansion backlog: authenticated Azure/Bedrock/Vertex/gateway resolvers, more -publisher feeds, waiver ownership/expiry, remote-feed trust/signing and overlay -composition, and broader file-format coverage. - -## Conventions - -- Prose in this repo uses " - " (space-hyphen-space), never em-dash glyphs. -- Every feed entry claiming a date must carry a `source` URL (feed-level or - entry-level) so claims are auditable. -- check.mjs stays a thin dispatcher; logic lives in lib/. (The original "checker - under ~200 lines" rule outgrew reality at v0.1.0 - the spirit survives as: the - reference implementation stays small, readable, and zero-dependency; complexity - goes in the spec discussion.) - -## Related work by the author - -- The companion essay draft and its research live in the author's content - workspace; the essay's mitigation ladder (harness -> evals -> structural seams) - is the strategic frame this tool slots into (rung two: "make retirement dates - fail CI"). -- `docs/RESEARCH.md` in this repo carries the public receipts - deprecation - history, policy floors, drift evidence - so this repo argues for itself without - external context. diff --git a/docs/DESIGN_BOT.md b/docs/DESIGN_BOT.md deleted file mode 100644 index a6fe21f..0000000 --- a/docs/DESIGN_BOT.md +++ /dev/null @@ -1,188 +0,0 @@ -# Design - the model-eol bot ("Dependabot for models") - -*Status 2026-08-18: the architecture below is shipped, adversarially reviewed, -and proven through a hosted plan/evaluate/publish UAT. Current releases add -commit-bound evaluator artifacts, trusted ownership leases, stale-work -reconciliation, per-reference routing, and permanent published-consumer tests. -Retired generic model references remain issue-only (`not-direct-api`).* - -## Decisions locked - -1. **Form factor**: cron-mode GitHub Action, no hosted app. Zero ops, installable - by anyone, builds on the existing composite action. -2. **Remediation unit**: a migration PR with full context, never auto-merged, - with a user-configurable eval hook seam. An npm bump is presumed safe until CI - fails; a model swap is presumed unsafe until evals pass. -3. **Tracks run in parallel**: Track A (repo-side bot) and Track B (feed refresh) - are independent and share only the schema. -4. **Language**: zero-dependency Node for the core, unchanged. Track B tooling - may carry its own dependencies (the zero-dep rule is scoped to the reference - checker). Rust is a possible v2 distribution story, not a v1 concern. - -## Architecture - three layers - -``` -core (zero-dep, this repo) bot adapter (zero-dep Node) feed refresh (Track B) -check/inventory/schedule/ -> GitHub REST orchestration, <- provider models endpoints -alert/plan/apply PR lifecycle, eval runner + deprecation pages -> feed PRs -``` - -The core stays executable spec documentation: scanning, lifecycle math, plan -generation, deterministic apply. All GitHub orchestration lives in a separate -bot adapter consuming versioned `plan --json`. Nothing stateful in the core. - -## Track A - repo-side bot - -### A1. Scanner hardening (prerequisite, found by field test 2026-08-01) - -- **Traversal**: use `git ls-files` (tracked + untracked-unignored) when the - target is inside a git work tree; fall back to a bounded walk (lstat, never - follow symlinks, expanded skip list, per-file size cap, file-count cap) for - non-git directories. Field test: a real repo had 414k files outside - node_modules/.git and hung the scanner for 2+ minutes. -- **Bad targets warn, not abort**: a missing path skips with a stderr warning; - exit 2 only when no valid target remains. -- **Dotfile extensions**: `path.extname('.env')` is `''`, so `.env` in CODE_EXT - is dead code today. Match dotfile basenames (`.env`, `.env.*`) explicitly. -- **Feed composition safety**: `loadFeeds` currently lets the last feed silently - win on id/alias collisions. Cross-feed duplicates become a hard load error - naming both files. -- **`--via` semantics**: a distribution entry for the requested distributor with - no `shutdown` means "no retirement scheduled on that channel" - it must not - inherit the publisher clock. Only when no distribution entry exists for the - requested distributor may the publisher clock apply, labelled - `publisher-fallback`. Fallback findings are never patchable. - -### A2. `plan` and `apply` (core commands) - -- `plan` emits a versioned plan document (`model-eol.plan/0.1`): patchable items - plus non-patchable issues, each with full feed context (dates, sources, notes). -- **Patch gating** (tightened per Codex review): an item is patchable only if - ALL hold - usage is `direct-api` at high confidence; status is retired or - retiring within threshold; the feed names a `replacement`; the replacement - resolves in the loaded feeds and is not itself retired/retiring; the lifecycle - date is not a `publisher-fallback`. Generic `model-reference`, cloud/gateway - hints, and candidates are issue-only. Candidates NEVER generate work items. -- **Apply safety**: each plan item carries the expected line content hash and - occurrence index. `apply` verifies before writing and refuses on any mismatch - (line drift, changed checkout, duplicate ambiguity). Idempotent re-runs. - -### A3. Bot adapter (separate task, after A1/A2 integrate) - -- Cron workflow scans the default branch; per retiring model and distributor - clock it maintains one open PR on a deterministic ref-safe branch - (`model-eol///-`). -- Lifecycle: trust work only when the `model-eol` label, valid metadata identity, - target repository, and expected branch agree. Malformed or user-authored - markers elsewhere never suppress findings. Before every skip or force-push, - independently fetch the branch and verify the recorded head, bot committer, - target default branch, and recorded base commit. -- Reconciliation: valid labelled bot PRs and issues are commented and closed - when their findings disappear, become ignored/retracted, change clock or kind, - or issue publication is disabled. A previously merged migration does not - suppress a retired model that is later reintroduced; a deleted historical bot - branch may be safely recreated after its recorded lease is checked. -- Workflow `concurrency` is keyed per repo. A machine-readable metadata block - (HTML comment JSON: canonical id, shutdown, replacement, clock, feed digest, - base/head commits) plus the label carries all state; the bot stays stateless. -- Dismissal: closed-unmerged PR with matching metadata means "don't reopen for - this (id, shutdown)"; a changed shutdown date reopens. -- PR body: announced/shutdown dates, days remaining, sources, feed notes, - distributor clock when configured, "replacement per feed as of " - timestamp, explicit warning that checks may be skipped under `GITHUB_TOKEN`. -- **Ownership label**: label creation and assignment are fail-closed. Publishing - unlabeled work would make future ownership decisions forgeable, so a label - policy failure produces a blocked non-zero outcome instead. -- **Token reality**: PRs created with `GITHUB_TOKEN` do not trigger - `on: pull_request` workflows. Document a fine-grained PAT / GitHub App token - as the recommended path; degrade gracefully (PR still opens, body warns). -- **Feed freshness**: the copy-ready workflow resolves the moving npm `0.x` line - once and carries that exact published package through all three jobs. Its - schema-validated bundled feeds are the default. Explicit remote feed URLs are - opt-in, fail closed, and may use a configured vendored fallback in degraded - report-only mode. - -### A4. Eval hook (security boundary, per Codex review - critical finding) - -Contract: `{"eval": {"command": "..."}}` in `.model-eol.json`; env vars -`MODEL_EOL_OLD_ID`, `MODEL_EOL_NEW_ID`, `MODEL_EOL_PLAN`, `MODEL_EOL_VIA`, -`MODEL_EOL_EVAL_MODE`; exit 0 = pass; -a regular bounded Markdown report is required at the `MODEL_EOL_REPORT` path for -an exit-zero run to count as passing. - -- `MODEL_EOL_VIA`, the distributor channel the plan was built for, or empty for the publisher's direct API. Use it to map the publisher's replacement ID to your client's ID on that channel. -- `MODEL_EOL_EVAL_MODE`, always `evaluate` today. The value `capture` is reserved for a future run on the unpatched checkout against the old ID. The bot does not run it, and the hook refuses any other value. - -The command is intentionally explicit rather than inferred from package files: -model-eol cannot know whether a unit suite proves tool calling, structured output, -retrieval quality, or another repository-specific behavior. The checked-in -[`examples/model-eol-eval.mjs`](../examples/model-eol-eval.mjs) is the -dependency-free starter. Consumers should customize its verification command and -semantic assertions, always write a non-secret receipt to `MODEL_EOL_REPORT`, and -run the evaluator locally against a clean committed checkout before enabling the -hosted workflow. A non-empty `MODEL_EOL_EVAL_COMMAND` workflow variable takes -precedence over `eval.command`; an absent command is recorded as unconfigured, -not silently treated as a passing eval. - -Execution rules: the workflow resolves the moving npm major once, validates and -records the exact version, and reuses it in all three jobs. The plan job emits a -versioned plan. A separate least-privilege evaluate job independently verifies -that plan, applies each migration in its own temporary checkout, then invokes -the configured command with its own old/new IDs and selected plan. It has no -write token; consumers add only the provider secrets their trusted eval needs, -and those names must also appear in `eval.pass_env`. Environment filtering controls -normal child-process forwarding, not same-user OS isolation; repository code and -its dependencies remain trusted inside the evaluate job. Timeout and report-size -limits apply per migration, and tracked eval workspace/history drift turns that -migration into a failure. - -The evaluate job always uploads one bounded result manifest containing an exact -pass/fail/timeout record per migration. The manifest is bound to the evaluated -Git commit plus stable plan and eval-configuration digests. The publish job -refuses a newer default-branch head, receives no provider keys, -independently regenerates the plan, verifies both artifacts before any GitHub -operation, publishes passing migrations, records failed peers as blocked, and -returns non-zero if any conflict, eval failure, lease stand-down, label failure, -or degraded report-only decision remains. Reports are untrusted content: capped -and fenced when embedded in PR bodies. The publish process never executes -repository-owned eval code. When `eval.command` is configured, a missing bound -result manifest fails before GitHub API access. - -The former inline `model-eol-bot --eval` mode and the unbound report/status -artifacts are intentionally refused: process-level environment scrubbing is not a -privilege boundary when evaluation and publication share a user and write token. -Consumers migrate to the plan/evaluate/publish workflow in `bot.yml.example`. - -### Config - `.model-eol.json` in the consuming repo - -Threshold days, scope, `via`, ignore list (canonical id ignores its aliases; -repo-relative globs), PR-vs-issue preferences, eval hook. - -## Track B - feed refresh (supply side) - -- Scheduled job regenerates `feeds/*.json` from two source classes per provider: - the **models endpoint** (live list of current models - this is what makes - absence-is-data real; field test showed the Anthropic feed carries only - deprecated models, so healthy repos get "unknown candidate" instead of - "you're clear") and the **deprecations page** (dates, replacements). -- Output: schema-valid feeds where current models appear as entries with no - `announced`/`shutdown`, every dated entry carries a `source` URL. -- Parse failures fail loudly and never emit a guessed feed or drop existing - entries. Human review stays in the loop: the job opens a PR only when - something changed, with a semantic diff (models added, dates moved, - replacements changed) in the body. -- Tests run offline against recorded fixtures. -- Workflow ships as `refresh.yml.example` (workflow-scoped-credential - constraint, same convention as ci.yml.example). - -## Spec questions raised (v0.2 candidates, not blocking) - -- Distribution entry without `shutdown`: define as "no retirement scheduled on - that channel as of `generated`" (matches absence-is-data). -- Model ids need a documented git-ref-safe slug rule for branch names. -- Plan document schema joins the published schemas. - -## Backlog (accepted, not gating v1) - -Monorepo ownership groups (per-path reviewers, per-component eval commands), -ignore-semantics precision (expiry dates, audit reasons), feed signing. diff --git a/docs/DESIGN_EVAL_BOUNDARY.md b/docs/DESIGN_EVAL_BOUNDARY.md index 7de23f2..cdd22f9 100644 --- a/docs/DESIGN_EVAL_BOUNDARY.md +++ b/docs/DESIGN_EVAL_BOUNDARY.md @@ -1,86 +1,93 @@ -# Design - the eval boundary (what model-eol verifies, and what it does not) +# What model-eol verifies, and what it doesn't -Status: decided 2026-09-09 (issue #98). This note records the line so that -future requests for "evals inside model-eol" start from the reasoning, not from -scratch. +Decided 2026-09-09 in issue #98. This note records where the line sits, so a +future request for "evals inside model-eol" can start from the reasoning. ## The decision -model-eol owns two things around migration verification: - -1. **The deadline.** The feeds carry the date a model stops answering on each - clock: the publisher's API, and each distributor (Bedrock, Azure, Vertex). - An exact date is the last day a baseline can be captured from the old - model. A tentative or earliest floor is only "at least until": the old - model may answer past it, and the bot words it that way. On the applied - clock the date is the finding's `shutdown`. When the feed lists - another clock on which the model still answers, `check` appends - `[still answers via until ]` to the retiring or retired - line, and the bot adds a `## Capture window` section to every PR and issue - it opens. The value comes from the feed only. When the feed has no date, - model-eol says nothing. The `check --json` document does not carry a - `capture` field yet: the 0.1 report shape published by 0.5.2 is kept - byte-compatible for default output, and adding the field is a separate - contract decision. -2. **The eval-hook contract.** The bot runs one repository-owned command on the - patched checkout and passes `MODEL_EOL_OLD_ID`, `MODEL_EOL_NEW_ID`, - `MODEL_EOL_PLAN`, `MODEL_EOL_REPORT`, `MODEL_EOL_VIA`, and - `MODEL_EOL_EVAL_MODE`. Exit zero plus a report file is a pass. That is the - whole verification interface. - -model-eol does not own verification itself. It ships no comparator, no +model-eol owns two things around verifying a migration: the deadline and the +eval-hook contract. + +### The deadline + +The feeds carry the date a model stops answering on each channel: the +publisher's own API, and each distributor (Bedrock, Azure AI Foundry, Vertex AI). +An exact date is the last day you can record a baseline from the old model. A +tentative or earliest date only means "at least until": the old model may +answer past it, and the bot says so in those words. + +On the channel being checked, that date is the finding's `shutdown`. When the +feed lists another channel where the model still answers, `check` adds +`[still answers via until ]` to the retired or retiring line, +and the bot adds a "Capture window" section to its migration PRs and issues. +The value comes only from the feed. When the feed has no date for any other +channel, `check` adds nothing, and for a retired model the bot's section says it +can't state a +capture window. + +`check --json` doesn't include the capture window yet. Adding it would change +the published 0.1 report shape for default output, and that is a separate +contract decision. + +### The eval-hook contract + +The bot runs one repository-owned command on the +patched checkout and passes it `MODEL_EOL_OLD_ID`, `MODEL_EOL_NEW_ID`, +`MODEL_EOL_PLAN`, `MODEL_EOL_REPORT`, `MODEL_EOL_VIA`, and +`MODEL_EOL_EVAL_MODE`. Exit 0 with a report file is a pass. That is the whole +verification interface. [BOT.md](BOT.md#the-eval-hook) has the details. + +model-eol does not do the verification itself. It has no comparator, no per-case results schema, no starter case set, no scorer, and no judge model. -Those live in the repository's own eval command and CI, or in tools built for -that job (promptfoo, Inspect AI, and the like). - -## Why the line sits here - -- **The Dependabot pattern.** Dependabot opens the PR; the repository's CI - decides. model-eol already works that way through the eval hook. Moving the - decision into model-eol would make it the first part of the tool that - guesses. -- **The hard parts are verification-domain problems, not lifecycle-data - problems.** A design pass on golden capture and replay found that every deep - question belongs to the eval side: which cases to run, how to score prose, - how many runs make a baseline stable, how much API budget a run may spend, - and how to map a publisher's model ID to a distributor's client ID. None of - those has settled numbers in the published literature or vendor guidance. - model-eol has no data that would improve them. -- **Zero dependencies.** Calling model APIs from model-eol would need SDKs, - keys, and per-vendor request formats. That ends the zero-dependency - contract that keeps the tool auditable. -- **No customer with a case set.** As of the decision, no consumer had a real - case set to design against. A comparator built without a user is - speculation. - -## What the feeds can say, and what they cannot - -The feed answers "until when does this model answer on this clock". It does -not answer "how close is the replacement". A model retired on the publisher -clock can still answer on Bedrock or Azure for months; `capture.alternatives` -lists those clocks with their dates. That is the one verification-adjacent -fact only this data set gives, so it is the one model-eol reports. - -Distribution rows carry no replacement ID. The feed's `replacement` is the -publisher's ID (for example `claude-sonnet-4-6`). A repository that calls -Bedrock holds a different string for the same weights. An eval command that -replays against the replacement on a distributor must map the publisher ID to +Those belong in the repository's own eval command and CI, or in tools built for +the job, such as promptfoo or Inspect AI. + +## Why the line is here + +Dependabot opens a PR and leaves the decision to the repository's CI, and +model-eol already works the same way through the eval hook. If model-eol scored +migrations itself, it would be guessing about behavior it can't observe. + +A design pass on capturing and replaying golden outputs found that every hard +question was about evaluation: which cases to run, how to score prose, how many +runs make a stable baseline, how much API budget a run may spend, and how to +map a publisher's model ID to a distributor's. None of these has settled +answers in published research or vendor guidance, and model-eol's lifecycle +data doesn't help answer them. + +Calling model APIs would also mean SDKs, keys, and per-vendor request formats, +which would end the zero-dependency rule that keeps the tool easy to audit. And +when this was decided, no consumer had a real set of eval cases to design +against, so a comparator would have been built without a user. + +## What the feeds can and can't say + +A feed says how long a model keeps answering on each channel. It says nothing +about how close the replacement is. A model retired on the publisher's API can +keep answering on Bedrock or Azure for months, and the capture window lists +those channels and dates. The capture window is the only verification fact the +feeds can supply, and it is the only one model-eol reports. + +Distribution entries carry no replacement ID. The feed's `replacement` is the +publisher's ID, for example `claude-sonnet-4-6`, while a repository that calls +Bedrock uses a different string for the same model. An eval command that +replays against the replacement on a distributor has to map the publisher ID to its own client ID. `MODEL_EOL_VIA` tells the command which channel the plan was -built for so it can do that mapping. +built for so it can do that. ## The reserved `capture` mode -`MODEL_EOL_EVAL_MODE` is `evaluate` today. The value `capture` is reserved for -a run on the unpatched checkout against the old ID, so a command could record -a baseline before the capture window closes. The bot does not run capture -mode, and `runEvalHook` refuses any mode other than `evaluate`. The name is -reserved so that a future opt-in does not have to change the contract. +`MODEL_EOL_EVAL_MODE` is always `evaluate` today. The value `capture` is +reserved for a run on the unpatched checkout against the old model, so a +command could record a baseline before the window closes. The bot doesn't run +capture mode, and the hook refuses any value other than `evaluate`. Reserving +the name now means a future opt-in won't need a contract change. ## What would reopen this -A consumer with a real case set, a concrete regressed-item gate they already -use, and a request for model-eol to schedule (not score) the capture run before -the deadline. Even then the scoring stays in the consumer's command. The -candidate addition would be an opt-in capture run keyed by old ID and case-set -hash, run at most once per key, with the existing timeout as the only cap +A consumer with a real case set, a regression gate they already use, and a +request for model-eol to schedule the capture run before the deadline, but not +to score it. Even then, scoring would stay in the consumer's command. The +likely addition is an opt-in capture run keyed by old model ID and case-set +hash, run at most once per key, with the existing timeout as the only limit model-eol can enforce. diff --git a/docs/FEEDS.md b/docs/FEEDS.md new file mode 100644 index 0000000..ff04122 --- /dev/null +++ b/docs/FEEDS.md @@ -0,0 +1,108 @@ +# Feeds + +The files in [`feeds/`](../feeds) follow the format in [`SPEC.md`](../SPEC.md). +The publisher feeds are generated by [`refresh/`](../refresh) and never edited +by hand. A hand edit would be overwritten by the next weekly run, and it would +have no source to point to. + +## Sources + +| Feed | Dates come from | Current models come from | +|---|---|---| +| `openai.json` | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations) | the models endpoint (needs `OPENAI_API_KEY`) | +| `anthropic.json` | [Anthropic model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations), including "not sooner than" dates for active models | the model overview pages listed in `llms.txt` (no key needed), and the models endpoint when `ANTHROPIC_API_KEY` is set | +| `google.json` | [Gemini API deprecations](https://ai.google.dev/gemini-api/docs/deprecations) | the models endpoint (`GEMINI_API_KEY`; locally `GOOGLE_API_KEY` also works) | +| `mistral.json` | [Mistral models](https://docs.mistral.ai/models) | the models endpoint (`MISTRAL_API_KEY`) | +| `cohere.json` | [Cohere deprecations](https://docs.cohere.com/docs/deprecations) | the models endpoint (`COHERE_API_KEY`) | +| `amazon.json` | Amazon's own models. The entries are carried forward from the committed file; only their `aws-bedrock` dates are refreshed from the [Bedrock lifecycle pages](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html). | | + +Distributor dates are added to the publisher feeds as `distributions` entries: + +- **Amazon Bedrock**: the [legacy lifecycle table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html) + for models launched before AWS changed its policy on 2026-09-07, and the + individual [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) + for newer ones. A card's "EOL no sooner than" date becomes a tentative date, + and an exact Model EOL date stays exact. +- **Vertex AI**: the [model versions page](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/model-versions). +- **Azure AI Foundry**: the [model retirement schedule](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule), + matched to the OpenAI, Anthropic, Mistral, and Cohere feeds. + +API keys are optional. Without them the refresh still reads every page and keeps +the current models already in the feeds, but it can't add new ones from the +OpenAI, Google, Mistral, or Cohere models endpoints. xAI isn't covered: +it has no lifecycle index, and its models API has no deprecation fields. + +How each page is parsed, and what makes a parser stop, is in +[`refresh/README.md`](../refresh/README.md). + +## The weekly refresh + +[`feed-refresh.yml`](../.github/workflows/feed-refresh.yml) runs every Monday +at 05:23 UTC. It fetches every source, regenerates the feeds, and compares them +with the committed ones. + +- **No material change:** the run records a receipt and the hosted feeds' + `last_checked` time moves forward. Nothing is committed. +- **Material change:** the run opens a PR with a semantic diff (models added, + dates moved, replacements changed) and an updated freshness line in the + README. If an earlier refresh PR is still open, nobody else has pushed to it, + and nobody has approved it or requested changes, the run updates that PR + instead of opening another. `last_checked` doesn't move while a refresh PR is + waiting to merge. +- **Parse failure:** the run fails and opens or updates one issue. It never + writes a partial or guessed feed. The next successful run closes the issue. + +A feed's `generated` date changes only when its data changes. `last_checked` +on the public site shows when the sources were last read. The refresh never +drops a model that is in a committed feed; a parser change that would do so +stops the run. + +The refresh also removes dates that a source withdraws. In August 2026 Google withdrew +the October 16 shutdown dates it had listed for the Gemini 2.5 Pro, Flash, and +Flash-Lite GA models. The refresh opened +[a PR that removed them](https://github.com/thossullivan/model-eol/pull/60), +and downstream checks stopped warning. + +The first feeds were compiled by hand, and the refresh replaced them because +hand-compiled data went wrong quickly. Within the first week of live runs it +caught the Anthropic feed drifting from Anthropic's recommendations and a +Bedrock date that was three months off. + +When every commit since the last release changed only feeds or the README, +the merge is released to npm automatically as a patch version, so +`npx model-eol` picks up new dates without a code release. Otherwise the next +release is dispatched by hand. + +## Running a refresh locally + +```sh +node refresh/refresh.mjs --check # compare live sources with feeds/ +node refresh/refresh.mjs # regenerate the publisher feeds +node refresh/refresh.mjs --distributor aws-bedrock,vertex-ai,azure-ai-foundry # distributor dates +node scripts/feed-changelog.mjs # render the Atom changelog locally +``` + +`--check` exits 0 when nothing material changed, 3 when something did, and 1 +on failure. `--fixtures DIR` runs against recorded pages instead of the +network. That's what the tests use. Never commit feeds generated from fixtures. + +## The public site + +[thossullivan.github.io/model-eol](https://thossullivan.github.io/model-eol/) +publishes the feeds with SHA-256 digests, the JSON schemas under canonical +IDs, an Atom changelog of feed changes, a publication manifest, and refresh +health. Each feed is at +`https://thossullivan.github.io/model-eol/feeds/.json`, for example +[`feeds/openai.json`](https://thossullivan.github.io/model-eol/feeds/openai.json). + +The CLI uses the feeds bundled in its npm package, or a local directory passed +with `--feeds DIR`. The bot can download hosted feeds instead, with +`model-eol-bot --feeds-url URL[,URL...]`. A failed download stops the bot +unless `feeds.allow_vendored_fallback` is set (see +[CONFIGURATION.md](CONFIGURATION.md#keys)). + +Publication is gated. Each live refresh writes a +[feed-refresh receipt](RECEIPTS.md) with the digest of every feed file. The +Pages workflow publishes only when a receipt matches the feeds on `main` +byte for byte, and afterwards it fetches every deployed file and compares it +with what it meant to publish. diff --git a/docs/PRODUCT_PLAN.md b/docs/PRODUCT_PLAN.md deleted file mode 100644 index 0c646a8..0000000 --- a/docs/PRODUCT_PLAN.md +++ /dev/null @@ -1,133 +0,0 @@ -# Product plan - direct-first model retirement inventory - -*Status 2026-08-18: the direct-first product is shipped and live-UAT proven. -The public-contract milestone stages canonical schema IDs and Pages publication, -plus zero-dependency document validation, explicit refresh receipts, status-aware -distributor diffs, and permanent published-consumer UAT. Hosted URLs become -authoritative only after Pages is enabled and its first exact-byte deployment is -green. Account-level resolvers remain deliberate expansion work under "Next -resolvers." Current state lives in docs/CONTEXT.md.* - -## Thesis - -The first useful product is not "one more CI check." It is a direct LLM dependency -inventory that can also fail CI when a visible model ID is near retirement. - -Provider clouds and gateways usually have their own lifecycle surfaces, but those -signals often land with account admins or platform owners, not the repo owner. Static -repo scanning should not pretend it can resolve deployment aliases like `chat-prod` -or gateway routes like `smart-chat`. It should record them and hand them to resolvers. - -## MVP scope - -1. Direct API references are actionable. - - Find tracked model IDs in code/config. - - Classify nearby SDK/API signals such as OpenAI, Anthropic, Gemini, Mistral, - Cohere, and xAI. - - Fail CI only for tracked model IDs that are retired or inside the warning - threshold. - -2. Cloud and gateway references are inventory hints. - - Detect Azure Foundry/OpenAI, Amazon Bedrock, Vertex AI, OpenRouter, LiteLLM, - Portkey, and generic AI gateway signals. - - Mark them as unresolved because static code usually only shows deployment names, - env vars, provider paths, or aliases. - - Keep them in `inventory` and `schedule` output so teams know where provider or - gateway resolvers would add value. - -3. Lifecycle data stays feed-first. - - `feeds/` remains the source of truth for shutdown dates, replacements, aliases, - and per-distributor clocks. - - The scanner consumes feeds; it does not scrape provider pages at runtime. - -## Commands - -```sh -node check.mjs . --days 90 -node check.mjs . --days 90 --scope direct -node check.mjs inventory . --json -node check.mjs schedule . --days 90 -node check.mjs alert . --days 90 --scope direct -node check.mjs . --days 90 --via azure-ai-foundry -node check.mjs validate feeds/openai.json plan.json -``` - -## Mixed repositories and generated code - -Repository-level `days`, `scope`, and `via` values are defaults. A monorepo can -refine them with path-scoped `overrides` and route each reference through the -clock that actually serves it: - -```json -{ - "days": 90, - "scope": "direct", - "overrides": [ - { "paths": ["services/**"], "scope": "all" }, - { "paths": ["services/bedrock/**"], "days": 180, "via": "aws-bedrock" } - ], - "routes": [ - { "paths": ["services/vertex/**"], "via": "vertex-ai" }, - { - "paths": ["services/bedrock/**"], - "match": "chat-prod", - "model": "claude-3-7-sonnet-20250219", - "via": "aws-bedrock" - } - ] -} -``` - -Path globs are repository-relative and support `*`, `**`, and `?`. Matching -overrides apply in array order: later scalar values win, while `ignore.models` -and `ignore.paths` remain additive. The last matching route wins. A route with -an exact `match` and canonical feed `model` turns a known deployment alias into -a tracked, non-direct reference without pretending it is safe to patch. Explicit -CLI flags still override repository and path policy. Machine reports carry the -effective threshold, scope, requested channel, and matching rule indexes when a -path rule or route changed behavior. - -The primary `check --json` report has its own public discriminator and Draft-07 -schema. CycloneDX exports identify components by canonical model plus requested -lifecycle channel, using `publisher-direct` for the un-routed publisher clock so -it cannot collide with a custom distributor named `publisher`. Azure, Bedrock, -and other clocks for one model cannot overwrite each other; occurrences remain -attached to the channel used. - -The scanner reads `.baml` source but skips conservative generated-code headers, -BAML-generated clients, and model-eol check/inventory/schedule/alert/plan artifacts. -CycloneDX is skipped only when its metadata carries model-eol generator -provenance, so third-party BOMs stay visible. Intentional artifact skips are -diagnostics, not incomplete-coverage warnings. - -## Next resolvers - -Add optional resolvers only when they can return exact deployed model/version data: - -- Azure Foundry: resolve deployment name -> model/version/update policy. -- Amazon Bedrock: resolve model ID and lifecycle from Bedrock API/account context. -- Vertex AI: resolve endpoints, publisher model names, and lifecycle pages. -- Gateways: resolve route alias -> provider/model from config or gateway API. - -Resolvers should enrich inventory output; they should not be required for the direct -API checker to work. - -## Alerting - -Alerting should consume `schedule --json`, not duplicate scanner logic. - -The first built-in alert target is `node check.mjs alert`, which can emit GitHub -Actions annotations or Markdown. Anything stateful, such as opening GitHub issues or -sending Slack messages, should consume `alert --json` or `schedule --json`. - -Useful first targets: - -- GitHub Actions annotations for PR/build feedback. -- Markdown summary artifact for humans. -- GitHub issue per repo/service owner. -- Slack/Teams message for findings inside a configured window. -- CI artifact containing `model-inventory.json`. - -For direct-first workflows, use `--scope direct` so cloud/gateway references remain -inventory hints until a resolver can prove the exact deployed model. The historical -`check` behavior remains available as `--scope all`. diff --git a/docs/RECEIPTS.md b/docs/RECEIPTS.md new file mode 100644 index 0000000..655f023 --- /dev/null +++ b/docs/RECEIPTS.md @@ -0,0 +1,118 @@ +# Receipts and `verify` + +model-eol writes three kinds of receipt. Each one records what a step did and +binds it to exact inputs by digest, so someone else can check the claim later +without trusting the run that wrote it. + +| Receipt | `schema` value | Written by (artifact name) | What it binds | +|---|---|---|---| +| [Eval results manifest](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.eval.schema.json) | `model-eol.eval/0.1` | `model-eol-bot evaluate` (`model-eol-eval-results`) | One pass, fail, or timeout result per migration, tied to the evaluated base commit, the plan digest, the eval settings digest, and each migration's feed digest. | +| [Feed-refresh receipt](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.feed-refresh-receipt.schema.json) | `model-eol/feed-refresh-receipt@0.1` | the weekly feed refresh (`feed-refresh-receipt`) | The run URL, the checked commit, and the check time, tied to the SHA-256 of every `feeds/*.json` file. A pending receipt also names the open feed PR. | +| [npm release result](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.npm-release-result.schema.json) | `model-eol/npm-release-result@0.2` | the release workflow (`npm-release-result`) | The source commit, and either "no release" or the exact version, tag, release commit, and sha512 integrity of the registry tarball. | + +Each receipt is uploaded as a workflow artifact with the name shown, so you can +fetch one with +`gh run download RUN_ID --repo OWNER/REPO --name ARTIFACT_NAME`. + +There are two commands for receipts: + +- `model-eol validate receipt.json` checks structure only. It picks the schema + from the receipt's `schema` value. +- `model-eol verify receipt.json` checks what the receipt claims against files + and values you supply. + +## Examples + +```bash +# a release result, against the exact tarball you hold (no registry lookup) +npm pack model-eol@X.Y.Z +model-eol verify npm-release-result.json --package model-eol-X.Y.Z.tgz \ + --commit RELEASE_SHA --source-commit SOURCE_SHA --version X.Y.Z --require-all + +# a refresh receipt, against a checkout's feeds directory +model-eol verify feed-refresh-receipt.json --feeds-dir ./feeds \ + --commit REFRESH_SHA --run-url https://github.com/OWNER/REPO/actions/runs/ID + +# an eval manifest, against the plan, config, and feeds the evaluator used +model-eol verify eval-results.json --commit BASE_SHA --plan plan.json \ + --config .model-eol.json --feeds-dir ./feeds +``` + +`verify` first validates the receipt's structure, then reports on each claim. +It never reads the current directory, Git state, or the network. Every input is +an explicit flag. + +## Flags + +| Flag | Checks | +|---|---| +| `--feeds-dir DIR` | Refresh receipts: the SHA-256 of every `DIR/*.json`, compared with the receipt's list as a set, so changed, missing, and extra files are all mismatches. `DIR` must be the feeds directory itself; a repository root is refused. Eval manifests: together with `--plan` and `--config`, recomputes each result's `feed_digest`. | +| `--commit SHA` | `refresh_commit`, `release_commit`, or the eval manifest's `base_sha`. | +| `--run-url URL` | `refresh_run`, after the same URL normalization the writer applies. | +| `--package FILE.tgz` | `registry_integrity`: the sha512 SRI of the tarball's bytes. | +| `--source-commit SHA` | `source_commit` of a release result. | +| `--version X.Y.Z` | `version` of a release result. `tag` must then be `vX.Y.Z`. | +| `--plan FILE` | The eval manifest's `plan_digest`, from a valid `model-eol.plan/0.1` document. | +| `--config FILE` | The eval manifest's `eval_config_digest` and `configured` value, resolved the same way the bot resolves settings. `MODEL_EOL_EVAL_COMMAND` in the environment overrides `eval.command`, and the output says when it did. For a repository with no `.model-eol.json`, pass a file containing `{}`. | +| `--require-all` | Exit 1 when the result is `partial`. | +| `--json` | Print a [`model-eol/verification@0.1`](https://thossullivan.github.io/model-eol/schema/0.1/model-eol.verification.schema.json) document. | + +A SHA is 40 to 64 hex characters, and uppercase is accepted. A flag that +doesn't apply to the receipt's kind is a usage error. + +## Results + +Each claim comes back as one of: + +- `verified`: the receipt matches what you supplied. +- `mismatch`: it doesn't. The output shows both values. +- `not-checked`: you didn't pass the flag that would check it. The output names + the flag. +- `not-applicable`: the claim doesn't exist for this receipt. A release result + with `released: false` makes no release claims, so only `source_commit` + applies. + +The overall result is `verified` when every applicable claim is verified, and +`mismatch` when any claim mismatched. It is `partial` when nothing mismatched +but something wasn't checked: the receipt agrees with what you gave it, but you +haven't proven all of it. + +Some details matter if you build a gate on this: + +- **Release gates** should pass `--package`, `--version`, `--commit`, and + `--require-all`. If you pass `--commit`, `--version`, or `--package` for a + `released: false` receipt, verify adds a `released` claim that mismatches, + because those inputs say a release happened. Without them, a no-release + receipt could verify through `--source-commit` alone. +- **Eval manifests**: `--plan` and `--config` together fix which results the + evaluator had to produce. A missing result, a result nobody planned, and a + duplicated result are each a mismatch. Without both flags the `results` claim + is `not-checked`, so a manifest with results removed can't verify. +- **Refresh receipts** show their `state`, `checked_at`, and `pending_pr` for + information. A pending receipt verifies by the same rules as a clean one. +- **Eval outcomes are not claims.** `verify` doesn't check `status`, + `exit_code`, `report`, or `generated`, because an eval can't be rerun + offline. A `verified` eval manifest is bound to the commit, plan, config, and + feeds you supplied. It does not mean the evals passed. A gate that needs + passing evals has to read the results itself. + +Text output escapes control characters in every value it prints. + +## Exit codes + +| Exit | Meaning | +|---|---| +| 0 | Nothing mismatched: `verified`, or `partial` without `--require-all`. | +| 1 | A claim mismatched, or the result was `partial` with `--require-all`. | +| 2 | Usage error, unreadable file, structurally invalid receipt, or an unsupported schema or version. No claims are reported. | +| 3 | Writing the output failed, when the result would otherwise have exited 0. | + +## Versioning + +Adding an optional field keeps a receipt's version. Anything that would make an +old receipt invalid, or that changes what a digest covers, gets a new version. +Validators reject versions they don't know. + +A few rules can't be written in JSON Schema Draft-07, such as the sort order of +feeds or `tag` being `v` plus `version`. Each schema's `description` lists +them, and `validate` enforces them in code. diff --git a/docs/RESEARCH.md b/docs/RESEARCH.md index 92ac96f..a46c4fa 100644 --- a/docs/RESEARCH.md +++ b/docs/RESEARCH.md @@ -1,88 +1,111 @@ -# Research receipts (public sources, compiled 2026-07-25) +# Background research -The evidence base this project stands on. Every claim carries its URL; verify dates -against primary sources before relying on them - several were compiled the week of -the July 2026 shutoff wave. +Public sources behind the project's claims, compiled in July 2026 and updated +since. Dates in this file are as published when they were collected. The feeds +carry the current values, so check those, or the primary source, before relying +on a date here. -## Deprecation events (selected timeline) +## Retirement history -| Provider | Model(s) | Notice window | Shutoff | +| Provider | Models | Notice | Shutdown | |---|---|---|---| -| OpenAI | Codex API | **~3 days** | 2023-03-23 | -| OpenAI | GPT-3 bases + all fine-tunes | ~6 mo | 2024-01-04 | -| OpenAI | GPT-4.5-preview (API) | 3 mo | 2025-07-14 (~4.5 mo after launch) | -| OpenAI | GPT-4o pulled from ChatGPT | **0 days** | 2025-08-07 (restored days later) | -| Anthropic | Claude 2/2.1/Sonnet 3 | 6 mo | 2025-07-21 | -| Anthropic | Claude 3.7 Sonnet | ~4 mo | 2026-02-19 (Bedrock: 2026-04-28) | -| OpenAI | o3-deep-research + 17 others | 3 mo | 2026-07-23 | -| OpenAI | gpt-4/o1/o3-mini wave | 6 mo | 2026-10-23 | -| OpenAI | o3, o3-pro, gpt-5 snapshots | 6 mo | 2026-12-11 | - -- OpenAI deprecations page (policy floors: 6 mo GA / 3 mo specialized / ~2 wk - preview): https://developers.openai.com/api/docs/deprecations -- Anthropic policy (60-day floor, lifecycle states): - https://platform.claude.com/docs/en/about-claude/model-deprecations -- Azure (12 mo from launch + 60 days notice; keeps models past OpenAI's own dates): +| OpenAI | Codex API | about 3 days | 2023-03-23 | +| OpenAI | GPT-3 base models and all their fine-tunes | about 6 months | 2024-01-04 | +| OpenAI | GPT-4.5-preview (API) | 3 months | 2025-07-14, about 4.5 months after launch | +| Anthropic | Claude 2, 2.1, and Sonnet 3 | 6 months | 2025-07-21 | +| OpenAI | GPT-4o removed from ChatGPT | none | 2025-08-07, restored days later | +| Anthropic | Claude 3.7 Sonnet | about 4 months | 2026-02-19 (Bedrock listed 2026-04-28 at the time) | +| OpenAI | o3-deep-research and 17 others | 3 months | 2026-07-23 | +| OpenAI | gpt-4, o1, o3-mini wave | 6 months | 2026-10-23 | +| OpenAI | o3, o3-pro, gpt-5 snapshots | 6 months | 2026-12-11 | + +## Provider lifecycle policies + +- **OpenAI** gives about 6 months' notice for GA models, 3 months for + specialized models, and about 2 weeks for previews: + https://developers.openai.com/api/docs/deprecations +- **Anthropic** commits to at least 60 days' notice and defines its lifecycle + states: https://platform.claude.com/docs/en/about-claude/model-deprecations +- **Azure** retires standard GA models 18 months after launch (closed + to new customers after 12), with 60 days' notice, and often serves them past + OpenAI's own dates. Anthropic, DeepSeek, Fireworks, and Mistral models on + Azure follow a 12-month lifecycle. Policy: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirements - The old `ai-foundry` URL redirects to this lifecycle policy article. -- Mistral (deprecation and retirement days per model, product-name alternatives): - https://docs.mistral.ai/models -- Cohere (dated announcements; deprecated means no new customers, shutdown assigned later): - https://docs.cohere.com/docs/deprecations -- xAI: no lifecycle index. Retirements appear only on per-event migration pages - (https://docs.x.ai/developers/migration/may-15-retirement, - https://docs.x.ai/developers/migration/imagine-image-quality-nov-2) and the - models API has no deprecation fields. Not ingested; Azure carries grok clocks - for Foundry only. -- Bedrock changed its lifecycle policy on 2026-09-07. - Models launched earlier retain the [legacy lifecycle table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html). - Newer models publish lifecycle dates on [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html). - Refresh reads both sources; the legacy table wins overlaps, and card not-before floors become tentative distribution dates. -- Cross-provider calendars (community scrapers - the demand signal): - https://benchr.org/deprecations and - https://hidekazu-konishi.com/entry/ai_model_deprecation_and_lifecycle_calendar.html - -## Behavior drift without version change (why pinning isn't enough even pre-shutoff) - -- Chen, Zaharia, Zou - GPT-4 prime-identification accuracy 97.6% -> 2.4% between - March and June 2023 versions, same endpoint: https://arxiv.org/abs/2307.09009 -- NTNU supply-chain paper - hosted models "behavior may change without version or - endpoint changes": https://arxiv.org/html/2604.27789v1 -- Anthropic postmortem (2025): infra bugs silently degraded output, up to 16% of - requests affected: referenced in the paper above; anthropic.com/engineering - -## Migration cost (why a CI gate pays for itself) - -- 58.8% of prompt+model combinations lost accuracy across API updates: + (the older `ai-foundry` URL redirects here). Schedule: + https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule +- **Mistral** lists deprecation and retirement dates per model, with + alternatives by product name: https://docs.mistral.ai/models +- **Cohere** dates its announcements. "Deprecated" means no new customers, and + a shutdown date is assigned later: https://docs.cohere.com/docs/deprecations +- **Amazon Bedrock** changed its lifecycle policy on 2026-09-07. Models launched + before then stay in the + [legacy lifecycle table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html). + Newer models publish dates on their + [model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html). +- **xAI** has no lifecycle index. Retirements appear only on one-off migration + pages (https://docs.x.ai/developers/migration/may-15-retirement, + https://docs.x.ai/developers/migration/imagine-image-quality-nov-2), and the + models API has no deprecation fields. Azure lists Grok dates for Foundry + only. + +Community calendars that track several providers, which show the demand for +shared data: https://benchr.org/deprecations and +https://hidekazu-konishi.com/entry/ai_model_deprecation_and_lifecycle_calendar.html + +## Behavior changes without a version change + +Pinning a model ID doesn't freeze its behavior, even before shutdown. + +- Chen, Zaharia, and Zou measured GPT-4's accuracy at identifying primes + falling from 97.6% to 2.4% between the March and June 2023 versions of the + same endpoint, in the first version of their paper: + https://arxiv.org/abs/2307.09009v1. The current revision reports 84% to 51% + for this comparison. +- A supply-chain study notes that hosted models' "behavior may change without + version or endpoint changes": https://arxiv.org/html/2604.27789v1 +- Anthropic's 2025 postmortem describes infrastructure bugs that degraded + output quality, affecting up to 16% of Sonnet 4 requests in the worst hour: + https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues + +## Migration cost + +- 58.8% of prompt and model combinations lost accuracy across API updates: https://arxiv.org/pdf/2311.11123 -- Underspecified prompts regress ~2x more than explicit ones: +- Underspecified prompts regress about twice as often as explicit ones: https://arxiv.org/pdf/2505.13360 -- Documented migration: several months ad hoc -> two weeks with a testbed: - https://arxiv.org/html/2507.05573v1 -- One estimate of the standing tax: ~40 engineer-weeks/year for a five-feature team - (single source, hedge): https://tianpan.co/blog/2026-04-27-model-deprecation-treadmill-pre-sunset-discipline +- One documented migration went from several months of ad hoc work to two weeks + with a test bed: https://arxiv.org/html/2507.05573v1 +- One blog post estimates the ongoing cost at about 40 engineer-weeks a year + for a team running five LLM features. It is a single, unreviewed estimate: + https://tianpan.co/blog/2026-04-27-model-deprecation-treadmill-pre-sunset-discipline -## Standards prior art +## Standards and prior art - CycloneDX ML-BOM (v1.5, June 2023): https://cyclonedx.org/capabilities/mlbom/ -- SPDX 3.0 AI profile + LF implementation guide: +- SPDX 3.0 AI profile and the Linux Foundation implementation guide: https://www.linuxfoundation.org/hubfs/LF%20Research/lfr_spdx_aibom_102524a.pdf -- endoflife.date (the shape of the solution): https://endoflife.date/ -- OSV (shared schema turning N scrapers into an ecosystem): https://osv.dev/ -- Anthropic deprecation commitments (weights preserved "for, at minimum, the - lifetime of Anthropic as a company" - preservation is not availability): +- endoflife.date: https://endoflife.date/ +- OSV: https://osv.dev/ +- Anthropic's deprecation commitments say it will preserve model weights "for, + at minimum, the lifetime of Anthropic as a company". Preserving weights is + not the same as keeping a model available: https://www.anthropic.com/research/deprecation-commitments -## The July 2026 event specifics (this project's founding receipt) +## The July 2026 shutdown + +The event that started this project. -- Notice 2026-04-22; shutoff 2026-07-23; recommended replacement gpt-5.6-sol, which - did not GA until 2026-07-09 - the migration target changed mid-window - ("treat that mapping as a snapshot in time, not a constant"): +- OpenAI announced the shutdown on 2026-04-22 and shut the models off on + 2026-07-23. The recommended replacement, gpt-5.6-sol, wasn't generally + available until 2026-07-09, so the migration target changed partway through + the notice period. One write-up advised readers to "treat that mapping as a + snapshot in time, not a constant": https://privatedevops.com/news/openai-retires-gpt-5-o3-snapshots-december-2026 -- Deep-research models were post-trained research agents with no structural - successor (community analysis): +- The deep-research models were post-trained research agents with no direct + successor (community discussion): https://community.openai.com/t/o4-mini-deep-research-o3-deep-research-deprecation/1379560 -- Azure keeps o3-deep-research until 2026-12-26 (retirement table): +- Azure listed o3-deep-research until 2026-12-26 when this was compiled. The + current feed shows 2026-11-19: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule -- Shutdown-day failure mode: "a production 404 on the shutdown day": - https://ecorpit.com/openai-model-shutdowns-23-july-2026-migration-map/ +- A migration guide described the failure as "a production 404 on the shutdown + day" (secondary source): https://ecorpit.com/openai-model-shutdowns-23-july-2026-migration-map/ diff --git a/docs/img/demo.gif b/docs/img/demo.gif index f61bb2d..960203b 100644 Binary files a/docs/img/demo.gif and b/docs/img/demo.gif differ diff --git a/docs/vhs/demo.tape b/docs/vhs/demo.tape index 00a841a..a56a19c 100644 --- a/docs/vhs/demo.tape +++ b/docs/vhs/demo.tape @@ -1,36 +1,42 @@ -# VHS tape: model-eol 30-second demo -# Run from repo root: vhs docs/vhs/demo.tape +# model-eol demo for the README +# Run from the repo root: vhs docs/vhs/demo.tape +# Output depends on today's date, so re-record when the README sample changes. Output docs/img/demo.gif Set Shell "bash" -Set FontSize 20 -Set Width 1400 -Set Height 660 +Set FontSize 16 +Set Width 1700 +Set Height 320 Set Theme "Catppuccin Mocha" Set Padding 24 Set TypingSpeed 40ms Set WindowBar Colorful Set WindowBarSize 40 -Type "# which of my models are already dead?" +Hide +Type `export PS1='$ '; alias model-eol="node $PWD/check.mjs"; cd test/fixture; clear` Enter -Type "node check.mjs test/fixture --days 90" +Show + +Type "# check one file against the publishers' dates" +Enter +Type "model-eol app.py --days 90" Enter Sleep 4.5s Type "clear" Enter -Type "# same repo, judged by Amazon Bedrock's clock instead" +Type "# the same file, using Amazon Bedrock's dates" Enter -Type "node check.mjs test/fixture --days 90 --via aws-bedrock" +Type "model-eol app.py --days 90 --via aws-bedrock" Enter Sleep 4.5s Type "clear" Enter -Type "# worst case date for the whole repo" +Type "# the earliest retirement in the file" Enter -Type "node check.mjs schedule test/fixture --days 90 | tail -2" +Type "model-eol schedule app.py --days 90 | tail -1" Enter -Sleep 4s +Sleep 3.5s diff --git a/refresh/README.md b/refresh/README.md new file mode 100644 index 0000000..1968165 --- /dev/null +++ b/refresh/README.md @@ -0,0 +1,315 @@ +# refresh + +This directory regenerates `feeds/*.json` from provider pages, models +endpoints, and distributor lifecycle pages. For what the feeds contain and how +the weekly workflow uses this code, see [docs/FEEDS.md](../docs/FEEDS.md). + +| File | Role | +|---|---| +| `refresh.mjs` | Entry point. Fetches, parses, merges, writes, and prints the semantic diff. | +| `providers.mjs` | Publisher parsers: OpenAI, Anthropic, Google, Mistral, Cohere. | +| `distributors.mjs` | Distributor parsers: Bedrock, Vertex AI, Azure AI Foundry. | +| `fetch.mjs` | Bounded HTTP fetches (30-second deadline, 8 MiB limit) for Anthropic model pages and distributor pages. | +| `diff.mjs` | The semantic diff used for `--check` and for refresh PR bodies. | +| `test/` | Offline tests and recorded pages under `test/fixture/`. | + +## The rule every parser follows + +A parser that isn't sure must stop the run. It never returns a partial or +best-guess result, so the refresh either writes a complete, validated feed or +nothing: + +- Every model ID that enters a feed passes the identifier grammar. +- Conflicting records for one model stop the run, unless a precedence rule + below says which source wins. +- A page on which no lifecycle table is recognized stops the run. +- A missing fixture page stops an offline run. It is never treated as empty. + +A few rows are skipped rather than stopping the run, and the sections below +say which: OpenAI endpoint and product rows, Gemini cells without a usable ID, +and Vertex IDs with whitespace. A Cohere section with only prose yields nothing +without stopping. + +How strictly tables are identified also differs. Mistral, Cohere, the Bedrock +model cards, and Azure check lifecycle headers exactly and stop on drift. +OpenAI, the Gemini API, the Bedrock legacy table, and Vertex AI find columns by +label patterns: tables without a matching header are skipped, and extra columns +are accepted. + +Every fetch has a 30-second deadline over the request and body, and an 8 MiB +body limit (`fetch.mjs`). Requests that carry an API key (the models endpoints) +refuse redirects, so a key never follows a redirect to another origin; page +fetches follow redirects. Error messages include at most 300 characters of the +error body on one line, with request header values redacted. Paged models +endpoints stop the run if they still have more pages after 20; Mistral's is +read in one request. + +A change that makes a parser accept more input has to show that the new input +can't produce a row bound to the wrong model. Add a fixture that would have +produced a misaligned row and assert that it is refused. + +## Anthropic + +The refresh reads the Anthropic model deprecations page for dates and the +model overview pages for IDs and aliases. + +- Dates come only from the + [model deprecations page](https://platform.claude.com/docs/en/about-claude/model-deprecations). + Its "not sooner than" dates for active models become `tentative` shutdowns. +- Model IDs and aliases come from the overview pages listed in + `https://platform.claude.com/docs/llms.txt`. Only `Claude API` and + `Claude API alias` rows are imported. Those pages supply no dates. +- The index may list at most 100 unique pages. Each fetch has a 30-second + deadline and an 8 MiB limit. +- When an alias is attached to a model or moves to another one, the run emits a + notice and the change appears in the semantic diff. +- Offline runs read `test/fixture/anthropic-llms.txt` and + `test/fixture/anthropic-model-pages/.md`. + +## OpenAI + +The refresh reads the +[OpenAI deprecations page](https://developers.openai.com/api/docs/deprecations) +and expects tables with a shutdown date column, a model column, and a +replacement column, each under a heading that carries its announcement date. + +- The URL fetched is the `source` of the committed `feeds/openai.json`. The + constant in `providers.mjs` is used only when the feed has no `source`. +- `script`, `style`, `nav`, `header`, and `footer` elements are removed before + headings and tables are read. +- The header is the first row with three labels: a shutdown, retirement, + sunset, or removal date; a "model" or "system" label that doesn't mention a + replacement; and a replacement, substitute, or successor label. +- A table whose date label sits beside an endpoint, product, service, + operation, or capability label is skipped as an endpoint table. +- The announcement date comes from the nearest heading above the table, or the + nearest earlier higher-level heading if that one has no date. A table with + no dated heading above it stops the run. +- Model IDs come from the `code` elements in the model cell, or from the cell + text when there are none. A token with a `YYYY-MM-DD` date or a valid `-MMDD` + suffix (optionally followed by `-preview`) is a snapshot and becomes the + record ID. The other tokens become its aliases. +- A cell with two or more snapshots plus an undated ID stops the run, because + the alias can't be assigned to one snapshot. +- A row is skipped only when every token is endpoint- or product-shaped + (`/vN/...`, `Name API`, `OpenAI-Beta: name=vN`). Any other token that fails + the ID grammar, or a real model ID beside an endpoint token, stops the run. +- Every row needs a shutdown date no earlier than the announcement. OpenAI + records never set `date_precision`. +- Replacement cells: `-`, `N/A`, and `none` give nothing. One code token becomes + `replacement`, several become `replacement_options`, and leftover prose + becomes `replacement_note`. A single replacement stays `replacement` only if + it resolves to an ID or alias in the feed; otherwise it becomes a one-item + `replacement_options`. +- Duplicate rows for one ID: identical dates merge aliases and replacements; + different announcement dates keep the later announcement, unless the dropped + row names an alias the kept row doesn't, which stops the run; the same + announcement with different shutdown dates stops the run. +- A row listed as `alias | snapshot`, where the snapshot ends in `-YYYY-MM-DD` + and the alias is that ID without the date, gives both committed entries the + row's lifecycle and merges neither. Two rows giving the alias different + dates, precision, or replacement targets stop the run, and so does a row + whose replacement is the alias or the alias's own newer snapshot. +- The models endpoint (`https://api.openai.com/v1/models`, `OPENAI_API_KEY`) is + paged for up to 20 pages. A repeated ID or a cursor that doesn't advance stops + the run. Without the key, committed current-model entries are kept. +- Offline runs read `test/fixture/openai-deprecations.html` and + `test/fixture/openai-models.json`. + +## Google (Gemini API) + +The refresh reads the +[Gemini API deprecations page](https://ai.google.dev/gemini-api/docs/deprecations) +and expects tables with a model column, a shutdown date column, and a +replacement column. The page has no announcement dates. + +- The URL fetched is the `source` of the committed `feeds/google.json`, with the + constant as a fallback. +- The header is the first row of a table, in `th` or `td` cells, with a + shutdown date label, a "model" or "system" label, and a replacement or + successor label. Endpoint tables are skipped as for OpenAI. +- The model ID is the first `code` element in the model cell, or the cell text + when it has no spaces. Rows without a usable ID or without a shutdown cell are + skipped. An ID that fails the grammar stops the run. The page supplies no + aliases. +- An empty, `N/A`, dash, or "No shutdown date (announced)" cell gives no + shutdown. Any other text must contain a date or the run stops. +- `date_precision` is `earliest` when "earliest possible", "not moved to an + earlier date", "or later", or "no sooner than" appears in the shutdown cell, + or in the prose of the table's section or any enclosing section (tables, + comments, and scripts excluded). The page's single note sits under the h1, so + today it covers every table. +- Replacement cells follow the OpenAI rules. +- Duplicate rows for one ID must agree on shutdown, precision, and replacement, + or the run stops. +- The models endpoint + (`https://generativelanguage.googleapis.com/v1beta/models`, `GEMINI_API_KEY`, + or `GOOGLE_API_KEY` when running locally) is paged, up to 20 pages. Names + must start with `models/`, which is stripped. A repeated page token stops the + run. +- Offline runs read `test/fixture/google-deprecations.html` and + `test/fixture/google-models.json`. + +## Mistral + +The refresh reads the [Mistral models page](https://docs.mistral.ai/models) +and expects lifecycle tables whose header is exactly five `th` cells (Model, +Version, API, Deprecation Retirement, Alternative), with each Deprecation +Retirement cell holding two M/D/YYYY dates. + +- A lifecycle header row that contains data cells or unexpected labels stops + the run. +- Mistral sometimes puts the header row in one table and the data rows in the + next. A table with a valid header and no data rows holds its header for the + next table. That table must come before any heading, have no header of its + own, and contain only `td` cells, or the run stops. A held header that is + never used also stops the run. +- A row counts as a lifecycle header when a label mentions deprecation or + retirement, or when two or more labels are exactly Model, Version, API, or + Alternative. In a table with `th` cells, every row with a `th` cell is + checked. In a table without `th` cells that isn't receiving a held header, + only the first row is checked, so a header written in `td` cells stops the + run. + +## Cohere + +The refresh reads the +[Cohere deprecations page](https://docs.cohere.com/docs/deprecations) and +expects a `Deprecation History` heading whose subsections each start with a +`YYYY-MM-DD:` date. A subsection holds an "Effective , the following +models will be retired:" list, a "Deprecated Models:" list, or tables headed +Shutdown Date, Deprecated Model, and Recommended Replacement. + +- Identities and aliases are collected from every section first. Records are + then merged in ascending order of the section's heading date. + +## Amazon Bedrock + +AWS changed its lifecycle policy on 2026-09-07. Models launched before then +stay in the +[legacy table](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html). +Newer models publish dates on individual +[model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html). +The refresh reads both. + +Precedence: + +- When the legacy table and a card cover the same model, the legacy table wins. + This also applies when two different Bedrock IDs resolve to the same + publisher model through aliases. +- Conflicting exact dates are listed under "Source conflicts" in the diff, and + the legacy table's date is used. +- A card's "EOL no sooner than" floor becomes a tentative `distributions` entry + with no `announced` date. An exact Model EOL date stays exact. +- When one card gives two different floors, the later day-precision date + applies. Two cards that disagree about the same model stop the run, including + after alias resolution. + +Card parsing: + +- Each card must contain exactly one Model ID table. A table is that table when + any cell in its first row, `th` or `td`, is exactly `Model ID`. +- HTML comments are removed before extraction. +- A lifecycle field is read only from a paragraph that starts with the + exact-case label and a colon. A label found anywhere else on the card, matched + as a whole label ignoring case and colon, stops the run. +- A standalone Model EOL date counts as lifecycle data. `N/A` and + `Not announced` count as no date. +- The feed has no Region dimension, so a card whose lifecycle reads `Legacy in + ; Active in other Regions` is legacy, with the EOL date given as + ` ( only)`. The two Region lists must match. Every other + region-qualified form stops the run. +- Cards with no lifecycle fields or no day-precision date are listed under + "Model cards without lifecycle fields" and emit notices. +- Links from the card index resolve inside the index's directory after the + query and fragment are removed. An invalid link stops the run. + +Limits: at most 400 unique cards, fetched one at a time, each with a 30-second +deadline and an 8 MiB limit. + +Offline runs read `bedrock-lifecycle.html`, `bedrock-model-cards.html`, and +`bedrock-model-cards/`. A missing card fixture stops the run. + +## Vertex AI + +The refresh reads the +[Vertex AI model versions page](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/model-versions) +and expects tables with a Model ID column and a retirement date column. It runs +only with `--distributor vertex-ai`. + +- Row and column spans are expanded before columns are indexed. +- The header is the first row with a "model ID" or "model identifier" label and + a retirement or discontinuation date label. Other tables are skipped and extra + columns are accepted. +- The model ID is the first `code` element in the model cell, or the cell text, + with trailing asterisks removed. Values containing whitespace are skipped. + Vertex rows never create feed entries, so no other ID check runs. +- A row with no retirement cell stops the run. Empty, dash, `N/A`, `na`, + `none`, "not available", "not applicable", and "no shutdown (or retirement) + date announced" cells give no date. Any other + text must contain a date or the run stops. +- `date_precision` is `earliest` under the same section-scoped test as the + Gemini API, plus "retirement timelines may be extended". The walk up the + sections stops at the retired-models archive (an `id` of `retired-models`, or + a heading reading "Retired models" or "The following table lists the retired + models"), so archive rows keep plain dates. Any other heading mentioning + retirement beneath a qualifier stops the run. +- A model listed with the same date in an available table and the archive keeps + the available row's precision. Any other conflict stops the run and names + both rows. +- A `publishers//models/` prefix is stripped, and its namespace binds the + row to that publisher's feed. A bare ID binds to whichever feed has it as an + ID or alias. An ID shared by two feeds stops the run. +- Rows that match no feed entry are listed in the diff and create nothing. Two + rows that resolve to one model with different dates stop the run. +- A matched model gets one `vertex-ai` distribution. Committed `vertex-ai` + entries the page no longer confirms are kept and reported as unconfirmed. The + merge may change only `vertex-ai` distributions; any other change stops the + run. +- Offline runs read `test/fixture/vertex-model-versions.html`. + +## Azure AI Foundry + +- Rows bind to publisher feeds through the Azure OpenAI, Anthropic, Mistral AI, + and Cohere section headings. +- OpenAI rows first try exact snapshot IDs and aliases. A bare model ID is used + only for models with a single row and no dated snapshot candidate. +- For OpenAI, four-digit versions need an exact snapshot ID or alias, and + versions `1`, `2`, `001`, and `-` use the bare ID. Mistral and Cohere + integer versions, including four-digit ones, and `-` use the bare ID. +- Cohere binding strips one leading `Cohere-` or `cohere-` prefix and keeps the + case of the rest. +- Model cells are counted before deduplication. Replacement cells are ignored. +- Rows that can't be bound stay unconfirmed. They appear in the diff but don't + count as a material change. +- Matching rows for different hosting options collapse into one. For Azure + OpenAI and Anthropic rows with two different retirement dates, the earlier + date is kept and listed under "Source conflicts". Conflicting Mistral or + Cohere rows stop the run. +- Fine-tuning tables and other unrelated tables are skipped. Header drift in a + lifecycle table stops the run. +- Azure's status labels are mapped to `distributions[].status` as described in + [SPEC.md](../SPEC.md#distributions). + +## All distributor tables + +- A span over 64 rows or columns, or a table over 10,000 cells, stops the run. +- When a refresh PR opens for another change, its body also lists source + conflicts. Conflicts alone don't open a PR. + +## Adding a source + +Start from the closest existing parser and keep its shape: strict table +identity, ID grammar checks, and a thrown error whenever the page doesn't look +the way the parser expects. + +1. Record the live page as a fixture under `test/fixture/`. +2. Write tests against the fixture. `node refresh/test/run.mjs` runs only the + refresh tests. +3. Regenerate the feeds from the live pages with `node refresh/refresh.mjs` + (add `--distributor ...` for a distributor parser), and include the semantic + diff in the PR description. Don't commit feeds + generated with `--fixtures`. + +The test rules in [CONTRIBUTING.md](../CONTRIBUTING.md#tests) apply: no +network, and no assertions on live feed values. diff --git a/scripts/test-verify.mjs b/scripts/test-verify.mjs index f7a97bf..d122768 100644 --- a/scripts/test-verify.mjs +++ b/scripts/test-verify.mjs @@ -491,8 +491,8 @@ try { for (const text of ['A release gate passes --package, --version', 'verify RECEIPT', '--feeds-dir DIR', '--commit SHA', '--run-url URL', '--package FILE.tgz', '--source-commit SHA', '--version X.Y.Z', '--require-all', 'partial', 'mismatch']) { assert(help.stdout.includes(text), `help documents ${text}`) } - const readme = fs.readFileSync(path.join(root, 'README.md'), 'utf8') - assert(!readme.includes('arrives in a later release') && readme.includes('model-eol verify') && readme.includes('--require-all') && readme.includes('`partial`'), 'README documents verify instead of promising it later') + const receiptsDoc = fs.readFileSync(path.join(root, 'docs', 'RECEIPTS.md'), 'utf8') + assert(!receiptsDoc.includes('arrives in a later release') && receiptsDoc.includes('model-eol verify') && receiptsDoc.includes('--require-all') && receiptsDoc.includes('`partial`'), 'docs/RECEIPTS.md documents verify instead of promising it later') assert(reports >= 35, `every non-refused verify run emitted a valid ${VERIFICATION_SCHEMA} document (${reports} checked)`) } finally { fs.rmSync(tempRoot, { recursive: true, force: true })