Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/feed-refresh.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Weekly feed refresh - regenerates feeds/ from the live provider pages and the
# AWS Bedrock and Google Vertex AI lifecycle pages, updates the README freshness
# AWS Bedrock, Google Vertex AI, and Azure Foundry lifecycle pages, updates the README freshness
# line, and opens a PR with the semantic diff for human review. Never pushes to
# main directly. PR reuse is ownership-checked and protected by an exact lease.
#
Expand Down
45 changes: 45 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Agent instructions

Read [CONTRIBUTING.md](CONTRIBUTING.md) first. Its layout table, test rules,
conventions, list of docs that tests check, and design decisions all apply to
agents. This file adds what an agent needs on top of that.

## Before you finish a change

1. Run `npm test`. It must pass offline.
2. For refresh changes, run `node refresh/refresh.mjs --check` against the live
sources and read the semantic diff. If you regenerated feeds, confirm
`--check` then exits 0.
3. For scanner or `--changed` changes, run `scripts/e2e-changed.mjs` as shown
in CONTRIBUTING.md.
4. Read your own diff for stacked `//` comment blocks, em-dashes, and tests that
assert live feed values. These are the mistakes agents make most often here.

## Feeds

- Never edit `feeds/*.json` by hand.
- If you have no network access, prove a parser change against fixtures and
leave the committed feeds untouched. Say in your report that the feeds need a
live regeneration. Never commit feeds generated with `--fixtures`.

## Commits

- No `Co-Authored-By` or other model-attribution trailers. Put
"Implemented by <model>; reviewed by <model>" in the commit body instead.
- Rationale goes in the commit body, not in code comments.
- Don't commit unless you were asked to.

## Releases and publishing

These steps are usually done by the maintainer, but agents are sometimes asked
to run them.

- A merge that changes only `feeds/` and the README freshness line releases a
patch to npm automatically.
- Any other commit since the last tag makes the next release a manual dispatch
of `npm-release.yml` with an explicit `release_version`. Choose the version
from the diff of the published `files` in `package.json`, not from the whole
repository diff. Changes to `docs/` count as code commits here.
- The Pages site publishes only feeds that match a feed-refresh receipt. After
merging feeds regenerated outside the refresh workflow, run
`gh workflow run feed-refresh.yml --ref main` so a matching receipt exists.
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@AGENTS.md
205 changes: 141 additions & 64 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,91 +1,168 @@
# Contributing to model-eol
# Contributing

Thanks for your interest in model-eol - a machine-readable deprecation feed
format for AI models, plus the tooling that turns it into a Dependabot for
models. This guide covers the contributions the project most needs and the
conventions it holds to.
## Feed corrections

By contributing, you agree that your contributions are licensed under the
project's [MIT License](LICENSE).
If a feed has a wrong date, is missing a model, or doesn't carry a retirement
you know about, that's the most useful thing you can report.

## The most valuable contribution: feed corrections
Don't edit `feeds/*.json` by hand. The files are generated by `refresh/` from
publisher and distributor pages, and the next weekly refresh would overwrite
your change. Instead:

Found a wrong date, a missing model, or a retirement we don't carry? That is
exactly the signal this project runs on. One rule matters:

**Feeds are generated - never hand-edit `feeds/*.json`.** Every entry comes
from `refresh/` parsing a provider's live page or endpoint. A hand-edited date
would be silently overwritten by the next weekly refresh.

Instead:

- **Wrong or missing data, source exists** - open an issue with the model ID,
the channel (publisher or distributor), the correct value, and the URL that
states it. If the parser should have caught it, that URL is the failing case.
- **Parser gap** - fix it in `refresh/providers.mjs` or
- **The data is wrong or missing and a source says so:** open a
[feed correction issue](https://github.com/thossullivan/model-eol/issues/new?template=feed-correction.md)
with the model ID, the channel (the publisher's own API or a distributor), the
correct value, and the URL that states it. If the parser should have picked
it up, that URL is the failing test case.
- **A parser missed something:** fix it in `refresh/providers.mjs` or
`refresh/distributors.mjs`, add a fixture under `refresh/test/fixture/`, and
regenerate with `node refresh/refresh.mjs`. The semantic diff it prints is
your PR description.
- **New publisher or distributor** - the highest-value PR there is. Copy the
shape of an existing parser: strict table identity, model-ID grammar
validation, refusal over guessing. A parser that cannot be certain must
throw, not produce plausible data.
regenerate from the live pages with `node refresh/refresh.mjs`. Put the
semantic diff it prints in the PR description. Don't commit feeds generated
with `--fixtures`: the fixture's test models and aliases end up in the feed
as if they were real.
- **A new publisher or distributor:** start from the closest existing parser.
[`refresh/README.md`](refresh/README.md) explains the rules parsers follow.
The main one is that a parser that isn't sure must throw. If your change
makes a parser accept more input, add a test showing it still refuses input
that would attach a row to the wrong model.

If you work at a model provider: you can make the scraping unnecessary by
serving your own feed at `/.well-known/model-eol.json`. `SPEC.md` is small on
purpose. Open an issue - that conversation is the whole point of the project.
If you work at a model provider, the most useful change is to publish your own
feed at `/.well-known/model-eol.json`. [`SPEC.md`](SPEC.md) is short. Please
open an issue to discuss it.

## Development setup
## Development

Node 22+, nothing else. There are no dependencies to install - not for the
tool, not for development.
You need Node 22 or later and nothing else. There are no dependencies to
install, for the tool or for development.

```bash
git clone https://github.com/thossullivan/model-eol
cd model-eol
npm test
```

## Quality gates
| Path | What lives there |
|---|---|
| `check.mjs` | CLI entry point. It only dispatches; the logic is in `lib/`. |
| `lib/` | Scanner, lifecycle dates, config, reports, plan and apply, validation, verify. |
| `bot/` | `model-eol-bot`: plan, evaluate, and publish against the GitHub API. |
| `refresh/` | Feed generation. See [`refresh/README.md`](refresh/README.md). |
| `feeds/` | Generated feeds. |
| `schema/` | Public JSON schemas. |
| `scripts/` | Release, Pages, changelog, and test helpers. |
| `docs/` | Reference docs linked from the README. |

`npm test` runs every suite: checker, refresh, bot, Action, schemas, the
example eval harness, the packed npm package, the changelog, and the public
site. It runs offline with no API keys, and CI runs the same command. For
changes to the scanner or `--changed`, also run the end-to-end harness against
a few real repositories:

```bash
node scripts/e2e-changed.mjs --checker ./check.mjs --repo ../some-repo --repo ../another-repo
```

It replays recent commits through `check --changed` and compares each result
with a full scan of the same commit.

`npm test` composes the checker, refresh, bot, Action, schema validation,
example eval-harness, packed-consumer, changelog, and public-site suites - it
must pass offline, with no API keys. CI runs exactly this. Every behavior change
needs a regression test that fails on the old code.
### Tests

Two testing conventions that bite newcomers:
Every behavior change needs a test that fails on the old code. Two things
catch people out:

- **Content-stable assertions.** Feeds regenerate weekly, so never assert
against a committed feed's specific values (counts, dates, the `generated`
timestamp). Build a small fixture feed inside the test instead.
- **No network in tests.** Parsers are tested against recorded fixtures;
fetch paths take an injectable `fetchImpl`.
- **Committed feed values change every week.** The refresh workflow runs
`npm test` against freshly regenerated feeds, so a test that checks a real
count, date, or `generated` timestamp breaks the weekly refresh. Build a
small feed inside the test. A test that runs `refresh --check` against the
committed feeds should accept exit code 0 or 3.
- **Tests don't use the network.** Parsers are tested against recorded pages,
and fetch functions take an injectable `fetchImpl`.

### Docs that tests check

Some docs are read by tests or rewritten by scripts:

- `README.md` has `<!-- feeds-status -->` markers. The refresh workflow
rewrites the text between them. Keep the markers.
- `README.md` must contain exactly one Action snippet using
`actions/checkout@v7` and `actions/setup-node@v7` with `node-version: 22`
and `package-manager-cache: false`, plus the feed-refresh badge.
- Fragments of the local evaluation script in `docs/BOT.md` are checked by
`bot/test/package.mjs`. Change the script and the test together.
- `docs/RECEIPTS.md` must document `model-eol verify`.

User-facing basics go in `README.md`, reference material in `docs/`, the feed
format in `SPEC.md`, and parser behavior in `refresh/README.md`.

## Conventions

- **Zero runtime dependencies** - Node builtins only. This is a feature, not
an accident; PRs adding a dependency will be declined.
- **Fail loudly.** Malformed input is refused with a named reason, never
coerced into plausible data. When in doubt, throw.
- Comments are single terse lines stating what the code cannot - no stacked
comment blocks. Rationale goes in the commit body.
- Use " - " rather than em-dashes in prose.
- Spec changes (`SPEC.md`, `schema/`) start as an issue, not a PR. Settled
design decisions live in `docs/CONTEXT.md` - read it first; it is the
record of what has already been decided and why.
- **No dependencies,** runtime or development. The test suite fails if one is
added. Everything the tool runs is in this repository, which keeps it easy to
audit.
- **Refuse bad input.** Malformed input fails with an error that names the
problem. It is never replaced with a default or a guess.
- **Comments are one short line** and say what the code can't. Put the
reasoning in the commit message.
- **No em-dashes in prose.** Use a comma, a colon, parentheses, or a new
sentence.
- **Write for a public repository.** Commits, issues, and docs contain no local
paths, private repository names, or personal data.
- Changes to `SPEC.md` or `schema/` start as an issue, not a PR.

## Design decisions

These are settled. To reopen one, open an issue that explains what has changed.
Decisions about the feed format itself are recorded in [`SPEC.md`](SPEC.md).

The checker reports only strings that appear in a feed. It doesn't try to
recognize anything that merely looks like a model name, because false alarms in
CI get a check switched off faster than misses do. Unknown model-like strings
are listed as candidates and never fail or create work.

Cloud and gateway references are hints until something resolves them. Static
analysis can see that code calls Bedrock or OpenRouter but usually can't see
which model sits behind a deployment name, so those references stay in the
inventory and don't fail `--scope direct`. The bot opens an issue for each such
provider instead, so a person can check what is deployed.

Patching is strict. `plan` patches only references it is confident are direct
API calls, and only when the feed's `replacement` is itself safe. Cloud and
gateway references, prose replacements, and dates that came from
`publisher-fallback` become issues for a person.

A distribution entry without a date means nothing is scheduled on that channel.
It doesn't inherit the publisher's date; only a missing entry falls back, and
the result is labeled `publisher-fallback`. An earlier, more cautious reading
told Azure users their models were gone five months early.

`refresh --check` exits 3 only when the generated files change materially.
Rows the parser can't confirm show up in the diff but don't count. Otherwise a
weekly run without API keys would open the same empty PR every Monday, which is
what the first live run did.

model-eol reports when a model stops answering and runs your eval command, but
it doesn't evaluate models itself. The reasons are in
[docs/DESIGN_EVAL_BOUNDARY.md](docs/DESIGN_EVAL_BOUNDARY.md). The eval command
runs only in a job without write access; see
[docs/BOT.md](docs/BOT.md#security-model).

## AI-assisted contributions

Much of this codebase was written by AI under human direction and adversarial
review - the commit history says which. Contributions are welcome however you
author them, under the same terms: you own what you submit, you have verified
it yourself, and it arrives with tests. "The model wrote it" explains nothing
in review; the diff stands or falls on its own.
Much of this codebase was written by AI models directed by the maintainer, with
adversarial review. When a model wrote or reviewed a change, the commit message
body says so, for example "Implemented by <model>; reviewed by <model>".
Commits don't use `Co-Authored-By` trailers for models.

Contributions are welcome however you write them, on the same terms: you own
what you submit, you've checked it yourself, and it comes with tests. Coding
agents should also read [AGENTS.md](AGENTS.md).

## Security

If you find a vulnerability - especially in the bot's trust boundaries (feed
ingestion, plan application, the eval hook) - please use GitHub's private
vulnerability reporting rather than a public issue. Two full adversarial
review cycles are dispositioned in the issue tracker (`adversarial-review`
label); reading them first will tell you what has already been considered.
Please report vulnerabilities privately; see [SECURITY.md](SECURITY.md). Past
adversarial reviews are tracked in issues with the `adversarial-review` label,
and reading them first shows what has already been considered.

## License

Contributions are licensed under the project's [MIT License](LICENSE).
Loading
Loading