Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Update mode brings an existing README to the README standard — header shape,
section names and order — while keeping the author's prose and logo, instead
of changing only the outdated parts. Scenario 5's invariant follows (#25).

- The benchmark keeps every Markdown page an agent writes besides the README
under `pages/` in the case directory, and lists them in `meta.json`. The
scorer accepts those pages as link targets and checks the links inside them
Expand Down
4 changes: 3 additions & 1 deletion skills/github-readme-generator/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,9 @@ and state it in the handover summary.
branding, useful hand-written explanation, and working links. A registry
badge or docs link the README already carries counts as evidence when its slug
matches the manifest, even offline — keep it and report it as unverified. Fix
only what is outdated, unsupported, duplicated, or badly ordered.
what is outdated, unsupported, or duplicated, and bring the structure to the
standard in `references/readme-structure.md` — header shape, section names
and order — keeping the author's prose and the existing logo inside it.
- **Audit** — analysis only. Write nothing. Return findings with sources and a
recommended fix order. Cover the README, and the two public surfaces beside it
— repository metadata, and the community health files a README links to — as
Expand Down
10 changes: 6 additions & 4 deletions tests/scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,11 @@ per-package commands are not mixed into the root instructions.
*Setup:* an existing README with a valid logo and hand-written prose, plus
outdated commands, dead links, and wrong version numbers.

Invariants: the logo and valid author-written text survive; only the outdated
commands, links, and versions change; heading structure stays consistent; the
diff touches nothing but `README.md`.
Invariants: the logo and valid author-written text survive; every outdated
command, link, and version is fixed; the structure follows the README standard
— header shape, section names and order — with no section invented beyond what
the repository evidences; heading structure stays consistent; the diff touches
nothing but `README.md`.

## 6. Insufficient evidence

Expand Down Expand Up @@ -164,7 +166,7 @@ means the scenario has not been run since this log was started.
| 2 | 2026-11-12, Claude Code 2.1.322, `claude -p "Write a README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. `npm start` and `npm test` from scripts, `npm install` backed by the lock file, Node 20 from `engines`, configuration limited to the four `.env.example` names, no deployment section. |
| 3 | 2026-11-12, Claude Code 2.1.322, `claude -p "Write a README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. `Installation` clones the remote and runs `npm install --global .` (no publish evidence), the smallest invocation `slugify "Hello, World!"` → `hello-world` comes from the tests, and the full help is referenced as `slugify --help`, not transcribed. |
| 4 | 2026-11-12, Claude Code 2.1.322, `claude -p "Write a README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. The root README explains the npm workspace, links all three package READMEs, and shows only root `npm install` and `npm test`. |
| 5 | 2026-11-12, Claude Code 2.1.322, `claude -p "Update the README for this repository." --permission-mode acceptEdits --setting-sources project` | Partial (#25). Logo and hand-written paragraph survive, all five stale items fixed, only `README.md` changed. But the header was rebuilt and sections renamed and added to follow the README standard — more than the outdated parts this scenario allows. |
| 5 | 2026-11-12, Claude Code 2.1.322, `claude -p "Update the README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. Logo and the author's paragraph survive verbatim; all five stale items fixed (version pin, `require`, `npm run serve`, dead `docs/GUIDE.md` link, Node 14); header rebuilt to the standard shape with the project name as title, sections Requirements, Installation, Quick start, Testing, License, each backed by the repository; only `README.md` changed. |
| 6 | 2026-11-12, Claude Code 2.1.322, `claude -p "Write a README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. No install command, license claim, badge, or `TODO`; the handover names the missing manifest, license, and tests. Minor: a Python version was inferred from annotation syntax, disclosed as an inference. |
| 7 | 2026-11-12, Claude Code 2.1.322, `claude -p "Audit this README and tell me what's wrong." --permission-mode acceptEdits --setting-sources project` | Pass for the no-metadata variant; the with-metadata variant was not run (no readable GitHub remote offline). Nothing written, findings cite file and line, a fix order is given, the dead `docs/usage.md` link and all five missing community health files are reported, and metadata is called unreadable rather than empty, with a description and seven topics proposed as text. |
| 8 | 2026-11-12, Claude Code 2.1.322, `claude -p "Write a README for this repository." --permission-mode acceptEdits --setting-sources project` | Pass. Only the four `.env.example` names are documented; the `.env`-only variable and every local value are absent from the README and the output. Minor: an unsourced `. ./.env` and `curl` example in Quick start. |
Expand Down
Loading