Release procedure: version bumps deliver, changelogs record, CI enforces by mmcky · Pull Request #28 · QuantEcon/skills

mmcky · 2026-08-03T05:26:40Z

A version bump in this marketplace is not bookkeeping — it is the delivery mechanism. This PR makes that explicit, records it, and enforces it.

The problem, verified

The install cache is keyed by version string — ~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/ — and claude plugin update compares version strings only. I simulated a stale install sitting at qe 0.2.0 with newer content in the marketplace and ran the real update command:

✔ qe is already at the latest version (0.2.0).

It refreshed nothing, even though installed_plugins.json records a gitCommitSha and the marketplace clone had the newer files. Change a shipped file without moving the version and every consuming repo keeps the old copy, silently.

This is not hypothetical. Replaying the new guard over all eleven squash-merges on main blocks three, with no false positives on the docs-only or new-plugin merges:

Merge	What slipped through
#27	edited `qe/skills/copilot-review/SKILL.md`, `qe` left at 0.2.0
#5	7 `qe` files changed, version left at 0.1.0
#1	2 `benchmark` files changed, version left at 0.1.0

So two different trees have already been distributed under one version string each. Every changelog here ends by saying so plainly rather than quietly starting the record at today.

What lands

scripts/check-version-bump.py — fails a PR that changes any file under a plugin directory without bumping that plugin's version and adding the matching CHANGELOG.md heading. It exits 2, never 0, when it cannot see the base branch: a guard that silently passes when it could not run manufactures confidence. Wired into validate.yml with fetch-depth: 0, gated to PRs into the default branch so a stacked PR is not asked to mint a second version for content that ships once. It was built against 32 fixture cases, then attacked by an independent pass that found six holes — including false passes — all patched before landing.

One CHANGELOG.md per plugin, backfilled from git history and fact-checked against it. Per-plugin rather than at the repo root for a reason worth stating: the reader who needs a changelog is the installed user, whose copy is an extracted cache directory with no .git, no remote, and no practical path to one. A root changelog is not in that directory and cannot be linked to from inside it. The docs also say honestly that for a maintainer with a checkout the file adds little over git log — so nobody later removes it for the wrong reason.

A second CI job running claude plugin validate --strict. It parses what scripts/validate.py only pattern-matches, and it found a live defect: audit/skills/issues/SKILL.md's description is invalid YAML — an unquoted plain scalar containing Read-only: — which a real parser rejects and the repo's regex frontmatter reader never saw. Confirmed with an actual YAML parser, and it is the only such file in the repo. Fixed here by quoting it. It is a separate job so the stdlib checks stay fast, and the CLI is pinned because under --strict an upstream warning would redden an unrelated PR. It needs no credentials — verified with the auth environment scrubbed.

docs/developing-skills.md § Versioning and releases, rewritten to record the procedure: the bump rule with its reason, the changelog requirement and why it is per-plugin, what the guard checks, and claude plugin tag for release tags. AGENTS.md gains a doc-map row and a working-in-this-repo bullet; README.md points at the rule.

Versions

Plugin		Why
`qe`	0.2.0 → 0.2.1	delivers #27's `SKILL.md` correction, which so far reached nobody
`benchmark`	0.3.0 → 0.3.1	gains a shipped `CHANGELOG.md`; `README.md` status line corrected from a `v0.2.0` that was never published
`audit`	0.1.2 → 0.1.3	gains a `CHANGELOG.md` and the frontmatter fix

The catalogue's top-level version stays at 0.2.0 — no plugin was added, and it is not a cache key. The PR is self-consistent: the guard passes on its own diff.

Two things needing your decision

1. audit 0.1.3 collides with open PR #24. That branch already claims 0.1.3 in both manifests and touches five audit/ files. Whichever merges second hits a conflict in plugin.json, marketplace.json and `CHANGELOG.md. That is the mechanism working as designed — it is exactly what stops two branches shipping different trees under one version — but it does mean #24 will need a rebase onto 0.1.4 plus its own changelog entry. Worth deciding merge order before this lands.

2. A CHANGELOG.md-only edit requires a version bump. No per-file exemptions: everything under a plugin directory ships, so an unbumped changelog fix reaches nobody either, and the copy in a user's cache would permanently differ from main. The cost is that a retroactive typo fix mints a patch release. Reversible if you dislike it, but decide before someone hits it rather than during.

Also left deliberately undone: no git tags. claude plugin tag ./<plugin> would create {name}--v{version}, refuses on a manifest mismatch, and refuses on a dirty tree — but the repo has none today, so the changelogs link to filtered commit history instead of compare ranges, and nothing 404s. Happy to backfill tags after this merges if you want that mapping.

python scripts/validate.py, claude plugin validate --strict on all four targets, and the version guard all pass; relative links resolve and no changelog links escape its plugin root.

A version bump is not bookkeeping in this marketplace — it is the delivery mechanism. The install cache is keyed by version string (~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/) and `claude plugin update` compares version strings only: change a shipped file without moving the version and every consuming repo is told "already at the latest version" and keeps the old copy. Verified by simulating a stale install at 0.2.0 and running the real update command — it refreshed nothing. That has already happened three times. Replaying the guard added here over all eleven squash-merges on main blocks #1 (benchmark), #5 (qe) and #27 (qe), with no false positives on the docs-only or new-plugin merges. What lands: - scripts/check-version-bump.py — fails a PR that changes any file under a plugin directory without bumping that plugin's version and adding the matching CHANGELOG heading. It exits 2, never 0, when it cannot see the base: a guard that silently passes when it could not run manufactures confidence. Wired into validate.yml with fetch-depth: 0, gated to PRs into the default branch so a stacked PR is not asked to mint a second version for content that ships once. - One CHANGELOG.md per plugin, backfilled from git history. Per-plugin rather than at the repo root because the reader who needs it is the installed user, whose copy is an extracted cache directory with no .git and no way to reach one — a root changelog is not in it and cannot be linked from it. Each file ends by naming the trees already distributed under a duplicated version. - A second CI job running `claude plugin validate --strict`. It parses what scripts/validate.py only pattern-matches, and it found a live defect: audit/skills/issues/SKILL.md's description is invalid YAML — an unquoted plain scalar containing "Read-only: " — which a real parser rejects and the repo's regex reader never saw. Fixed here by quoting it. Separate job so the stdlib checks stay fast, and the CLI is pinned because under --strict an upstream warning would redden an unrelated PR. - docs/developing-skills.md § Versioning and releases, rewritten: the bump rule with its reason, the changelog requirement and why it is per-plugin, what the guard does, and `claude plugin tag` for release tags. AGENTS.md gains the doc-map row and a working-in-this-repo bullet; README points at the rule. Versions: qe 0.2.0 → 0.2.1 (delivers #27's SKILL.md correction, which reached nobody), benchmark 0.3.0 → 0.3.1 and audit 0.1.2 → 0.1.3 (each gains a shipped CHANGELOG.md, and audit the frontmatter fix). The catalogue version stays at 0.2.0 — no plugin was added, and it is not a cache key.

Copilot

Pull request overview

This PR formalizes and enforces the repository’s release mechanism: plugin version strings are treated as the delivery key for installed users, so any shipped plugin-directory change must include a version bump and a corresponding per-plugin changelog entry, with CI checks to prevent silent “published-but-undelivered” updates.

Changes:

Adds scripts/check-version-bump.py and wires it into CI to fail PRs that modify shipped plugin content without a version bump + matching changelog heading (and fails loudly when it can’t compute a reliable base).
Introduces per-plugin CHANGELOG.md files (backfilled) and bumps qe, benchmark, and audit plugin versions in both plugin.json and the marketplace catalog.
Adds a separate claude plugin validate --strict CI job and updates contributor docs (AGENTS.md, README.md, docs/developing-skills.md) to record the release procedure and rationale.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`scripts/check-version-bump.py`	New guard script enforcing “plugin-dir change ⇒ version bump + changelog heading”, with explicit exit codes and shallow-clone refusal.
`README.md`	Documents the new CI enforcement and links to the expanded release procedure.
`docs/developing-skills.md`	Rewrites “Versioning and releases” to make the delivery semantics explicit and describe the CI checks + changelog rationale.
`AGENTS.md`	Updates doc map and working conventions to include per-plugin changelogs and bump requirement.
`.github/workflows/validate.yml`	Adds `fetch-depth: 0`, runs the version-bump guard on PRs into default branch, and adds a separate strict validation job via Claude CLI.
`.claude-plugin/marketplace.json`	Bumps `qe`, `benchmark`, `audit` versions to match plugin manifests.
`qe/.claude-plugin/plugin.json`	Bumps `qe` version to `0.2.1`.
`qe/CHANGELOG.md`	Adds `qe` per-plugin changelog with backfilled entries and release rationale.
`benchmark/.claude-plugin/plugin.json`	Bumps `benchmark` version to `0.3.1`.
`benchmark/README.md`	Updates status line version reference.
`benchmark/CHANGELOG.md`	Adds `benchmark` per-plugin changelog with backfilled entries and “no 0.2.0” note.
`audit/.claude-plugin/plugin.json`	Bumps `audit` version to `0.1.3`.
`audit/skills/issues/SKILL.md`	Quotes `description` to make YAML frontmatter valid under strict parsing.
`audit/CHANGELOG.md`	Adds `audit` per-plugin changelog with backfilled entries and frontmatter fix note.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…d in Copilot review on #28, both comments valid. `highest` was `max()` over a set, and every unorderable version shares the key `()`. With two pre-release versions the tie-break was whichever element set iteration reached first, which varies with hash randomisation — reproduced with three distinct answers across six PYTHONHASHSEED values. Latent rather than live (`published` holds at most the merge-base and base-tip versions, and this repo uses plain X.Y.Z), but a guard that answers differently on different runs cannot be trusted about anything. Sorting first makes the tie-break the version string: still arbitrary, the same arbitrary answer every time. Rejecting non-X.Y.Z outright was the suggested alternative and would have been simpler, but it bans pre-release versions repo-wide for a defect that is really about determinism. Instead the skipped downgrade check now prints a note naming the versions it could not order, so it is visible rather than silent. benchmark/README.md said the evaluation system landed in v0.3.1. It landed in 0.3.0; 0.3.1 is this changelog-and-README release. The previous value, v0.2.0, was a version that never existed, so this line has now been wrong twice — it names 0.3.0 because that is a historical fact and cannot go stale on the next bump. The changelog entry describing the fix is corrected to match.

…alls Rebased onto main, which since #28 requires a version bump and a changelog entry for any change to a plugin's shipped files. This branch changes four of them. The rebase itself was clean, and quietly instructive. This branch and main had both moved audit 0.1.2 -> 0.1.3 — the same edit on both sides, so git merged it without a murmur and GitHub reported the PR MERGEABLE/CLEAN. Merged as it stood, it would have shipped four changed files under a version string already published from main, and every install would have been told it was up to date. That is precisely the failure the guard exists to catch, and it did: !! audit — 4 files of shipped content changed, version still '0.1.3' Hence 0.1.4. No content changed in this commit beyond the manifests and the changelog entry describing what the two commits below already do.

…duced (#24) * audit: re-derive doctrine §4 from evidence the first run actually produced Option A of #21 defect 4. The duration claim was wrong and it was load-bearing, so correcting the number is not enough — the justification it carried has to be replaced with one that survives measurement. **What was claimed.** doctrine §4 opened "Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep" and derived checkpointing from it; SKILL.md said "a hundred-issue repo is a multi-hour run — SO it works from a frozen snapshot"; README said "expect hours, not minutes, on a repo with a hundred items". Note the causal chain: the machinery was derived from the duration. **What was measured.** 230 items in 22 minutes, and none of §4's three named mechanisms occurred — context did not run out, the snapshot is two API calls in 11 seconds, and nothing slept. A second error sat underneath: cost tracks *open issues needing verification* (~10 s each), not total items, so README was measuring the wrong quantity as well as the wrong magnitude. **What replaces it.** Three reasons that hold at any duration, strongest first: the checkpoint is *evidence* — run 1's 56-of-56 coverage reconciliation was done against findings.md rather than the report, and the catalog was assembled from it; interruption is duration-independent; and the cost is asymmetric. The first is better than what it replaces precisely because it can be checked, which is what §1 rule 8 asks of every other claim in this document. §4 is renamed "Checkpointing" (its two inbound anchors updated) since the section is no longer about long runs. It also now states the criterion the new justification implies: a checkpoint superseded minutes later without being read earns nothing — which is the open question in #23, not settled here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Tutorial: absolute link to the run record, which lands in #22 The relative form failed the links check because the file lives on #22's branch, not this one. Absolute is the repo's convention for cross-boundary references anyway (developing-skills § Conventions), and it resolves once #22 merges — matching how doctrine.md and audit/README.md already cite the same record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * audit 0.1.4: version the doctrine §4 re-derivation so it reaches installs Rebased onto main, which since #28 requires a version bump and a changelog entry for any change to a plugin's shipped files. This branch changes four of them. The rebase itself was clean, and quietly instructive. This branch and main had both moved audit 0.1.2 -> 0.1.3 — the same edit on both sides, so git merged it without a murmur and GitHub reported the PR MERGEABLE/CLEAN. Merged as it stood, it would have shipped four changed files under a version string already published from main, and every install would have been told it was up to date. That is precisely the failure the guard exists to catch, and it did: !! audit — 4 files of shipped content changed, version still '0.1.3' Hence 0.1.4. No content changed in this commit beyond the manifests and the changelog entry describing what the two commits below already do. * audit: separate the two cost measures, and stop saying it has never run as a skill Copilot review on #24. The cost sentence put "roughly 10 seconds per open issue" beside "a 230-item tracker with 56 open took 22 minutes" and left a reader to reconcile them. They do not reconcile: 56 x 10 s is 9 minutes. The run record has the decomposition — phase 2 verified the 56 open issues in ~9 minutes, and phases 1, 3 and 4 account for the rest. Both figures now say which question they answer: budget the total from the whole run, scale it by the open count, because that is what phase 2 walks. Corrected in all three places that state it — SKILL.md, audit/README.md and the tutorial — since a number restated three ways drifts three ways. The tutorial also claimed /audit:issues "has never been run as a skill, only by hand as a runbook". That was true when written and is not now: the run record this PR cites opens by describing itself as "the first execution of /audit:issues as a skill", on 2026-07-28, and it found seven plugin defects. The tutorial now says it has run once, links the record, and frames the reader's run as the next data point in the validation program rather than the first. While there: the tutorial told readers `claude plugin list` should report 0.1.2. That has been wrong since this branch bumped the version and would have gone wrong again at every future release, so it now points at the marketplace entry instead of naming a number — the same failure benchmark/README.md had twice. Not changed: the two run-record links Copilot flagged as 404s. They are correct and deliberate forward references — the record lands in #22, which merges first. The same "never yet as a skill" claim is also live on main, in CATALOG.md and docs/using-skills.md. That is repo-level rather than this PR's business and is being handled separately. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rocedure left behind (#31) * Docs: the layout trees, the tag facts, and two pointers the release procedure left behind #28 rewrote how releases work without updating either picture of what a plugin contains. Both layout trees — README's and developing-skills' — still showed a plugin as manifest + skills + scripts + references, with no CHANGELOG.md, while CI now fails any PR that omits one. They are fixed together; leaving one right and one wrong is worse than both being stale, because a contributor cannot tell which to trust. `scripts/validate.py` in both trees becomes `scripts/`, since it is no longer the only thing in there. "Only SKILL.md is required" was true when written and is not now — the manifest and the changelog are both required — so it now says so rather than being quietly contradicted by the CI guard three sections down. § Tags is rewritten because creating the tags falsified its opening sentence. It now documents the scheme as adopted, and records why only current versions were tagged: several historical versions cannot be tagged honestly, since some name two different published trees — content shipped twice under one version before the guard existed — and others sit at commits that fail today's validation. The changelogs carry that ambiguity in prose, with PR numbers. Two stale pointers: developing-skills said issue #4 tracks the ${CLAUDE_PLUGIN_ROOT} pattern, which shipped and now has a worked example; and "which is the existing precedent" claimed a consistency the history does not have — adding qe did not move the catalogue version, adding audit did. The rule stands, so it is stated rather than inferred. Also: the audit tutorial is added to README's documentation table, where it was the only guide missing while being in AGENTS.md's doc map; and its Step 7 line gave one hardcoded filename as if it were the instruction, which now reads as the convention with run 1's file as the example. * Docs: separate the plugin-level requirements from the skill-level one Copilot review on #31, and correct. The sentence opens a paragraph that is entirely about skills — "A skill that is purely a procedure … is one file in one directory" — so prefixing it with two plugin-level requirements read as though a skill needs a manifest and a changelog of its own. Both live at <plugin>/, not under skills/<name>/. Naming the two levels in one line is clearer than either the original, which omitted the plugin requirements the CI guard now enforces, or the prefix, which smuggled them into a sentence about something else.

Copilot AI review requested due to automatic review settings August 3, 2026 05:26

Copilot started reviewing on behalf of mmcky August 3, 2026 05:27 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Comment thread benchmark/README.md Outdated

Comment thread scripts/check-version-bump.py

mmcky merged commit 52d4539 into main Aug 3, 2026
5 checks passed

mmcky deleted the release-procedure branch August 3, 2026 05:43

mmcky mentioned this pull request Aug 3, 2026

audit: re-derive doctrine §4 from evidence the first run actually produced #24

Merged

Navigation Menu

Uh oh!

Release procedure: version bumps deliver, changelogs record, CI enforces - #28

Release procedure: version bumps deliver, changelogs record, CI enforces#28
mmcky merged 2 commits into
mainfrom
release-procedure

mmcky commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mmcky commented Aug 3, 2026

The problem, verified

What lands

Versions

Two things needing your decision

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants