audit: re-derive doctrine §4 from evidence the first run actually produced by mmcky · Pull Request #24 · QuantEcon/skills

mmcky · 2026-07-28T07:36:32Z

Option A of #21 defect 4. audit 0.1.3. Option B — whether the phase apparatus itself should shrink — is deferred to #23, pending run 2.

The claim was load-bearing, so correcting the number is not enough

The three statements formed a causal chain, not three loose estimates:

doctrine §4 — "Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep. One rule follows from that…" → therefore checkpoint.
SKILL.md — "a hundred-issue repo is a multi-hour run — so it works from a frozen snapshot and checkpoints every phase to disk."
audit/README.md — "Expect hours, not minutes, on a repo with a hundred items."

The machinery was derived from the duration. Fixing the number alone would have left the derivation dangling.

What run 1 measured

230 items in 22 minutes, and all three of §4's named mechanisms were absent: context did not run out, the snapshot is two API calls in 11 seconds, nothing slept.

A second error sat underneath the first. Cost tracks open issues needing verification — about 10 s each — not total items. So README was measuring the wrong quantity as well as the wrong magnitude, which is why a 1000-item repo with a small open set extrapolates cheaper than a 300-item repo with a large one.

What replaces the justification

Three reasons that hold at any duration, strongest first:

The checkpoint is evidence, not insurance. Run 1's 56-of-56 coverage reconciliation was performed against findings.md, not against the report, and the catalog was assembled from it.
Interruption is duration-independent — a cancel, a tool error, a rate limit.
The cost is asymmetric — a line per item is nearly free; the phase is not.

Reason 1 is better than what it replaces precisely because it can be checked, which is what §1 rule 8 ("match confidence to inspection") asks of every other claim in that document. A doctrine carrying an unevidenced, false, load-bearing empirical claim was not practising what it demands — that, rather than the wrong number, is what made this worth a version bump.

Also

§4 is renamed Checkpointing — it is no longer about surviving long runs — with both inbound anchors updated. It now also states the criterion its new justification implies: a checkpoint superseded minutes later without being read earns nothing. That criterion has a concrete instance (phase 3's links.md, superseded eight minutes later and never read), which is exactly what #23 exists to decide — flagged here, not settled here.

Duration claims corrected in four places; validate.py green.

🤖 Generated with Claude Code

Copilot

Pull request overview

Updates the audit plugin’s documentation and doctrine to remove a load-bearing but false “multi-hour run” premise, re-deriving checkpointing (§4) from evidence-based, duration-independent reasons, and bumps the plugin version to 0.1.3.

Changes:

Rewrites doctrine §4 as “Checkpointing” with a new justification grounded in run-1 evidence and a clearer criterion for when checkpoints “earn” their cost.
Updates user-facing docs (tutorial, SKILL.md, plugin README) to replace “hours” expectations with “tens of minutes” guidance and per-open-issue budgeting language.
Bumps audit plugin version from 0.1.2 → 0.1.3 (and marketplace entry to match).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/tutorial-run-an-audit.md	Updates expected duration and run-record guidance for the audit tutorial.
audit/skills/issues/SKILL.md	Replaces the “multi-hour” premise with measured-cost language and updates doctrine anchor.
audit/scripts/README.md	Updates doctrine §4 anchor reference to the renamed section.
audit/references/doctrine.md	Renames §4 to “Checkpointing” and rewrites its justification based on measured evidence.
audit/README.md	Updates the top-level audit plugin README duration/cost guidance and links to the run record.
audit/.claude-plugin/plugin.json	Bumps `audit` plugin version to 0.1.3.
.claude-plugin/marketplace.json	Bumps marketplace `audit` version to 0.1.3.

Comments suppressed due to low confidence (3)

docs/tutorial-run-an-audit.md:14

This line combines an end-to-end wall clock (22 minutes) with a per-open-issue rate (10 seconds) without clarifying that the per-issue rate is about phase-2 verification. Tweaking the wording avoids readers trying to reconcile 56×10s with 22 minutes.

- **Tens of minutes**, and a session you can afford to interrupt — interrupting it is one of the tests. Run 1 took 22 minutes for 230 items; budget by *open* issues, at roughly 10 seconds each.

docs/tutorial-run-an-audit.md:14

The tutorial still tells readers claude plugin list should show audit version 0.1.2, but this PR bumps the plugin to 0.1.3. Update the expected version so the install check matches the published plugin version.

- **Tens of minutes**, and a session you can afford to interrupt — interrupting it is one of the tests. Run 1 took 22 minutes for 230 items; budget by *open* issues, at roughly 10 seconds each.

docs/tutorial-run-an-audit.md:111

This links to ../reviews/audit-run-action-translation-2026-07-28.md, but that file is not present under reviews/ in this branch, so the link will be broken. Either add the run record file under reviews/ or link to an existing permanent record (e.g., issue #21) so the tutorial stays navigable.

- **Cost** — wall clock and rough token spend per phase. Run 1's figures are in [its record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md); a second data point at a different repo type is what turns one measurement into an estimate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…duced Option A of #21 defect 4. The duration claim was wrong and it was load-bearing, so correcting the number is not enough — the justification it carried has to be replaced with one that survives measurement. **What was claimed.** doctrine §4 opened "Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep" and derived checkpointing from it; SKILL.md said "a hundred-issue repo is a multi-hour run — SO it works from a frozen snapshot"; README said "expect hours, not minutes, on a repo with a hundred items". Note the causal chain: the machinery was derived from the duration. **What was measured.** 230 items in 22 minutes, and none of §4's three named mechanisms occurred — context did not run out, the snapshot is two API calls in 11 seconds, and nothing slept. A second error sat underneath: cost tracks *open issues needing verification* (~10 s each), not total items, so README was measuring the wrong quantity as well as the wrong magnitude. **What replaces it.** Three reasons that hold at any duration, strongest first: the checkpoint is *evidence* — run 1's 56-of-56 coverage reconciliation was done against findings.md rather than the report, and the catalog was assembled from it; interruption is duration-independent; and the cost is asymmetric. The first is better than what it replaces precisely because it can be checked, which is what §1 rule 8 asks of every other claim in this document. §4 is renamed "Checkpointing" (its two inbound anchors updated) since the section is no longer about long runs. It also now states the criterion the new justification implies: a checkpoint superseded minutes later without being read earns nothing — which is the open question in #23, not settled here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The relative form failed the links check because the file lives on #22's branch, not this one. Absolute is the repo's convention for cross-boundary references anyway (developing-skills § Conventions), and it resolves once #22 merges — matching how doctrine.md and audit/README.md already cite the same record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…alls Rebased onto main, which since #28 requires a version bump and a changelog entry for any change to a plugin's shipped files. This branch changes four of them. The rebase itself was clean, and quietly instructive. This branch and main had both moved audit 0.1.2 -> 0.1.3 — the same edit on both sides, so git merged it without a murmur and GitHub reported the PR MERGEABLE/CLEAN. Merged as it stood, it would have shipped four changed files under a version string already published from main, and every install would have been told it was up to date. That is precisely the failure the guard exists to catch, and it did: !! audit — 4 files of shipped content changed, version still '0.1.3' Hence 0.1.4. No content changed in this commit beyond the manifests and the changelog entry describing what the two commits below already do.

mmcky · 2026-08-03T05:53:49Z

Rebased onto main and bumped audit to 0.1.4 so this actually reaches installs. Force-pushed, so the branch history changed — two original commits replayed unchanged, plus one commit carrying the manifests and the changelog entry.

Worth recording why, because the failure mode was invisible. This branch and main had both moved audit 0.1.2 → 0.1.3 — the identical edit on both sides, so git merged it without complaint and GitHub reported this PR MERGEABLE/CLEAN throughout. Merged as it stood, it would have shipped four changed files (README.md, doctrine.md, scripts/README.md, SKILL.md) under a version string already published from main, and every consuming repo would have been told "already at the latest version" and kept the old files. No conflict, no warning, no delivery.

The version guard added in #28 caught it: audit — 4 files of shipped content changed, version still '0.1.3'. That is the first thing it has caught in anger, and it caught something git's own conflict detection structurally cannot see.

The rebase was otherwise clean, and main's frontmatter fix to SKILL.md survived it. Content is unchanged from what you wrote — the new commit only adds the version bump and a changelog entry describing the doctrine §4 re-derivation and the measured cost figures.

…un as a skill Copilot review on #24. The cost sentence put "roughly 10 seconds per open issue" beside "a 230-item tracker with 56 open took 22 minutes" and left a reader to reconcile them. They do not reconcile: 56 x 10 s is 9 minutes. The run record has the decomposition — phase 2 verified the 56 open issues in ~9 minutes, and phases 1, 3 and 4 account for the rest. Both figures now say which question they answer: budget the total from the whole run, scale it by the open count, because that is what phase 2 walks. Corrected in all three places that state it — SKILL.md, audit/README.md and the tutorial — since a number restated three ways drifts three ways. The tutorial also claimed /audit:issues "has never been run as a skill, only by hand as a runbook". That was true when written and is not now: the run record this PR cites opens by describing itself as "the first execution of /audit:issues as a skill", on 2026-07-28, and it found seven plugin defects. The tutorial now says it has run once, links the record, and frames the reader's run as the next data point in the validation program rather than the first. While there: the tutorial told readers `claude plugin list` should report 0.1.2. That has been wrong since this branch bumped the version and would have gone wrong again at every future release, so it now points at the marketplace entry instead of naming a number — the same failure benchmark/README.md had twice. Not changed: the two run-record links Copilot flagged as 404s. They are correct and deliberate forward references — the record lands in #22, which merges first. The same "never yet as a skill" claim is also live on main, in CATALOG.md and docs/using-skills.md. That is repo-level rather than this PR's business and is being handled separately.

… citations Copilot review on #22, both comments valid, plus one they did not catch. The Cost table's "Completed" column carried bare clock times while Setup records the snapshot as 2026-07-28T06:45:58Z. The two are ten hours apart and look contradictory; they are the same instant in AEST. In a measurement record that ambiguity is not cosmetic, so the column now names its timezone. The total was wrong against the table's own timestamps. 16:45:59 to 17:08:05 is 22 minutes 6 seconds, and the per-phase elapsed figures sum to about the same, but the Total row said ~23 min. It matters beyond arithmetic: audit/README.md and #24 both cite "22 minutes" sourced from this record, so as written the citation disagreed with its evidence — the same defect this record criticises elsewhere, one level down. Corrected here and in the two places downstream that restate it as prose. Both doctrine citations pointed at the top of doctrine.md rather than the section being cited, which is a poor look in a document whose complaint #1 is a citation that does not verify. §2 is now #2-evidence-classes. §4 is anchored to #4-checkpointing, which is deliberately the post-#24 name. #24 renames that section from "Surviving a long run", and merges immediately after this — so the alternative anchor would be correct for minutes and wrong permanently. lychee runs without --include-fragments, so CI checks neither.

…026-07-28) (#22) * Reviews: run 1 of the audit validation program (action-translation, 2026-07-28) The first execution of /audit:issues as a skill rather than as a hand-run runbook, recorded per #16's protocol. Result: the bundle is good and the plugin has seven defects (#21), one of them in the checkpoint fix that shipped the day before in #17. Eight of the ten review checks held, including the read-only boundary — git status stayed empty across a run that wrote 127 KB into the audited repo — and a coverage table that matches coverage.json on every number. Two findings are worth naming here because they cut against decisions this repo made recently. A [verified] citation on the audit's headline finding pointed at a commit that is not on main, which doctrine §2 permits because it never says to check reachability. And the run took ~23 minutes for 230 items, against docs promising "hours, not minutes" — which undercuts doctrine §4's stated premise that bulk audits outlive sessions, and so the stated justification for the checkpointing machinery. The honest correction is to keep the machinery and fix the premise, not to pretend the measurement did not happen. Checks 9 and 10 — is the tiering right, would you act on this — are the maintainer's and remain open. Resumability is still untested: the run was not interrupted, so the program's headline claim survives run 1 unvalidated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Run record: label the clock, reconcile the total, anchor the doctrine citations Copilot review on #22, both comments valid, plus one they did not catch. The Cost table's "Completed" column carried bare clock times while Setup records the snapshot as 2026-07-28T06:45:58Z. The two are ten hours apart and look contradictory; they are the same instant in AEST. In a measurement record that ambiguity is not cosmetic, so the column now names its timezone. The total was wrong against the table's own timestamps. 16:45:59 to 17:08:05 is 22 minutes 6 seconds, and the per-phase elapsed figures sum to about the same, but the Total row said ~23 min. It matters beyond arithmetic: audit/README.md and #24 both cite "22 minutes" sourced from this record, so as written the citation disagreed with its evidence — the same defect this record criticises elsewhere, one level down. Corrected here and in the two places downstream that restate it as prose. Both doctrine citations pointed at the top of doctrine.md rather than the section being cited, which is a poor look in a document whose complaint #1 is a citation that does not verify. §2 is now #2-evidence-classes. §4 is anchored to #4-checkpointing, which is deliberately the post-#24 name. #24 renames that section from "Surviving a long run", and merges immediately after this — so the alternative anchor would be correct for minutes and wrong permanently. lychee runs without --include-fragments, so CI checks neither. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…#30) Every place still calling a shipped skill unfinished, or pointing a reader at a tracker for work that has landed. Each plugin-side claim is fixed in the same commit as its repo-side twin — splitting them is how the pairs drifted apart in the first place. - CATALOG.md said /audit:issues had been executed "by hand, never yet as a skill". False since 2026-07-28: run 1 was the first execution as a skill, took 22 minutes over a 230-item tracker and found seven plugin defects. #24 already says the opposite in two *shipped* audit files, so the catalogue was contradicting the plugin it describes. Corrected, and it keeps the honest half — resumability is the method's central claim and remains untested, because the run was never interrupted. - /qe:check-math's description read \\top, \\tag and \\mathbb where it meant \top, \tag and \mathbb. YAML plain scalars do no escape processing, so those were literal double backslashes in the string that natural-language invocation matches against. It stays unquoted deliberately: \t and \m are not valid double-quoted YAML escapes, so "fixing" it by quoting would break the file. - /qe:copilot-review's banner sent readers to issue #3 for "plan and open questions" about a skill that issue never mentions — #3 tracks the style surface. - benchmark/README.md said skill wiring was "tracked in skills#4". It shipped in 0.3.0. 0.3.1 corrected the version number in that same sentence and left the stale clause, which is how half-fixed lines survive review. - docs/using-skills.md's three status cells, matching the above. qe 0.2.1 -> 0.2.2 and benchmark 0.3.1 -> 0.3.2: both patch, both corrections that leave the procedures unchanged. audit is untouched at 0.1.4.

…ndings (#69–#74) (#76) * workplan-project: replace the plan table with a phase table, and write the tracker once (#69) The mandated `Phase | Issue | Work item` table was a roster of the sub-issue list — membership and order restated in the body, which QEP-6 §7 forbids and its Alternatives section rejects by name. It is replaced with Appendix A's `Phase | Intent | Exit criterion`, which carries what the list cannot. With no Issue column there is nothing to backfill, so step 5's backfill edit and the placeholder-then-fill creation order go, and the tracking issue is written once. The read-back no longer guards a second body write. The construct was C2-conformant: C2 only rules that checkbox progress is never read. The skill now says where the two documents disagree and cites QEP-6 Adoption clause 3, which rules QEP-6 authoritative for structure until the handover, rather than resolving the gap itself. See #69. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: state plan order, link with the native verb, and place re-run items in position (#70) QEP-6 §3 makes the sub-issue list the plan — position is sequence, topmost open item is next — and the skill never said so. A first run produced plan order by accident (create in draft order, link in draft order, linking appends); the documented re-run path broke it, because a recovered item is appended to the bottom regardless of its phase. The invariant is now stated in step 4 (draft file order is plan order; the NN- prefix never reaches a title). Step 5 links with `gh issue edit --add-sub-issue`, which takes plain numbers, in draft order, and the read-back asserts list order. A re-run places what it recovers with the reprioritise API and `after_id`, which is the one sub-issue operation with no gh verb and the one place a database id is needed — the id gotcha moves there, off the common path. The read-back says to check on GitHub rather than the dashboard, whose collector re-sorts children by issue number until status-projects#19 ships. See #70. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: read an issue's parent before linking it, since linking is claiming (#72) Sub-issue membership is single-parent, and gh's `--add-sub-issue` and `--parent` both send replace_parent=true with no opt-out (cli/cli v2.98.0, api/queries_issue.go), so linking an already-parented issue silently detaches it from its tracker. The field test emptied the org's QEP-2 rollout tracker this way and left it publishing 100%. The skill now reads `.parent.number` for every work item that is an existing issue — at draft time, so the method note names the parent the item would leave and the approval covers it knowingly — and a re-run that discovers an issue the drafts did not know stops before linking. Detachments that go ahead are recorded. Step 5 documents the verb's replace behaviour and states why one link method with one guard replaces the two methods that behaved differently on the one case that matters. Cites QEP-6 Adoption clause 2, which makes the read an obligation on conform tooling. See #72. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: add a membership gate between validation and drafting (#71) The extraction bar (actionable, evidenced, material) and the validation step are a quality bar and a truth bar; nothing asked whether a survivor belongs to this project. Drafted from a report bundle, the goal described whatever survived instead of constraining it — the repository-sweep failure the QEP-6 field test produced on cli#20, reproduced by this skill from a different direction. New step 4: write the definition of done before membership is decided, test each survivor against QEP-6 §1's criterion (cited, with only its goal-widening diagnostic restated), split by definition of done rather than by count or phase, file non-members unparented, and record membership exclusions in the method note separately from verification drops. The ~15 gotcha now points at that question instead of at "raise the bar or split by phase". Draft and file become steps 5 and 6; cross-references renumbered. See #71. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: cite the exemplar for content and the evidence bar, and Appendix A for shape (#73) QuantEcon.py#925 was cited as the shape to follow in the same step that calls the stamp form exact. Its heading has been fixed since the finding was filed (it now reads `verified 2026-08-31 14:53 AEST`, parseable under C2 §3.4), but its plan table still carries an Issue and a Status column, so it remains non-conformant for shape under QEP-6 §7 — and once the skill's own table goes (#69) the exemplar would teach what the skill just stopped doing. The shape now comes from QEP-6 Appendix A; #925 is cited for body content and #926 for a sub-issue's evidence bar, with the exemplar's table named as the thing not to copy. See #73. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: give split packages Related work and Gates (#74) The skill's own remedy for an oversized package produces the multi-project case and then had nowhere to record how the packages relate — the relationship lived in the author's head at split time. GitHub records only that a mention happened, so a deliberate relationship between projects is an assertion with no native carrier, which is what the body is for. A multi-package run now drafts a Related work section in each package naming its siblings (projects, never work items), and a gate between packages is written once, on the waiting side, at phase granularity by default — the field result on cli#20/#24 was that a tracker-to-tracker edge would have asserted a constraint that did not exist. Cites QEP-6 §7 and Appendix A rather than restating them. See #74. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: point the status banner at #65 and the report source at /qe:audit-issues The banner said first-run validation is tracked in #3; it is #65, under the family tracker #63. The intro linked the retired `audit` plugin directory, which folded into `qe` at 0.7.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * qe 0.10.0: version bump and changelog for the workplan-project QEP-6 conformance release Minor, because the procedure changes materially: the plan table goes, plan order and single-parent guarding come in, and a membership step is added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * workplan-project: list non-member filings in the mutating-calls table, and use <parent-number> throughout step 6 Copilot's two review comments on #76: the "What this skill writes" table did not mention the unparented issues step 4 now files, and its type row had no target; and step 6 alternated between <parent> and <parent-number> in a section whose point is that the parent is a number and the children in the reprioritise body are database ids. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 28, 2026 07:36

Copilot started reviewing on behalf of mmcky July 28, 2026 07:37 View session

This was referenced Jul 28, 2026

TESTING: audit run 1 on action-translation — seven plugin defects, and a cost claim off by an order of magnitude #21

Open

TRACKING: work plan from 2026-07-29 — run 2 against meta, and the block-2 judgement calls #25

Open

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Comment thread audit/skills/issues/SKILL.md Outdated

Comment thread audit/references/doctrine.md

Comment thread audit/README.md Outdated

Comment thread docs/tutorial-run-an-audit.md Outdated

mmcky mentioned this pull request Aug 3, 2026

Release procedure: version bumps deliver, changelogs record, CI enforces #28

Merged

mmcky and others added 3 commits August 3, 2026 15:52

mmcky force-pushed the doctrine-cost-correction branch from 481e412 to 71bcbee Compare August 3, 2026 05:53

mmcky merged commit 9156b2e into main Aug 3, 2026
5 checks passed

mmcky deleted the doctrine-cost-correction branch August 3, 2026 06:22

Navigation Menu

Uh oh!

audit: re-derive doctrine §4 from evidence the first run actually produced - #24

audit: re-derive doctrine §4 from evidence the first run actually produced#24
mmcky merged 4 commits into
mainfrom
doctrine-cost-correction

mmcky commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmcky commented Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mmcky commented Jul 28, 2026

The claim was load-bearing, so correcting the number is not enough

What run 1 measured

What replaces the justification

Also

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmcky commented Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants