Skip to content

feat(knowledge): hand course-digest rule candidates to /corrections:codify #17175

Description

@kyle-sexton

Parent

Refs #15703
Source: #16029, part 2 of #15729. The list recorded in #16602 gives /knowledge:course-digest the route "call /corrections:codify, presence-gated, with the strongest-first fallback".

Depends on

Outcome

Course-digest's Phase 5 sorts each recommendation into fixed buckets, and its action-items template sends every rule to a "CLAUDE.md / .claude/rules/ candidates" list with "Add to {target file}", which picks the instruction rung for every rule a course suggests. After this change a rule candidate is handed to /corrections:codify when it is available, which picks the strongest rung that can decide it, with the shared strongest-first fallback when it is not. Extraction, analysis and the other recommendation buckets do not change.

Demo: with a corrections stand-in plugin loaded and an already extracted fixture course, run Phase 5. analysis/action-items.md lists a course rule such as "validate request bodies at the boundary" as a rule candidate handed to /corrections:codify, not as a line to add to a named instruction file.

Where the text changes

Line numbers as read on main at a483a2c1; they may move.

File and lines Present wording Change
plugins/knowledge/skills/course-digest/context/workflow.md:273-280 (Phase 5, step 4) "Categorize recommendations ...: CLAUDE.md / .claude/rules/ rule candidates; Skill candidates ...; CI/CD improvements" The first bucket becomes "Rule candidates", each handed to /corrections:codify (gate plus shared fallback)
plugins/knowledge/skills/course-digest/reference/analysis-template.md:56-61 (action-items.md template) "## CLAUDE.md / .claude/rules/ candidates. Rules or conventions worth codifying: - [ ] {Rule description} — Add to {target file}." Heading "Rule candidates"; each entry names the codify hand-off instead of a target file

Call form: /corrections:codify "<the rule>". The digest names the call in its output; it does not run codify unasked.

Decisions

Key interfaces

  • "A skill that carries no ladder copy calls /corrections:codify, presence-gated, with a two-sentence strongest-first fallback. Only a skill that applies its own selection rule holds a synced copy and gets a row in the ladder's Selection rules table." (List enforcement-placing skills and give each one route to the ladder #15729)
  • The seam-phrasing convention: an availability gate, a stated fallback beside it, the plugin named and never the marketplace.

Done when

  • Phase 5 and the action-items template hand each rule candidate to /corrections:codify behind the availability gate, followed by the shared fallback, and no longer send it to an instruction file by default.
  • The skill gains no ladder copy or Selection rules row, and no rung list is restated.
  • The pull request body quotes the sentence that picked a mechanism from a private list before the change. If the recheck finds none, the skill is left unchanged and the finding is reported on List enforcement-placing skills and give each one route to the ladder - part 2 of 2 #16029.
  • One new eval case loads a corrections stand-in plugin and a pre-extracted fixture course (no browser, no download), in the shape of session-flow's re-mechanical-finding-check-proposal case, and grades that a course rule lands as a rule candidate handed to /corrections:codify.
  • The pull request body names the existing case whose run reads the changed text, or says none does (the current cd-* cases stop before Phase 5). The lane runs no eval.
  • A changelog fragment under .changes/knowledge/ describes the change.
  • AC10: no new enforcement-placing skill is added here.
  • Open the pull request with Closes this issue in its body, then stop: the lane adds no label.

After the pull request opens (host session, not the lane)

The pull request stays held with do-not-merge for the eval; a host session runs only the eval and posts its record; the merge lane releases the hold when the record matches the head.

  • Hold line: the hub's sweep (or a host session) adds do-not-merge and posts, from a listed human's account, a comment with the line <!-- pr-hold reason=eval --> on a line of its own.
  • Run budget. List enforcement-placing skills and give each one route to the ladder #15729 asks for "that skill's cases on the opus and sonnet aliases, three runs per side, and no more than 240 runs per plugin". The owner decision of 2026-11-16 in the owner-decisions record (hop 4 Q15, which names List enforcement-placing skills and give each one route to the ladder #15729, amended by hop 7 Q3) replaces the run counts: per changed skill, one targeted before-and-after case on sonnet at 3 runs a side; opus runs only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first. Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error. Ceiling for this plugin: 1 skill x (3 smoke + 6 sonnet + 6 opus confirm) = 15 runs, under the parent's 240.
  • Control: the replaced bucket is a partial cut, so the pull request body carries a detection-control result. No paired no-loss compare (#15703 comment 6906800940).
  • On a LOSS reading: a trace read of each dropped case and an opus confirm; only a loss traced to the changed text counts.
  • Record: the verdict and the control result go into the pull request body. When the run is VALID and shows no loss, the host session posts, from a listed human's account, a comment with the line <!-- eval-pair verdict=no-loss validity=VALID head=<40-hex head SHA> --> naming the current head, keeping "do not merge" and do-not-merge out of that comment (docs/contracts/loop-lane/README.md, "Cross-lane PR hold"). A loss or a run that is not VALID leaves the hold in place; say so on the pull request.
  • Release: pr-merge's scheduled step "Release eval holds whose record matches the head" removes do-not-merge once that record names the current head and no other hold stands. A push after the record needs a new record.

Out of scope

  • Extraction, adapters, the other recommendation buckets, and every other knowledge skill.

Related

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 18, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 18, 2026
  3. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none
    Blast radius: course-digest's Phase 5 workflow text, its action-items template, one new eval case and a changelog fragment; read by anyone running /knowledge:course-digest.
    Reversible: yes; revert the text edits and remove the eval case.
    Open questions: none
    I read this as: Phase 5 and the action-items template should hand each course rule candidate to /corrections:codify, behind the availability gate and with the shared fallback, instead of naming an instruction file. The outcome and done-when are exact. The pull request body quote and the eval run need a person or host, which does not hold this issue. No open issue asks for the same thing; #17174, #17172, #17163, #17179, #17162, #17158, #17160 and #17159 are sibling follow-ons for other skills. No merged pull request in the list already fixes it.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 18, 2026
  5. melodic-automation-lanes commented on Nov 18, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (tracker-output: "The pull request body quotes the sentence that picked a mechanism from a priv..."), item 4 (eval: "One new eval case loads a corrections stand-in plugin and a pre-extracted fix..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. melodic-automation-lanes commented on Nov 18, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Course-digest's Phase 5 and action-items template hand each rule candidate to /corrections:codify behind the availability gate with the shared fallback, instead of naming an instruction file.

    Split. Done-when items 3, 4 of the issue body need a person: they are checks a person runs, listed below, and the lane does not do them. This issue keeps the rest of its done-when.

    Not done by the lane: checks a person runs

    External state: none

    Dependencies

    • none open; the knowledge cut #​16766 is closed

    Related work

    Sources read

    • plugins/knowledge/skills/course-digest/context/workflow.md:274: Phase 5 step 4 lists the bucket 'CLAUDE.md / .claude/rules/ rule candidates' and still has it as of the base checkout
    • plugins/knowledge/skills/course-digest/reference/analysis-template.md:56: action-items template still has the heading '## CLAUDE.md / .claude/rules/ candidates' with 'Rules or conventions worth codifying' at line 58
    • plugins/knowledge/skills/course-digest/evals/evals.json:64: the existing Phase 5 case grades a 'CLAUDE.md/rule' bucket, so the existing case that reads the changed text may need its wording checked
    • plugins/knowledge/skills/course-digest/SKILL.md:264: describes action-items.md as holding rule candidates; no codify reference exists in the skill yet

    Automated refinement by the lanes App. Intake triages the issue again with it.

  7. melodic-automation-lanes commented on Nov 18, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"f00eae579e176637fdeef23812e01a1b0c1f1ad1857879a65187e54f1aeca69d","body_sha256":"bfea3d39d8a3c093fc61d2050ed866f7f453156fd7b7590fa2261c216b199582"}

  8. melodic-automation-lanes commented on Nov 18, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"f00eae579e176637fdeef23812e01a1b0c1f1ad1857879a65187e54f1aeca69d","body_sha256":"bfea3d39d8a3c093fc61d2050ed866f7f453156fd7b7590fa2261c216b199582"}

  9. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor
    No description provided.
  10. kyle-sexton commented on Nov 19, 2026

    @kyle-sexton
    MemberAuthor

    Closed by #17216 (merged 2026-11-19 09:49 UTC). For the record, what the host eval did and did not run:

    • Run: one targeted before-and-after case on sonnet, 3 runs a side (the eval record on feat(knowledge): hand course-digest rule candidates to corrections:codify #17216, issuecomment-6997295013): PASS, 0.714 before, 1.000 after; rule-candidate-to-codify and no-instruction-target went from 0 of 3 to 3 of 3. The reference/analysis-template.md row was measured; the context/workflow.md row was not (opened in 0 of 3 runs before, 1 of 3 after).
    • Not run: the opus confirmation and the detection control. The gate in force is sonnet at 3 runs a side.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.

    Fields

    Work status

    needs-triage

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions