Point the harness at a project, and let an item produce an answer - #185
Merged
Conversation
Two gaps found by a run that delivered six items of worthless work. ## survey: the first run can generate the plan (#181) The harness could be told exactly one thing — here is a PLAN.md, execute it. A user may instead want to say "review and generate a plan to upgrade to Node v22" and have the first run produce the plan, in the form the harness consumes. Mostly assembly rather than invention, because the deliverable is a file the parser already understands: - **The gate is `parse_plan`**, the same reader a hand-written plan goes through. A generated plan the queue would read differently from how it looks is caught here rather than three commands later, which beats a model's opinion of its own output. - **The output is a PLAN.md, never queue rows.** Writing to the queue would fork the pipeline into a generated path and a hand-written one that diverge forever. - **Nothing external happens.** No issues, branches or rows; without `--out` it prints and writes nothing. What is new is that it *reads the project*: documents named with `--doc`, or a short list of guesses, plus the tracked tree and recent history. A named document that is missing is reported rather than skipped silently — the failure this exists to prevent is a confident plan built without the file that states the project's direction. That is exactly what happened: a real repository kept its roadmap in two documents nothing in the harness had ever opened, so seven items were hand-written instead, all real, all delivered, none of them the work that mattered. ## deliverable: an item can produce an answer instead of a diff (#182) Every outcome path assumed a diff. An item like "compare these three approaches" left a clean worktree, hit the clean-tree path, and was recorded `escalated / no_target` — a finished investigation was indistinguishable from an agent that did nothing. Writing the answer to a file instead met the rest of the pipeline: the checks ran against a document, and the reviewer graded it with the diff rubric. An item now declares what it produces. `deliverable: findings` tells the agent to write its answer to `.harness-findings.md` and change nothing else; the answer becomes the item's result and the file is never committed. `code` is the default and behaves exactly as before. **The plan declares this; the agent never chooses it.** Otherwise the first hard test failure becomes an essay about why the test was wrong. A findings item can still refuse — "this question cannot be answered from this repository" is an escalation whatever it was asked to produce — and one that answers nothing lands in the clean-tree path, which is where an agent that did nothing belongs. ## Three defects the tests caught, not the reasoning **Phase headings were read back as work items.** `## P0 Upgrade` matches the item pattern, so every generated plan carried a phantom item whose brief was the phase's *rationale*. `render_plan` gained `phases_as_items`; `inception`'s default is unchanged, because that behaviour is deliberate and documented there and reversing it on one run's evidence from a different context is not mine to do. #184 records what would settle it. **A refusal on a findings item left the answer in the tree**, which then read as an ordinary change and ran the whole code path over it. Both harness notes are now taken unconditionally. **`deliverable:` never parsed.** `_META` whitelists keys, so the handler was unreachable and the value silently defaulted. Closes #181, closes #182. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #181, closes #182. Both found by a run that delivered six items of worthless work.
survey(#181)The gate is
parse_plan— the same reader a hand-written plan goes through — so a plan the queue would read differently from how it looks is caught here, not three commands later. Output is a PLAN.md, never queue rows. Nothing external happens; without--outit writes nothing.A
--docthat is missing is reported, not skipped. That is the failure this exists to prevent: a real repository kept its roadmap in two files nothing in the harness had ever opened.deliverable: findings(#182)An investigation left a clean worktree and was recorded
escalated / no_target— indistinguishable from an agent that did nothing. Write the answer to a file instead and the checks ran against a document while the reviewer graded it with the diff rubric.An item now declares what it produces. The plan declares it; the agent never chooses it, or the first hard test failure becomes an essay about why the test was wrong.
Three defects the tests caught
## P0 Upgradematches the item pattern, so every generated plan carried a phantom item whose brief was the phase's rationale.inception's default left unchanged deliberately; see A generated phase heading is both a tracking umbrella and a claimable item, and it cannot be both #184.deliverable:never parsed —_METAwhitelists keys, so the handler was unreachable.Verified end-to-end against this repository with a scripted model. All four gates green locally.
🤖 Generated with Claude Code