Hold done to a checklist and to a reason for every changed test - #65
Merged
Merged
Conversation
added 5 commits
September 29, 2026 23:00
…o to v0.0.9 jolt-crypto moves with http-client, which is released against v0.0.9; our top-level pin was 15 commits behind it and would have won resolution.
An answer could ship a requirement by never mentioning it, and a run could change a test that existed before it started without saying so. Both are now mechanical checks at done. The checklist is built from data that exists before the work: the list items of what the branch was asked (the problem, or a board piece's task contract), the task's tests when they are prose, what the branch declared with plan, and for a branch holding no task the operator's acceptance criteria. done must carry an entry per item - met, not_met or n/a, with a reason - and the accounted list is appended to the shipped answer. Every test that was at the run's baseline and is not in the tree unchanged has to be named in done's changed_tests with a reason. Clojure test files are compared as whole forms, so a moved test or a whitespace edit is not flagged; other files fall back to their assertion lines. Also fixes gitdiff reporting repository-relative paths, which was wrong for a project nested inside a larger repository.
Run 582980ef on endless-flight showed four problems: - A board piece's contract is one line, so it had no list items and the piece owed nothing. A contract with no list items is now one item. - A revision branch carries its task while the claim stays with the original branch id, so it was read as the run's answer and owed every acceptance criterion. Carrying a task now counts as working one. - The model labelled its checklist items c1..c4 and reworded them on each re-plan. The union grew to c8 while it kept answering c1..c4. A labelled item that names an existing one is now a restatement. - changed_tests named the test as ns/name and as name (file). Both now match the bare test name. The scripted models in the board, feature and team tests share one checklist helper instead of three copies.
…em text
Run 6e3eda8a sent plan's checklist as [{"c1": "..."}] and stored the
braces as the item text, then named its done entries by sentence instead
of id. A one-entry map is a labelled item now, and an entry whose key is
an item's exact text counts for that item; a paraphrase still does not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two model-free checks at done, for karamazov-dsfx and karamazov-fgsb.
An answer could ship a requirement by never mentioning it. done now owes one entry per checklist item, met / not_met / n/a with a reason, and a missing item is refused. The items come from data that exists before the work: the list items of what the branch was asked (the problem, or a board piece's contract; a contract without list items is itself one item), the task's tests when they are prose, what the branch declared with plan, and for a branch working no task the operator's acceptance criteria. The accounted list is appended to the shipped answer for the critic.
A run could also change a test that existed when it started without saying so. Every test at the run's baseline that is not in the tree unchanged now needs a reason in done's changed_tests. Clojure test files are compared as whole forms, so moves and whitespace are not flagged; other files fall back to their assertion lines.
Before settling on this I measured the model route with lev on 50 labeled cases per question: the BERT encoders were at chance and never answered "absent", Qwen3.5-4B called four of five silent answers "met" at 0.95+, and MiniCPM5 thinking got 92% at about 7 s per item.
Two endless-flight runs on GLM-5.3 exercised it. The first found four rough edges (one-line contracts owed nothing, revision branches were held to the run's criteria, relabelled checklist items piled up, loosely named tests did not match), fixed here; the second found two more (checklist items sent as maps, entries named by text), also fixed. In the second run both combo pieces shipped through the checklist and the ratchet with the rewritten scoring test explained, and a partial end-to-end piece shipped honestly as partial.
Also: gitdiff printed repository-relative paths, which broke a project nested in a larger repo; http-client v0.0.17, ring-chez-adapter v0.7.10 and jolt-crypto v0.0.9.