fit-for-purpose: integration, first run, outcome taxonomy, resumable attempts, budgets, holds, telemetry - #145
Open
thedancingdeveloper wants to merge 61 commits into
Open
Conversation
added 30 commits
August 4, 2026 10:35
…e CLI Chains shipped in the CLI and in the stored role map, and `RoleRoute` -- the only way to configure a deployed pod -- could not express one. Found while repointing Node B: the models an operator had chosen could be set locally and not on the machine that runs the work, which makes the feature unshipped where it matters. `models` names the chain in preference order; `model` stays the preferred one and is filled in from the first when omitted, so every existing map, client and reader that knows only `model` is unaffected. Giving both is allowed and checked: a route whose two fields contradict each other behaves differently depending on which one a reader consults, so it is refused with the mismatch named rather than silently resolved.
Three ways a well-formed patch was called corrupt, all found by pointing the harness at a real backlog rather than a toy repository. **A patch that creates a file.** `new file mode 100644` is what git emits for the ordinary way to add a module, and it was not in the list of lines allowed between hunks -- so the validator read it as body content and refused with `line inside a hunk starts with 'n'`. `deleted file mode`, `similarity index`, `rename from`/`to` and `copy from`/`to` were missing for the same reason. **A hunk header that over-declares, mid-diff.** Both places that detect it marked the problem fatal, which contradicts what `PatchProblem` documents about itself: line counts that are wrong are "routinely repaired, notably hunk headers whose line counts are wrong... refusing the second would throw away work `--unidiff-zero` rescues every day". They are now reported and not refused, because another hunk or another file follows -- so nothing was cut off, and `recount_hunks` derives the right counts from the body. The end-of-input case is unchanged: there, a genuine truncation is possible, so it is still refused unless both shortfalls match. **A patch with no hunks at all.** A rename, a delete or a mode change is a complete patch with nothing to hunk. The no-hunk check exists to catch a model answering in prose, so it now looks for a real file header before refusing. Live effect on NGMS: two items that failed at `parse` now reach `apply` and fail, honestly, on what git actually says -- `Cargo.toml: patch does not apply` rather than `corrupt patch at line 53`.
…ent must not assume Four stages sit finished on their own branches and none is merged. This says where each one is, what its report proves, what it leaves unmeasured, and which numbers must not be carried forward.
`Provider` classified failures. The CLI's transport separately assumed one gateway's completion path, one authentication header and one response envelope. Neither knew about the other, so "add a vendor" meant editing the transport, and changing the classifier changed nothing about the wire. A route now names a preset, and a preset supplies the request adapter, the authentication strategy, the response/usage reader and the classifier together. The core preset is generic and claims nothing about any vendor; `chat-completions` and `claw-bay` moved to adapters and are resolved by name through entry-point metadata, so no core module imports one and adding a vendor is a module plus a name. The older `provider` field keeps working and keeps meaning what it always meant -- the classifier only -- so a role map written before this calls the same URL with the same credential and classifies the same way.
Stage A's deterministic transport becomes the classifier conformance fixture. Four configurations -- the generic core route, two adapter presets and one declared outside the package entirely -- are asserted against the same six documented failure shapes: the classification, and then the reaction. What was retried, with which injected backoff and jitter; what was refused; what parked an endpoint and what did not; and what the event said about the route, protocol, classifier and role. Each configuration declares what it produces for each shape, so a build cannot claim a distinction it does not make: a classifier with only HTTP to read collapses caps and refusals into "going too fast", and the table says so rather than excusing it. The suite also proves an unknown price stays unknown instead of becoming zero, that usage reading stays conservative, and that a preset registered from outside runs the whole suite unchanged.
What a route is made of, how a preset is named, discovered and configured, what the older `provider` field still means, and why detection from an endpoint host is printed for a human and acted on by nobody.
Four route configurations through one conformance suite, what each §7.3 criterion is proven by, the migration for a role map written before presets existed, and the limits: no gateway was contacted, no money was spent, and session-mode implementer traffic is still invisible to this telemetry (issue #128).
added 26 commits
August 4, 2026 10:36
What each acceptance criterion is proven by, and the three things this result does not establish: real GitHub service behaviour, assessor model quality, and adoption against a backlog anyone has actually lived in. Records one deviation honestly — the first report of a project is not byte-identical to later ones, because the queue rows it created are new facts the later reports state. Idempotence holds on the effects and as a fixed point from the second adoption onwards.
docs/backlog.json read like a record of current state and is not one: no state field, 44 issues on github that it never held, and every item filed under the superseded P0-P4 phase order. renamed to docs/backlog-seed-2026-08-02.json so the name says what it is, T43 backfilled so the seed matches what was created, and D1's ruling -- github is the sole source of truth for issues -- written into the AGENTS.md table. no attempt is made to sync the two.
…nothing is merged
… nothing init --demo builds a real git repository, a plan, a queue and a route in one directory and leaves the project stopped. one documented command then runs the real executor against a deterministic in-process transport: plan, implement, apply, checks, commit, review. only the transport is replaced, so what is exercised is the harness rather than a demonstration of the demo. the answers are fixed, and it says so — it proves wiring, not model quality. doctor reports route completeness, the resolved protocol and classifier, worktree availability, whether a check command is runnable at all, reviewer independence, cost visibility and whether anything here can mutate github. nothing it reports costs a request; asking a model is opt-in and is otherwise reported as not asked, never as passing. two defects the demo found on the way. run --project X set X running and then claimed from the default project, so a full queue reported nothing to do. and a relative --work failed mid-apply with a git error naming the wrong thing.
README §0 shows init --demo and doctor before anything is configured, and says plainly that a green demo proves wiring and not model quality. USAGE gains the worked first run with its real output, and a separate opt-in path for a local OpenAI-compatible server: who supplies the model, what the endpoint shape is, that authentication is sent and ignored rather than disabled, and exactly what offline does and does not mean. README also now defines tested, observed and proven and says which claims are which, because a reader cannot otherwise tell a deterministic pass from a live one. tests assert the documented flags exist and that the command init prints is the command that works.
Checks.run returned (bool, str). that one bit collapsed four situations: a
check that could not run, a check that ran and the diff is wrong, a failure a
mechanical fix would clear, and a condition needing a person. it now returns a
typed result carrying one of pass, retry, fail, fix_available or escalate, and
still unpacks as (ok, detail) so a caller that only wants the bit is unchanged.
the classification is structural and never semantic: how the subprocess ended,
never what a project's output said. a timeout is a question that was not
answered rather than an answer of no. a missing interpreter and a full disk are
a machine to fix rather than a diff to reject. a declared fix is recorded and
never run.
an attempt now carries a disposition -- completed, refused, crashed, withheld,
escalated -- and a reason kind, both stored on the row and published on
GET /api/work/{id}. a reviewer rejection and a crashed worker are both failed
and are no longer indistinguishable.
attempt accounting is deliberately unchanged. only the two outcomes that did
not exist before decline to consume an attempt; whether a crash should is D11,
it is open, and naming a distinction is not the place to answer it by moving a
number. D8 is untouched: there is no registry here, and a test says so.
a fix is keyed to a check command verbatim and refused if it names one that is not a check -- dead configuration is almost always a typo, and a fix that silently never applies is worse than none because somebody believes it is there. AGENTS.md and USAGE gain the disposition table, the reason kinds and the five check outcomes.
an attempt was monolithic: any crash sent the next claim back to the planner, re-paying for the plan, the context selection and the implementer. a durable per-attempt record now holds one row per stage reached -- planned, implemented, applied, checked, checkpointed, reviewed -- with the artefact that makes it resumable and the graph revision it was admitted at. recording a stage is not the same as resuming at it, and RESUMES_AT says which is which rather than leaving it implied: an uncommitted working tree does not survive a crash, so applied re-applies its stored diff, while a checkpoint is already in git. resumption is never replay -- a model is not deterministic, so a recorded verdict is reused rather than re-asked, which also stops a crash being a way to shop for a different answer. durability is a policy: nothing until exit, at each stage boundary, or with the intent to perform an external effect recorded before it happens. the pre-review git checkpoint is unaffected by all three -- the mode governs the record, not the commit. D11, resolved: a resumed attempt continues the existing one, so max_attempts goes on bounding genuine failures. the consequence is named -- a crash loop is then bounded by a budget, which stage L owes. three defects found on the way. a killed worker left its half-applied diff in the tree and the next attempt's patch failed against a tree already containing it, which read as a bad diff and was not. an operator's retry would have resumed into the verdict it was retrying, so a decision now seals the attempt and a requeue forgets it entirely. and a brief that moves under a live claim discards the position loudly rather than judging old work against a new question.
--durability on run beats the project's setting beats the deployment default. an unknown mode is refused at both doors rather than defaulted: a typo that silently downgraded to exit would look exactly like a harness that had stopped resuming, and nobody would know which. USAGE gains the resume table, the sealing rule and what a killed worker now costs.
the lease bounds a worker's absence, not an item's duration: a heartbeat proves a process is alive and proves nothing about progress. stage H made that worse before it made it better -- D11 ruled that a resumed attempt continues the existing one, so max_attempts stopped counting crashes and an item that crashes in a loop was bounded by nothing. this bounds it. a per-item wall-clock ceiling and a per-item spend ceiling, declared on the project and overridable per item, both defaulting to unlimited so an existing database upgrades with no behaviour change. exceeding either stops the item at the next boundary -- never mid-stage, because stopping in flight destroys the context and leaves a half-finished worktree -- as blocked / escalated, with the ceiling named, and without consuming an attempt. a spend ceiling is not a provider cost cap and is never classified as one. it does not park the endpoint: window_cap and terminal_cap are a provider's statement about our account and belong in the never-retry set, while this is our statement about one item, and parking a shared endpoint because one item was expensive would be that conflation made real. unknown cost stays unknown. a call that reported no usage is unpriced, not free, and while any call is unpriced the recorded spend is a lower bound and the ceiling is reported as unenforceable rather than as met. the wall clock is still checked in that case: an item that has run for a week should stop for the reason that is knowable. the ceilings are readable through the api and through doctor, which warns that unlimited is safe on upgrade and unsafe unattended. whether it should stay the default is D14 and remains open.
waiting_for_input was a projection over recent events. the row stayed claimed, the heartbeat kept stamping, and the lease kept renewing -- so a lease whose whole purpose is to distinguish slow from dead was holding open a human's inbox. nothing bounded it, nothing survived the worker dying, and the answer could only come from the process that happened to be attached. issue #103 is that hole seen from outside. a held item is now a state of the work item, carrying the question, who may answer, and a single-use resume token. D12, resolved: the hold suspends the lease and keeps the claim, so the worktree and the agent's context survive and answering resumes where the item stopped. no other worker can claim it while a person is thinking. the cost of that ruling is named rather than hidden -- a worker slot is tied up for the whole hold -- which is why the maximum duration is not decoration and why, unlike the budgets, its default is not unlimited. expiry returns the item to blocked with the question preserved, never to ready: a hold that times out has not been approved. nothing interprets the answer and nothing writes into a session. both are asserted against the module's code with its prose stripped, because both are the kind of thing somebody adds later while being helpful. GET /api/holds is the inbox, oldest first. an item that is claimed, making no progress and holding no question is a hang, and now reads as one by omission.
an adapter in adapters/, opt-in behind run --otel and lazily imported, so no core module reaches for the SDK and a build without it runs identically. the whole test suite runs with the SDK deliberately not installed, which is how that claim is asserted rather than hoped. model calls, gates and item lifecycle become spans, each carrying the run, item, project and sequence that join it back to the append-only record. a span nobody can join to the events is a second story about the same run. D13, and the safe answer: export only. nothing is written back, nothing is read back, and a test greps the module for the types it must never reach for. a broken collector costs a counter increment -- observation never stops work -- and the tap writes the event before it exports, so telemetry can never come between an event and the record of it. it does not close #128 and says so with a number rather than a caveat. coverage() reports the fraction of known model traffic exported, and a session-mode agent self-reporting a bare count LOWERS that fraction, because a count with no per-call record is honestly less than a span. traffic that self-reports nothing is in neither number, and the note says that too.
not a stage. §13 of the extension proposal records this as worth doing and explicitly not fitness-for-purpose work, so it is here on its own rather than smuggled into one. one machine, one file, one writer, worktrees multiplying peak disk with the worker count, and a backup that is a file copy only after a WAL checkpoint. where the ceiling actually is has not been measured and the paragraph says so: no load test has been run, and the concurrency at which sqlite contention becomes the limit is unknown.
…ocument that matches reality one number derived on one tree -- 1063 -- and one sentence for a hurried reader: every stage that could be run has been run, and not one line of it is evidence that the harness works against a real model. eight defects, all found by a test and all fixed, three of them by pre-existing tests failing. the status document now says where it got to before it says where it was, and names the three things a later agent would otherwise assume: nothing here is live evidence, session mode is behind on stages H and L, and the per-branch test counts are history.
…ot keep them main requires linear history and the verified branch had four merge commits, so no way of landing it preserves the shas the evidence reports cite. the rebased history has a byte-identical tree; the original is preserved on an archive branch and the seven stage branches. a report citing a sha nobody can resolve is a report nobody can check.
thedancingdeveloper
force-pushed
the
fix/validator-rejects-valid-patches
branch
from
August 4, 2026 10:37
cf5a0e6 to
8324210
Compare
thedancingdeveloper
changed the title
…rting from AGENTS.md carried three claims that were flatly wrong. it said this repository does not contain the model client until a later phase relocates it -- it is here and central. it gave a local-run snippet using an ingest --logs flag that does not exist. and its core-module list was a second, drifting copy of the one tests/test_generic.py enforces, in the document that forbids exactly that. it now points at the test, records D10-D14, and states the four gates and the TMPDIR precondition that otherwise costs an afternoon. USAGE gains a Which way in? table, because the four routes -- demo, inception, a plan you wrote, adopt -- were all documented and nothing told a reader which was theirs. each route now says where it goes next; adopt in particular ended without mentioning that it writes queue rows and does not register or start a project. inception is marked API-only, because it has no subcommand and a reader following it would have curled a port with nothing listening. ARCHITECTURE listed project inception as designed and not built; it is built. its component diagram and table were missing graph, adoption, inception, outcomes, attempts, budgets, holds, preflight and doctor. INTERNALS gains the dispositions and the two different budget outcomes, and says the event table is the shape rather than the schema. COORDINATION-PLANE and MULTI-PROJECT-PLAN both said proposed, not implemented at the very top while parts of both were live. each now carries a per-section state table. the stage E1 evidence report gets an appended correction for the link the backlog rename broke, rather than an edit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened as a three-part fix to the patch validator. It has since become the branch the fit-for-purpose programme landed on, and now carries stages F, D, K, H, L, J and M as well. The original validator fix is unchanged and is described at the bottom.
1063 tests,
ruff check,ruff format --checkand full-project strictmypyall clean.Full context:
docs/evidence/2026-08-04-programme-report.md— the §11 report over every delivered stage. Each stage has its own report beside it.What this contains
backlog.jsonrenamed tobacklog-seed-2026-08-02.json; #146 closed.agent-harness init --demo→run --demotakes an item plan→implement→apply→checks→commit→review with no credentials and no network.agent-harness doctorreports what a real run would need and contacts nothing. Local-provider path documented.Checks.run()returned one bit; it now returns one of five typed outcomes. An attempt carries a disposition and a reason kind, so a reviewer rejection and a crashed worker stopped being the samefailed.heldas a real state: survives worker death, answerable from any process, expires toblockedwith the question preserved.Decisions taken
D11 — a resumed attempt continues the existing one, so
max_attemptsbounds genuine failures rather than crashes. D12 — a hold suspends the lease and keeps the claim. Both were put to the owner and recorded in the proposal's §11.1 before the code depending on them was written. D7, D8 and D9 remain open and untouched.Eight defects found on the way, all by a test
Three of them by pre-existing tests failing, which is the system working:
verify:timeout reachedadoptionasfailed, meaning this item is not done — a materially wrong answer;Also:
run --project Xset X running and then claimed fromdefault; a killed worker left its half-applied diff in the tree so the next attempt's patch failed against a tree already containing it; a relative--workfailed mid-apply with a git error naming the wrong thing.What this is not
Every stage report carries a blind-spots section saying which of its own claims are untested.
A note on the commit identifiers
This branch was rebased into a linear history because
mainrequires one and the integration had four merge commits. The rebased tree is byte-identical to the verified commit (3607b82deither way). The original, with its merge commits, is preserved onarchive/fit-for-purpose-2026-08-04, and each stage on itscodex/fit-stage-*branch — that is where the shas quoted in the evidence reports resolve. Explained in the programme report.The original validator fix
Found by pointing the harness at a real backlog instead of a toy repo. Three ways a well-formed patch was refused before git ever saw it:
new file mode 100644and its siblings were missing from_BETWEEN_HUNKSso a file creation read as corrupt; a hunk header that over-declares was marked fatal, contradictingPatchProblem's own docstring; and the recount path discarded a patch git would have applied.