Skip to content

Start a step's answer window when its session starts (nxf 6j6v.1wep) - #39

Merged
cabcookie merged 8 commits into
fix/9g8j-hand-off-one-step-per-ordered-runfrom
fix/1wep-answer-window-starts-with-the-session
Oct 10, 2026
Merged

cabcookie merged 8 commits into
fix/9g8j-hand-off-one-step-per-ordered-runfrom
fix/1wep-answer-window-starts-with-the-session

Conversation

@cabcookie

Copy link
Copy Markdown
Member

Stacked on #38 (6j6v.9g8j, fix/9g8j-hand-off-one-step-per-ordered-run). Merge #38 first; this PR's base is that branch.

The bug

Measured in the agents workspace on 2026-10-09/10 (0.206.1). Operation v05a's build step was queued behind th4p's working-copy lease for six hours and never started. At its declared window the tick logged step "build" gave no answer inside its declared window; the flow advanced to step "verify" without it, and a verify slot was opened for a build that never ran.

The rule

A step's answer window starts when its session starts. A step whose trigger still waits in the working-copy queue has not been asked, so it cannot lapse.

  • supervised_member_threads (the set the supervisor waits for, read by the tick's tick_move and by the push path) clears stale for any member whose thread has a row in working_tree_queue. That covers both cases from the item:
  • ThreadQuorum::stale is not redefined. M2 §4.3 says its derivation "is unchanged and still the only one", so the quorum SQL is untouched, and nxc threads, nxc status and the opener's wake read it as before. Only the supervisor's reading of its own set changes, the same kind of check as the 10yb liveness gate.
  • A started step is not in the queue, so a step that started and then fell silent lapses on its declared timeout: exactly as before.

Same root, fixed here

  • Lease bound at hand-off. fire_queued_trigger used to restart the member's clock after trigger_role. The inherit inside trigger_role therefore derived the lease bound (operation_lease_expiry, which takes the latest outstanding window) from the window stamped at queue time. Mutation B below shows the result: the lease expired at 01:01, while the hand-off happened at 07:00:30. The restart now runs before the trigger, so the started member's window is the latest in its area. It stays best-effort and never returns early. If the start fails or is queued again, the restarted clock does no harm: a queued member cannot lapse and is restarted again when it really starts.
  • Nothing watched the restarted window. The tick armed at fan-out fires at the old instant, finds the member queued and declines, and a past instant is never re-armed. So a member that starts and then falls silent would only have been noticed by chance. After a Spawned admission, fire_queued_trigger now re-arms the channel thread's tick through rearm_after_the_members_moved. This is best-effort, after the trigger.

Tests

New file crates/chat/tests/a_queued_step_is_not_asked_until_it_starts.rs, at the library seam. It follows the conventions of a_hand_off_starts_one_step_per_ordered_run.rs: a recording worker that answers liveness, and the tick through the compute layer. The hand-off goes through the tick's sweep, as in the field: the holder declares no window, so its lease ran out at the two-hour fallback and only its live builder pinned the copy.

  1. a_step_queued_behind_the_working_copy_does_not_lapse_and_its_window_starts_when_it_does
    • The coder is queued for six hours. A tick past its window does not advance the flow, reports no lapse, and opens no verifier.
    • The holder lets go and the coder starts. The hand-off arms a job for the channel at start + 1m.
    • A tick 30 s into that window does not advance.
    • A tick past it advances, with one StepUnanswered.
  2. a_started_step_that_falls_silent_still_lapses_on_its_window is the guard for the declared timeout: of a started step. It is green with and without the fix.
  3. the_lease_handed_on_after_a_long_wait_is_not_already_expired: the lease row expires after the hand-off instant, and working_tree_holder(now) returns the round.

How they were shown red:

  • Without any production change, test 1 failed at its first assertion (reason: "advanced", with the field's exact step "coder" gave no answer… finding).
  • With the fix, three mutations were applied, then restored:
    • A: the queue check is disabled (if false && members.iter().any(|m| m.stale)). Test 1 goes red.
    • B: the clock restart is moved back after trigger_role. Test 3 goes red: expires at 2026-10-10T01:01:00Z, handed on at 2026-10-10T07:00:30Z.
    • C: the re-arm after the start is disabled (if false && matches!(admission, …)). Test 1 goes red on the armed-job assertion.

#38's tests were adapted. a_hand_off_starts_one_step_per_ordered_run.rs built its "two steps of one run in line together" premise from the 1wep lapse, which this PR removes. It now builds the premise by posting an answer under the queued coder's session before that session ever starts. This settles the step and opens the verifier behind it. All five tests keep their assertions. The coder's start then carries no reply obligation, so the tests take the coder's slot from the queue entry instead of from reply_thread. The prose in that file and in one_step_per_ordered_run's doc now says the lapse route is closed.

Run locally

  • cargo fmt
  • cargo test -p nexus-chat --test a_queued_step_is_not_asked_until_it_starts --test a_hand_off_starts_one_step_per_ordered_run
  • Gates: --test read_surface --test seam_disposition --test verb_seam. No read_surface row changes, because the set of calls that reach trigger_and_bind is unchanged.
  • Neighbouring flow, timeout and lease suites (after cargo build -p nxs):
    • channel_member_timeout, channel_tick, channel_flow, channel_supervisor
    • the_flow_waits_for_a_live_session, a_clock_watches_the_liveness_gate
    • a_start_that_failed_is_not_an_orphan, a_step_ends_when_its_session_ends, a_holder_past_its_bound_is_parked, an_unanswered_escalation_parks_and_comes_back
    • withdraw_a_running_round, a_way_out_of_a_held_lease, orchestration_trigger, working_tree_claim_scope
  • cargo test -p nexus-chat --lib -- orchestration::tests working_tree::tests and -- guide
  • cargo xtask changelog check origin/main

No clippy and no full suite were run locally; CI is the gate.

Recorded, not changed here

Open decisions

  1. A failed start pending retry is not treated as queued. Its trigger is off the queue and in failed_starts, so its window can still lapse. A permanently failed start that could never lapse would stall its flow forever, so I kept to "queued or deferred". The clock restart now runs before the trigger, so a failed start gets a fresh window from the attempt instead of keeping the one from queue time. Should a start with a retry due also hold off the lapse?
  2. Replicas. The queue is device-local, so a sync peer still sees the queued member's op-folded deadline and could read it as lapsed. This is the same bound the member_deadline.rs module docs already record for the resettable clock.
  3. The tick's reason for a queued member is still not_due. A distinct reason, such as waiting_for_the_working_copy, would tell an operator why. That would add a reason the CLI branches on, so I left it out.

No spec under docs/specs/ is contradicted: §4.3's derivation of stale is untouched, and the change is to what the supervisor counts as settled. The guide's channels page (EN/DE) now says that a window starts when the member's session starts.

🤖 Generated with Claude Code

https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ

cabcookie and others added 2 commits October 10, 2026 08:44
A build step queued six hours behind another operation's working-copy
lease reached its declared window, still queued and never started. The
tick read that as silence, reported that the step gave no answer, and
opened the verify step for a build that never ran.

- supervised_member_threads no longer counts a member as stale while its
  trigger waits in the working-copy queue. That covers a step queued
  behind another lease and a later step of an ordered run put back in
  line at the hand-off (Promotions::deferred). ThreadQuorum::stale is
  still derived exactly as M2 section 4.3 says; only the supervisor's
  reading of its set changes. A started step that falls silent lapses
  on its declared timeout as before.
- fire_queued_trigger restarts the member's clock before the trigger,
  so the lease the inherit derives is bounded from the start instead of
  from windows that ran out in the queue (it could be expired the
  moment it was granted).
- After a queued member starts, the tick for its channel is re-armed at
  the restarted window, which nothing watched before.
- PR #38's hand-off tests build their two-steps-in-line shape with an
  answer posted under the queued coder's session, since the lapse no
  longer produces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
…v.ys54)

A delivered operation kept the working-copy lease to its bound. A verify step
of an ordered run had escalated; the requester answered on the thread above
it, the run started again in new sibling step threads and delivered, and
nothing was written in the escalating thread again. Its escalation stayed the
last reply there, so the release rule read it as unanswered for ever.

A hand-back now counts as answered once a thread above it in the claim area
was declared again after it (the turn watermark, read upward, vouched
declarations only). An escalation nothing above answered still holds the copy,
and so does a question two levels down under consolidating levels: replies go
up, declarations come down.

`nxc status` names the hand-back that keeps a copy with no thread open:
`StatusOperation::held_by_hand_back` (additive, the struct is
`#[non_exhaustive]`) and a detail line in the human output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
cabcookie and others added 3 commits October 10, 2026 10:29
…ad (nxf 6j6v.ys54)

The first cut let any hand-back be answered by a re-declaration above it,
which broke the pinned guard
`working_tree::tests::an_escalating_board_is_handed_back_and_a_session_scope_has_nothing_to_ask`:
a question on a DM under a board was cleared when the board was asked again
about something else and reported.

The guide's rule is the reason to split them: `--escalate` goes UP to whoever
commissioned the sender, so the commissioner continuing above it is the
answer. A question is not handed upward and waits in its own thread. The
second door now applies to an escalation only; the field case stays green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
- A failed start with a retry still due counts as not asked: its window
  no longer lapses before the retry, so the flow does not open the next
  step and the retry does not start the earlier step beside it. A failed
  start with no retry due lapses as before. (Code #1, Integrity #2)
- A step whose window has passed while its start still waits is named
  on the declining tick's receipt and on stderr, as the new
  ConsequenceClass::StepStillWaiting, pointing at nxc status and
  nxc withdraw. (Integrity #1)
- The re-arm after a queued start reports a failure as a
  tick_unscheduled finding through the callers' warnings instead of
  dropping it, and the nest is flattened. (Code #2, #3, Integrity #5)
- Tests: a failed start with a retry due, one with none, the still-
  waiting finding, a later step put back in line by the hand-off, a
  failed start arms no window, and the lease runs at least to the end of
  the started window. (Test #1, #2, #4)
- #38's hand-off tests rebuild the two-steps-in-line shape as an earlier
  build left it, with the coder still owing its answer. (Code #4, Test #6)
- Docs: supervised_member_threads names failed starts and what ends the
  wait; the channels guide (EN/DE) covers retried starts, the finding,
  withdraw, the board's stale flag, replicas and absolute instants;
  channel_member_timeout's comment no longer describes the retired
  behaviour. (Code #5, Integrity #3, Test #7)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
@cabcookie

Copy link
Copy Markdown
Member Author

Review mitigation and new base (head fa5a8a7)

Base merged: origin/fix/9g8j-hand-off-one-step-per-ordered-run at ee75b7f. Conflicts:

All 12 tests in #38's file and all 7 in this PR's file pass.

Finding Outcome Note
Code #1 / Integrity #2: a failed start with retries left can lapse Fixed #38's change did not settle this here. The new test was red on the merged base: reason: "advanced". A failed_starts row with retry_at set now counts as waiting; a row with no retry due lapses as before.
Code #2 / Integrity #5: the re-arm finding is discarded Fixed fire_queued_trigger takes a noticed sink, and both callers pass their receipt warnings.
Code #3: the re-arm nest Fixed The redundant member_deadline pre-check is gone, and a supervising_parent error is reported.
Code #4: the coder starts for an already-settled step Fixed #38's tests no longer answer the coder by hand. Filed nxf 6j6v.fzd5 for the hand-settled-while-queued case, and added a note on 55b1.
Code #5: wedge risk undocumented Fixed supervised_member_threads docs and the guide name what ends the wait (copy freed, retry, nxc withdraw).
Test #1: a later step put back in line is untested Fixed a_later_step_put_back_in_line_does_not_lapse_while_it_waits. Red without the queue check: the tick went for the consolidation (no_return_address).
Test #2: failed or re-queued start untested Fixed (failed) Covered by the two failed-start tests. A re-queued start needs a re-parented thread; see Test #3.
Test #3: Spawned-only re-arm checked one way Declined (in part) The failed path returns at trigger_role(..)? before the re-arm, so it cannot reach the gate. A Queued admission from the queue needs a thread re-parented between grant and fire, which has no seam at the library handle. A "failed start arms no window" assertion was added, but it cannot tell the Spawned gate from the ?.
Test #4: lease assertion too loose Fixed Now expires >= FRESH_WINDOW_ENDS.
Test #5: guard test green both ways Declined It is a guard by design. The queued→started→silent path is test 1's tail.
Test #6: names in #38's tests Fixed The premise once again lapses the coder at LAPSED, and the coder answers its own owed slot.
Test #7: stale comment in channel_member_timeout.rs Fixed
Integrity #1: a queued-forever wait is silent Fixed New ConsequenceClass::StepStillWaiting on the declining tick's receipt, plus stderr. Test red without it.
Integrity #3: absolute-instant members Fixed (docs) The guide says an absolute instant does not move. Changing that in code would contradict the declared meaning: "this moment, whatever happens".
Integrity #4: cross-process gap between queue delete and clock restart Declined The gap lies between the hand-off's commit and fire_queued_trigger's first statement, in one process. Closing it means restarting member clocks inside hand_the_working_tree_on's transaction, in #38's store code. The same window already exists for the lease grant (documented on fire_queued_trigger).

Red without the fix, shown by mutation:

  • Queue check removed: 5 tests red.
  • Retry-due failed starts not counted as waiting: the retry test goes red.
  • Every failed start counted as waiting: the no-retry test goes red.
  • Finding not attached: the finding test and the deferred-step test go red.
  • Re-arm disabled: test 1 goes red.
  • Clock restarted after the trigger: the lease test goes red.

Run locally:

The fragment is now facade: changed, because of the new enum variant (the enum is #[non_exhaustive]).

cabcookie and others added 3 commits October 10, 2026 16:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
An escalation is answered from above only when a thread above it was BOTH
written in by its commissioner and declared again after it. A supervisor or
engine re-declaration with no word from the commissioner (a channel
concluding, a delivery retarget, a withdrawal) answers nothing; an empty
register asks nobody; an engine opener is nobody's commissioner. Questions
are still answered only in their own thread.

- status: `held_by_hand_back` carries the kind (`HandBackStatus { thread,
  kind }`) and reads the lease holder's claim area, the release's own input;
  the human line says where the answer goes, differently for a question.
- status: `needs_decision` no longer counts an escalation answered from
  above, so a delivered run stops showing NEEDS DECISION.
- tests: seam test for a question under a commissioner carrying on above it
  (held, named, non-vacuous), mid-redo hold, needs_decision after delivery;
  unit tests for the supervisor re-declaration, the note that asks nobody,
  the whatever-it-says decision, the question branch, two levels up, the
  area edge, a parent cycle, vouching, the empty register, engine openers,
  and the CLI line.
- docs: guide (en/de) commands and limits-and-safety, changelog fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184D5PqXGRbwMKDkMkRggdZ
…red-hand-back

Let an escalation answered above it release the working copy (nxf 6j6v.ys54)
@cabcookie
cabcookie merged commit 92f7590 into fix/9g8j-hand-off-one-step-per-ordered-run Oct 10, 2026
cabcookie added a commit that referenced this pull request Oct 10, 2026
…swered escalations release, store-free lanes, next filters and paging, per-type ranking, self-update restarts the service (nxf 6j6v.9g8j, 1wep, ys54, t1ym, 15ed, z9jk, c92y) (#38)

Folds the reviewed and mitigated PRs #38 (9g8j), #39 (1wep), #40 (ys54), #41 (c92y), #42 (t1ym), #43 (15ed) and #44 (z9jk).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant