Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions docs/SURFACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -724,7 +724,11 @@ compiler/SDK validation limit; the kernel validates positive milliseconds
fitting in `i64` and carries the declaration to the worker.

```ts
const repair = await f.agent('repair', { task: 'Repair failing checks.', timeout: '45m' });
const repair = await f.agent('repair', {
task: 'Repair failing checks.',
timeout: '45m',
transportRetries: 0, // the deadline is the result; do not continue
});
if (repair.completionReason === 'timeout') {
await f.run('git push'); // publish what the agent already committed
}
Expand All @@ -751,11 +755,18 @@ native artifact paths remain under `trajectory_tail.transcript.artifacts.paths`
in the journal (a bounded list). Wrappers discard partial stdout on timeout
and provide the deadline message; they have no transcript artifact list. A
wrapper result envelope emitted before the deadline still supplies its usage.
No workspace reset occurs: committed work and uncommitted edits remain, and
the following step sees that potentially dirty tree even with
`recoveryMode: 'reset'`. Unlike crash/lease recovery in RFC Appendix A, timeout
settles the step with no successor attempt. Resume replays the recorded timeout
and does not execute that agent again. Incurred spend remains charged:
No workspace reset occurs on the settling attempt: committed work and
uncommitted edits remain, and the following step sees that potentially dirty
tree. The default transport budget is one. A deadline then journals `timeout`
with disposition `retry`, the dirty end pins, and the trajectory tail, and the
next attempt recovers as `inspect` even when `recoveryMode` is `reset`. The
continuation sees the edits and the journaled tail. The dirty revision's
stamp includes file contents and deletions, so a deletion-only edit and two
different edits of the same paths are distinct pins. After a human answers a
`manual` park, that attempt inspects the dirty pins. A second deadline seals
the step. Set `transportRetries: 0` when the deadline itself is the result you
branch on: that attempt settles with no successor, and resume replays the
recorded timeout without executing the agent again. Incurred spend remains charged:
reported usage is priced as usual, and usage never reported before the deadline
is journaled as dollar-unmetered rather than as a measured $0.

Expand Down
27 changes: 23 additions & 4 deletions kernel/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,9 +296,25 @@ retry; `inspect` resumes inside the dirty workspace with the failed attempt's
tail injected; `manual` parks as `wait.human` with `diff_ref`. Infrastructure retry
is a separate, explicit budget (`retry.max_transport_retries`, default one so a
single process loss remains resumable; set zero to disable).
Only `crashed` and `lease_expired` consume it. `worker_error`, timeout, budget,
cancellation, and an ordinary nonzero CLI exit are terminal regardless of the
budget; semantic verification retry remains bounded only by `max_iterations`.
`crashed`, `lease_expired`, and an agent CLI `timeout` consume it. A deadline
is not a judged result: it does not increment `semantic_executions`, and the
completion keeps the worker's end pins and trajectory tail. The end revision
is the start revision plus a stamp of the workspace delta: each added or
changed file by its content signature, each deletion by its path. An empty
diff keeps the start pin. The next attempt recovers as `inspect` even when
the step declared `reset`, so the dirty pin and the tail are what the
continuation sees. The attempt-started record still carries the declared
mode. `manual` still parks for a human. After that answer the attempt
inspects the dirty pins: the attached worker was advanced to them, and a
retry that asked for the clean start pin would be refused. A second deadline
seals when the transport budget is the default one. A local agent with a
deadline declares the surface `local-workspace`, which that worker holds at
revision `local` until a deadline stamps it. Selection prefers a worker
already standing at the attempt's revisions, so a fresh worker still at
`local` does not take the continuation. A deterministic command timeout,
`worker_error`, budget, cancellation, and an ordinary nonzero CLI exit stay
terminal regardless of the budget; semantic verification retry remains
bounded only by `max_iterations`.
One kernel-side `worker_error` is not terminal: a dispatch the kernel refuses
because the attached worker does not hold the attempt's journaled pins
(Appendix A rule 2, `DispatchOutcome::PinMismatch`) ran nothing, so the step is
Expand All @@ -308,7 +324,10 @@ re-elected against those same pins under `max_iterations`
The kernel learns of a dead attempt two ways, and the recovery mode applies to
both: the kernel notices an abandoned lease (`abandonment_actions`), or the
worker reports its own loss through `step.complete` with `crashed` /
`lease_expired` (`completion_actions`). Under `manual` either path journals
`lease_expired` (`completion_actions`). A CLI deadline is a third report,
`step.complete` with `timeout`: unlike abandonment it keeps the end pins and
the tail, because the worker still holds the dirty tree. Under `manual` a
crash, an expired lease, or a CLI deadline journals
`step.completed` with `disposition: park` and then the `wait.human` a human
answers, each its own append. A process death between the two leaves the step
folded to the placeholder wait id `park-<step>-<attempt>` with nothing to
Expand Down
18 changes: 14 additions & 4 deletions kernel/relayflowd-core/src/machine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,9 @@ fn start_actions(state: &RunState, step: &StepSpec, attempt: u32, now_ms: i64) -
surfaces,
..
} => {
let retry_pins = match recovery_mode {
let effective_mode =
recovery::recovery_mode_for_attempt(*recovery_mode, runtime.last_completion_reason);
let retry_pins = match effective_mode {
RecoveryMode::Inspect => runtime
.last_end_pins
.as_ref()
Expand All @@ -267,9 +269,9 @@ fn start_actions(state: &RunState, step: &StepSpec, attempt: u32, now_ms: i64) -
None => carried_pins(state.current_pins.as_ref(), surfaces),
};
let recovery = RecoveryInstruction {
mode: *recovery_mode,
mode: effective_mode,
restore_pins: (runtime.last_completion_reason.is_some()
&& *recovery_mode == RecoveryMode::Reset)
&& effective_mode == RecoveryMode::Reset)
.then(|| pins.clone()),
previous_completion_reason: runtime.last_completion_reason,
trajectory_tail: runtime.trajectory_tail.clone(),
Expand Down Expand Up @@ -445,10 +447,15 @@ fn settle_attempt(
result.failure_reason,
None | Some(CompletionReason::VerificationFailed)
);
// A CLI deadline stopped the process; it is not a result a gate judged.
// A deterministic command timeout stays that command's result and does
// not join this budget.
let agent_deadline = matches!(step.kind, StepKind::Agent { .. })
&& matches!(result.failure_reason, Some(CompletionReason::Timeout));
let transport_failure = matches!(
result.failure_reason,
Some(CompletionReason::Crashed | CompletionReason::LeaseExpired)
);
) || agent_deadline;
// Appendix A rule 4: a dead attempt under `manual` parks as `needs_human`
// with a diff, whichever way the kernel learned of the death. The
// abandoned-lease path (`abandonment_actions`) always honoured that; a
Expand Down Expand Up @@ -622,10 +629,13 @@ fn deterministic_ulid(
}

mod recovery;
pub(crate) use recovery::recovery_mode_for_attempt;
pub use recovery::{abandonment_actions, recovery_actions, recovery_actions_filtered};

mod parallel;

#[cfg(test)]
mod deadline_tests;
#[cfg(test)]
mod parallel_tests;
#[cfg(test)]
Expand Down
Loading
Loading