feat(exec): retain terminal sessions after exit - #1146
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe execution subsystem now tracks live, retained, and tombstone states. It supports terminal output summaries, exclusive attachment leases, ID validation, reservation-based startup, timeout cancellation, lifecycle pruning, shutdown cleanup, and explicit initialization registration failures. ChangesExecution lifecycle
Estimated code review effort: 5 (Critical) | ~90+ minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant ExecutionService
participant ExecutionRegistry
participant ExecutionState
participant OutputManager
Client->>ExecutionService: start execution
ExecutionService->>ExecutionRegistry: reserve execution ID
ExecutionService->>ExecutionState: spawn and configure execution
ExecutionService->>ExecutionRegistry: publish execution state
ExecutionRegistry->>ExecutionState: observe terminal state
ExecutionState->>OutputManager: collect terminal output summary
Client->>ExecutionService: attach or wait
ExecutionService->>ExecutionRegistry: look up lifecycle state
ExecutionRegistry-->>ExecutionService: return live, retained, or tombstone state
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
📦 BoxLite review — couldn't completepowered by BoxLite |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/guest/src/service/exec/state.rs (1)
416-462: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winA displaced
output_taskcan trip the assert and leak its handle.
ConsumerLease::dropinsrc/guest/src/service/exec/output.rsLines 55-60 releases the lease when theAttachStreamis dropped. TheAttachStreamis owned by the forwarder task spawned at Line 437, so it drops as that task body ends, beforeJoinHandle::is_finished()reportstrue.In that window a second
attach_outputcall can:
- run
join_finished_output_task, which returnsNonebecause the previous handle is not finished yet,- pass the
releasedcheck,- claim the now-free consumer lease at Line 431 or Line 432,
- reach Line 451 with
inner.output_taskstillSome(..).
debug_assert!(inner.output_task.is_none())then panics in debug builds. In release builds Line 452 overwrites the previous handle, so that handle is dropped without being awaited.Take the existing handle instead of asserting it is absent, and await it after the lock is released.
🐛 Proposed fix to displace the previous handle safely
- let task_to_abort = { + let (task_to_abort, displaced) = { let mut inner = self.inner.lock().await; if inner.released { - Some(task) + (Some(task), None) } else { - debug_assert!(inner.output_task.is_none()); - inner.output_task = Some(task); - None + let displaced = inner.output_task.replace(task); + (None, displaced) } }; + if let Some(displaced) = displaced { + // The lease was released as the previous forwarder ended; let it finish. + let _ = displaced.await; + } if let Some(task) = task_to_abort { task.abort(); let _ = task.await; return Err(ExecutionError::HandleUnavailable); } Ok(rx)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/guest/src/service/exec/state.rs` around lines 416 - 462, Update attach_output to safely displace any existing inner.output_task instead of asserting it is absent. While holding the lock, replace the previous handle with the newly spawned task, then release the lock and await the displaced handle before returning or completing the attachment, preserving the released-state cleanup and HandleUnavailable behavior.
🧹 Nitpick comments (4)
src/guest/src/service/exec/output.rs (1)
253-294: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract the duplicated summary construction.
terminal_summaryandsealed_terminal_summarybuild the identicalOutputTerminalSummaryfromstate; only the precondition differs. A private helper onOutputStateremoves the duplication and keeps the two preconditions visible.♻️ Proposed refactor
pub(crate) async fn terminal_summary(&self) -> Option<OutputTerminalSummary> { let state = self.inner.lock().await; if !state.stdout.finished || !state.stderr.finished { return None; } - - Some(OutputTerminalSummary { - stdout: OutputStreamSummary { - enabled: state.stdout.enabled, - total_bytes: state.stdout.total_bytes, - }, - stderr: OutputStreamSummary { - enabled: state.stderr.enabled, - total_bytes: state.stderr.total_bytes, - }, - reader_failure: state - .failure - .as_ref() - .map(|failure| failure.message.clone()), - }) + Some(state.summary()) } pub(crate) async fn sealed_terminal_summary(&self) -> Option<OutputTerminalSummary> { let state = self.inner.lock().await; if !state.sealed { return None; } - Some(OutputTerminalSummary { - stdout: OutputStreamSummary { - enabled: state.stdout.enabled, - total_bytes: state.stdout.total_bytes, - }, - stderr: OutputStreamSummary { - enabled: state.stderr.enabled, - total_bytes: state.stderr.total_bytes, - }, - reader_failure: state - .failure - .as_ref() - .map(|failure| failure.message.clone()), - }) + Some(state.summary()) }Add to
impl OutputState:fn summary(&self) -> OutputTerminalSummary { OutputTerminalSummary { stdout: OutputStreamSummary { enabled: self.stdout.enabled, total_bytes: self.stdout.total_bytes, }, stderr: OutputStreamSummary { enabled: self.stderr.enabled, total_bytes: self.stderr.total_bytes, }, reader_failure: self.failure.as_ref().map(|failure| failure.message.clone()), } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/guest/src/service/exec/output.rs` around lines 253 - 294, Add a private OutputState::summary helper containing the shared OutputTerminalSummary construction, then update terminal_summary and sealed_terminal_summary to retain their existing precondition checks and return state.summary().src/guest/src/service/exec/mod.rs (1)
306-321: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winCouple the channel capacity to the event count.
The capacity 2 at Line 309 is correct only because
output::terminal_eventsemits at most one event per stream. That bound lives insrc/guest/src/service/exec/output.rsLines 450-459. If a third terminal event is ever added,try_sendreturnsFulland theexpectat Line 316 panics on a request path.Build the events first and size the channel from their count.
♻️ Proposed refactor
fn terminal_output_receiver( summary: output::OutputTerminalSummary, ) -> mpsc::Receiver<Result<ExecOutput, Status>> { - let (tx, rx) = mpsc::channel(2); if let Some(failure) = summary.reader_failure { + let (tx, rx) = mpsc::channel(1); tx.try_send(Err(Status::internal(failure))) .expect("terminal attach receiver must be live"); - } else { - for event in output::terminal_events(&summary) { - tx.try_send(Ok(event)) - .expect("terminal attach receiver must be live"); - } + return rx; + } + let events = output::terminal_events(&summary); + let (tx, rx) = mpsc::channel(events.len().max(1)); + for event in events { + tx.try_send(Ok(event)) + .expect("terminal attach receiver must be live"); } rx }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/guest/src/service/exec/mod.rs` around lines 306 - 321, Update terminal_output_receiver to build the terminal events before creating the channel, then size the channel capacity from the resulting event count; preserve the reader_failure branch and existing try_send behavior while ensuring all events can be queued without expect panicking due to a full channel.src/guest/src/service/exec/registry.rs (2)
367-371: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCollapse the duplicate test-only prune wrappers.
prune_atandprune_for_testare both#[cfg(test)], andprune_for_testonly forwards toprune_at. Keep one name.♻️ Proposed refactor
#[cfg(test)] - async fn prune_at(&self, now: Instant) { + async fn prune_for_test(&self, now: Instant) { Self::prune_inner(&self.inner, now).await; }Then remove the separate
prune_for_testwrapper at Lines 409-412.Also applies to: 409-412
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/guest/src/service/exec/registry.rs` around lines 367 - 371, Collapse the duplicate test-only wrappers by keeping a single #[cfg(test)] method, preferably prune_at, and removing prune_for_test and its forwarding call. Update all test call sites to use the retained method name.
323-357: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAvoid cloning every retained entry on the common no-eviction path.
Lines 323-342 clone the id, the
ExecutionState, and the wholeTerminalSnapshotfor every retained entry on each call, while the registry mutex is held.TerminalSnapshotcarries two diagnosticStringvalues, andMAX_RETAINED_ENTRIESis 64, so up to 64 snapshot clones run per completed execution. In the common case the limits are not exceeded and every clone is discarded.Collect only
(id, retained_bytes, last_access)for the sort, then remove and clone the full entry for the ids that are actually evicted.♻️ Proposed refactor sketch
- let mut retained: Vec<_> = inner - .entries - .iter() - .filter_map(|(id, entry)| match entry { - ExecutionEntry::Retained { - state, - snapshot, - retained_bytes, - last_access, - .. - } => Some(( - id.clone(), - state.clone(), - snapshot.clone(), - *retained_bytes, - *last_access, - )), - _ => None, - }) - .collect(); - retained.sort_by_key(|(_, _, _, _, last_access)| *last_access); - let mut total_bytes: usize = - retained.iter().map(|(_, _, _, bytes, _)| *bytes).sum(); + let mut retained: Vec<_> = inner + .entries + .iter() + .filter_map(|(id, entry)| match entry { + ExecutionEntry::Retained { + retained_bytes, + last_access, + .. + } => Some((id.clone(), *retained_bytes, *last_access)), + _ => None, + }) + .collect(); + retained.sort_by_key(|(_, _, last_access)| *last_access); + let mut total_bytes: usize = retained.iter().map(|(_, bytes, _)| *bytes).sum(); let mut evicted = Vec::new(); while retained.len() > MAX_RETAINED_ENTRIES || total_bytes > MAX_RETAINED_BYTES { - let (id, state, snapshot, bytes, _) = retained.remove(0); + let (id, bytes, _) = retained.remove(0); total_bytes -= bytes; + let Some(ExecutionEntry::Retained { state, snapshot, .. }) = + inner.entries.remove(&id) + else { + continue; + }; let access = next_access(&mut inner); inner.entries.insert( id, tombstone_entry(snapshot, Instant::now() + TOMBSTONE_TTL, access), ); evicted.push(state); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/guest/src/service/exec/registry.rs` around lines 323 - 357, Update the retained-entry eviction logic to collect only each entry’s id, retained_bytes, and last_access for sorting, avoiding clones of ExecutionState and TerminalSnapshot on the no-eviction path. After determining eviction candidates in the existing while loop, remove each selected entry from inner.entries and clone the state and snapshot only for entries actually evicted, then create the tombstone and preserve the existing byte and tombstone-limit behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/guest/src/reaper.rs`:
- Around line 1133-1149: Serialize all three real-child tests with the shared
test guard to prevent concurrent interaction with REAP_LOCK: in
src/guest/src/reaper.rs lines 1133-1149, acquire the guard before reap_fence()
in signal_leader_terminates_the_live_claimed_process; in
src/guest/src/service/exec/state.rs lines 689-719, wrap child.wait() in
reap_fence() and execute it via spawn_blocking; in
src/guest/src/service/exec/timeout.rs lines 90-123, add the same guard while
retaining its existing reap_fence() and spawn_blocking usage.
In `@src/guest/src/service/exec/mod.rs`:
- Around line 441-463: In the execution registry branch following the
reservation publish path, replace the nested `else { if ... }` around
`server.registry.register` with an `else if` or equivalent single conditional
expression. Preserve the existing shutdown error response and
`state.abort_unpublished().await` behavior when registration fails.
In `@src/guest/src/service/exec/registry.rs`:
- Around line 1129-1162: Update
shutdown_releases_live_and_retained_state_resources to avoid assigning
fabricated live PIDs that shutdown_all may signal on the host. Use
guaranteed-nonexistent PIDs for the settled_state handles, or invoke
release_remaining_states directly while preserving the assertions that both
states release their resources and registry entries.
- Around line 520-557: Update shutdown_all to route both SIGTERM and SIGKILL
through each execution state’s ExitClaimOwner and Reaper::signal_leader_if_live,
matching the behavior used by TimeoutTarget::signal_if_live. Remove raw
state.get_pid() and kill(pid, None) probing for signal decisions, while
preserving the existing wait loop and graceful-exit handling.
- Around line 448-470: Update observe_terminal so wait_terminal_output_summary
is bounded by the execution’s existing deadline or timeout mechanism after the
leader exits, while preserving the normal completed-summary path. Ensure a stuck
stdout/stderr reader cannot leave the ExecutionEntry in Live indefinitely and
that the flow still reaches retain with the available output, producing Retained
or Tombstone as appropriate.
In `@src/guest/src/service/exec/state.rs`:
- Around line 278-293: Update wait_terminal_output_summary to await
output.seal() unconditionally, bind its boolean result, and apply debug_assert!
to that bound result so release builds still seal the OutputManager.
---
Outside diff comments:
In `@src/guest/src/service/exec/state.rs`:
- Around line 416-462: Update attach_output to safely displace any existing
inner.output_task instead of asserting it is absent. While holding the lock,
replace the previous handle with the newly spawned task, then release the lock
and await the displaced handle before returning or completing the attachment,
preserving the released-state cleanup and HandleUnavailable behavior.
---
Nitpick comments:
In `@src/guest/src/service/exec/mod.rs`:
- Around line 306-321: Update terminal_output_receiver to build the terminal
events before creating the channel, then size the channel capacity from the
resulting event count; preserve the reader_failure branch and existing try_send
behavior while ensuring all events can be queued without expect panicking due to
a full channel.
In `@src/guest/src/service/exec/output.rs`:
- Around line 253-294: Add a private OutputState::summary helper containing the
shared OutputTerminalSummary construction, then update terminal_summary and
sealed_terminal_summary to retain their existing precondition checks and return
state.summary().
In `@src/guest/src/service/exec/registry.rs`:
- Around line 367-371: Collapse the duplicate test-only wrappers by keeping a
single #[cfg(test)] method, preferably prune_at, and removing prune_for_test and
its forwarding call. Update all test call sites to use the retained method name.
- Around line 323-357: Update the retained-entry eviction logic to collect only
each entry’s id, retained_bytes, and last_access for sorting, avoiding clones of
ExecutionState and TerminalSnapshot on the no-eviction path. After determining
eviction candidates in the existing while loop, remove each selected entry from
inner.entries and clone the state and snapshot only for entries actually
evicted, then create the tombstone and preserve the existing byte and
tombstone-limit behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a51d5bb4-bde3-477d-a31c-d445aa547607
📒 Files selected for processing (7)
src/guest/src/reaper.rssrc/guest/src/service/container.rssrc/guest/src/service/exec/mod.rssrc/guest/src/service/exec/output.rssrc/guest/src/service/exec/registry.rssrc/guest/src/service/exec/state.rssrc/guest/src/service/exec/timeout.rs
Review dispositionAll six inline findings plus the outside-diff one are handled in 9db68d1 / 2ace576. Per-thread replies are inline; the two findings with no inline thread are covered here. Displaced let displaced = inner.output_task.replace(task);
// ... lock released ...
if let Some(displaced) = displaced { let _ = displaced.await; }The join is bounded, which is why it is safe inside the handler:
Deliberately out of scope: the unbounded terminal-summary wait ( Stated plainly, three parts carry no test: the Verification: 9db68d1 went green on all three Clippy platforms and all three Rust Tests platforms; guest unit tests are 305/305 on Linux, with the two new shutdown/kill tests demonstrated red-then-green against a full revert of the production changes. 2ace576 is a one-line |
2ace576 to
fc46370
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/guest/src/service/container.rs`:
- Around line 486-499: Update the failed registration branch in the container
initialization flow to pass state.clone() into registry.register, then await
state.abort_unpublished() before returning the error response. Preserve
reaper.release_slot(&exit) and the existing response behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7295d341-b2e1-47aa-bc8b-716d563366f9
📒 Files selected for processing (6)
src/guest/src/service/container.rssrc/guest/src/service/exec/mod.rssrc/guest/src/service/exec/output.rssrc/guest/src/service/exec/registry.rssrc/guest/src/service/exec/state.rssrc/guest/src/service/exec/timeout.rs
🚧 Files skipped from review as they are similar to previous changes (4)
- src/guest/src/service/exec/output.rs
- src/guest/src/service/exec/state.rs
- src/guest/src/service/exec/mod.rs
- src/guest/src/service/exec/registry.rs
fc46370 to
31bfe5d
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@coderabbitai review |
✅ Action performedReview finished.
|
| ssh_workload, | ||
| reservation.as_ref(), | ||
| ) | ||
| .await |
There was a problem hiding this comment.
Cancellation here can leave the reservation and spawned process unowned.
There was a problem hiding this comment.
Half fixed in be0b994, half deliberately left.
The reservation half is fixed. You are right that nothing owned it: the release only ran on the Err arm below, so a dropped request future skipped it entirely, and I confirmed prune_inner never touches Reserved — it only tombstones Retained and removes expired Tombstone. release_ephemeral puts a Reserved entry straight back, and release_remaining_states maps it to None. So the entry survived to guest shutdown and exists() stayed true for an id lookup() reported as absent.
ExecutionReservation is now an RAII guard: it carries a registry handle and releases on drop, publish and release_reservation disarm it so the normal paths do not double-release, and the new release_reserved_id only removes the entry while that exact ticket still owns it, so a late release cannot evict a replacement. The Drop spawns its release because the registry sits behind an async mutex — same shape as ChannelBridge::drop, with Handle::try_current() added so a drop during runtime teardown cannot panic. The explicit release on the Err arm stays: it is deterministic and visible at the call site, and Drop is only the cancellation net.
The spawned-process half is not fixed, and not by oversight. That window pre-exists on main: spawn_with_executor returns the child before registry.register/publish is reached, so a cancellation in between orphans it there too — this PR narrows the non-cancelled path via abort_unpublished but does not close the cancelled one. Closing it needs something that owns the child and kills it on drop; ExecutionState is cloned freely so a Drop on it is wrong, and detaching the spawn onto its own task would need GuestServer: Clone, which it is not. That is an ownership change rather than a fix, so it is scoped out of this PR by agreement and belongs in its own.
Test: dropping_an_unpublished_reservation_frees_its_execution_id, written before the fix and observed red as a dropped reservation must free its execution id: Elapsed(()).
| } | ||
| }; | ||
| for state in evicted { | ||
| state.release_resources().await; |
There was a problem hiding this comment.
Eviction here can end an active Attach before its final output frames.
There was a problem hiding this comment.
Fixed in be0b994, and your finding turned out to be a class rather than an instance — both retirement paths had it, so both are guarded now.
This site (cap eviction). The loop no longer takes the oldest candidate outright; it takes the oldest one with no reader, and warns and stops when every candidate is being read, leaving the caps breached until those readers finish. Testing the lease from inside the loop meant hoisting the output consumer flag out from under ExecutionState's mutex — the loop holds the registry lock, and awaiting a state's own lock there would invert lock order — so ExecutionState now shares the Arc<AtomicBool> with its OutputManager and exposes a synchronous has_active_reader.
The sibling (grace expiry). prune_inner selected on expires_at <= now alone and truncated an active replay exactly as this site used to. It was reachable rather than theoretical: lookup refreshes last_access for LRU but never extends expires_at, so a reader attaching near the end of the retain grace was cut when the grace elapsed. It now requires !state.has_active_reader() too, and the next tick retires the entry once the reader detaches. I had first set this one aside as scope and that was wrong — same shape, same file, so it belonged in the same pass.
Both are covered, each red before its fix: eviction_spares_a_retained_session_with_an_active_reader (a retained session with an active reader must not be evicted) and grace_expiry_spares_a_retained_session_with_an_active_reader (grace expiry must not tombstone a session with an active reader).
Two things I checked and left alone. release_ephemeral and release_remaining_states still abort a forwarder, but in both the caller has explicitly asked for teardown, so I read those as intended rather than as this defect. And I briefly suspected a race where execution_lookup releases the registry lock before attach_retained claims the lease — it is not one: attach_output re-checks released after spawning the forwarder and aborts, and attach_execution catches HandleUnavailable on the Retained branch and falls back to the snapshot summary, so a reader that loses that race gets the summary rather than a truncated stream.
What is still missing, stated plainly. When truncation does happen on those two remaining paths, the client gets no signal — the sender drops and the stream simply ends, which looks like a normal end of output. Skipping readers removes causes of truncation; it adds no diagnostic. The plumbing exists without touching the proto, since attach hands back mpsc::Receiver<Result<ExecOutput, Status>>, but an aborted forwarder cannot send, so supplying one means replacing the abort inside release_resources with a cooperative finish and that changes the contract for all four of its callers. Noted in the PR description as a deliberate follow-up rather than folded in here.
| inner.lifecycle_manager.take() | ||
| }; | ||
| if let Some(lifecycle_manager) = lifecycle_manager { | ||
| lifecycle_manager.abort(); |
There was a problem hiding this comment.
Aborting here can interrupt cleanup after the entry becomes a tombstone.
There was a problem hiding this comment.
Fixed in be0b994.
The abort is gone. RegistryInner now carries a watch::Sender<bool>; the manager loop selects between its one-second sleep and that signal. Both of the loop's exits — the signal arm and the Weak upgrade failing once the registry is dropped — sit ahead of prune_inner, and the receiver is polled only inside the select!, so a signal arriving mid-prune waits in the channel and is seen on the next turn. Being precise about the guarantee: that removes deliberate cancellation, not all of it. A runtime teardown still cancels at the next await, but by then the process is exiting and the resources go with it. stop_lifecycle_manager sends the signal and awaits the handle instead of aborting it, taking the handle and releasing the registry lock before that await so the manager can still take the lock it needs to finish. The join result now drives a warn! when the manager does not end cleanly rather than being discarded. The watch channel follows SshManager's existing shutdown_tx: watch::Sender<bool>; using it rather than polling a flag also means shutdown no longer waits out a full tick.
Worth being precise about what the test binds. stopping_the_lifecycle_manager_lets_it_exit_on_its_own asserts the manager finished its own loop rather than being cancelled — it was red as the manager must finish its own loop rather than be aborted. It does not reproduce the interleaving you described, because I cannot force the abort to land on that specific await deterministically in a unit test. So the test binds the mechanism, not the race.
On reachability, for whoever reads this later: stop_lifecycle_manager has one caller, shutdown_all, which itself runs only from the host-driven Shutdown RPC and from the guest's own power-off after init exits. The stranded resources would therefore have been reclaimed by process exit moments later. What made this worth fixing is not the leak but the state it allowed — an entry already tombstoned while its resources were still live is an invariant break, and cooperative stop makes it unreachable instead of merely unlikely.
b4abc11 to
b34788a
Compare
An execution's registry entry disappeared as soon as its output stream ended, so a Wait or Attach arriving afterwards found nothing and the caller could not tell "never existed" from "already finished". The registry now holds three states per execution. Live behaves as before. Once an execution's exit and output summary are both in, it becomes Retained and its buffered output stays readable. After the retain grace it degrades to a Tombstone holding only the classified exit and a truncated diagnosis. Retention is bounded on grace, TTL, entry count and retained bytes, with LRU eviction, so a long-lived box cannot grow the registry without limit. Reservations close the window between spawn and register: an execution that cannot be published is SIGKILLed and torn down, and the reservation releases on drop so a request cancelled mid-spawn does not strand its id in a Reserved entry nothing prunes. Neither retirement path drops a retained session while a reader is still streaming it, because aborting that forwarder truncates the replay indistinguishably from a normal end of output: the caps skip it and the grace defers it until the reader detaches. Shutdown signals the retention pruner rather than aborting it, so an entry cannot be left tombstoned while still holding resources. The timeout watcher now returns its handle so a session that already exited cancels it rather than leaving a task parked on a deadline.
b34788a to
be0b994
Compare
2 participants
Summary
A finished execution loses two things it should still be able to answer for. Its output is gone the moment the stream ends — a late
Attachgets an empty stream over pipes already at EOF, not the bytes the execution produced. And its exit is only classifiable once, because the container-death diagnosis drains init's pipes, so a repeatWaitgets a different answer than the first. Meanwhile the entry itself is never removed at all: only the SSH bridges callrelease_ephemeral, so every SDK exec the guest ever ran stays in the map.This gives an execution three lifecycle states. It stays readable for a bounded window after it exits, then decays to a small record, then goes away.
Call graph
Before
After
Changes
ExecutionRegistryentries becomeLive→Retained→Tombstone. Retained keeps the buffered output readable; a tombstone keeps only the classified exit and a truncated diagnosis. Retention is bounded on grace, TTL, entry count and retained bytes, with LRU eviction, so the registry can no longer grow without limit.ExecutionStateclassifies its exit once into aOnceCell, which is what makes a repeatWaitreturn the same container-death diagnosis instead of an emptied one.prune_innertombstones entries under the lock and releases their resources after dropping it, so an abort in between left an entry tombstoned while its resources were still live.How to verify
318 tests pass. The retention behavior is covered by
terminal_observer_retains_a_completed_live_execution,retained_entry_exposes_its_terminal_snapshot_before_tombstoning, anda_tombstone_keeps_its_terminal_snapshot_repeatable; the bounds by the grace/TTL/LRU/byte-cap eviction tests inregistry.rs; the repeatability fix byrepeated_wait_caches_the_init_exit_diagnosis, which countsdiagnose_exitcalls and fails onmainbecause the secondWaitre-drains init's pipes; and the reader-versus-retirement rule byeviction_spares_a_retained_session_with_an_active_readerandgrace_expiry_spares_a_retained_session_with_an_active_reader, one per retirement path.Risks / rollout
Retained output is memory-bounded and expires. After eviction, terminal RPCs answer from the tombstone and then return NotFound — a caller that waits longer than the tombstone TTL sees a not-found where it previously saw a stale live entry.
Rebased onto #1185, which was split out of this branch and merged first. That PR's claim/token design was replaced during review by a
ProcessInstanceidentity check, so this branch was re-ported onto the merged design rather than rebased hunk-by-hunk; it no longer touchesreaper.rsorexec_handle.rsat all.Two gaps are left open deliberately.
release_ephemeralreturning the reaper slot has no unit test. That path goes through the globalREAPER, whichmainalso leaves untested, and making it testable means re-adding API surface this branch does not otherwise need.Neither retirement path drops a retained session while a reader is streaming it — the caps skip it and the grace defers it — but the two paths that still abort a forwarder,
release_ephemeraland shutdown, give the client no signal: the stream simply ends, which is indistinguishable from a normal end of output. Both are cases where the caller knowingly tore the session down, so this is a missing diagnostic rather than a surprise. Supplying one needsrelease_resourcesto finish cooperatively so aStatuscan go out over the existingResult<ExecOutput, Status>channel, which changes the contract for all four of its callers; that belongs in its own change rather than here.