bug(tui): session picker refuses saved sessions whose runtime store exists but is not the running host's own

[7jrxt42BxFZo4iAnN4CX]

Description

In the TUI, opening a saved session fails with:

Failed to restore session: This session belongs to another Runtime host. Resume it in a new
Codewhale process to reopen its saved store.

The session's runtime store directory exists, its ownership validates, and no other Codewhale
process is running — so nothing else holds the store. The refusal sits in the session-switch guard,
which treats "the bound directory exists" as "another live host owns it".

Steps to reproduce

1. Launch Codewhale in a workspace; sessions created there are bound to the host's store,
   e.g. ~/.codewhale/sessions/<host-session-id>/runtime.
2. Lose the host session record while its store directory stays on disk. A conversation created in
   that host keeps the durable binding to it.
3. Start a new Codewhale process (new host, its own store) and open the saved session (picker or
   --resume from the TUI).
4. The refusal above.

Expected behavior

Opening the session should succeed, because the bound store both validates and is free:

• RuntimeStoreBinding::validate_existing_store() (crates/tui/src/runtime_threads.rs:2918) accepts
  it — the owner record parses and the recomputed scope
  sha256(owner_id ‖ 0x00 ‖ <store>/events.lock) equals the scope saved in the session;
• the store's process-owner lock (runtime-process.owner.lock) is free — no live host holds it;
• RuntimeThreadManager::open_for_session (crates/tui/src/runtime_threads.rs:4636-4668) already
  takes this exact path when a process opens a session with a binding: existing store + passing
  validation ⇒ adopt it.

So either adopt the store the same way in the switch path, or fall back to transcript recovery,
instead of refusing because the running process happens to be bound to a different store. At
minimum, the message should name the real condition and the remedy that works.

Actual behavior

Fail-closed, in apply_loaded_session_with_goal (crates/tui/src/tui/ui/apply.rs:3561-3573):

if let Some(binding) = session.metadata.runtime_store.as_ref()
    && let Some(tasks) = app.runtime_services.task_manager.as_ref()
    && tasks.session_store_binding().as_ref() != Some(binding)
{
    if binding
        .is_missing_session_store()            // apply.rs:3566-3567
        .map_err(|error| error.to_string())?
    {
        recovered_binding = tasks.session_store_binding();
    }
    if recovered_binding.is_none() {
        return Err("This session belongs to another Runtime host. …".into());   // apply.rs:3573
    }
}

is_missing_session_store() (crates/tui/src/runtime_threads.rs:2891) returns true only when
symlink_metadata yields NotFound; an existing directory returns false no matter who — if
anyone — holds it. The recovery branch added for missing stores (commit 1de7d138, issue #6102)
therefore never covers "directory exists, its host is gone", and the guard's message asserts a live
rival host that does not exist.

Evidence from this machine

~/.codewhale/sessions held 57 saved sessions:

• 25 carry a binding to a store directory that is not their own;
• 22 of those directories still exist (this failure mode); 3 are missing (already recovered by bug(session): resuming an interrupted session fails with ENOENT while resolving its runtime store dir #6102).

Concrete case:

• session c035ff00-ea47-4109-bee4-154b98f67514, 921 messages, workspace
  /home/f2/Documents/Shared/ProL/Rust/pipeline-orchestrator;
• binding data_dir: …/sessions/e27ea9fe-75ab-47d1-b443-794ef523f638/runtime,
  execution_scope: 27271aff…;
• that directory exists and holds owner.json (owner_4acdc4dd…), state.json
  ({"schema_version":2,"next_seq":1}) and two lock files — 4 files, zero events;
• no session record e27ea9fe-….json exists: the host that created the store is gone, the
  conversation it hosted is not;
• recomputing the scope from owner_id and …/runtime/events.lock reproduces 27271aff… exactly —
  this is the session's own scope, not a foreign one;
• ps showed no running codewhale process.

Second case: session 6b57f767-… (0 messages) is bound to …/6b57f767-…/runtime-recovered-session
— a store minted by the #6102 recovery path itself (runtime_threads.rs:4657-4658). If the next
launch refuses an existing runtime-recovered-session directory the same way, recovery hands the
session a binding the following launch rejects. Worth confirming: it looks self-inflicted, but I did
not run a second launch against it.

Caveat on this report

The failure is the TUI path (Failed to restore session: …, tui/ui/apply.rs). I did not run a live
codewhale resume <id>, so I cannot say from observation that a fresh process adopting the binding
succeeds — but read as code, open_for_session validates this same store and adopts it, which is why
the guard's advice ("resume in a new Codewhale process") looks plausible while the operator, opening
the session from the picker, only sees the refusal. I also did not mutate live session state to test
repairs.

Impact

Blocks opening older sessions from the picker with an error that points at a rival process that is
not there, and offers no in-product way to reach the session. 22 of 57 sessions on this machine are
in the triggering state.

Related: #6102 (missing-store recovery — the case this one falls outside), #6014 (session-picker UX
around empty sessions).

Environment

• OS: Arch Linux
• codewhale version: 0.9.13 (dev)
• Install method: source build
• Model/provider: DeepSeek
• Terminal app: tmux (TERM=tmux-256color)
• Shell: fish

Logs, screenshots, or recordings

No crash log — the failure is a UI message on session restore. The on-disk state above is the
reproduction artifact.

[github-actions]

Thanks @7jrxt42BxFZo4iAnN4CX for the report.

This issue is staying open for maintainer triage. CodeWhale gets better because people bring us real edge cases from real machines, providers, regions, and workflows.

If you can add a reproduction, logs, version output, screenshots, or the provider/model involved, that makes it much easier for us to verify and harvest the fix. Maintainers may comment /lgtmi to mark recurring issue reporters as approved so this intake note is skipped next time.

[Hmbown]

Confirmed, with one correction that changes what the fix should be.

Every line you cited says what you say it says — is_missing_session_store at the refusal, validate_existing_store accepting the binding, open_for_session already taking the adopt path for existing+valid stores, and the owner lock being free. We opened all of them.

The correction: this is a policy with a missing case, not a wrong predicate

The switch path's recovery escape hatch is deliberately scoped to "there is nothing on disk to abandon". That is documented at runtime_threads.rs:2890-2891 and apply.rs:3595-3597, and locked in by an assertion at tests/runtime_store_binding.rs:284. The reason is that the switch path can rebind the conversation but cannot carry the old store's durable work across — queued tasks, pending approvals, agent mail, and automations whose execution scope matches the binding. The guard exists to stop that work being silently dropped.

So the predicate is not "wrong" — it is protecting something real. What it lacks is a case for the state you actually have: a store that exists but is empty and ownerless. In your repro that is four files — owner.json, state.json with {"next_seq":1}, and two lock files — and zero events. There is nothing to abandon, so the guarantee the guard protects is not at stake, and the refusal is wrong for that state.

This also means the message is misdescribing the situation: it says the session belongs to another Runtime host, when what it actually means is "this path cannot carry a store's durable work across". That is worth fixing regardless of the predicate.

Why lock-freeness is not the discriminator to use

Adopting on "the owner lock is free" would be a genuine policy reversal, not a bug fix. Lock-freeness does not distinguish "abandon nothing" from "abandon a session's queued tasks, approvals, mail and scope-pinned automations" — a crashed host leaves both states unlocked. Taking that route means rewriting the assertion at tests/runtime_store_binding.rs:284 and stating the lost guarantee explicitly, rather than discovering it in CI.

The predicate we think is right is no durable work: no events beyond next_seq == 1, no tasks, no approvals, no mail, and no automation whose execution_scope matches the binding. That covers your case and, we think, the broader set you found in #6102 without weakening the guard.

(For the record, one sub-claim we tested and rejected along the way: the exclusive flock does not make adoption impossible. It is per store directory, and runtime_chat_relay.rs:293-303 already runs a second manager on a different store in-process. So "adopt" stays on the table; it just costs more rewiring than the emptiness predicate.)

Possible duplicate

#6185 reports an empty transcript on resume after a force-quit, with an intact journal. The six early returns in apply_loaded_session_with_goal all produce exactly that shape, and this guard is one of them. If the reporter there confirms the status line reads Failed to restore session: This session belongs to another Runtime host…, the two issues are the same defect and this fix closes both.

Thanks for the precise write-up — the validate_existing_store / open_for_session contrast is what made the policy framing visible.

[Hmbown]

Fixed on branch v0914-chunk2 in 68431ba097, taking the emptiness route described above rather than lock-freeness.

RuntimeStoreBinding::has_no_durable_work adopts an existing store only when it is confined to <state>/sessions/<id>/runtime[-recovered-*] with no symlink on the path, every work directory (threads, turns, items, events, goals, agent-mail, turn-operations) is empty, and state.json's sequence is still at its initial value — events can be appended and later pruned, and the sequence remembers. Anything unreadable, unconfined or non-empty fails closed.

The seven pre-existing tests that pin the "cannot carry durable work across" guarantee pass untouched, which is the evidence that this widens the policy by exactly the empty case and nothing more.

The refusal message changed too. It claimed another host owned the session; what it actually means is that the saved store holds work this process cannot carry across, so it now says that.

Checks: ./scripts/dev-test.sh tui runtime_store_binding — 8 tests run, 8 passed, including the new empty_existing_runtime_store_reports_no_durable_work, which writes and removes a marker in each work directory in turn so every directory is proven load-bearing on its own. Full suite: ./scripts/dev-test.sh tui — 12708 tests run, 12708 passed, 19 skipped.

This is on the branch in PR #6175, which is open, not merged — so it is not in a release yet. Leaving this issue open until that lands.

[Hmbown]

Correcting myself: the fix I posted above is reverted in bc77a0ff5d. It was wrong, and the way it was wrong matters for whoever picks this up.

What broke

runtime_store_binding_survives_launch_snapshot_and_resume failed on CI (macOS, then ubuntu) at its line-284 assertion, while passing here in isolation even under --all-features. Passing alone and failing under the full suite is the signature of a race, and the race was in my predicate.

has_no_durable_work samples emptiness at a single instant and has no liveness check. That makes two very different states look identical:

• a store that is empty because it was abandoned (safe to adopt), and
• a store that is empty right now because the host holding it has not flushed yet (absolutely not safe to adopt).

That test has a live foreign TaskManager holding the store. Under load the adopt path wins the race, the switch succeeds where it must refuse, and the assertion sees a different error. In production that is someone's in-flight work being dropped.

A second, independent gap

My predicate only walks the runtime store's own directories (threads, turns, items, events, goals, agent-mail, turn-operations) plus state.json's sequence. Scope-pinned automations are recorded outside that tree, so an automation whose execution_scope matches the binding is durable work the check cannot see. The analysis I posted earlier in this thread said the predicate needed "no automation whose execution_scope matches the binding" — and then I did not implement it. That is on me.

Also correcting a claim

I wrote that "the seven pre-existing tests that pin the guarantee pass untouched, which is the evidence that this widens the policy by exactly the empty case and nothing more." That was true of the run I did and false in general. A green targeted run is not evidence about a test whose failure mode is ordering-dependent, and I presented it as though it were.

The corrected design

Adopt only when the store is both:

1. provably unheld — a liveness check on the store's process-owner lock, so a live host is never raced; and
2. provably empty — the seven work directories, state.json's sequence and no automation whose execution_scope matches the binding.

Emptiness alone is insufficient, which is the thing I got wrong. Lock-freeness alone is also insufficient, which is the thing the earlier analysis got right. It needs both, and the liveness check's shape (what it does about a crashed host that still holds a stale lock) should be settled before any of it is written.

Behaviour returns to refusing — the bug this issue reports. That is a worse experience than the fix and a better one than silently dropping scheduled work, so it stays that way until the predicate is right.

Issue stays open. Sorry for the noise on the thread.

[SparkofSpike]

Glad to find this issue — it matches a problem I ran into exactly: /resume refuses to bring the conversation back. Below is what I found while digging through 0.9.13 and current main. I hope it offers some help to you, @Hmbown.

One note first, so this is not read as re-reporting something closed: I see 68431ba09 was reverted by bc77a0ff5 nineteen minutes later, and PR #6175 landed with both of them on the same branch — so the merge referenced from this thread is a net zero, and the guard on main still returns the old refusal. The notes below aim at the corrected design in your third comment.

1. There is no stale-lock case to settle.

RuntimeProcessOwnerLock holds a kernel-owned exclusive lock (flock(LOCK_EX | LOCK_NB) on unix, LockFile on windows — runtime_threads.rs:4222-4246). The lock's lifetime is bound to the open descriptor, so a host that dies, including SIGKILL or a force-quit, has its lock released by the kernel. A lock file that outlives its host is not a held lock.

That is already the semantics this tree relies on. recover_dead_executions_locked (task_manager.rs:2963) probes an execution lease with try_acquire_file(&path, false) and reads a successful acquire as "prior process is not attached". The doc comment on try_acquire_file (runtime_threads.rs:4157-4160) draws the line on the other side — "a missing existing lease is uncertainty, never proof that an owner died" — so missing is the undecidable case, and present and acquirable is the decidable one.

So the liveness check can be: one non-blocking exclusive acquire on <store>/runtime-process.owner.lock. Acquired ⇒ provably unheld. Contended ⇒ refuse. Absent ⇒ undecidable, fail closed. No crashed-host branch is needed.

2. Hold the probe's guard across the emptiness scan.

try_acquire_file hands the lock back as Option<RuntimeProcessOwnerLock> — an RAII value. Dropping it before the emptiness scan starts reopens the window that produced the race: a freshly launched host can begin flushing into the store between the probe and the walk, and the walk then reads a store that is empty only because it is not yet written. Holding the guard until the scan returns costs nothing and closes that window.

3. Where "automation scope" actually lives.

default_automations_dir() (automation_manager.rs:2800-2819) resolves to $CODEWHALE_HOME/automations or ~/.codewhale/automations, and AutomationManager::open (:1072) creates three trees under it: automations/, runs/, triggers/. All three sit outside the runtime store, which is why a seven-directory walk inside the store cannot see any of them.

Definitions are <root>/automations/<id>.json, deserialized as AutomationRecord, which carries its own execution_scope (:106) and embeds an AutomationDispatch carrying a second one (:249). An emptiness predicate that includes automations has to read those records and match either scope against the binding, and account for triggers/ as well.

4. One case the tests should cover, from the second example in this report.

open_for_session (runtime_threads.rs:4658) rebinds a missing store to a runtime-recovered-session directory, and the confinement whitelist accepts that prefix (:2900). But is_missing_session_store (:2891) only answers NotFound, so an existing recovered directory falls through to the refusal. A store this code path minted should not be refused by the next launch.

5. Scope of the fix.

Both other reports on this guard are the same early return (apply.rs:3617-3631): #6185(a)'s empty transcript after a force-quit, and #6225's "belongs to another Runtime host" from the picker. Whichever shape lands, it closes all three.

[Hmbown]

Reframing this, because I think the whole predicate approach — mine and the original proposal's — is the wrong shape, and that is why my attempt raced.

The adopt path already enforces liveness, atomically

open_inner acquires the store's process-owner lock at runtime_threads.rs:4190 and bails at :4210 with:

This runtime is already active in another process. Close the other Codewhale session and try again, or set CODEWHALE_RUNTIME_DIR to a different directory.

So opening a store is the atomic test for whether another host holds it. There is no window: either you take the lock or you do not.

Every predicate discussed on this issue — "is the owner lock free", "is the store empty", mine included — is an attempt to predict the outcome of that atomic operation from a snapshot taken beforehand. That is check-then-act, and it is precisely what CI caught in 68431ba097: a store that is empty only because its holder has not flushed yet is indistinguishable from an abandoned one, and the adopt path wins the race.

This is the same shape as two other bugs in this milestone. In #6184, timeout_at drop-cancels a parked future without needing the future to cooperate, so reasoning about what the child "would" do was wrong. In #6169, tcgetpgrp() cannot prevent SIGTTIN because the read(2) itself is the atomic foreground test. The operation is the test; predicting it is a race.

What that implies for the fix

Stop predicting. Let the switch path attempt the adoption and handle the outcome:

1. Attempt to open the saved store through the path that already adopts an existing, validating store (open_for_session, runtime_threads.rs:4636-4668).
2. If the lock is held, that is a genuine "another host is using this" — surface it, and say so honestly rather than claiming the session belongs elsewhere.
3. If it opens, the store is ours and the question of whether it was "really" abandoned is answered by the lock rather than by a directory listing.

Emptiness then stops being a liveness proxy and becomes what it actually is: a question about whether adopting would lose anything — which only matters for the store being left behind, not the one being taken.

The message is a separate, smaller bug

Failed to restore session: This session belongs to another Runtime host. Resume it in a new Codewhale process to reopen its saved store.

#6225 reports this from a plain quit-and-resume on a clean install, and the advice is impossible to follow — they were in a new process. Whatever happens to the predicate, this string should describe the actual condition. That part is safe to fix independently and worth doing first, since it is currently sending users in circles.

Status

68431ba097 is reverted; main still refuses. #6225 is the same bug reached by the ordinary path — start, /quit, start, /resume — which is a stronger reproduction than anything on this thread, and raises the priority accordingly.

I am not writing the predicate version again. The attempt-and-handle shape above needs someone to confirm that routing the switch path through open_for_session does not disturb the conversation rebinding it also performs, and that is worth reading carefully rather than inferring — which is how the last two attempts here went wrong.

[Hmbown]

claiming: taking the message half of this — the part you called out as "safe to fix independently and worth doing first". Scope: apply_loaded_session_with_goal in crates/tui/src/tui/ui/apply.rs, and the assertion in crates/tui/src/tui/ui/tests/runtime_store_binding.rs that pins the old string.

I am not attempting the adopt/predicate half, and I will say why in the follow-up comment rather than leave it implied — the switch path cannot perform the atomic open you describe without replacing the live TaskManager, which is not a batch-A slice. Issue stays open for that.

[Hmbown]

Landed the message half as 0cc0bdef8 on fix/lost-work-batch-a-20260915 (local branch; not pushed — pushing is founder authority).

What changed. apply_loaded_session_with_goal (crates/tui/src/tui/ui/apply.rs) no longer asserts a rival host. It now names the bound store, states the real condition — a running process cannot switch stores under its own tasks, mail and automations — and gives the remedy that works, codewhale --resume <id>. When validate_existing_store() says that adoption would itself fail, it reports that reason instead, so the advice is never a circle.

Why --resume is the honest remedy, checked rather than assumed: at launch, apply_loaded_session_with_goal runs at event_loop.rs:786, before TaskManager::start at :849. app.runtime_services.task_manager is still None there, so this guard is skipped entirely and the binding is adopted by open_for_session. The failure is in-session switching only (picker / SessionSelected), never --resume.

Tests. runtime_store_binding_survives_launch_snapshot_and_resume already reached this refusal with a free store — its tasks host is shut down before the assertion — so it was the right place to pin the new contract. Its old assertion (error.contains("Resume it in a new Codewhale process")) is replaced by four: the message must not contain belongs to another Runtime host, must name the store directory, must name the condition, and must name codewhale --resume. The first of those is the old literal verbatim, so it cannot pass against the old code.

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 12730 filtered out; finished in 0.74s

What I did not do, and why — issue stays open. I did not attempt the adopt-and-handle shape. Reading the switch path: the process holds its own store's RuntimeProcessOwnerLock for the lifetime of the TaskManager, which is created once at event_loop.rs:849 and held by _task_shutdown for the whole event loop; AutomationManager and the scheduler are bound to that same instance. Routing the switch through open_for_session therefore means tearing down and rebuilding the live TaskManager mid-session — not a message fix, and not something to infer my way through. Your read stands: the operation is the test, and the operation here is a TaskManager replacement nobody has built yet.

I also did not write the predicate version. You said you would not write it again; I am not going to land it behind your back.