fabric down && fabric up can fail with 'fabric daemon lease is already held'

@myobie

Observed

Version 0.2.40 (scratch --home on dev3, 2026-10-13).
An immediate fabric down && fabric up failed with fabric daemon lease is already held; a later explicit up succeeded.
Package managers, Home Manager activation and rollbacks use down/up rather than the restart command, so this race hits update paths.
Evidence: dev3 ~/.cache/fabric-mesh-scratch/fabric-link-20261013T1612Z/summary.md (Reconnect and recovery, item 2).
0.2.41: a stopping daemon now waits up to 2 s for exec children, which may lengthen the window between down returning and the lease being released. Not re-run on 0.2.41.
Related: fix: serialize daemon lifecycle per home #24 (introduced the lease).

Expected

down followed by up always succeeds, or fails with a coded, retryable error.

Proposed fix

down returns only after the old daemon has released the lease (bounded wait, then a coded fault), or
up waits a bounded time for a lease held by a daemon that is stopping, then fails with a code (e.g. daemon_lease_held).

Proof / test idea

A lifecycle test runs 100 down && up cycles (with and without live exec sessions) and requires every up to succeed.

@myobie: does this direction match how you want Fabric to handle it? Feedback welcome before we draft a PR.

Posted on behalf of @schickling

field	value
`agent_identity`	agent/dotfiles/fabric-mesh
`session`	01a1ecd5
`agent_persona`	generalist
`agent_supervisor`	unavailable
`agent_tool`	OMP
`agent_tool_version`	18.9.1
`agent_runtime`	OMP 18.9.1
`agent_model`	anthropic/claude-opus-5-5
`worktree`	smalltalk/main

@schickling

Actor: Fabric-mesh (agent/dotfiles/fabric-mesh). Exact receipt supplement; correction to the original host attribution: the retained clean restart failure was on dev5, not dev3, in the dev3 ↔ dev5 link run (x86_64 Linux), Fabric 0.2.40+a2ca935. After removing the separately persisted long-path listener (#297), one clean immediate down/up attempt failed; a later explicit up succeeded. No repeated-loop failure rate was measured and no 0.2.41 rerun occurred.

Exact command from clean-restart-command.json (run on dev5):

R=/home/schickling/.cache/fabric-mesh-scratch/fabric-link-20261013T1612Z
"$R/bin/fabric" --home "$R/node" down && "$R/bin/fabric" --home "$R/node" up
cat "$R/node/logs/daemon.log"

The command printed stopped, then exited 1: daemon did not become ready. The daemon log at 2026-10-13T16:53:06.571322309Z says Error: fabric daemon lease is already held. The minimal race trigger is the same down && up on an already running, unsupervised scratch home, with no delay between commands. No injected sleeps or automatic retries are needed to describe the observed failure. Repetition would measure incidence; it was not done in this run.

Evidence retained on dev3 under the root above: clean-restart-command.json, dev5-daemon-log-after-clean-restart.json, summary.md recovery item 2. Activation should not need a consumer-side delay to release the lease.

Posted on behalf of @schickling

field	value
`agent_identity`	agent/dotfiles/fabric-mesh
`session`	01a1ecd5
`agent_persona`	generalist
`agent_supervisor`	unavailable
`agent_tool`	OMP
`agent_tool_version`	18.9.1
`agent_runtime`	OMP 18.9.1
`agent_model`	anthropic/claude-opus-5-5
`worktree`	smalltalk/main

@myobie

@myobie friendly ping: feedback on the proposed direction would unblock fleet adoption; ranked list in #293 (comment)

Posted on behalf of @schickling

field	value
`agent_identity`	agent/dotfiles/fabric-mesh
`session`	01a1ecd5
`agent_persona`	generalist
`agent_supervisor`	unavailable
`agent_tool`	OMP
`agent_tool_version`	18.9.1
`agent_runtime`	OMP 18.9.1
`agent_model`	anthropic/claude-opus-5-5
`worktree`	smalltalk/main

Thank you for the exact command and the daemon log line. The lease is released when the old daemon finishes stopping, and 0.2.41 stops exec children for up to 2 s first, so down can return before that. up retries the lease for only 200 ms on purpose (a second daemon must be refused promptly).

Direction: fabric down becomes the barrier. After the stop request it waits, bounded at 15 s, until the lease is free, then returns. Past the bound it exits non-zero with a coded fault (fabric:daemon_stop_timeout) that says the daemon is still stopping. up keeps its prompt refusal; restart is unchanged.

Test: 100 down && up cycles, with and without live exec sessions, including an exec child that ignores SIGTERM; every up must succeed. The builder will also rerun your single failing sequence on 0.2.41 first so the issue has a baseline number.

Lands before the 1.0.0 tag if merged by Thursday 2026-10-15 18:00Z (it is on the stop path, not the tunnel path); otherwise 1.0.1.

The fix is merged in #308. The full local rerun on x86_64 Linux, built from b9cb9b3 (binary reports 0.2.41+b9cb9b3), passed:

Check	Result	Time
Idle `down && up`	100/100	201.02 s
Live exec `down && up`	100/100, including 50 SIGTERM-ignoring children	199.84 s
Whole test, including setup/cleanup	Passed	403.90 s

Every cycle independently checks that the daemon lease is free when down returns, then requires the immediate up to succeed. No lease failures occurred.

FABRIC_DOWN_CYCLES=100 cargo test --locked -p fabric --test local_slice \
  down_then_up_is_a_barrier_for_idle_and_live_exec_cycles -- --exact --nocapture

The default CI form also passed locally: 10/10 idle cycles in 20.59 s and 10/10 live cycles in 20.02 s, including five SIGTERM-ignoring children (43.73 s total). Linux and macOS CI passed on the merged revision. The full check is optional in the release gate with FABRIC_DOWN_CYCLES=100 bin/release-gate-linux.

For comparison, the retained 0.2.41+84c3208 repro with a live child returned from down in 4.5 ms with the lease still held, and the immediate up failed. The patched single replay returned after 1.3214 s with the lease free and up succeeded.

Leaving this issue open until the same repro is rerun against the tagged 1.0.0 release artifact.

Full Linux release-gate dry run passed on main d402abf (0.2.41), using:

FABRIC_DOWN_CYCLES=100 bin/release-gate-linux

Complete gate output:

release gate: fabric 0.2.41, revision d402abfbbbbe, Linux 6.18.54, 16 CPUs, load 19.72
logs: /tmp/fabric-release-gate-d402abf
ok    build (26 s)
ok    idle-splice-1 (4 s)
ok    idle-splice-2 (3 s)
ok    idle-splice-3 (3 s)
ok    idle-splice-4 (4 s)
ok    idle-splice-5 (3 s)
ok    socket-soak-250x400ms (105 s)
ok    down-up-100 (403 s)
ok    no-leftover-test-processes
PASS: fabric 0.2.41 at d402abfbbbbe (5 idle-splice runs, soak 250 rounds)

The gate ran five idle-splice checks and the default socket soak (four workers × 250 rounds = 1,000 cycles). Its full shutdown check passed 100/100 idle cycles in 199.42 s and 100/100 live exec cycles in 199.77 s, including 50 SIGTERM-ignoring children. The test took 402.44 s including setup/cleanup; the gate step took 403 s including Cargo overhead. Every down returned with the daemon lease free and every immediate up succeeded. The gate's leftover-process check also passed.

This is the pre-tag dry-run receipt. The release owner will rerun the gate against the release commit and verify the tagged artifact before closing #297/#298.

Full Linux release-gate dry run passed on merged main e5126b9, including the concurrent exec/shell socket fix from #317, using:

FABRIC_DOWN_CYCLES=100 bin/release-gate-linux

Complete gate output:

release gate: fabric 0.2.41, revision e5126b9e5331, Linux 6.18.54, 16 CPUs, load 29.77
logs: /tmp/fabric-release-gate-e5126b9
ok    build (27 s)
ok    idle-splice-1 (4 s)
ok    idle-splice-2 (3 s)
ok    idle-splice-3 (4 s)
ok    idle-splice-4 (3 s)
ok    idle-splice-5 (3 s)
ok    socket-soak-250x400ms (109 s)
ok    down-up-100 (399 s)
ok    no-leftover-test-processes
PASS: fabric 0.2.41 at e5126b9e5331 (5 idle-splice runs, soak 250 rounds)

The gate passed all five idle-splice checks and the default socket soak (four workers × 250 rounds = 1,000 cycles). Shutdown/restart passed 100/100 idle cycles in 196.30 s and 100/100 live exec cycles in 199.08 s, including 50 SIGTERM-ignoring children. The shutdown test took 398.51 s including setup/cleanup; the gate step took 399 s including Cargo overhead. Every down returned with its lease free and every immediate up succeeded. The leftover-process check passed, and the memory-throttle watchdog did not trip.

This is the pre-tag dry run on the merged fix. The release owner will run the final gate on the release commit; #297, #298 and #316 remain open for tagged-artifact verification. No further source changes are included in this run.

@schickling

Field confirmation from the st replication benchmark (Fabric-mesh agent, agent/dotfiles/fabric-mesh): released 0.2.41+84c3208 still hits this on dev5 (NixOS x86_64, load1 ~40). Scratch --home, fabric down then an immediate fabric up (~117 ms after the old daemon logged peer closure). The new daemon logged Error: fabric daemon lease is already held 0.2 s later; up failed after ~10 s with fabric:internal daemon did not become ready. After cleanup: no daemon process, no kernel lock, a 0-byte run/daemon.lock left. Control socket path is 106 bytes, but no long-path dial error appeared (#297 not involved here).

This blocks our daemon-restart recovery case. Is a release with #308 (and #317) planned soon (1.0.0?), so we can repin and rerun on a released build?

Posted on behalf of @schickling

field	value
`agent_identity`	agent/dotfiles/fabric-mesh
`session`	01a1ecd5
`agent_persona`	generalist
`agent_supervisor`	unavailable
`agent_tool`	OMP
`agent_tool_version`	18.9.1
`agent_runtime`	OMP 18.9.1
`agent_model`	anthropic/claude-opus-5-5
`worktree`	smalltalk/main

Thanks for the field confirmation. 0.2.41+84c3208 predates the lease-release barrier in #308 and the concurrent exec/shell socket fix in #317. Both fixes are merged on main.

The release plan has changed to a 0.9.0 preview, targeted after #309 merges. Once that artifact is published and the release owner gives the go-ahead, I will rerun the released-artifact checks in order: #316, #297, then this issue, and post the exact version and results here. The merged-source release gate already passed 100 idle and 100 live down/up cycles, including 50 children ignoring SIGTERM; those numbers are source-build evidence, not yet released-artifact verification.

Release-PR source gate passed for 0.9.0, exact head 4e07e35b61c961c8b93bf82b87af3d07dd8bd6d7, using FABRIC_DOWN_CYCLES=100 bin/release-gate-linux on Linux. This is the source gate for #318; the tagged-artifact replay is still separate.

Idle down/up: 100/100, 197.558951888 s.
Live exec down/up: 100/100, 197.603064909 s, including 50 children ignoring SIGTERM.
Each down was checked for lease availability before the immediate up; every live cycle first verified the child's ready output.
Five idle-splice checks and the 4 × 250-cycle socket soak passed.
memory.high counters did not increase in the build cgroup or its monitored ancestors. The throttle stop did not trigger. No local-slice test process remained.

Full gate stdout:

release gate: fabric 0.9.0, revision 4e07e35b61c9, Linux 6.18.54, 16 CPUs, load 8.87
logs: /tmp/fabric-release-gate-090-4e07e35
ok    build (46 s)
ok    idle-splice-1 (4 s)
ok    idle-splice-2 (3 s)
ok    idle-splice-3 (4 s)
ok    idle-splice-4 (4 s)
ok    idle-splice-5 (3 s)
ok    socket-soak-250x400ms (104 s)
ok    down-up-100 (399 s)
ok    no-leftover-test-processes
PASS: fabric 0.9.0 at 4e07e35b61c9 (5 idle-splice runs, soak 250 rounds)

The stop-barrier test's final output:

idle down/up: 100/100 in 197.558951888s
live exec down/up: 100/100 (50 SIGTERM-ignoring children) in 197.603064909s
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 46 filtered out; finished in 398.21s

Tagged-artifact replay for #298: passed.

Version: 0.9.0+eb237b7. Binary SHA-256: 0b64611b45646729845ff554bbf3244de81fde9773b51a65ee845924a3b745ec.

Downloaded the published Linux x86_64 archive for release commit eb237b7. Archive SHA-256 a9a651f5f02fc73c27982a6223b696ab170b8868456627c27d9830b94df64cfd matches both the published sidecar and the release owner's supplied digest. Replayed in order #316, #297, #298 on isolated homes; production daemons were untouched.

Isolated scratch nodes; exact replay output:

[
  {
    "down_exit": 0,
    "down_seconds": 1.3385,
    "lease_free_when_down_returned": true,
    "immediate_up_exit": 0,
    "immediate_up_stdout": "started\nreachability: client\t8ea94d0058c0317b6e10c3a1ff68f1807045a13026e1ee28266f2f666805d536\tinbound-only\tnot connected",
    "immediate_up_stderr": ""
  }
]

Release owner retains issue closure.

Fixed in fabric 0.9.0 (tag v0.9.0, commit eb237b7), merged in #308. Numbers on the released artifact: fabric down returned in 1.34 s with the lease already free and the immediate fabric up exited 0. The release gate on the tagged tree ran 100 idle and 100 live-exec down/up cycles (50 children ignoring SIGTERM) with every up succeeding, in 197.6 s each. Details in the comments above.

Release-PR source gate passed for 0.9.1, exact head 40addd0c10a6966ca3b3d87a1b4c3a78a4a14304, using FABRIC_DOWN_CYCLES=100 bin/release-gate-linux on Linux. This is the source gate for #327; it is separate from a tagged-artifact replay.

Idle down/up: 100/100, 197.2307151s.
Live exec down/up: 100/100, 200.895945966s, including 50 children ignoring SIGTERM.
Each cycle checks lease availability when down returns, followed immediately by up; live cycles verify the child is ready first.
Five idle-splice checks and the 4 × 250-cycle socket soak passed; no local-slice test process remained.
All monitored memory.high counters stayed unchanged; the throttle stop did not trigger. Other builds from this seat were paused for the gate.

Full gate stdout:

release gate: fabric 0.9.1, revision 40addd0c10a6, Linux 6.18.54, 16 CPUs, load 16.36
logs: /tmp/fabric-release-gate-091-40addd0
ok    build (57 s)
ok    idle-splice-1 (6 s)
ok    idle-splice-2 (4 s)
ok    idle-splice-3 (6 s)
ok    idle-splice-4 (4 s)
ok    idle-splice-5 (5 s)
ok    socket-soak-250x400ms (104 s)
ok    down-up-100 (404 s)
ok    no-leftover-test-processes
PASS: fabric 0.9.1 at 40addd0c10a6 (5 idle-splice runs, soak 250 rounds)

Stop-barrier test totals:

idle down/up: 100/100 in 197.2307151s
live exec down/up: 100/100 (50 SIGTERM-ignoring children) in 200.895945966s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 47 filtered out; finished in 402.93s

@schickling

Field result on the released 0.9.0+eb237b7, from the st replication benchmark (Fabric-mesh agent, agent/dotfiles/fabric-mesh). Setup: dev4 (aarch64) and dev5 (x86_64), 35 daemon restarts (fabric down then an immediate fabric up) under real st replication, with dev5 load1 between 16 and 60.

No lease failures: 35/35 restarts came back, and replication recovered every time.
down: p50 30 ms, p90 54 ms, max 1.02 s.
up until the peer is reachable: p50 3.18 s, p90 3.23 s, max 3.59 s.
Replication recovered: p50 3.56 s, p90 4.49 s, max 4.89 s.

The ~3.2 s from up until the peer is reachable is now most of the restart cost. Is that floor expected (endpoint bind plus relay/home discovery), or could the daemon reach a known peer sooner after restart? We are not blocked on this; we are reporting it for Fabric 1.0 planning.

Posted on behalf of @schickling

field	value
`agent_identity`	agent/dotfiles/fabric-mesh
`session`	01a1ecd5
`agent_persona`	generalist
`agent_supervisor`	unavailable
`agent_tool`	OMP
`agent_tool_version`	18.9.1
`agent_runtime`	OMP 18.9.1
`agent_model`	anthropic/claude-opus-5-5
`worktree`	smalltalk/main

Release-PR source gate passed for 0.9.2, exact head e5491e0e42ed268978cdb92ea60130433f40d067, using FABRIC_DOWN_CYCLES=100 bin/release-gate-linux on Linux. This is the source gate for #331; it is separate from a tagged-artifact replay.

Idle down/up: 100/100, 197.611060521s.
Live exec down/up: 100/100, 198.421191158s, including 50 children ignoring SIGTERM.
Each cycle checks lease availability when down returns, followed immediately by up; live cycles verify the child is ready first.
Five idle-splice checks and the 4 × 250-cycle socket soak passed; no local-slice test process remained.
All monitored memory.high counters stayed unchanged; the throttle stop did not trigger. Other builds from this seat were paused for the gate.

Full gate stdout:

release gate: fabric 0.9.2, revision e5491e0e42ed, Linux 6.18.54, 16 CPUs, load 51.25
logs: /tmp/fabric-release-gate-092-e5491e0
ok    build (62 s)
ok    idle-splice-1 (5 s)
ok    idle-splice-2 (4 s)
ok    idle-splice-3 (4 s)
ok    idle-splice-4 (4 s)
ok    idle-splice-5 (4 s)
ok    socket-soak-250x400ms (105 s)
ok    down-up-100 (400 s)
ok    no-leftover-test-processes
PASS: fabric 0.9.2 at e5491e0e42ed (5 idle-splice runs, soak 250 rounds)

Stop-barrier test totals:

idle down/up: 100/100 in 197.611060521s
live exec down/up: 100/100 (50 SIGTERM-ignoring children) in 198.421191158s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 47 filtered out; finished in 399.40s

Release gate for 0.9.3 (PR #344, revision b823e2e) run on hetz2 with FABRIC_DOWN_CYCLES=100:

release gate: fabric 0.9.3, revision b823e2e88557, Linux 6.18.54, 16 CPUs, load 28.32
logs: /var/tmp/gate-093/logs
ok    build (102 s)
ok    idle-splice-1 (4 s)
ok    idle-splice-2 (4 s)
ok    idle-splice-3 (5 s)
ok    idle-splice-4 (4 s)
ok    idle-splice-5 (5 s)
ok    socket-soak-250x400ms (104 s)
ok    down-up-100 (415 s)
ok    no-leftover-test-processes
PASS: fabric 0.9.3 at b823e2e88557 (5 idle-splice runs, soak 250 rounds)

Release gate for 0.9.4 (tag v0.9.4, revision 19e7f89) run on a Linux host with FABRIC_DOWN_CYCLES=100, after the release, because the earlier gate host was offline:

release gate: fabric 0.9.4, revision 19e7f8964ce8, Linux 7.0.0-22-generic, 16 CPUs, load 13.64
ok    build (97 s)
ok    idle-splice-1 (4 s)
ok    idle-splice-2 (3 s)
ok    idle-splice-3 (3 s)
ok    idle-splice-4 (3 s)
ok    idle-splice-5 (4 s)
ok    socket-soak-250x400ms (104 s)
ok    down-up-100 (366 s)
ok    no-leftover-test-processes
PASS: fabric 0.9.4 at 19e7f8964ce8 (5 idle-splice runs, soak 250 rounds)

No release change needed.

Navigation Menu

Repository navigation

`fabric down && fabric up` can fail with 'fabric daemon lease is already held' #298

Observed

Expected

Proposed fix

Proof / test idea

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fabric down && fabric up can fail with 'fabric daemon lease is already held' #298

Description

Observed

Expected

Proposed fix

Proof / test idea

Activity

schickling-assistant commented on Oct 14, 2026

schickling-assistant commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 15, 2026

schickling-assistant commented on Oct 16, 2026

myobie commented on Oct 16, 2026

myobie commented on Oct 16, 2026

myobie commented on Oct 16, 2026

myobie commented on Oct 16, 2026

myobie commented on Oct 16, 2026

schickling-assistant commented on Oct 16, 2026

myobie commented on Oct 16, 2026

myobie commented on Oct 28, 2026

myobie commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`fabric down && fabric up` can fail with 'fabric daemon lease is already held' #298