Rollout ordering: server expires cursors (cloud#196) but deployed mount clients predate the 410 handler (#476, v0.10.56)

[kjgbot]

Sandbox agent-relay cloud run workflows fail after their command steps succeed, because the post-step mount flush dies on 410 cursor_expired and never performs the full resync the error asks for. Two independent occurrences on 2026-09-09, ~20 min apart, on different runs.

This is a rollout ordering problem, not a missing fix

relayfile-cloud#196  "expire cursors and retain event history"   merged 2026-09-07 21:54Z   <- server starts emitting 410
relayfile#476        "resync expired event cursors"              merged 2026-09-08 05:37Z   <- client handler
                                                                  released v0.10.56 05:51Z

The server began expiring cursors roughly eight hours before the client-side handler was released. #476 is exactly the missing behaviour — "recognize 410 cursor_expired / action: full_resync … clear the stale event cursor and force exactly one authoritative full-tree reconciliation" — and its own description flags the dependency:

makes retained-event pruning safe for mount clients once both PRs are reviewed, merged, released, and intentionally enabled by a human

The deployed clients are all older than the handler

sandbox snapshot:  relay-orchestrator-sdk-11.10.3-relayfile-v0.10.55-...   <- one release short
sf-mini CLI:       0.10.41
finn-mini CLI:     0.10.22

That answers the reporter's question of why cursor expiry does not trigger the resync its own log message names: the code that would do it is not in v0.10.55.

Suggested fix

Bump the sandbox snapshot's pinned relayfile to v0.10.56 or later, and audit any other pinned client. Until then, every mount whose event cursor expires mid-run fails non-deterministically, which blocks both demo workflows and the flows review-swarm CI gate.

The other half — the WebSocket 403 — is probably separate

The reported sequence is: WebSocket handshake returns 403 → falls back to polling → polling's cursor is rejected with 410 → no resync. The 410 half is explained above. The 403 is not, and should not be assumed to share a cause.

Worth noting alongside it, without claiming causation: delegated credential minting is currently failing fleet-wide for CLI clients —

mint delegated relayfile credentials: http 502 relayauth_mint_failed:
RelayAuth request failed (500) /v1/identities;
unable to verify whether RelayAuth identity creation committed

A 403 on a WebSocket upgrade is consistent with an auth/token problem, and the reporter separately observed Relaycast workspace key repair failed: 503 database_overloaded the same day. Those may share a root cause in the session/credential path, or may not — it is being investigated separately and should be established rather than assumed.

Relayfile itself is up. Verified from the deployed factory's authenticated evidence: Relayfile workspace mirror preflight: mounted=1 failed=0 routedRepos=19, with successful ensureSubRoot and listTree operations. Clients holding a valid credential and a current version work fine.

[kjgbot]

Correction: this is not sandbox-only, and the sandbox rebuild is not what closes it.

I framed this around the Daytona snapshot pin. That is real but it is the smaller half. The reported failures also occur running the mount locally, where no snapshot is involved — and there the same version gap exists:

npm dist-tag latest:  0.10.56    <- contains the #476 cursor handler
sf-mini:              0.10.41
finn-mini:            0.10.22
sandbox snapshot:     v0.10.55

Every client we run is older than the release that handles 410 cursor_expired. The server started emitting it on 2026-09-07 21:54Z (relayfile-cloud#196); the handler shipped 2026-09-08 05:51Z (v0.10.56). Anything installed before that, anywhere, fails the same way.

What actually closes this

Two independent bumps, and neither substitutes for the other:

1. Local / CI clients → upgrade to relayfile@0.10.56 or later. This is the one that matters for the reported occurrences. latest already points at 0.10.56, so this is an upgrade, not a release.
2. Daytona sandbox → rebuild the snapshot. The mount binary is image-owned, so the version only moves with a rebuilt image (Rebuild Daytona Snapshot action in AgentWorkforce/cloud). SNAPSHOT.md explicitly warns against editing the name pins by hand — that would point Daytona at an image that does not exist.

Acceptance for both: a run whose event cursor expires mid-flight performs exactly one full resync and completes, instead of failing the step.

Still not explained

The WebSocket 403 remains unaccounted for. Neither bump addresses it. If it has an auth cause it will survive both, and the reported sequence (403 → polling fallback → 410) means fixing only the 410 leaves runs depending on a fallback path that should not have been needed.

Worth stating plainly so it does not get lost when the version bumps land and the symptom partially disappears.

[khaliqgant]

Fresh hosted evidence from Relay PR #1683 exact head 54dd0819102a77c5ace65af596729d2363f293ed: Actions run 34364211839 dispatched Cloud run 77b370d9-2d9e-48e6-9284-5377ad078077. The base arm completed and validated the bug; the head arm later failed when the Relayfile mount notify flush returned HTTP 410 cursor_expired. The Daytona snapshot was still pinned to relayfile 0.10.55. WebSocket setup also returned 403, and the stop-and-once fallback retried with the same expired cursor instead of completing a full resync. Both step sandboxes were cleaned. The terminal failed outer sandbox remained started and was manually deleted after exact run-label verification. This independently confirms that the Cloud Daytona snapshot must be rebuilt with relayfile 0.10.56 or newer and then proven to perform exactly one full resync after cursor expiry.

[khaliqgant]

Cloud now has a concrete v0.10.56 full/lite snapshot pair from run 34320569038, but it is not qualification evidence: both mount-convergence probes skipped because every RELAYFILE_SMOKE_* input was empty while the workflow still concluded success. Cloud issue #3495 tracks that false-green. The deployed selector remains on v0.10.55. We will use the v0.10.56 image only as an unpromoted candidate and require the schema-v3 cold plus two-concurrent exact 851-file / 270532608-byte acceptance, including one expired-cursor full resync, before selector promotion.