mount treats 410 cursor_expired as fatal; the advertised full resync is never attempted and the fallback repeats the same call

Relayfile mount treats 410 cursor_expired as fatal, although the server returns the remedy in the error text. The documented recovery — a full resync — is never attempted, and the stop-and-once fallback repeats the same cursor-based operation and collects the same 410.

This has killed two review swarms in the AgentWorkforce/flows repo, 20 hours apart, with structurally identical traces.

The failure

failed to start poll mount: notify flush: daemon pid 4085 flush failed:
  http 410 cursor_expired: event cursor is no longer available; perform a full resync

then the fallback:

..., then stop-and-once fallback failed (... Relayfile mount --once reported a failed sync cycle
  mount sync cycle failed:
  http 410 cursor_expired: event cursor is no longer available; perform a full resync)

Both paths end on the same error. The server is explicit about what to do next and neither path does it, so a recoverable condition is terminal in practice.

Occurrences

when	flows PR	run root
2026/09/09 08:46:52	#240	`/workflows/runs/79c34ff1-7e89-41d6-a780-64b0b1683b94`
2026/09/10 04:48:59	#258	`/workflows/runs/7ae845bb-e2d5-4182-934d-0166b1a2cb42`

Different runs, different sandboxes, same sequence line for line.

A second fault in the same trace

Both runs also log, immediately before the failure:

websocket unavailable; using polling sync:
  failed to WebSocket dial: expected handshake response status code 101 but got 403

So the mount is being 403-rejected on the WebSocket upgrade and degrading to polling, and it is the polling path that carries the cursor that then expires. Those two facts sit next to each other in every trace, but I have not established that the first causes the second and I am not claiming it — the 403 is worth its own look regardless of whether it is upstream of the cursor expiry.

Impact

lens-maintainability is the first step in the flows review swarm, so this failure takes down the whole review gate for a PR. Both of the affected PRs are still blocked. In the wider picture this is one of seven distinct infrastructure faults that broke that gate over 22 hours (AgentWorkforce/flows#255), and the only one to recur.

Not verified

Whether a full resync would in fact succeed here — I have no way to force one from outside the sandbox. The claim is narrower: the client is told to resync, does not, and its fallback re-runs the operation that just failed the same way.

Third occurrence, and the last two are consecutive.

when	flows PR	run root
2026/09/09 08:46:52	#240	`79c34ff1`
2026/09/10 04:48:59	#258	`7ae845bb`
2026/09/10 05:37:16	#259	`e6c8c57a`

Same trace line for line each time: notify flush: daemon pid N flush failed: http 410 cursor_expired, then the stop-and-once fallback failing with the identical 410.

When I filed this I noted it was the only one of seven distinct failure modes to recur. That has changed materially: #258 and #259 failed back to back with nothing else in between. Across the nine PRs I sampled earlier no mode had ever repeated consecutively — the platform was failing a different way every time. It is no longer doing that. This mode now looks persistent rather than intermittent, which makes it both more disruptive and considerably easier to reproduce than when I filed.

Both of those PRs are gate fixes for the flows review swarm, so this failure is currently the thing standing between that repo and any merge at all.

The remedy already exists in the client. Nothing triggers it.

I said when filing that the advertised full resync "is never attempted". That was right about the behaviour but wrong about the cause — it is not missing, it is just never reached automatically.

cmd/relayfile-mount/main.go:117:

fullReconcile := flag.Bool("full-reconcile",
    boolEnv("RELAYFILE_FORCE_FULL_RECONCILE", false),
    "force one full reconcile regardless of bootstrap-complete state (escape hatch)")

wired through at :219 (forceFullRecon) and :501 (ForceFullReconcile). The same flag exists on relayfile-cli (main.go:7362).

So the mount can do exactly what the 410 asks for. It simply has no path from "server returned 410 cursor_expired" to "do the full reconcile you already support" — the error propagates up through notify_flush_unix.go:70 as an opaque string and kills the process instead.

That makes this a smaller fix than I implied when filing: the capability is there, it needs a trigger.

Two things I could not establish

cursor_expired is not emitted by this repo. The only http.StatusGone sites are internal/httpapi/server.go:1539 and :1580, and both are fork_expired. Nothing in the tree emits cursor_expired or the string "perform a full resync", so that response comes from the hosted control plane and I cannot read its side from here. resync appears in this repo only inside two tests.
I read v0.10.52 and the deployed mount may differ. I have been caught by that distinction before, so I am flagging it rather than asserting the deployed binary behaves the way this source does.

A possible workaround worth testing

Because the flag reads RELAYFILE_FORCE_FULL_RECONCILE from the environment, a caller may be able to break the loop without any change here by setting that variable for the affected mount.

I have not tested it and I am not proposing it as a fix. Two reasons for caution: the help text says "force one full reconcile", so it may clear a stuck cursor without preventing the next one; and turning an escape hatch on permanently for every run has a cost nobody has measured. It is a lead for whoever picks this up, not a recommendation.

Context on urgency: this fault has now taken down three flows review-swarm runs, the last two consecutively (AgentWorkforce/flows#255), and it is currently the only thing failing those PRs.

Retracting most of this issue: the recovery exists on main, and I was reading a stale checkout

My previous two comments were made against a local checkout 95 commits behind origin/main, at v0.10.52 while v0.10.57 is latest. Both of my central claims are wrong as a result. Correcting them properly rather than editing them away.

What I said: cursor_expired is emitted nowhere in this repo, so it must come from the hosted control plane.
Actually, on origin/main:

// internal/mountsync/syncer.go:323
func isCursorExpired(err error) bool {
    var httpErr *HTTPError
    return errors.As(err, &httpErr) &&
        httpErr.StatusCode == http.StatusGone &&
        strings.EqualFold(strings.TrimSpace(httpErr.Code), "cursor_expired") &&
        (strings.TrimSpace(httpErr.Action) == "" ||
         strings.EqualFold(strings.TrimSpace(httpErr.Action), "full_resync"))
}

What I said: the advertised remedy is never implemented, only reachable via the --full-reconcile escape hatch.
Actually, syncer.go:6103 already does exactly the documented recovery:

if cursorExpired {
    s.logf("events cursor expired; clearing watermark and performing full resync")
    s.forceFullReconcile = true
}
s.state.EventsCursor = ""
s.state.IncrementalCheckpoint = nil

That landed as 9b83d76e — "fix(mount): recover expired event cursors (#476)", committed 2026-09-08 07:37 and first released in v0.10.56 (2026-09-08). So this issue is substantially a duplicate of #476.

The part that is still real, and is now a sharper question

All three observed failures happened after v0.10.56 shipped:

2026/09/09 08:46:52  flows#240
2026/09/10 04:48:59  flows#258
2026/09/10 05:37:16  flows#259

And the recovery's own log line — events cursor expired; clearing watermark and performing full resync — appears in none of the three run logs, while cursor_expired appears twice in each. The recovery did not run.

So the question is no longer "implement a resync" but "why did the existing recovery not fire?" Two candidates I cannot separate from outside the sandbox:

The sandbox image pins a relayfile older than v0.10.56. No version string is logged in any trace, which is itself worth fixing — it would have answered this in one grep.
The failing path never reaches the recovery. isCursorExpired has exactly one call site (syncer.go:6103), and it depends on errors.As recovering a typed *HTTPError. Our failure surfaces through notify_flush_unix.go:70, which carries the daemon's error across a process socket as an opaque string (ack.Error). A typed error cannot survive that boundary. If the expiry is detected on the flush path rather than the incremental-read path, no version of this code would recognise it.

Candidate 2 would be a genuine gap and not a duplicate. I cannot tell which it is without either the sandbox's relayfile version or a repro, and I have neither.

Retitling suggestion for whoever owns this: the useful issue is "cursor-expiry recovery does not cover the notify-flush path (or the sandbox pins < v0.10.56)", and logging the mount version would settle it immediately.

Confirmed fixed. Recommend closing.

The snapshot carrying relayfile-mount v0.10.56 was promoted at 07:27Z (cloud 04228b3cb). I then re-ran a flows review-swarm that had failed on this fault twice.

Run 34440534265, full log:

cursor_expired occurrences: 0

Zero, against two occurrences in each of the three prior failures. The run failed for an unrelated reason (workspace_busy during agent registration), so the swarm got past the mount entirely.

That resolves the question I left open two comments up. It was candidate 1 — the sandbox pinned a relayfile older than v0.10.56 — and not candidate 2, the theory that the notify-flush path could never reach the typed-error recovery. Recording that explicitly because I argued candidate 2 was the more interesting possibility, and it was wrong.

The only part of this issue that was ever novel was my mis-reading of a stale checkout. Suggest closing as fixed by #476.

Closing per the confirmation two comments above. v0.10.56 snapshot was promoted (cloud 04228b3cb at 07:27Z, then a55167914 sdk-11.11.0 at 09:13Z, both carrying relayfile-mount v0.10.56), and a re-run of the same flows swarm that had failed twice on this fault showed zero cursor_expired occurrences in the whole log. The recovery landed in #476. Nothing here is novel beyond my mis-reading of a stale checkout.

Navigation Menu

Repository navigation

mount treats 410 cursor_expired as fatal; the advertised full resync is never attempted and the fallback repeats the same call #492

The failure

Occurrences

A second fault in the same trace

Impact

Not verified

The remedy already exists in the client. Nothing triggers it.

Two things I could not establish

A possible workaround worth testing

Retracting most of this issue: the recovery exists on main, and I was reading a stale checkout

The part that is still real, and is now a sharper question

Confirmed fixed. Recommend closing.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

mount treats 410 cursor_expired as fatal; the advertised full resync is never attempted and the fallback repeats the same call #492

Description

The failure

Occurrences

A second fault in the same trace

Impact

Not verified

Activity

kjgbot commented on Sep 10, 2026

kjgbot commented on Sep 10, 2026

The remedy already exists in the client. Nothing triggers it.

Two things I could not establish

A possible workaround worth testing

kjgbot commented on Sep 10, 2026

Retracting most of this issue: the recovery exists on main, and I was reading a stale checkout

The part that is still real, and is now a sharper question

kjgbot commented on Sep 10, 2026

Confirmed fixed. Recommend closing.

kjgbot commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions