Skip to content

fix(client): promote a follower after the leader yields leadership - #319

Closed
prisis wants to merge 10 commits into
alphafrom
fix/client-leader-handoff
Closed

prisis wants to merge 10 commits into
alphafrom
fix/client-leader-handoff

Conversation

@prisis

@prisis prisis commented Aug 2, 2026

Copy link
Copy Markdown
Member

Work from an agent worktree, pushed under a descriptive name (local branch was worktree-agent-a296f70ca717ffc10).
Branched before #288/#289 landed, so it needs a rebase onto current alpha before merge.

Commits

  • fix(client): promote a follower after the leader yields leadership
  • fix(client): mirror leader connection status so follower tabs queue offline writes
  • fix(client): scope the cross-tab settled watermark to the owning clientId
  • fix(client,do,db): carry lastMutationId on list data frames and consume it
  • fix(client): scope the cross-tab channel to deployment and identity
  • fix(client): stamp and verify leader identity on cross-tab data frames
  • test(client): close every fake/real-timer LunoraClient in socket-lifecycle tests
  • fix(db): stop a zero mutation watermark from disabling the RPC-ack fallback
  • fix(client): promote a leader tab's fresh coordinator immediately
  • chore(api-snapshots): sync client and shard-engine snapshots

prisis added 10 commits August 1, 2026 17:33
When the leader tab closes (sign-out, SPA teardown, HMR dispose), its
yield-leadership broadcast cleared knownLeader on every follower but
nothing ever claimed leadership again, freezing every remaining tab's
live queries until reload. Add claimAndPromote(), shared by the yield
handler (claims immediately) and checkLeaderHealth's belt-and-braces
else-branch (claims when knownLeader is stranded undefined for any
other reason), guarded by a promotionPending flag so the two triggers
can't stack becomeLeader timeouts.
…ffline writes

A follower tab never had a ShardConnection record, so the offline write
queue was silently disabled on every follower (a mutation during a
network drop rejected and rolled back instead of queueing), and
connectionStatus() reported "idle" forever regardless of the leader's
real socket state. Add a connection-status broadcast the leader sends
on every status change (and once on becoming leader), mirrored by
followers into leaderStatus/leaderWasEverConnected; computeStatus() and
the mutation() offline-queue gate both read the mirror instead of the
(always-empty) connections map on a follower. A follower now flushes
its own queued writes over HTTP once the mirrored status reports
"connected" again.

Also extract the demotion connection-teardown sequence into
teardownConnection() (clearing reconnect/connect timers and stopping
the heartbeat, not just closing the socket) so onStopBeingLeader
matches close()'s teardown and stops leaking timers on a demoted
leader.
…ntId

The leader's settled broadcast carried its own per-client mutation
watermark, and followers applied it unconditionally to their own
lastMutationId and @lunora/db checkpoint gates — dropping a follower's
optimistic overlay for writes its own client never had confirmed (or
that were rejected), since the watermark is scoped server-side to the
leader's own clientId.

broadcastSubscriptionSettled/onSubscriptionSettled now carry an
optional clientId; a follower applies the mutationId half only when it
matches its own clientId (Math.max-guarded for monotonicity), and
otherwise forwards only the checkpoint half. An absent clientId
(mixed-version leader) also skips the mutationId half, relying on the
follower's own RPC-ack watermark path and the CheckpointRegistry's
bounded fallback.

Updates the existing CLIENT-01 "onCheckpoint fires with the
leader-broadcast lastMutationId" test to stamp a matching clientId
(client.clientIdentifier()), since that behavior is now conditional on
the clientId matching.
…me it

ServerDataMessage.lastMutationId was documented as the outbox-drop
signal but never sent by the server or read by the client. @lunora/db
compensated by resolving the checkpoint gate from the provisional
RPC-ack watermark (confirmedMutationWatermark) whenever any data frame
arrived, regardless of whether that frame's rows actually reflect the
most recently acked write — a write B acked while only write A's rows
have synced drops B's still-pending overlay early.

The plain (non-delta) list data frame now stamps the same per-client
watermark the sibling settled frame already carried (shard-do.ts,
socketClientWatermark). handleDataMessage consumes it (monotonic
Math.max guard) and fans it out to checkpointCallbacks BEFORE
notifySubscription, so @lunora/db's onRows sees a frame-scoped
watermark before it would otherwise fall back to the provisional
signal. lunoraCollectionOptions now prefers that frame-carried
watermark once observed, keeping confirmedMutationWatermark only as
the fallback for an un-upgraded server or the very first frame.

The delta-frame path is out of scope (follow-up); the cross-tab
broadcastSubscriptionData rebroadcast deliberately does not carry the
field — a follower's watermark comes only from the clientId-scoped
settled path (plan 266 S3).
Every LunoraClient with crossTabSync: true joined the same
BroadcastChannel ("lunora-bridge") regardless of signed-in identity or
backend url, so a follower adopted any leader-broadcast subscription
value matching its own (fn, args, shardKey) key with no identity or
deployment check — two same-origin tabs signed in as different users
(or two different Lunora apps on one origin) could leak authoritative
query results across identities, and unrelated apps could fight over
leader election.

The coordinator's channel name now derives from
`lunora-bridge::<url>::<identityFingerprint() ?? "anon">`, computed in
a new createTabCoordinator() (extracted from the constructor so
setAuthToken can rebuild it). setAuthToken restarts the coordinator on
its own re-derived channel whenever the fingerprint changes (a genuine
credential change or a subject resolving), after the existing
identity-change queue handling so drained/restamped writes settle
before the new group's leader election begins.

Updates the CLIENT-01/plan-266 tests that posted raw frames to the
hardcoded "lunora-bridge" channel to derive the client's actual
(now identity-scoped) channel instead.
The channel-name scoping (previous commit) is the primary defence
against cross-identity leaks, but has a race: setAuthToken in one tab
can move it to a new channel while a stale subscription-data/-settled/
connection-status frame from the OLD leader is already queued in the
message task queue.

subscription-data, subscription-settled, and connection-status frames
now carry the leader's own identityFingerprint() (null = signed out,
distinct from an absent field). A follower drops a frame whose stamp
is present and doesn't match its own fingerprint; an absent stamp
(mixed-version leader) is still accepted, since the channel split
already separates version groups.
…cycle tests

Several reconnect/heartbeat/wsToken-provider tests constructed a
LunoraClient with an armed reconnect timer (some via vi.useFakeTimers(),
some with a real short reconnect delay) and never called client.close().

This is a genuine pre-existing flake, reproducible on baseline
(071c6a2) at a comparable rate to what was observed on this branch
(measured ~25-30% baseline failure rate across 27 runs in this
environment, both before and independent of this branch's diff): an
unclosed client's dangling reconnectTimer/connectTimer/heartbeatTimer
survives past vi.useRealTimers() and can fire during a LATER,
unrelated test's execution (same file, same module-level `sockets`
array reset only per-test by beforeEach), constructing a stray
WebSocket that pollutes whichever test happens to be running —
exactly the class of flake several tests' own comments already
described and worked around with sockets.find(...) token matching
instead of fixing at the source.

Adds client.close() to every affected test (verified via a script
auditing every `new LunoraClient(...)` call against a matching
`.close()` within its own test body), closing while fake timers are
still installed where applicable so close()'s clearTimeout/
clearInterval calls target the still-active fake clock. Verified
clean across 47 consecutive full-suite runs (vs. a reproducible flake
on both this branch and true baseline before the fix).
…llback

frameWatermark ?? confirmedMutationWatermark(...) treated a genuine 0
mutationId (the value readClientWatermark returns for a client's first
frame) as already-resolved, and the value was sticky across onCheckpoint
calls, so once any frame carried a watermark the RPC-ack fallback never
ran again — including after a tab is demoted and its own onRows never
receives a fresh checkpoint. Consume the watermark once per onRows call
instead of accumulating it, so a follower's stale leader-era value can't
mask a missing one and every write still gets an RPC-ack path.

Delta frames never carried lastMutationId at all (only the snapshot
branch did), which made the gap permanent for @lunora/db list
collections, since those move into delta mode after their first frame.
Thread the per-socket watermark through sendDeltaFrames so every delta
frame stamps it too, and hoist the socketClientWatermark point read to
once per flush instead of once per subscription.
setAuthToken's identity-change restart tears down the old
TabCoordinator (closing every shard socket via onStopBeingLeader when
this tab was leader) and constructs a fresh one on the re-derived
channel, which then sits through the full leaderTimeout before
self-promoting. That froze every live query for up to 3s on any
identity change, including a routine token refresh with no stable
subject passed — contradicting setAuthToken's own documented contract
that it does not touch the socket.

A tab that was leading before the restart is overwhelmingly likely to
still be alone on the new channel, so promote it immediately instead of
waiting out the timeout; a sibling tab going through the normal start()
dance for the same transition still defers to the heartbeat, and the
existing leader-vs-leader tie-break covers the rare double-promotion
case.
Reflects TabCoordinator.promoteImmediately, the identity params added to
broadcastSubscriptionData/broadcastSubscriptionSettled, the new
broadcastConnectionStatus, and sendDeltaFrames' lastMutationId param.
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9827f5d9-f7e6-4364-b2aa-3316c61b1212

📥 Commits

Reviewing files that changed from the base of the PR and between 6a4dd72 and 19c5685.

⛔ Files ignored due to path filters (6)
  • api-snapshots/client.api.md is excluded by none and included by none
  • api-snapshots/shard-engine.api.md is excluded by none and included by none
  • packages/client/__tests__/cross-tab.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/client/__tests__/lunora-client.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/db/__tests__/collection-options.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/subscription-data-watermark.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (5)
  • packages/client/src/cross-tab.ts
  • packages/client/src/lunora-client.ts
  • packages/db/src/collection-options.ts
  • packages/do/src/shard-do.ts
  • packages/shard-engine/src/subscription-delivery.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@netlify

netlify Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

❌ Deploy Preview for lunorash failed.

Name Link
🔨 Latest commit 19c5685
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6f0c0c0ba5cf0008156813

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@prisis

prisis commented Aug 2, 2026

Copy link
Copy Markdown
Member Author

Closing: this PR was opened in error by an automated run, outside the scope of its task.

No work is lost — the branch is retained on origin and this PR can be reopened at any time.

Context: this branch predates the merges of #288 and #289, so much of its content is already on alpha, and it would need a rebase before being reviewable. Reopening deliberately, after a rebase, is preferable to leaving it open in this state.

@prisis prisis closed this Aug 2, 2026
@prisis
prisis deleted the fix/client-leader-handoff branch August 2, 2026 09:47
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant