Standby replication: forward-sequence gap is masked as consistent → silent divergence and wrong replay floor at promotion #498

The standby's forward-sequence consistency tracking has no gap detection: a forwarded command lost mid-stream is silently treated as applied, so the standby diverges from the primary and, when promoted, recovers from a checkpoint floor that is above the missing data — the data is never replayed. This is the "command-chain gap → unrecoverable" failure mode.

The masking

CcShard::UpdateLastReceivedStandbySequenceId (tx_service/src/cc/cc_shard.cpp:3498-3499) advances the consistency marker with a plain max, with no check that seq_id == last_consistent + 1:

seq_grp_info.last_consistent_standby_sequence_id_ =
    std::max(seq_id, seq_grp_info.last_consistent_standby_sequence_id_);

The fields that exist to detect a gap — missing_standby_seqeunce_ids_, next_expecting_standby_sequence_id_ (standby.h:121-126) — are dead code: nothing reads them on the receive path (only declared / reset). So if forward N is lost but N+1 arrives, last_consistent jumps to N+1, marking the never-applied N as consistent.

How a forward is lost without resync

Primary side advances last_sent_seq_id on StreamWrite success (buffer-accept, not delivery):

tx_service/src/cc/cc_shard.cpp:3194-3200 (ForwardStandbyMessage):

write_succ = stream_sender_->SendStandbyMessageToNode(node_id, entry_ptr->Message());
if (write_succ) { last_sent_seq_id++; }   // advanced even though delivery isn't guaranteed

Resend after a failure starts at last_sent_seq_id + 1 (cc_shard.cpp:3063-3071). When a stream breaks, messages already accepted into the brpc stream write buffer but not yet delivered are lost, yet last_sent has advanced past them, so they are never resent — and there is no resubscribe/resync on on_closed for this path. The standby then receives a later seq id and masks the hole via the max above.

Impact

last_consistent_standby_sequence_id_ feeds the standby consistent ts, which becomes the replay_start_ts floor at promotion. A masked gap → the missing command's effect is neither applied on the standby nor replayed from WAL after promotion → permanent silent divergence / lost committed update on a former standby that becomes leader.

Fix: make the receive path reject/track out-of-order seq ids (wire up the existing missing-sequence machinery), trigger a repair/resubscribe on a detected gap, and/or advance last_sent only on confirmed delivery.

Found during a code audit (PR #493). The masking (no gap check, dead missing-seq fields) and the send-side last_sent-on-StreamWrite-success are verified against source; the loss trigger depends on brpc stream-break dropping buffered-but-unacked writes.

🤖 Found with Claude Code