helm: the chart cannot deploy a replicated range, so nothing in Kubernetes exercises replication

[allamiro]

Update (2026-08-08): the first half of this landed in #286. The chart can now
render a replicated range — data.topology: replicated picks leader/follower/standalone,
the leader gets a followers list of per-pod headless FQDNs, and the followers get a
lease block. Option (1) below is DONE.

Two things remain, and they are different sizes, so they are split:

• Nothing DEPLOYS the replicated shape. — DONE in meta,helm: follow the metadata leader, and make the replicated topology work at all (#292) #293. scripts/k8s-smoke.sh never sets
  data.topology, so CI only ever deploys the standalone default. The replicated
  shape is exercised by helm template in the render job and by nothing else — it
  has never actually run in a cluster. Rendering is not evidence; this whole
  exercise keeps proving that. Small, and it is what makes the node: complete the failover recovery protocol — epoch-qualified truncation, pre-probe fencing, transition evidence #240 arc reachable
  outside the live-chaos harness.
• Failover still needs an operator — option (2) below. This is the real
  remaining scope of this issue, and it interacts with node: vtop-node ignores SIGTERM, so every orderly stop is a crash stop #280 (a departing leader
  releasing its lease), so the two probably belong in the same milestone.

The original text follows unchanged.

---

The chart deploys three independent standalone ranges, not one replicated range. Three replicas are three separate logs, which is why --durability quorum is refused outright — "Quorum durability requires a configured replica set" — and why a pod nobody produced to stays empty. scripts/k8s-smoke.sh asserts both, so the behaviour is pinned rather than incidental.

That means the deployment shape which exercises replication, fencing, promotion, and the whole #240 epoch arc is not reachable from the chart at all. Everything that arc fixed is currently only exercised by the live-chaos harness.

The blocker the chart cited is gone

The template said:

Replicated ranges under co-location wait on follower-side epoch propagation upstream.

That landed in #239 (lease_watcher.rs — followers learn granted epochs without a restart), and the rest of the recovery arc landed as #240's stack. DataNodeConfig already supports role: leader with a followers list, role: follower, and a lease block that is explicitly valid on followers. The engine is no longer the constraint.

The constraint that is real

role is a static per-node config field, and a StatefulSet renders one config per ordinal. So a replicated range means:

• ordinal 0 → role: leader, with a followers list naming the other ordinals' headless FQDNs
• ordinals 1..n → role: follower, each with a lease block so it adopts granted epochs

That renders fine. What does not work is failover: when metadata grants the range to a different node, the new holder needs role: leader and the old one needs role: follower, and nothing re-renders a ConfigMap in response to a lease change. The live-chaos harness handles this by restarting the promoted node with a new config (scenario 09 does exactly that), which is a harness move, not an operator one.

Options, roughly in order of honesty

1. Render the static shape and document the limitation. Leader is ordinal 0; failover requires an operator to re-render and restart. Useful for testing replication and for deployments that accept manual failover, and it makes the node: complete the failover recovery protocol — epoch-qualified truncation, pre-probe fencing, transition evidence #240 work reachable outside the chaos harness.
2. Make the role dynamic in the binary — every node starts as a candidate and its role follows the lease, rather than being fixed at config time. This is the shape Kubernetes actually wants, and it would make the chart's job trivial. It is also the larger change, and it interacts with node: vtop-node ignores SIGTERM, so every orderly stop is a crash stop #280 (a departing leader releasing its lease).
3. Leave it standalone and keep replication exercised only by live-chaos.

(2) is where this should end up. (1) is worth having in the meantime, because without it nothing in Kubernetes ever exercises the replication path.

Also worth testing while here

Cross-namespace. The chart parameterises the namespace and every peer FQDN includes it, so two releases in different namespaces should be independent by construction — but "should be" is what this whole exercise keeps disproving. A second release in another namespace, both streaming, neither seeing the other's records, belongs in the smoke test as a variant.

---

Update (2026-08-08): the first half is done and verified live

#293 landed it. scripts/k8s-smoke.sh now deploys the replicated topology as a third shape and asserts the things only that shape can prove:

[k8s-smoke] producing with QUORUM durability, which a standalone range refuses
[k8s-smoke] quorum produce accepted 60 records
[k8s-smoke] follower 1 has durably applied all 60 records
[k8s-smoke] follower 2 has durably applied all 60 records
[k8s-smoke] replication verified in Kubernetes: quorum durability works and both followers hold the data

The assertions are deliberately the INVERSE of the standalone ones — quorum succeeds where it is refused, and a follower nobody produced to holds everything instead of staying empty — because that is what proves the topology took effect rather than merely being accepted. Both followers are checked, since quorum only proves a majority acked (with three replicas, the leader plus one), and it reads local_committed_offset, what the follower has FSYNCED, not next_offset.

Getting there needed three bugs fixed, all in #293: the topology rendered invalid YAML and had never once worked; a replicated install needs meta create-topic and register-node per node, which nothing documented; and nothing followed a Raft leader redirect (#292), so two of three replicas failed closed permanently.

What remains under this number is option (2) alone: failover still needs an operator. role is fixed at render time, so when metadata grants the range elsewhere nothing re-renders the ConfigMap. That interacts with #280 (a departing leader releasing its lease) and the two probably belong in the same milestone.

[allamiro]

Moved to v0.4.0.

v0.3.0's claim is the transfer arc — "segment transfer and what it unblocks: verbatim record transfer between replicas (#255), live replica replacement (#242), and the follower-restart removal that completes #239." This issue is not part of that; it got attached to the milestone later.

What remains under this number is option (2) alone — failover needs an operator, because role is fixed at render time. That is deployment capability, and it belongs with the rest of it: #294 (TLS as a configured capability), #287 (separated deployment mode), and #280 (SIGTERM), which it directly interacts with — a departing leader that cannot release its lease makes dynamic roles worse rather than better.

The half of this issue that mattered for v0.3.0 is already done and verified: #293 made the replicated topology work at all and scripts/k8s-smoke.sh now deploys it and asserts quorum durability with both followers durably applied. Nothing in the transfer arc is waiting on what is left here.

[allamiro]

Design for option (2), dynamic role — before any code, per the #265 lesson

Traced the full surface first. The shape that falls out is contained entirely in vtop-node: no broker-crate surgery, no protocol change, and half of "role follows the lease" turns out to already work.

What already works

• Leader → fenced is a live, tested transition. BrokerLeasePublisher::promote/demote/suspend toggle three atomics; a LocalBroker whose lease view is inactive refuses every produce and reports not-ready. Nothing about demotion needs inventing.
• The lease agent is not leader-shaped. Its publisher is a trait (a follower implementation already exists), its probe is Option, and its admin client is the same helper the follower's watcher uses. The only reason followers don't run it is stated policy in lease_watcher.rs — candidate mode is precisely the decision to reverse that sentence, and the machinery that makes N candidates safe already landed: single linearizable acquisition, the §5.4.1 election restriction, and the stand-aside hold-off (node: promotion enforces Raft's election restriction — a majority must vouch for the candidate (#240) #342).
• Both roles already bind replica_listen, and the chart gives both 0.0.0.0:9300. A follower learns granted epochs (node: follower-side lease watcher — propagate granted epochs to followers #239); the lease view also carries holder_node_uuid, currently discarded — "who leads" is free for any node with a lease block. Peer addresses stay config (metadata's admin surface doesn't expose them, and shouldn't need to).

The one hard problem, and the chosen answer

InProcessFollower and LocalBroker each take the SegmentSet by value — one process cannot hold both over one range. Three options considered; choosing (A): a role supervisor in vtop-node, with two structural commitments that remove (A)'s classic weaknesses:

1. Listeners bind once, in every role, and never rebind. native_listen binds unconditionally (a lease-inactive broker fails closed already); replica_listen serves a role-switching ReplicaPeerHandler — an RwLock-swapped delegate pointing at InProcessFollower while following and LeaderStatusReplica while leading. No socket ever moves, no EADDRINUSE window, and — the sharpest lesson from scenario 09's choreography — no node ever takes over another node's addresses, which is what a Kubernetes pod cannot do anyway.
2. Transitions run in node: vtop-node ignores SIGTERM, so every orderly stop is a crash stop #280 order, in-process. Demotion is "stop admitting → drain native sessions → quiesce() → swap the handler → rebuild as follower", with the same ReleaseRangeLease proposal on voluntary handoff. Promotion rebuilds open_range + NetworkedReplicaSet from the peer list minus self, keeping the existing 10s follower-stream wait off the lease-renewal critical path.

Rejected: (B) storage handover between the two state machines (touches the most safety-critical code in the workspace for a cleaner steady state we don't need yet) and (C) impl ReplicaPeerHandler for LocalBroker (re-implements the reconciliation logic #261/#315 hardened — the end state, maybe, but not the increment).

Config: symmetric, one new variant

role: candidate + peers: (every replica including self; self filtered by node_uuid — the leader path reuses its NetworkFollowerConfig construction verbatim). lease: becomes mandatory for candidates; fencing_epoch: 0 floor; native_tls/principal_id on every node. transfer_allowed becomes "peers minus self", which is what #270/#301 wanted anyway. Standalone stays exactly as-is.

Gotchas already scoped

• One role-agnostic replica collector replacing BrokerCollector/FollowerCollector for candidates (they export the same metric names; registering both is a duplicate-descriptor error — and one collector keeps dashboard series continuous across a promotion instead of gapping).
• node_info{role="data-candidate"} set once; the live verdict stays in the lease gauges, not a mutating label.
• Readiness: gate on "listener bound ∧ first metadata read", then a live probe whose ready verdict is "I lead with a live view" or "I follow at the granted epoch" — otherwise healthy followers drop out of every Service.
• Ready markers: data_node_ready role=candidate plus a new data_node_role_changed role=… epoch=… line, so the harness asserts transitions instead of restarts.

Proof plan

• Transition state machine unit-tested against the existing LeasePublisher/QuorumProbe traits (they exist precisely so this is assertable without a disk).
• Scenario 09 rewritten to restart nothing: kill the leader, assert a surviving candidate takes the range and serves on its own native address — the current restart choreography (role flip, port takeover, follower list, TLS, epoch floor) collapses to zero config deltas.
• k8s-smoke gains the assertion it cannot write today: kubectl delete pod the leader in the replicated namespace and watch quorum produce resume against another pod with no helm upgrade. data.leaderOrdinal and the "failover is manual" caveat disappear from the chart.

Follow-on (not this slice)

With per-pod /readyz already meaning "fenced ⇒ not ready", a Service selecting on readiness over the native port becomes viable for the first time — "which pod do I produce to" gets a Kubernetes-native answer. Worth its own issue once candidate mode lands.

Building this next unless there's an objection to the (A)-with-fixed-listeners shape.