tls: make transport security a configured capability rather than a hard dependency, consistently across every deployment method

[allamiro]

What is true today

Audited across crates/, helm/, docker-compose.yml, examples/ and docs/.

Plane	TLS today	Can it be turned off?	Verify toggle	Custom CA
Metadata Raft peer	mandatory mTLS	no	no	yes — ca is a required path
Metadata admin	mandatory mTLS	no	no	yes
Replication (leader↔follower)	mandatory mTLS	no	no	yes
Native client (produce/fetch)	mandatory mTLS	no	no	yes
Observability (/metrics, /healthz, /readyz)	none — plaintext only	n/a (cannot be enabled)	n/a	n/a
S3 / object store	optional	yes — verify_tls	partial	system trust store
PostgreSQL	required for remote	yes, loopback only	sslmode	sslrootcert

Evidence:

• TlsPaths { ca, cert, key } — all three are non-optional PathBuf (crates/vtop-node/src/config.rs).
• meta.tls and data.replica_tls are non-optional fields.
• data.native_tls is Option, but a leader/standalone rejects None at startup: .ok_or("leader/standalone requires native_tls")?. The Option exists only because a follower serves no clients — not to allow plaintext.
• grep -rniE "insecure|skip_verify|danger_accept|no_verify|accept_invalid" crates/ returns nothing. There is no verification toggle anywhere in the cluster planes.
• No production code path binds a cluster listener without TLS; every plaintext TcpListener::bind is in a test.

So: the four cluster planes are TLS-mandatory with no configuration surface, the observability plane is plaintext with no way to secure it, and only the object-store client has the knobs. That is the inverse of a coherent design, and it is scattered across 37 TLS construction sites.

One thing to decide before building, because it reverses a documented decision

mTLS-only is not an accident of implementation — it is a stated security property:

• docs/SECURITY_MODEL.md: "Native client ↔ broker | TLS 1.3 with mandatory client certificates"
• docs/SECURITY_MODEL.md: "The native broker transport in vtop-broker is TLS 1.3 mTLS only."

And the existing verify_tls flag deliberately refuses the "disable verification" case:

Honest scope: this flag does NOT disable certificate verification for https:// endpoints… A self-signed or private-CA endpoint needs its CA in the system trust store; skipping verification is deliberately unsupported.

That stance is defensible: on the cluster planes the certificate CN is the authorization identity — a metadata node id, a broker UUID, a client principal. Disabling verification there does not merely weaken confidentiality, it removes authentication and authorization, because there would be nothing left binding a connection to an identity. Fencing, lease ownership and admin authorization all rest on it.

So the request splits into two parts with very different risk:

1. Plaintext mode (TLS off entirely) — coherent, and worth having for local development, CI, and single-node evaluation. Needs an explicit, loud, refuse-by-default opt-in.
2. TLS on but verification off — on these planes this is not "TLS without checks", it is "authentication removed while looking encrypted". If it is wanted, it should be scoped narrowly (a named test-only mode that refuses to start unless the listener is loopback, or gated behind a build feature) rather than a general config flag, and SECURITY_MODEL.md has to be amended to say so.

My recommendation: build (1) properly, and for (2) support custom CA / private PKI as the answer to the real underlying need (self-signed certs in test environments), which the codebase already treats as the correct path. If a true no-verify mode is still wanted after that, do it as its own decision with the doc change attached.

Proposed design

One TransportSecurity type, resolved once from configuration and threaded through, replacing the scattered TlsPaths requirements:

mode: mtls | tls | plaintext        # default mtls; plaintext must be opt-in
ca:   <path>                        # custom/private CA, per plane
cert, key: <path>                   # required unless plaintext
client_auth: required | none        # `tls` mode = server auth only
verify: full | ca-only              # hostname verification; NOT a way to skip it

Non-negotiables that fall out of the audit:

• Refuse, never warn, on a downgrade the operator did not ask for. This is the pattern validate_endpoint_scheme already sets, and it is the right one.
• Plaintext must be visible at runtime: a startup warning naming the plane, and a metric or /readyz detail an operator can alert on. A cluster silently running plaintext is worse than one that refuses.
• A plaintext plane must refuse to talk to a TLS peer and vice versa, with an error naming both sides. Half-configured is the state that produces unexplainable handshake failures.
• Per plane, not global. Plaintext client access with mTLS replication is a legitimate shape; one global switch cannot express it.
• Observability gains the same surface so /metrics can be TLS where required. Today it cannot be, which is a real gap in the opposite direction (security: metrics endpoint spawns an unbounded task per connection #78 notes it is unauthenticated by design, but "no option" is different from "off by default").
• Environment-variable overrides for every field, matching the VTOP_S3_VERIFY_TLS precedent, so a container can be configured without rewriting a mounted file.

Deployment surfaces, all four

• Bare metal — the config file is the source of truth; document a plaintext single-node quickstart, since there is currently no way to run the engine at all without minting certificates.
• Docker / compose — docker-compose.yml does not deploy vtop-node at all today, so the cluster planes are unexercised there. A plaintext compose profile is the cheapest way to make the engine runnable in the lab.
• Helm — tls.metaSecretName / tls.dataSecretName are required and the chart refuses to render without them (security: lab compose binds to all interfaces with default credentials #81, deliberately). Under a plaintext mode they must become conditional, and the refusal must still fire whenever mode is not plaintext.
• Kubernetes — the per-pod certificate CN convention is load-bearing for identity; plaintext mode has to explicitly drop the SAN/CN requirements rather than leave them half-checked.

Acceptance

• TransportSecurity resolved from config + env, one type, no per-plane ad-hoc requirements
• plaintext runs a full cluster: metadata quorum, replication, produce/fetch, observability
• Mixed plaintext/TLS between peers is refused with an error naming both sides
• Custom CA honoured on every plane, verified against a private PKI in a test
• Helm renders and deploys both modes; k8s-smoke.sh covers plaintext as a variant
• Compose gains a runnable plaintext cluster profile
• SECURITY_MODEL.md states the modes, the defaults, and what each gives up
• Enabling or disabling TLS changes no other behaviour — the existing suites pass in both modes

Milestone v0.4.0 (hardening), since v0.3.0 is gated on the durability arc.

[allamiro]

Design, with the cost measured rather than estimated

Having read the whole surface, this is bigger than a config change and worth breaking up before anyone starts.

The obstacle is the stream type, not the configuration. Every plane does:

let stream = acceptor.accept(tcp).await?;      // server
let stream = connector.connect(name, tcp).await?;  // client

and then hands that concrete TlsStream to read_frame / write_frame. Making TLS optional means the thing passed to those becomes a MaybeTls<TcpStream> enum implementing AsyncRead/AsyncWrite, which touches every call site that names the stream type.

Measured: 37 acceptor/connector construction sites across 7 files —

crates/vtop-broker/src/lib.rs
crates/vtop-broker/src/replication/network.rs
crates/vtop-broker/src/replication/transfer.rs
crates/vtop-meta/src/transport/admin.rs
crates/vtop-meta/src/transport/peer.rs
crates/vtop-meta/src/transport/tls.rs
crates/vtop-node/src/client.rs

It is mechanical but wide, and half-doing it is worse than not starting: a plane that looks configurable and silently keeps requiring certificates is a worse bug than one that plainly requires them.

Slices, each independently shippable

1. MaybeTls stream seam in vtop-meta/src/transport, plus TransportSecurity resolved from config and environment. Prove it on the metadata admin plane only — the smallest complete vertical, and the one an operator hits first through vtopctl. Ends with: vtopctl meta status against a plaintext admin endpoint.
2. Metadata peer plane, so a Raft group forms without certificates. Ends with: a single-node and a three-node group in plaintext.
3. Replica plane, then native client plane in vtop-broker. Ends with: quorum durability with no TLS anywhere.
4. Observability TLS — the inverse gap. /metrics, /healthz, /readyz are plaintext-only today and cannot be secured at all, which security: metrics endpoint spawns an unbounded task per connection #78 notes is unauthenticated by design; "off by default" and "no option" are different things.
5. Deployment surfaces: helm (tls.metaSecretName is required and must become conditional, while still refusing whenever the mode is not plaintext), a runnable plaintext compose profile — docker-compose.yml does not deploy vtop-node at all today — and a bare-metal quickstart, since there is currently no way to start the engine without minting certificates.
6. Cross-mode refusal: a plaintext plane talking to a TLS peer must fail with an error naming both sides. This is the one that prevents the worst outcome, which is a half-configured cluster producing unattributable handshake errors.

Non-negotiables carried into every slice

• Refuse, never warn, on a downgrade nobody asked for. validate_endpoint_scheme in vtop-upload already sets this pattern and it is the right one.
• Plaintext must be visible at runtime — a startup warning naming the plane, and something in the observability surface an operator can alert on. A cluster silently running plaintext is worse than one that refuses to start.
• Per plane, not global. Plaintext client access with mTLS replication is a legitimate shape and one switch cannot express it.
• Environment overrides for every field, matching the VTOP_S3_VERIFY_TLS precedent, so a container is configurable without rewriting a mounted file.

The scope decision, restated so it is on the record

Building plaintext and custom CA. NOT building a verification-off mode on the cluster planes, unless someone explicitly asks for it with the SECURITY_MODEL.md change attached.

The reason is specific to this system rather than general security caution: on these planes the certificate CN is the authorization identity — a metadata node id, a broker UUID, a client principal. Fencing, lease ownership and admin authorization all read it. Disabling verification there does not weaken confidentiality, it removes authentication, and every one of those checks would then be reading an attacker-supplied string. Custom CA is the correct answer to the real underlying need, which is self-signed certificates in test environments, and the codebase already treats it that way.

[allamiro]

Slice 2 analysis: the peer plane is not the admin plane, and the difference is a safety decision

Slice 1 (#299) is done. Before writing slice 2 I read the peer plane's identity model, and it needs a decision made deliberately rather than by momentum.

The admin plane's identity is an authorization input. The peer plane's is a Raft safety input.

fn peer_id_from_server_stream(...) -> MetaNodeId   // certificate CN → node id

fn assert_sender_identity(peer_id: MetaNodeId, vote: &HardState) {
    match vote.voted_for {
        Some(claimed) if claimed == peer_id => Ok(()),
        Some(claimed) => Err("certificate CN maps to node {peer_id}, but request claims sender {claimed}"),
        None          => Err("peer request HardState is missing voted_for"),
    }
}

Every peer RPC's claimed sender is cross-checked against the certificate. Remove TLS and there is no certificate, so nothing binds the claim to anything — a peer can assert it is node 1, vote as node 1, and append entries as node 1. That is not a weaker channel, it is the removal of the check that makes Raft membership mean something.

So plaintext on this plane is a different proposition from plaintext on the admin plane, and "consistent across deployment methods" should not quietly mean "equally unauthenticated everywhere".

The options, with what each actually costs

1. Plaintext peer, loopback only. Safe by the same argument as slice 1 — the blast radius is the machine. But a multi-node Raft group is by definition not loopback, so this makes plaintext useful for a single-node dev cluster and useless for the three-node compose lab, which is a shape people will want.
2. Plaintext peer, explicit non-loopback opt-in (plaintext_on_any_interface, as slice 1 has). The sender claim becomes self-asserted and the cluster's integrity rests entirely on network isolation. This is what a plaintext etcd or Kafka dev cluster does, and it is defensible if it is stated, not discovered.
3. No plaintext on the peer plane at all. The metadata group always requires mTLS; plaintext covers the admin, client, and observability planes only. Costs the certificate-free multi-node dev cluster; keeps Raft identity a property nobody can turn off.

Recommendation

(2), with the same shape as slice 1: loopback by default, a separately-named constructor for anything routable, a startup warning naming the plane, and assert_sender_identity explicitly skipped with a comment saying the sender is UNVERIFIED rather than silently passing.

I am not going to implement it on momentum, because the failure mode is not a crash or a bad error message — it is a Raft group that works perfectly in testing and has no authenticated membership. That deserves an explicit yes rather than being carried along by "make TLS optional".

Slices 3–6 (replica, native, observability TLS, deployment surfaces) do not depend on this decision and can proceed either way. The native client plane has a similar but weaker version of the same question — the client principal is a certificate CN too — and the replica plane's fencing checks read the peer's identity, so both will want the same explicit treatment when they come.

[allamiro]

Decision: option (2) for the peer plane, and it is v0.4.0 rather than release-blocking

Taking the recommendation from the analysis above — loopback by default, with a separately-named constructor for anything routable, matching the shape slice 1 already shipped for the admin plane.

Concretely, when it is built:

• PeerServer::plaintext binds loopback only and refuses anything else at bind time, so a misconfigured deployment fails to start rather than serving until the first unwanted peer arrives.
• PeerServer::plaintext_on_any_interface exists for the trusted-segment and sidecar-mesh cases, and has to be typed deliberately.
• assert_sender_identity is explicitly skipped in plaintext with a comment saying the sender is UNVERIFIED, rather than silently passing. A reader must not be able to mistake "the check ran and agreed" for "there was nothing to check against".
• A startup warning naming the plane, every time, because silence makes "no policy" indistinguishable from a policy that happens to permit what is being observed.

The reason it is (2) and not (3): a certificate-free multi-node dev cluster is a real need — the compose lab wants exactly that shape — and refusing it outright pushes people toward disabling security in worse ways. The reason it is not (1): loopback-only would make the mode useless for the very shape people want it for.

Not release-blocking. TLS-optional is hardening (v0.4.0). v0.3.0 is the durability and repair arc, and slice 1 already landed the piece that mattered for operator ergonomics — vtopctl against a plaintext admin endpoint.