Ship Keldra 0.16.1 with three-node recovery fixes

[zcourts]

Outcome

Prepares Keldra 0.16.1 as a fresh-volume corrective release. It keeps the measured single-lane ingest/index path, removes the failed speculative multi-lane/pipelined implementation from the final tree, and closes the security, recovery, catalog, cluster-growth and failover defects exposed while qualifying 0.16.0.

The high-level production architecture and authority boundaries are recorded in docs/rfcs/arch.md.

Cluster lifecycle

• fresh clusters record protocol/storage capability 2/2 regardless of node count;
• a joining node becomes publicly usable after Raft acknowledges membership;
• gateway operations route through the existing peer APIs;
• stale placement or atomic-executor responses refresh placement and retry with the original request identity/deadline;
• migration and replica activation continue in the background;
• no proxy authority, readiness subsystem, or additional public gate was introduced.

Correctness and security

• binds every current-object quorum observation to the requested stable tenant ID, bucket ID and exact path before quorum selection, preventing cross-object/cross-tenant peer substitution;
• recovers alias-expanded source-journal entries from the canonical mutation quorum proof after interruption between metadata commit and direct settlement, including page boundaries and restart;
• retries atomic finalization after bounded reference backlog rather than exposing a local-reference-cursor/source-journal-tail mismatch;
• treats stale local and remote atomic-executor results, including stale executor lookup, as refresh-and-retry conditions for public calls while preserving peer-routed single-hop behavior;
• makes stale journal settlement idempotent and preserves ordered reference progress at journal capacity;
• keeps the approved clean 0.16 storage boundary; compatibility with earlier on-disk formats is not claimed.

Index catalog correction

• fixes physical TypedJson families assembled from several valid logical definitions;
• assigns temporary dense field IDs to the fingerprint-ordered union before existing canonicalization;
• preserves public definitions and the durable format;
• directly covers the Worka case where several JSON definitions share one tenant/bucket/source scope.

Three-node durability, growth and failover

• fixes learner joins after the Raft leader has purged old logs;
• allows fresh authorization, credential and logical-name reads to fail over to any selected quorum replica;
• authorizes realm read/repair RPCs between selected replicas while retaining rank-zero mutation coordination;
• routes joining-node gateway mutations using stable identity;
• propagates BulkWrite deadlines through peer routes;
• deduplicates distributed payload placement and mutation reconciliation;
• preserves typed authorization handoff frames and destination errors;
• reconciles local retention during handoff while excluding replica-local retention from quorum authority;
• retains full release qualification for large replicated payloads, growth from one to three nodes, every single-node outage, rolling restart, atomic programs, accounting, PersonalDB, S3, Git and CAS.

Qualification and release surface

• adds the manually dispatched GitHub three-node workflow, pinned to an immutable release tag and exact published GHCR image;
• release scripts now verify that fresh clusters are already active at 2/2 and no longer perform obsolete capability activation;
• test fixtures advertise the production binary capability range;
• captures retained journal/handoff evidence and preserves source-version, image-identity and file-size checks;
• updates the Keldra API/client and pinned Zanzibar 0.4.1 compatibility source to exact Keldra 0.16.1.

Performance lineage

• keeps the successful single-lane publication improvements: batched derived artifact staging, removal of redundant local replica replay, grouped source-journal settlement and single-pass component-delta lookup;
• retains low-overhead cumulative diagnostic counters;
• preserves Git history for the unsuccessful preparation-lane/pipeline experiment, while the final production tree contains none of that implementation;
• the retained single-lane SSD baseline is approximately 4,499–4,524 accepted ingest operations/second with continuous index progress.

Validation

Passed for exact final candidate b671e7abb1c4b2706532815d2b2be48b0eca7b31:

• static release checks and source-size limits;
• complete Keldra workspace tests;
• native amd64 image build and gRPC smoke;
• native arm64 image build and gRPC smoke;
• focused current-snapshot identity tests;
• focused three-node alias crash/restart recovery tests (4/4);
• focused TypedJson physical-family regression;
• focused public atomic-executor refresh/retry tests.

Exact CI: run 33852522246.

Still required before deployment-qualified publication:

• full local three-node release qualification against the exact b671 image;
• merge/tag publication at the same validated commit;
• GitHub three-node qualification against the immutable published 0.16.1 tag/image;
• crates.io publication and verification for keldra-api 0.16.1, keldra 0.16.1, then zanzibar 0.4.1.

Only failures of the required release contract block this PR. Bounded non-release limitations are tracked separately in #266 and #267.