Repository navigation
Local identity rotation API for in-place consumer re-key (no overlap) #338
Description
Problem
Smalltalk needs an operator-triggered in-place Fabric identity rotation: st fleet transports rekey fabric should obtain a new local NodeID, re-expose its existing replication protocol, and publish the replacement endpoint without changing the Smalltalk member identity.
At Fabric main 97ba98678c580e455d85f96ed70bab6fd18c7ba0, src/main.rs exposes only key gen --out PATH for identity key management; src/control.rs has restart/recycle operations but no identity rotation. Generating a separate key file is not a daemon identity cutover. Smalltalk must not rewrite Fabric-owned identity files or guess managed-service restart semantics.
Remote policy replacement is already available: docs/runtime-peers.md describes one atomic batch removing the old owner record, adding the new NodeID, and setting exact grants. Smalltalk's membership adapter already emits these operations in one batch. This request is only the missing local identity rotation capability, not overlap/expiry support.
Smallest proposed surface — direction review requested
Add one Fabric-owned local command/control operation that durably generates and installs a new identity, restarts/rebinds the endpoint using that identity, and returns machine-readable old/new public NodeIDs after the new daemon identity is active. Command naming and response schema need Fabric owner agreement before Smalltalk consumes them. No secret key material should cross this interface.
Required properties:
- Preserve the resolved Fabric home, static/runtime peer policy and existing service declarations.
- Immediate replacement only; no old/new authorization overlap requested.
- Serialize against concurrent rotation/restart; no success while the live endpoint still uses the old NodeID.
- Return explicit failure/uncertain-outcome information so a caller cannot blindly rotate again after a lost reply.
- Keep key persistence, permissions, old-key retirement and managed-service behavior inside Fabric.
Open questions for Nathan
- What command/control request and JSON result should Smalltalk consume? Is a durable request ID or compare-against-old-NodeID guard appropriate for lost-reply recovery?
- Should rotation restart through the existing service-aware path, or can Fabric safely recreate its endpoint in-process? How does success prove the active NodeID?
- How should other Fabric consumers (sync author identity, durable sessions/dials and runtime receipts) behave across identity replacement? Which state is preserved and which must be explicitly retired?
- What is the failure contract after the new key is durable but before the new endpoint is serving, including process crash or managed-service restart failure?
Acceptance
- A fake-free Fabric integration test rotates a running isolated home, observes a different live NodeID and verifies persisted identity after restart.
- Existing peer policy and exposure declarations survive; the old endpoint is no longer serving after success.
- Failure/crash and concurrent-operation tests establish the agreed recovery contract.
- Published JSON/CLI contract lets Smalltalk implement its wrapper without reading or writing secret identity files.
Related: #312; Smalltalk transport ordering prerequisite compoundingtech/fractal#3642.
Posted on behalf of @schickling
| field | value |
|---|---|
agent_identity |
agent/dotfiles/fabric-mesh |
session |
01a1ecd5 |
agent_persona |
generalist |
agent_supervisor |
unavailable |
agent_tool |
OMP |
agent_tool_version |
18.9.1 |
agent_runtime |
OMP 18.9.1 |
agent_model |
anthropic/claude-opus-5-5 |
worktree |
smalltalk/2026-10-22-fleet-fabric-rekey-q37 |
Activity
schickling-assistant commentedon Oct 22, 2026
@myobie, a direction input for this issue from Fabric-mesh (
agent/dotfiles/fabric-mesh).Johannes decided on 2026-10-22 (axe q39) that an st member re-keys in place with a bounded overlap of at most 10 minutes. The reason: once Fabric is the only st carrier, an immediate cutover strands the machine. Peers can learn the new NodeID only over Fabric, and they do not trust the new NodeID yet.
The st side needs two Fabric pieces:
docs/runtime-peers.md159-162: install the new NodeID, and set an immutable expiry of at most 10 minutes on the old record.The st adapter already sends the NodeID replace as one batch (smalltalk#3644 draft,
reconcile_tests.rs:167), so the st side waits on the command contract for these two pieces. Please confirm or change this shape. Fabric-mesh can open the Fabric PR once you agree on the shape.myobie commentedon Oct 25, 2026
Direction for #338 (decided; open the PR against this).
Shape: prepare, publish over the old identity, activate. No dual-identity serving. A second live endpoint in the daemon is a large change and is not needed. The strand risk comes from activating before peers trust the new NodeID, so the order removes it: the member prepares a new identity, tells peers over the still-working old identity, peers install it, and only then does the machine switch.
fabric identity prepare [--json]generates the new key durably (0600, inside Fabric's home, never printed) and returns{"current":"<id>","next":"<id>"}. Idempotent: while a prepared key exists it returns the samenext. Nothing changes for the running daemon.nextwith the exact grants, set an immutable expiry of at most 10 minutes on the old record (the planned overlap batch indocs/runtime-peers.md). That expiry is a separate runtime-peers slice in this repo; open it as its own PR first (record fieldexpires_at, immutable once set, revocation receipts on expiry like any other revoke).fabric identity activate --expect-current <id> --expect-next <id> [--json]swaps the key by one atomic rename, restarts through the existing service-aware path, and returns only after the live daemon reportsnextas its NodeID:{"previous":"<id>","active":"<id>","outcome":"activated"}.fabric identity status [--json]returns{"current":"<id>","next":"<id>|null","state":"active|prepared"}.Lost reply / guard. The expect flags are the guard; no request IDs. If
currentequals the expectednext, activate returnsoutcome:"already_active"(success). Ifcurrentis neither, it fails withidentity_mismatchand changes nothing. Operations serialize behind one identity lock; concurrent callers getidentity_busy.Crash contract. The swap is a single rename, so after a crash the home is either
prepared(old key active, new key pending) oractive. The daemon starts from whichever the rename left;statustells the caller which. Restart failure after the rename leavesactivewith a daemon that is not serving: activate returns a non-zerorestart_failedwith both IDs, and repeating the same command retries the restart only. The old key is moved to a retired file and deleted when the old record's expiry has passed; the daemon never serves it again after activation.Other consumers (the PR must state and test each). Preserved: static and runtime peer policy, exposures, receipts and revocation history, audit log. Retired on activation: tunnel sessions, mux connections and queued dials bound to the old NodeID (peers see a new identity, so they re-establish). Sync author identity: if it is derived from the NodeID, document the behavior; if independent, it is unchanged.
Acceptance. Your list, plus: prepare is idempotent, activate lost-reply retry is a no-op success, crash between rename and restart recovers, and the old NodeID does not accept a connection after success.
schickling-assistant commentedon Oct 25, 2026
Implementation drafts are open bottom-up: #340 (immutable runtime-peer expiry) → #341 (prepare/publish/activate, no dual serving). #340 has independent correctness/security source-review GOs at 3578f43. #341 includes review fixes but still needs exact-head re-review. Both remain draft: targeted local execution could not find Cargo in the host environment, and hosted CI is still running. The two conservative open questions are recorded in #341; no merge or auto-merge is enabled.
Posted on behalf of @schickling
agent_identitysessionagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimeagent_modelworktreeschickling-assistant commentedon Oct 25, 2026
Implemented Nathan’s prepare → publish over the old identity → activate direction, with no dual-identity serving:
1b546240840cf6a06563ce9587919b18a1d5353e, base main): immutable ≤600-second runtime-peer expiry in the atomic add-next/exact-grants/expire-old batch; expiry uses normal revocation/cancellation receipts.96c1b08dfd3afd302aa2ff823d0d41fd96c5b4db, stacked on Expire runtime peer records through atomic overlap batches #340): durable private prepare/status, guarded single-rename activation, held daemon-startup lease, restart-only recovery, old transport/dial retirement, preserved policy/exposures/receipts/audit and NodeID-derived sync history transition.Both exact heads have independent correctness and security GOs. Gated local acceptance checks pass; #341 test and Nix CI are green, and #340 Nix/deterministic/public-text are green. Acceptance-test names, evidence, conservative decisions and the two B open questions are in the PR bodies.
#340 macOS CI is now green after two authorized failed-job reruns without changing either timing-flake test: initially
continuous_events_are_bounded_by_max_coalesce_window, then the knowna_peer_returns_after_minutes_and_hours_without_errors_or_a_helper. First rerun; final successful rerun. Both PRs are ready for bottom-up review. Nothing merged and no auto-merge enabled.Posted on behalf of @schickling
agent_identitysessionagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimeagent_modelworktreemyobie commentedon Oct 29, 2026
Implemented by #340 (expiry) and #341 (prepare/activate/status), both merged to main. Not in a release yet: 0.9.4 follows the small hardening PR requested on #341. Keeping this open until then.