Expire runtime peer records through atomic overlap batches

CHANGELOG.md

@@ -18,6 +18,17 @@ Deprecated, Fixed or Security.
 
 ### Added
 
+- **Provisional:** runtime peer records support immutable `expires_at` deadlines
+  within the next 600 seconds. Existing atomic batches can install a prepared
+  next NodeID's exact grants and schedule the old owner record's retirement.
+  Timed policy generations close expired access independently of blocked disk
+  writers, preserving static/other-owner grants; automatic removal reuses
+  admission/session cancellation, connection retirement and durable completion
+  receipts. Expiry reserves conflict-checked receipt IDs and capacity, fences
+  stale sweep generations, survives restart, and uses a re-anchored progressing
+  monotonic floor across forward wall steps and rollback. Runtime state that
+  has used expiry cannot be
+  downgraded to strict readers lacking these fields.
 - **Provisional:** opt-in pairing permit issue/query/cancel control requests,
   with private durable state, immutable exact policy with a versioned digest,
   bounded lifetime and reserved terminal receipt capacity (1,024 globally,

docs/runtime-peer-api.md

@@ -269,11 +269,14 @@ The `operation` field selects one of:
 | `list` | Optional `cursor` from the preceding page. |
 | `add_peer` | `owner`, `request_id`, `expected_revision`, `node_id`; optional `name`, `direction`, `addr`. |
 | `set_grants` | `owner`, `request_id`, `expected_revision`, `node_id`, `grants`. |
+| `set_expiry` | `owner`, `request_id`, `expected_revision`, `node_id`, `expires_at`. |
 | `remove_peer` | `owner`, `request_id`, `expected_revision`, `node_id`. |
 | `batch` | `owner`, `request_id`, `expected_revision`, `operations`. |
 | `cancellation` | `request_id` of a retained mutation receipt. |
 
 An owner and request ID each contain 1–128 bytes without control characters.
+The `expiry:` request-ID prefix is system-reserved: owner mutations cannot use
+it, including before a deadline is scheduled or after its receipt is retained.
 A new record starts with no grants and normal direction. An omitted name,
 direction, or address preserves that owner's existing value. `set_grants`
 replaces the entire owner's grant set; `[]` withdraws all services while
@@ -312,6 +315,69 @@ an intermediate identity or unintended dual access:
 Static and other-owner records on the old NodeID remain visible and effective;
 the caller must inspect the result before claiming complete retirement.
 
+### Bounded overlap expiry
+
+To publish a prepared replacement over the still-working old identity, use one
+batch with `add_peer` for the next NodeID, `set_grants` for its **exact** grants,
+and `set_expiry` for the old owner's record. For example, replace the immediate
+removal in the example above with:
+
+```json
+{"op":"set_expiry","node_id":"<OLD_NODE_ID>","expires_at":1792887000}
+```
+
+The integer is illustrative: use an absolute UTC Unix-seconds deadline strictly
+after the daemon's current observation and no more than 600 seconds ahead.
+The signed 64-bit wire/storage representation rejects overflow; zero, negative,
+past and over-limit deadlines return `invalid_argument` without committing any
+part of the batch. The old and next NodeIDs must use distinct labels during
+overlap. This grants overlap on the receiving peer only, not dual-identity
+serving by the rotating daemon.
+
+The owner's record starts with no expiry. Once set, `expires_at` cannot be
+cleared, shortened or extended, including by updating grants, re-adding the
+same existing record, or replaying an old request. An identical setter is a
+no-op. Explicit removal and re-addition creates a new record; it is not an
+implicit renewal. Static and other-owner access is never modified.
+
+Runtime entries in `trust_sources`, `direction_sources`, and service provenance
+include `expires_at` and `expiry_request_id` when set. Save `expiry_request_id`
+from the scheduling reply/list before expiry; query it with `cancellation` to
+observe automatic removal, residual access, selected work and completion. The
+reserved ID belongs to that scheduling transaction, so a separately re-created
+record cannot replay an earlier expiry receipt. Until cleanup commits, its
+cancellation query returns an unknown receipt, never a premature completion.
+Reservation conflicts with retained receipt IDs return `request_id_conflict`
+before any part of the scheduling batch commits.
+
+Authorization and attach select a prevalidated effective generation at the
+deadline, without disk I/O or waiting for the mutation sequencer. A separate
+50 ms deadline sweep uses the same admission/session cancellation and connection
+retirement paths as removal. The serialized 100 ms cleanup sweep writes the
+owner-scoped removal and advances the policy revision; idle ticks do not fsync.
+Expiry withdrawals are not blocked by stale sync selectors. A blocked writer
+cannot extend expired authorization or live work; residual grants preserve
+still-permitted work. Cancellation monitoring retains its two-second deadline
+and missed-bound evidence even when the automatic receipt is persisted later.
+Each automatic receipt names only cancelled work for its own expiring NodeID;
+other records expiring in the same sweep retain their own cancellation evidence.
+Sweep selection and signaling share an in-memory publication-generation guard;
+they cannot revoke against a stale snapshot after newer residual grants publish.
+This guard never covers filesystem work.
+
+Running deadline observations combine wall and monotonic time and never move
+backward, including reload reconciliation. Each forward wall observation anchors
+a progressing monotonic floor, so rollback after scheduling cannot extend the
+600-second elapsed bound. Restart removes elapsed records durably before
+accepting traffic. A clock below the persisted last expiry
+observation conservatively expires outstanding records. Arbitrary rollback
+across power loss is not assumed detectable. Cleanup write failures cannot
+restore expired access; uncertain commits use ordinary policy quarantine.
+
+Homes without expiry keep the old representation. Once expiry is used, its
+record fields and retained clock anchor make runtime state incompatible with
+older strict readers lacking this feature.
+
 ## Revisions, persistence, and lost replies
 
 Shutdown allows up to three seconds for in-flight policy/config writes to drain,
@@ -354,9 +420,11 @@ request; do not invent a new request ID for an uncertain outcome.
 The store permits at most 1,024 owner/NodeID records in total. Receipts are
 retained for at least 24 hours in a pool of 1,024 slots. Ordinary mutations may
 use 960 slots; the remaining 64 are reserved for batches containing only
-`remove_peer` or `set_grants` that remove services without adding any. Pure
-no-ops consume no slot, even when the pool is full. Expired receipts are pruned
-by a subsequent changed mutation. When the applicable capacity is full, that
+`remove_peer` or `set_grants` that remove services without adding any.
+Scheduling with `set_expiry` uses ordinary capacity and also reserves a future automatic receipt slot; all
+mutations account for those reservations before committing. Pure no-ops consume
+no slot, even when the pool is full. Expired receipts are pruned by a subsequent
+changed mutation. When the applicable capacity is full, that
 mutation is refused before commit. After retention, reconcile current state
 and revision before deciding on a new operation.
 

docs/runtime-peers.md

@@ -2,9 +2,9 @@
 
 Status: approved design. Runtime mode, direct peer/grant transactions, revisions,
 and revocation receipts are implemented as a provisional API; see the
-[runtime peer API](runtime-peer-api.md). Rotation expiry, policy watches,
-pairing and the typed client are follow-up slices whose request names and
-limits below remain proposals. The surface remains provisional through at
+[runtime peer API](runtime-peer-api.md). Immutable rotation expiry and pairing
+are implemented; policy watches and the typed client remain follow-up slices.
+The surface remains provisional through at
 least one minor release of real use, with an explicit review before promotion.
 This note covers [#312](https://github.com/compoundingtech/fabric/issues/312)
 and its typed control client, [#310](https://github.com/compoundingtech/fabric/issues/310).
@@ -112,7 +112,7 @@ static or other-owner access to that NodeID is reported by the same rule.
 
 ## Transactions, replacement, and expiry
 
-The proposed typed operations are `add_peer`, `set_grants`, `remove_peer`,
+The typed operations are `add_peer`, `set_grants`, `set_expiry`, `remove_peer`,
 `list`, and `batch`. `set_grants` replaces that owner's entire set; `[]`
 withdraws it. `add_peer` for a new ID starts with zero grants. Renaming an
 existing ID preserves its grants. A batch has one owner, a fresh request ID,
@@ -163,28 +163,55 @@ reinstalling access; the client must reconcile instead of inventing a new ID
 for an uncertain request. No-op mutations return the current revision without
 storing a receipt.
 
-Rotation defaults to immediate replacement. Explicit overlap uses a batch that
-installs the new record and sets an immutable expiry on the old owner's record,
-with a maximum overlap of 10 minutes. Overlap cannot silently extend on retry;
-a further change requires a new explicit transaction. The old ID retains a
-distinct label during overlap. Admission and session attach enforce the expiry
-even if the cleanup timer or disk writer is late. The timer removes the expired
-record and advances policy revision. Admissions and sessions that rely on an
-expiring record also carry its deadline: they recheck the remaining effective
-grants and signal cancellation independently of the persistence queue. A stalled
-cleanup writer cannot extend live work that lost its last grant. The same
-2-second completion target below is measured from the expiry deadline for
-automatic withdrawal; a missed bound is reported. Persistence failure cannot
-keep expired access live. Restart filters expired records before
-admitting traffic and durably records their removal before mutation success.
-
-Expiry stores an absolute UTC deadline and uses a monotonic deadline while
-running, taking the earlier limit so a backward wall-clock step cannot extend
-access. Recovery needs a trustworthy clock; detected rollback or uncertain
-time closes expiring access and unconfirmed permits instead of renewing them.
-This bound does not assume arbitrary clock rollback can be measured across a
-power loss. Static or other-owner grants have their own provenance and are
-never removed by an overlap expiry.
+Rotation defaults to immediate replacement. Explicit overlap uses the existing
+atomic batch to add the new NodeID, set its exact grants, and `set_expiry` on
+the old owner's record. This changes peer policy only: the rotating machine
+prepares its key, publishes that batch over its still-working old identity, and
+then activates its new identity. Fabric never needs to serve both identities.
+The old ID retains a distinct label during overlap.
+
+`expires_at` is an absolute UTC Unix-seconds signed 64-bit integer, strictly
+in the future and at most 600 seconds away when first set. Negative, zero,
+out-of-range and later deadlines are refused atomically. The deadline is
+immutable for that owner/NodeID record, including through `add_peer` and
+`set_grants`; an identical setter is a no-op and a historical batch retry
+cannot renew or reinstall access. Removing and explicitly re-adding a record
+creates a new record. Runtime source provenance reports `expires_at` and the
+reserved `expiry_request_id` used to query automatic cancellation.
+
+Policy generations precompute the effective union at each distinct deadline.
+Their shared wall/monotonic observation never moves backward during one daemon
+lifetime, including reconciliation. Every forward observation re-anchors a
+progressing monotonic floor: a later wall rollback cannot freeze the clock or
+extend a newly installed overlap beyond 600 elapsed seconds.
+Admission, service authorization, outbound
+policy lookup and session attach use the applicable generation without waiting
+for a timer, mutation sequencer or disk writer. The deadline sweep independently
+signals the existing admission/session revocation path, including connection
+retirement, while the serialized writer records owner-scoped removal and
+advances policy revision. Still-permitted static and other-owner work survives.
+Selection and signaling hold a separate in-memory generation read guard;
+publication fences it only after durability, never while waiting on disk.
+A sweep cannot cancel work against a superseded policy lacking newer grants.
+The two-second cancellation target is measured from expiry, not a delayed
+cleanup commit; timeout evidence survives through the automatic receipt.
+
+Setting expiry reserves an automatic receipt slot in the existing bounded pool.
+Generated receipt reservations are checked against retained receipts and the
+final candidate before committing any scheduling operation; collision refuses
+the complete batch rather than poisoning automatic cleanup or restart.
+An idle sweep performs no disk writes. Failed cleanup persistence cannot
+reopen expired access; indeterminate outcomes use ordinary quarantine and reload
+reconciliation. Cleanup withdrawals cannot be blocked by stale sync selectors.
+Restart filters and durably removes elapsed records before admitting traffic,
+with complete empty cancellation receipts because no work from the new daemon
+was admitted. The persisted expiry clock detects rollback below its last
+observation and conservatively expires outstanding records. This does not claim
+that arbitrary clock rollback across a power loss can be measured.
+
+Expiry fields and the clock anchor are omitted until expiry is used. Existing
+state without expiry remains readable by older strict readers; state that has
+used expiry is not downgrade-compatible with releases lacking this slice.
 
 ## Revision and watch contract
 

src/admission.rs

@@ -120,6 +120,7 @@ impl Admissions {
             })
             .map(|(id, entry)| Revocation {
                 id: *id,
+                peer: entry.peer,
                 revoked: entry.revoked.clone(),
                 done: entry.done.clone(),
                 children: entry.children.clone(),
@@ -178,6 +179,7 @@ impl Requirements {
 #[derive(Debug, Clone)]
 pub(crate) struct Revocation {
     pub id: u64,
+    pub peer: EndpointId,
     revoked: CancellationToken,
     done: CancellationToken,
     children: fabric_service_api::child_tracking::Children,