Report registrar certificate lifetimes and classify lapses

[sehkone]

Report registrar certificate lifetimes and classify lapses

Context

The registrar endpoint depends on two certificate/key pairs: the registrar client leaf used by callers and the endpoint server leaf presented by the daemon. A failed renewal must be visible before either leaf expires, and a leaf that has expired must not be mistaken for a transient connection failure that an automatic retry can repair.

On origin/main, the daemon already prepares and runs renewal for both leaves. During preparation, RegistrarCertRenewalState::initialize observes each enabled leaf's not_after from the certificate already on disk, records never_attempted with no attempt timestamp, and refuses preparation if either leaf cannot be observed. Consequently, an enabled endpoint cannot serve without the two seeded entries; this seed is not a renewal attempt and contacts neither OpenBao nor the CA. Later renewal-loop ticks own changes to the observed expiry, outcome, and attempt timestamp, and the accessor exposes those values without a request-time certificate read. A successful server renewal also swaps the active TLS configuration without restarting the endpoint. The endpoint client already reloads its client certificate/key pair for every dial, but deliberately does not inspect the leaf expiry; an expired client leaf or an expiry rejection during the TLS handshake therefore still reaches callers as an undifferentiated material or handshake failure.

The endpoint already serializes a daemon-owned RegistrarHealth snapshot on the response shapes that carry registrar health. This issue extends that existing snapshot; it does not change which responses carry it or the existing limiter and audit-capacity members.

Scope

• Add a certificates member to RegistrarHealth containing exactly two entries, in this order: registrar_client and endpoint_server. Each entry contains leaf, not_after, remaining_seconds, last_renewal_outcome, and last_renewal_at. leaf is the closed string enum registrar_client or endpoint_server. last_renewal_outcome is the closed string enum never_attempted, succeeded, or failed. Timestamps are RFC 3339 UTC strings. last_renewal_at is omitted, not null, when the outcome is never_attempted and is present for succeeded and failed.
• remaining_seconds is a signed whole-second value calculated as floor(not_after - now) over the exact fractional duration. Compute that floor explicitly with div_euclid over a sub-second representation; do not use a whole-seconds accessor or a conversion that truncates a negative duration toward zero. At now == not_after, it is 0 and the leaf is not yet lapsed. Strictly after expiry it is negative, including -1 when the leaf expired by less than one second; before expiry it never overstates the time remaining.
• certificates is present with both seeded entries in the first response carrying the populated RegistrarHealth snapshot, before the first daemon maintenance tick completes. Until a real renewal attempt, each entry carries the observed not_after, never_attempted, and an omitted last_renewal_at; do not fabricate an attempt or omit either entry. This uses the existing preparation seed, not delayed endpoint readiness or an explicit initial renewal refresh.
• Append certificates after audit_capacity in RegistrarHealth. The complete encoded member order is limiter, audit_capacity, certificates; update the ordered wire example and serialization fixtures to pin that sequence.
• On the daemon's existing health-maintenance cadence, copy both entries from RegistrarCertRenewalState into the shared RegistrarHealth snapshot and calculate remaining_seconds from that tick's clock. Responses must serialize only that snapshot and must not read, stat, or parse certificate files on the request path.
• Add the certificates schema to the bootroot-owned portion of the checked-in registrar wire reference and extend the existing serialization fixtures additively wherever the non-empty RegistrarHealth snapshot is already encoded. Preserve the order and wire representation of existing health members.
• Introduce one typed endpoint-client lapse error, carrying which local role lapsed. A caller must detect an expired registrar client leaf while loading its per-dial material and return that error before opening the Unix socket. When TLS verification identifies expiry of the endpoint server leaf, classify that handshake failure as the same typed lapse error for endpoint_server rather than exposing a generic handshake error. The endpoint's pinned verifier does not distinguish that verdict today: an expired presented leaf reaches the client as the same generic chain failure an unpinned peer produces, and the only CertificateError::Expired the verifier emits is the pinned anchor's own lapse, whose repair is the caller's pin file rather than the daemon's renewal. So the verifier must report the presented leaf's expiry as a rejection of its own, and the anchor's lapse must stop sharing that spelling, or the client cannot tell the two apart. The anchor's lapse remains a pin refusal. The error is non-retryable: retrying with unchanged material cannot repair it, and its diagnostic must direct the operator to restore the registrar daemon's certificate maintenance rather than re-provisioning the host.
• Preserve the observable precedence implied by the call sequence: a lapsed registrar client leaf is reported before a dial; an unavailable daemon is reported only when the client material is valid enough to reach the connect step; a server-leaf lapse is reported only after a connection reaches TLS verification.
• Update both localized operator manuals, docs/en/operations.md and docs/ko/operations.md, to explain the health signal, the typed lapse diagnosis, and the recovery distinction: starting a stopped daemon repairs an already-expired leaf before endpoint startup, while a still-valid leaf with a failed renewal needs the running daemon's subsequent renewal work.

Acceptance criteria

• Every enabled registrar endpoint response that currently carries the populated RegistrarHealth snapshot includes certificates with the two entries and exact ordering and field-presence rules specified above; existing health members and response placement are unchanged.
• The first such response, before any daemon maintenance tick, carries the two preparation-seeded entries with observed not_after, never_attempted, and no last_renewal_at.
• RegistrarHealth serializes its top-level members in the full fixed order limiter, audit_capacity, certificates. The checked-in ordered wire example and golden serialization fixtures pin that order.
• remaining_seconds equals the floor of the exact not_after - now duration: it is 0 at expiry, -1 when expired by less than one second, and strictly negative at every instant after expiry.
• The certificates values reflect the renewal state for both leaves, including a failed renewal while the affected leaf remains valid.
• Serving a registrar request performs no certificate filesystem read or parsing to produce certificates; the daemon refreshes the shared snapshot from RegistrarCertRenewalState on its maintenance cadence.
• The local registrar wire reference and golden serialization fixtures pin the member names, enum spellings, timestamp representation, signed remaining_seconds value, fixed entry order, and omission of last_renewal_at for never_attempted.
• An expired registrar client leaf returns the typed lapse error before any socket connection is attempted.
• A TLS expiry verdict for the endpoint server leaf returns the typed lapse error for endpoint_server rather than a generic handshake error.
• The typed lapse error is distinct from retryable connection failures and identifies certificate maintenance on the registrar daemon as the recovery path; no wire error identifier, wire refusal class, or externally defined unavailability reason is added, renamed, or reclassified.
• Both docs/en/operations.md and docs/ko/operations.md describe pre-expiry reporting and post-expiry recovery without recommending host re-provisioning.
• cargo fmt -- --check --config group_imports=StdExternalCrate, cargo clippy --all-targets -- -D warnings, and scripts/preflight/run-all.sh pass. If the E2E matrix cannot complete locally because passwordless sudo is unavailable, run it as far as possible and record the skipped arm and reason for CI to gate.

Constraints

• Reuse the existing renewal-state accessor and shared RegistrarHealth holder. Do not create a separate certificate-health cache or derive certificate lifetime on the request path.
• The preparation seed is the sole pre-attempt state: it observes the certificates already needed for mTLS, contacts neither OpenBao nor the CA, and must make both entries available before serving. Do not delay endpoint readiness, trigger an explicit initial refresh, or turn the seed into a renewal attempt.
• After preparation, the renewal tick is the sole writer of changed observed expiry, outcome, and attempt timestamp values; this issue reads all three through the accessor without filesystem access and must not re-derive any of them.
• Calculate remaining_seconds with an explicit mathematical floor over sub-second precision. Do not use time::Duration::whole_seconds or any equivalent truncation-toward-zero operation for negative durations.
• Append certificates after audit_capacity; do not reorder, remove, or otherwise alter limiter or audit_capacity.
• Do not issue, renew, publish, reload, or repair certificates from reporting or client error handling. Existing renewal and startup-repair behavior is standing functionality.
• Do not change the placement of RegistrarHealth or modify the transcribed portion of the protocol reference.
• Do not broaden this work into a redesign of transport, codec, pin, or other TLS-error classification. Non-expiry errors retain their existing behavior.
• Do not test by starting an endpoint with expired server material: startup renewal repairs expired material before the endpoint loads. Exercise the server-lapse classification through the existing active-TLS swap seam or a focused verifier-error test.

Out of scope

• Certificate issuance, renewal scheduling, publication rollback, and the active TLS reload contract.
• Changes to the endpoint health container's response placement, limiter data, audit-capacity data, or any other health member.
• New wire identifiers, changes to the external control plane's unavailable-reason mapping, and client behavior outside this repository.

Test plan

• Add unit and serialization tests for the certificates member's complete shape, fixed entry order, enum spellings, optional timestamp rule, and complete top-level registrar_health order: limiter, audit_capacity, certificates.
• Pin remaining_seconds fixtures at exact expiry (0), less than one second after expiry (-1), and a fractional live lifetime that demonstrates the value is floored rather than rounded up.
• Exercise renewal-state preparation and the first populated health response before any maintenance tick; verify it exposes both observed entries as never_attempted without an attempt timestamp.
• Exercise a failed renewal while valid, then verify the next health refresh exposes the correct two entries without request-time file access.
• Update the ordered registrar_health wire example and golden fixtures to encode certificates after audit_capacity without changing the existing member bytes or response placement.
• Use an expired local client certificate pair and a listener that records connection attempts to prove the typed registrar_client lapse error is returned before dialing.
• Exercise endpoint-server expiry through the active TLS swap seam or a focused TLS verification error and assert the typed endpoint_server lapse variant rather than the generic handshake variant.
• Run cargo fmt -- --check --config group_imports=StdExternalCrate, cargo clippy --all-targets -- -D warnings, and ./scripts/check-docs.sh. Run scripts/preflight/run-all.sh, including scripts/preflight/ci/e2e-matrix.sh. If passwordless sudo prevents matrix step 13, report the local arm that passed, the arm that could not run, and why; CI remains the gate for the unavailable arm.

Dependencies

Part of #770. This work relies on the existing RegistrarCertRenewalState accessor for both leaves and the existing daemon-owned RegistrarHealth snapshot. Preparation seeds the accessor with each observed expiry, never_attempted, and no timestamp before an enabled endpoint serves. Apart from that one non-attempt seed, the renewal tick writes the observed expiry, outcome, and attempt timestamp; all three are readable through the accessor without touching the filesystem, and this issue must not re-derive any of them.

Pointers

• src/registrar_renewal.rs: RegistrarCertRenewalState, LeafRenewalState, and preparation
• src/daemon.rs: registrar service composition, RegistrarHealth construction, and maintenance refresh
• src/registrar/endpoint/protocol.rs: RegistrarHealth and response serialization
• src/registrar/endpoint/client.rs: per-dial material loading and handshake classification
• src/registrar/endpoint_pin.rs: EndpointVerifyRejection and its rustls::Error mapping, where the presented leaf's expiry has to become distinguishable from the pinned anchor's
• src/registrar_certs.rs: SurfaceLeaf, the existing two-leaf enum the leaf member and the lapse diagnostic both name a leaf with
• src/registrar/endpoint/tls.rs: active TLS swap seam
• docs/reference/registrar-wire-contract.md, docs/reference/registrar-endpoint-client.md, docs/en/operations.md, and docs/ko/operations.md

[AcoPiper]

Verification summary

• Delivery: COMPLETE — the merged code delivers what the issue asked for. (the auditor's own verdict)
• Extra changes: NO_EXTRA_CHANGES — the merged code changed nothing beyond what the issue asked for. (the auditor's own verdict)
• Defects: NO_DEFECT — the merged pull requests left no defect of their own behind. (the auditor's own verdict)

Delivery decides whether this issue is closed or reopened. Defects widens what the follow-up filing considers. Extra changes gates nothing — it is an observation for a human; the report names what went beyond the issue.

Implementing pull requests, at the merge commits this audit read:

• Report registrar certificate lifetimes and classify lapses (#769) #970 — f67f3ec2920825248bcf8d26efa27970fa67e443 (the tree the audit read)

The full report — the assessment included verbatim — stays local, at:
~/.agentcoop/runs/aicers/bootroot/verify/NzY5/report.md

Findings this run did not file:

Finding not filed — The supplied evidence could not verify this finding.
Summary (the filing turn's statement):
The target-specific cause and affected test of the failed Unit & CLI Smoke check are unverified.
Requirement (quotation from the target issue):
`cargo fmt -- --check --config group_imports=StdExternalCrate`, `cargo clippy --all-targets -- -D warnings`, and `scripts/preflight/run-all.sh` pass.
Missing evidence (the filing turn's statement):
The failed Unit & CLI Smoke job log, failing test name, and output for f67f3ec2920825248bcf8d26efa27970fa67e443 were not supplied.

Finding not filed — The supplied evidence could not verify this finding.
Summary (the filing turn's statement):
Successful local execution of scripts/preflight/run-all.sh, including its E2E matrix invocation, is unverified.
Requirement (quotation from the target issue):
`cargo fmt -- --check --config group_imports=StdExternalCrate`, `cargo clippy --all-targets -- -D warnings`, and `scripts/preflight/run-all.sh` pass.
Missing evidence (the filing turn's statement):
No local scripts/preflight/run-all.sh output was supplied; the checked-in runner invokes the matrix at scripts/preflight/run-all.sh:32.