Add self-healing TLS certificate lifecycle

[magiccodingman]

What changed

• separates certificate serving from renewal orchestration so Kestrel handshakes only read the in-memory certificate cache
• keeps still-valid certificates serving throughout the 30-day renewal window
• introduces explicit certificate inspection states for fresh, renewal-due, expired, missing, corrupt, wrong-host, missing-key, and not-yet-valid certificates
• automatically repairs expired, missing, corrupt, or mismatched PFX files without requiring an administrator to delete them
• makes per-host ACME issuance atomic single-flight and rechecks certificate state after acquiring the global issuance throttle
• replaces direct PFX overwrites with validated temporary writes and atomic replacement
• quarantines unreadable PFX files and continues into automatic issuance
• corrects HTTP-01 ordering so challenge content is published before preflight and validation, with cleanup in a finally block
• persists and rotates the self-signed fallback certificate instead of replacing its identity on every restart or allowing it to expire in memory
• tightens certificate-directory, PFX, and ACME account-key permissions on Unix
• replaces swallowed watcher exceptions with structured TLS lifecycle logs and retry state

Root cause

LiveCertProvider.TryLoadIssued treated certificates within 30 days of expiration as unusable. That caused TruthGate to immediately fall back to its self-signed certificate while renewal was still pending. If renewal failed once, an otherwise valid public certificate stopped being served early and an expired PFX could remain permanently stuck on disk.

The production evidence matches that failure mode: truthgate.io.pfx was issued August 17, 2025, expired November 15, 2025, and was never replaced, while new IPNS certificates were still issued successfully in 2026.

Recovery behavior

After this version is deployed, the existing expired truthgate.io.pfx is loaded and classified as expired, the HTTP listener is allowed to start, and the reconciliation worker immediately requests a replacement. A successful replacement is validated, atomically written, reloaded, and switched into the live SNI cache without another restart.

Valid certificates that are merely renewal-due remain live while the replacement attempt runs. Failed renewal attempts use bounded backoff and do not displace the currently valid certificate.

Validation

A dedicated .NET 9 test project and GitHub Actions workflow cover:

• serving a renewal-due certificate while renewal is in flight
• automatic replacement of an expired certificate
• one ACME order under 100 concurrent reconciliation requests
• quarantine and repair of corrupt certificate storage
• rejection of wrong-host certificates
• reloadability and cleanup of atomic PFX writes

GitHub Actions successfully restored and compiled the full web project and passed all 6 TLS lifecycle tests on .NET 9.

Deployment note

No manual deletion of /opt/truthgate/certs/truthgate.io.pfx should be required. The first post-deployment reconciliation should repair it automatically, assuming HTTP port 80 and DNS still allow Let’s Encrypt HTTP-01 validation.

[magiccodingman]

validated the fix and code.