Skip to content

refactor(release): add pre-tag production readiness, single promotion entrypoint, shared executor and durable deployment ledger #167

Description

@balejosg

Summary

Follow-up to #160 and #162 after the production path was proven successfully with v1.2.379.

The core release architecture is now considered correct and operationally proven:

build once
identify once
test exact release
promote exact release
rollback exact previous release

This issue does not reopen #162 and does not introduce another round of generic release hardening.

The remaining goal is operational simplification:

make it harder for an operator to start an invalid promotion
make staging exercise the same deploy executor as production
make promotion creation reproducible
make historical deployment identity trivial to recover

The work is intentionally bounded to four improvements:

  1. pre-tag production-readiness preflight;
  2. one canonical promotion entrypoint;
  3. staging/production convergence on the same hermetic executor;
  4. durable bounded deployment ledger.

No blue/green, Kubernetes, GitOps migration, new state machine or automatic rollback selection is required.


Context

#160 established immutable Release Bundle v2 and exact release identity.

#162 established and operationally proved:

  • explicit minimal production host contract;
  • no host Node/npm dependency in the critical path;
  • hermetic verifier/runtime;
  • durable transactional deployment state;
  • explicit mutation boundary;
  • exact runtime projection;
  • live identity validation before commit;
  • stable recovery authority R;
  • deterministic rollback in isolated K;
  • post-boundary mandatory recovery;
  • always-run diagnostics;
  • fail-closed production promotion.

The successful production promotion of v1.2.379 proved the normal production path end-to-end.

Do not weaken or redesign those invariants.

Operational friction still exists:

operator creates production tag
        ↓
workflow discovers missing/invalid promotion prerequisites
        ↓
production remains safe
BUT
immutable tag/version has already been consumed

A concrete example was a promotion that failed pre-boundary because PRODUCTION_RECOVERY_SHA was not configured. Another pre-boundary failure exposed a workflow-only recovery bundle assertion defect.

Both failed safely, but both should be discoverable before creating the production tag.

The current workflow also intentionally has an asymmetry:

staging    -> local npm run deploy:staging path
production -> tag-only GitHub Actions + hermetic production executor

Now that the production executor is proven, staging should converge on that same executor semantics so normal release traffic continuously exercises the production path.


Core invariants

Invariant 1 — no tag before production readiness

A production tag must not be created until a non-mutating readiness check proves all prerequisites required to start the production promotion.

Conceptually:

RC selected
  -> exact release identity resolved
  -> production readiness preflight
  -> operator sees exact C/R/release identity
  -> only then production tag may be created

A failed readiness preflight must leave:

no production mutation
no production tag
no new release candidate

Invariant 2 — promotion remains exact and immutable

The preflight/promotion helper must not select latest, rebuild artifacts, recompute OpenPath pins independently or choose an alternative release.

It must consume one exact RC and resolve exactly:

C
RC_RUN_ID
RELEASE_ID
OpenPath SHA
OpenPath contract SHA256
OCI digests
Windows template tuple where present

The annotated tag identity must match those exact values.

Invariant 3 — production remains tag-only

Do not change production deployment into a branch push, manual mutable checkout or direct SSH command.

The canonical promotion entrypoint may create the annotated tag, but the production deploy itself must remain triggered by the immutable v* tag and use the existing #160/#162 semantics.

Invariant 4 — R remains explicit authority

Do not infer recovery authority from:

previous tag
previous main
latest known-good
candidate parent

PRODUCTION_RECOVERY_SHA remains an explicit independently validated authority.

The readiness preflight must prove it before tag creation.

Invariant 5 — staging and production share executor semantics

Staging and production should consume the same canonical release payload and the same hermetic deployment executor/state semantics.

Environment-specific differences may include:

host
credentials
URLs
compose/project identity
production authorization
fault-injection allowance
retention/evidence policy

They must not maintain separate implementations of release projection, switch, verification or transaction semantics.

Invariant 6 — deployment evidence is durable but bounded

Persist a small safe deployment ledger sufficient to answer:

What exact release was deployed?
From which RC/tag?
What was previous?
Which recovery authority was prepared?
Did the transaction COMMIT or ROLLBACK?
What immutable images/contract were live?
Which workflow run proves it?

Do not retain secrets or full runtime envs.


Phase 0 — read before editing

Read completely:

  1. root AGENTS.md;
  2. refactor(release): consume exact OpenPath contract and promote immutable release bundles #160 completion/current implementation;
  3. hardening(release): make production promotion and rollback hermetic and operationally provable #162 completion thread and final production evidence;
  4. current .github/workflows/deploy.yml;
  5. current release-candidate workflow(s);
  6. current production tag creation/operator procedure;
  7. scripts/deploy-production-remote.sh;
  8. scripts/lib/deploy-production-runtime.sh;
  9. production recovery authority/bundle scripts;
  10. deployment-state / transaction helpers;
  11. staging deployment scripts and helpers;
  12. release evidence/promotion evidence tooling;
  13. relevant deployment/release automation tests.

Do not modify OpenPath from this issue.

Do not deploy staging/production, create tags or release anything during implementation unless separately authorized.


Phase 1 — pre-tag production readiness

Create one canonical non-mutating production-readiness command/workflow callable before a production tag exists.

Repository conventions may determine whether this is primarily a script, reusable workflow, or both, but there must be one authoritative implementation.

Suggested conceptual interface:

production-readiness --rc-run-id <RC>

or equivalent.

It must resolve the exact RC and fail closed if any required prerequisite is unavailable.

Required checks

Exact candidate identity

Resolve and verify:

CANDIDATE_SHA
RC_RUN_ID
RELEASE_ID
OpenPath SHA
OpenPath contract SHA256
Release Bundle v2 bytes/hash
OCI image refs/digests
required platform compatibility

The selected RC must be successful and promotion-eligible according to the current repository contract.

Do not select another RC automatically.

Staging eligibility

Require the exact release identity to have the staging evidence currently required by production promotion.

The preflight must prove whatever the current production workflow would later require from verify-staging-release-state.

Do not duplicate staging release-selection logic; reuse the existing canonical evidence parser/contract.

Production recovery authority

Before tag creation prove:

PRODUCTION_RECOVERY_SHA present
full lowercase SHA
R != C
R checkout/source resolvable
contract/source versions supported
exact recovery bundle packageable
recovery executor present
artifact/executor hashes valid
recovery authority preflight passes

Use the same recovery packaging/preflight code that production uses.

Do not create a second approximation of recovery readiness.

The preflight must catch a missing/invalid PRODUCTION_RECOVERY_SHA before tag creation.

Production configuration contract

Validate all configuration that can be safely checked without mutating production, including at minimum the mandatory variables/secrets/identities required by the deploy job.

Do not print secret values.

For secret-backed requirements, only report presence/absence and safe metadata.

Host/readiness contract

Where practical and non-mutating, run the same production host preflight/read-only connectivity checks used by the deploy path:

SSH reachable
required host tools
Docker/Compose available
Docker daemon reachable
deploy root contract
filesystem permissions/state prerequisites
required immutable artifacts pullable/authenticated where safe

Do not cross the production mutation boundary.

Do not stop/recreate containers.

Output

On success emit one bounded human-readable and machine-readable summary, conceptually:

PRODUCTION READINESS PASSED
C=<sha>
RC=<run>
releaseId=<id>
OpenPath=<sha>
contract=<sha256>
R=<sha>
stagingEvidence=<verified>
recoveryArtifact=<verified>
productionHost=<ready>

On failure identify the concrete blocker class:

RC_BLOCKER
STAGING_BLOCKER
CONFIG_BLOCKER
RECOVERY_BLOCKER
HOST_BLOCKER
ARTIFACT_BLOCKER

Do not emit generic exit 1 when a safe diagnostic can be provided.


Phase 2 — one canonical promotion entrypoint

Add one operator-facing promotion command/workflow that consumes an explicit RC and is the normal way to create a production tag.

Conceptually:

promote-production <RC_RUN_ID>

Required sequence:

resolve exact RC
    ↓
run Phase 1 production readiness
    ↓
render exact proposed identity
    ↓
require explicit operator approval where repository policy requires it
    ↓
create exactly one annotated v* tag
    ↓
push tag
    ↓
existing tag-only production workflow executes

Tag identity

Reuse the current canonical annotated-tag schema.

The entrypoint must not hand-build a divergent schema.

Before pushing, verify locally/in-run that the tag identity contains and matches the exact resolved values required by the current deploy workflow, including at least:

CLASSROOMPATH_SHA
RC_RUN_ID
RELEASE_ID

If additional canonical identity fields already exist, preserve them.

Version allocation

Use the repository's existing release version policy.

Do not silently retag an existing version.

Do not move existing tags.

If version allocation remains operator-supplied, validate uniqueness before creation.

If automated, make the allocation deterministic and collision-safe.

Failure semantics

Anything failing before the tag push must result in no tag.

Once the tag is pushed, the existing immutable production workflow remains authoritative.

Do not add logic that deletes a pushed production tag after a failed deploy.


Phase 3 — converge staging on the hermetic production executor

Refactor staging so it uses the same canonical deployment runtime/executor semantics proven by #162.

This does not mean production and staging share mutable state or credentials.

They must remain isolated environments.

Desired model

                  exact Release Bundle
                         |
                 deploy payload/runtime
                         |
                hermetic deploy executor
                   /              \
              staging          production

The shared path should own:

release identity validation
runtime projection
transaction/state semantics
container switch/recreate orchestration
health
ready
live identity validation
commit semantics
diagnostics contract
recovery decision semantics where applicable

Environment adapters/config should own:

URLs
host credentials
deploy root
compose project/environment identity
production authorization flag
fault injection policy
recovery authority scope

Preserve production fences

Production-only protections must remain explicit and fail closed.

Do not make a staging environment capable of satisfying production identity accidentally.

The existing K environment fencing concepts should guide the separation, but do not introduce K-only coupling into normal staging.

Staging transaction model

Normal staging success should follow the same relevant state progression as production:

PREPARED
-> SWITCHING
-> ACTIVATED_UNVERIFIED
-> VERIFIED
-> COMMITTED

A post-boundary staging failure should produce the same truthful failure/recovery semantics appropriate to staging.

No duplicated runtime projection

There must be one canonical implementation/list for release runtime projection.

Do not keep separate production and staging copies that can drift.

Migration path

Remove or reduce the legacy npm run deploy:staging implementation only after the shared executor path is covered and proven.

A thin wrapper command may remain for operator UX, but it should call the canonical executor rather than implement separate deploy semantics.


Phase 4 — durable deployment ledger

Add a bounded safe deployment ledger independent of short-lived GitHub artifact retention.

Prefer the smallest repository-consistent storage mechanism.

Do not introduce a new external database solely for this issue unless already available infrastructure makes that clearly simpler.

One record per completed/failed production promotion is sufficient.

Minimum fields

Persist safe values such as:

timestamp
environment
tag
CANDIDATE_SHA
RC_RUN_ID
RELEASE_ID
OPENPATH_SHA
OPENPATH_CONTRACT_SHA256
PRODUCTION_RECOVERY_SHA
transactionId
phase/result
previous releaseId/SHA where canonical
current releaseId/SHA where canonical
OCI image digests
health result
ready result
rollback attempted
rollback result
GitHub workflow run ID/URL

Use current canonical names where they differ.

Security

Never persist:

secrets
tokens
Authorization/Cookie values
private keys
download refs
personalized installer payloads
full environment dumps
raw config containing credentials

Integrity

Ledger entries should be immutable/append-only in normal operation or otherwise tamper-evident according to existing repository conventions.

Do not allow a failed later job to rewrite an earlier successful deployment as a different identity.

Retention

Keep existing bounded debug artifacts as appropriate.

The ledger is the long-lived compact identity/audit summary; it is not a replacement for all debug artifacts.


Phase 5 — tests and regression coverage

Write failing tests before or alongside implementation for each new invariant.

Pre-tag readiness tests

Required cases:

  • exact valid RC passes;
  • missing RC fails;
  • wrong releaseId/C mismatch fails;
  • staging evidence for a different release fails;
  • missing PRODUCTION_RECOVERY_SHA fails before tag creation;
  • R == C fails;
  • recovery source mismatch fails;
  • recovery artifact package/preflight failure fails;
  • required production config missing fails safely;
  • host preflight failure fails without mutation;
  • no failed readiness path creates/pushes a tag.

Promotion entrypoint tests

  • runs readiness before tag creation;
  • exact RC/C/releaseId embedded in tag;
  • tag SHA matches C;
  • existing version/tag collision fails;
  • preflight failure => no tag;
  • successful promotion creates one tag only;
  • no rebuild/re-resolution after RC selection;
  • no latest fallback.

Use fixtures or an isolated test repository where appropriate; tests must not create real production tags.

Shared executor tests

Prove staging and production use the same canonical runtime/deploy helper for:

  • runtime projection;
  • release identity validation;
  • state transitions;
  • health/readiness;
  • live identity validation.

Add a regression that fails if staging reintroduces a divergent projection/switch implementation.

Test environment fences so staging cannot identify as production.

Ledger tests

  • successful COMMITTED promotion produces exact identity record;
  • ROLLED_BACK promotion records rollback truthfully;
  • pre-boundary failure records no false production mutation;
  • no secrets included;
  • records cannot silently overwrite an unrelated deployment identity;
  • workflow run locator preserved.

Existing suites

Run at minimum the repository-current equivalents of:

npm run test:deployment
npm run test:release-automation
npm run test:ci-regression
npm test
verify:static / relevant static gates
shell syntax/shellcheck for modified shell

Follow AGENTS.md for exact required commands.


Operational proof

Implementation completion and operational adoption are separate.

Before claiming implemented

All local/CI tests above must pass and main must contain the implementation.

Staging proof

Only with explicit authorization, run one normal staging deployment through the new shared executor.

Prove:

exact selected release
COMMITTED
runtime/checkout/images exact
health 200
ready 200 / true
live identity exact
ledger entry exact

A deliberate staging fault is only required if the implementation changes recovery/failure semantics materially; do not repeat #162 K automatically for UI/refactoring-only changes.

Production proof

Do not deploy production as part of implementation without separate explicit authorization.

The first authorized production promotion using the new entrypoint should demonstrate:

pre-tag readiness PASSED
tag created only after readiness
normal tag-only deploy
COMMITTED
exact live identity
health/ready/smoke green
ledger record written

Do not deliberately inject production faults.


Acceptance criteria

  • One canonical non-mutating production-readiness preflight exists.
  • Missing/invalid recovery authority is detected before production tag creation.
  • Recovery bundle/package/preflight is exercised before production tag creation.
  • Exact RC/C/releaseId/OpenPath/contract/image identity is resolved before tagging.
  • Current staging eligibility is checked before tagging without duplicating release-selection logic.
  • Production config/host prerequisites that can be checked safely are verified before tagging.
  • Preflight failures produce clear blocker classification and no tag.
  • One canonical operator promotion entrypoint exists.
  • Promotion entrypoint runs readiness first and creates exactly one immutable annotated tag.
  • Production remains tag-only.
  • No latest, rebuild or alternative RC selection is introduced.
  • R remains explicit; it is never inferred automatically from previous/current releases.
  • Staging uses the same canonical hermetic deploy/runtime semantics as production.
  • Staging and production retain strict environment/authorization separation.
  • Runtime projection has one canonical implementation/list.
  • Staging state progression matches the proven transactional semantics where applicable.
  • A durable bounded deployment ledger exists.
  • Ledger contains exact release/recovery/transaction/workflow identity.
  • Ledger contains no secrets or personalized payloads.
  • Existing refactor(release): consume exact OpenPath contract and promote immutable release bundles #160/hardening(release): make production promotion and rollback hermetic and operationally provable #162 invariants remain intact.
  • Required deployment/release regression suites pass.
  • No production deployment/tag is performed during implementation without explicit authorization.

Non-goals

Do not in this issue:


Implementation guidance

Prefer deletion/convergence over adding another abstraction layer.

Examples:

GOOD:
existing release resolver + existing recovery authority + one readiness orchestrator

BAD:
new parallel release resolver
new recovery model
new deployment state machine
GOOD:
npm run deploy:staging -> thin wrapper -> canonical hermetic executor

BAD:
production executor + staging executor that merely happen to look similar

The goal after this issue is that a routine promotion is boring:

select RC
-> readiness green
-> create tag
-> deploy
-> COMMITTED
-> smoke
-> ledger

and an invalid promotion stops before an immutable production version is consumed.


Completion report required

The implementing agent must report separately:

  1. PRE-TAG READINESS

    • command/workflow;
    • exact checks;
    • failure classifications;
    • proof that failure creates no tag.
  2. PROMOTION ENTRYPOINT

    • exact invocation;
    • tag schema reuse;
    • version collision behavior;
    • no rebuild/latest behavior.
  3. SHARED EXECUTOR

    • code paths removed/reused;
    • staging vs production environment adapters;
    • runtime projection ownership;
    • state transition evidence.
  4. DEPLOYMENT LEDGER

    • storage location/format;
    • fields;
    • retention/integrity behavior;
    • secret exclusion tests.
  5. TESTS

    • focused commands/results;
    • complete deployment/release suite results.
  6. OPERATIONAL PROOF

    • staging only if separately authorized/executed;
    • production only if separately authorized/executed.

Do not collapse implementation completion into production proof.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions