diff --git a/docs/development/ENGINEERING_PLATFORM_ROADMAP.md b/docs/development/ENGINEERING_PLATFORM_ROADMAP.md index a72299d3..babd06ff 100644 --- a/docs/development/ENGINEERING_PLATFORM_ROADMAP.md +++ b/docs/development/ENGINEERING_PLATFORM_ROADMAP.md @@ -461,6 +461,32 @@ Not required unless the canary proves otherwise: generalized Agent separation; m After standalone + self-development proofs, continue separately with generalized Project-Agent separation/dispatch, multi-host routing, richer queue policy/capacity/ordering, multi-repository leases/parallelism, broader recovery/retry and remaining B8E parity/productization. +## Bounded clean-CENTRAL preparation + +`EP::CENTRAL_OPERATIONAL_RESET_V1` is the small product-owned maintenance +slice needed before a later clean-state Mission preflight. It preserves +installation/configuration/security authority, archives and removes only +schema-classified operational history, advances dataset generation, retains +anti-replay tombstones and uses a verified protected backup plus durable +writer fence. Its implementation does not require the full Operations Console, +generalized Project-Agent topology, universal installer or Workspace UI. + +The lane is: + +```text +schema-68 implementation + fixture qualification + -> protected review/full EP gates + -> 2.3.81 exact-artifact release and authorized installation + -> installed read-only preview + -> later explicit coordinated Forge+EP clean-CENTRAL preflight +``` + +Source implementation, qualification, release, installation, preview and +destructive apply remain distinct evidence states. The present slice must not +turn an unstarted Mission/telemetry trial into a reported runtime failure and +does not reopen the closed #271/#272 reporting/telemetry work without a new +defect. + ## Forge integration after standalone/self-development proof ```text diff --git a/docs/engineering/EP_CENTRAL_OPERATIONAL_RESET_V1.md b/docs/engineering/EP_CENTRAL_OPERATIONAL_RESET_V1.md new file mode 100644 index 00000000..e7e35ef8 --- /dev/null +++ b/docs/engineering/EP_CENTRAL_OPERATIONAL_RESET_V1.md @@ -0,0 +1,469 @@ +# EP CENTRAL operational reset V1 + +**Owning product:** Engineering Platform. **Profile:** +`EP_CENTRAL_OPERATIONAL_HISTORY_V1`. **Server schema:** 68. **Candidate +release:** 2.3.81. + +This is a bounded local maintenance route for removing EP operational history +while retaining installation identity, project and repository attachment, +consumer authority, credentials, security audit and configuration. It is not a +factory reset, a generic database administration surface, or a Forge Mission. +It never opens Forge storage and has no remote HTTP route or Console button. + +The installed command is `engineering-platform-maintenance`. It is separate +from `engineering-platform-server` because a stopped Server is a precondition, +and separate from the submission CLI because maintenance authorization is the +local installation owner's authority rather than a consumer bearer. Every +mutating command acquires the existing `operational-installation.lock` and is +also fenced by durable CENTRAL maintenance state. + +## State and authorization contract + +`preview` opens the selected `epdata.sqlite` with SQLite URI `mode=ro`. It does +not call `server.initialize`, create a maintenance row, migrate schema, run a +Mission, issue a grant, checkpoint WAL, vacuum, or write an artifact. A schema +other than 68 is a reported blocker rather than an implicit upgrade. + +The plan binds the exact instance ID, resolved data/database paths, filesystem +database identity, schema, profile, dataset generation, complete logical data +revision, preserved-binding digest, table/record classification, external +file inventory, integrity findings and effect set. Its canonical SHA-256 +digest is the approval identity. `prepare` re-reads that plan while acquiring +the installation lock, derives the real local actor as `uid::`, +persists `PREPARING`, and thereby activates database triggers that reject +normal INSERT/UPDATE/DELETE entry points after any process restart. + +The durable states are: + +```text +PREPARING -> AUTHORIZED -> ARTIFACTS_ARCHIVING -> ARTIFACTS_ARCHIVED + -> DB_APPLIED -> VERIFIED -> COMPLETED +``` + +`FAILED` is deliberately still writer-blocking. Only `finish` may enter +`COMPLETED`, and only from `VERIFIED`. A changed operation request or plan +digest is rejected; there is no `--force`. Known foreign-key findings may be +approved only by repeating every exact `fk::::` +identity emitted by that plan. Findings touching preserved or unknown state +remain blockers. + +`apply` accepts only `AUTHORIZED`, `ARTIFACTS_ARCHIVING` or +`ARTIFACTS_ARCHIVED`; its safe idempotent readback states are `DB_APPLIED` and +`VERIFIED`. It rejects `ABORTED`, `COMPLETED`, `FAILED` and every unknown state. +Every transition is compare-and-set and is independently enforced by a +maintenance-owner state trigger. Reset-operation bindings and tombstones are +immutable to ordinary CENTRAL connections; a connection-local owning +capability is present only on the maintenance service's connections. Preview +requires the complete schema-derived normal-writer trigger set plus every +maintenance-table guard before it can be allowed. + +Before the first external effect, `abort` may move `PREPARING` or `AUTHORIZED` +to terminal `ABORTED` and release the fence. It is rejected after artifact +archiving or database apply starts. A failed-backup resume accepts only the +originally authorized backup-root binding; changing that binding requires +aborting safely and preparing a newly previewed operation. + +The CLI emits `operational-reset-v1` JSON with stable top-level product, +command, operation, state, allowed, target, profile, generation, plan/revision, +backup, counts, blockers, integrity and preserved-binding fields. Protected +credential verifiers and row contents never appear in that receipt. +Argument, storage and unexpected command failures use the same top-level +shape and a stable `error_code`; exception text, raw verifier material and +operator-selected paths are not copied into error receipts. + +## Schema-owned classification + +The implementation enumerates every schema-68 application table; an extra or +missing application table blocks prepare. SQLite indexes, triggers and views +are separately inventoried with definition digests and a preserve effect; +SQLite-internal objects are not classified as application data to purge. + +`INSTALLATION_AND_CONFIGURATION` is preserved: + +- `engineering_schema_migrations`, `ep_installations`, + `ep_project_registrations`, `ep_repository_registrations`, + `ep_agent_registrations`, `ep_agent_repository_attachments`, + `ep_local_repository_bindings`, `ep_external_producer_bindings`, and + `execution_migration_provenance`; +- configuration records in `engineering_metadata` and records in + `execution_projections` whose explicit classification is `CONFIGURATION`. + +`SECURITY_AND_AUTHORITY_LEDGER` is preserved: + +- `ep_agent_pairing_codes`, `ep_consumer_credentials`, + `ep_consumer_registrations`, + `ep_consumer_credential_recovery_operations`, `ep_control_provenance`, + `ep_external_producer_binding_audit`, and `ep_operator_capabilities`. + +This preserves verifier bytes in their owning database but never copies a +Keychain item, bearer, provider login or credential into a receipt or portable +export. Revocations, recovery decisions, operator capabilities and external +producer bindings are not reset into reusable authority. + +`OPERATIONAL_HISTORY` is purged from the active dataset: + +- CENTRAL submissions/events/prompt indexes, execution runs/leases, + lifecycle dispatches, queue dispositions, receipt provenance, Forge + exchange/action/planning envelopes, execution-host/technical/reconciliation + evidence; +- retained-host transactions, submissions/attempts/links, leases/events, + prompt history/chat, receipts/artifacts, admission/readiness/reconciliation, + phase timing, validation/qualification, governance/assurance, provider + invocation/recovery, Dependabot admission and terminal telemetry outbox; +- legacy `engineering_artifacts` and every other table enumerated in + `OPERATIONAL_HISTORY` in `central_operational_reset.py`. + +`DERIVED_CACHE_OR_PROJECTION` is purged: + +- daily/execution activity and provider-usage telemetry, component logs, + `engineering_status`, non-configuration `execution_projections`, and the + two operational metadata keys + `central_database.maintenance_last_attempt_at` and + `ep.provider_capacity_history.v1`. + +`MAINTENANCE_AUDIT` is retained and excluded from the relevant-source revision: + +- `ep_operational_reset_operations`, `ep_operational_dataset_state`, and + `ep_operational_identity_tombstones`. + +Tombstones retain one-way identities for deleted submissions, runs, provider +invocations and used idempotency keys. The original prompt is not retained in +the tombstone. Reusing the same idempotency request is rejected as +`IDEMPOTENCY_RETIRED`; using different bytes with that key is +`IDEMPOTENCY_CONFLICT`. Dataset generation advances once for one applied +operation; installation/runtime identity does not change. Integer SQLite +rowids are internal storage identities, not external run/submission +namespaces. External textual identities are random/immutable and tombstoned. + +## External storage + +The active `artifacts/`, `file-inbox/` and +`runtime/central-data-imports/` trees are hashed into the plan and protected +backup, moved beneath +`operational-reset-archive//`, and recreated empty. This prevents +an Inbox watcher or artifact lookup from immediately re-projecting the old +dataset. A symlink or non-regular entry fails closed. + +The exact known installation/configuration/runtime/recovery names are +preserved. In particular, `central.sqlite` and the bounded +`epdata.sqlite.pre-X.Y.Z.backup` form are +`FORENSIC_OR_RECOVERY`; `.forge-ep-consumer-recovery.lock` and the installation +lock are `SYSTEM_RUNTIME_CONTROL`. `server.json`, `runtime-identity.json`, +`runtime/`, update `operations/`, `migration/`, `recovery/` and existing +`backups/` stay outside the purge. A new top-level name is +`UNKNOWN_OR_UNSUPPORTED` and blocks prepare. Source repositories, canonical +documents, Git worktrees, user workspaces and other installations are never +targets. + +Known preserved directories are also inventoried recursively. Only exact +product-owned runtime records, installer journal/staging shapes, migration +receipts, legacy backups and reset-archive layouts receive a preserve +classification. An arbitrary file hidden inside `runtime/`, `operations/`, +`recovery/`, `migration/`, `backups/` or a reset archive is unsupported and +blocks prepare without being moved or deleted. A pending CENTRAL import is an +explicit active-ingest blocker. + +## Protected backup and recovery + +`prepare` requires an operator-selected backup root outside the active data +root. It rejects links, unsafe permissions, insufficient free space and +unwritable destinations. SQLite's online backup API captures the complete +database, including committed WAL content, into `central.sqlite`; no raw main +file copy is used. The private operation directory and files are mode 0700 and +0600. Its manifest binds source instance/schema, operation/plan/revision, +database hash and row counts, exact included external files/hashes, integrity +and known FK findings, and excluded secret stores/runtime/source trees. + +Verification reopens the backup in an isolated read-only connection, runs +`integrity_check`, compares every table count and re-hashes every protected +file. Known purge-local FK defects may remain faithfully present in the backup; +that is recorded, not called semantically clean. + +Backup publication is staged in a new private sibling and atomically renamed +to the operation-ID destination only after full restore verification. Nested +source or destination symlinks and pre-existing destination bytes are never +followed or overwritten. A complete destination left by a crash is reusable +only when its operation, plan, database and every file verify exactly; partial +staging remains preserved for diagnosis while `resume` creates a fresh staging +snapshot for the same durable operation. The stored manifest-file digest is +rechecked by `apply`, `verify`, readback and finalization, not merely recomputed +from whichever manifest happens to be present. + +After an interruption, use `status` and `resume` with the same operation ID +and plan digest. A partial external move is reconciled by exact planned hashes. +The database purge and generation change are one `BEGIN IMMEDIATE` transaction; +immutable-evidence and maintenance-block triggers are removed and recreated +inside that uncommitted transaction, so another connection never observes an +unfenced write window. A pre-commit crash leaves no partial DB purge. A +post-commit crash resumes at verification. Forward reconciliation is the +default. + +`finish` does not trust an earlier `VERIFIED` receipt by itself. While the +installation lock and durable writer fence are still active it re-verifies the +protected backup, quick/FK checks, operational emptiness, dataset generation, +preserved bindings and all external ingest routes. It then durably binds an +operation- and generation-specific finish boundary and atomically renames the +complete active `artifacts/`, `file-inbox/` and +`runtime/central-data-imports/` roots into that protected boundary before it +creates empty roots for the new generation. Every replacement root is opened +without following symlinks and immediately frozen at mode `0500`. The final +`VERIFIED` proof requires all three roots to remain exactly `0500` and empty; +only after the durable `COMPLETED` commit are they idempotently thawed to +`0700`. A file that arrives after the last empty scan through the old path is +therefore preserved behind the boundary; +a writer holding an old directory descriptor also remains isolated in the +renamed inode. Unknown files are never deleted. The boundary manifest, file +hashes and digest are verified before `COMPLETED` and on later verification. + +The boundary-path binding is persisted while the state remains `VERIFIED` +before the first filesystem rename. A crash during the three renames therefore +keeps the durable writer fence active. Re-running `finish` for the same +operation recognizes already-rotated roots, rotates only the remaining roots, +recovers its exact marker staging file and completes forward. A pre-existing +unbound boundary or a changed marker fails closed. + +A crash after the `COMPLETED` commit but before every thaw leaves the new roots +non-writable. Re-running `finish`, or `resume` for that same completed +operation, verifies the exact operation/plan/target identity, database inode, +dataset generation and boundary digest before thawing the safe mix of `0500` +and already-thawed `0700` roots. This completed-operation proof deliberately +does not demand that current operational tables or active roots still be +empty: legitimate new-generation work may exist after a successful thaw. +Symlinks, unexpected modes and changed boundary content always fail closed. + +Do not automatically copy the backup over live CENTRAL: an old database could +undo later revocations, consumed authority or external effects. An exceptional +restore is a separately authorized, offline operation: keep writers stopped, +copy the protected backup to an isolated test root, verify manifest/hash/counts +and schema, establish how later security effects will be preserved, then use +the owning recovery/import route. This reset command deliberately has no +generic rollback switch. + +## Chat-integrity defect and migration + +Schema 67 declared +`execution_chat_messages.run_id -> prompt_execution_history.run_id`, while the +CENTRAL Console deliberately authorized a terminal chat through +`ep_parity_lifecycle_dispatches`/`ep_execution_runs` even when the legacy +prompt-history writer had not produced a row. The direct CENTRAL Console path +therefore wrote a schema-declared orphan whenever this supported no-prompt-index +state occurred. That owning defect shape is reproducible and could create the +observed shape. + +Schema 68 changes the declared chat parent to canonical `ep_execution_runs` +and makes the owning Console writer enable and read back `foreign_keys=ON` on +the exact write connection, lock the write transaction, and verify the +canonical run before insert. +Reset-owned connections enforce foreign keys throughout maintenance. Migration +copies chat rows unchanged and refuses any row without that canonical run; it +creates no fictitious prompt parent and deletes no chat row. Correct chat +evidence whose legacy prompt index is absent is therefore preserved. The +currently observed production rows were confirmed read-only to have canonical +run and lifecycle parents, but their exact historical writer/provenance was +not retained. +Consequently: + +```text +REPRODUCIBLE_OWNING_DEFECT = CONFIRMED +EXACT_HISTORICAL_ROW_CAUSE = ROOT_CAUSE_UNCONFIRMED +ORPHAN_PREVENTION = CANONICAL_PARENT_PLUS_FOREIGN_KEYS +ORPHAN_RESET_HANDLING = PREVIEW_CLASSIFY_EXACTLY; NO AUTOMATIC ROW DELETION +``` + +## Installed command recipe + +Preview and status are non-destructive: + +```bash +engineering-platform-maintenance preview \ + --data-root "/exact/Engineering Platform Server/data" + +engineering-platform-maintenance status \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" +``` + +The later clean-CENTRAL operation is explicitly destructive from `apply` +onward. Do not run these commands during installation or as part of an +ordinary Server start: + +```bash +# 1. Stop/verify every writer, save PREVIEW_PLAN_DIGEST and every exact +# review_required FK identity from the preview. +engineering-platform-maintenance prepare \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" \ + --backup-root "/exact/protected/recovery/root" \ + --allow-operational-fk "fk::::" + +# 2. DESTRUCTIVE: archive active artifacts/Inbox and purge the approved DB set. +engineering-platform-maintenance apply \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" + +# 3. Prove physical/referential integrity, operational emptiness, backup and bindings. +engineering-platform-maintenance verify \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" + +# 4. Only after the coordinated Forge+EP verification authorizes normal writers. +engineering-platform-maintenance finish \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" +``` + +If interrupted, run `status`, then: + +```bash +engineering-platform-maintenance resume \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" \ + --backup-root "/exact/protected/recovery/root" +``` + +`--backup-root` is needed during resume only if the durable state is still +`PREPARING`. The command never rotates credentials, changes historical +acceptance, creates a submission/run/Mission, or resumes writers by itself. + +To safely end preparation before any reset effect has begun: + +```bash +engineering-platform-maintenance abort \ + --data-root "/exact/Engineering Platform Server/data" \ + --operation-id "central-clean-" \ + --plan-digest "$PREVIEW_PLAN_DIGEST" +``` + +## Joint qualification coordinator + +`tools/qualification/operational_reset_coordinator.py` is the bounded operator +harness for the later coordinated Forge and EP maintenance window. It is not a +third product service and it does not open either database. It records only a +secret-free joint progress receipt and invokes the two installed, owning CLIs +as subprocesses. Its receipt directory is mode `0700`, its receipt and lock are +mode `0600`, updates are atomic and fsynced, and symlinks or a concurrent +coordinator process are rejected. + +Choose a new joint reference and two distinct product operation IDs. Bind the +exact installed CLI files, target roots and protected EP backup root during the +initial preview: + +```bash +COORDINATOR_ID="central-clean-coordinator-" +COORDINATOR_RECEIPTS="/exact/protected/coordinator-receipts" +FORGE_OPERATION_ID="forge-clean-" +EP_OPERATION_ID="ep-clean-" + +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" \ + --coordinator-id "$COORDINATOR_ID" \ + preview \ + --forge-cli "/exact/installed/bin/forge" \ + --forge-data-root "/exact/Forge/data" \ + --forge-operation-id "$FORGE_OPERATION_ID" \ + --ep-cli "/exact/installed/bin/engineering-platform-maintenance" \ + --ep-data-root "/exact/Engineering Platform Server/data" \ + --ep-operation-id "$EP_OPERATION_ID" \ + --ep-backup-root "/exact/protected/ep-recovery" +``` + +`preview` is read-only for both product datasets. If either preview reports an +explicitly reviewable operational foreign-key finding, retain that receipt as +read-only evidence, choose a new coordinator reference before any `prepare`, +and repeat the initial preview with the exact finding identity bound as +`--forge-fk-acknowledgement ` or +`--ep-fk-acknowledgement `. The acknowledgement is used only by the +later owning `prepare`; it is not a general integrity bypass. + +The remaining commands load the already-bound configuration from the receipt: + +```bash +# MUTATING: both products enter durable maintenance and make verified backups. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" prepare + +# Read-only under both writer fences: prove both exact plans are still current. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" revalidate + +# DESTRUCTIVE: invoke one owning reset at a time. Order is an operator choice. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" \ + apply --product forge +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" \ + apply --product engineering-platform + +# Read-only proofs. Authorization is local to the joint receipt. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" verify +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" authorize-resume + +# MUTATING: owning products may leave maintenance only after both verified. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" \ + finish --product forge +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" \ + finish --product engineering-platform +``` + +These destructive commands are qualified only against isolated fixture stores +in this delivery; do not run them against CENTRAL as part of installation. The +joint receipt reaches `COMPLETE` only after both owning verifications, explicit +resume authorization and both owning finishes. + +Immediately before every individual `finish`, the harness calls both owning +`verify` commands again and requires each product to report `VERIFIED` or +`COMPLETED`. This is a two-phase pre-finish readiness check, not a distributed +transaction: the owning finishes remain sequential. If the first product has +already reached `COMPLETED` and the second finish fails, the receipt records +`RECONCILIATION_REQUIRED`; reconciliation truthfully returns the pair +`COMPLETED`/`VERIFIED`, keeps the unfinished product fenced, and requires an +explicit retry of that same second finish. It never rolls the first product +back or automatically resumes the second. + +After any interruption or owning failure, inspect without creating a new +operation and reconcile the same product operation IDs: + +```bash +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" status +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" reconcile + +# Use only when the receipt proves PLANS_REVALIDATED and durably records that +# this exact product's apply was admitted before the process was interrupted. +python3 tools/qualification/operational_reset_coordinator.py \ + --receipt-root "$COORDINATOR_RECEIPTS" --coordinator-id "$COORDINATOR_ID" \ + resume --product engineering-platform +``` + +Once an owning apply may have started, every command failure is +`RECONCILIATION_REQUIRED`: both products stay in maintenance, there is no +automatic finish and no automatic destructive rollback. The harness never +calls a peer CLI from product code, imports Forge or EP, creates a Mission, +provider invocation or submission, or treats delayed callbacks as new work. +`resume` is fail-closed from `BOTH_PREVIEWED`, `BACKUPS_VERIFIED`, and from +`PLANS_REVALIDATED` until the selected product has its durable apply-admission +milestone. A subprocess crash after apply can be reconciled because that +milestone was fsynced before the owning command ran. + +The repository document `missions/MISSION-0003.md`, a future test label and an +operational Forge Mission are distinct namespaces. EP reset readback names its +own runtime run/submission namespace and does not make a Mission-ID allocation +decision. + +## Delivery status + +At source-candidate creation this slice is `IMPLEMENTED` and locally fixture +qualified, while protected PR review, hosted full gates, release publication, +artifact-byte qualification and installed live preview remain separate +evidence. Installation may activate schema 68 and the chat relationship repair +but must not automatically prepare/apply a reset or delete historical rows. diff --git a/docs/engineering/EP_SERVER_PRODUCTION_RELEASE.md b/docs/engineering/EP_SERVER_PRODUCTION_RELEASE.md index a02c90ca..59b86279 100644 --- a/docs/engineering/EP_SERVER_PRODUCTION_RELEASE.md +++ b/docs/engineering/EP_SERVER_PRODUCTION_RELEASE.md @@ -9,6 +9,14 @@ the exclusive source of a published version change. Any historic branch-create workflow remains evidence for prior releases only. It is not authority to publish, re-publish, or rewrite EP release bytes. +The 2.3.81 patch candidate adds schema-68 CENTRAL operational-reset support. +Its release uses this unchanged protected main-first policy. Schema activation +may retarget chat evidence to its canonical run parent and install dormant +maintenance controls; neither publication nor installation authorizes +`prepare`/`apply`, orphan deletion, submission creation or a Mission. Exact +artifact qualification and an installed read-only preview are required before +the capability can be reported installed. + ## Release-operation lifecycle V1 After both the production-wheel and dashboard qualifications succeed, the diff --git a/docs/engineering/EXECUTION_HOST_OPERATIONS.md b/docs/engineering/EXECUTION_HOST_OPERATIONS.md index 5bdcce7b..ea497291 100644 --- a/docs/engineering/EXECUTION_HOST_OPERATIONS.md +++ b/docs/engineering/EXECUTION_HOST_OPERATIONS.md @@ -458,3 +458,12 @@ creating an autonomous retry loop. Neither control rewrites history, stashes work, or deletes a branch without the explicit second confirmation. + +## CENTRAL operational-history maintenance + +The product-owned local reset route, exact preservation/purge mapping, +protected backup protocol and stopped-Server command sequence are specified in +[EP CENTRAL operational reset V1](EP_CENTRAL_OPERATIONAL_RESET_V1.md). The +installed entry point is `engineering-platform-maintenance`; preview is +read-only, while prepare activates a durable writer fence and apply is +destructive. Normal Server startup and package installation never invoke it. diff --git a/package-lock.json b/package-lock.json index d6d798f1..e336ceab 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "engineering-platform-browser-validation", - "version": "2.3.80", + "version": "2.3.81", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "engineering-platform-browser-validation", - "version": "2.3.80", + "version": "2.3.81", "devDependencies": { "@playwright/test": "1.63.0" } diff --git a/package.json b/package.json index eddb8e0a..703e3b40 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "engineering-platform-browser-validation", "private": true, - "version": "2.3.80", + "version": "2.3.81", "scripts": { "test:engineering-dashboard": "PYTHONPATH=src python3 -m engineering_platform.dashboard_browser_validation", "test:engineering-dashboard-logic": "node --test tests/engineering/dashboard_status_store.test.mjs tests/engineering/ui_localization_contract.test.mjs tests/engineering/dashboard_translation.test.mjs", diff --git a/pyproject.toml b/pyproject.toml index 83bf7cf9..4965dbdc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "engineering-platform" -version = "2.3.80" +version = "2.3.81" description = "Local-first Engineering Platform execution operations runtime" readme = "README.md" requires-python = ">=3.14,<3.15" @@ -19,6 +19,7 @@ engineering-platform-server = "engineering_platform.server:main" engineering-platform = "engineering_platform.submission_cli:main" engineering-project-agent = "engineering_platform.project_agent:main" engineering-reconciliation-adopt = "engineering_platform.reconciliation_adoption:main" +engineering-platform-maintenance = "engineering_platform.central_operational_reset:main" [tool.setuptools] package-dir = {"" = "src"} diff --git a/src/engineering_platform/ENGINEERING_PLATFORM_CONFIG.json b/src/engineering_platform/ENGINEERING_PLATFORM_CONFIG.json index 42ac3009..ab357d7e 100644 --- a/src/engineering_platform/ENGINEERING_PLATFORM_CONFIG.json +++ b/src/engineering_platform/ENGINEERING_PLATFORM_CONFIG.json @@ -3,7 +3,7 @@ "platform": { "id": "engineering-platform", "name": "Engineering Platform", - "version": "2.3.80", + "version": "2.3.81", "generation": 2, "documentation_namespace": "engineering-platform", "capability_registry_version": 1 diff --git a/src/engineering_platform/ENGINEERING_PLATFORM_VERSION.json b/src/engineering_platform/ENGINEERING_PLATFORM_VERSION.json index 3acd5867..133f80c7 100644 --- a/src/engineering_platform/ENGINEERING_PLATFORM_VERSION.json +++ b/src/engineering_platform/ENGINEERING_PLATFORM_VERSION.json @@ -1,15 +1,15 @@ { "bootstrap_contract": "2026.12", "checkpoint_format": 1, - "dashboard_version": "2.3.80", + "dashboard_version": "2.3.81", "handoff_protocol": 1, "memory_format": 2, "minimum_codex_cli": "0.146.0", "inbox_protocol": 1, - "platform_version": "2.3.80", + "platform_version": "2.3.81", "report_format": 2, - "runner_version": "2.3.80", + "runner_version": "2.3.81", "status_model": 1, "storage_schema": 45, - "watcher_version": "2.3.80" + "watcher_version": "2.3.81" } diff --git a/src/engineering_platform/assets/dashboard.css b/src/engineering_platform/assets/dashboard.css index b9b991e4..d0c8d27d 100644 --- a/src/engineering_platform/assets/dashboard.css +++ b/src/engineering_platform/assets/dashboard.css @@ -1213,7 +1213,7 @@ html[data-theme="light"] .dashboard-health__tooltip .dashboard-health__reason{co /* A missing provider is a blocking error. The generic button hover/focus rule above otherwise wins by specificity and turns the repair action orange, which incorrectly reads as a recoverable warning. */ -.dashboard-status-banner--provider-unavailable .provider-readiness-banner__action{background:#7a2230!important;border-color:#ff91a1!important;color:#fff0f2!important} +.dashboard-status-banner--provider-unavailable .provider-readiness-banner__action{background:#7a2230!important;border-color:#ff91a1!important;color:#ffb4c0!important} .dashboard-status-banner--provider-unavailable .provider-readiness-banner__action:hover:not(:disabled){background:#a92b40!important;border-color:#ffc0c9!important;color:#fff!important} .dashboard-status-banner--provider-unavailable .provider-readiness-banner__action:is(:focus,:focus-visible){box-shadow:0 0 0 4px #ff91a188!important;outline:2px solid #ff91a1!important;outline-color:#ff91a1!important;outline-offset:2px!important} :is(#codexProviderReadinessBanner,#githubProviderReadinessBanner).dashboard-status-banner--provider-unavailable .provider-readiness-banner__action:is(:focus,:focus-visible){box-shadow:0 0 0 4px #ff91a188!important;outline:2px solid #ff91a1!important;outline-color:#ff91a1!important;outline-offset:2px!important} diff --git a/src/engineering_platform/central_operational_reset.py b/src/engineering_platform/central_operational_reset.py new file mode 100644 index 00000000..cb58adae --- /dev/null +++ b/src/engineering_platform/central_operational_reset.py @@ -0,0 +1,2144 @@ +"""Product-owned, fail-closed operational reset for installed EP CENTRAL. + +The service is deliberately local and installation-owner only. It has no +remote route, never discovers a target, and treats every unclassified table or +external path as a blocker. Preview is read-only; all destructive work is +bound to a prepared operation, a verified SQLite backup, and durable +maintenance state. +""" + +from __future__ import annotations + +from contextlib import contextmanager +from datetime import datetime, timezone +import argparse +import hashlib +import json +import os +from pathlib import Path +import pwd +import re +import shutil +import sqlite3 +import stat +import tempfile +from typing import Iterator + +from . import central_database +from .platform_admin import require_installation_owner +from .operational_installation_lock import ( + OperationalInstallationLock, OperationalInstallationLockError, +) +from .storage import sqlite_connection + + +PROFILE = "EP_CENTRAL_OPERATIONAL_HISTORY_V1" +PLAN_VERSION = 1 +SCHEMA_VERSION = 68 +_OPERATION = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{7,127}") +_ACTIVE_STATES = frozenset({ + "PREPARING", "AUTHORIZED", "ARTIFACTS_ARCHIVING", "ARTIFACTS_ARCHIVED", + "DB_APPLIED", "VERIFIED", "FAILED", +}) +_APPLY_ENTRY_STATES = frozenset({ + "AUTHORIZED", "ARTIFACTS_ARCHIVING", "ARTIFACTS_ARCHIVED", +}) +_APPLY_IDEMPOTENT_STATES = frozenset({"DB_APPLIED", "VERIFIED"}) +_RESET_TABLES = frozenset({ + "ep_operational_reset_operations", "ep_operational_dataset_state", + "ep_operational_identity_tombstones", +}) + +# These sets are the schema-owned contract. Prefix matching and implicit +# "everything but configuration" are intentionally forbidden. +INSTALLATION_AND_CONFIGURATION = frozenset({ + "engineering_schema_migrations", "engineering_metadata", "ep_installations", + "ep_project_registrations", "ep_repository_registrations", + "ep_agent_registrations", "ep_agent_repository_attachments", + "ep_local_repository_bindings", "ep_external_producer_bindings", + "execution_migration_provenance", +}) +SECURITY_AND_AUTHORITY_LEDGER = frozenset({ + "ep_agent_pairing_codes", "ep_consumer_credential_recovery_operations", + "ep_consumer_credentials", "ep_consumer_registrations", + "ep_control_provenance", "ep_external_producer_binding_audit", + "ep_operator_capabilities", +}) +DERIVED_CACHE_OR_PROJECTION = frozenset({ + "daily_execution_statistics", "engineering_component_logs", "engineering_status", + "execution_activity_summaries", "execution_projections", "execution_runs", + "provider_usage_snapshots", +}) +OPERATIONAL_HISTORY = frozenset({ + "dependabot_admission_events", "engineering_artifacts", "engineering_transactions", + "ep_execution_host_evidence", "ep_execution_leases", "ep_execution_runs", + "ep_forge_action_context_envelopes", "ep_forge_exchange_audit", + "ep_forge_planning_context_envelopes", "ep_parity_lifecycle_dispatches", + "ep_queue_disposition_operations", "ep_receipt_run_provenance", + "ep_submission_events", "ep_submission_prompt_history", "ep_submissions", + "ep_technical_diagnostics", "ep_terminal_evidence_reconciliation_operations", + "execution_admission_decisions", "execution_artifact_records", + "execution_chat_messages", "execution_dismissals", "execution_emergency_recoveries", + "execution_lease_events", "execution_lifecycle_events", "execution_phase_spans", + "execution_pr_evidence_backfills", "execution_readiness_evaluations", + "execution_receipts", "execution_run_leases", "execution_run_qualification_context", + "execution_run_qualification_snapshots", "execution_run_reconciliations", + "execution_submission_attempt_links", "execution_submission_attempts", + "execution_submission_links", "execution_submissions", + "execution_validation_command_invocations", "execution_validation_command_terminals", + "execution_validation_control_results", "execution_validation_profile_identities", + "execution_validation_profiles", "managed_autonomy_actions", + "managed_governance_gates", "managed_pr_check_observations", + "managed_validation_observations", "prompt_execution_history", + "provider_invocation_receipts", "provider_invocations", + "provider_recovery_attempts", "terminal_telemetry_outbox", +}) +MAINTENANCE_AUDIT = _RESET_TABLES +MAPPED_TABLES = ( + INSTALLATION_AND_CONFIGURATION | SECURITY_AND_AUTHORITY_LEDGER | + DERIVED_CACHE_OR_PROJECTION | OPERATIONAL_HISTORY | MAINTENANCE_AUDIT +) + +# Child-to-parent order for all schema-68 operational records. Deleting a +# table as one statement also handles its self-references. Foreign keys stay +# enabled throughout. +PURGE_ORDER = ( + "ep_terminal_evidence_reconciliation_operations", + "ep_forge_action_context_envelopes", "ep_forge_planning_context_envelopes", + "ep_forge_exchange_audit", "ep_queue_disposition_operations", + "ep_receipt_run_provenance", "ep_parity_lifecycle_dispatches", + "ep_submission_events", "ep_submission_prompt_history", + "execution_chat_messages", "execution_dismissals", "execution_emergency_recoveries", + "execution_validation_command_terminals", "execution_validation_command_invocations", + "execution_admission_decisions", "execution_submission_attempt_links", + "execution_submission_attempts", "execution_submission_links", + "execution_artifact_records", "provider_usage_snapshots", + "provider_invocation_receipts", "provider_recovery_attempts", + "execution_lease_events", "execution_run_reconciliations", + "execution_readiness_evaluations", "execution_run_leases", + "execution_lifecycle_events", "execution_pr_evidence_backfills", + "ep_execution_host_evidence", "ep_execution_leases", + "execution_submissions", "execution_runs", "ep_execution_runs", "ep_submissions", + "engineering_transactions", "prompt_execution_history", + "provider_invocations", "dependabot_admission_events", "engineering_artifacts", + "ep_technical_diagnostics", "execution_phase_spans", "execution_receipts", + "execution_run_qualification_context", "execution_run_qualification_snapshots", + "execution_validation_control_results", "execution_validation_profile_identities", + "execution_validation_profiles", "managed_autonomy_actions", + "managed_governance_gates", "managed_pr_check_observations", + "managed_validation_observations", "terminal_telemetry_outbox", + "engineering_component_logs", "engineering_status", "execution_activity_summaries", + "daily_execution_statistics", "provider_usage_snapshots", "execution_projections", +) + +_OPERATIONAL_METADATA_KEYS = frozenset({ + central_database.MAINTENANCE_LAST_ATTEMPT_KEY, + central_database.PROVIDER_CAPACITY_HISTORY_KEY, +}) +_EFFECT_DIRECTORIES = ("artifacts", "file-inbox", "runtime/central-data-imports") +_ACTIVE_ROOT_MODE = 0o700 +_FROZEN_ROOT_MODE = 0o500 +_KNOWN_TOP_LEVEL = frozenset({ + central_database.DATABASE_FILENAME, f"{central_database.DATABASE_FILENAME}-journal", + f"{central_database.DATABASE_FILENAME}-shm", f"{central_database.DATABASE_FILENAME}-wal", + "server.json", "runtime-identity.json", "runtime.json", "runtime", + "artifacts", "file-inbox", "operations", "recovery", "migration", "backups", + "operational-reset-archive", "operational-reset.lock", "operational-installation.lock", + "operational-installation.json", "legacy-installation-adoption.json", + "development-profile.json", "dependabot-producer-heartbeat.json", + "central.sqlite", ".forge-ep-consumer-recovery.lock", +}) +_VERSIONED_RECOVERY_BACKUP = re.compile( + r"epdata\.sqlite\.pre-(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.backup" +) +_MAINTENANCE_GUARD_TRIGGERS = frozenset({ + "ep_operational_reset_operations_insert_owner", + "ep_operational_reset_operations_update_owner", + "ep_operational_reset_operations_delete_immutable", + "ep_operational_dataset_state_insert_owner", + "ep_operational_dataset_state_update_owner", + "ep_operational_dataset_state_delete_immutable", + "ep_operational_identity_tombstones_insert_owner", + "ep_operational_identity_tombstones_update_immutable", + "ep_operational_identity_tombstones_delete_immutable", +}) + + +class OperationalResetError(RuntimeError): + """Stable fail-closed maintenance error.""" + + def __init__(self, code: str, detail: str = "") -> None: + super().__init__(code if not detail else f"{code}: {detail}") + self.code = code + + +@contextmanager +def _central_connection(*arguments: object, **keywords: object) -> Iterator[sqlite3.Connection]: + """Open owning CENTRAL with enforced referential integrity for maintenance.""" + with sqlite_connection(*arguments, **keywords) as connection: + # Persistent maintenance triggers call this connection-local function. + # Normal product connections cannot forge owning reset/audit writes. + connection.create_function("ep_reset_maintenance_owner", 0, lambda: 1) + connection.execute("PRAGMA foreign_keys=ON") + connection.execute("PRAGMA busy_timeout=10000") + yield connection + + +def _utcnow() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _canonical(value: object) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("ascii") + + +def _digest(value: object) -> str: + return "sha256:" + hashlib.sha256(_canonical(value)).hexdigest() + + +def _file_digest(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(1024 * 1024): + digest.update(chunk) + return "sha256:" + digest.hexdigest() + + +def _quote(identifier: str) -> str: + return '"' + identifier.replace('"', '""') + '"' + + +def _trusted_directory(path: Path, *, code: str, must_exist: bool = True) -> Path: + """Resolve one operator-selected directory without following symlink components.""" + candidate = path.expanduser().absolute() + # macOS exposes the root-owned compatibility alias /var -> /private/var. + # Normalize that fixed system alias before rejecting any operator-selected + # symlink component below it. + if len(candidate.parts) >= 2 and candidate.parts[:2] == ("/", "var"): + system_var = Path("/var") + if system_var.is_symlink() and system_var.resolve() == Path("/private/var"): + candidate = Path("/private/var", *candidate.parts[2:]) + component = Path(candidate.anchor) + for part in candidate.parts[1:]: + component /= part + if component.is_symlink(): + raise OperationalResetError(code) + if not component.exists(): + break + try: + resolved = candidate.resolve(strict=must_exist) + except OSError as error: + raise OperationalResetError(code) from error + if must_exist and not resolved.is_dir(): + raise OperationalResetError(code) + return resolved + + +def _tables(connection: sqlite3.Connection) -> set[str]: + return { + str(row[0]) for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'" + ) + } + + +def _schema_objects(connection: sqlite3.Connection) -> list[dict[str, object]]: + """Inventory non-table schema objects without treating them as datasets.""" + return [ + { + "type": str(kind), "name": str(name), "table": str(table), + "definition_digest": _digest(str(sql)) if sql is not None else None, + "effect": "PRESERVE", + } + for kind, name, table, sql in connection.execute( + "SELECT type,name,tbl_name,sql FROM sqlite_master " + "WHERE type IN ('index','trigger','view') AND name NOT LIKE 'sqlite_%' " + "ORDER BY type,name" + ) + ] + + +def _install_writer_blocks(connection: sqlite3.Connection) -> None: + for table in sorted(_tables(connection) - _RESET_TABLES): + for operation in ("INSERT", "UPDATE", "DELETE"): + name = f"ep_operational_reset_block_{table}_{operation.casefold()}" + connection.execute( + f"CREATE TRIGGER IF NOT EXISTS {_quote(name)} BEFORE {operation} ON {_quote(table)} " + "WHEN EXISTS(SELECT 1 FROM ep_operational_reset_operations " + "WHERE state NOT IN ('COMPLETED','ABORTED')) BEGIN " + "SELECT RAISE(ABORT,'EP_OPERATIONAL_MAINTENANCE_ACTIVE'); END" + ) + + +def _expected_writer_fences(tables: set[str]) -> set[str]: + return { + f"ep_operational_reset_block_{table}_{operation.casefold()}" + for table in tables - _RESET_TABLES + for operation in ("INSERT", "UPDATE", "DELETE") + } | set(_MAINTENANCE_GUARD_TRIGGERS) + + +def _install_maintenance_guards(connection: sqlite3.Connection) -> None: + """Make reset state, audit bindings and tombstones owner-only/immutable.""" + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_reset_operations_insert_owner " + "BEFORE INSERT ON ep_operational_reset_operations " + "WHEN ep_reset_maintenance_owner()!=1 BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_MAINTENANCE_OWNER_REQUIRED'); END" + ) + # Recreate this schema-owned trigger when the maintenance audit gains a + # new immutable effect field within the same unreleased schema revision. + connection.execute("DROP TRIGGER IF EXISTS ep_operational_reset_operations_update_owner") + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_reset_operations_update_owner " + "BEFORE UPDATE ON ep_operational_reset_operations BEGIN " + "SELECT CASE WHEN ep_reset_maintenance_owner()!=1 THEN " + "RAISE(ABORT,'EP_RESET_MAINTENANCE_OWNER_REQUIRED') END; " + "SELECT CASE WHEN NEW.operation_id!=OLD.operation_id OR NEW.actor!=OLD.actor " + "OR NEW.plan_digest!=OLD.plan_digest OR NEW.target_digest!=OLD.target_digest " + "OR NEW.source_revision!=OLD.source_revision OR NEW.request_digest!=OLD.request_digest " + "OR NEW.plan_json!=OLD.plan_json OR NEW.allowed_fk_json!=OLD.allowed_fk_json " + "OR NEW.backup_root!=OLD.backup_root OR NEW.generation_before!=OLD.generation_before " + "OR NEW.created_at!=OLD.created_at THEN " + "RAISE(ABORT,'EP_RESET_AUDIT_BINDING_IMMUTABLE') END; " + "SELECT CASE WHEN ((NEW.backup_path IS NOT OLD.backup_path " + "OR NEW.backup_sha256 IS NOT OLD.backup_sha256 " + "OR NEW.backup_manifest_digest IS NOT OLD.backup_manifest_digest) " + "AND NOT (OLD.state='PREPARING' AND NEW.state='AUTHORIZED')) " + "OR (OLD.state='PREPARING' AND NEW.state='AUTHORIZED' AND " + "(NEW.backup_path IS NULL OR NEW.backup_sha256 IS NULL " + "OR NEW.backup_manifest_digest IS NULL)) " + "OR (NEW.generation_after IS NOT OLD.generation_after AND NOT " + "(OLD.state='ARTIFACTS_ARCHIVED' AND NEW.state='DB_APPLIED')) " + "OR (NEW.finish_boundary_path IS NOT OLD.finish_boundary_path AND NOT " + "(OLD.state='VERIFIED' AND NEW.state='VERIFIED' " + "AND OLD.finish_boundary_path IS NULL AND NEW.finish_boundary_path IS NOT NULL " + "AND NEW.finish_boundary_digest IS OLD.finish_boundary_digest)) " + "OR (NEW.finish_boundary_digest IS NOT OLD.finish_boundary_digest AND NOT " + "(OLD.state='VERIFIED' AND NEW.state='COMPLETED' " + "AND OLD.finish_boundary_digest IS NULL AND NEW.finish_boundary_digest IS NOT NULL)) " + "OR (OLD.state='VERIFIED' AND NEW.state='COMPLETED' AND " + "(NEW.finish_boundary_path IS NULL OR NEW.finish_boundary_digest IS NULL)) " + "OR (NEW.verification_json IS NOT OLD.verification_json AND NOT " + "((OLD.state='DB_APPLIED' AND NEW.state='VERIFIED') OR " + "(OLD.state='VERIFIED' AND NEW.state='COMPLETED'))) THEN " + "RAISE(ABORT,'EP_RESET_AUDIT_EFFECT_IMMUTABLE') END; " + "SELECT CASE WHEN NEW.state!=OLD.state AND NOT (" + "(OLD.state='PREPARING' AND NEW.state IN ('AUTHORIZED','ABORTED','FAILED')) OR " + "(OLD.state='AUTHORIZED' AND NEW.state IN ('ARTIFACTS_ARCHIVING','ABORTED','FAILED')) OR " + "(OLD.state='ARTIFACTS_ARCHIVING' AND NEW.state IN ('ARTIFACTS_ARCHIVED','FAILED')) OR " + "(OLD.state='ARTIFACTS_ARCHIVED' AND NEW.state IN ('DB_APPLIED','FAILED')) OR " + "(OLD.state='DB_APPLIED' AND NEW.state IN ('VERIFIED','FAILED')) OR " + "(OLD.state='VERIFIED' AND NEW.state IN ('COMPLETED','FAILED'))) THEN " + "RAISE(ABORT,'EP_RESET_STATE_TRANSITION_INVALID') END; END" + ) + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_reset_operations_delete_immutable " + "BEFORE DELETE ON ep_operational_reset_operations BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_AUDIT_IMMUTABLE'); END" + ) + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_dataset_state_insert_owner " + "BEFORE INSERT ON ep_operational_dataset_state " + "WHEN ep_reset_maintenance_owner()!=1 BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_MAINTENANCE_OWNER_REQUIRED'); END" + ) + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_dataset_state_update_owner " + "BEFORE UPDATE ON ep_operational_dataset_state BEGIN " + "SELECT CASE WHEN ep_reset_maintenance_owner()!=1 THEN " + "RAISE(ABORT,'EP_RESET_MAINTENANCE_OWNER_REQUIRED') END; " + "SELECT CASE WHEN NEW.singleton!=OLD.singleton OR NEW.generation NOT IN " + "(OLD.generation,OLD.generation+1) THEN " + "RAISE(ABORT,'EP_RESET_DATASET_GENERATION_INVALID') END; END" + ) + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_dataset_state_delete_immutable " + "BEFORE DELETE ON ep_operational_dataset_state BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_DATASET_STATE_IMMUTABLE'); END" + ) + connection.execute( + "CREATE TRIGGER IF NOT EXISTS ep_operational_identity_tombstones_insert_owner " + "BEFORE INSERT ON ep_operational_identity_tombstones " + "WHEN ep_reset_maintenance_owner()!=1 BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_MAINTENANCE_OWNER_REQUIRED'); END" + ) + for operation in ("UPDATE", "DELETE"): + connection.execute( + f"CREATE TRIGGER IF NOT EXISTS ep_operational_identity_tombstones_{operation.casefold()}_immutable " + f"BEFORE {operation} ON ep_operational_identity_tombstones BEGIN " + "SELECT RAISE(ABORT,'EP_RESET_TOMBSTONE_IMMUTABLE'); END" + ) + + +def install_schema(connection: sqlite3.Connection) -> None: + """Install schema-68 maintenance state and the canonical chat relation.""" + tables = _tables(connection) + if "execution_chat_messages" in tables: + foreign_keys = list(connection.execute("PRAGMA foreign_key_list(execution_chat_messages)")) + parent = str(foreign_keys[0][2]) if foreign_keys else "" + if parent != "ep_execution_runs": + missing = connection.execute( + "SELECT chat.id FROM execution_chat_messages AS chat " + "LEFT JOIN ep_execution_runs AS run ON run.run_id=chat.run_id " + "WHERE run.run_id IS NULL LIMIT 1" + ).fetchone() + if missing is not None: + raise OperationalResetError( + "CHAT_CANONICAL_RUN_PARENT_MISSING", str(missing[0]) + ) + connection.execute("DROP TRIGGER IF EXISTS execution_chat_messages_immutable_update") + connection.execute("DROP INDEX IF EXISTS execution_chat_messages_run_created") + connection.execute( + "CREATE TABLE execution_chat_messages_schema68 (" + "id INTEGER PRIMARY KEY,run_id TEXT NOT NULL REFERENCES ep_execution_runs(run_id)," + "role TEXT NOT NULL CHECK(role IN ('user','assistant')),content TEXT NOT NULL," + "model TEXT,created_at TEXT NOT NULL)" + ) + connection.execute( + "INSERT INTO execution_chat_messages_schema68(id,run_id,role,content,model,created_at) " + "SELECT id,run_id,role,content,model,created_at FROM execution_chat_messages" + ) + connection.execute("DROP TABLE execution_chat_messages") + connection.execute( + "ALTER TABLE execution_chat_messages_schema68 RENAME TO execution_chat_messages" + ) + connection.execute( + "CREATE INDEX execution_chat_messages_run_created " + "ON execution_chat_messages(run_id,id)" + ) + connection.execute( + "CREATE TRIGGER execution_chat_messages_immutable_update " + "BEFORE UPDATE ON execution_chat_messages BEGIN " + "SELECT RAISE(ABORT,'Chat transcript messages are immutable.'); END" + ) + connection.execute( + "CREATE TABLE IF NOT EXISTS ep_operational_dataset_state (" + "singleton INTEGER PRIMARY KEY CHECK(singleton=1),generation INTEGER NOT NULL CHECK(generation>=0)," + "updated_at TEXT NOT NULL)" + ) + if connection.execute( + "SELECT 1 FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone() is None: + connection.execute( + "INSERT INTO ep_operational_dataset_state(singleton,generation,updated_at) " + "VALUES(1,0,CURRENT_TIMESTAMP)" + ) + connection.execute( + "CREATE TABLE IF NOT EXISTS ep_operational_reset_operations (" + "operation_id TEXT PRIMARY KEY,state TEXT NOT NULL CHECK(state IN (" + "'PREPARING','AUTHORIZED','ARTIFACTS_ARCHIVING','ARTIFACTS_ARCHIVED'," + "'DB_APPLIED','VERIFIED','COMPLETED','ABORTED','FAILED'))," \ + "actor TEXT NOT NULL,plan_digest TEXT NOT NULL,target_digest TEXT NOT NULL," + "source_revision TEXT NOT NULL,request_digest TEXT NOT NULL,plan_json TEXT NOT NULL," + "allowed_fk_json TEXT NOT NULL,backup_root TEXT NOT NULL,backup_path TEXT,backup_sha256 TEXT," + "backup_manifest_digest TEXT,generation_before INTEGER NOT NULL,generation_after INTEGER," + "finish_boundary_path TEXT,finish_boundary_digest TEXT," + "created_at TEXT NOT NULL,updated_at TEXT NOT NULL,verification_json TEXT)" + ) + operation_columns = { + str(row[1]) for row in connection.execute( + "PRAGMA table_info(ep_operational_reset_operations)" + ) + } + for column in ("finish_boundary_path", "finish_boundary_digest"): + if column not in operation_columns: + connection.execute( + f"ALTER TABLE ep_operational_reset_operations ADD COLUMN {column} TEXT" + ) + connection.execute( + "CREATE UNIQUE INDEX IF NOT EXISTS ep_operational_reset_one_active " + "ON ep_operational_reset_operations((1)) WHERE state NOT IN ('COMPLETED','ABORTED')" + ) + connection.execute( + "CREATE TABLE IF NOT EXISTS ep_operational_identity_tombstones (" + "identity_kind TEXT NOT NULL,identity_digest TEXT NOT NULL,request_digest TEXT," + "operation_id TEXT NOT NULL REFERENCES ep_operational_reset_operations(operation_id)," + "recorded_at TEXT NOT NULL,PRIMARY KEY(identity_kind,identity_digest))" + ) + _install_maintenance_guards(connection) + _install_writer_blocks(connection) + + +def _actor(data_root: Path) -> str: + uid = require_installation_owner(data_root) + try: + name = pwd.getpwuid(int(uid)).pw_name + except (KeyError, ValueError): + name = "unknown" + return f"uid:{uid}:{name}" + + +def _identity(data_root: Path, connection: sqlite3.Connection) -> dict[str, object]: + try: + file_identity = json.loads((data_root / "runtime-identity.json").read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise OperationalResetError("TARGET_IDENTITY_UNAVAILABLE") from error + row = connection.execute("SELECT instance_id,schema_version FROM ep_installations").fetchone() + metadata = connection.execute( + "SELECT value FROM engineering_metadata WHERE key='installation.instance_id'" + ).fetchone() + if ( + not isinstance(file_identity, dict) or not isinstance(file_identity.get("instance_id"), str) + or row is None or metadata is None or file_identity["instance_id"] != row[0] != "" + or row[0] != metadata[0] + ): + raise OperationalResetError("TARGET_IDENTITY_CONFLICT") + database = (data_root / central_database.DATABASE_FILENAME).resolve() + stat = database.stat() + return { + "product": "engineering-platform", "instance_id": str(row[0]), + "data_root": str(data_root), "database": str(database), "schema_version": int(row[1]), + "database_device": stat.st_dev, "database_inode": stat.st_ino, + } + + +def _row_count(connection: sqlite3.Connection, table: str) -> int: + if table == "execution_projections": + return int(connection.execute( + "SELECT COUNT(*) FROM execution_projections WHERE classification!='CONFIGURATION'" + ).fetchone()[0]) + if table == "engineering_metadata": + placeholders = ",".join("?" for _ in _OPERATIONAL_METADATA_KEYS) + return int(connection.execute( + f"SELECT COUNT(*) FROM engineering_metadata WHERE key IN ({placeholders})", + tuple(sorted(_OPERATIONAL_METADATA_KEYS)), + ).fetchone()[0]) + return int(connection.execute(f"SELECT COUNT(*) FROM {_quote(table)}").fetchone()[0]) + + +def _total_count(connection: sqlite3.Connection, table: str) -> int: + return int(connection.execute(f"SELECT COUNT(*) FROM {_quote(table)}").fetchone()[0]) + + +def _value_bytes(value: object) -> bytes: + if value is None: + return b"N" + if isinstance(value, bytes): + return b"B" + value + return (type(value).__name__[0] + str(value)).encode("utf-8", "surrogateescape") + + +def _logical_digest( + connection: sqlite3.Connection, tables: set[str], *, preserved_only: bool = False, +) -> str: + digest = hashlib.sha256() + for table in sorted(tables - _RESET_TABLES): + columns = [str(row[1]) for row in connection.execute(f"PRAGMA table_info({_quote(table)})")] + pk = [str(row[1]) for row in connection.execute(f"PRAGMA table_info({_quote(table)})") if int(row[5])] + where, parameters = "", () + if table == "engineering_metadata" and preserved_only: + placeholders = ",".join("?" for _ in _OPERATIONAL_METADATA_KEYS) + where, parameters = f" WHERE key NOT IN ({placeholders})", tuple(sorted(_OPERATIONAL_METADATA_KEYS)) + elif table == "execution_projections" and preserved_only: + where = " WHERE classification='CONFIGURATION'" + elif preserved_only and table != "execution_projections" and table not in INSTALLATION_AND_CONFIGURATION | SECURITY_AND_AUTHORITY_LEDGER: + continue + order = ",".join(_quote(item) for item in (pk or columns)) + query = f"SELECT {','.join(_quote(item) for item in columns)} FROM {_quote(table)}{where} ORDER BY {order}" + digest.update(table.encode("utf-8") + b"\0") + for row in connection.execute(query, parameters): + for value in row: + encoded = _value_bytes(value) + digest.update(len(encoded).to_bytes(8, "big") + encoded) + return "sha256:" + digest.hexdigest() + + +def _walk_regular_files(directory: Path, *, code: str) -> list[Path]: + """Walk without following any symlink or special-file boundary.""" + if not directory.exists(): + return [] + if directory.is_symlink() or not directory.is_dir(): + raise OperationalResetError(code) + files: list[Path] = [] + pending = [directory] + while pending: + current = pending.pop() + try: + entries = sorted(os.scandir(current), key=lambda entry: entry.name) + except OSError as error: + raise OperationalResetError(code) from error + for entry in entries: + try: + if entry.is_symlink(): + raise OperationalResetError(code) + if entry.is_dir(follow_symlinks=False): + pending.append(Path(entry.path)) + elif entry.is_file(follow_symlinks=False): + files.append(Path(entry.path)) + else: + raise OperationalResetError(code) + except OSError as error: + raise OperationalResetError(code) from error + return sorted(files) + + +def _safe_files(directory: Path, logical_root: str) -> list[dict[str, object]]: + rows: list[dict[str, object]] = [] + for path in _walk_regular_files(directory, code="EXTERNAL_SYMLINK_UNSAFE"): + relative = path.relative_to(directory).as_posix() + rows.append({ + "root": logical_root, "path": relative, "size_bytes": path.stat().st_size, + "sha256": _file_digest(path), "effect": "ARCHIVE_OUTSIDE_ACTIVE_ROUTE", + }) + return rows + + +def _known_top_level(name: str) -> bool: + return name in _KNOWN_TOP_LEVEL or _VERSIONED_RECOVERY_BACKUP.fullmatch(name) is not None + + +def _nested_preserved_classification(root_name: str, relative: str, path: Path) -> str | None: + """Classify only product-owned shapes inside preserved top-level roots.""" + parts = Path(relative).parts + if root_name == "runtime": + if relative in { + "store-authority.json", "engineering-dashboard-relay", + "server-launchagent.out.log", "server-launchagent.err.log", + }: + return "INSTALLATION_RUNTIME" + if relative == "pending-central-data-import.json": + return "ACTIVE_INGEST_CONTROL" + return None + if root_name == "operations": + if len(parts) == 1 and path.is_file() and parts[0].endswith(".json"): + return "INSTALLATION_OPERATION_AUDIT" + if parts and _OPERATION.fullmatch(parts[0]) is not None: + operation_root = path.parents[len(parts) - 2] + if (operation_root / "operation.json").is_file() and len(parts) >= 2: + if parts[1] in {"operation.json", "candidate-runtime.json"} and len(parts) == 2: + return "INSTALLATION_OPERATION_AUDIT" + if parts[1:] == ("backup", "central.sqlite"): + return "FORENSIC_OR_RECOVERY" + if parts[1] in {"build", "download", "pip-cache", "candidate-venv"}: + return "INSTALLATION_RUNTIME_STAGING" + return None + if root_name == "migration": + if (len(parts) == 1 and parts[0].endswith(".json")) or ( + len(parts) == 2 and parts[0] == "contamination-attestations" + and parts[1].endswith(".json") + ): + return "MIGRATION_AUDIT" + return None + if root_name == "backups": + if len(parts) == 1 and re.fullmatch(r"legacy-schema40-[A-Za-z0-9._-]+\.db", parts[0]): + return "FORENSIC_OR_RECOVERY" + return None + if root_name == "recovery": + # The topology reserves this root, but no current product writer owns + # an arbitrary nested payload shape. Fail closed until one is defined. + return None + if root_name == "operational-reset-archive": + if ( + len(parts) >= 3 and _OPERATION.fullmatch(parts[0]) is not None + and (parts[1] in {"artifacts", "file-inbox"} + or parts[1:3] == ("runtime", "central-data-imports")) + ): + return "MAINTENANCE_AUDIT_OR_RECOVERY" + if ( + len(parts) >= 4 and _OPERATION.fullmatch(parts[0]) is not None + and parts[1] == "finish-boundary" + and re.fullmatch(r"generation-(?:0|[1-9][0-9]*)", parts[2]) is not None + and ( + parts[3] in { + "artifacts", "file-inbox", "manifest.json", "manifest.json.partial", + } + or parts[3:5] == ("runtime", "central-data-imports") + ) + ): + return "MAINTENANCE_AUDIT_OR_RECOVERY" + return None + return None + + +def _external_inventory( + data_root: Path, +) -> tuple[list[dict[str, object]], list[dict[str, object]], list[str], list[str]]: + unknown = sorted(path.name for path in data_root.iterdir() if not _known_top_level(path.name)) + rows: list[dict[str, object]] = [] + for name in _EFFECT_DIRECTORIES: + rows.extend(_safe_files(data_root / Path(name), name)) + preserved: list[dict[str, object]] = [] + active_ingest_controls: list[str] = [] + for path in sorted(data_root.iterdir(), key=lambda item: item.name): + if path.name in {Path(name).parts[0] for name in _EFFECT_DIRECTORIES} or not _known_top_level(path.name): + if path.name != "runtime": + continue + if path.name in {"artifacts", "file-inbox"}: + continue + if path.name == central_database.DATABASE_FILENAME: + classification = "INSTALLATION_AND_CONFIGURATION" + elif path.name in {f"{central_database.DATABASE_FILENAME}-journal", + f"{central_database.DATABASE_FILENAME}-shm", + f"{central_database.DATABASE_FILENAME}-wal"}: + classification = "SYSTEM_RUNTIME_CONTROL" + elif path.name in {"central.sqlite"} or _VERSIONED_RECOVERY_BACKUP.fullmatch(path.name): + classification = "FORENSIC_OR_RECOVERY" + elif path.name.endswith(".lock") or path.name in { + "runtime.json", "dependabot-producer-heartbeat.json", + }: + classification = "SYSTEM_RUNTIME_CONTROL" + elif path.name in {"operations", "recovery", "migration", "backups", "operational-reset-archive"}: + classification = "MAINTENANCE_AUDIT_OR_RECOVERY" + else: + classification = "INSTALLATION_AND_CONFIGURATION" + entry: dict[str, object] = { + "path": path.name, "classification": classification, "effect": "PRESERVE", + "kind": "symlink" if path.is_symlink() else ("directory" if path.is_dir() else "file"), + } + if path.is_file() and not path.is_symlink() and path.name != central_database.DATABASE_FILENAME: + entry.update({"size_bytes": path.stat().st_size, "sha256": _file_digest(path)}) + preserved.append(entry) + if path.is_dir() and not path.is_symlink() and path.name in { + "runtime", "operations", "recovery", "migration", "backups", + "operational-reset-archive", + }: + for nested in _walk_regular_files(path, code="EXTERNAL_SYMLINK_UNSAFE"): + relative = nested.relative_to(path).as_posix() + if path.name == "runtime" and relative.startswith("central-data-imports/"): + continue + classification = _nested_preserved_classification(path.name, relative, nested) + if classification is None: + unknown.append(f"{path.name}/{relative}") + continue + preserved.append({ + "path": f"{path.name}/{relative}", "classification": classification, + "effect": "PRESERVE", "kind": "file", "size_bytes": nested.stat().st_size, + "sha256": _file_digest(nested), + }) + if classification == "ACTIVE_INGEST_CONTROL": + active_ingest_controls.append(f"{path.name}/{relative}") + return rows, preserved, sorted(set(unknown)), active_ingest_controls + + +def _runtime_activity(data_root: Path) -> dict[str, object]: + try: + payload = json.loads((data_root / "runtime.json").read_text(encoding="utf-8")) + except FileNotFoundError: + return {"active": False, "reason": "NO_RUNTIME_RECORD"} + except (OSError, json.JSONDecodeError): + return {"active": True, "reason": "RUNTIME_IDENTITY_UNREADABLE"} + pid = payload.get("pid") if isinstance(payload, dict) else None + if not isinstance(pid, int) or pid <= 0: + return {"active": True, "reason": "RUNTIME_IDENTITY_INVALID"} + try: + os.kill(pid, 0) + except ProcessLookupError: + return {"active": False, "reason": "STALE_RUNTIME_RECORD"} + except PermissionError: + pass + return {"active": True, "reason": "TARGET_RUNTIME_ACTIVE", "pid": pid} + + +def _foreign_key_findings(connection: sqlite3.Connection) -> list[dict[str, object]]: + findings: list[dict[str, object]] = [] + for table, rowid, parent, fk_index in connection.execute("PRAGMA foreign_key_check"): + classification = ( + "OPERATIONAL_PURGE_SET" if table in OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION + and parent in OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION else + "PRESERVATION_OR_UNKNOWN_SCOPE" + ) + finding_id = f"fk:{table}:{rowid}:{parent}:{fk_index}" + findings.append({ + "id": finding_id, "table": str(table), "rowid": rowid, "parent": str(parent), + "foreign_key_index": int(fk_index), "classification": classification, + }) + return findings + + +def preview(data_root: Path) -> dict[str, object]: + """Build a read-only, canonical plan for exactly one installed CENTRAL.""" + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + database = root / central_database.DATABASE_FILENAME + try: + with _central_connection(f"file:{database}?mode=ro", uri=True) as connection: + connection.execute("PRAGMA query_only=ON") + tables = _tables(connection) + unknown_tables = sorted(tables - MAPPED_TABLES) + missing_tables = sorted(MAPPED_TABLES - tables) + identity = _identity(root, connection) + schema = max((int(row[0]) for row in connection.execute( + "SELECT version FROM engineering_schema_migrations" + )), default=0) + quick = [str(row[0]) for row in connection.execute("PRAGMA quick_check")] + foreign_keys = _foreign_key_findings(connection) + counts = { + "INSTALLATION_AND_CONFIGURATION": { + table: ( + _total_count(connection, table) - _row_count(connection, table) + if table == "engineering_metadata" else _total_count(connection, table) + ) for table in sorted(INSTALLATION_AND_CONFIGURATION & tables) + }, + "SECURITY_AND_AUTHORITY_LEDGER": { + table: _row_count(connection, table) for table in sorted(SECURITY_AND_AUTHORITY_LEDGER & tables) + }, + "OPERATIONAL_HISTORY": { + table: _row_count(connection, table) for table in sorted(OPERATIONAL_HISTORY & tables) + }, + "DERIVED_CACHE_OR_PROJECTION": { + table: _row_count(connection, table) for table in sorted(DERIVED_CACHE_OR_PROJECTION & tables) + }, + # Own audit writes cannot invalidate their operation's plan. + "MAINTENANCE_AUDIT": { + table: "EXCLUDED_FROM_PLAN_REVISION" for table in sorted(MAINTENANCE_AUDIT & tables) + }, + } + counts["INSTALLATION_AND_CONFIGURATION"]["execution_projections[CONFIGURATION]"] = ( + _total_count(connection, "execution_projections") - _row_count(connection, "execution_projections") + ) if "execution_projections" in tables else 0 + counts["DERIVED_CACHE_OR_PROJECTION"]["engineering_metadata[operational_keys]"] = ( + _row_count(connection, "engineering_metadata") + ) if "engineering_metadata" in tables else 0 + source_revision = _logical_digest(connection, tables) + preserved_digest = _logical_digest(connection, tables, preserved_only=True) + generation = int(connection.execute( + "SELECT generation FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone()[0]) if "ep_operational_dataset_state" in tables else -1 + schema_objects = _schema_objects(connection) + installed_triggers = { + str(row[0]) for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type='trigger'" + ) + } + missing_writer_fences = sorted( + _expected_writer_fences(tables) - installed_triggers + ) + except (OSError, sqlite3.DatabaseError, TypeError, ValueError) as error: + raise OperationalResetError("CENTRAL_INSPECTION_FAILED") from error + external_error: str | None = None + try: + external, preserved_external, unknown_paths, active_ingest_controls = _external_inventory(root) + except OperationalResetError as error: + external, preserved_external, unknown_paths, active_ingest_controls = [], [], [], [] + external_error = error.code + unsafe_preserved_paths = [ + str(item["path"]) for item in preserved_external if item["kind"] == "symlink" + ] + activity = _runtime_activity(root) + blockers: list[str] = [] + if schema != SCHEMA_VERSION or int(identity["schema_version"]) != SCHEMA_VERSION: + blockers.append("SCHEMA_UNSUPPORTED") + if quick != ["ok"]: + blockers.append("PHYSICAL_INTEGRITY_FAILED") + if unknown_tables or missing_tables: + blockers.append("TABLE_CLASSIFICATION_INCOMPLETE") + if unknown_paths: + blockers.append("EXTERNAL_CLASSIFICATION_INCOMPLETE") + if active_ingest_controls: + blockers.append("ACTIVE_INGEST_PENDING") + if missing_writer_fences: + blockers.append("WRITER_FENCE_INCOMPLETE") + if unsafe_preserved_paths: + blockers.append("EXTERNAL_SYMLINK_UNSAFE") + if external_error is not None: + blockers.append( + "EXTERNAL_SYMLINK_UNSAFE" if external_error in { + "EXTERNAL_PATH_UNSAFE", "EXTERNAL_SYMLINK_UNSAFE", + } else external_error + ) + if any(item["classification"] != "OPERATIONAL_PURGE_SET" for item in foreign_keys): + blockers.append("PRESERVATION_INTEGRITY_FAILED") + if activity["active"]: + blockers.append("TARGET_WRITER_ACTIVE") + review = [str(item["id"]) for item in foreign_keys if item["classification"] == "OPERATIONAL_PURGE_SET"] + target_digest = _digest(identity) + effect_set = { + "database_tables": sorted(OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION), + "mixed_record_rules": { + "engineering_metadata": {"purge_keys": sorted(_OPERATIONAL_METADATA_KEYS)}, + "execution_projections": {"preserve_classification": "CONFIGURATION"}, + }, + "external_roots": list(_EFFECT_DIRECTORIES), + } + plan_basis = { + "plan_version": PLAN_VERSION, "profile": PROFILE, "target": identity, + "target_digest": target_digest, "source_revision": source_revision, + "preserved_bindings_digest": preserved_digest, "dataset_generation": generation, + "classification_counts": counts, "foreign_key_findings": foreign_keys, + "external_files": external, + "preserved_external": [ + item for item in preserved_external + if item["classification"] != "SYSTEM_RUNTIME_CONTROL" + ], + "unsafe_preserved_paths": unsafe_preserved_paths, + "schema_objects": schema_objects, + "missing_writer_fences": missing_writer_fences, + "external_inventory_error": external_error, + "active_ingest_controls": active_ingest_controls, + "unknown_tables": unknown_tables, + "missing_tables": missing_tables, "unknown_external_paths": unknown_paths, + "effect_set": effect_set, + } + return { + **plan_basis, "plan_digest": _digest(plan_basis), "quick_check": quick, + "runtime_activity": activity, "blocking_codes": sorted(set(blockers)), + "review_required": review, + "runtime_control_external": [ + item for item in preserved_external + if item["classification"] == "SYSTEM_RUNTIME_CONTROL" + ], + "execution_state": "BLOCKED" if blockers else ("REVIEW_REQUIRED" if review else "ALLOWED"), + "namespace": { + "runtime_runs": "EP CENTRAL run/submission identifiers", + "forge_missions": "external Forge runtime namespace; not created or reset here", + "repository_document": "missions/MISSION-0003.md is repository documentation only", + }, + } + + +@contextmanager +def _operation_lock(data_root: Path, operation_id: str) -> Iterator[None]: + if _OPERATION.fullmatch(operation_id) is None: + raise OperationalResetError("OPERATION_ID_INVALID") + lock = OperationalInstallationLock(data_root) + try: + try: + lock.acquire(operation_id) + except OperationalInstallationLockError as error: + raise OperationalResetError("MAINTENANCE_LOCK_BUSY") from error + yield + finally: + try: + lock.release(operation_id) + except OperationalInstallationLockError: + pass + + +def _backup_destination(backup_root: Path, operation_id: str) -> tuple[Path, Path | None]: + expanded = backup_root.expanduser().absolute() + if expanded.is_symlink(): + raise OperationalResetError("BACKUP_PATH_UNSAFE") + existed = expanded.exists() + root = _trusted_directory( + expanded, code="BACKUP_PATH_UNSAFE", must_exist=existed, + ) + if existed: + if root.stat().st_mode & 0o077: + raise OperationalResetError("BACKUP_PERMISSIONS_UNSAFE") + else: + root.mkdir(mode=0o700, parents=True, exist_ok=False) + root.chmod(0o700) + destination = root / operation_id + if destination.is_symlink(): + raise OperationalResetError("BACKUP_PATH_UNSAFE") + if destination.exists(): + if not destination.is_dir(): + raise OperationalResetError("BACKUP_PATH_UNSAFE") + try: + if next(destination.iterdir(), None) is not None: + # A crash may have atomically published the exact same complete + # backup before its DB binding was stored. The caller may only + # reuse it after full operation/plan verification. + return destination, None + except OSError as error: + raise OperationalResetError("BACKUP_PATH_UNSAFE") from error + raise OperationalResetError("BACKUP_DESTINATION_PREEXISTS") + staging = Path(tempfile.mkdtemp(prefix=f".{operation_id}.", suffix=".partial", dir=root)) + staging.chmod(0o700) + return staging, destination + + +def _bound_regular_file(root: Path, relative: str, *, code: str) -> Path: + logical = Path(relative) + if logical.is_absolute() or not logical.parts or any(part in {"", ".", ".."} for part in logical.parts): + raise OperationalResetError(code) + parent = _trusted_directory(root / logical.parent, code=code) + try: + parent.relative_to(root) + except ValueError as error: + raise OperationalResetError(code) from error + candidate = parent / logical.name + try: + metadata = candidate.lstat() + except OSError as error: + raise OperationalResetError(code) from error + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OperationalResetError(code) + return candidate + + +def _secure_mkdirs(root: Path, relative: Path, *, code: str) -> Path: + """Create a relative directory tree only after checking every component.""" + if relative.is_absolute() or any(part in {"", ".", ".."} for part in relative.parts): + raise OperationalResetError(code) + current = root + for part in relative.parts: + current = current / part + try: + metadata = current.lstat() + except FileNotFoundError: + try: + current.mkdir(mode=0o700) + metadata = current.lstat() + except OSError as error: + raise OperationalResetError(code) from error + except OSError as error: + raise OperationalResetError(code) from error + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OperationalResetError(code) + return current + + +def _exclusive_copy(source: Path, target: Path) -> None: + """Copy one regular file without following source or destination symlinks.""" + flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) + source_fd = os.open(source, flags) + try: + metadata = os.fstat(source_fd) + if not stat.S_ISREG(metadata.st_mode): + raise OperationalResetError("BACKUP_SOURCE_CHANGED") + target_fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(source_fd, "rb", closefd=False) as source_handle, os.fdopen( + target_fd, "wb", closefd=False, + ) as target_handle: + shutil.copyfileobj(source_handle, target_handle, length=1024 * 1024) + finally: + os.close(target_fd) + except FileExistsError as error: + raise OperationalResetError("BACKUP_DESTINATION_CONFLICT") from error + finally: + os.close(source_fd) + + +def _copy_external(data_root: Path, destination: Path, entries: list[dict[str, object]]) -> None: + for entry in entries: + source_root = _trusted_directory( + data_root / Path(str(entry["root"])), code="BACKUP_SOURCE_CHANGED", + ) + source = _bound_regular_file( + source_root, str(entry["path"]), code="BACKUP_SOURCE_CHANGED", + ) + relative_target = Path("files") / Path(str(entry["root"])) / Path(str(entry["path"])) + if relative_target.is_absolute() or ".." in relative_target.parts: + raise OperationalResetError("BACKUP_DESTINATION_CONFLICT") + target = destination / relative_target + _secure_mkdirs( + destination, relative_target.parent, code="BACKUP_PATH_UNSAFE", + ) + if _file_digest(source) != entry["sha256"]: + raise OperationalResetError("BACKUP_SOURCE_CHANGED") + _exclusive_copy(source, target) + if _file_digest(source) != entry["sha256"] or _file_digest(target) != entry["sha256"]: + raise OperationalResetError("BACKUP_SOURCE_CHANGED") + + +def _backup( + data_root: Path, operation_id: str, backup_root: Path, plan: dict[str, object], +) -> dict[str, object]: + destination, publish_destination = _backup_destination(backup_root, operation_id) + selected_destination = publish_destination or destination + try: + selected_destination.relative_to(data_root) + except ValueError: + pass + else: + raise OperationalResetError("BACKUP_PATH_INSIDE_ACTIVE_DATA_ROOT") + if publish_destination is None: + verified = verify_backup( + destination, operation_id=operation_id, plan_digest=str(plan["plan_digest"]), + ) + return { + "backup_path": str(destination), + "backup_sha256": _file_digest(destination / "central.sqlite"), + "backup_manifest_digest": verified["manifest_digest"], + } + database_backup = destination / "central.sqlite" + temporary = destination / ".central.sqlite.partial" + if temporary.exists() or temporary.is_symlink() or database_backup.exists() or database_backup.is_symlink(): + raise OperationalResetError("BACKUP_DESTINATION_CONFLICT") + required = (data_root / central_database.DATABASE_FILENAME).stat().st_size + sum( + int(item["size_bytes"]) for item in plan["external_files"] # type: ignore[index] + ) + if shutil.disk_usage(destination).free < required * 2 + 1024 * 1024: + raise OperationalResetError("BACKUP_SPACE_INSUFFICIENT") + try: + with _central_connection( + f"file:{data_root / central_database.DATABASE_FILENAME}?mode=ro", uri=True, + ) as source, _central_connection(temporary) as target: + source.backup(target) + temporary.chmod(0o600) + os.replace(temporary, database_backup) + _copy_external(data_root, destination, list(plan["external_files"])) # type: ignore[arg-type] + with _central_connection(f"file:{database_backup}?mode=ro", uri=True) as restored: + restored.execute("PRAGMA query_only=ON") + if [str(row[0]) for row in restored.execute("PRAGMA integrity_check")] != ["ok"]: + raise OperationalResetError("BACKUP_RESTORE_INTEGRITY_FAILED") + counts = {table: _row_count(restored, table) for table in sorted(_tables(restored))} + foreign_keys = _foreign_key_findings(restored) + entries = [] + for path in sorted((destination / "files").rglob("*")) if (destination / "files").exists() else []: + if path.is_file(): + entries.append({ + "path": path.relative_to(destination).as_posix(), "size_bytes": path.stat().st_size, + "sha256": _file_digest(path), + }) + manifest = { + "manifest_version": 1, "kind": "EP_PROTECTED_OPERATIONAL_RESET_BACKUP", + "operation_id": operation_id, "profile": PROFILE, "target": plan["target"], + "target_digest": plan["target_digest"], "plan_digest": plan["plan_digest"], + "source_revision": plan["source_revision"], "schema_version": SCHEMA_VERSION, + "database": {"path": "central.sqlite", "sha256": _file_digest(database_backup), + "size_bytes": database_backup.stat().st_size, "row_counts": counts, + "foreign_key_findings": foreign_keys}, + "included_files": entries, "excluded": [ + "provider logins", "Keychain items", "runtime virtual environments", "source repositories", + ], + "created_at": _utcnow(), + } + manifest_path = destination / "manifest.json" + manifest_path.write_bytes(_canonical(manifest) + b"\n") + manifest_path.chmod(0o600) + manifest_digest = _file_digest(manifest_path) + # Re-read every protected byte; a manifest hash alone is not restore proof. + verify_backup(destination, operation_id=operation_id, plan_digest=str(plan["plan_digest"])) + os.replace(destination, publish_destination) + destination = publish_destination + return { + "backup_path": str(destination), "backup_sha256": manifest["database"]["sha256"], + "backup_manifest_digest": manifest_digest, + } + except OperationalResetError: + raise + except (OSError, sqlite3.DatabaseError) as error: + raise OperationalResetError("BACKUP_WRITE_OR_VERIFY_FAILED") from error + finally: + temporary.unlink(missing_ok=True) + + +def verify_backup( + path: Path, *, operation_id: str, plan_digest: str, + expected_manifest_digest: str | None = None, +) -> dict[str, object]: + root = _trusted_directory(path, code="BACKUP_PATH_UNSAFE") + try: + if root.is_symlink() or root.stat().st_mode & 0o077: + raise OperationalResetError("BACKUP_PERMISSIONS_UNSAFE") + manifest_path = _bound_regular_file(root, "manifest.json", code="BACKUP_INVALID") + manifest_digest = _file_digest(manifest_path) + if expected_manifest_digest is not None and manifest_digest != expected_manifest_digest: + raise OperationalResetError("BACKUP_MANIFEST_DIGEST_MISMATCH") + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + database = _bound_regular_file(root, "central.sqlite", code="BACKUP_INVALID") + if ( + manifest.get("operation_id") != operation_id or manifest.get("plan_digest") != plan_digest + or manifest.get("database", {}).get("sha256") != _file_digest(database) + ): + raise OperationalResetError("BACKUP_BINDING_INVALID") + included_files = manifest.get("included_files", []) + if not isinstance(included_files, list): + raise OperationalResetError("BACKUP_INVALID") + expected_files = {"manifest.json", "central.sqlite"} + for entry in included_files: + if not isinstance(entry, dict) or not isinstance(entry.get("path"), str): + raise OperationalResetError("BACKUP_INVALID") + candidate = _bound_regular_file(root, str(entry["path"]), code="BACKUP_FILE_INVALID") + if _file_digest(candidate) != entry["sha256"]: + raise OperationalResetError("BACKUP_FILE_INVALID") + expected_files.add(str(entry["path"])) + actual_files = { + candidate.relative_to(root).as_posix() + for candidate in _walk_regular_files(root, code="BACKUP_FILE_INVALID") + } + if actual_files != expected_files: + raise OperationalResetError("BACKUP_FILE_SET_INVALID") + with _central_connection(f"file:{database}?mode=ro", uri=True) as connection: + connection.execute("PRAGMA query_only=ON") + integrity = [str(row[0]) for row in connection.execute("PRAGMA integrity_check")] + counts = {table: _row_count(connection, table) for table in sorted(_tables(connection))} + if integrity != ["ok"] or counts != manifest["database"]["row_counts"]: + raise OperationalResetError("BACKUP_RESTORE_INTEGRITY_FAILED") + return {"state": "VERIFIED", "operation_id": operation_id, + "manifest_digest": manifest_digest} + except OperationalResetError: + raise + except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError, sqlite3.DatabaseError) as error: + raise OperationalResetError("BACKUP_INVALID") from error + + +def _operation(connection: sqlite3.Connection, operation_id: str) -> dict[str, object] | None: + connection.row_factory = sqlite3.Row + row = connection.execute( + "SELECT * FROM ep_operational_reset_operations WHERE operation_id=?", (operation_id,) + ).fetchone() + return dict(row) if row is not None else None + + +def _transition( + connection: sqlite3.Connection, operation_id: str, before: str, after: str, + assignments: str = "", parameters: tuple[object, ...] = (), +) -> None: + prefix = f"state='{after}',updated_at=?" + if assignments: + prefix += "," + assignments + cursor = connection.execute( + f"UPDATE ep_operational_reset_operations SET {prefix} " + "WHERE operation_id=? AND state=?", + (_utcnow(), *parameters, operation_id, before), + ) + if cursor.rowcount != 1: + raise OperationalResetError("OPERATION_STATE_TRANSITION_CONFLICT") + + +def prepare( + data_root: Path, *, operation_id: str, plan_digest: str, backup_root: Path, + allowed_fk_findings: tuple[str, ...] = (), +) -> dict[str, object]: + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + actor = _actor(root) + plan = preview(root) + if plan["plan_digest"] != plan_digest: + raise OperationalResetError("PLAN_DIGEST_MISMATCH") + if plan["blocking_codes"]: + raise OperationalResetError("PLAN_BLOCKED", ",".join(plan["blocking_codes"])) + expected_findings = tuple(sorted(str(item) for item in plan["review_required"])) + allowed = tuple(sorted(set(allowed_fk_findings))) + if allowed != expected_findings: + raise OperationalResetError("OPERATIONAL_FK_APPROVAL_MISMATCH") + request = { + "operation_id": operation_id, "actor": actor, "plan_digest": plan_digest, + "target_digest": plan["target_digest"], "profile": PROFILE, + "effect_set": plan["effect_set"], "backup_root": str(backup_root.expanduser().resolve(strict=False)), + "allowed_fk_findings": list(allowed), + } + request_digest = _digest(request) + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + existing = _operation(connection, operation_id) + if existing is not None: + if existing["request_digest"] != request_digest: + raise OperationalResetError("OPERATION_REQUEST_CONFLICT") + return _public_status(existing) + generation = int(connection.execute( + "SELECT generation FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone()[0]) + now = _utcnow() + connection.execute("BEGIN IMMEDIATE") + connection.execute( + "INSERT INTO ep_operational_reset_operations(" + "operation_id,state,actor,plan_digest,target_digest,source_revision,request_digest," + "plan_json,allowed_fk_json,backup_root,generation_before,created_at,updated_at) " + "VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)", + (operation_id, "PREPARING", actor, plan_digest, plan["target_digest"], + plan["source_revision"], request_digest, json.dumps(plan, sort_keys=True), + json.dumps(allowed), request["backup_root"], generation, now, now), + ) + connection.execute("COMMIT") + try: + backup = _backup(root, operation_id, backup_root, plan) + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition( + connection, operation_id, "PREPARING", "AUTHORIZED", + "backup_path=?,backup_sha256=?,backup_manifest_digest=?", + (backup["backup_path"], backup["backup_sha256"], + backup["backup_manifest_digest"]), + ) + connection.execute("COMMIT") + return _public_status(_operation(connection, operation_id) or {}) + except Exception: + # PREPARING intentionally remains durable and writer-blocking. A + # resume retries the same backup instead of inventing a new reset. + raise + + +def _public_status(row: dict[str, object]) -> dict[str, object]: + allowed = { + "operation_id", "state", "actor", "plan_digest", "target_digest", "source_revision", + "backup_root", "backup_path", "backup_sha256", "backup_manifest_digest", "generation_before", + "generation_after", "finish_boundary_path", "finish_boundary_digest", + "created_at", "updated_at", "verification_json", + } + result = {key: value for key, value in row.items() if key in allowed} + if isinstance(result.get("verification_json"), str): + result["verification"] = json.loads(str(result.pop("verification_json"))) + result["credentials_included_in_receipt"] = False + return result + + +def status(data_root: Path, *, operation_id: str | None = None) -> dict[str, object]: + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + _actor(root) + with _central_connection(f"file:{root / central_database.DATABASE_FILENAME}?mode=ro", uri=True) as connection: + connection.row_factory = sqlite3.Row + if operation_id is None: + row = connection.execute( + "SELECT * FROM ep_operational_reset_operations ORDER BY created_at DESC LIMIT 1" + ).fetchone() + else: + row = connection.execute( + "SELECT * FROM ep_operational_reset_operations WHERE operation_id=?", (operation_id,) + ).fetchone() + generation = int(connection.execute( + "SELECT generation FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone()[0]) + return {"dataset_generation": generation, + "operation": _public_status(dict(row)) if row is not None else None} + + +def maintenance_active(data_root: Path) -> bool: + """Read the durable fence without initializing or migrating CENTRAL.""" + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + database = root / central_database.DATABASE_FILENAME + try: + with _central_connection(f"file:{database}?mode=ro", uri=True) as connection: + if "ep_operational_reset_operations" not in _tables(connection): + return False + return connection.execute( + "SELECT 1 FROM ep_operational_reset_operations " + "WHERE state NOT IN ('COMPLETED','ABORTED') LIMIT 1" + ).fetchone() is not None + except sqlite3.DatabaseError as error: + raise OperationalResetError("MAINTENANCE_STATE_UNAVAILABLE") from error + + +def _archive_effects(data_root: Path, operation_id: str, plan: dict[str, object]) -> None: + archive = _secure_mkdirs( + data_root, Path("operational-reset-archive") / operation_id, + code="ARCHIVE_PATH_UNSAFE", + ) + expected = list(plan["external_files"]) # type: ignore[arg-type] + for name in _EFFECT_DIRECTORIES: + relative = Path(name) + source, target = data_root / relative, archive / relative + _secure_mkdirs(archive, relative.parent, code="ARCHIVE_PATH_UNSAFE") + if target.exists(): + if target.is_symlink() or not target.is_dir(): + raise OperationalResetError("ARCHIVE_PATH_UNSAFE") + if source.exists() and any(source.iterdir()): + raise OperationalResetError("ARCHIVE_RECONCILIATION_AMBIGUOUS", name) + elif source.exists(): + if source.is_symlink() or not source.is_dir(): + raise OperationalResetError("EXTERNAL_PATH_UNSAFE", name) + os.replace(source, target) + source.mkdir(mode=0o700, parents=True, exist_ok=True) + observed: list[dict[str, object]] = [] + for name in _EFFECT_DIRECTORIES: + observed.extend(_safe_files(archive / name, name)) + comparable = lambda rows: sorted((row["root"], row["path"], row["size_bytes"], row["sha256"]) for row in rows) + if comparable(observed) != comparable(expected): + raise OperationalResetError("ARCHIVE_RECONCILIATION_FAILED") + + +def _finish_boundary_relative(operation_id: str, generation: int) -> Path: + return ( + Path("operational-reset-archive") / operation_id / "finish-boundary" + / f"generation-{generation}" + ) + + +def _finish_boundary_path(data_root: Path, operation_id: str, generation: int) -> Path: + return data_root / _finish_boundary_relative(operation_id, generation) + + +def _finish_boundary_files(boundary: Path) -> list[dict[str, object]]: + files: list[dict[str, object]] = [] + for name in _EFFECT_DIRECTORIES: + files.extend(_safe_files(boundary / Path(name), name)) + return files + + +def _active_root_mode(path: Path) -> int: + flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(path, flags) + except OSError as error: + raise OperationalResetError("ACTIVE_ROOT_UNSAFE") from error + try: + metadata = os.fstat(descriptor) + if not stat.S_ISDIR(metadata.st_mode): + raise OperationalResetError("ACTIVE_ROOT_UNSAFE") + return stat.S_IMODE(metadata.st_mode) + finally: + os.close(descriptor) + + +def _set_active_root_mode(path: Path, mode: int) -> None: + flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(path, flags) + except OSError as error: + raise OperationalResetError("ACTIVE_ROOT_UNSAFE") from error + try: + metadata = os.fstat(descriptor) + if not stat.S_ISDIR(metadata.st_mode): + raise OperationalResetError("ACTIVE_ROOT_UNSAFE") + os.fchmod(descriptor, mode) + os.fsync(descriptor) + except OSError as error: + raise OperationalResetError("ACTIVE_ROOT_MODE_CHANGE_FAILED") from error + finally: + os.close(descriptor) + + +def _active_roots_state(data_root: Path) -> dict[str, int]: + return { + name: _active_root_mode(data_root / Path(name)) + for name in _EFFECT_DIRECTORIES + } + + +def _freeze_empty_active_roots(data_root: Path) -> dict[str, int]: + for name in _EFFECT_DIRECTORIES: + path = data_root / Path(name) + if path.is_symlink() or not path.is_dir(): + raise OperationalResetError("ACTIVE_ROOT_UNSAFE", name) + try: + if next(path.iterdir(), None) is not None: + raise OperationalResetError("FINISH_ACTIVE_ROOT_NOT_EMPTY", name) + except OSError as error: + raise OperationalResetError("ACTIVE_ROOT_UNSAFE", name) from error + _set_active_root_mode(path, _FROZEN_ROOT_MODE) + modes = _active_roots_state(data_root) + if any(mode != _FROZEN_ROOT_MODE for mode in modes.values()): + raise OperationalResetError("ACTIVE_ROOT_FREEZE_FAILED") + return modes + + +def _verify_frozen_empty_active_roots(data_root: Path) -> dict[str, int]: + modes = _active_roots_state(data_root) + if any(mode != _FROZEN_ROOT_MODE for mode in modes.values()): + raise OperationalResetError("ACTIVE_ROOT_NOT_FROZEN") + for name in _EFFECT_DIRECTORIES: + path = data_root / Path(name) + try: + if next(path.iterdir(), None) is not None: + raise OperationalResetError("FINISH_ACTIVE_ROOT_NOT_EMPTY", name) + except OSError as error: + raise OperationalResetError("ACTIVE_ROOT_UNSAFE", name) from error + return modes + + +def _thaw_active_roots(data_root: Path) -> dict[str, int]: + # COMPLETED may have been committed just before process loss. Accept an + # idempotent mix of already-thawed and still-frozen roots, but no other + # mode or path type. + modes = _active_roots_state(data_root) + if any(mode not in {_FROZEN_ROOT_MODE, _ACTIVE_ROOT_MODE} for mode in modes.values()): + raise OperationalResetError("ACTIVE_ROOT_MODE_INVALID") + for name, mode in modes.items(): + if mode == _FROZEN_ROOT_MODE: + _set_active_root_mode(data_root / Path(name), _ACTIVE_ROOT_MODE) + thawed = _active_roots_state(data_root) + if any(mode != _ACTIVE_ROOT_MODE for mode in thawed.values()): + raise OperationalResetError("ACTIVE_ROOT_THAW_FAILED") + return thawed + + +def _verify_finish_boundary( + boundary: Path, *, operation_id: str, generation: int, + expected_digest: str | None = None, +) -> dict[str, object]: + if boundary.is_symlink() or not boundary.is_dir(): + raise OperationalResetError("FINISH_BOUNDARY_UNSAFE") + marker = _bound_regular_file( + boundary, "manifest.json", code="FINISH_BOUNDARY_INVALID", + ) + digest = _file_digest(marker) + if expected_digest is not None and digest != expected_digest: + raise OperationalResetError("FINISH_BOUNDARY_DIGEST_MISMATCH") + try: + manifest = json.loads(marker.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as error: + raise OperationalResetError("FINISH_BOUNDARY_INVALID") from error + if ( + not isinstance(manifest, dict) + or manifest.get("manifest_version") != 1 + or manifest.get("kind") != "EP_OPERATIONAL_RESET_FINISH_BOUNDARY" + or manifest.get("operation_id") != operation_id + or manifest.get("dataset_generation") != generation + or manifest.get("active_roots") != list(_EFFECT_DIRECTORIES) + ): + raise OperationalResetError("FINISH_BOUNDARY_INVALID") + observed = _finish_boundary_files(boundary) + comparable = lambda rows: sorted( + (item["root"], item["path"], item["size_bytes"], item["sha256"]) + for item in rows + ) + expected = manifest.get("preserved_files") + if not isinstance(expected, list) or comparable(observed) != comparable(expected): + raise OperationalResetError("FINISH_BOUNDARY_CHANGED") + return { + "path": str(boundary), "digest": digest, + "preserved_file_count": len(observed), "verified": True, + } + + +def _rotate_finish_boundary( + data_root: Path, *, operation_id: str, generation: int, boundary: Path, +) -> dict[str, object]: + expected = _finish_boundary_path(data_root, operation_id, generation) + if boundary != expected: + raise OperationalResetError("FINISH_BOUNDARY_BINDING_MISMATCH") + _secure_mkdirs( + data_root, _finish_boundary_relative(operation_id, generation), + code="FINISH_BOUNDARY_UNSAFE", + ) + marker = boundary / "manifest.json" + if marker.exists() or marker.is_symlink(): + _freeze_empty_active_roots(data_root) + return _verify_finish_boundary( + boundary, operation_id=operation_id, generation=generation, + ) + for name in _EFFECT_DIRECTORIES: + relative = Path(name) + source = data_root / relative + target = boundary / relative + _secure_mkdirs(data_root, relative.parent, code="FINISH_BOUNDARY_UNSAFE") + _secure_mkdirs(boundary, relative.parent, code="FINISH_BOUNDARY_UNSAFE") + if target.exists() or target.is_symlink(): + if target.is_symlink() or not target.is_dir(): + raise OperationalResetError("FINISH_BOUNDARY_UNSAFE") + else: + if source.exists() or source.is_symlink(): + if source.is_symlink() or not source.is_dir(): + raise OperationalResetError("EXTERNAL_PATH_UNSAFE", name) + else: + _secure_mkdirs(data_root, relative, code="FINISH_BOUNDARY_UNSAFE") + try: + os.rename(source, target) + except OSError as error: + raise OperationalResetError("FINISH_BOUNDARY_ROTATION_FAILED", name) from error + # A path lookup after the atomic rename belongs to the new generation; + # writers with an already-open directory descriptor remain isolated in + # the archived inode. The replacement stays frozen through the DB + # COMPLETED commit, closing the post-rename/pre-commit race. + _secure_mkdirs(data_root, relative, code="FINISH_BOUNDARY_UNSAFE") + _set_active_root_mode(source, _FROZEN_ROOT_MODE) + _verify_frozen_empty_active_roots(data_root) + manifest = { + "manifest_version": 1, "kind": "EP_OPERATIONAL_RESET_FINISH_BOUNDARY", + "operation_id": operation_id, "dataset_generation": generation, + "active_roots": list(_EFFECT_DIRECTORIES), + "preserved_files": _finish_boundary_files(boundary), + } + manifest_bytes = _canonical(manifest) + b"\n" + temporary = boundary / "manifest.json.partial" + if temporary.exists() or temporary.is_symlink(): + if temporary.is_symlink() or not temporary.is_file(): + raise OperationalResetError("FINISH_BOUNDARY_DESTINATION_CONFLICT") + # This exact operation-owned staging name cannot predate the durable + # boundary intent. Recover both a fully written pre-rename marker and + # an interrupted partial write without touching any archived payload. + if temporary.read_bytes() == manifest_bytes: + os.rename(temporary, marker) + return _verify_finish_boundary( + boundary, operation_id=operation_id, generation=generation, + ) + temporary.unlink() + try: + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except OSError as error: + raise OperationalResetError("FINISH_BOUNDARY_DESTINATION_CONFLICT") from error + try: + with os.fdopen(descriptor, "wb") as stream: + stream.write(manifest_bytes) + stream.flush() + os.fsync(stream.fileno()) + if marker.exists() or marker.is_symlink(): + raise OperationalResetError("FINISH_BOUNDARY_DESTINATION_CONFLICT") + os.rename(temporary, marker) + directory_descriptor = os.open(boundary, os.O_RDONLY) + try: + os.fsync(directory_descriptor) + finally: + os.close(directory_descriptor) + finally: + temporary.unlink(missing_ok=True) + return _verify_finish_boundary( + boundary, operation_id=operation_id, generation=generation, + ) + + +def _tombstone(connection: sqlite3.Connection, kind: str, identity: str, + operation_id: str, request_digest: str | None = None) -> None: + digest = hashlib.sha256(identity.encode("utf-8")).hexdigest() + connection.execute( + "INSERT OR IGNORE INTO ep_operational_identity_tombstones(" + "identity_kind,identity_digest,request_digest,operation_id,recorded_at) VALUES(?,?,?,?,?)", + (kind, digest, request_digest, operation_id, _utcnow()), + ) + + +def _submission_request_digest(row: sqlite3.Row) -> str: + return hashlib.sha256(_canonical({ + "repository_id": row["repository_id"], "producer_id": row["producer_id"], + "producer_type": row["producer_type"], "producer_version": row["producer_version"], + "prompt": row["prompt"], "constraints": row["constraints"], + "correlation_id": row["correlation_id"], "mission_id": row["mission_id"], + "engineering_action_id": row["engineering_action_id"], + "transport_receipt_id": row["transport_receipt_id"], + })).hexdigest() + + +def _record_tombstones(connection: sqlite3.Connection, operation_id: str) -> None: + connection.row_factory = sqlite3.Row + for row in connection.execute("SELECT * FROM ep_submissions"): + _tombstone(connection, "submission_id", str(row["submission_id"]), operation_id) + if row["idempotency_key"] is not None: + identity = f"{row['project_id']}\0{row['idempotency_key']}" + _tombstone(connection, "submission_idempotency", identity, operation_id, + _submission_request_digest(row)) + for table, column, kind in ( + ("ep_execution_runs", "run_id", "run_id"), + ("execution_runs", "run_id", "run_id"), + ("engineering_transactions", "run_id", "run_id"), + ("prompt_execution_history", "run_id", "run_id"), + ("provider_invocations", "invocation_id", "provider_invocation_id"), + ): + for row in connection.execute(f"SELECT {_quote(column)} FROM {_quote(table)}"): + _tombstone(connection, kind, str(row[0]), operation_id) + + +def _delete_operational(connection: sqlite3.Connection) -> None: + placeholders = ",".join("?" for _ in _OPERATIONAL_METADATA_KEYS) + connection.execute( + f"DELETE FROM engineering_metadata WHERE key IN ({placeholders})", + tuple(sorted(_OPERATIONAL_METADATA_KEYS)), + ) + for table in PURGE_ORDER: + if table == "execution_projections": + connection.execute("DELETE FROM execution_projections WHERE classification!='CONFIGURATION'") + else: + connection.execute(f"DELETE FROM {_quote(table)}") + + +def apply(data_root: Path, *, operation_id: str, plan_digest: str) -> dict[str, object]: + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + _actor(root) + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + row = _operation(connection, operation_id) + if row is None or row["plan_digest"] != plan_digest: + raise OperationalResetError("AUTHORIZED_OPERATION_NOT_FOUND") + state = str(row["state"]) + if state == "PREPARING": + raise OperationalResetError("BACKUP_NOT_AUTHORIZED") + if state not in _APPLY_ENTRY_STATES | _APPLY_IDEMPOTENT_STATES: + raise OperationalResetError("OPERATION_STATE_INVALID_FOR_APPLY") + if not isinstance(row.get("backup_path"), str) or not isinstance( + row.get("backup_manifest_digest"), str + ): + raise OperationalResetError("BACKUP_BINDING_INVALID") + verify_backup( + Path(str(row["backup_path"])), operation_id=operation_id, + plan_digest=plan_digest, + expected_manifest_digest=str(row["backup_manifest_digest"]), + ) + if state in _APPLY_IDEMPOTENT_STATES: + return _public_status(row) + plan = json.loads(str(row["plan_json"])) + if state == "AUTHORIZED": + current = preview(root) + if current["source_revision"] != row["source_revision"] or current["plan_digest"] != plan_digest: + raise OperationalResetError("SOURCE_REVISION_CHANGED") + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition(connection, operation_id, "AUTHORIZED", "ARTIFACTS_ARCHIVING") + connection.execute("COMMIT") + state = "ARTIFACTS_ARCHIVING" + if state != "ARTIFACTS_ARCHIVING" and state != "ARTIFACTS_ARCHIVED": + raise OperationalResetError("OPERATION_STATE_INVALID_FOR_APPLY") + _archive_effects(root, operation_id, plan) + if state == "ARTIFACTS_ARCHIVING": + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition(connection, operation_id, "ARTIFACTS_ARCHIVING", "ARTIFACTS_ARCHIVED") + connection.execute("COMMIT") + verify_backup( + Path(str(row["backup_path"])), operation_id=operation_id, + plan_digest=plan_digest, + expected_manifest_digest=str(row["backup_manifest_digest"]), + ) + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + try: + current = _operation(connection, operation_id) + if current is None or current["state"] != "ARTIFACTS_ARCHIVED": + raise OperationalResetError("OPERATION_STATE_TRANSITION_CONFLICT") + # Immutable-evidence and maintenance-block triggers are removed and + # recreated inside this one uncommitted transaction. Other writers + # cannot observe an unfenced schema window. + triggers = [ + (str(name), str(sql)) for name, sql in connection.execute( + "SELECT name,sql FROM sqlite_master WHERE type='trigger' AND tbl_name IN (" + + ",".join("?" for _ in (OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION)) + ")", + tuple(sorted(OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION)), + ).fetchall() if sql is not None + ] + for name, _sql in triggers: + connection.execute(f"DROP TRIGGER {_quote(name)}") + _record_tombstones(connection, operation_id) + _delete_operational(connection) + generation = int(row["generation_before"]) + 1 + cursor = connection.execute( + "UPDATE ep_operational_dataset_state SET generation=?,updated_at=? " + "WHERE singleton=1 AND generation=?", + (generation, _utcnow(), row["generation_before"]), + ) + if cursor.rowcount != 1: + raise OperationalResetError("DATASET_GENERATION_CONFLICT") + for _name, sql in triggers: + connection.execute(sql) + if list(connection.execute("PRAGMA foreign_key_check")): + raise OperationalResetError("POST_RESET_FOREIGN_KEY_FAILED") + _transition( + connection, operation_id, "ARTIFACTS_ARCHIVED", "DB_APPLIED", + "generation_after=?", (generation,), + ) + connection.execute("COMMIT") + return _public_status(_operation(connection, operation_id) or {}) + except Exception: + if connection.in_transaction: + connection.execute("ROLLBACK") + raise + + +def _verify_under_lock( + root: Path, *, operation_id: str, plan_digest: str, promote: bool, + allow_boundary_recovery: bool = False, +) -> dict[str, object]: + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + row = _operation(connection, operation_id) + if ( + row is None or row["plan_digest"] != plan_digest + or row["state"] not in {"DB_APPLIED", "VERIFIED", "COMPLETED"} + ): + raise OperationalResetError("RESET_NOT_APPLIED") + plan = json.loads(str(row["plan_json"])) + quick = [str(item[0]) for item in connection.execute("PRAGMA quick_check")] + foreign_keys = list(connection.execute("PRAGMA foreign_key_check")) + remaining = { + table: _row_count(connection, table) + for table in sorted(OPERATIONAL_HISTORY | DERIVED_CACHE_OR_PROJECTION) + } + remaining["engineering_metadata[operational_keys]"] = _row_count( + connection, "engineering_metadata" + ) + preserved_digest = _logical_digest(connection, _tables(connection), preserved_only=True) + generation = int(connection.execute( + "SELECT generation FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone()[0]) + credentials = int(connection.execute("SELECT COUNT(*) FROM ep_consumer_credentials").fetchone()[0]) + projects = int(connection.execute("SELECT COUNT(*) FROM ep_project_registrations").fetchone()[0]) + repositories = int(connection.execute("SELECT COUNT(*) FROM ep_repository_registrations").fetchone()[0]) + archive = root / "operational-reset-archive" / operation_id + boundary: dict[str, object] | None = None + boundary_path_value = row.get("finish_boundary_path") + frozen_modes: dict[str, int] | None = None + if boundary_path_value is None: + if allow_boundary_recovery: + raise OperationalResetError("FINISH_BOUNDARY_RECOVERY_INVALID") + _archive_effects(root, operation_id, plan) + else: + expected_boundary = _finish_boundary_path(root, operation_id, generation) + if str(expected_boundary) != boundary_path_value: + raise OperationalResetError("FINISH_BOUNDARY_BINDING_MISMATCH") + marker = expected_boundary / "manifest.json" + if marker.exists() or marker.is_symlink(): + boundary = _verify_finish_boundary( + expected_boundary, operation_id=operation_id, generation=generation, + expected_digest=( + str(row["finish_boundary_digest"]) + if row.get("finish_boundary_digest") is not None else None + ), + ) + elif row["state"] == "COMPLETED": + raise OperationalResetError("FINISH_BOUNDARY_INVALID") + if not allow_boundary_recovery: + frozen_modes = _verify_frozen_empty_active_roots(root) + external, _preserved, unknown, active_controls = _external_inventory(root) + backup = verify_backup( + Path(str(row["backup_path"])), operation_id=operation_id, plan_digest=plan_digest, + expected_manifest_digest=str(row["backup_manifest_digest"]), + ) + failures = {table: count for table, count in remaining.items() if count} + result = { + "quick_check": quick, "foreign_key_errors": len(foreign_keys), + "operational_rows_remaining": failures, + "preserved_bindings_digest": preserved_digest, + "expected_preserved_bindings_digest": plan["preserved_bindings_digest"], + "dataset_generation": generation, "expected_generation": row["generation_after"], + "project_registrations": projects, "repository_registrations": repositories, + "consumer_credentials": credentials, "backup": backup, + "active_artifact_roots_empty": not external, + "active_ingest_controls": active_controls, + "unknown_external_paths": unknown, + "archive_path": str(archive), + "finish_boundary": boundary, + "active_root_modes": frozen_modes, + } + if ( + quick != ["ok"] or foreign_keys or failures + or preserved_digest != plan["preserved_bindings_digest"] + or generation != row["generation_after"] + or (external and not allow_boundary_recovery) or active_controls or unknown + ): + raise OperationalResetError("POST_RESET_VERIFICATION_FAILED") + if promote and row["state"] == "DB_APPLIED": + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition( + connection, operation_id, "DB_APPLIED", "VERIFIED", + "verification_json=?", (json.dumps(result, sort_keys=True),), + ) + connection.execute("COMMIT") + return result + + +def _verify_completed_finish_binding( + root: Path, *, operation_id: str, plan_digest: str, +) -> tuple[dict[str, object], dict[str, object]]: + """Verify immutable finish proof without reasserting old-generation emptiness.""" + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + row = _operation(connection, operation_id) + if ( + row is None or row["state"] != "COMPLETED" + or row["plan_digest"] != plan_digest + ): + raise OperationalResetError("COMPLETED_OPERATION_BINDING_INVALID") + try: + plan = json.loads(str(row["plan_json"])) + except (TypeError, ValueError, json.JSONDecodeError) as error: + raise OperationalResetError("COMPLETED_OPERATION_BINDING_INVALID") from error + generation = int(connection.execute( + "SELECT generation FROM ep_operational_dataset_state WHERE singleton=1" + ).fetchone()[0]) + identity = _identity(root, connection) + quick = [str(item[0]) for item in connection.execute("PRAGMA quick_check")] + foreign_keys = list(connection.execute("PRAGMA foreign_key_check")) + expected_boundary = _finish_boundary_path(root, operation_id, generation) + digest = row.get("finish_boundary_digest") + if ( + generation != row.get("generation_after") + or plan.get("target_digest") != row.get("target_digest") + or _digest(identity) != row.get("target_digest") + or row.get("finish_boundary_path") != str(expected_boundary) + or not isinstance(digest, str) + or quick != ["ok"] or foreign_keys + ): + raise OperationalResetError("COMPLETED_OPERATION_BINDING_INVALID") + boundary = _verify_finish_boundary( + expected_boundary, operation_id=operation_id, generation=generation, + expected_digest=digest, + ) + modes = _active_roots_state(root) + if any(mode not in {_FROZEN_ROOT_MODE, _ACTIVE_ROOT_MODE} for mode in modes.values()): + raise OperationalResetError("ACTIVE_ROOT_MODE_INVALID") + return row, { + "state": "COMPLETED", "operation_id": operation_id, + "dataset_generation": generation, "target_digest": row["target_digest"], + "finish_boundary": boundary, "active_root_modes": modes, + "quick_check": quick, "foreign_key_errors": 0, + } + + +def verify(data_root: Path, *, operation_id: str, plan_digest: str) -> dict[str, object]: + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + _actor(root) + with _central_connection(root / central_database.DATABASE_FILENAME) as connection: + row = _operation(connection, operation_id) + if row is not None and row.get("state") == "COMPLETED": + _row, proof = _verify_completed_finish_binding( + root, operation_id=operation_id, plan_digest=plan_digest, + ) + return proof + return _verify_under_lock( + root, operation_id=operation_id, plan_digest=plan_digest, promote=True, + ) + + +def finish(data_root: Path, *, operation_id: str, plan_digest: str) -> dict[str, object]: + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + _actor(root) + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + row = _operation(connection, operation_id) + if row is None or row["plan_digest"] != plan_digest: + raise OperationalResetError("AUTHORIZED_OPERATION_NOT_FOUND") + if row["state"] == "COMPLETED": + completed, _proof = _verify_completed_finish_binding( + root, operation_id=operation_id, plan_digest=plan_digest, + ) + _thaw_active_roots(root) + return _public_status(completed) + if row["state"] != "VERIFIED": + raise OperationalResetError("VERIFICATION_REQUIRED_BEFORE_FINISH") + # First re-prove the exact database, backup, bindings and active routes. + # If a prior crash already persisted the boundary intent this is a + # recovery verification and the still-fenced active paths may contain + # post-boundary arrivals. + _verify_under_lock( + root, operation_id=operation_id, plan_digest=plan_digest, promote=False, + allow_boundary_recovery=row.get("finish_boundary_path") is not None, + ) + generation = int(row["generation_after"]) + boundary = _finish_boundary_path(root, operation_id, generation) + if row.get("finish_boundary_path") is None: + if boundary.exists() or boundary.is_symlink(): + raise OperationalResetError("FINISH_BOUNDARY_DESTINATION_CONFLICT") + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition( + connection, operation_id, "VERIFIED", "VERIFIED", + "finish_boundary_path=?", (str(boundary),), + ) + connection.execute("COMMIT") + row["finish_boundary_path"] = str(boundary) + elif row["finish_boundary_path"] != str(boundary): + raise OperationalResetError("FINISH_BOUNDARY_BINDING_MISMATCH") + boundary_proof = _rotate_finish_boundary( + root, operation_id=operation_id, generation=generation, boundary=boundary, + ) + verification = _verify_under_lock( + root, operation_id=operation_id, plan_digest=plan_digest, promote=False, + ) + if verification.get("finish_boundary") != boundary_proof: + raise OperationalResetError("FINISH_BOUNDARY_CHANGED") + with _central_connection(database) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition( + connection, operation_id, "VERIFIED", "COMPLETED", + "finish_boundary_digest=?,verification_json=?", + (boundary_proof["digest"], json.dumps(verification, sort_keys=True)), + ) + connection.execute("COMMIT") + completed = _operation(connection, operation_id) or {} + _thaw_active_roots(root) + return _public_status(completed) + + +def abort(data_root: Path, *, operation_id: str, plan_digest: str) -> dict[str, object]: + """End a prepared operation only while no reset effect has started.""" + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + _actor(root) + archive = root / "operational-reset-archive" / operation_id + if archive.exists(): + raise OperationalResetError("ABORT_AFTER_EFFECT_FORBIDDEN") + database = root / central_database.DATABASE_FILENAME + with _central_connection(database) as connection: + row = _operation(connection, operation_id) + if row is None or row["plan_digest"] != plan_digest: + raise OperationalResetError("AUTHORIZED_OPERATION_NOT_FOUND") + if row["state"] == "ABORTED": + return _public_status(row) + if row["state"] not in {"PREPARING", "AUTHORIZED"}: + raise OperationalResetError("ABORT_AFTER_EFFECT_FORBIDDEN") + connection.execute("BEGIN IMMEDIATE") + connection.execute( + "UPDATE ep_operational_reset_operations SET state='ABORTED',updated_at=? " + "WHERE operation_id=? AND state IN ('PREPARING','AUTHORIZED')", + (_utcnow(), operation_id), + ) + connection.execute("COMMIT") + return _public_status(_operation(connection, operation_id) or {}) + + +def resume(data_root: Path, *, operation_id: str, plan_digest: str, + backup_root: Path | None = None) -> dict[str, object]: + current = status(data_root, operation_id=operation_id).get("operation") + if not isinstance(current, dict): + raise OperationalResetError("AUTHORIZED_OPERATION_NOT_FOUND") + state = current.get("state") + if state == "PREPARING": + if backup_root is None: + raise OperationalResetError("BACKUP_ROOT_REQUIRED_FOR_RESUME") + root = _trusted_directory(data_root, code="DATA_ROOT_UNSAFE") + with _operation_lock(root, operation_id): + _actor(root) + with _central_connection(root / central_database.DATABASE_FILENAME) as connection: + row = _operation(connection, operation_id) or {} + if row.get("state") != "PREPARING" or row.get("plan_digest") != plan_digest: + raise OperationalResetError("OPERATION_STATE_INVALID_FOR_RESUME") + plan = json.loads(str(row["plan_json"])) + requested_backup_root = str(backup_root.expanduser().resolve(strict=False)) + if requested_backup_root != row.get("backup_root"): + raise OperationalResetError("BACKUP_ROOT_BINDING_MISMATCH") + backup = _backup(root, operation_id, backup_root, plan) + with _central_connection(root / central_database.DATABASE_FILENAME) as connection: + connection.execute("BEGIN IMMEDIATE") + _transition( + connection, operation_id, "PREPARING", "AUTHORIZED", + "backup_path=?,backup_sha256=?,backup_manifest_digest=?", + (backup["backup_path"], backup["backup_sha256"], + backup["backup_manifest_digest"]), + ) + connection.execute("COMMIT") + state = "AUTHORIZED" + if state in {"AUTHORIZED", "ARTIFACTS_ARCHIVING", "ARTIFACTS_ARCHIVED"}: + apply(data_root, operation_id=operation_id, plan_digest=plan_digest) + state = "DB_APPLIED" + if state == "DB_APPLIED": + verify(data_root, operation_id=operation_id, plan_digest=plan_digest) + elif state == "COMPLETED": + # A process may have died after the durable completion commit but + # before thawing the new-generation ingest roots. The same operation + # verifies its immutable boundary and idempotently finishes the thaw. + finish(data_root, operation_id=operation_id, plan_digest=plan_digest) + elif state not in {"VERIFIED", "COMPLETED"}: + raise OperationalResetError("OPERATION_STATE_INVALID_FOR_RESUME") + return status(data_root, operation_id=operation_id) + + +def contract_readback( + data_root: Path, *, command: str, operation_id: str | None, +) -> dict[str, object]: + """Project the shared operator contract from persisted, public state only. + + Command return values can contain implementation diagnostics or protected + configuration. They are deliberately not accepted by this projection and + can therefore never be copied to stdout by the maintenance CLI. + """ + observed = preview(data_root) + persisted: object = None + if operation_id is not None: + persisted = status(data_root, operation_id=operation_id).get("operation") + operation = persisted if isinstance(persisted, dict) else {} + if not isinstance(operation, dict): + operation = {} + selected_operation = operation_id or ( + str(operation["operation_id"]) if isinstance(operation.get("operation_id"), str) else None + ) + backup: dict[str, object] | None = None + if isinstance(operation.get("backup_path"), str): + verified = False + try: + verify_backup( + Path(str(operation["backup_path"])), operation_id=str(selected_operation), + plan_digest=str(operation.get("plan_digest")), + expected_manifest_digest=( + str(operation["backup_manifest_digest"]) + if isinstance(operation.get("backup_manifest_digest"), str) else None + ), + ) + verified = True + except OperationalResetError: + verified = False + backup = { + "manifest": str(Path(str(operation["backup_path"])) / "manifest.json"), + "digest": operation.get("backup_manifest_digest"), "verified": verified, + } + target = observed["target"] + blockers = list(observed["blocking_codes"]) + if operation.get("state") == "FAILED": + blockers.append("OPERATION_FAILED_RECONCILIATION_REQUIRED") + return { + "contract_version": "operational-reset-v1", "product": "engineering-platform", + "command": command, "operation_id": selected_operation, + "state": operation.get("state", observed["execution_state"]), + "allowed": not blockers, + "target": { + "instance_id": target["instance_id"], "database_path": target["database"], + "database_identity": observed["target_digest"], + "schema_version": target["schema_version"], + }, + "profile": PROFILE, "dataset_generation": observed["dataset_generation"], + "plan_digest": operation.get("plan_digest", observed["plan_digest"]), + "relevant_revision_digest": observed["source_revision"], "backup": backup, + "counts": observed["classification_counts"], "blockers": blockers, + "integrity": { + "quick_check": observed["quick_check"], + "foreign_key_findings": observed["foreign_key_findings"], + }, + "preserved_bindings_digest": observed["preserved_bindings_digest"], + "details": { + "credentials_included_in_receipt": False, + "projection": "PERSISTED_PUBLIC_MAINTENANCE_STATE", + }, + } + + +class _ReceiptArgumentParser(argparse.ArgumentParser): + def error(self, message: str) -> None: + raise OperationalResetError("CLI_ARGUMENT_INVALID") + + +def _error_receipt(command: str, operation_id: str | None, code: str) -> dict[str, object]: + """Return the shared contract shape without paths, exception text or secrets.""" + return { + "contract_version": "operational-reset-v1", "product": "engineering-platform", + "command": command, "operation_id": operation_id, "state": "ERROR", "allowed": False, + "target": { + "instance_id": None, "database_path": None, "database_identity": None, + "schema_version": None, + }, + "profile": PROFILE, "dataset_generation": None, "plan_digest": None, + "relevant_revision_digest": None, "backup": None, "counts": {}, + "blockers": [code], "integrity": {"quick_check": [], "foreign_key_findings": []}, + "preserved_bindings_digest": None, "error": code, "error_code": code, + "details": {"credentials_included_in_receipt": False}, + } + + +def main(argv: list[str] | None = None) -> int: + parser = _ReceiptArgumentParser(prog="engineering-platform-maintenance") + parser.add_argument( + "command", + choices=("preview", "prepare", "apply", "status", "resume", "verify", "finish", "abort"), + ) + parser.add_argument("--data-root", type=Path, required=True) + parser.add_argument("--operation-id") + parser.add_argument("--plan-digest") + parser.add_argument("--backup-root", type=Path) + parser.add_argument("--allow-operational-fk", action="append", default=[]) + raw = list(argv) if argv is not None else [] + command = raw[0] if raw and not raw[0].startswith("-") else "unknown" + operation_id: str | None = None + try: + args = parser.parse_args(argv) + command = args.command + operation_id = args.operation_id + if args.command == "preview": + result = preview(args.data_root) + elif args.command == "status": + result = status(args.data_root, operation_id=args.operation_id) + else: + if not args.operation_id or not args.plan_digest: + parser.error("--operation-id and --plan-digest are required") + if args.command == "prepare": + if args.backup_root is None: + parser.error("--backup-root is required") + result = prepare(args.data_root, operation_id=args.operation_id, + plan_digest=args.plan_digest, backup_root=args.backup_root, + allowed_fk_findings=tuple(args.allow_operational_fk)) + elif args.command == "apply": + result = apply(args.data_root, operation_id=args.operation_id, plan_digest=args.plan_digest) + elif args.command == "resume": + result = resume(args.data_root, operation_id=args.operation_id, + plan_digest=args.plan_digest, backup_root=args.backup_root) + elif args.command == "verify": + result = verify(args.data_root, operation_id=args.operation_id, plan_digest=args.plan_digest) + elif args.command == "abort": + result = abort(args.data_root, operation_id=args.operation_id, plan_digest=args.plan_digest) + else: + result = finish(args.data_root, operation_id=args.operation_id, plan_digest=args.plan_digest) + # Do not serialize ``result``. The shared receipt is reconstructed + # from the schema-owned public projection after the command completes. + print(json.dumps(contract_readback( + args.data_root, command=args.command, operation_id=args.operation_id, + ), sort_keys=True)) + return 0 + except OperationalResetError as error: + print(json.dumps(_error_receipt(command, operation_id, error.code), sort_keys=True)) + return 2 + except (OSError, sqlite3.DatabaseError, TypeError, ValueError, json.JSONDecodeError): + print(json.dumps( + _error_receipt(command, operation_id, "MAINTENANCE_COMMAND_FAILED"), sort_keys=True, + )) + return 2 + except Exception: + print(json.dumps( + _error_receipt(command, operation_id, "INTERNAL_MAINTENANCE_ERROR"), sort_keys=True, + )) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/engineering_platform/parity_lifecycle_dispatcher.py b/src/engineering_platform/parity_lifecycle_dispatcher.py index ca125ea7..bad886b0 100644 --- a/src/engineering_platform/parity_lifecycle_dispatcher.py +++ b/src/engineering_platform/parity_lifecycle_dispatcher.py @@ -10,6 +10,7 @@ from contextlib import contextmanager from dataclasses import dataclass, replace from datetime import datetime, timezone +import hashlib import json import logging import os @@ -225,9 +226,22 @@ def _utcnow() -> str: return datetime.now(timezone.utc).isoformat() -def _allocate_run_id() -> str: - """Allocate a canonical CENTRAL lifecycle run identity.""" - return f"inbox-{uuid.uuid4().hex}" +def _allocate_run_id(connection: sqlite3.Connection) -> str: + """Allocate a canonical run identity without reusing a retired value.""" + for _attempt in range(32): + candidate = f"inbox-{uuid.uuid4().hex}" + if connection.execute( + "SELECT 1 FROM ep_execution_runs WHERE run_id=?", (candidate,) + ).fetchone() is not None: + continue + retired = connection.execute( + "SELECT 1 FROM ep_operational_identity_tombstones " + "WHERE identity_kind='run_id' AND identity_digest=?", + (hashlib.sha256(candidate.encode("utf-8")).hexdigest(),), + ).fetchone() + if retired is None: + return candidate + raise ParityLifecycleDispatchError("RUN_IDENTITY_ALLOCATION_EXHAUSTED") def _lifecycle_logger(data_root: Path): @@ -400,7 +414,7 @@ def _claim(self, submission_id: str) -> tuple[ParityProjectContext, HistoricalCa connection.execute("ROLLBACK") raise ParityLifecycleDispatchError("PROJECT_RUN_ALREADY_ACTIVE") candidate = historical_candidate(connection, context=context, submission_id=submission_id) - run_id = _allocate_run_id() + run_id = _allocate_run_id(connection) prompt = self._prompt_path(context, run_id) now = _utcnow() connection.execute( diff --git a/src/engineering_platform/platform_version.py b/src/engineering_platform/platform_version.py index efe5705f..3228ad1d 100644 --- a/src/engineering_platform/platform_version.py +++ b/src/engineering_platform/platform_version.py @@ -12,7 +12,7 @@ SEMVER = re.compile(r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$") CONTRACT = re.compile(r"^(\d{4})\.(0[1-9]|1[0-2])$") -CURRENT_PLATFORM_VERSION = "2.3.80" +CURRENT_PLATFORM_VERSION = "2.3.81" MANIFEST_FIELDS = frozenset( { "platform_version", diff --git a/src/engineering_platform/server.py b/src/engineering_platform/server.py index 7c1237e3..46836b0d 100644 --- a/src/engineering_platform/server.py +++ b/src/engineering_platform/server.py @@ -40,6 +40,7 @@ from . import agent_trust from . import central_database from . import central_data_transfer +from . import central_operational_reset from . import console_route_ownership from . import console_presentation from . import development_profile @@ -139,7 +140,7 @@ # bootstrap is deliberately separate from the retired predecessor migration # machinery: it creates a clean installation only and never accepts a source # database path. -SERVER_STORE_SCHEMA_VERSION = 67 +SERVER_STORE_SCHEMA_VERSION = 68 SERVER_ENVIRONMENT_DATA_ROOT = "EP_SERVER_DATA_ROOT" FILE_INBOX_DIRECTORY = "file-inbox" HTTP_JSON_OPENAPI_PATH = "/v1/openapi.json" @@ -616,6 +617,9 @@ class ServerConfigurationError(ValueError): "ep_execution_host_evidence", "ep_technical_diagnostics", "ep_terminal_evidence_reconciliation_operations", + "ep_operational_reset_operations", + "ep_operational_dataset_state", + "ep_operational_identity_tombstones", } ) SERVER_REQUIRED_INDEXES = frozenset( @@ -840,6 +844,7 @@ def _install_current_schema(connection: sqlite3.Connection, identity: RuntimeIde _install_technical_diagnostics_schema(connection) owner_credential_recovery.install_schema(connection) submission_service.install_terminal_evidence_reconciliation_schema(connection) + central_operational_reset.install_schema(connection) connection.execute( "INSERT INTO engineering_schema_migrations(version) VALUES(?)", @@ -2006,6 +2011,29 @@ def _migrate_schema_67(connection: sqlite3.Connection) -> None: connection.execute("UPDATE ep_installations SET schema_version=67") +def _migrate_schema_68(connection: sqlite3.Connection) -> None: + """Install product-owned reset control and repair the CENTRAL chat parent. + + Existing chat rows are copied unchanged. The migration validates that + every row belongs to its canonical execution run; it never fabricates a + prompt-history parent and never deletes historical chat evidence. + """ + connection.execute("ALTER TABLE ep_installations RENAME TO ep_installations_schema67") + connection.execute( + "CREATE TABLE ep_installations (instance_id TEXT PRIMARY KEY,created_at TEXT NOT NULL," + "schema_version INTEGER NOT NULL CHECK(schema_version BETWEEN 41 AND 68))" + ) + connection.execute( + "INSERT INTO ep_installations SELECT instance_id,created_at,68 " + "FROM ep_installations_schema67" + ) + connection.execute("DROP TABLE ep_installations_schema67") + central_operational_reset.install_schema(connection) + connection.execute("INSERT OR IGNORE INTO engineering_schema_migrations(version) VALUES(68)") + connection.execute("UPDATE engineering_metadata SET value='68' WHERE key='installation.schema_version'") + connection.execute("UPDATE ep_installations SET schema_version=68") + + _SERVER_SCHEMA_UPGRADE_STEPS = ( (42, _migrate_schema_42), (43, _migrate_schema_43), @@ -2033,6 +2061,7 @@ def _migrate_schema_67(connection: sqlite3.Connection) -> None: (65, _migrate_schema_65), (66, _migrate_schema_66), (67, _migrate_schema_67), + (68, _migrate_schema_68), ) _SUPPORTED_SERVER_SCHEMA_VERSIONS = frozenset( range(41, SERVER_STORE_SCHEMA_VERSION + 1) @@ -4588,7 +4617,9 @@ def _central_console_chat_history(data_root: Path, project_id: str, run_id: str) """Return a project-authorized CENTRAL transcript; no root fallback exists.""" with storage.sqlite_connection(data_root / SERVER_DATABASE_FILENAME) as connection: belongs = connection.execute( - "SELECT 1 FROM ep_parity_lifecycle_dispatches WHERE project_id=? AND run_id=?", + "SELECT 1 FROM ep_parity_lifecycle_dispatches AS dispatch " + "JOIN ep_execution_runs AS run ON run.run_id=dispatch.run_id " + "WHERE dispatch.project_id=? AND dispatch.run_id=?", (project_id, run_id), ).fetchone() if belongs is None: @@ -4681,8 +4712,14 @@ def _central_console_append_chat_message( raise CodexChatError("Het chatbericht bevat geen bewaarbare tekst.", code="CHAT_REQUEST_INVALID") cutoff = (datetime.now(timezone.utc) - timedelta(days=CHAT_RETENTION_DAYS)).isoformat() with storage.sqlite_connection(data_root / SERVER_DATABASE_FILENAME) as connection: + connection.execute("PRAGMA foreign_keys=ON") + if connection.execute("PRAGMA foreign_keys").fetchone() != (1,): + raise ValueError("CHAT_FOREIGN_KEY_ENFORCEMENT_UNAVAILABLE") + connection.execute("BEGIN IMMEDIATE") belongs = connection.execute( - "SELECT 1 FROM ep_parity_lifecycle_dispatches WHERE project_id=? AND run_id=?", + "SELECT 1 FROM ep_parity_lifecycle_dispatches AS dispatch " + "JOIN ep_execution_runs AS run ON run.run_id=dispatch.run_id " + "WHERE dispatch.project_id=? AND dispatch.run_id=?", (project_id, run_id), ).fetchone() if belongs is None: @@ -7063,6 +7100,8 @@ def log_message(self, _format: str, *_args: object) -> None: def serve(data_root: Path, *, development: development_profile.DevelopmentProfile | None = None) -> int: + if central_operational_reset.maintenance_active(data_root): + raise ServerConfigurationError("EP_OPERATIONAL_MAINTENANCE_ACTIVE") relocation = installation_relocation.apply_pending(data_root) if relocation is not None: data_root = Path(relocation["value"]) @@ -7183,6 +7222,8 @@ def start(data_root: Path, *, development: development_profile.DevelopmentProfil current = status(data_root) if current["running"]: return current + if central_operational_reset.maintenance_active(data_root): + raise ServerConfigurationError("EP_OPERATIONAL_MAINTENANCE_ACTIVE") # The installed entrypoint supplies the interpreter. Run from the # installation-owned data root and discard Python import overrides so a # caller's checkout can never become the child Server's import authority. @@ -7776,6 +7817,8 @@ def main(argv: list[str] | None = None) -> int: result = {"instance_id": initialize(args.data_root, bind_host=args.bind_host, bind_port=args.bind_port).instance_id, "initialized": True} elif args.command == "start": initialize(args.data_root) + if central_operational_reset.maintenance_active(args.data_root): + raise ServerConfigurationError("EP_OPERATIONAL_MAINTENANCE_ACTIVE") configuration = ServerConfiguration.load(args.data_root) if (configuration.bind_host, configuration.bind_port) != (args.bind_host, args.bind_port): _write_json(args.data_root / SERVER_CONFIGURATION_FILENAME, asdict(ServerConfiguration( diff --git a/src/engineering_platform/submission_service.py b/src/engineering_platform/submission_service.py index 67db21c1..06b26b1b 100644 --- a/src/engineering_platform/submission_service.py +++ b/src/engineering_platform/submission_service.py @@ -460,6 +460,44 @@ def _same_idempotent_request(row: tuple[object, ...], request: SubmissionRequest return stored == requested +def _idempotency_request_digest(request: SubmissionRequest) -> str: + """Bind retired idempotency evidence without retaining request content.""" + payload = { + "repository_id": request.repository_id, "producer_id": request.producer_id, + "producer_type": request.producer_type, "producer_version": request.producer_version, + "prompt": request.prompt, "constraints": json.dumps(request.constraints or {}, sort_keys=True), + "correlation_id": request.correlation_id, "mission_id": request.mission_id, + "engineering_action_id": request.engineering_action_id, + "transport_receipt_id": request.transport_receipt_id, + } + return hashlib.sha256( + json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("ascii") + ).hexdigest() + + +def _allocate_submission_id(connection: sqlite3.Connection) -> str: + """Allocate without reusing an active or reset-retired external identity.""" + tables = { + str(row[0]) for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ) + } + for _attempt in range(32): + candidate = "sub-" + secrets.token_hex(16) + if connection.execute( + "SELECT 1 FROM ep_submissions WHERE submission_id=?", (candidate,) + ).fetchone() is not None: + continue + retired = connection.execute( + "SELECT 1 FROM ep_operational_identity_tombstones " + "WHERE identity_kind='submission_id' AND identity_digest=?", + (hashlib.sha256(candidate.encode("utf-8")).hexdigest(),), + ).fetchone() if "ep_operational_identity_tombstones" in tables else None + if retired is None: + return candidate + raise SubmissionError("IDENTITY_ALLOCATION_EXHAUSTED", 503) + + def _persist_lifecycle_events( connection: sqlite3.Connection, *, submission_id: str, @@ -503,6 +541,16 @@ def lifecycle(connection: sqlite3.Connection, submission_id: str) -> dict[str, s def submit(connection: sqlite3.Connection, request: SubmissionRequest, *, audit_forge_exchange: bool = True) -> SubmissionResult: """Persist and admit one request; no provider or Agent is selected here.""" + tables = { + str(row[0]) for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ) + } + if "ep_operational_reset_operations" in tables and connection.execute( + "SELECT 1 FROM ep_operational_reset_operations " + "WHERE state NOT IN ('COMPLETED','ABORTED') LIMIT 1" + ).fetchone() is not None: + raise SubmissionError("PLATFORM_MAINTENANCE_ACTIVE", 503) _transport(request.transport) _validate_execution_mode(request) _forge_provenance(request) @@ -521,6 +569,20 @@ def submit(connection: sqlite3.Connection, request: SubmissionRequest, *, audit_ if repository[0] != request.project_id: raise SubmissionError("REPOSITORY_PROJECT_CONFLICT", 409) if request.idempotency_key: + identity = hashlib.sha256( + f"{request.project_id}\0{request.idempotency_key}".encode("utf-8") + ).hexdigest() + retired = connection.execute( + "SELECT request_digest FROM ep_operational_identity_tombstones " + "WHERE identity_kind='submission_idempotency' AND identity_digest=?", + (identity,), + ).fetchone() if "ep_operational_identity_tombstones" in tables else None + if retired is not None: + code = ( + "IDEMPOTENCY_RETIRED" if retired[0] == _idempotency_request_digest(request) + else "IDEMPOTENCY_CONFLICT" + ) + raise SubmissionError(code, 409) duplicate = connection.execute( "SELECT submission_id,created_at,state,admission,repository_id,producer_id,producer_type,producer_version,prompt,constraints,correlation_id,mission_id,engineering_action_id,transport_receipt_id,transport " "FROM ep_submissions WHERE project_id=? AND idempotency_key=?", @@ -538,7 +600,7 @@ def submit(connection: sqlite3.Connection, request: SubmissionRequest, *, audit_ str(duplicate[1]), str(duplicate[3]), str(duplicate[14]), str(duplicate[5]), True, receipt, ) - submission_id, created_at = "sub-" + secrets.token_hex(16), _now() + submission_id, created_at = _allocate_submission_id(connection), _now() # Admission intentionally validates CENTRAL topology only at submission # time. Agent selection, leases and provider execution remain downstream. admission, state = "ADMITTED", "QUEUED" diff --git a/src/engineering_platform/templates/workspace-config.json b/src/engineering_platform/templates/workspace-config.json index c2313b8c..fc44f87f 100644 --- a/src/engineering_platform/templates/workspace-config.json +++ b/src/engineering_platform/templates/workspace-config.json @@ -3,7 +3,7 @@ "platform": { "id": "engineering-platform", "name": "Engineering Platform", - "version": "2.3.80", + "version": "2.3.81", "generation": 2, "documentation_namespace": "engineering-platform", "capability_registry_version": 1 diff --git a/tests/engineering/dashboard.spec.mjs b/tests/engineering/dashboard.spec.mjs index 49a038a4..81018931 100644 --- a/tests/engineering/dashboard.spec.mjs +++ b/tests/engineering/dashboard.spec.mjs @@ -9475,7 +9475,14 @@ test.describe("Engineering Status browser smoke", () => { }); test("never renders a white focus ring on visible interactive elements", async ({ page }) => { + await page.route("**/api/provider-login-status", (route) => route.fulfill({ json: { + providers: { + codex: { provider: "CODEX", state: "UNAVAILABLE" }, + github: { provider: "GITHUB", state: "READY" }, + }, + } })); await page.goto(dashboardUrl, { waitUntil: "domcontentloaded" }); + await page.waitForFunction(() => document.body?.classList.contains("dashboard-ready")); await page.evaluate(() => { document.querySelectorAll("details").forEach((element) => { element.open = true; }); }); diff --git a/tests/engineering/test_central_operational_reset.py b/tests/engineering/test_central_operational_reset.py new file mode 100644 index 00000000..3150a0fb --- /dev/null +++ b/tests/engineering/test_central_operational_reset.py @@ -0,0 +1,956 @@ +from __future__ import annotations + +from collections import namedtuple +from contextlib import closing, contextmanager, redirect_stdout +import gc +import hashlib +import io +import json +import os +from pathlib import Path +import sqlite3 +import tempfile +import unittest +import warnings +from unittest.mock import patch + +from engineering_platform import central_operational_reset as reset +from engineering_platform import server, submission_service +from engineering_platform.operational_installation_lock import OperationalInstallationLock +from engineering_platform.storage import sqlite_connection + + +class SimulatedPostCommitFinishCrash(BaseException): + """Model process death after durable COMPLETED and before active-root thaw.""" + + +class CentralOperationalResetTests(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory( + dir=Path(tempfile.gettempdir()).resolve() + ) + self.root = Path(self.temporary.name) / "data" + self.backups = Path(self.temporary.name) / "protected-backups" + server.initialize(self.root) + + def tearDown(self) -> None: + self.temporary.cleanup() + + def _populate(self) -> submission_service.SubmissionRequest: + with sqlite_connection(self.root / "epdata.sqlite") as connection: + connection.execute( + "INSERT INTO ep_project_registrations VALUES(?,?,?,?,?)", + ("project-a", "{}", "ACTIVE", "now", "now"), + ) + connection.execute( + "INSERT INTO ep_repository_registrations VALUES(?,?,?,?,?,?,?)", + ("repo-a", "project-a", "repo-a", "authority", "{}", "now", "now"), + ) + connection.execute( + "INSERT INTO ep_consumer_registrations VALUES(?,?,?,?,?,?,?,?)", + ("consumer-a", "project-a", "ACTIVE", "now", "now", None, None, "{}"), + ) + connection.execute( + "INSERT INTO ep_consumer_credentials VALUES(?,?,?,?,?,?,?,?,?)", + ("credential-a", "consumer-a", "project-a", b"v" * 32, b"f" * 32, + "now", None, None, None), + ) + connection.execute( + "INSERT INTO execution_projections VALUES(?,?,?,?,?)", + ("configuration-a", "CONFIGURATION", "{}", "digest", "now"), + ) + connection.execute( + "INSERT INTO execution_projections VALUES(?,?,?,?,?)", + ("cache-a", "PROJECTION", "{}", "digest", "now"), + ) + request = submission_service.SubmissionRequest( + "project-a", "repo-a", "human", "HUMAN", "1", "synthetic prompt", "HTTP", + idempotency_key="already-used", + ) + submission_service.submit(connection, request) + (self.root / "artifacts").mkdir() + (self.root / "artifacts" / "terminal.json").write_text("{}", encoding="utf-8") + (self.root / "file-inbox").mkdir() + (self.root / "file-inbox" / "historical.json").write_text("{}", encoding="utf-8") + return request + + def _prepared(self, *, operation_id: str = "reset-fixture-0001") -> tuple[str, str]: + plan = reset.preview(self.root) + reset.prepare( + self.root, operation_id=operation_id, plan_digest=str(plan["plan_digest"]), + backup_root=self.backups, + allowed_fk_findings=tuple(plan["review_required"]), + ) + return operation_id, str(plan["plan_digest"]) + + def test_schema_owned_inventory_is_exhaustive_and_preview_is_read_only(self) -> None: + before = hashlib.sha256((self.root / "epdata.sqlite").read_bytes()).hexdigest() + plan = reset.preview(self.root) + after = hashlib.sha256((self.root / "epdata.sqlite").read_bytes()).hexdigest() + self.assertEqual(before, after) + self.assertEqual(plan["execution_state"], "ALLOWED") + self.assertFalse(plan["unknown_tables"]) + self.assertFalse(plan["missing_tables"]) + self.assertTrue(plan["schema_objects"]) + self.assertFalse(any( + str(item["name"]).startswith("sqlite_") for item in plan["schema_objects"] + )) + with sqlite_connection(f"file:{self.root / 'epdata.sqlite'}?mode=ro", uri=True) as connection: + self.assertEqual(reset._tables(connection), set(reset.MAPPED_TABLES)) + + def test_preview_of_older_schema_blocks_without_migrating(self) -> None: + with closing(sqlite3.connect(self.root / "epdata.sqlite")) as connection: + connection.execute("DELETE FROM engineering_schema_migrations WHERE version=68") + connection.execute("INSERT INTO engineering_schema_migrations(version) VALUES(67)") + connection.commit() + before = hashlib.sha256((self.root / "epdata.sqlite").read_bytes()).hexdigest() + plan = reset.preview(self.root) + self.assertEqual(hashlib.sha256((self.root / "epdata.sqlite").read_bytes()).hexdigest(), before) + self.assertIn("SCHEMA_UNSUPPORTED", plan["blocking_codes"]) + + def test_unknown_table_and_unknown_root_file_block_but_known_recovery_files_do_not(self) -> None: + (self.root / "central.sqlite").write_bytes(b"") + (self.root / "epdata.sqlite.pre-2.3.32.backup").write_bytes(b"backup") + (self.root / ".forge-ep-consumer-recovery.lock").write_text("", encoding="utf-8") + self.assertNotIn("EXTERNAL_CLASSIFICATION_INCOMPLETE", reset.preview(self.root)["blocking_codes"]) + (self.root / "user-notes.txt").write_text("preserve", encoding="utf-8") + plan = reset.preview(self.root) + self.assertIn("EXTERNAL_CLASSIFICATION_INCOMPLETE", plan["blocking_codes"]) + self.assertTrue((self.root / "user-notes.txt").is_file()) + (self.root / "user-notes.txt").unlink() + with sqlite_connection(self.root / "epdata.sqlite") as connection: + connection.execute("CREATE TABLE plugin_unknown(id INTEGER PRIMARY KEY)") + self.assertIn("TABLE_CLASSIFICATION_INCOMPLETE", reset.preview(self.root)["blocking_codes"]) + + def test_complete_reset_preserves_bindings_credentials_and_configuration(self) -> None: + request = self._populate() + with sqlite_connection(self.root / "epdata.sqlite") as connection: + retired_submission_id = str(connection.execute( + "SELECT submission_id FROM ep_submissions" + ).fetchone()[0]) + operation_id, digest = self._prepared() + with sqlite_connection(self.root / "epdata.sqlite") as connection: + with self.assertRaisesRegex(submission_service.SubmissionError, "PLATFORM_MAINTENANCE_ACTIVE"): + submission_service.submit(connection, request) + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + result = reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(result["quick_check"], ["ok"]) + self.assertEqual(result["foreign_key_errors"], 0) + self.assertFalse(result["operational_rows_remaining"]) + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + self.assertEqual(connection.execute("SELECT COUNT(*) FROM ep_project_registrations").fetchone(), (1,)) + self.assertEqual(connection.execute("SELECT COUNT(*) FROM ep_repository_registrations").fetchone(), (1,)) + self.assertEqual(connection.execute("SELECT verifier FROM ep_consumer_credentials").fetchone(), (b"v" * 32,)) + self.assertEqual(connection.execute( + "SELECT COUNT(*) FROM execution_projections WHERE classification='CONFIGURATION'" + ).fetchone(), (1,)) + self.assertEqual(connection.execute("SELECT COUNT(*) FROM ep_submissions").fetchone(), (0,)) + with self.assertRaisesRegex(submission_service.SubmissionError, "IDEMPOTENCY_RETIRED"): + submission_service.submit(connection, request) + changed = submission_service.SubmissionRequest( + **{**request.__dict__, "prompt": "different bytes"} + ) + with self.assertRaisesRegex(submission_service.SubmissionError, "IDEMPOTENCY_CONFLICT"): + submission_service.submit(connection, changed) + fresh = submission_service.SubmissionRequest( + **{**request.__dict__, "idempotency_key": None, "prompt": "new work"} + ) + with patch.object( + submission_service.secrets, "token_hex", + side_effect=[retired_submission_id.removeprefix("sub-"), "n" * 32], + ): + allocated = submission_service.submit(connection, fresh) + self.assertEqual(allocated.submission_id, "sub-" + "n" * 32) + archive = self.root / "operational-reset-archive" / operation_id + self.assertTrue((archive / "artifacts" / "terminal.json").is_file()) + self.assertTrue((archive / "file-inbox" / "historical.json").is_file()) + + def test_empty_reset_is_meaningful_and_idempotent_for_same_operation(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-empty-0001") + first = reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + second = reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(first["generation_after"], 1) + self.assertEqual(second["generation_after"], 1) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + + def test_changed_source_plan_target_and_request_are_rejected(self) -> None: + plan = reset.preview(self.root) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + connection.execute( + "INSERT INTO engineering_component_logs(component,payload,created_at) VALUES('ep_server','{}','now')" + ) + with self.assertRaisesRegex(reset.OperationalResetError, "PLAN_DIGEST_MISMATCH"): + reset.prepare(self.root, operation_id="reset-stale-0001", + plan_digest=str(plan["plan_digest"]), backup_root=self.backups) + other = Path(self.temporary.name) / "other" + server.initialize(other) + with self.assertRaisesRegex(reset.OperationalResetError, "PLAN_DIGEST_MISMATCH"): + reset.prepare(other, operation_id="reset-other-0001", + plan_digest=str(plan["plan_digest"]), backup_root=self.backups) + + def test_existing_installation_lock_blocks_prepare(self) -> None: + plan = reset.preview(self.root) + lock = OperationalInstallationLock(self.root) + lock.acquire("update-lock-0001") + try: + with self.assertRaisesRegex(reset.OperationalResetError, "MAINTENANCE_LOCK_BUSY"): + reset.prepare(self.root, operation_id="reset-locked-0001", + plan_digest=str(plan["plan_digest"]), backup_root=self.backups) + finally: + lock.release("update-lock-0001") + + def test_crash_before_database_commit_has_no_half_purge_and_resumes_same_operation(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-crash-0001") + with patch.object(reset, "_delete_operational", side_effect=RuntimeError("crash")): + with self.assertRaisesRegex(RuntimeError, "crash"): + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + self.assertEqual(connection.execute("SELECT COUNT(*) FROM ep_submissions").fetchone(), (1,)) + resumed = reset.resume(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(resumed["operation"]["state"], "VERIFIED") + + def test_resume_after_database_commit_verifies_without_a_second_reset(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-after-commit-0001") + applied = reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(applied["state"], "DB_APPLIED") + with self.assertRaisesRegex(reset.OperationalResetError, "ABORT_AFTER_EFFECT_FORBIDDEN"): + reset.abort(self.root, operation_id=operation_id, plan_digest=digest) + resumed = reset.resume(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(resumed["operation"]["state"], "VERIFIED") + self.assertEqual(resumed["dataset_generation"], 1) + + def test_durable_fence_survives_new_connection_and_blocks_server_restart(self) -> None: + operation_id, _digest = self._prepared(operation_id="reset-fence-0001") + self.assertTrue(reset.maintenance_active(self.root)) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + with self.assertRaisesRegex(sqlite3.IntegrityError, "EP_OPERATIONAL_MAINTENANCE_ACTIVE"): + connection.execute( + "INSERT INTO engineering_component_logs(component,payload,created_at) " + "VALUES('ep_server','{}','now')" + ) + with self.assertRaisesRegex(server.ServerConfigurationError, + "EP_OPERATIONAL_MAINTENANCE_ACTIVE"): + server.start(self.root) + self.assertEqual(reset.status(self.root, operation_id=operation_id)["operation"]["state"], + "AUTHORIZED") + + def test_partial_artifact_move_reconciles_forward(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-artifact-0001") + with reset._central_connection(self.root / "epdata.sqlite") as connection: + connection.execute("BEGIN IMMEDIATE") + reset._transition( + connection, operation_id, "AUTHORIZED", "ARTIFACTS_ARCHIVING", + ) + connection.execute("COMMIT") + archive = self.root / "operational-reset-archive" / operation_id + archive.mkdir(parents=True) + (self.root / "artifacts").replace(archive / "artifacts") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + self.assertTrue((archive / "artifacts" / "terminal.json").is_file()) + self.assertTrue((self.root / "artifacts").is_dir()) + + def test_operational_fk_requires_exact_approval_and_preservation_fk_blocks(self) -> None: + connection = sqlite3.connect(self.root / "epdata.sqlite") + connection.execute("PRAGMA foreign_keys=OFF") + connection.execute( + "INSERT INTO ep_execution_leases VALUES(?,?,?,?,?,?)", + ("lease-orphan", "run-missing", "host", "now", "later", None), + ) + connection.commit(); connection.close() + plan = reset.preview(self.root) + self.assertEqual(plan["execution_state"], "REVIEW_REQUIRED") + with self.assertRaisesRegex(reset.OperationalResetError, "OPERATIONAL_FK_APPROVAL_MISMATCH"): + reset.prepare(self.root, operation_id="reset-fk-0001", plan_digest=str(plan["plan_digest"]), + backup_root=self.backups) + operation_id = "reset-fk-0002" + reset.prepare(self.root, operation_id=operation_id, plan_digest=str(plan["plan_digest"]), + backup_root=self.backups, allowed_fk_findings=tuple(plan["review_required"])) + reset.apply(self.root, operation_id=operation_id, plan_digest=str(plan["plan_digest"])) + self.assertEqual(reset.verify(self.root, operation_id=operation_id, + plan_digest=str(plan["plan_digest"]))["foreign_key_errors"], 0) + + other = Path(self.temporary.name) / "preserve-fk" + server.initialize(other) + connection = sqlite3.connect(other / "epdata.sqlite") + connection.execute("PRAGMA foreign_keys=OFF") + connection.execute( + "INSERT INTO ep_external_producer_bindings VALUES(?,?,?,?,?,?,?,?,?,?,?,?)", + ("binding", "GITHUB", "REPOSITORY", "owner/repo", "missing-project", + "missing-repo", "ACTIVE", 1, "now", "actor", "now", "{}"), + ) + connection.commit(); connection.close() + blocked = reset.preview(other) + self.assertIn("PRESERVATION_INTEGRITY_FAILED", blocked["blocking_codes"]) + + def test_backup_paths_data_root_symlink_and_active_runtime_fail_closed(self) -> None: + plan = reset.preview(self.root) + inside = self.root / "backups" + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PATH_INSIDE_ACTIVE_DATA_ROOT"): + reset.prepare(self.root, operation_id="reset-inside-0001", + plan_digest=str(plan["plan_digest"]), backup_root=inside) + + other = Path(self.temporary.name) / "symlink-root" + server.initialize(other) + (Path(self.temporary.name) / "outside").mkdir() + (other / "artifacts").symlink_to(Path(self.temporary.name) / "outside", target_is_directory=True) + self.assertIn("EXTERNAL_SYMLINK_UNSAFE", reset.preview(other)["blocking_codes"]) + + active = Path(self.temporary.name) / "active" + server.initialize(active) + (active / "runtime.json").write_text(json.dumps({"pid": os.getpid()}), encoding="utf-8") + self.assertIn("TARGET_WRITER_ACTIVE", reset.preview(active)["blocking_codes"]) + + backup_target = Path(self.temporary.name) / "backup-target" + backup_target.mkdir() + backup_link = Path(self.temporary.name) / "backup-link" + backup_link.symlink_to(backup_target, target_is_directory=True) + other = Path(self.temporary.name) / "backup-fixture" + server.initialize(other) + other_plan = reset.preview(other) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PATH_UNSAFE"): + reset.prepare(other, operation_id="reset-backup-link-0001", + plan_digest=str(other_plan["plan_digest"]), backup_root=backup_link) + self.assertTrue(reset.maintenance_active(other)) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_ROOT_BINDING_MISMATCH"): + reset.resume(other, operation_id="reset-backup-link-0001", + plan_digest=str(other_plan["plan_digest"]), + backup_root=Path(self.temporary.name) / "different-backup") + aborted = reset.abort(other, operation_id="reset-backup-link-0001", + plan_digest=str(other_plan["plan_digest"])) + self.assertEqual(aborted["state"], "ABORTED") + self.assertFalse(reset.maintenance_active(other)) + + backup_parent_link = Path(self.temporary.name) / "backup-parent-link" + backup_parent_link.symlink_to(backup_target, target_is_directory=True) + parent_fixture = Path(self.temporary.name) / "backup-parent-fixture" + server.initialize(parent_fixture) + parent_plan = reset.preview(parent_fixture) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PATH_UNSAFE"): + reset.prepare(parent_fixture, operation_id="reset-backup-parent-link-0001", + plan_digest=str(parent_plan["plan_digest"]), + backup_root=backup_parent_link / "not-created") + + data_link = Path(self.temporary.name) / "data-link" + data_link.symlink_to(self.root, target_is_directory=True) + with self.assertRaisesRegex(reset.OperationalResetError, "DATA_ROOT_UNSAFE"): + reset.preview(data_link) + + def test_backup_restore_validation_permissions_space_and_invalid_archive(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-backup-proof-0001") + backup = self.backups / operation_id + backup.chmod(0o755) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PERMISSIONS_UNSAFE"): + reset.verify_backup(backup, operation_id=operation_id, plan_digest=digest) + backup.chmod(0o700) + database = backup / "central.sqlite" + database.write_bytes(database.read_bytes() + b"tamper") + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_BINDING_INVALID"): + reset.verify_backup(backup, operation_id=operation_id, plan_digest=digest) + + low_space = Path(self.temporary.name) / "low-space" + server.initialize(low_space) + low_plan = reset.preview(low_space) + usage = namedtuple("usage", "total used free")(1024, 1024, 0) + with patch.object(reset.shutil, "disk_usage", return_value=usage): + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_SPACE_INSUFFICIENT"): + reset.prepare(low_space, operation_id="reset-low-space-0001", + plan_digest=str(low_plan["plan_digest"]), + backup_root=Path(self.temporary.name) / "low-space-backup") + + unwritable_shape = Path(self.temporary.name) / "not-a-directory" + unwritable_shape.write_text("file", encoding="utf-8") + unsafe = Path(self.temporary.name) / "unsafe-shape" + server.initialize(unsafe) + unsafe_plan = reset.preview(unsafe) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PATH_UNSAFE"): + reset.prepare(unsafe, operation_id="reset-unwritable-0001", + plan_digest=str(unsafe_plan["plan_digest"]), + backup_root=unwritable_shape) + + archive_root = Path(self.temporary.name) / "archive-fixture" + archive_backup = Path(self.temporary.name) / "archive-backup" + server.initialize(archive_root) + (archive_root / "artifacts").mkdir() + (archive_root / "artifacts" / "evidence.json").write_text("{}", encoding="utf-8") + archive_plan = reset.preview(archive_root) + reset.prepare(archive_root, operation_id="reset-invalid-archive-0001", + plan_digest=str(archive_plan["plan_digest"]), backup_root=archive_backup) + reset.apply(archive_root, operation_id="reset-invalid-archive-0001", + plan_digest=str(archive_plan["plan_digest"])) + archived = archive_root / "operational-reset-archive" / "reset-invalid-archive-0001" + (archived / "artifacts" / "evidence.json").write_text("changed", encoding="utf-8") + with self.assertRaisesRegex(reset.OperationalResetError, + "ARCHIVE_RECONCILIATION_FAILED"): + reset.verify(archive_root, operation_id="reset-invalid-archive-0001", + plan_digest=str(archive_plan["plan_digest"])) + + def test_chat_parent_migration_preserves_rows_and_enforces_canonical_run(self) -> None: + with closing(sqlite3.connect(self.root / "epdata.sqlite")) as connection: + connection.execute("PRAGMA foreign_keys=OFF") + connection.execute("DROP TRIGGER execution_chat_messages_immutable_update") + connection.execute("DROP INDEX execution_chat_messages_run_created") + connection.execute("ALTER TABLE execution_chat_messages RENAME TO chat_schema68") + connection.execute( + "CREATE TABLE execution_chat_messages(id INTEGER PRIMARY KEY," + "run_id TEXT NOT NULL REFERENCES prompt_execution_history(run_id)," + "role TEXT NOT NULL,content TEXT NOT NULL,model TEXT,created_at TEXT NOT NULL)" + ) + connection.execute("DROP TABLE chat_schema68") + connection.execute( + "INSERT INTO ep_project_registrations VALUES('p','{}','ACTIVE','now','now')" + ) + connection.execute( + "INSERT INTO ep_execution_runs VALUES('run-chat','p','BLOCKED','now','now',NULL)" + ) + connection.execute( + "INSERT INTO execution_chat_messages VALUES(1,'run-chat','user','kept',NULL,'now')" + ) + connection.execute("DELETE FROM engineering_schema_migrations WHERE version=68") + connection.execute("INSERT INTO engineering_schema_migrations(version) VALUES(67)") + connection.execute("UPDATE engineering_metadata SET value='67' WHERE key='installation.schema_version'") + connection.execute("ALTER TABLE ep_installations RENAME TO installation_schema68") + connection.execute( + "CREATE TABLE ep_installations(instance_id TEXT PRIMARY KEY,created_at TEXT NOT NULL," + "schema_version INTEGER NOT NULL CHECK(schema_version=67))" + ) + connection.execute( + "INSERT INTO ep_installations SELECT instance_id,created_at,67 FROM installation_schema68" + ) + connection.execute("DROP TABLE installation_schema68") + connection.commit() + server.initialize(self.root) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + connection.execute("PRAGMA foreign_keys=ON") + parent = connection.execute("PRAGMA foreign_key_list(execution_chat_messages)").fetchone()[2] + self.assertEqual(parent, "ep_execution_runs") + self.assertEqual(connection.execute("SELECT content FROM execution_chat_messages").fetchone(), ("kept",)) + self.assertFalse(list(connection.execute("PRAGMA foreign_key_check"))) + with self.assertRaises(sqlite3.IntegrityError): + connection.execute( + "INSERT INTO execution_chat_messages(run_id,role,content,created_at) " + "VALUES('missing','user','blocked','now')" + ) + + def test_apply_refuses_terminal_states_and_aborted_is_not_an_active_submission_fence(self) -> None: + request = self._populate() + operation_id, digest = self._prepared(operation_id="reset-aborted-0001") + reset.abort(self.root, operation_id=operation_id, plan_digest=digest) + self.assertFalse(reset.maintenance_active(self.root)) + with self.assertRaisesRegex(reset.OperationalResetError, + "OPERATION_STATE_INVALID_FOR_APPLY"): + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + fresh = submission_service.SubmissionRequest( + **{**request.__dict__, "idempotency_key": None, "prompt": "after abort"} + ) + self.assertTrue(submission_service.submit(connection, fresh).submission_id) + + completed = Path(self.temporary.name) / "completed" + completed_backup = Path(self.temporary.name) / "completed-backup" + server.initialize(completed) + plan = reset.preview(completed) + reset.prepare( + completed, operation_id="reset-completed-0001", + plan_digest=str(plan["plan_digest"]), backup_root=completed_backup, + ) + reset.apply(completed, operation_id="reset-completed-0001", + plan_digest=str(plan["plan_digest"])) + reset.verify(completed, operation_id="reset-completed-0001", + plan_digest=str(plan["plan_digest"])) + reset.finish(completed, operation_id="reset-completed-0001", + plan_digest=str(plan["plan_digest"])) + with self.assertRaisesRegex(reset.OperationalResetError, + "OPERATION_STATE_INVALID_FOR_APPLY"): + reset.apply(completed, operation_id="reset-completed-0001", + plan_digest=str(plan["plan_digest"])) + + def test_backup_copy_rejects_existing_content_and_nested_symlinks_and_resumes_after_copy_crash(self) -> None: + plan = reset.preview(self.root) + operation_id = "reset-existing-backup-0001" + occupied = self.backups / operation_id + occupied.mkdir(parents=True) + self.backups.chmod(0o700) + occupied.chmod(0o700) + marker = occupied / "foreign.txt" + marker.write_text("do not overwrite", encoding="utf-8") + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_INVALID"): + reset.prepare( + self.root, operation_id=operation_id, + plan_digest=str(plan["plan_digest"]), backup_root=self.backups, + ) + self.assertEqual(marker.read_text(encoding="utf-8"), "do not overwrite") + reset.abort(self.root, operation_id=operation_id, plan_digest=str(plan["plan_digest"])) + + crash_root = Path(self.temporary.name) / "copy-crash" + crash_backup = Path(self.temporary.name) / "copy-crash-backup" + server.initialize(crash_root) + crash_plan = reset.preview(crash_root) + with patch.object(reset, "_copy_external", side_effect=RuntimeError("synthetic crash")): + with self.assertRaisesRegex(RuntimeError, "synthetic crash"): + reset.prepare( + crash_root, operation_id="reset-copy-crash-0001", + plan_digest=str(crash_plan["plan_digest"]), backup_root=crash_backup, + ) + self.assertEqual( + reset.status(crash_root, operation_id="reset-copy-crash-0001")["operation"]["state"], + "PREPARING", + ) + resumed = reset.resume( + crash_root, operation_id="reset-copy-crash-0001", + plan_digest=str(crash_plan["plan_digest"]), backup_root=crash_backup, + ) + self.assertEqual(resumed["operation"]["state"], "VERIFIED") + self.assertTrue((crash_backup / "reset-copy-crash-0001" / "manifest.json").is_file()) + self.assertTrue(any(path.name.endswith(".partial") for path in crash_backup.iterdir())) + + source_root = Path(self.temporary.name) / "copy-source" + outside = Path(self.temporary.name) / "copy-outside" + destination = Path(self.temporary.name) / "copy-destination" + (source_root / "artifacts").mkdir(parents=True) + outside.mkdir() + (outside / "secret").write_text("secret", encoding="utf-8") + (source_root / "artifacts" / "nested").symlink_to(outside, target_is_directory=True) + destination.mkdir() + entry = [{ + "root": "artifacts", "path": "nested/secret", "sha256": reset._file_digest(outside / "secret"), + }] + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_SOURCE_CHANGED"): + reset._copy_external(source_root, destination, entry) + (source_root / "artifacts" / "nested").unlink() + (source_root / "artifacts" / "nested").mkdir() + (source_root / "artifacts" / "nested" / "secret").write_text("secret", encoding="utf-8") + (destination / "files").mkdir() + (destination / "files" / "artifacts").symlink_to(outside, target_is_directory=True) + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_PATH_UNSAFE"): + reset._copy_external(source_root, destination, entry) + self.assertFalse((outside / "nested").exists()) + + def test_stored_manifest_digest_is_mandatory_for_apply_and_verify(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-manifest-binding-0001") + manifest = self.backups / operation_id / "manifest.json" + original = manifest.read_bytes() + unexpected = self.backups / operation_id / "unexpected.bin" + unexpected.write_bytes(b"foreign") + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_FILE_SET_INVALID"): + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + unexpected.unlink() + manifest.write_bytes(original.rstrip() + b" \n") + with self.assertRaisesRegex(reset.OperationalResetError, + "BACKUP_MANIFEST_DIGEST_MISMATCH"): + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + manifest.write_bytes(original) + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + manifest.write_bytes(original.rstrip() + b" \n") + with self.assertRaisesRegex(reset.OperationalResetError, + "BACKUP_MANIFEST_DIGEST_MISMATCH"): + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + manifest.unlink() + with self.assertRaisesRegex(reset.OperationalResetError, "BACKUP_INVALID"): + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + + def test_maintenance_tables_are_owner_guarded_and_preview_requires_every_fence(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-guard-0001") + with sqlite_connection(self.root / "epdata.sqlite") as connection: + with self.assertRaises(sqlite3.DatabaseError): + connection.execute( + "UPDATE ep_operational_reset_operations SET updated_at='forged' WHERE operation_id=?", + (operation_id,), + ) + with self.assertRaises(sqlite3.DatabaseError): + connection.execute( + "INSERT INTO ep_operational_identity_tombstones VALUES(?,?,?,?,?)", + ("run_id", "forged", None, operation_id, "now"), + ) + self.assertEqual( + reset.abort(self.root, operation_id=operation_id, plan_digest=digest)["state"], + "ABORTED", + ) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + connection.execute("DROP TRIGGER ep_operational_reset_block_ep_submissions_insert") + plan = reset.preview(self.root) + self.assertIn("WRITER_FENCE_INCOMPLETE", plan["blocking_codes"]) + self.assertIn("ep_operational_reset_block_ep_submissions_insert", + plan["missing_writer_fences"]) + + def test_finish_reproves_backup_database_bindings_and_all_active_ingest_routes(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-finish-fence-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + delayed = self.root / "file-inbox" / "delayed.json" + delayed.write_text("{}", encoding="utf-8") + with self.assertRaisesRegex(reset.OperationalResetError, + "ARCHIVE_RECONCILIATION_AMBIGUOUS"): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual(reset.status(self.root, operation_id=operation_id)["operation"]["state"], + "VERIFIED") + delayed.unlink() + pending = self.root / "runtime" / "pending-central-data-import.json" + pending.parent.mkdir(exist_ok=True) + pending.write_text("{}", encoding="utf-8") + with self.assertRaisesRegex(reset.OperationalResetError, + "POST_RESET_VERIFICATION_FAILED"): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + pending.unlink() + self.assertEqual( + reset.finish(self.root, operation_id=operation_id, plan_digest=digest)["state"], + "COMPLETED", + ) + + def test_finish_atomically_isolates_file_arriving_after_last_empty_scan(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-finish-race-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + original_verify = reset._verify_under_lock + injected = False + + def inject_after_empty_scan(*args: object, **kwargs: object) -> dict[str, object]: + nonlocal injected + result = original_verify(*args, **kwargs) + if not injected: + injected = True + delayed = self.root / "file-inbox" / "delayed-old-event.json" + delayed.write_text('{"event":"old"}', encoding="utf-8") + return result + + with patch.object(reset, "_verify_under_lock", side_effect=inject_after_empty_scan): + completed = reset.finish( + self.root, operation_id=operation_id, plan_digest=digest, + ) + + self.assertEqual("COMPLETED", completed["state"]) + self.assertFalse(reset.maintenance_active(self.root)) + self.assertFalse((self.root / "file-inbox" / "delayed-old-event.json").exists()) + boundary = Path(str(completed["finish_boundary_path"])) + isolated = boundary / "file-inbox" / "delayed-old-event.json" + self.assertEqual('{"event":"old"}', isolated.read_text(encoding="utf-8")) + self.assertRegex(str(completed["finish_boundary_digest"]), r"^sha256:[0-9a-f]{64}$") + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + + def test_finish_boundary_rotation_resumes_same_operation_after_partial_crash(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-finish-crash-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + real_rename = os.rename + rotated = 0 + + def crash_during_second_route(source: object, target: object, *args: object, + **kwargs: object) -> object: + nonlocal rotated + source_path, target_path = Path(source), Path(target) + if "finish-boundary" in target_path.parts and source_path.is_dir(): + rotated += 1 + if rotated == 2: + raise OSError("synthetic process loss during route rotation") + return real_rename(source, target, *args, **kwargs) + + with patch.object(reset.os, "rename", side_effect=crash_during_second_route): + with self.assertRaisesRegex( + reset.OperationalResetError, "FINISH_BOUNDARY_ROTATION_FAILED", + ): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + + interrupted = reset.status(self.root, operation_id=operation_id)["operation"] + self.assertEqual("VERIFIED", interrupted["state"]) + self.assertIsNotNone(interrupted["finish_boundary_path"]) + self.assertTrue(reset.maintenance_active(self.root)) + completed = reset.finish( + self.root, operation_id=operation_id, plan_digest=digest, + ) + self.assertEqual("COMPLETED", completed["state"]) + self.assertFalse(reset.maintenance_active(self.root)) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + + def test_finish_freezes_new_roots_and_rejects_post_rotation_active_file(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-finish-frozen-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + real_rotate = reset._rotate_finish_boundary + + def inject_after_real_rotation(*args: object, **kwargs: object) -> dict[str, object]: + proof = real_rotate(*args, **kwargs) + inbox = self.root / "file-inbox" + self.assertEqual(0o500, inbox.stat().st_mode & 0o777) + # Simulate an installation-owner writer bypassing the ordinary + # permission failure. The post-rotation proof must still prevent + # COMPLETED rather than accepting this active old event. + inbox.chmod(0o700) + (inbox / "late-after-rotation.json").write_text("{}", encoding="utf-8") + return proof + + with patch.object( + reset, "_rotate_finish_boundary", side_effect=inject_after_real_rotation, + ): + with self.assertRaisesRegex( + reset.OperationalResetError, "ACTIVE_ROOT_NOT_FROZEN", + ): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + + operation = reset.status(self.root, operation_id=operation_id)["operation"] + self.assertEqual("VERIFIED", operation["state"]) + self.assertTrue(reset.maintenance_active(self.root)) + self.assertTrue((self.root / "file-inbox" / "late-after-rotation.json").is_file()) + + def test_finish_roots_are_frozen_until_commit_then_thawed_for_new_generation(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-finish-modes-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + observed_during_commit: dict[str, int] = {} + real_transition = reset._transition + + def inspect_completed_transition( + connection: sqlite3.Connection, selected: str, before: str, after: str, + assignments: str = "", parameters: tuple[object, ...] = (), + ) -> None: + if before == "VERIFIED" and after == "COMPLETED": + observed_during_commit.update(reset._active_roots_state(self.root)) + real_transition( + connection, selected, before, after, assignments, parameters, + ) + + with patch.object(reset, "_transition", side_effect=inspect_completed_transition): + completed = reset.finish( + self.root, operation_id=operation_id, plan_digest=digest, + ) + + self.assertEqual("COMPLETED", completed["state"]) + self.assertEqual({name: 0o500 for name in reset._EFFECT_DIRECTORIES}, + observed_during_commit) + self.assertEqual({name: 0o700 for name in reset._EFFECT_DIRECTORIES}, + reset._active_roots_state(self.root)) + late = self.root / "file-inbox" / "new-generation.json" + late.write_text("{}", encoding="utf-8") + self.assertEqual( + "COMPLETED", + reset.finish(self.root, operation_id=operation_id, plan_digest=digest)["state"], + ) + self.assertTrue(late.is_file()) + self.assertEqual( + "COMPLETED", + reset.verify(self.root, operation_id=operation_id, plan_digest=digest)["state"], + ) + + def test_post_commit_pre_thaw_crash_resumes_same_completed_operation(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-finish-thaw-crash-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + + with patch.object( + reset, "_thaw_active_roots", side_effect=SimulatedPostCommitFinishCrash(), + ): + with self.assertRaises(SimulatedPostCommitFinishCrash): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + + operation = reset.status(self.root, operation_id=operation_id)["operation"] + self.assertEqual("COMPLETED", operation["state"]) + self.assertFalse(reset.maintenance_active(self.root)) + self.assertEqual({name: 0o500 for name in reset._EFFECT_DIRECTORIES}, + reset._active_roots_state(self.root)) + with self.assertRaises(PermissionError): + (self.root / "file-inbox" / "blocked.json").write_text("{}", encoding="utf-8") + + resumed = reset.resume( + self.root, operation_id=operation_id, plan_digest=digest, + ) + self.assertEqual("COMPLETED", resumed["operation"]["state"]) + self.assertEqual({name: 0o700 for name in reset._EFFECT_DIRECTORIES}, + reset._active_roots_state(self.root)) + + def test_completed_finish_refuses_active_root_symlink_without_touching_target(self) -> None: + operation_id, digest = self._prepared(operation_id="reset-finish-symlink-0001") + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + reset.verify(self.root, operation_id=operation_id, plan_digest=digest) + with patch.object( + reset, "_thaw_active_roots", side_effect=SimulatedPostCommitFinishCrash(), + ): + with self.assertRaises(SimulatedPostCommitFinishCrash): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + + inbox = self.root / "file-inbox" + inbox.rmdir() + outside = Path(self.temporary.name) / "outside-inbox" + outside.mkdir() + sentinel = outside / "keep.txt" + sentinel.write_text("preserve", encoding="utf-8") + inbox.symlink_to(outside, target_is_directory=True) + with self.assertRaisesRegex(reset.OperationalResetError, "ACTIVE_ROOT_UNSAFE"): + reset.finish(self.root, operation_id=operation_id, plan_digest=digest) + self.assertEqual("preserve", sentinel.read_text(encoding="utf-8")) + + def test_nested_preserved_external_inventory_is_explicit_and_unknown_appdata_blocks(self) -> None: + operations = self.root / "operations" / "update-known-0001" + operations.mkdir(parents=True) + (operations / "operation.json").write_text("{}", encoding="utf-8") + plan = reset.preview(self.root) + self.assertNotIn("EXTERNAL_CLASSIFICATION_INCOMPLETE", plan["blocking_codes"]) + classified = {str(item["path"]): item["classification"] for item in plan["preserved_external"]} + self.assertEqual(classified["operations/update-known-0001/operation.json"], + "INSTALLATION_OPERATION_AUDIT") + operation_unknown = operations / "user-payload.bin" + operation_unknown.write_bytes(b"keep") + self.assertIn("operations/update-known-0001/user-payload.bin", + reset.preview(self.root)["unknown_external_paths"]) + operation_unknown.unlink() + runtime = self.root / "runtime" + runtime.mkdir(exist_ok=True) + unknown = runtime / "user-payload.bin" + unknown.write_bytes(b"keep") + blocked = reset.preview(self.root) + self.assertIn("EXTERNAL_CLASSIFICATION_INCOMPLETE", blocked["blocking_codes"]) + self.assertIn("runtime/user-payload.bin", blocked["unknown_external_paths"]) + self.assertEqual(unknown.read_bytes(), b"keep") + + def test_cli_argument_and_internal_errors_are_stable_secret_free_contracts(self) -> None: + def invoke(arguments: list[str]) -> tuple[int, dict[str, object]]: + output = io.StringIO() + with redirect_stdout(output): + code = reset.main(arguments) + return code, json.loads(output.getvalue()) + + code, receipt = invoke([]) + self.assertEqual((code, receipt["error_code"]), (2, "CLI_ARGUMENT_INVALID")) + required = { + "contract_version", "product", "command", "operation_id", "state", "allowed", + "target", "profile", "dataset_generation", "plan_digest", + "relevant_revision_digest", "backup", "counts", "blockers", "integrity", + "preserved_bindings_digest", "error_code", + } + self.assertTrue(required.issubset(receipt)) + with patch.object(reset, "preview", side_effect=ValueError("secret=do-not-print")): + code, receipt = invoke(["preview", "--data-root", str(self.root)]) + self.assertEqual((code, receipt["error_code"]), (2, "MAINTENANCE_COMMAND_FAILED")) + self.assertNotIn("do-not-print", json.dumps(receipt)) + self.assertIsNone(receipt["target"]["database_path"]) + + def test_central_chat_writer_enables_foreign_keys_on_the_used_connection(self) -> None: + self._populate() + statements: list[str] = [] + with sqlite_connection(self.root / "epdata.sqlite") as connection: + submission_id = str(connection.execute( + "SELECT submission_id FROM ep_submissions ORDER BY created_at LIMIT 1" + ).fetchone()[0]) + connection.execute( + "INSERT INTO ep_execution_runs VALUES(?,?,?,?,?,?)", + ("run-chat-fk", "project-a", "BLOCKED", "now", "now", None), + ) + connection.execute( + "INSERT INTO ep_parity_lifecycle_dispatches(" + "submission_id,project_id,repository_id,run_id,state,prompt_path,claimed_at,updated_at) " + "VALUES(?,?,?,?,?,?,?,?)", + (submission_id, "project-a", "repo-a", "run-chat-fk", "BLOCKED", + "CENTRAL:prompt", "now", "now"), + ) + + real_connection = sqlite_connection + + @contextmanager + def observed_connection(*args: object, **kwargs: object): + with real_connection(*args, **kwargs) as connection: + connection.set_trace_callback(statements.append) + yield connection + + with patch.object(server.storage, "sqlite_connection", observed_connection): + server._central_console_append_chat_message( + self.root, "project-a", "run-chat-fk", "user", "safe", + ) + self.assertTrue(any(statement.casefold() == "pragma foreign_keys=on" + for statement in statements)) + with sqlite_connection(self.root / "epdata.sqlite") as connection: + self.assertEqual(connection.execute( + "SELECT content FROM execution_chat_messages WHERE run_id='run-chat-fk'" + ).fetchone(), ("safe",)) + + def test_reset_exception_paths_release_sqlite_connections_without_resource_warnings(self) -> None: + self._populate() + operation_id, digest = self._prepared(operation_id="reset-resource-0001") + with warnings.catch_warnings(record=True) as observed: + warnings.simplefilter("always", ResourceWarning) + with patch.object(reset, "_delete_operational", side_effect=RuntimeError("crash")): + with self.assertRaises(RuntimeError): + reset.apply(self.root, operation_id=operation_id, plan_digest=digest) + gc.collect() + self.assertFalse([item for item in observed if item.category is ResourceWarning]) + + def test_shared_json_contract_contains_no_verifier(self) -> None: + self._populate() + plan = reset.preview(self.root) + contract = reset.contract_readback( + self.root, command="preview", operation_id=None, + ) + self.assertEqual(contract["contract_version"], "operational-reset-v1") + self.assertEqual(contract["product"], "engineering-platform") + self.assertEqual(contract["target"]["instance_id"], plan["target"]["instance_id"]) + rendered = json.dumps(contract) + self.assertNotIn((b"v" * 32).hex(), rendered) + self.assertNotIn((b"f" * 32).hex(), rendered) + self.assertEqual( + contract["details"], + { + "credentials_included_in_receipt": False, + "projection": "PERSISTED_PUBLIC_MAINTENANCE_STATE", + }, + ) + + def test_cli_state_machine_emits_stable_shared_contract(self) -> None: + def invoke(*arguments: str) -> tuple[int, dict[str, object]]: + output = io.StringIO() + with redirect_stdout(output): + code = reset.main(list(arguments)) + return code, json.loads(output.getvalue()) + + common = ("--data-root", str(self.root)) + code, preview = invoke("preview", *common) + self.assertEqual(code, 0) + operation_id = "reset-cli-0001" + mutation = ( + "--operation-id", operation_id, + "--plan-digest", str(preview["plan_digest"]), + ) + code, prepared = invoke( + "prepare", *common, *mutation, "--backup-root", str(self.backups), + ) + self.assertEqual((code, prepared["state"]), (0, "AUTHORIZED")) + self.assertTrue(prepared["backup"]["verified"]) + code, status = invoke("status", *common, "--operation-id", operation_id) + self.assertEqual((code, status["state"]), (0, "AUTHORIZED")) + self.assertEqual(invoke("apply", *common, *mutation)[1]["state"], "DB_APPLIED") + self.assertEqual(invoke("verify", *common, *mutation)[1]["state"], "VERIFIED") + self.assertEqual(invoke("finish", *common, *mutation)[1]["state"], "COMPLETED") + required = { + "contract_version", "product", "command", "operation_id", "state", "allowed", + "target", "profile", "dataset_generation", "plan_digest", + "relevant_revision_digest", "backup", "counts", "blockers", "integrity", + "preserved_bindings_digest", + } + self.assertTrue(required.issubset(prepared)) + + code, error = invoke( + "prepare", *common, "--operation-id", "reset-cli-error-0001", + "--plan-digest", "sha256:wrong", "--backup-root", str(self.backups), + ) + self.assertEqual((code, error["error"]), (2, "PLAN_DIGEST_MISMATCH")) + + abort_root = Path(self.temporary.name) / "cli-abort" + abort_backup = Path(self.temporary.name) / "cli-abort-backup" + server.initialize(abort_root) + abort_preview = reset.preview(abort_root) + abort_common = ("--data-root", str(abort_root)) + abort_mutation = ( + "--operation-id", "reset-cli-abort-0001", + "--plan-digest", str(abort_preview["plan_digest"]), + ) + self.assertEqual(invoke( + "prepare", *abort_common, *abort_mutation, "--backup-root", str(abort_backup), + )[1]["state"], "AUTHORIZED") + self.assertEqual(invoke("abort", *abort_common, *abort_mutation)[1]["state"], "ABORTED") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/engineering/test_engineering_operational_documentation.py b/tests/engineering/test_engineering_operational_documentation.py index b06e987a..f63d79d1 100644 --- a/tests/engineering/test_engineering_operational_documentation.py +++ b/tests/engineering/test_engineering_operational_documentation.py @@ -82,6 +82,30 @@ def test_execution_host_operations_documents_the_recent_host_increments(self) -> self.assertIn("Retry Execution", operations) self.assertIn("Queue Recovery", operations) + def test_central_operational_reset_runbook_is_product_owned_and_explicit(self) -> None: + reset = (ROOT / "docs" / "engineering" / "EP_CENTRAL_OPERATIONAL_RESET_V1.md").read_text( + encoding="utf-8" + ) + for command in ( + "preview", "prepare", "apply", "status", "resume", "verify", "finish", "abort", + ): + self.assertIn(f"engineering-platform-maintenance {command}", reset) + self.assertIn("ROOT_CAUSE_UNCONFIRMED", reset) + self.assertIn("ep_consumer_credentials", reset) + self.assertIn("operational-reset-v1", reset) + self.assertIn("DESTRUCTIVE", reset) + self.assertIn("must not automatically prepare/apply", reset) + self.assertIn("tools/qualification/operational_reset_coordinator.py", reset) + for command in ( + "preview", "prepare", "revalidate", "apply --product forge", + "apply --product engineering-platform", "verify", "authorize-resume", + "finish --product forge", "finish --product engineering-platform", + "status", "reconcile", "resume --product engineering-platform", + ): + self.assertIn(command, reset) + self.assertIn("RECONCILIATION_REQUIRED", reset) + self.assertIn("no automatic destructive rollback", reset) + def test_execution_host_architecture_bounds_provider_interruption_recovery(self) -> None: architecture = (ROOT / "docs" / "engineering" / "EXECUTION_HOST_ARCHITECTURE.md").read_text( encoding="utf-8" diff --git a/tests/engineering/test_operational_reset_coordinator.py b/tests/engineering/test_operational_reset_coordinator.py new file mode 100644 index 00000000..106330f0 --- /dev/null +++ b/tests/engineering/test_operational_reset_coordinator.py @@ -0,0 +1,418 @@ +"""Qualification matrix for the subprocess-only operational reset coordinator.""" +from __future__ import annotations + +import ast +import importlib.util +import json +import os +from pathlib import Path +import shutil +import stat +import subprocess +import sys +import tempfile +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +MODULE = ROOT / "tools" / "qualification" / "operational_reset_coordinator.py" +FIXTURE = ROOT / "tests" / "fixtures" / "operational_reset_coordinator_fixture_cli.py" +SPEC = importlib.util.spec_from_file_location("operational_reset_coordinator", MODULE) +assert SPEC and SPEC.loader +coordinator_module = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = coordinator_module +SPEC.loader.exec_module(coordinator_module) + + +class SimulatedCoordinatorCrash(BaseException): + """Model process death after an owning effect and before receipt update.""" + + +class OperationalResetCoordinatorTests(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory() + # macOS exposes /var as a compatibility symlink. Qualification uses + # the canonical temporary path so the receipt store can reject every + # caller-supplied symlink component without weakening the test. + self.root = Path(self.temporary.name).resolve() + self.receipts = self.root / "coordinator-receipts" + self.forge_root = self.root / "forge-data" + self.ep_root = self.root / "ep-data" + self.ep_backup = self.root / "ep-backups" + self.forge_cli = self.root / "forge-reset-fixture" + self.ep_cli = self.root / "ep-reset-fixture" + for source, target in ((FIXTURE, self.forge_cli), (FIXTURE, self.ep_cli)): + shutil.copy2(source, target) + target.chmod(0o700) + self.config = coordinator_module.CoordinationConfig.create( + coordinator_id="joint-reset-fixture-001", + forge_cli=self.forge_cli, + forge_data_root=self.forge_root, + forge_operation_id="forge-reset-fixture-001", + ep_cli=self.ep_cli, + ep_data_root=self.ep_root, + ep_operation_id="ep-reset-fixture-001", + ep_backup_root=self.ep_backup, + ) + self.store = coordinator_module.ReceiptStore( + self.receipts, self.config.coordinator_id, + ) + self.coordinator = coordinator_module.OperationalResetCoordinator( + self.store, command_timeout_seconds=10, + ) + + def tearDown(self) -> None: + self.temporary.cleanup() + + def prepare_both(self) -> None: + self.assertEqual("BOTH_PREVIEWED", self.coordinator.preview(self.config)["state"]) + self.assertEqual("BACKUPS_VERIFIED", self.coordinator.prepare()["state"]) + self.assertEqual("PLANS_REVALIDATED", self.coordinator.revalidate()["state"]) + + def apply_both(self, first: str = "forge") -> None: + second = "engineering-platform" if first == "forge" else "forge" + expected = "FORGE_APPLIED" if first == "forge" else "EP_APPLIED" + self.assertEqual(expected, self.coordinator.apply(first)["state"]) + self.assertEqual("BOTH_APPLIED", self.coordinator.apply(second)["state"]) + + def complete_both(self, first: str = "forge") -> dict[str, object]: + self.prepare_both() + self.apply_both(first) + self.assertEqual("BOTH_VERIFIED", self.coordinator.verify()["state"]) + self.assertEqual("RESUME_AUTHORIZED", self.coordinator.authorize_resume()["state"]) + self.assertEqual("RESUME_AUTHORIZED", self.coordinator.finish(first)["state"]) + second = "engineering-platform" if first == "forge" else "forge" + return self.coordinator.finish(second) + + def owning_state(self, root: Path) -> dict[str, object]: + return json.loads((root / "fixture-owning-state.json").read_text(encoding="utf-8")) + + def command_log(self, root: Path) -> list[dict[str, object]]: + return [ + json.loads(line) for line in (root / "fixture-command-log.jsonl").read_text( + encoding="utf-8" + ).splitlines() + ] + + def test_both_owning_resets_succeed_and_receipt_is_private_and_complete(self) -> None: + result = self.complete_both() + + self.assertEqual("COMPLETE", result["state"]) + self.assertEqual(self.config.forge_operation_id, result["products"]["forge"]["operation_id"]) + self.assertEqual( + self.config.ep_operation_id, + result["products"]["engineering-platform"]["operation_id"], + ) + for product in coordinator_module.PRODUCTS: + observed = result["products"][product] + for field in ("target_digest", "plan_digest", "backup_digest", "result_digest"): + self.assertRegex(observed[field], r"^sha256:[0-9a-f]{64}$") + self.assertEqual(0o700, stat.S_IMODE(self.store.directory.stat().st_mode)) + self.assertEqual(0o600, stat.S_IMODE(self.store.receipt_path.stat().st_mode)) + + def test_forge_succeeds_ep_fails_and_neither_product_is_finished(self) -> None: + self.prepare_both() + self.coordinator.apply("forge") + (self.ep_root / "fail-apply").touch() + + with self.assertRaises(coordinator_module.ProductCommandError): + self.coordinator.apply("engineering-platform") + + receipt = self.store.load() + self.assertEqual("RECONCILIATION_REQUIRED", receipt["state"]) + self.assertEqual("APPLIED", self.owning_state(self.forge_root)["state"]) + self.assertEqual("AUTHORIZED", self.owning_state(self.ep_root)["state"]) + self.assertNotIn("finish", {item["action"] for item in self.command_log(self.forge_root)}) + self.assertNotIn("finish", {item["action"] for item in self.command_log(self.ep_root)}) + + def test_ep_succeeds_forge_fails_and_neither_product_is_finished(self) -> None: + self.prepare_both() + self.coordinator.apply("engineering-platform") + (self.forge_root / "fail-apply").touch() + + with self.assertRaises(coordinator_module.ProductCommandError): + self.coordinator.apply("forge") + + receipt = self.store.load() + self.assertEqual("RECONCILIATION_REQUIRED", receipt["state"]) + self.assertEqual("BACKUP_VERIFIED", self.owning_state(self.forge_root)["state"]) + self.assertEqual("DB_APPLIED", self.owning_state(self.ep_root)["state"]) + self.assertNotIn("finish", {item["action"] for item in self.command_log(self.forge_root)}) + self.assertNotIn("finish", {item["action"] for item in self.command_log(self.ep_root)}) + + def test_coordinator_crash_after_owning_apply_reconciles_same_operations(self) -> None: + self.prepare_both() + + def crash(boundary: str) -> None: + if boundary == "after-forge-apply": + raise SimulatedCoordinatorCrash() + + crashing = coordinator_module.OperationalResetCoordinator( + self.store, command_timeout_seconds=10, fault_hook=crash, + ) + with self.assertRaises(SimulatedCoordinatorCrash): + crashing.apply("forge") + self.assertEqual("PLANS_REVALIDATED", self.store.load()["state"]) + self.assertTrue(self.store.load()["milestones"]["forge_apply_started"]) + self.assertEqual("APPLIED", self.owning_state(self.forge_root)["state"]) + + restarted = coordinator_module.OperationalResetCoordinator( + self.store, command_timeout_seconds=10, + ) + self.assertEqual("FORGE_APPLIED", restarted.reconcile()["state"]) + self.assertEqual("BOTH_APPLIED", restarted.apply("engineering-platform")["state"]) + + def test_resume_is_forbidden_before_revalidation_and_before_product_apply_admission(self) -> None: + self.assertEqual("BOTH_PREVIEWED", self.coordinator.preview(self.config)["state"]) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "plan revalidation"): + self.coordinator.resume("forge") + self.assertEqual(["preview"], [item["action"] for item in self.command_log(self.forge_root)]) + + self.assertEqual("BACKUPS_VERIFIED", self.coordinator.prepare()["state"]) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "plan revalidation"): + self.coordinator.resume("engineering-platform") + self.assertNotIn("resume", {item["action"] for item in self.command_log(self.ep_root)}) + + self.assertEqual("PLANS_REVALIDATED", self.coordinator.revalidate()["state"]) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "apply admission"): + self.coordinator.resume("engineering-platform") + self.assertNotIn("resume", {item["action"] for item in self.command_log(self.ep_root)}) + + def test_resume_reconciles_only_the_durably_admitted_crashed_apply(self) -> None: + self.prepare_both() + + def crash(boundary: str) -> None: + if boundary == "after-forge-apply": + raise SimulatedCoordinatorCrash() + + crashing = coordinator_module.OperationalResetCoordinator( + self.store, command_timeout_seconds=10, fault_hook=crash, + ) + with self.assertRaises(SimulatedCoordinatorCrash): + crashing.apply("forge") + + restarted = coordinator_module.OperationalResetCoordinator( + self.store, command_timeout_seconds=10, + ) + self.assertEqual("FORGE_APPLIED", restarted.resume("forge")["state"]) + self.assertIn("resume", {item["action"] for item in self.command_log(self.forge_root)}) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "apply admission"): + restarted.resume("engineering-platform") + + def test_product_process_restart_and_status_readback_use_durable_state(self) -> None: + self.prepare_both() + restarted = coordinator_module.OperationalResetCoordinator( + coordinator_module.ReceiptStore(self.receipts, self.config.coordinator_id), + command_timeout_seconds=10, + ) + + status = restarted.status() + + self.assertEqual("PLANS_REVALIDATED", status["state"]) + self.assertEqual("BACKUP_VERIFIED", status["owning_readback"]["forge"]["state"]) + self.assertEqual("AUTHORIZED", status["owning_readback"]["engineering-platform"]["state"]) + for root in (self.forge_root, self.ep_root): + pids = [item["pid"] for item in self.command_log(root)] + self.assertEqual(len(pids), len(set(pids))) + + def test_bound_cli_bytes_are_rechecked_before_maintenance(self) -> None: + self.coordinator.preview(self.config) + with self.forge_cli.open("a", encoding="utf-8") as stream: + stream.write("\n# changed after coordinated preview\n") + + with self.assertRaisesRegex( + coordinator_module.CoordinatorError, "owning CLI bytes changed", + ): + self.coordinator.prepare() + + self.assertEqual("BOTH_PREVIEWED", self.store.load()["state"]) + for root in (self.forge_root, self.ep_root): + self.assertEqual(["preview"], [item["action"] for item in self.command_log(root)]) + + def test_target_identity_change_is_rejected_during_revalidation(self) -> None: + self.coordinator.preview(self.config) + self.coordinator.prepare() + (self.forge_root / "changed-target").touch() + + with self.assertRaisesRegex( + coordinator_module.ProductCommandError, "REVALIDATION_CHANGED", + ): + self.coordinator.revalidate() + + self.assertEqual("RECONCILIATION_REQUIRED", self.store.load()["state"]) + + def test_delayed_old_callbacks_are_projected_historical_without_new_work(self) -> None: + result = self.complete_both() + self.assertEqual("COMPLETE", result["state"]) + (self.forge_root / "delayed-old-callback").touch() + (self.ep_root / "delayed-old-callback").touch() + + status = self.coordinator.status() + + for product in coordinator_module.PRODUCTS: + owning = status["owning_readback"][product] + self.assertEqual("REJECTED_HISTORICAL", owning["details"]["old_callback_projection"]) + self.assertEqual(0, owning["details"]["new_missions"]) + self.assertEqual(0, owning["details"]["new_provider_invocations"]) + self.assertEqual(0, owning["details"]["new_submissions"]) + + def test_no_hidden_mission_provider_or_submission_command_is_invoked(self) -> None: + self.complete_both(first="engineering-platform") + allowed = {"preview", "prepare", "apply", "verify", "finish", "status"} + for root in (self.forge_root, self.ep_root): + log = self.command_log(root) + self.assertTrue({item["action"] for item in log} <= allowed) + state = self.coordinator.status()["owning_readback"] + for owning in state.values(): + self.assertEqual( + {"missions": 0, "provider_invocations": 0, "submissions": 0}, + owning["counts"], + ) + + def test_finish_is_forbidden_until_both_products_verify(self) -> None: + self.prepare_both() + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "both verifications"): + self.coordinator.authorize_resume() + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "forbidden"): + self.coordinator.finish("forge") + self.coordinator.apply("forge") + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "both owning resets"): + self.coordinator.verify() + self.coordinator.apply("engineering-platform") + self.coordinator.verify() + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "forbidden"): + self.coordinator.finish("forge") + + def test_finish_reproves_both_product_readiness_before_first_finish(self) -> None: + self.prepare_both() + self.apply_both() + self.coordinator.verify() + self.coordinator.authorize_resume() + (self.ep_root / "fail-verify").touch() + + with self.assertRaises(coordinator_module.ProductCommandError): + self.coordinator.finish("forge") + + self.assertEqual("RECONCILIATION_REQUIRED", self.store.load()["state"]) + self.assertNotIn("finish", {item["action"] for item in self.command_log(self.forge_root)}) + self.assertEqual("VERIFIED", self.owning_state(self.forge_root)["state"]) + self.assertEqual("VERIFIED", self.owning_state(self.ep_root)["state"]) + + def test_sequential_finish_failure_preserves_partial_state_for_explicit_reconcile(self) -> None: + self.prepare_both() + self.apply_both() + self.coordinator.verify() + self.coordinator.authorize_resume() + self.assertEqual("RESUME_AUTHORIZED", self.coordinator.finish("forge")["state"]) + (self.ep_root / "fail-finish").touch() + + with self.assertRaises(coordinator_module.ProductCommandError): + self.coordinator.finish("engineering-platform") + + self.assertEqual("COMPLETED", self.owning_state(self.forge_root)["state"]) + self.assertEqual("VERIFIED", self.owning_state(self.ep_root)["state"]) + self.assertEqual("RESUME_AUTHORIZED", self.coordinator.reconcile()["state"]) + (self.ep_root / "fail-finish").unlink() + self.assertEqual("COMPLETE", self.coordinator.finish("engineering-platform")["state"]) + forge_actions = [item["action"] for item in self.command_log(self.forge_root)] + self.assertGreaterEqual(forge_actions.count("verify"), 4) + + def test_receipt_symlink_and_concurrent_lock_fail_closed(self) -> None: + unsafe_root = self.root / "unsafe-receipts" + unsafe_root.symlink_to(self.receipts, target_is_directory=True) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "symlink"): + coordinator_module.ReceiptStore(unsafe_root, "joint-reset-fixture-002") + self.coordinator.preview(self.config) + with self.store.locked(): + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "another coordinator"): + coordinator_module.OperationalResetCoordinator( + coordinator_module.ReceiptStore( + self.receipts, self.config.coordinator_id, + ), command_timeout_seconds=10, + ).status() + + def test_status_rejects_a_symlink_in_an_existing_receipt_parent(self) -> None: + self.coordinator.preview(self.config) + unsafe_parent = self.root / "unsafe-parent" + unsafe_parent.symlink_to(self.root, target_is_directory=True) + + def unsafe_status() -> object: + unsafe_store = coordinator_module.ReceiptStore( + unsafe_parent / self.receipts.name, self.config.coordinator_id, + ) + return coordinator_module.OperationalResetCoordinator( + unsafe_store, command_timeout_seconds=10, + ).status() + + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "symlink component"): + unsafe_status() + + relocated = self.root / "relocated-receipts" + self.receipts.rename(relocated) + self.receipts.symlink_to(relocated, target_is_directory=True) + with self.assertRaisesRegex(coordinator_module.CoordinatorError, "symlink component"): + self.coordinator.status() + + def test_coordinator_source_has_no_product_import_or_database_access(self) -> None: + source = MODULE.read_text(encoding="utf-8") + tree = ast.parse(source) + imported = { + alias.name.split(".")[0] + for node in ast.walk(tree) + if isinstance(node, ast.Import) + for alias in node.names + } | { + (node.module or "").split(".")[0] + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) + } + self.assertTrue({"engineering_platform", "forge", "sqlite3"}.isdisjoint(imported)) + self.assertNotIn("sqlite3.connect", source) + self.assertIn("subprocess.run", source) + + def test_real_ep_owning_preview_smoke_uses_subprocess_boundary(self) -> None: + real_ep_root = self.root / "real-ep-data" + environment = { + **os.environ, + "PYTHONPATH": str(ROOT / "src"), + "PYTHONNOUSERSITE": "1", + "PYTHONSAFEPATH": "1", + } + initialized = subprocess.run( + [ + sys.executable, "-m", "engineering_platform.server", "init", + "--data-root", str(real_ep_root), + ], + cwd=ROOT, env=environment, capture_output=True, text=True, check=False, timeout=60, + ) + self.assertEqual(0, initialized.returncode, initialized.stderr) + real_ep_cli = self.root / "engineering-platform-maintenance-real" + real_ep_cli.write_text( + "#!/bin/sh\n" + f"exec {sys.executable!s} -m engineering_platform.central_operational_reset \"$@\"\n", + encoding="utf-8", + ) + real_ep_cli.chmod(0o700) + real_config = coordinator_module.CoordinationConfig.create( + coordinator_id="joint-reset-real-ep-001", + forge_cli=self.forge_cli, forge_data_root=self.forge_root, + forge_operation_id="forge-reset-real-ep-001", + ep_cli=real_ep_cli, ep_data_root=real_ep_root, + ep_operation_id="ep-reset-real-ep-001", ep_backup_root=self.ep_backup, + ) + real = coordinator_module.OperationalResetCoordinator( + coordinator_module.ReceiptStore(self.receipts, real_config.coordinator_id), + command_timeout_seconds=30, + runner=lambda command, **kwargs: subprocess.run(command, env=environment, **kwargs), + ) + + result = real.preview(real_config) + + self.assertEqual("BOTH_PREVIEWED", result["state"]) + self.assertEqual( + 68, result["products"]["engineering-platform"]["target"]["schema_version"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/engineering/test_owner_credential_recovery.py b/tests/engineering/test_owner_credential_recovery.py index 0107ddce..da6634e6 100644 --- a/tests/engineering/test_owner_credential_recovery.py +++ b/tests/engineering/test_owner_credential_recovery.py @@ -200,7 +200,7 @@ def setUp(self) -> None: self.temporary = tempfile.TemporaryDirectory() self.root = Path(self.temporary.name) / "ep" identity = server.initialize(self.root) - self.assertEqual(server.SERVER_STORE_SCHEMA_VERSION, 67) + self.assertEqual(server.SERVER_STORE_SCHEMA_VERSION, 68) self.instance = identity.instance_id with self.connection() as connection: connection.execute( diff --git a/tests/fixtures/operational_reset_coordinator_fixture_cli.py b/tests/fixtures/operational_reset_coordinator_fixture_cli.py new file mode 100644 index 00000000..73a970c7 --- /dev/null +++ b/tests/fixtures/operational_reset_coordinator_fixture_cli.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Subprocess-only owning CLI fixture for coordinator fault qualification.""" +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +import sys + + +def _digest(label: str) -> str: + return "sha256:" + hashlib.sha256(label.encode("utf-8")).hexdigest() + + +def _argument(name: str) -> str | None: + try: + return sys.argv[sys.argv.index(name) + 1] + except (ValueError, IndexError): + return None + + +def main() -> int: + product = "forge" if "forge" in Path(sys.argv[0]).name else "engineering-platform" + if product == "forge": + action = sys.argv[sys.argv.index("reset") + 1] + else: + action = sys.argv[1] + root_value = _argument("--data-root") + if root_value is None: + raise SystemExit("fixture requires --data-root") + root = Path(root_value) + root.mkdir(mode=0o700, parents=True, exist_ok=True) + state_path = root / "fixture-owning-state.json" + log_path = root / "fixture-command-log.jsonl" + with log_path.open("a", encoding="utf-8") as stream: + stream.write(json.dumps({ + "action": action, "pid": os.getpid(), "product": product, + }, sort_keys=True) + "\n") + operation_id = _argument("--operation-id") + if (root / f"fail-{action}").exists(): + print(json.dumps({"error": f"INJECTED_{action.upper()}_FAILURE"}, sort_keys=True)) + return 2 + if state_path.exists(): + state = json.loads(state_path.read_text(encoding="utf-8")) + else: + state = {"state": "ALLOWED", "operation_id": None} + if action == "prepare": + state = { + "state": "BACKUP_VERIFIED" if product == "forge" else "AUTHORIZED", + "operation_id": operation_id, + } + state_path.write_text(json.dumps(state, sort_keys=True), encoding="utf-8") + elif action == "apply": + state["state"] = "APPLIED" if product == "forge" else "DB_APPLIED" + state_path.write_text(json.dumps(state, sort_keys=True), encoding="utf-8") + elif action in {"verify", "resume"}: + # A readiness readback must never regress an already-finished owner. + if state["state"] != "COMPLETED": + state["state"] = "VERIFIED" + state_path.write_text(json.dumps(state, sort_keys=True), encoding="utf-8") + elif action == "finish": + state["state"] = "COMPLETED" + state_path.write_text(json.dumps(state, sort_keys=True), encoding="utf-8") + + plan_digest = _digest(product + ":plan") + backup_digest = _digest(product + ":backup") + verification_digest = _digest(product + ":verification") + target = { + "instance_id": product + "-fixture-instance" + ( + "-changed" if (root / "changed-target").exists() else "" + ), + "database_path": str(root / ("forge.db" if product == "forge" else "epdata.sqlite")), + "database_identity": _digest(product + ":database"), + "schema_version": 38 if product == "forge" else 68, + } + details: dict[str, object] = { + "request_digest": _digest(product + ":request"), + "old_callback_projection": ( + "REJECTED_HISTORICAL" if (root / "delayed-old-callback").exists() else "NOT_OBSERVED" + ), + "new_missions": 0, + "new_provider_invocations": 0, + "new_submissions": 0, + } + if state["state"] in {"VERIFIED", "COMPLETED"} and product == "forge": + details["verification_digest"] = verification_digest + envelope = { + "contract_version": "operational-reset-v1", + "product": product, + "command": action, + "operation_id": operation_id or state.get("operation_id"), + "state": state["state"], + "allowed": True, + "target": target, + "profile": ( + "forge-operational-history-v1" if product == "forge" + else "EP_CENTRAL_OPERATIONAL_HISTORY_V1" + ), + "dataset_generation": 1 if state["state"] in { + "APPLIED", "DB_APPLIED", "VERIFIED", "COMPLETED", + } else 0, + "plan_digest": plan_digest, + "relevant_revision_digest": _digest(product + ":source"), + "backup": ( + None if state["state"] == "ALLOWED" else { + "manifest": str(root / "fixture-backup" / "manifest.json"), + "digest": backup_digest, + "verified": True, + } + ), + "counts": {"missions": 0, "provider_invocations": 0, "submissions": 0}, + "blockers": [], + "integrity": {"quick_check": "ok", "foreign_key_errors": 0}, + "preserved_bindings_digest": _digest(product + ":bindings"), + "details": details, + } + print(json.dumps(envelope, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/qualification/operational_reset_coordinator.py b/tools/qualification/operational_reset_coordinator.py new file mode 100644 index 00000000..02b9f9d7 --- /dev/null +++ b/tools/qualification/operational_reset_coordinator.py @@ -0,0 +1,1041 @@ +#!/usr/bin/env python3 +"""Qualification-only coordinator for two product-owned operational resets. + +The coordinator owns only its secret-free progress receipt. Forge and +Engineering Platform remain the sole authorities for preview, maintenance, +backup, apply, verification, recovery and finish. They are invoked only as +subprocesses through their installed local CLIs; this module never imports a +product package or opens either product database. +""" +from __future__ import annotations + +import argparse +from contextlib import contextmanager +from dataclasses import dataclass +from datetime import datetime, timezone +import fcntl +import hashlib +import json +import os +from pathlib import Path +import re +import stat +import subprocess +import tempfile +from typing import Callable, Iterator, Mapping, Sequence + + +CONTRACT_VERSION = "cross-product-operational-reset-coordinator-v1" +PRODUCT_CONTRACT = "operational-reset-v1" +PRODUCTS = ("forge", "engineering-platform") +STATES = frozenset({ + "DISCOVERED", "BOTH_PREVIEWED", "BOTH_MAINTENANCE", "BACKUPS_VERIFIED", + "PLANS_REVALIDATED", "FORGE_APPLIED", "EP_APPLIED", "BOTH_APPLIED", + "BOTH_VERIFIED", "RESUME_AUTHORIZED", "COMPLETE", "RECONCILIATION_REQUIRED", +}) +_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{7,127}$") +_DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$") +_MAX_OUTPUT = 2 * 1024 * 1024 +_MILESTONE_KEYS = frozenset({ + "previewed", "maintenance", "backups_verified", "plans_revalidated", + "forge_apply_started", "ep_apply_started", "forge_applied", "ep_applied", + "forge_verified", "ep_verified", "resume_authorized", "forge_finished", + "ep_finished", +}) + + +class CoordinatorError(ValueError): + """The coordinated operation cannot progress safely.""" + + +class ProductCommandError(CoordinatorError): + """One owning CLI rejected or failed its exact operation.""" + + def __init__(self, product: str, action: str, code: str) -> None: + super().__init__(f"{product} {action} failed: {code}") + self.product, self.action, self.code = product, action, code + + +def _utcnow() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _canonical(value: object) -> bytes: + return json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False, + ).encode("ascii") + + +def _digest(value: object) -> str: + return "sha256:" + hashlib.sha256(_canonical(value)).hexdigest() + + +def _file_digest(path: Path) -> str: + candidate = path.expanduser().resolve(strict=True) + if not candidate.is_file(): + raise CoordinatorError(f"owning CLI is not a regular file: {path}") + hasher = hashlib.sha256() + with candidate.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + hasher.update(chunk) + return "sha256:" + hasher.hexdigest() + + +def _validated_identifier(value: str, label: str) -> str: + if not isinstance(value, str) or _IDENTIFIER.fullmatch(value) is None: + raise CoordinatorError(f"{label} is invalid") + return value + + +def _absolute_path(value: Path, label: str, *, executable: bool = False) -> Path: + candidate = value.expanduser().absolute() + if not candidate.is_absolute(): + raise CoordinatorError(f"{label} must be absolute") + if executable and (not candidate.exists() or not os.access(candidate, os.X_OK)): + raise CoordinatorError(f"{label} is not an executable file") + return candidate + + +@dataclass(frozen=True) +class CoordinationConfig: + coordinator_id: str + forge_cli: Path + forge_data_root: Path + forge_operation_id: str + ep_cli: Path + ep_data_root: Path + ep_operation_id: str + ep_backup_root: Path + forge_fk_acknowledgements: tuple[str, ...] = () + ep_fk_acknowledgements: tuple[str, ...] = () + + @classmethod + def create( + cls, + *, + coordinator_id: str, + forge_cli: Path, + forge_data_root: Path, + forge_operation_id: str, + ep_cli: Path, + ep_data_root: Path, + ep_operation_id: str, + ep_backup_root: Path, + forge_fk_acknowledgements: Sequence[str] = (), + ep_fk_acknowledgements: Sequence[str] = (), + ) -> "CoordinationConfig": + coordinator_id = _validated_identifier(coordinator_id, "coordinator ID") + forge_operation_id = _validated_identifier(forge_operation_id, "Forge operation ID") + ep_operation_id = _validated_identifier(ep_operation_id, "EP operation ID") + if forge_operation_id == ep_operation_id: + raise CoordinatorError("owning operation IDs must be distinct") + return cls( + coordinator_id=coordinator_id, + forge_cli=_absolute_path(forge_cli, "Forge CLI", executable=True), + forge_data_root=_absolute_path(forge_data_root, "Forge data root"), + forge_operation_id=forge_operation_id, + ep_cli=_absolute_path(ep_cli, "EP CLI", executable=True), + ep_data_root=_absolute_path(ep_data_root, "EP data root"), + ep_operation_id=ep_operation_id, + ep_backup_root=_absolute_path(ep_backup_root, "EP backup root"), + forge_fk_acknowledgements=tuple(sorted(set(forge_fk_acknowledgements))), + ep_fk_acknowledgements=tuple(sorted(set(ep_fk_acknowledgements))), + ) + + def payload(self) -> dict[str, object]: + return { + "coordinator_id": self.coordinator_id, + "forge": { + "cli": str(self.forge_cli), "cli_digest": _file_digest(self.forge_cli), + "data_root": str(self.forge_data_root), + "operation_id": self.forge_operation_id, + "fk_acknowledgements": list(self.forge_fk_acknowledgements), + }, + "engineering-platform": { + "cli": str(self.ep_cli), "cli_digest": _file_digest(self.ep_cli), + "data_root": str(self.ep_data_root), + "operation_id": self.ep_operation_id, + "backup_root": str(self.ep_backup_root), + "fk_acknowledgements": list(self.ep_fk_acknowledgements), + }, + } + + @classmethod + def parse(cls, value: object) -> "CoordinationConfig": + if not isinstance(value, Mapping): + raise CoordinatorError("coordinator configuration is invalid") + forge, ep = value.get("forge"), value.get("engineering-platform") + if not isinstance(forge, Mapping) or not isinstance(ep, Mapping): + raise CoordinatorError("coordinator product configuration is invalid") + config = cls.create( + coordinator_id=str(value.get("coordinator_id", "")), + forge_cli=Path(str(forge.get("cli", ""))), + forge_data_root=Path(str(forge.get("data_root", ""))), + forge_operation_id=str(forge.get("operation_id", "")), + ep_cli=Path(str(ep.get("cli", ""))), + ep_data_root=Path(str(ep.get("data_root", ""))), + ep_operation_id=str(ep.get("operation_id", "")), + ep_backup_root=Path(str(ep.get("backup_root", ""))), + forge_fk_acknowledgements=tuple(forge.get("fk_acknowledgements", ())), + ep_fk_acknowledgements=tuple(ep.get("fk_acknowledgements", ())), + ) + expected = config.payload() + if dict(value) != expected: + raise CoordinatorError("coordinator configuration or owning CLI bytes changed") + return config + + +class ReceiptStore: + """Crash-safe one-operation coordinator receipt with a local flock.""" + + def __init__(self, root: Path, coordinator_id: str) -> None: + self.root = root.expanduser().absolute() + self.coordinator_id = _validated_identifier(coordinator_id, "coordinator ID") + self.directory = self.root / coordinator_id + self.receipt_path = self.directory / "receipt.json" + self.lock_path = self.directory / "coordinator.lock" + self._prepare_directory(self.root) + self._prepare_directory(self.directory) + + @staticmethod + def _reject_symlink_components(path: Path) -> None: + current = Path(path.anchor) + for component in path.parts[1:]: + current /= component + try: + metadata = os.lstat(current) + except FileNotFoundError: + continue + if stat.S_ISLNK(metadata.st_mode): + raise CoordinatorError( + f"coordinator receipt path contains a symlink component: {current}" + ) + + @classmethod + def _prepare_directory(cls, path: Path) -> None: + cls._reject_symlink_components(path) + if path.exists() and not path.is_dir(): + raise CoordinatorError(f"coordinator receipt path is not a directory: {path}") + path.mkdir(mode=0o700, parents=True, exist_ok=True) + cls._reject_symlink_components(path) + os.chmod(path, 0o700) + mode = stat.S_IMODE(path.stat().st_mode) + if mode != 0o700 or path.stat().st_uid != os.getuid(): + raise CoordinatorError("coordinator receipt directory permissions are unsafe") + + @contextmanager + def locked(self) -> Iterator[None]: + self._reject_symlink_components(self.directory) + if self.lock_path.is_symlink(): + raise CoordinatorError("coordinator lock path is a symlink") + flags = os.O_RDWR | os.O_CREAT + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(self.lock_path, flags, 0o600) + try: + os.fchmod(descriptor, 0o600) + try: + fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError as error: + raise CoordinatorError("another coordinator process owns this receipt") from error + os.ftruncate(descriptor, 0) + os.write(descriptor, (self.coordinator_id + "\n").encode("ascii")) + os.fsync(descriptor) + yield + finally: + try: + fcntl.flock(descriptor, fcntl.LOCK_UN) + finally: + os.close(descriptor) + + def load(self) -> dict[str, object] | None: + self._reject_symlink_components(self.directory) + if self.receipt_path.is_symlink(): + raise CoordinatorError("coordinator receipt is a symlink") + if not self.receipt_path.exists(): + return None + metadata = self.receipt_path.stat() + if not stat.S_ISREG(metadata.st_mode) or stat.S_IMODE(metadata.st_mode) != 0o600: + raise CoordinatorError("coordinator receipt permissions or type are unsafe") + try: + value = json.loads(self.receipt_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise CoordinatorError("coordinator receipt is unreadable") from error + _validate_receipt(value, self.coordinator_id) + return value + + def save(self, receipt: Mapping[str, object]) -> None: + _validate_receipt(receipt, self.coordinator_id) + self._reject_symlink_components(self.directory) + if self.receipt_path.is_symlink(): + raise CoordinatorError("coordinator receipt is a symlink") + descriptor, temporary_name = tempfile.mkstemp( + prefix=".receipt.", suffix=".json", dir=self.directory, + ) + try: + with os.fdopen(descriptor, "wb") as stream: + stream.write(_canonical(receipt) + b"\n") + stream.flush() + os.fsync(stream.fileno()) + os.chmod(temporary_name, 0o600) + os.replace(temporary_name, self.receipt_path) + directory_descriptor = os.open(self.directory, os.O_RDONLY) + try: + os.fsync(directory_descriptor) + finally: + os.close(directory_descriptor) + except BaseException: + Path(temporary_name).unlink(missing_ok=True) + raise + + +def _validate_receipt(value: object, coordinator_id: str) -> None: + if not isinstance(value, Mapping): + raise CoordinatorError("coordinator receipt is not an object") + required = { + "contract_version", "coordinator_id", "state", "configuration", + "configuration_digest", "products", "milestones", "events", "created_at", "updated_at", + } + if set(value) != required or value.get("contract_version") != CONTRACT_VERSION: + raise CoordinatorError("coordinator receipt contract is invalid") + if value.get("coordinator_id") != coordinator_id or value.get("state") not in STATES: + raise CoordinatorError("coordinator receipt identity or state is invalid") + configuration = value.get("configuration") + if value.get("configuration_digest") != _digest(configuration): + raise CoordinatorError("coordinator configuration digest does not match") + if not isinstance(value.get("products"), Mapping) or set(value["products"]) != set(PRODUCTS): + raise CoordinatorError("coordinator product receipt is invalid") + if not isinstance(value.get("milestones"), Mapping) or not isinstance(value.get("events"), list): + raise CoordinatorError("coordinator progress receipt is invalid") + milestones = value["milestones"] + if set(milestones) != _MILESTONE_KEYS or not all( + isinstance(item, bool) for item in milestones.values() + ): + raise CoordinatorError("coordinator milestone receipt is invalid") + for index, event in enumerate(value["events"], start=1): + if not isinstance(event, Mapping) or event.get("index") != index: + raise CoordinatorError("coordinator event sequence is invalid") + has_revalidation = any( + event.get("state") == "PLANS_REVALIDATED" for event in value["events"] + if isinstance(event, Mapping) + ) + if milestones["plans_revalidated"] != has_revalidation: + raise CoordinatorError("coordinator plan revalidation evidence is invalid") + for product in ("forge", "ep"): + if milestones[f"{product}_apply_started"] and not milestones["plans_revalidated"]: + raise CoordinatorError("coordinator apply admission precedes plan revalidation") + if milestones[f"{product}_applied"] and not milestones[f"{product}_apply_started"]: + raise CoordinatorError("coordinator apply result lacks durable admission evidence") + + +def _empty_product(product: str, operation_id: str) -> dict[str, object]: + return { + "product": product, "operation_id": operation_id, "state": "UNOBSERVED", + "target": None, "target_digest": None, "plan_digest": None, + "relevant_revision_digest": None, "preserved_bindings_digest": None, + "backup_digest": None, "request_digest": None, "result_digest": None, + "dataset_generation": None, "last_receipt_digest": None, "finished": False, + } + + +def _new_receipt(config: CoordinationConfig) -> dict[str, object]: + configuration = config.payload() + now = _utcnow() + receipt: dict[str, object] = { + "contract_version": CONTRACT_VERSION, + "coordinator_id": config.coordinator_id, + "state": "DISCOVERED", + "configuration": configuration, + "configuration_digest": _digest(configuration), + "products": { + "forge": _empty_product("forge", config.forge_operation_id), + "engineering-platform": _empty_product( + "engineering-platform", config.ep_operation_id, + ), + }, + "milestones": { + "previewed": False, "maintenance": False, "backups_verified": False, + "plans_revalidated": False, "forge_apply_started": False, + "ep_apply_started": False, "forge_applied": False, "ep_applied": False, + "forge_verified": False, "ep_verified": False, "resume_authorized": False, + "forge_finished": False, "ep_finished": False, + }, + "events": [], "created_at": now, "updated_at": now, + } + _event(receipt, "DISCOVERED", "coordinator operation discovered") + return receipt + + +def _event( + receipt: dict[str, object], + state: str, + event: str, + *, + product: str | None = None, + evidence: object | None = None, +) -> None: + if state not in STATES: + raise CoordinatorError("coordinator transition state is invalid") + events = receipt["events"] + if not isinstance(events, list): + raise CoordinatorError("coordinator event journal is invalid") + events.append({ + "index": len(events) + 1, "at": _utcnow(), "state": state, "event": event, + "product": product, "evidence_digest": None if evidence is None else _digest(evidence), + }) + receipt["state"] = state + receipt["updated_at"] = _utcnow() + + +def _find_digest(value: object, key: str) -> str | None: + if isinstance(value, Mapping): + candidate = value.get(key) + if isinstance(candidate, str) and _DIGEST.fullmatch(candidate): + return candidate + for item in value.values(): + found = _find_digest(item, key) + if found is not None: + return found + elif isinstance(value, list): + for item in value: + found = _find_digest(item, key) + if found is not None: + return found + return None + + +def _validate_envelope( + product: str, + action: str, + envelope: object, + operation_id: str | None, +) -> dict[str, object]: + if not isinstance(envelope, dict): + raise ProductCommandError(product, action, "NON_OBJECT_RECEIPT") + if envelope.get("contract_version") != PRODUCT_CONTRACT or envelope.get("product") != product: + raise ProductCommandError(product, action, "WRONG_PRODUCT_CONTRACT") + if envelope.get("command") != action: + raise ProductCommandError(product, action, "COMMAND_RECEIPT_MISMATCH") + if operation_id is not None and envelope.get("operation_id") != operation_id: + raise ProductCommandError(product, action, "OPERATION_ID_MISMATCH") + target = envelope.get("target") + if not isinstance(target, dict) or set(target) != { + "instance_id", "database_path", "database_identity", "schema_version", + }: + raise ProductCommandError(product, action, "TARGET_IDENTITY_INVALID") + if not isinstance(target.get("instance_id"), str) or not target["instance_id"]: + raise ProductCommandError(product, action, "TARGET_INSTANCE_INVALID") + plan_digest = envelope.get("plan_digest") + if not isinstance(plan_digest, str) or _DIGEST.fullmatch(plan_digest) is None: + raise ProductCommandError(product, action, "PLAN_DIGEST_INVALID") + if not isinstance(envelope.get("state"), str): + raise ProductCommandError(product, action, "PRODUCT_STATE_INVALID") + return envelope + + +Runner = Callable[..., subprocess.CompletedProcess[str]] +FaultHook = Callable[[str], None] + + +class OperationalResetCoordinator: + """Durably coordinate only the two owning product maintenance CLIs.""" + + def __init__( + self, + store: ReceiptStore, + *, + runner: Runner = subprocess.run, + command_timeout_seconds: float = 1800, + fault_hook: FaultHook | None = None, + ) -> None: + if not 1 <= command_timeout_seconds <= 7200: + raise CoordinatorError("owning CLI timeout is outside the qualified bound") + self.store, self.runner = store, runner + self.command_timeout_seconds = command_timeout_seconds + self.fault_hook = fault_hook or (lambda _boundary: None) + + def _config(self, receipt: Mapping[str, object]) -> CoordinationConfig: + return CoordinationConfig.parse(receipt["configuration"]) + + def _command( + self, + config: CoordinationConfig, + receipt: Mapping[str, object], + product: str, + action: str, + ) -> list[str]: + products = receipt["products"] + current = products[product] + if not isinstance(current, Mapping): + raise CoordinatorError("coordinator product state is invalid") + if product == "forge": + command = [ + str(config.forge_cli), "--data-root", str(config.forge_data_root), + "server", "reset", action, + ] + operation_id = config.forge_operation_id + else: + command = [str(config.ep_cli), action, "--data-root", str(config.ep_data_root)] + operation_id = config.ep_operation_id + if action == "preview": + return command + command.extend(("--operation-id", operation_id)) + if action == "status": + return command + plan_digest = current.get("plan_digest") + if not isinstance(plan_digest, str): + raise CoordinatorError(f"{product} plan digest is unavailable") + command.extend(("--plan-digest", plan_digest)) + if action == "prepare": + acknowledgements = ( + config.forge_fk_acknowledgements if product == "forge" + else config.ep_fk_acknowledgements + ) + flag = ( + "--acknowledge-operational-fk" if product == "forge" + else "--allow-operational-fk" + ) + for finding in acknowledgements: + command.extend((flag, finding)) + if product == "engineering-platform": + command.extend(("--backup-root", str(config.ep_backup_root))) + elif product == "forge" and action in {"apply", "verify", "resume"}: + request_digest = current.get("request_digest") + if not isinstance(request_digest, str): + raise CoordinatorError("Forge request digest is unavailable") + command.extend(("--request-digest", request_digest)) + backup_digest = current.get("backup_digest") + if isinstance(backup_digest, str): + command.extend(("--backup-digest", backup_digest)) + elif action != "resume": + raise CoordinatorError("Forge backup digest is unavailable") + elif product == "engineering-platform" and action == "resume": + command.extend(("--backup-root", str(config.ep_backup_root))) + elif action == "finish" and product == "forge": + result_digest = current.get("result_digest") + if not isinstance(result_digest, str): + raise CoordinatorError("Forge verification digest is unavailable") + # The owning Forge finish command binds its own verification digest. + command = command[:-2] + ["--verification-digest", result_digest] + return command + + def _invoke( + self, + config: CoordinationConfig, + receipt: Mapping[str, object], + product: str, + action: str, + ) -> dict[str, object]: + product_config = config.payload()[product] + if not isinstance(product_config, Mapping): + raise CoordinatorError("owning product configuration is invalid") + cli = Path(str(product_config["cli"])) + if _file_digest(cli) != product_config["cli_digest"]: + raise ProductCommandError(product, action, "OWNING_CLI_CHANGED") + command = self._command(config, receipt, product, action) + try: + completed = self.runner( + command, capture_output=True, text=True, check=False, + timeout=self.command_timeout_seconds, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise ProductCommandError(product, action, type(error).__name__) from error + stdout = completed.stdout.encode("utf-8", errors="replace") + stderr = completed.stderr.encode("utf-8", errors="replace") + if len(stdout) > _MAX_OUTPUT or len(stderr) > _MAX_OUTPUT: + raise ProductCommandError(product, action, "OWNING_CLI_OUTPUT_TOO_LARGE") + lines = [line for line in completed.stdout.splitlines() if line.strip()] + parsed: object = None + if len(lines) == 1: + try: + parsed = json.loads(lines[0]) + except json.JSONDecodeError: + parsed = None + if completed.returncode != 0: + code = parsed.get("error") if isinstance(parsed, Mapping) else None + raise ProductCommandError( + product, action, str(code) if isinstance(code, str) else f"EXIT_{completed.returncode}", + ) + operation_id = None if action == "preview" else str(product_config["operation_id"]) + return _validate_envelope(product, action, parsed, operation_id) + + @staticmethod + def _record_product( + receipt: dict[str, object], product: str, envelope: Mapping[str, object], + ) -> None: + products = receipt["products"] + current = products[product] + target = dict(envelope["target"]) + target_digest = _digest(target) + prior_target = current.get("target_digest") + if prior_target is not None and prior_target != target_digest: + raise ProductCommandError(product, str(envelope["command"]), "TARGET_CHANGED") + prior_plan = current.get("plan_digest") + plan_digest = envelope.get("plan_digest") + if prior_plan is not None and prior_plan != plan_digest: + raise ProductCommandError(product, str(envelope["command"]), "PLAN_CHANGED") + current.update({ + "state": envelope["state"], "target": target, "target_digest": target_digest, + "plan_digest": plan_digest, + "relevant_revision_digest": envelope.get("relevant_revision_digest"), + "preserved_bindings_digest": envelope.get("preserved_bindings_digest"), + "dataset_generation": envelope.get("dataset_generation"), + "last_receipt_digest": _digest(envelope), + }) + backup = envelope.get("backup") + if isinstance(backup, Mapping) and backup.get("verified") is True: + backup_digest = backup.get("digest") + if not isinstance(backup_digest, str) or _DIGEST.fullmatch(backup_digest) is None: + raise ProductCommandError(product, str(envelope["command"]), "BACKUP_DIGEST_INVALID") + previous = current.get("backup_digest") + if previous is not None and previous != backup_digest: + raise ProductCommandError(product, str(envelope["command"]), "BACKUP_CHANGED") + current["backup_digest"] = backup_digest + request_digest = _find_digest(envelope, "request_digest") + if request_digest is not None: + previous = current.get("request_digest") + if previous is not None and previous != request_digest: + raise ProductCommandError(product, str(envelope["command"]), "REQUEST_CHANGED") + current["request_digest"] = request_digest + if envelope.get("state") in {"VERIFIED", "COMPLETED"}: + previous = current.get("result_digest") + explicit = _find_digest(envelope, "verification_digest") + if explicit is not None: + if previous is not None and previous != explicit: + raise ProductCommandError( + product, str(envelope["command"]), "RESULT_CHANGED", + ) + current["result_digest"] = explicit + elif previous is None: + # EP does not currently expose a separate verification digest. Bind + # its first verified semantic result once; later status/finish + # receipts contain command-specific fields and must not silently + # redefine that result identity. + current["result_digest"] = _digest({ + "product": product, "operation_id": current["operation_id"], + "target_digest": target_digest, "plan_digest": plan_digest, + "backup_digest": current.get("backup_digest"), + "dataset_generation": envelope.get("dataset_generation"), + "integrity": envelope.get("integrity"), "details": envelope.get("details"), + "preserved_bindings_digest": envelope.get("preserved_bindings_digest"), + }) + current["finished"] = envelope.get("state") == "COMPLETED" + + @staticmethod + def _product_stage(product: str, state: object) -> str: + if state == "COMPLETED": + return "finished" + if state == "VERIFIED": + return "verified" + if state in ({"DATABASE_APPLIED", "APPLIED"} if product == "forge" else {"DB_APPLIED"}): + return "applied" + if state in ( + {"BACKUP_VERIFIED"} if product == "forge" + else {"AUTHORIZED", "ARTIFACTS_ARCHIVING", "ARTIFACTS_ARCHIVED"} + ): + return "backup" + if state in ({"PREPARED"} if product == "forge" else {"PREPARING"}): + return "maintenance" + if state in {"ERROR", "FAILED", "ABORTED"}: + return "failed" + return "other" + + def _save_observation( + self, + receipt: dict[str, object], + product: str, + action: str, + envelope: Mapping[str, object], + ) -> None: + self._record_product(receipt, product, envelope) + _event( + receipt, str(receipt["state"]), f"{product} {action} observed", + product=product, evidence=envelope, + ) + self.store.save(receipt) + + def _mark_failure( + self, receipt: dict[str, object], error: ProductCommandError, + ) -> None: + evidence = {"product": error.product, "action": error.action, "code": error.code} + _event( + receipt, "RECONCILIATION_REQUIRED", "owning command requires reconciliation", + product=error.product, evidence=evidence, + ) + self.store.save(receipt) + + def preview(self, config: CoordinationConfig) -> dict[str, object]: + with self.store.locked(): + receipt = self.store.load() + if receipt is None: + receipt = _new_receipt(config) + self.store.save(receipt) + elif receipt["configuration_digest"] != _digest(config.payload()): + raise CoordinatorError("coordinator ID already binds another request") + if receipt["state"] != "DISCOVERED": + return self._readback(receipt) + try: + for product in PRODUCTS: + envelope = self._invoke(config, receipt, product, "preview") + self._save_observation(receipt, product, "preview", envelope) + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + milestones = receipt["milestones"] + milestones["previewed"] = True + _event(receipt, "BOTH_PREVIEWED", "both exact owning plans previewed") + self.store.save(receipt) + return self._readback(receipt) + + def prepare(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] not in {"BOTH_PREVIEWED", "RECONCILIATION_REQUIRED"}: + raise CoordinatorError("both previews are required before maintenance") + config = self._config(receipt) + try: + for product in PRODUCTS: + stage = self._product_stage(product, receipt["products"][product]["state"]) + if stage not in {"maintenance", "backup", "applied", "verified", "finished"}: + envelope = self._invoke(config, receipt, product, "prepare") + self.fault_hook(f"after-{product}-prepare") + self._save_observation(receipt, product, "prepare", envelope) + for product in PRODUCTS: + product_receipt = receipt["products"][product] + if ( + self._product_stage(product, product_receipt["state"]) != "backup" + or not isinstance(product_receipt["backup_digest"], str) + or _DIGEST.fullmatch(product_receipt["backup_digest"]) is None + ): + raise ProductCommandError( + product, "prepare", "VERIFIED_BACKUP_REQUIRED", + ) + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + milestones = receipt["milestones"] + milestones["maintenance"] = True + _event(receipt, "BOTH_MAINTENANCE", "both owning writer fences observed") + milestones["backups_verified"] = True + _event(receipt, "BACKUPS_VERIFIED", "both owning backups verified") + self.store.save(receipt) + return self._readback(receipt) + + def revalidate(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] != "BACKUPS_VERIFIED": + raise CoordinatorError("both backups must be verified before plan revalidation") + config = self._config(receipt) + try: + for product in PRODUCTS: + envelope = self._invoke(config, receipt, product, "preview") + current = receipt["products"][product] + if ( + _digest(envelope["target"]) != current["target_digest"] + or envelope["plan_digest"] != current["plan_digest"] + or envelope.get("relevant_revision_digest") != current["relevant_revision_digest"] + or envelope.get("preserved_bindings_digest") != current["preserved_bindings_digest"] + ): + raise ProductCommandError(product, "preview", "REVALIDATION_CHANGED") + _event( + receipt, str(receipt["state"]), f"{product} plan revalidated", + product=product, evidence=envelope, + ) + self.store.save(receipt) + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + receipt["milestones"]["plans_revalidated"] = True + _event(receipt, "PLANS_REVALIDATED", "both plans remain exact under writer fences") + self.store.save(receipt) + return self._readback(receipt) + + def apply(self, product: str) -> dict[str, object]: + if product not in PRODUCTS: + raise CoordinatorError("owning product is invalid") + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] not in {"PLANS_REVALIDATED", "FORGE_APPLIED", "EP_APPLIED"}: + raise CoordinatorError("both exact plans must be revalidated before apply") + config = self._config(receipt) + other = "engineering-platform" if product == "forge" else "forge" + current_stage = self._product_stage(product, receipt["products"][product]["state"]) + started_milestone = ( + "forge_apply_started" if product == "forge" else "ep_apply_started" + ) + try: + if current_stage not in {"applied", "verified", "finished"}: + if not receipt["milestones"][started_milestone]: + receipt["milestones"][started_milestone] = True + _event( + receipt, str(receipt["state"]), + f"{product} owning apply durably admitted", + product=product, + evidence={"joint_gate": "PLANS_REVALIDATED"}, + ) + # Persist the recovery authority before a subprocess can + # commit an owning effect. + self.store.save(receipt) + envelope = self._invoke(config, receipt, product, "apply") + self.fault_hook(f"after-{product}-apply") + self._save_observation(receipt, product, "apply", envelope) + if self._product_stage(product, receipt["products"][product]["state"]) not in { + "applied", "verified", "finished", + }: + raise ProductCommandError(product, "apply", "APPLIED_STATE_REQUIRED") + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + milestone = "forge_applied" if product == "forge" else "ep_applied" + receipt["milestones"][milestone] = True + other_stage = self._product_stage(other, receipt["products"][other]["state"]) + if other_stage in {"applied", "verified", "finished"}: + state = "BOTH_APPLIED" + else: + state = "FORGE_APPLIED" if product == "forge" else "EP_APPLIED" + _event(receipt, state, f"{product} owning reset applied", product=product) + self.store.save(receipt) + return self._readback(receipt) + + def verify(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] != "BOTH_APPLIED": + raise CoordinatorError("both owning resets must be applied before verification") + config = self._config(receipt) + try: + for product in PRODUCTS: + envelope = self._invoke(config, receipt, product, "verify") + self.fault_hook(f"after-{product}-verify") + self._save_observation(receipt, product, "verify", envelope) + if self._product_stage(product, envelope["state"]) != "verified": + raise ProductCommandError(product, "verify", "VERIFIED_STATE_REQUIRED") + receipt["milestones"][ + "forge_verified" if product == "forge" else "ep_verified" + ] = True + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + _event(receipt, "BOTH_VERIFIED", "both owning reset results verified") + self.store.save(receipt) + return self._readback(receipt) + + def authorize_resume(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] != "BOTH_VERIFIED": + raise CoordinatorError("normal writers cannot resume before both verifications") + receipt["milestones"]["resume_authorized"] = True + _event(receipt, "RESUME_AUTHORIZED", "joint verification authorizes owning finishes") + self.store.save(receipt) + return self._readback(receipt) + + def finish(self, product: str) -> dict[str, object]: + if product not in PRODUCTS: + raise CoordinatorError("owning product is invalid") + with self.store.locked(): + receipt = self._required_receipt() + if receipt["state"] != "RESUME_AUTHORIZED": + raise CoordinatorError("finish is forbidden until both reset results are verified") + config = self._config(receipt) + try: + # Re-prove both owning results immediately before each + # sequential finish. This is the two-phase readiness boundary; + # a failed peer check cannot be followed by the requested finish. + for candidate in PRODUCTS: + readiness = self._invoke(config, receipt, candidate, "verify") + if self._product_stage(candidate, readiness["state"]) not in { + "verified", "finished", + }: + raise ProductCommandError( + candidate, "verify", "FINISH_READINESS_REQUIRED", + ) + self._save_observation( + receipt, candidate, "pre-finish readiness", readiness, + ) + if not receipt["products"][product]["finished"]: + envelope = self._invoke(config, receipt, product, "finish") + self.fault_hook(f"after-{product}-finish") + self._save_observation(receipt, product, "finish", envelope) + if not receipt["products"][product]["finished"]: + raise ProductCommandError(product, "finish", "COMPLETED_STATE_REQUIRED") + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + milestone = "forge_finished" if product == "forge" else "ep_finished" + receipt["milestones"][milestone] = True + both = all(receipt["milestones"][name] for name in ("forge_finished", "ep_finished")) + _event( + receipt, "COMPLETE" if both else "RESUME_AUTHORIZED", + f"{product} owning maintenance finished", product=product, + ) + self.store.save(receipt) + return self._readback(receipt) + + def resume(self, product: str) -> dict[str, object]: + if product not in PRODUCTS: + raise CoordinatorError("owning product is invalid") + with self.store.locked(): + receipt = self._required_receipt() + milestone_prefix = "forge" if product == "forge" else "ep" + milestones = receipt["milestones"] + has_revalidation = any( + event.get("state") == "PLANS_REVALIDATED" + for event in receipt["events"] if isinstance(event, Mapping) + ) + if not milestones["plans_revalidated"] or not has_revalidation: + raise CoordinatorError( + "owning resume is forbidden before joint plan revalidation", + ) + if not ( + milestones[f"{milestone_prefix}_apply_started"] + or milestones[f"{milestone_prefix}_applied"] + ): + raise CoordinatorError( + "owning resume is forbidden before durable product apply admission", + ) + config = self._config(receipt) + try: + envelope = self._invoke(config, receipt, product, "resume") + self.fault_hook(f"after-{product}-resume") + self._save_observation(receipt, product, "resume", envelope) + return self._reconcile_locked(receipt, config) + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + + def reconcile(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + return self._reconcile_locked(receipt, self._config(receipt)) + + def _reconcile_locked( + self, receipt: dict[str, object], config: CoordinationConfig, + ) -> dict[str, object]: + try: + for product in PRODUCTS: + envelope = self._invoke(config, receipt, product, "status") + self._record_product(receipt, product, envelope) + except ProductCommandError as error: + self._mark_failure(receipt, error) + raise + forge = self._product_stage("forge", receipt["products"]["forge"]["state"]) + ep = self._product_stage( + "engineering-platform", receipt["products"]["engineering-platform"]["state"], + ) + milestones = receipt["milestones"] + if forge == ep == "finished": + milestones["forge_finished"] = milestones["ep_finished"] = True + state = "COMPLETE" + elif forge in {"finished", "verified"} and ep in {"finished", "verified"}: + milestones["forge_verified"] = milestones["ep_verified"] = True + state = "RESUME_AUTHORIZED" if milestones["resume_authorized"] else "BOTH_VERIFIED" + elif forge in {"applied", "verified"} and ep in {"applied", "verified"}: + milestones["forge_applied"] = milestones["ep_applied"] = True + state = "BOTH_APPLIED" + elif forge in {"applied", "verified"} and ep == "backup": + milestones["forge_applied"] = True + state = "FORGE_APPLIED" + elif ep in {"applied", "verified"} and forge == "backup": + milestones["ep_applied"] = True + state = "EP_APPLIED" + elif forge == ep == "backup": + milestones["maintenance"] = milestones["backups_verified"] = True + state = "PLANS_REVALIDATED" if milestones["plans_revalidated"] else "BACKUPS_VERIFIED" + elif forge in {"maintenance", "backup"} and ep in {"maintenance", "backup"}: + milestones["maintenance"] = True + state = "BOTH_MAINTENANCE" + else: + state = "RECONCILIATION_REQUIRED" + _event(receipt, state, "owning status reconciled", evidence={"forge": forge, "ep": ep}) + self.store.save(receipt) + return self._readback(receipt) + + def status(self) -> dict[str, object]: + with self.store.locked(): + receipt = self._required_receipt() + config = self._config(receipt) + owning = { + product: self._invoke(config, receipt, product, "status") for product in PRODUCTS + } + return {**self._readback(receipt), "owning_readback": owning} + + def _required_receipt(self) -> dict[str, object]: + receipt = self.store.load() + if receipt is None: + raise CoordinatorError("coordinator receipt does not exist; run preview first") + return receipt + + def _readback(self, receipt: Mapping[str, object]) -> dict[str, object]: + return { + "contract_version": CONTRACT_VERSION, + "coordinator_id": receipt["coordinator_id"], "state": receipt["state"], + "receipt": str(self.store.receipt_path), "receipt_digest": _digest(receipt), + "configuration_digest": receipt["configuration_digest"], + "products": receipt["products"], "milestones": receipt["milestones"], + } + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="operational-reset-coordinator", + description="Coordinate only installed Forge and EP owning reset CLIs.", + ) + parser.add_argument("--receipt-root", type=Path, required=True) + parser.add_argument("--coordinator-id", required=True) + parser.add_argument("--command-timeout-seconds", type=float, default=1800) + commands = parser.add_subparsers(dest="command", required=True) + preview = commands.add_parser("preview") + preview.add_argument("--forge-cli", type=Path, required=True) + preview.add_argument("--forge-data-root", type=Path, required=True) + preview.add_argument("--forge-operation-id", required=True) + preview.add_argument("--ep-cli", type=Path, required=True) + preview.add_argument("--ep-data-root", type=Path, required=True) + preview.add_argument("--ep-operation-id", required=True) + preview.add_argument("--ep-backup-root", type=Path, required=True) + preview.add_argument("--forge-fk-acknowledgement", action="append", default=[]) + preview.add_argument("--ep-fk-acknowledgement", action="append", default=[]) + for name in ("prepare", "revalidate", "verify", "authorize-resume", "reconcile", "status"): + commands.add_parser(name) + for name in ("apply", "resume", "finish"): + product_command = commands.add_parser(name) + product_command.add_argument("--product", choices=PRODUCTS, required=True) + return parser + + +def main(argv: list[str] | None = None) -> int: + parser = _parser() + args = parser.parse_args(argv) + try: + store = ReceiptStore(args.receipt_root, args.coordinator_id) + coordinator = OperationalResetCoordinator( + store, command_timeout_seconds=args.command_timeout_seconds, + ) + if args.command == "preview": + config = CoordinationConfig.create( + coordinator_id=args.coordinator_id, + forge_cli=args.forge_cli, forge_data_root=args.forge_data_root, + forge_operation_id=args.forge_operation_id, + ep_cli=args.ep_cli, ep_data_root=args.ep_data_root, + ep_operation_id=args.ep_operation_id, ep_backup_root=args.ep_backup_root, + forge_fk_acknowledgements=args.forge_fk_acknowledgement, + ep_fk_acknowledgements=args.ep_fk_acknowledgement, + ) + result = coordinator.preview(config) + elif args.command in {"apply", "resume", "finish"}: + result = getattr(coordinator, args.command)(args.product) + else: + result = getattr(coordinator, args.command.replace("-", "_"))() + print(json.dumps(result, sort_keys=True)) + return 0 + except (CoordinatorError, OSError) as error: + print(json.dumps({ + "contract_version": CONTRACT_VERSION, "coordinator_id": args.coordinator_id, + "state": "ERROR", "allowed": False, "error": str(error), + }, sort_keys=True)) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main())