Skip to content

fix: complete V1.1 clean-room reliability review - #31

Merged
magetsu002 merged 77 commits into
mainfrom
fix/v1-1-clean-room-reliability
Aug 1, 2026
Merged

fix: complete V1.1 clean-room reliability review#31
magetsu002 merged 77 commits into
mainfrom
fix/v1-1-clean-room-reliability

Conversation

@magetsu002

@magetsu002 magetsu002 commented Aug 1, 2026

Copy link
Copy Markdown
Owner

V1.1 reliability review

Starting main SHA: 22ecacc25021d06faa052d92ad3c7e9abe37acb5

Final exact certified head: 545949e90149cc693732340c16d4eeb35a8ede6d

Reproduced real failures and root causes

  • Installed command packaging: clean TypeScript builds recreated npm bin targets without executable permissions, and the MCP bin target did not point at a stable executable entry.
  • Error containment: embedded Python and subprocess failures could expose traceback text, temporary staging paths, or generic errors without a safe failed-stage/recovery description.
  • Canonical state: a partial installed generation without a transaction marker could be detected by setup while repair returned no action; doctor, boundary verification, and deployment did not consistently prioritize the same installation-health state.
  • Security integration: replacing the installed dispatcher invalidates canonical receipt evidence. The old Ubuntu integration expected an ordinary setup synchronization instead of the required evidence-preserving repair flow.
  • CodeQL strict gate: three js/file-system-race findings came from checking a path with lstat and reopening it later by pathname in bin preparation, package inspection, and a clean-room regression test.
  • Deployment plan drift: observedFingerprint() included exact availableDiskBytes. Normal free-space drift between plan and apply falsely marked an otherwise unchanged exact plan stale.
  • Clean-room deployment fixture: the disposable Ubuntu image did not expose the reviewed /usr/bin/node and /usr/bin/npm paths, initially provisioned root-owned synthetic release paths, and generated an invalid escaped npm build command.
  • Fixed command environment: reviewed npm commands ran with no fixed home, making npm cache/config behavior unreliable under the stripped environment.
  • Controlled privilege boundary: the controlled forced-command wrapper set Linux no-new-privs, which made the exact allowlisted sudo service restart impossible. This was a product-level contradiction between confinement and controlled deployment.

Repairs

Packaging

  • Added stable opshaven and opshaven-mcp entry modules.
  • Build now verifies regular non-symlink bin targets through stable file handles, validates shebangs, and applies executable mode during every clean build.
  • Added installed-command, rebuild-after-link, clean, and reinstall checks; no manual chmod or shell hash reset is required.

Canonical installation health and recovery

  • Added one canonical installation-health evaluator used by setup, repair, doctor, boundary verification, deployment plan, and deployment apply.
  • Added explicit classifications for fresh, healthy, legacy, partial, invalid-receipt, interrupted-transaction, staged-generation, and unknown-footprint states.
  • Added repair-first doctor output and blocked boundary/deployment behavior for damaged state.
  • Added whole-installation evidence-preserving reinstall with bounded evidence capture, copied-evidence verification, reviewed artifact installation, canonical receipt generation, post-repair doctor/boundary checks, and audit recording.
  • Documented state sources, precedence, migration results, repair actions, and manual-recovery cases.

Error boundary

  • Sanitized embedded Python and subprocess diagnostics in normal output.
  • Added safe stage, cause, mutation, rollback, known-state, and next-command presentation.
  • Deployment reviewed commands now preserve a bounded stage label without exposing executable arguments, command payloads, output, or private paths.

CodeQL

  • Replaced path-check-then-open sequences with O_NOFOLLOW file-handle inspection, handle-based reads, stats, and chmod.
  • Added stable-handle regression coverage.
  • No query, rule, file, or strict post-analysis check was suppressed or excluded.

Deployment identity and volatile readiness

Digest-bound stale-plan identity includes the current revision, active release and rollback target, source revision/cleanliness, service identity and state, runtime readiness, target-revision verification, application binding, pinned host identity, policy version, authorization profile, operation definitions, release layout, and health-check definition.

The plan still records observed free disk for human review and audit. Exact free-byte counts are audit-only volatile observations and are excluded from the observed-state fingerprint. Apply re-inspects current disk and requires availableDiskBytes >= DEPLOYMENT_MINIMUM_DISK_BYTES before any mutation.

Regression coverage proves:

  • 10 GiB at plan and 9.9 GiB at apply preserves the exact-plan identity and succeeds;
  • below-minimum disk fails before mutation;
  • current revision, active release, service/runtime readiness, policy, authorization, application binding, rollback target, and health-check-definition drift remain blocked;
  • identical stable state and constraints produce the same deterministic plan identity.

Controlled and read-only confinement

  • Read-only dispatch retains no-new-privs, dropped inherited/ambient capabilities, reset environment, forced SSH command, and no sudo authority.
  • Controlled dispatch retains forced SSH command, signed exact-operation approval, capability/resource allowlists, dropped inherited/ambient capabilities, and a reset fixed environment, but does not set no-new-privs because that kernel flag makes an approved sudo transition impossible.
  • The restricted account remains outside privileged groups. Host policy must allow only exact reviewed service restart commands; arbitrary sudo or shell execution is not introduced.
  • The disposable Ubuntu fixture verifies the restricted account, reviewed npm paths, writable synthetic release roots, exact empty command environment, and exact service restart rule before deployment.

Clean-room lifecycle

The final installed-CLI lifecycle completed:

  • clean dependency installation and local linking;
  • opshaven --version and MCP executable checks;
  • initialization and application registration;
  • remote setup and canonical generation installation;
  • doctor and boundary certification;
  • exact deterministic plan and interactive exact-plan apply;
  • post-deployment doctor health;
  • partial-generation diagnosis;
  • evidence-preserving repair and post-repair verification;
  • clean rebuild with both installed commands still executable.

Final full lifecycle time: 89 seconds on Ubuntu 24.04 / Node 22.

Measured major stages from the final run:

  • disposable host startup: about 30 seconds;
  • remote generation setup: about 10 seconds;
  • initial doctor/boundary verification: about 8 seconds;
  • exact plan creation: about 2 seconds;
  • exact apply and verification: about 5 seconds;
  • evidence-preserving repair: about 19 seconds;
  • final clean rebuild: about 1 second.

Validation

The execution environment available to this review could not clone GitHub directly, so no unsupported claim of a complete local clone-based run is made. Product changes were committed forward and validated on fresh GitHub-hosted runners.

Final unchanged-head runs:

  • CI 30718880268 — success, including release check and reproducible build.
  • Security 30718880266 — success, including npm audit, repository scan, lockfile/workflow verification, synchronization, rollback, receipt tampering, cancellation, progress modes, disposable VPS, remote MCP, Ubuntu setup, trust, deployment, and onboarding integrations.
  • CodeQL 30718880267 — success, including strict zero-finding post-analysis gate.
  • Clean Room 30718880274 — success, including installed CLI, canonical state matrix, and full operator lifecycle.

Security invariants preserved

Restricted SSH, pinned host identity, signed authorization, signed requests and responses, least privilege, capability-scoped dispatch, deterministic exact plans, exact-plan apply, replay protection, bounded execution, redaction, transactional synchronization, generation receipt integrity, rollback, evidence-preserving repair, tamper-evident audit, and fail-closed behavior remain enforced.

No arbitrary shell execution, broad sudo access, scan exclusion, CodeQL suppression, weakened audit level, skipped integration, release publication, or tag change is included.

Known limitations

  • Controlled service restart depends on a host-owned exact sudoers rule matching the configured service identity; incorrect host policy fails closed.
  • Distribution-specific sudo, systemd, filesystem, and confinement behavior still requires validation on the target operating system.
  • This PR does not publish a release or modify v1.0.0.

OpsHaven Reliability Bot and others added 28 commits August 1, 2026 20:39
@magetsu002
magetsu002 marked this pull request as ready for review August 1, 2026 21:22
@magetsu002
magetsu002 merged commit 1a66dfd into main Aug 1, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant