Shared Rust Machine installer and bounded CLI/Cloud upgrades #822

Problem Statement

Ployz operators need one dependable way to install Machine software and request explicit upgrades from the CLI or Cloud. Setup currently combines a shared shell Machine installer with Rust provisioning and enrollment code. It does not yet provide the bounded, observable upgrade attempt needed when restarting the daemon interrupts the management connection.

The maintenance goal is one owner for Machine installation behavior. The existing setup paths already share an installer: this work is not justified as eliminating several independent installers or guaranteeing fewer lines of code. Its value is locality, removal of embedded-script transport glue, and reuse of installation machinery for upgrades.

Solution

Provide a Rust Machine installation module shipped in ployzd, with ployzd install as its local command interface. SSH provisioning, local setup, Cloud enrollment and explicit upgrades reuse that implementation. Cloud-init invokes the existing setup/enrollment commands rather than implementing another installer.

An upgrade is a bounded Machine-local attempt: stage and verify the requested release, activate it, restart affected Ployz services, and report observed readiness and version. Run it independently under systemd so daemon restart and client disconnection do not terminate it. Existing applications must continue serving while management may briefly disconnect.

There is no automatic rollback or rollback guarantee. Preserve the previous binary for explicit operator recovery, but do not claim it can interpret state touched by another release. Failed activation may require local or SSH repair.

User Stories

1. As an operator, I want to install Ployz locally with one command, so that I do not assemble host setup steps manually.
2. As an operator, I want SSH setup to invoke the same installation implementation, so that remote and local setup behave consistently.
3. As a Cloud user, I want the enrollment command to prepare the required daemon, so that onboarding retains its existing convenience.
4. As an operator using cloud-init, I want to invoke the same setup/enrollment flow, so that first-boot provisioning needs no separate Ployz installer.
5. As an operator who provisions hosts externally, I want to skip installation where supported, so that Ployz respects my host-management process.
6. As an operator, I want initial setup to prepare explicitly requested prerequisites and storage, so that the Machine can support my selected configuration.
7. As an operator, I want ordinary Ployz upgrades to leave Docker, OS packages and ZFS upgrades to explicit maintenance, so that upgrading Ployz does not silently become host maintenance.
8. As a CLI user, I want to request an exact published daemon version, so that I control what runs on my Machines.
9. As a Cloud user, I want to request an upgrade through the existing authenticated Machine connection and view its outcome, so that routine upgrades require no SSH login.
10. As an operator, I want channel selection to resolve to one exact release per attempt, so that a moving channel cannot change the target midway.
11. As an operator, I want to select any available target version without forced intermediate upgrades, so that version selection remains under my control.
12. As an operator, I want download and verification failures to leave the running installation intact, so that acquisition failures do not interrupt management.
13. As an operator, I want overlapping installers and upgrades during active Machine mutations to be refused, so that installation does not race with other changes.
14. As an application owner, I want existing workloads to continue serving during daemon upgrades, so that management maintenance does not become an application outage.
15. As an operator, I want an accepted upgrade to survive client disconnection and daemon restart, so that losing the request connection does not terminate installation.
16. As an operator, I want to reconnect and inspect the same attempt, so that uncertainty does not force me to launch a duplicate upgrade.
17. As an operator, I want an upgrade to preserve Machine identity, pairing, configuration and data, so that software replacement does not implicitly reset or re-enroll the Machine.
18. As an operator, I want success to identify the verified running version, so that launching an installer is not mistaken for completing an upgrade.
19. As an operator, I want failures and interruptions to name the observed stage and provide diagnostic access, so that I can repair the Machine without guessed causes.
20. As an operator, I want the previous binary retained for manual repair, while being told that downgrades and data rollback are not guaranteed.
21. As a maintainer, I want installation behavior and its checks concentrated behind one interface, so that changes do not spread across SSH, enrollment and Cloud implementations.
22. As a maintainer, I want the old Machine shell implementation removed after migration, so that there is only one installation implementation to maintain.

Implementation Decisions

Module and seam

• One Machine installation module owns installation decisions, release acquisition and verification, host inspection/preparation, staging, activation, systemd integration, and outcome recording. Start within ployzd; do not introduce a separate installer crate or generic provisioning framework without a concrete need.
• The external interface requests installation with explicit setup choices and exposes an attempt's observed outcome. Callers do not supply or orchestrate individual installation steps.
• Internally, prefer ordinary functions and existing dependencies to a public Step trait, plugin registry, or distribution-provider abstraction. Continue to invoke package managers and systemd for their native responsibilities.
• Keep prerequisite preparation explicit and distinct from ordinary software replacement even though both belong to the installation module. An upgrade that cannot proceed with current prerequisites reports an actionable failure instead of upgrading unrelated host software.
• Preserve applicable existing setup capabilities and supported platform behavior. Migrating languages is not authorization to drop storage preparation, installation-only behavior used by tooling, or systemd hardening.

Setup and enrollment adapters

• Expose installation through ployzd install; use system SSH for remote setup. Remove embedded Machine shell scripts, base64 script transport, and their obsolete command-construction code when callers migrate.
• Retain the minimal shell bootstrap needed to acquire and verify the first executable on a fresh host. Do not require Rust tooling on the target Machine. Small unavoidable download logic in this bootstrap is acceptable.
• Keep the operator CLI bootstrap distinct from Machine installation where platform support requires it; Machines do not acquire a permanent CLI dependency merely for daemon upgrades.
• Enrollment remains responsible for Cloud identity exchange, pairing, founding/joining, and its existing completion evidence. It invokes installation as needed; the installer never performs enrollment or resets Machine identity.
• Preserve existing enrollment behavior that installs a missing local daemon, synchronizes the required version, reconnects, and verifies the observed daemon version. Preserve external provisioning paths and their documented limitations.
• Use the existing authenticated Machine transport for Cloud/CLI upgrades. Do not create an additional update server or Cloud SSH credential store.

Release selection and activation

• Accept any available published target version; no forced sequential version stepping or compatibility-based source/target allowlist. Availability, platform, artifact integrity and executability still require validation. Accepting a version does not guarantee that it can run against existing state.
• Resolve a requested channel once into an exact release and carry that release through the attempt and outcome.
• Validate version input and construct downloads from the trusted release source rather than exposing arbitrary privileged command or download-URL execution through the upgrade interface.
• Download to private staging, verify the published checksum, and perform meaningful bounded preflight before changing the running installation. Checksum verification protects integrity under the existing trusted release-distribution model; it is not independent proof against compromise of that source.
• Use staged files and same-filesystem atomic replacement, with appropriate file/directory durability. Do not overwrite a running executable in place or claim the entire multi-file installation is transactional.
• Preserve the previous binary for manual recovery. Keep retention bounded; no historical-release archive manager is required.
• Preserve Machine identity, pairing, operator configuration and application data. Do not run resets, storage reconfiguration or Docker/OS/ZFS upgrades as incidental parts of ordinary daemon replacement.
• Account for all affected Ployz processes, including the volume plugin sharing the daemon executable. Preserve ongoing application serving and volume availability; a naive restart of every dependency is not acceptable.

Execution and admission

• Run accepted upgrade installation in a separate bounded systemd service, independent of the daemon's process group and service lifecycle. The executable used by that worker must remain available throughout activation. No permanent keeper is introduced.
• Use a Machine-local installation lock across local, SSH and remote request paths. Admission must also prevent upgrades racing with active Machine mutations, with a clear busy response rather than queued draining.
• Refuse upgrades while relevant local mutations are active and reject new mutations during activation. This is a Machine-local exclusion rule, not a Cluster-wide lock, Deploy lease, or consensus mechanism. Existing workloads continue running.
• A Deploy spanning Machines can still return partial results if a later target becomes unavailable or refuses work. Do not promise survival of an in-flight Deploy or Cluster-wide upgrade atomicity.
• Explicit multi-Machine upgrades, where exposed, are client-driven bounded sequences with per-Machine outcomes, stopping on failure or uncertainty. Do not persist desired fleet versions or continue a fleet rollout as a reconciler.

Observation and failure contract

• Keep a small Machine-local attempt record containing its identity, exact target, observed stage and terminal outcome. Use journald for detailed diagnostics rather than building a separate log service.
• Request retry/reconnection must be able to find the accepted attempt rather than silently start duplicate work. Do not add a generic durable workflow engine or unbounded attempt history.
• Distinguish accepted/running from succeeded. Success requires verifying the requested running daemon version and local readiness. A created process, an accepted socket, or a successful service launch alone is insufficient.
• Cloud reachability is separately observed: a Cloud outage does not prove local activation failed. If management is unreachable, report the known evidence and uncertainty rather than inventing a terminal result.
• Failed attempts preserve the failed stage and underlying error. If the worker is no longer running without a recorded terminal result, report interruption rather than leaving the attempt indefinitely running.
• Power loss does not trigger automatic downgrade, installation resumption, or boot-time rollback. Boot follows the selected installation and normal systemd behavior; incomplete attempts remain inspectable for explicit repair/retry.
• No automatic fallback, persisted-state compatibility guarantee, migration framework, or old-new-old qualification requirement is introduced. Local/SSH repair remains available when the daemon cannot start. This supersedes the earlier conversational exploration of automatic rollback.

Testing Decisions

• Test observable installation outcomes through the installation interface. Keep setup/enrollment adapter checks focused on their own behavior; do not replicate installer implementation tests in each caller.
• Follow the repository's testing rungs and product-path evidence ledger. Use the lowest honest rung that can fail for the behavior; name the rung and test when implementing changes.
• Rung 1: replace existing Machine installer shell contracts with equivalent checks through the new implementation, retaining the separate CLI bootstrap checks. Cover same-target behavior, prerequisite scope, invalid/missing/corrupt artifacts, staging failure without replacing the running installation, attempt exclusion and clear failure outcomes.
• Reuse the existing enrollment daemon-synchronization tests as prior art for matching versions, installation failure before enrollment, reconnection and stale-version rejection. Verify all setup adapters use the shared behavior without preparing storage during a mere version synchronization.
• Use CLI-shape checks where command parsing and public error/exit behavior are the behavior under test.
• Real systemd verification is required for installer survival across daemon restart and client disconnection. Mock systemctl calls cannot prove that property. If an informing test is added, include its binary in the existing informing-test runner.
• Extend Authority qualification against real musl release artifacts on a real Machine: install the source release, initialize and deploy a small app with persistent data, request an upgrade, sever the management connection, reconnect and inspect the attempt, and verify target version, unchanged Machine identity, retained pairing/data and continued application traffic. Include relevant volume-plugin behavior for a storage-enabled Machine.
• Exercise a failed-activation case to prove honest reporting and diagnostic access without automatic rollback. Distinguish this from preflight rejection. Do not add a downgrade compatibility matrix or universal persisted-state fixture framework.
• Retain existing prerequisite/storage contract coverage where the behavior is ported. Replace obsolete tests when their implementation disappears rather than maintaining parallel shell/Rust suites forever.

Out of Scope

• Automatic rollback/fallback, boot-time rollback, data restoration, and downgrade safety guarantees.
• General migrations, automatic previous-release compatibility enforcement, historical compatibility matrices, and forced intermediate-version upgrades.
• Permanent keeper/recovery daemons, periodic update polling, desired-version reconciliation, and durable fleet workflow orchestration.
• DEB/RPM repositories, rootless-runtime adoption, daemon containerization, WASM, and custom connection handoff.
• Automatic Docker, OS package or ZFS upgrades during ordinary Ployz upgrades.
• A second management transport, generic Step/provider frameworks, package-manager abstractions, and a generic installer SDK.
• Changes to enrollment authority or Cluster semantics; implicit reset/re-enrollment; general rollback or atomicity across Machines.
• New ADRs or broad agent-skill configuration changes.

Further Notes

This is a maintenance-constrained design. Shipping Rust is not itself a simplification: most host setup logic moves rather than vanishes. The implementation succeeds when existing setup and new upgrades share one deep module, callers shed installation details, native system primitives handle lifecycle, and obsolete shell glue is deleted.

The independent installer is a bounded Machine-local operation whose lifetime spans a daemon restart; it is not a permanent background reconciler. Its acceptance and final outcome are separate observations.

The rollout must preserve existing behavior while honestly distinguishing preflight success, local daemon readiness, application availability and restored Cloud connectivity. None of those observations implies the others without evidence.