Skip to content

on-prem: install, update and roll back the real engine offline in CI - #1918

Open
abrichr wants to merge 4 commits into
agent/142-cfrom
agent/142-c2
Open

abrichr wants to merge 4 commits into
agent/142-cfrom
agent/142-c2

Conversation

@abrichr

@abrichr abrichr commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Part of the on-prem deployment work. Stacked on #1915 (base agent/142-c), which adds bin/run-egress-blocked.sh.

What this does

The on-prem kit already had a signed offline update and rollback, but its only test (bin/test-update.sh) fakes the venv build, and no CI job ran it. Nothing proved that a release archive actually holds everything the engine needs, or that the real engine installs and upgrades on a machine with no network.

  • deploy/on-prem/bin/build-release.sh builds a release archive on a connected host from packages/flow/uv.lock: it downloads exactly the pinned wheels for openadapt-flow[privacy], checks each against the lock's hashes (pip download --require-hashes), adds the openadapt-flow wheel, and writes the archive, a .sha256, and an openssl .sig when given the key. UPDATE.md's hand recipe resolves the newest matching dependencies instead; it now points at the script first.
  • deploy/on-prem/bin/test-offline-install.sh stage DIR builds two signed releases (this checkout as V1, the same source re-versioned as V1.post1 as V2) and a throwaway key, and deletes the private key afterwards.
  • test-offline-install.sh verify DIR runs as root inside run-egress-blocked.sh and refuses to run outside it. It checks the machine has no route out, then: first install from V1's wheelhouse with install.sh --wheelhouse (pip has --no-index and no network, so any wheel missing from the archive fails it); a tampered V2 archive is refused and leaves V1 current; a signed update to V2 (current and previous pointers, engine and privacy extra import at the new version, bundle and queued job unchanged); rollback to V1; then verify-airgap.sh --probe --audit (the egress probe fails to reach the internet, and the audit hash chain is intact).
  • New job flow-onprem-offline in packages.yml runs bin/test-update.sh, then stage, then verify. It is wired into packages-gate for flow changes, and the workflows README lists it.

Testing

  • shellcheck, actionlint and bash -n are clean.
  • bin/test-update.sh passes locally on macOS: 45 passed, 0 failed.
  • bin/build-release.sh locally on macOS: builds an unsigned release with 89 wheels from the lock.
  • CI contract tests locally: test_ci_required_check, test_flow_shards, test_ci_locked_installs and test_nightly (46 passed), plus packages/flow test_ci_workflow_contract and test_onprem_service (43 passed). tests/test_release_artifacts.py has 3 failures locally because the openadapt launcher distribution isn't installed in the venv I used (PackageNotFoundError). That is a gap in my environment; this branch doesn't touch those files.
  • First CI run of the offline job: 20 passed, 1 failed. Install from the locked wheelhouse with egress blocked, refusing a tampered update, the signed update, rollback, and the air-gap probe and audit chain all passed. The failure was the version check after the update: it reported 2.6.0, not 2.6.0.post1. The check ran python -c from packages/flow, so it imported the source checkout instead of the installed engine. Fixed in f5eb3c3: the check now runs isolated from /, requires the module to live inside the release venv, and compares the module version, the installed distribution version and openadapt-flow --version.
  • On the run for f5eb3c3, the offline job failed one check in the existing test-update.sh: "concurrent audit writers preserve one hash chain" (44 passed, 1 failed). It failed the same way on a rerun. This branch doesn't touch audit-log.sh; on-prem locks: hold on Ubuntu 26.04 (uutils mkdir) and never break a live writer's lock #1922 fixes it, and its two commits are cherry-picked here as normal commits (efbb57c, 7770839), so they become no-ops once on-prem locks: hold on Ubuntu 26.04 (uutils mkdir) and never break a live writer's lock #1922 merges. The first fix (a stale-lock race) gave one green run, but the check failed again on the stacked branch. The actual cause was the runner image: ubuntu-26.04 ships uutils mkdir, which can report success to two processes racing to create one directory, so the mkdir-based audit and update locks did not exclude. The second commit takes ownership with an exclusive file create instead. On the run for 7770839 (run 41940061341) the offline job passed, and on the stacked on-prem: build the runner image from a signed release with no network, test it in CI #1923 test-update.sh passed 51/0 on two consecutive runs.

Not tested

  • The verify step needs Linux network namespaces and root, so CI is its first real run. On macOS I couldn't run it beyond a syntax check.
  • The container image path (the Dockerfile wheelhouse is still a stub) is not covered here.

bin/build-release.sh builds a release archive from uv.lock (pinned wheels,
hash-checked) plus the flow wheel, a checksum and an openssl signature.
bin/test-offline-install.sh builds two signed releases, then as root inside
run-egress-blocked.sh installs the first from its wheelhouse, refuses a
tampered second, updates to the signed second, rolls back and runs the
air-gap attestation. The new flow-onprem-offline job runs it, and runs
bin/test-update.sh, which no job ran before.
@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
…kout

The version check ran python -c from packages/flow, so it imported the
source checkout and reported 2.6.0 after the update to 2.6.0.post1. Run it
isolated from /, require the module to live in the venv, and compare the
module with the installed distribution and the CLI's --version.
A writer that found the audit lock owned by an exited process removed it
without re-checking. If the owner released between the read and the
removal and another writer took the lock, the breaker deleted the live
lock and two writers could chain to the same parent. Break stale locks
only under a second lock and only if the lock still names the same dead
owner. Add a deterministic test that forces the handover, and print the
verifier output when the concurrent-writer check fails.
…r(1)

Ubuntu 26.04 ships uutils mkdir, which reports success to both callers when
two processes race to create one directory. The audit-log writer lock and the
release update lock both treated a zero exit from mkdir as ownership, so on
that image concurrent audit writers chained to the same parent (the
"concurrent audit writers preserve one hash chain" CI failure) and a second
update could proceed beside a live one.

Ownership is now the shell's own noclobber create of the pid file inside the
lock directory (O_CREAT|O_EXCL), and the stale-lock breaker lock is an
exclusive file too. Tests model the uutils behavior with a mkdir shim that
never reports EEXIST; on the previous code they fail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-needed Needs an independent review from another vendor's agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant