Skip to content

deploy: a check that only checks, every 4h, and a header that says what it found (#438) - #1364

Merged
dp-web4 merged 1 commit into
mainfrom
tristan/deploy-check-only
Oct 22, 2026
Merged

dp-web4 merged 1 commit into
mainfrom
tristan/deploy-check-only

Conversation

@lewisfab

Copy link
Copy Markdown
Collaborator

Why

The operator on tl-dell (2026-10-21) found that the dashboard's two deployment words mean something other than what they say:

What does "deployment current" actually mean? To me, that means there are no updates available. Check for update means ONLY to check.

They were right on both counts:

What the operator asked for:

  1. Make the status say what it means, including when it last checked.
  2. Check automatically every 4 hours, but install nothing unless auto-update is on.
  3. "check for updates" should only check, and should turn into "update" when there's an update.

What

Deploy script (deploy/from-main/hestia-deploy.sh)

  • Every cycle that syncs records its verdict in $HESTIA_HOME/deploy-check.tsv, as <state> <checked_at> <running> <latest>. The states are checking, up-to-date, available and failed, and the file is written atomically.
    • A full cycle writes the verdict too, and rewrites it as up-to-date after DEPLOYED.
    • A write failure logs a warning and never stops a deploy.
  • --check no longer self-updates the script, and a hold no longer skips it. A hold protects builds and restarts, and a check does neither.
  • A check that dies before reaching a verdict (fetch failed, no checkout) records failed.
  • A full cycle installs the check units if the box has the deploy service but not the check units. That's how existing boxes get the check. The check timer is enabled only on first install, so an operator who turns it off later isn't overridden.

New units

  • hestia-deploy-check.service: hestia-deploy --check, with SuccessExitStatus=3 because "update available" is a result, not a failure.
  • hestia-deploy-check.timer: 02/4:17, two hours off the deploy timer's marks so the two never contend for the lock.
  • deploy/fleet/install.sh installs both and enables the check timer on first install. The deploy timer stays off by default, as before.

Daemon

Dashboard header

header button
● up to date · checked <time> check for updates (checks only)
! update available · checked <time> update
● checking for updates… (disabled)
! update check failed · <time> retry check
● not checked yet / ● not checked since this build check for updates
! install incomplete (the old "stale") update
● update status unknown (no check unit: macOS, or a box before its next update) update

The tooltip lists the running build, main's build at the last check, the check schedule and the last deploy run. The words "deployment current" are gone.

Rolling out to an existing box

The first update after this merges installs the new script but runs the old one, which has already been read into memory, so it doesn't install the check units. The new daemon then shows ● update status unknown with an update button. One more update installs and enables the check timer. After that, the header is driven by the check. Alternatively, re-run deploy/fleet/install.sh.

Tests

  • New deploy/from-main/check_mode_test.py runs the real script in --check against a throwaway HOME, local bare origins and a stub daemon, with no network and no real unit.
    • It checks the up-to-date and available verdicts, no self-update, a hold not stopping a check, and an unsyncable check recording failed.
    • Run against main's script, 8 of its 12 checks fail. That confirms the old --check self-updated, recorded nothing, and was stopped by a hold.
  • deploy/fleet/deploy_supervisor_step_test.py now also checks that the check units are installed, that the check timer is enabled on first install and left alone on a re-run, and that the deploy timer still isn't enabled.
  • New Rust tests:
    • deploy_check_reports_the_supervisors_verdict_against_the_running_build
    • deployment_check_starts_the_check_unit_and_never_the_deploy
    • deployment_check_in_flight_only_while_requested_or_checking
  • Existing checks, all passing: tools/dashboard_deployment_update_contract_test.py (now with 7 check assertions), operator_surfaces_spec_test.py, operator_surfaces_ratchet_test.py, no_path_literals_test.py (the new units registered, the check service at 1 literal), dashboard_no_undef.mjs and the other dashboard contract tests.
  • cargo test --lib (pinned web4 31990db7): 1256 passed, 0 failed, 2 ignored. The three new tests were also run by name and pass.
  • Not tested:
    • the endpoint and header against a live daemon, because I didn't want to replace this box's governing daemon with an unreviewed build;
    • the check units under real systemd;
    • macOS, where the dashboard falls back to the "update" button with an honest "update status unknown".

Review gate

surface: operator_deployment_check   act: start the check-only supervisor run (sync deploy checkout, write deploy-check.tsv)
S: low/reversible [construct: hestia-deploy --check: no build, no install, no restart, no self-update; deploy-check.tsv is overwritten by the next check]
R: pass [construct: operator_gate route_layer on operator_surface]   W: pass [construct: current_operator_context + attach_operator_provenance]
O: pass [construct: deployment_check_requested append_chain before write_deployment_update_file / deployment_trigger_output]   A: pass [construct: deployment_check_requested + deployment_check_triggered|_trigger_failed, with `witnessed` in the reply]
V: n/a (no irreversible act; installing stays with operator_deployment_update)
verdict: PASS
surface: hestia-deploy full cycle (timer / update button)   act: install + enable hestia-deploy-check units where the deploy service is installed
S: low/reversible [construct: install_check_units; the timer only runs --check; systemctl --user disable undoes it, and a later cycle does not re-enable]
R/W: unchanged (same supervisor, same operator-originated or timer trigger)   O: pass [construct: runs after sync, before the BIN guard and any build]
A: pass [construct: deploy.log lines "installed the update-check units" / "enabled hestia-deploy-check.timer"]   V: n/a
verdict: PASS

Addresses #438: the header now answers "is the daemon current with main?" from a dated check. It doesn't add #438's second axis (lag size and whether it touches core/src). Follows #1334.

🤖 Generated with Claude Code

…at it found

The dashboard said "deployment current" when the running build matched this box's own
install record, which says nothing about hestia main (#438), and "check for updates"
ran a full deploy cycle. Nothing checked on a schedule.

- hestia-deploy records each sync's verdict in deploy-check.tsv (up-to-date /
  available / checking / failed, with the time); --check no longer self-updates and a
  hold no longer stops it
- hestia-deploy-check.{service,timer}: --check every 4h (02/4:17), installed by
  install.sh and by the next full cycle on boxes that predate it
- daemon: DeploymentHealth.check / check_capable and POST /api/operator/deployment/check,
  witnessed before it starts only the check unit
- dashboard: "up to date · checked <time>", "update available", "checking…",
  "install incomplete"; the button checks, and becomes "update" when there is one

Asked for by the operator on tl-dell, 2026-10-21. Review-gate blocks in the PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

dp-web4 commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Reviewed current head: separates update check from deployment and surfaces checked/version state with scheduled check cadence. Three CI workflows green. No blocker found, contingent on check-only path truly not installing/restarting (as specified).

@dp-web4
dp-web4 merged commit 3b7690d into main Oct 22, 2026
10 checks passed
dp-web4 added a commit that referenced this pull request Oct 22, 2026
…here (Nova hold); merge main

Merges origin/main (#1364, deploy check-only). The one conflict was the
DEPLOYED log line: it keeps post_deploy=$post_deploy and #1364's
write_check_result after it.

Nova's hold: the post-deploy directory extends the deploy's privileged
execution surface. Every check is now on the RESOLVED path:
- $HESTIA_HOME and post-deploy.d must be owned by the deploy user and not
  group/world writable, and post-deploy.d must not be a symlink; otherwise
  nothing runs: refused(post-deploy.d not operator-only).
- a step's realpath must be a regular file directly in post-deploy.d, owned
  by the deploy user and not group/world writable. A symlink out of the
  directory is skipped ("escapes post-deploy.d"), always: no allowlist. A link
  into a repo a governed member can write (SAGE) would let that member choose
  what the deploy runs, so the operator installs a copy.
- the checked path is the one executed, with stdin from /dev/null and output
  to the log file (no pipe a lingering grandchild can hold), bounded by
  timeout -k (KILL after POST_DEPLOY_KILL_SECS for a step ignoring TERM) where
  timeout(1) has -k. A timeout is logged as one. A failure is caught with
  `|| rc=$?`, so the caller's next line runs under set -euo pipefail.

post_deploy_step_test.py now extracts the two helpers with the function and
covers: escape skipped and not run; a link inside the dir runs; a
group-writable target, directly and via an inside link, skipped; a
group-writable dir, group-writable home, and symlinked dir each refuse all;
a failing step and a TERM-ignoring hung step that leaves a grandchild holding
its output each return within the bound, the next step runs, and a sentinel
after the call prints. Sabotaging each check (escape, target perms, dir/home
refusal, symlinked dir, home, kill-after, failure containment, build export)
fails the test.

README: the CBP install is `install -m 0755` of the snapshot script, which
finds the workspace from HESTIA_WORKSPACE (set in the deploy unit for the
inventory step).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014LYVQN8U1xqy6U6pbyUDvH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants