fix(apps): report a startup version pin that reverts a running app

[nilsmechtel]

A worker pod restart can roll a production app backwards by several versions with nothing anywhere reporting it. The app comes back up, passes every health check and serves traffic normally — it is just running older code than it was five minutes earlier. deNBI lost model-runner from 2.7.2 to 2.4.2 this way on 2026-09-01, and it was found only because someone compared versions by hand before an unrelated deployment.

There are two sources of truth for "what version is this app". The worker's runtime state, set by deploy_app() over the API and reported by get_app_status(); and the Deployment's --startup-applications argument, rendered from the cluster's values.yaml and replayed by deploy_startup_applications() at boot. An out-of-band roll changes only the first, so the next restart — node drain, OOM kill, liveness kill, helm upgrade, image roll — reinstates the committed pin. Nothing catches it because that redeploy is indistinguishable from a normal successful startup: RUNNING, HEALTHY, no warning. Every probe passes, because the older version is not broken.

The worker already holds both numbers at that moment: recover_deployed_applications() runs first and has just enumerated what survived the restart. It simply never compared them.

What changed

deploy_startup_applications() logs a WARNING before redeploying at the pin when the app was found running at a different version of the same artifact. That converts a silent revert into a greppable line naming both versions, and it alone would have caught the deNBI instance.

get_app_status() returns pinned_version alongside version / running_version / version_verified, so the divergence is checkable over the API — assertable in CI or a monitor — without reading cluster config. It is None when the app is not a startup application, and short artifact ids in the pin are resolved to the workspace-qualified form before matching.

Both are pure observability. Neither changes what gets deployed, so neither can make a rollback worse than it is today.

Three conditions deliberately stay silent, because in each the pin is not reverting anything: a pin whose version matches what is running; a pin carrying no version at all (deploy_app then inherits the running version); and a pin whose application_id matches but whose artifact does not, where the two version strings belong to different release lines and comparing them says nothing.

What is deliberately not here

The issue's third candidate fix — refuse a downgrade unless an explicit allow_downgrade flag is passed — is out of scope. It is not a stronger version of the two above, it is a different decision: refusing a downgrade also blocks deliberate rollback, and the last thing you want failing closed during an incident is the command that puts the old version back. That tradeoff is yours to make, and the WARNING added here is the evidence that argument needs — it shows how often divergence actually occurs before anyone commits to blocking on it.

Note that none of this removes the need to bump the pin in the same session as an out-of-band roll. It makes the omission visible instead of silent.

Verification

Seven new tests in tests/apps/test_startup_pin_divergence.py, covering the warning, the three silent cases, and the status field including short-id resolution. 184 tests pass across tests/_app, tests/apps and tests/worker (the 32 errors under tests/apps/model-runner and tests/apps/cellpose are the pre-existing --noconftest fixture errors, unrelated).

Both halves were positive-controlled rather than merely asserted. Removing the _warn_on_startup_pin_divergence call makes the warning test fail with assert [] — no log record at all, which is exactly the production symptom. Removing the pinned_version key makes the three status tests fail with KeyError: 'pinned_version'.

tests/apps/test_status_app_missing_from_cluster.py needed one line: its hand-built manager now sets startup_applications = [], since _get_app_status reads that attribute.

Related

#0023 describes the same "two sources of truth, only one reported" shape on the deploy path rather than the restart path, and GH #157 adds the frontend variant. Whatever surfaces head-vs-deployed there should reuse this field.