Fix job execution breaking on upgrade and restart (2026.07.1) - #3
Merged
Conversation
The Kubernetes self-registration branch of provision_instance hardcoded node_type='control' and re-registered the default queue as a ContainerGroup. The task pod re-runs that command every time it starts, and both calls overwrite what is already in the database -- so a restart, an eviction or a rolling upgrade silently undid whatever the installer had configured. On an upgrade the two combine into a platform that runs nothing: the default group is left without an execution-capable member, launches are accepted and then sit in "pending" forever with only "not enough available capacity" to go on. Measured on a live cluster, one variable at a time -- a regular group with no member hangs, a regular group whose only member is node_type=control hangs, and the same group with a hybrid member runs the job in four seconds. Take the intent from FORAIL_NODE_TYPE, which the chart and the Compose stack already set, and derive the default queue from it: an execution-capable pod runs jobs itself and needs a regular group that contains it, a control-only pod dispatches to a ContainerGroup. Both defaults are unchanged when the variable is unset, so a multi-node install that has no opinion behaves exactly as before.
The test job runs only tests_standalone/, whose modules stub Django out, so nothing in CI ever exercised the real application -- the instance and instance-group tests that cover how jobs get routed included. Installing the full requirements on a runner is impractical (python-ldap and friends), but the image about to be published already has the application installed. Run them there, between build and push, so a regression stops the release instead of shipping in it.
Also closes the changelog section that was still labelled [Unreleased] while 2026.07.0 was already published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrading 2026.06.0 → 2026.07.0 left a platform that accepts jobs and runs none of them: every launch sat in
pending, with only "not enough available capacity" to explain it.Cause
For jobs to run on the node itself,
defaulthas to be a regular instance group holding an execution-capable instance. Two things broke that, and either alone was enough to hang every launch:register_queueassigns instances only when it creates the group, so on an upgrade it assigned nothing.provision_instanceon every start, and that call hardcodednode_type='control'and re-registereddefaultas a ContainerGroup — overwriting whatever the installer configured. That is also why the published workaround did not survive a restart.Measured one variable at a time on a live cluster:
defaultgroupnode_typeChange
Registration takes its intent from
FORAIL_NODE_TYPE, which the chart and the Compose stack already set, and derives the default queue from it: an execution-capable pod gets a regular group containing itself, a control-only pod keeps the ContainerGroup. Both defaults are unchanged when the variable is unset.Also adds a CI step that runs the Django functional tests inside the image being published — the
testjob only ever rantests_standalone/, which stubs Django out.Verification
helm upgrade→ job successful in 78 s with a cleanPLAY RECAP, no manual intervention; tworollout restart deploy/forail-taskleft the state untouched and the next job succeeded