diff --git a/docs/RELEASE_NOTES_v2026.07.0.md b/docs/RELEASE_NOTES_v2026.07.0.md index 1686200..26294d5 100644 --- a/docs/RELEASE_NOTES_v2026.07.0.md +++ b/docs/RELEASE_NOTES_v2026.07.0.md @@ -23,6 +23,66 @@ with the tenancy work; both only drop and re-create PostgreSQL row-level-securit policies, so they apply to an existing database without touching table schemas or rows. +## ⚠️ Known issue — upgrading breaks job execution + +> **Fixed in 2026.07.1.** Upgrade to it instead of applying the workaround +> below — see the [2026.07.1 release notes](RELEASE_NOTES_v2026.07.1.md). The +> rest of this section describes what happens if you stay on 2026.07.0. + +**If you upgrade an existing 2026.06.0 install, jobs stop running: they are +accepted and then stay in `pending` indefinitely.** The only hint is the job's +`job_explanation`, *"This job is not ready to start because there is not enough +available capacity"* — accurate, but it does not point at the cause. Fresh +installs are unaffected. The workaround below is verified on a live cluster. + +The `default` instance group has to satisfy two conditions at once for a job to +run locally, and an upgrade breaks both: + +1. **It must contain an instance.** The chart's init Job calls `register_queue + --queuename=default`, which on an upgrade finds the group already there, + prints `Instance Group already registered default` and assigns nothing. +2. **That instance must be able to execute.** The task pod re-registers itself + as `node_type=control` on every start, and a control node only orchestrates. + +Either one alone is enough to hang every launch — both were measured +individually, holding the other fixed. + +Project updates keep working, because they run in `controlplane`, which does +have the instance. The install therefore looks healthy right up until someone +launches a job. + +**Workaround, after `helm upgrade` completes:** + +```bash +kubectl -n forail exec deploy/forail-web -- forail-manage shell -c " +from forail.main.models import Instance, InstanceGroup +i = Instance.objects.get(hostname='forail-node') +i.node_type='hybrid'; i.save(update_fields=['node_type']) +InstanceGroup.objects.get(name='default').instances.add(i)" +``` + +Substitute your own instance hostname if you did not install with the chart +defaults. Any job already sitting in `pending` starts on its own within about a +minute; a fresh short job should return a normal `PLAY RECAP` in a few seconds. + +> **The workaround does not survive a restart of `forail-task`.** That pod +> re-runs `provision_instance` every time it starts — after a node reboot, an +> eviction, or the next `helm upgrade` — and that call resets both `node_type` +> and the group's execution mode. Re-apply it, and re-check job execution, after +> any task-pod restart until the fix ships. + +**Also re-apply your role assignments after upgrading.** Any role assignment +attempted on 2026.06.0 failed silently (the `ScanFinding` / +`TenantIsolationEvent` `FieldDoesNotExist` bug fixed in this release, see +*Fixed*). The upgrade fixes the cause but does not recreate the assignments that +were lost, and *Fixed* saying "no data migration is required" refers to the +schema only. Check the members of every role you rely on and re-grant what is +missing. + +What the upgrade does do correctly: it succeeds, migrations `0209` and `0210` +apply cleanly, and no data is lost — object counts and names are identical +before and after. + ## Security advisories **Upgrade from 2026.06.0 or earlier is strongly recommended.** Several of the @@ -259,6 +319,18 @@ front of the ingress, or set `forail.cookieSecure: "false"` for a lab install. ## Fixed +- **The task dispatcher crash-looped on 2026.06.0, so no job could finish.** + The periodic schedule runs `update_active_jobs_gauge_task` every 30 seconds + unconditionally, but in 2026.06.0 that function carried Celery's + `@shared_task` instead of Forail's own `@task()`. Dispatching it raised + `ValueError: ... is not decorated with @task()`, the dispatcher exited, and + whatever was running died with *"Task was canceled due to receiving a + shutdown signal"* — typically surfacing as a failed project update and a job + in `error`. Measured on a fresh 2026.06.0 install: the dispatcher restarted + roughly every 50 seconds, indefinitely. **Anyone still on 2026.06.0 should + upgrade**; there is no configuration that avoids this, since the schedule + entry is not conditional. Fixed by registering the task properly + (`@task(queue=get_task_queuename)`). - **In-cluster job execution.** Two pieces were missing from the chart, and each failed a launch on its own. Note this is not "out of the box": project updates and control-plane jobs still run through podman inside the task pod, so they @@ -326,6 +398,10 @@ helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \ Before upgrading: +- **Any deployment that runs jobs** — read **Known issue — upgrading breaks job + execution** at the top of these notes, and plan to apply the workaround (and + re-apply role assignments) as part of the upgrade. Without it the platform + comes up healthy but executes nothing. - **SAML deployments** — review **Breaking changes — SAML** above and reconfigure the IdP if needed. - **Any deployment** — review **Breaking changes — deployment defaults**; an diff --git a/docs/RELEASE_NOTES_v2026.07.1.md b/docs/RELEASE_NOTES_v2026.07.1.md new file mode 100644 index 0000000..afbb068 --- /dev/null +++ b/docs/RELEASE_NOTES_v2026.07.1.md @@ -0,0 +1,90 @@ +# Forail 2026.07.1 — Release Notes + +**Release date:** 2026-07-26 +**Based on:** Forail 2026.07.0 +**License:** Apache License 2.0 + +--- + +## Overview + +2026.07.1 is a **patch release with one purpose**: an upgrade no longer leaves a +platform that accepts jobs and runs none of them. It fixes the known issue +published with 2026.07.0, and it removes the manual workaround that release +asked for. + +Nothing else changes. There are no migrations, no configuration changes, no +breaking changes. The frontend, the operator and the assistant are unchanged and +keep their 2026.07.0 / 2026.07.1 / 2026.06.0 versions respectively; only the +backend image and the Helm chart move. + +| Component | Version | +|---|---| +| `forail-backend` | **2026.07.1** | +| Helm chart | **2026.7.1** (pins the backend above) | +| `forail-frontend` | 2026.07.0, unchanged | +| `forail-operator` | 2026.07.1, unchanged | +| `forail-assistant` | 2026.06.0, unchanged | + +## Fixed + +### Jobs no longer stop running after an upgrade or a restart + +Upgrading 2026.06.0 → 2026.07.0 left every launch sitting in `pending` +indefinitely, with only `job_explanation` — *"This job is not ready to start +because there is not enough available capacity"* — to explain it. Project +updates kept working, so the install looked healthy right up until someone +launched a job. + +**What was wrong.** For jobs to run on the node itself, the `default` instance +group has to be a regular group that contains an execution-capable instance. +Two things conspired against that, and either one alone was enough to hang every +launch: + +- `register_queue` assigns instances only when it *creates* a group. On an + upgrade the group already exists, so it assigned nothing and left it empty. +- The task pod re-ran `provision_instance` on **every start** — restart, + eviction, rolling upgrade — and that call hardcoded `node_type='control'` and + re-registered `default` as a ContainerGroup, overwriting whatever the + installer had configured. A control node only orchestrates; it does not + execute. + +The second half is why the 2026.07.0 workaround did not stick: the next task-pod +restart quietly undid it. + +**What changed.** Registration now takes its intent from `FORAIL_NODE_TYPE` — +which the Helm chart and the Compose stack already set — and derives the default +queue from it. An execution-capable pod (`hybrid`, `execution`) gets a regular +instance group containing itself; a control-only pod keeps the ContainerGroup, +exactly as before. Both defaults are unchanged when the variable is unset, so a +multi-node install that has no opinion behaves as it did. The chart's init Job +additionally asserts group membership rather than trusting `register_queue`, so +a newer chart paired with an older image still converges. + +**Verified, not assumed.** On a freshly created cluster: install 2026.06.0, +`helm upgrade` to this release, launch a job — successful in 78 s with a normal +`PLAY RECAP`, with no manual intervention at any point. Two subsequent +`kubectl rollout restart deploy/forail-task` left the state untouched and the +next job succeeded as well. + +## Upgrade + +From **2026.07.0**, a straight image re-point. No migrations, no new required +values: + +```bash +helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \ + --version 2026.7.1 -n forail \ + --set secrets.forailAdminPassword='' \ + --set 'forail.allowedHosts=forail.example.com\,127.0.0.1\,localhost' \ + --set task.privileged=true --set task.hostCgroup=true # only if you run jobs in-pod +``` + +**If you applied the 2026.07.0 workaround**, you can leave it in place — it sets +exactly the state this release converges to on its own. Nothing needs to be +undone. + +From **2026.06.0**, read the [2026.07.0 release +notes](RELEASE_NOTES_v2026.07.0.md) first: the breaking changes, the required +admin password and the SAML defaults all still apply. The known issue documented +there no longer does.