Docs: 2026.07.1 release notes and the 2026.06.0 dispatcher crash loop

docs/RELEASE_NOTES_v2026.07.0.md

@@ -23,6 +23,66 @@ with the tenancy work; both only drop and re-create PostgreSQL row-level-securit
 policies, so they apply to an existing database without touching table schemas or
 rows.
 
+## ⚠️ Known issue — upgrading breaks job execution
+
+> **Fixed in 2026.07.1.** Upgrade to it instead of applying the workaround
+> below — see the [2026.07.1 release notes](RELEASE_NOTES_v2026.07.1.md). The
+> rest of this section describes what happens if you stay on 2026.07.0.
+
+**If you upgrade an existing 2026.06.0 install, jobs stop running: they are
+accepted and then stay in `pending` indefinitely.** The only hint is the job's
+`job_explanation`, *"This job is not ready to start because there is not enough
+available capacity"* — accurate, but it does not point at the cause. Fresh
+installs are unaffected. The workaround below is verified on a live cluster.
+
+The `default` instance group has to satisfy two conditions at once for a job to
+run locally, and an upgrade breaks both:
+
+1. **It must contain an instance.** The chart's init Job calls `register_queue
+   --queuename=default`, which on an upgrade finds the group already there,
+   prints `Instance Group already registered default` and assigns nothing.
+2. **That instance must be able to execute.** The task pod re-registers itself
+   as `node_type=control` on every start, and a control node only orchestrates.
+
+Either one alone is enough to hang every launch — both were measured
+individually, holding the other fixed.
+
+Project updates keep working, because they run in `controlplane`, which does
+have the instance. The install therefore looks healthy right up until someone
+launches a job.
+
+**Workaround, after `helm upgrade` completes:**
+
+```bash
+kubectl -n forail exec deploy/forail-web -- forail-manage shell -c "
+from forail.main.models import Instance, InstanceGroup
+i = Instance.objects.get(hostname='forail-node')
+i.node_type='hybrid'; i.save(update_fields=['node_type'])
+InstanceGroup.objects.get(name='default').instances.add(i)"
+```
+
+Substitute your own instance hostname if you did not install with the chart
+defaults. Any job already sitting in `pending` starts on its own within about a
+minute; a fresh short job should return a normal `PLAY RECAP` in a few seconds.
+
+> **The workaround does not survive a restart of `forail-task`.** That pod
+> re-runs `provision_instance` every time it starts — after a node reboot, an
+> eviction, or the next `helm upgrade` — and that call resets both `node_type`
+> and the group's execution mode. Re-apply it, and re-check job execution, after
+> any task-pod restart until the fix ships.
+
+**Also re-apply your role assignments after upgrading.** Any role assignment
+attempted on 2026.06.0 failed silently (the `ScanFinding` /
+`TenantIsolationEvent` `FieldDoesNotExist` bug fixed in this release, see
+*Fixed*). The upgrade fixes the cause but does not recreate the assignments that
+were lost, and *Fixed* saying "no data migration is required" refers to the
+schema only. Check the members of every role you rely on and re-grant what is
+missing.
+
+What the upgrade does do correctly: it succeeds, migrations `0209` and `0210`
+apply cleanly, and no data is lost — object counts and names are identical
+before and after.
+
 ## Security advisories
 
 **Upgrade from 2026.06.0 or earlier is strongly recommended.** Several of the
@@ -259,6 +319,18 @@ front of the ingress, or set `forail.cookieSecure: "false"` for a lab install.
 
 ## Fixed
 
+- **The task dispatcher crash-looped on 2026.06.0, so no job could finish.**
+  The periodic schedule runs `update_active_jobs_gauge_task` every 30 seconds
+  unconditionally, but in 2026.06.0 that function carried Celery's
+  `@shared_task` instead of Forail's own `@task()`. Dispatching it raised
+  `ValueError: ... is not decorated with @task()`, the dispatcher exited, and
+  whatever was running died with *"Task was canceled due to receiving a
+  shutdown signal"* — typically surfacing as a failed project update and a job
+  in `error`. Measured on a fresh 2026.06.0 install: the dispatcher restarted
+  roughly every 50 seconds, indefinitely. **Anyone still on 2026.06.0 should
+  upgrade**; there is no configuration that avoids this, since the schedule
+  entry is not conditional. Fixed by registering the task properly
+  (`@task(queue=get_task_queuename)`).
 - **In-cluster job execution.** Two pieces were missing from the chart, and each
   failed a launch on its own. Note this is not "out of the box": project updates
   and control-plane jobs still run through podman inside the task pod, so they
@@ -326,6 +398,10 @@ helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \
 
 Before upgrading:
 
+- **Any deployment that runs jobs** — read **Known issue — upgrading breaks job
+  execution** at the top of these notes, and plan to apply the workaround (and
+  re-apply role assignments) as part of the upgrade. Without it the platform
+  comes up healthy but executes nothing.
 - **SAML deployments** — review **Breaking changes — SAML** above and reconfigure
   the IdP if needed.
 - **Any deployment** — review **Breaking changes — deployment defaults**; an

docs/RELEASE_NOTES_v2026.07.1.md

@@ -0,0 +1,90 @@
+# Forail 2026.07.1 — Release Notes
+
+**Release date:** 2026-07-26
+**Based on:** Forail 2026.07.0
+**License:** Apache License 2.0
+
+---
+
+## Overview
+
+2026.07.1 is a **patch release with one purpose**: an upgrade no longer leaves a
+platform that accepts jobs and runs none of them. It fixes the known issue
+published with 2026.07.0, and it removes the manual workaround that release
+asked for.
+
+Nothing else changes. There are no migrations, no configuration changes, no
+breaking changes. The frontend, the operator and the assistant are unchanged and
+keep their 2026.07.0 / 2026.07.1 / 2026.06.0 versions respectively; only the
+backend image and the Helm chart move.
+
+| Component | Version |
+|---|---|
+| `forail-backend` | **2026.07.1** |
+| Helm chart | **2026.7.1** (pins the backend above) |
+| `forail-frontend` | 2026.07.0, unchanged |
+| `forail-operator` | 2026.07.1, unchanged |
+| `forail-assistant` | 2026.06.0, unchanged |
+
+## Fixed
+
+### Jobs no longer stop running after an upgrade or a restart
+
+Upgrading 2026.06.0 → 2026.07.0 left every launch sitting in `pending`
+indefinitely, with only `job_explanation` — *"This job is not ready to start
+because there is not enough available capacity"* — to explain it. Project
+updates kept working, so the install looked healthy right up until someone
+launched a job.
+
+**What was wrong.** For jobs to run on the node itself, the `default` instance
+group has to be a regular group that contains an execution-capable instance.
+Two things conspired against that, and either one alone was enough to hang every
+launch:
+
+- `register_queue` assigns instances only when it *creates* a group. On an
+  upgrade the group already exists, so it assigned nothing and left it empty.
+- The task pod re-ran `provision_instance` on **every start** — restart,
+  eviction, rolling upgrade — and that call hardcoded `node_type='control'` and
+  re-registered `default` as a ContainerGroup, overwriting whatever the
+  installer had configured. A control node only orchestrates; it does not
+  execute.
+
+The second half is why the 2026.07.0 workaround did not stick: the next task-pod
+restart quietly undid it.
+
+**What changed.** Registration now takes its intent from `FORAIL_NODE_TYPE` —
+which the Helm chart and the Compose stack already set — and derives the default
+queue from it. An execution-capable pod (`hybrid`, `execution`) gets a regular
+instance group containing itself; a control-only pod keeps the ContainerGroup,
+exactly as before. Both defaults are unchanged when the variable is unset, so a
+multi-node install that has no opinion behaves as it did. The chart's init Job
+additionally asserts group membership rather than trusting `register_queue`, so
+a newer chart paired with an older image still converges.
+
+**Verified, not assumed.** On a freshly created cluster: install 2026.06.0,
+`helm upgrade` to this release, launch a job — successful in 78 s with a normal
+`PLAY RECAP`, with no manual intervention at any point. Two subsequent
+`kubectl rollout restart deploy/forail-task` left the state untouched and the
+next job succeeded as well.
+
+## Upgrade
+
+From **2026.07.0**, a straight image re-point. No migrations, no new required
+values:
+
+```bash
+helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \
+    --version 2026.7.1 -n forail \
+    --set secrets.forailAdminPassword='<strong-password>' \
+    --set 'forail.allowedHosts=forail.example.com\,127.0.0.1\,localhost' \
+    --set task.privileged=true --set task.hostCgroup=true   # only if you run jobs in-pod
+```
+
+**If you applied the 2026.07.0 workaround**, you can leave it in place — it sets
+exactly the state this release converges to on its own. Nothing needs to be
+undone.
+
+From **2026.06.0**, read the [2026.07.0 release
+notes](RELEASE_NOTES_v2026.07.0.md) first: the breaking changes, the required
+admin password and the SAML defaults all still apply. The known issue documented
+there no longer does.