Console e2e: a worker is drained from Fleet with no dialog, and Undo undrains it fails on the row's drained text #886

Activity

1. V3RON

   added

   bug:newReported. Nobody has looked yet.Reported. Nobody has looked yet.

   flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.Names a test that fails without a code change. Delivery skips its base-commit check.

   on Oct 22, 2026
2. V3RON

   added a commit that references this issue on Oct 23, 2026

   fix(e2e): move the mouse off the menu so a new toast is not hovered (#…

   ...

   bfdf9c6
3. V3RON

   added

   bug:triageAn agent may reproduce it and write the triage report.An agent may reproduce it and write the triage report.

   on Oct 23, 2026
4. V3RON

   self-assigned this

   on Oct 23, 2026
5. github-actions

   removed

   bug:newReported. Nobody has looked yet.Reported. Nobody has looked yet.

   on Oct 23, 2026
6. V3RON commented on Oct 23, 2026

   

   V3RON

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Reproduction

   Defect in the test, not the console: the drain test reads the real host's load, and on a host whose data volume is 95% or more full the Fleet row's state says disk 98% instead of drained. That is why it fails on one machine and passes on CI. It still fails on origin/main after #903, and #903 is not the cause: this test does not use act and has no toast timer.

   bug/886-repro (9bd3260) adds one line to e2e/console/actions.spec.ts 'a worker is drained from Fleet with no dialog, and Undo undrains it': answerWithLoad(page, { cpu: 12, disk: 98, memory: 54 }), the reading of the machine #886 was seen on. Now it fails on every machine. With disk: 40 it passes.

   pnpm test:console --project=chromium e2e/console/actions.spec.ts -g 'drained from Fleet'
   
   Expected substring: "drained"
   Received string:    "drain-amacOS 26.6.1 · simlock 0.3.0disk 98%0 / 1612%54%98%"
   > 187 |       await expect(row).toContainText("drained", { timeout: 10_000 });
   

   On origin/main with no pin, this host fails 3 of 3 runs on the same line (received …disk 98%0 / 16—54%98%). df -h /System/Volumes/Data shows 99% used.

   Root cause

   ui/src/model/fleet.ts:18 workerHealth shows only the worst finding. A disk at 95% or more (load-thresholds.ts:7, errorPercent: 95) is error and ranks above drained (warn, fleet.ts:37), as DESIGN.md says at line 99. The console is right. The bug is at e2e/console/actions.spec.ts:166: unlike the six Fleet state tests in workers.spec.ts, it does not call answerWithoutLoad. The doc comment on that helper (e2e/console/workers-answers.ts:68) names this exact case: a drain hidden by the host's disk.

   Simplest fix

   Replace the repro's answerWithLoad(...) line with await answerWithoutLoad(page);. I checked this on this 98%-full host: 5 of 5 passed (--repeat-each=5).

   Alternatives rejected

   • Rank drained above load findings: this changes the documented worst-first rule (DESIGN.md:99) to fix a test.
   • Assert the drain through the menu's "Undrain" label: the row's word stops being checked end to end, and the precedent is answerWithoutLoad.
   • Pin host load in the daemon via an env seam: production code for a problem the page-level helper already solves.

   Risk

   • After the fix, CI cannot show the regression, since its disks are not full. Only a run on a full host proves it.
   • The Undo step's not.toContainText("drained") passes on a row that shows a load word. Stripping the load also makes that step meaningful.

   Written by an agent.
7. V3RON

   removed their assignment

   on Oct 23, 2026
8. V3RON commented on Oct 23, 2026

   

   V3RON

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Root cause, found during the #906 delivery run: the test reads the real host disk. On a Mac with a 99% full data volume, the Fleet row shows "disk 99%" in place of "drained", because a disk at 85% or more outranks drained in the row's status (ui/src/model/fleet.test.ts:84). So the test is not flaky. It fails on any machine whose disk is at 85% or more, and passes on CI.

   Fix: pin the fake worker's disk numbers in the test fleet (or in startFleet) so no console e2e test depends on the host's free space. Other console tests that read a row's status text have the same exposure.

   Written by an agent.
9. V3RON

   removed

   flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.Names a test that fails without a code change. Delivery skips its base-commit check.

   on Oct 23, 2026
10. V3RON

    closed this as completedin #938on Oct 23, 2026
11. V3RON

    added a commit that references this issue on Oct 23, 2026

    feat(inbox): the daemon says what needs the operator: GET /v1/inbox (#…

    ...

    02d4aab