test(qa): fix topology warm-up flake in the blacklist script tests

[dborup]

Summary

Fixes a timing flake in the QA blacklist script's unit tests, qa/scripts/test-blacklist-sql.sh. The change is test-only: qa/scripts/blacklist-test.sh and everything under cmd/, public/ and the workflows are unchanged.

Reference: master CI run 36250468641 on b3e44761 failed the step "QA blacklist script unit tests" with 895 passed and 2 failed:

FAIL: topology warm-up then 200: passes — expected '0' got '1'
FAIL: topology warm-up then 200: clean

The same tree passed 897/0 in PR CI, and a rerun of the step was green.

Cause

fetch_topology in blacklist-test.sh retries 503s until a deadline:

local deadline=$(( $(date +%s) + RESTART_WAIT_S ))
...
(( $(date +%s) < deadline )) || return 0
sleep 3

The deadline has whole-second resolution, so the wait lasts between RESTART_WAIT_S - 1 and RESTART_WAIT_S seconds. It is shortest when the wait starts late in a second.

The warm-up case runs with RESTART_WAIT_S=4 and two 503s. The retry after the second 503 is decided about 3 s plus two requests after the wait starts, so a wait that starts late enough in a second gives up with the 503.

Reproduction (commit 1 02a25f87, red on purpose)

The fake target can now run the script on a fake clock. Real time plays no part.

• Fake clock: with FAKE_CLOCK_START_MS, the logexec shim answers date +%s from a millisecond clock file, and sleep N advances that clock instead of waiting.
• Request cost: each fake curl request advances the clock by FAKE_CLOCK_CALL_MS and records its start time in clock.log.
• Test case: the warm-up case starts the topology wait at .950 s, with 50 ms per request. A fixture guard asserts that start time.
• Result: the second 503 returns at .050 s past the deadline second, so fetch_topology returns 503. The run reports ❌ hide-failed: /api/analytics/topology HTTP 503, which is the same pair of failures as in CI, every time.

With 50 ms requests, RESTART_WAIT_S=4 loses the retry for any wait that starts at .900 s or later.

Fix (commit 2 2b37fa0e)

The warm-up and stuck cases pass RESTART_WAIT_S=5, which leaves at least 4 s for a retry needed at about 3.1 s. Both cases run on the fake clock from two starts, .950 s and .000 s, and each has a fixture guard. They still prove both behaviours:

• Warm-up, two 503s then 200: exit 0, ✅ topology clean, and exactly three topology requests.
• Stuck, 503 forever: exit 1, classified as /api/analytics/topology HTTP 503, retried at least once, and the usual common_after checks: config restored, databases and files unchanged, nothing sensitive in argv.

These cases no longer sleep, so the suite also got faster.

Mutation checks

Each variant was run through the fixed suite; the variant files were not committed.

Variant	Result
The fixed cases with RESTART_WAIT_S=4	3 failed, all in the .950 s warm-up case
blacklist-test.sh without the 503 retry	8 failed
blacklist-test.sh without the deadline	4 failed: stuck no longer fails

Results

Tree	bash qa/scripts/test-blacklist-sql.sh
master 97cebd98	897 passed, 0 failed (40.0 s)
commit 1 (reproduction)	896 passed, 2 failed, 3 of 3 runs, same two failures as CI 36250468641
commit 2 (fix)	919 passed, 0 failed (24.4 s)

• Repeated runs: 20 consecutive runs of the fixed suite: 20/20 exit 0, each 919 passed, 0 failed.
• ShellCheck: shellcheck -x -P SCRIPTDIR qa/scripts/blacklist-test.sh qa/scripts/test-blacklist-sql.sh (ShellCheck 0.9.0) produces no findings.

Note on blacklist-test.sh

The whole-second deadline also shortens the real tool's wait by up to one second. With the default RESTART_WAIT_S=120 and a 3 s poll interval, that does not matter. The tool is left unchanged; any improvement to it would be a separate change.

🤖 Generated with Claude Code

https://claude.ai/code/session_019TcZHooUiiknVWbECVWzk8

---

Generated by Claude Code