Surfaced by review of #50. Both are pre-existing, but #50 made them likelier to matter by running the six scenarios concurrently (contention, and a real chance of an OOM kill).
1. host-env asserts the effect of a restart, not its cause
ansible/molecule/host-env/verify.yml:159 asserts pid_after != pid_before. The unit it is asserting against sets Restart=always (ansible/roles/decdn_node/templates/decdn-node.service.j2:47), so any independent death of the daemon between side_effect and the second converge satisfies it — systemd restarts it and the PID differs.
Failure mode: the role's env-file-change → restart notify regresses, the stub happens to get OOM-killed once under parallel load, systemd restarts it, the PID differs, the assertion passes, the regression ships. Serially this was near-impossible; at JOBS=6 it is a live false-pass.
Fix direction: assert the cause, not the effect — e.g. capture systemctl show -p NRestarts (or ActiveEnterTimestamp) and require exactly one transition attributable to the converge, or assert on the restart handler having fired in the converge result.
2. host-env and schema never assert the daemon became ready
The role's /metrics readiness probe is deliberately advisory — ansible/roles/decdn_node/tasks/main.yml:1583 retries decdn_readiness_retries × decdn_readiness_delay (30 × 2s ≈ 60s, roles/decdn_node/defaults/main.yml:328-329) and then only warns.
default compensates by positively asserting the socket (molecule/default/verify.yml:320, '127.0.0.1:9090' in listeners.stdout), and slow-readiness asserts the negative. host-env and schema assert neither — grep for 9090/metric in their verify.yml returns only config-text matches.
Failure mode: a daemon that never binds metrics produces one warn line and a green scenario, identical output to a real regression in metrics binding. Interleaved parallel logs make that warn even easier to miss.
Fix direction: add the default-style socket assertion to host-env/schema, or make the warn machine-detectable (assert decdn_metrics_probe.status == 200 everywhere except slow-readiness).
Surfaced by review of #50. Both are pre-existing, but #50 made them likelier to matter by running the six scenarios concurrently (contention, and a real chance of an OOM kill).
1.
host-envasserts the effect of a restart, not its causeansible/molecule/host-env/verify.yml:159assertspid_after != pid_before. The unit it is asserting against setsRestart=always(ansible/roles/decdn_node/templates/decdn-node.service.j2:47), so any independent death of the daemon betweenside_effectand the second converge satisfies it — systemd restarts it and the PID differs.Failure mode: the role's env-file-change → restart notify regresses, the stub happens to get OOM-killed once under parallel load, systemd restarts it, the PID differs, the assertion passes, the regression ships. Serially this was near-impossible; at
JOBS=6it is a live false-pass.Fix direction: assert the cause, not the effect — e.g. capture
systemctl show -p NRestarts(orActiveEnterTimestamp) and require exactly one transition attributable to the converge, or assert on the restart handler having fired in the converge result.2.
host-envandschemanever assert the daemon became readyThe role's
/metricsreadiness probe is deliberately advisory —ansible/roles/decdn_node/tasks/main.yml:1583retriesdecdn_readiness_retries×decdn_readiness_delay(30 × 2s ≈ 60s,roles/decdn_node/defaults/main.yml:328-329) and then only warns.defaultcompensates by positively asserting the socket (molecule/default/verify.yml:320,'127.0.0.1:9090' in listeners.stdout), andslow-readinessasserts the negative.host-envandschemaassert neither — grep for9090/metricin theirverify.ymlreturns only config-text matches.Failure mode: a daemon that never binds metrics produces one warn line and a green scenario, identical output to a real regression in metrics binding. Interleaved parallel logs make that warn even easier to miss.
Fix direction: add the
default-style socket assertion tohost-env/schema, or make the warn machine-detectable (assertdecdn_metrics_probe.status == 200everywhere exceptslow-readiness).