Skip to content

molecule: two host-env/schema assertions can pass for the wrong reason #51

Description

@thiras

Surfaced by review of #50. Both are pre-existing, but #50 made them likelier to matter by running the six scenarios concurrently (contention, and a real chance of an OOM kill).

1. host-env asserts the effect of a restart, not its cause

ansible/molecule/host-env/verify.yml:159 asserts pid_after != pid_before. The unit it is asserting against sets Restart=always (ansible/roles/decdn_node/templates/decdn-node.service.j2:47), so any independent death of the daemon between side_effect and the second converge satisfies it — systemd restarts it and the PID differs.

Failure mode: the role's env-file-change → restart notify regresses, the stub happens to get OOM-killed once under parallel load, systemd restarts it, the PID differs, the assertion passes, the regression ships. Serially this was near-impossible; at JOBS=6 it is a live false-pass.

Fix direction: assert the cause, not the effect — e.g. capture systemctl show -p NRestarts (or ActiveEnterTimestamp) and require exactly one transition attributable to the converge, or assert on the restart handler having fired in the converge result.

2. host-env and schema never assert the daemon became ready

The role's /metrics readiness probe is deliberately advisory — ansible/roles/decdn_node/tasks/main.yml:1583 retries decdn_readiness_retries × decdn_readiness_delay (30 × 2s ≈ 60s, roles/decdn_node/defaults/main.yml:328-329) and then only warns.

default compensates by positively asserting the socket (molecule/default/verify.yml:320, '127.0.0.1:9090' in listeners.stdout), and slow-readiness asserts the negative. host-env and schema assert neither — grep for 9090/metric in their verify.yml returns only config-text matches.

Failure mode: a daemon that never binds metrics produces one warn line and a green scenario, identical output to a real regression in metrics binding. Interleaved parallel logs make that warn even easier to miss.

Fix direction: add the default-style socket assertion to host-env/schema, or make the warn machine-detectable (assert decdn_metrics_probe.status == 200 everywhere except slow-readiness).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions