[NovrusShehaj]

Summary

backend-ci has been failing intermittently for reasons unrelated to the PRs it runs on. Two separate flakes, each fixed at its root cause. Test-only and CI-only changes: no biosim_server/ code changes, nothing to release or deploy.

1. Keycloak testcontainer readiness (e.g. PR #126, run 38454571680)

All 13 errors were 404 Client Error: Not Found for url: http://localhost:<port>/health/ready. In testcontainers 4.14.2, KeycloakContainer._readiness_probe:

1. GETs <management port 9000>/health/ready;
2. if that port isn't listening yet (ConnectionError), falls back to <http port 8080>/health/ready, which is a 404 on Keycloak ≥ 25;
3. raise_for_status() raises HTTPError, which is not one of the exceptions @wait_container_is_ready(ConnectionError, ReadTimeout) retries, so startup fails immediately.

The outcome depends on whether the management port is up on the first poll, which is why it was intermittent. On top of that, the image was the unpinned quay.io/keycloak/keycloak:latest.

Fix (backend/tests/fixtures/keycloak/container.py): a small KeycloakContainer subclass whose probe polls only the management port and retries any RequestException until a 180 s deadline. If the container exits first, the probe fails immediately instead of waiting out the deadline, and both failures include the last 50 lines of Keycloak's output. The image is pinned to 26.7.5.

2. Temporal dev-server download (main, run 39444224018)

11 errors: RuntimeError: Failed starting Temporal dev server: error decoding response body. WorkflowEnvironment.start_local downloads the dev server from temporal.download on every CI run.

Fix:

• ci.yaml installs a pinned Temporal CLI (1.9.1) from the GitHub release. The download uses curl --retry, is verified against the release's SHA-256, and is cached per version with actions/cache. The step then exports TEMPORAL_CLI_PATH.
• temporal_fixtures.py passes it as dev_server_existing_path. When it's unset (local dev), the SDK downloads exactly as before.

Test plan

• uv run ruff check .: clean
• uv run mypy biosim_server tests: clean (193 files)
• The 13 previously failing Keycloak tests plus the Temporal-backed tests (test_temporal.py, test_simulations_run.py, test_verification_completion.py), using a pre-installed CLI (macOS build of the same 1.9.1): 84 passed
• Full suite with TEMPORAL_CLI_PATH set: 1478 passed, 17 skipped (same credential-gated skips as CI)
• Keycloak tests 3 further times back-to-back: 11/11 passed each time
• Probe outcomes against a real Keycloak container: ready (a realm user gets a token the moment the probe returns), exited (fails in 1.3 s with Keycloak's own startup error), forced 3 s timeout (error includes the log tail)
• backend-ci green on this PR and stable across re-runs: 5/5 attempts of run 39642239080 on 0609708, plus run 39657354708 on 2f710f1 (1478 passed, 17 skipped each)

[NovrusShehaj]

No bugs found; this looks safe to merge. Notes below are non-blocking.

Verified

• Keycloak root cause matches the locked testcontainers 4.14.2 source: _readiness_probe falls back to the HTTP port's /health/ready on ConnectionError (a 404 on Keycloak ≥ 25), and the resulting HTTPError isn't in @wait_container_is_ready(ConnectionError, ReadTimeout). The same code is in 4.13.3 and 4.14.0.
• quay.io/keycloak/keycloak:26.7.5 exists on quay.
• TEMPORAL_CLI_SHA256 matches temporal_cli_1.9.1_linux_amd64.tar.gz in the v1.9.1 release's checksums.txt. temporalio 1.30.0's start_local accepts dev_server_existing_path.
• CI: run 39642239080 went through 5 attempts and all passed (1478 passed, 17 skipped). Attempt 1 had no cache: it downloaded the CLI, verified the checksum (temporal_cli.tar.gz: OK) and saved the cache. Attempts 2–5 restored the CLI from the cache. That covers the last unticked test-plan item.

Suggestions

1. The override drops upstream's log waits (container.py#L67). In start-dev mode, upstream also waits for the "started in …" and "Created temporary admin user|Added user '" log lines. Skipping them is fine today: the tests only call the realm's token endpoint (tokens.py), never the admin client (get_client()), and 8 green runs show the realm import has finished by the time /health/ready passes. Nothing records the deviation, though, so a docstring sentence would warn whoever first reaches for the admin client.

2. The 180 s comment overstates its scope (container.py#L24). It says the deadline covers a cold runner pulling the image, but the pull happens in super().start() (docker_client.run) before _readiness_probe runs. The deadline only covers boot plus the realm import. For reference, upstream's default is 120 s (TC_MAX_TRIES=120 × 1 s).

3. A timeout leaves little to debug (container.py#L75-L78). The TimeoutError carries only the last request error, usually "connection refused", when the Keycloak log is what you'd want from a failed CI run. Appending the tail of self.get_logs() would fix that. Checking whether the container has exited would also make a bad realm.json fail at once instead of after 180 s.

4. A cached CLI is never re-verified (ci.yaml#L39). The checksum runs only on a cache miss; a cache hit trusts the binary as is. Cache scoping (PR runs can't write caches that main reads) keeps the risk low. Caching the tarball and verifying it on every run would give the full guarantee for about a second.

5. CI's CLI version can drift from local (ci.yaml#L19). CI pins CLI 1.9.1, while local runs get the SDK's "default" dev-server version, which follows the temporalio version. A temporalio upgrade won't move CI unless someone remembers. A comment next to TEMPORAL_CLI_VERSION to bump it with temporalio would help. Optionally, backend/CLAUDE.md could mention that TEMPORAL_CLI_PATH also skips the download in local dev.

I'll push follow-ups for 1–3 to this branch.

[NovrusShehaj]

Re-review at 2f710f1: items 1–3 from the earlier review are addressed, with no new issues.

• 1: the docstring records the skipped log waits and says to restore the admin-user wait before using get_client().
• 2: the timeout comment is corrected.
• 3: an exited container now fails at once (1.3 s instead of 180 s against a real container), and both errors include the last 50 lines of Keycloak's output.

backend-ci is green on 2f710f1 (1478 passed, 17 skipped). One trade-off: the loop now calls Docker (reload(), get_logs()) inside the error handler, so a one-off Docker API error ends startup instead of being retried. testcontainers' own log waits check the container on every poll the same way, so this is fine as is.

Still open, both optional: 4 (a cached CLI isn't re-verified) and 5 (CI's CLI version can drift from local). I've updated the description with the new behaviour and the CI results.

[jcschaff]

looks very good and save - approved.