Flaky: test_live_stream_fences_within_ten_seconds_and_keeps_liability[revoke] locks engine_modelrequestreservation on SQLite

[Brad-Edwards]

Summary

tests/engine/services/test_model_broker_http.py::test_live_stream_fences_within_ten_seconds_and_keeps_liability[revoke]
fails intermittently with a SQLite write-lock error, failing the whole
Quality / Tests (shifter_platform) job and therefore gating deploys.

Evidence

Run 35823199983
(Deploy -> orthanc, branch orthanc, sha 0f9b3d50):

FAILED tests/engine/services/test_model_broker_http.py::test_live_stream_fences_within_ten_seconds_and_keeps_liability[revoke]
  - django.db.utils.OperationalError: database table is locked: engine_modelrequestreservation

====== 1 failed, 9063 passed, 1 skipped, 7 warnings in 1083.05s (0:18:03) ======

Underlying error surfaces first as:

E  sqlite3.OperationalError: database table is locked: engine_modelrequestreservation
   .venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py:359: OperationalError

Only the revoke parameterisation failed; the test is parameterised
["revoke", "control_loss", "disconnect", "drain", "deadline"].

Why it matters

gcp-dev is gated on needs.quality.result == 'success', so a single flake in
this test skips the deploy job entirely. During an orthanc tenant standup this
cost a full ~24 minute cycle and blocked a time-boxed range rollout.

Suspected cause

The test drives a live streaming request and then calls
revoke_model_generation(...) through sync_to_async while the ASGI server
task still holds a transaction touching engine_modelrequestreservation. On
SQLite that is a writer/writer conflict rather than a serialisable wait, so the
second writer raises table is locked instead of blocking. The
PostgreSQL-backed job (Quality / Tests (shifter_platform, PostgreSQL)) does
not appear to exhibit it.

Suggested investigation

• Confirm whether the failure reproduces under --count / repeated runs on
  SQLite only, and whether it is specific to the revoke fence.
• Check whether the revoke path and the streaming fence path can be made to
  share one connection/transaction, or whether the test should await quiescence
  of the server task before revoking.
• If the race is genuinely test-only, fix the fixture ordering rather than
  adding a retry or marking it flaky: the assertion it makes (fence within 10s,
  liability retained) is worth keeping strict.

Please do not resolve this by skipping, xfail-ing, or retrying the test without
first establishing whether the lock contention can also occur in production
against a real database.