service-template-897: One committed command, one emission, under concurrency

[vladmesh]

Card: service-template-897 — One committed command, one emission, under concurrency
Branch: pipeline/service-template-897 → main

What the card asks for

Goal

Close the two windows in the released core jobs contract where a fire can execute a behaviour that
no committed command records, or execute it twice. The sprint's Definition of Done requires that
command identity and terminal evidence survive retry and that replay cannot create an unbounded
duplicate execution; service-template-896 delivered that for sequential retries, and these are the
two paths where it does not yet hold.

Both were found by the reviewer of 896 and neither violated its acceptance criteria — they are
released behaviour now, and they are cheaper to close before any product provides jobs.fire than
after.

Context

Repository: /home/dev/projects/service-template, main at d4b07a5 (896 merged).
/home/dev/projects/codegen_orchestrator is not touched by this card, and neither is
scheduler.service_template_ref — the pin moves in the next card, to this card's candidate.

The released code:

• template/services/backend/src/controllers/jobs.py::JobsController.fire — validates, calls
  repository.record(...), returns early when the command is already DISPATCHED, otherwise calls
  _emit(command) and, on success, repository.mark_dispatched(...).
• template/services/backend/src/app/repositories/job_command.py — get, record (read, then
  insert inside begin_nested(), IntegrityError → re-read), mark_dispatched (a flush, no
  commit).
• The request transaction is committed by get_async_db() after the controller returns, so
  _emit() publishes while the row is still uncommitted.
• template/services/backend/src/app/models/job_command.py — identity is the unique
  (fired_by_product, command_id); DispatchStatus is undelivered / dispatched.
• Tests: template/services/backend/tests/unit/test_jobs.py, whose suite runs on
  sqlite+aiosqlite (template/services/backend/tests/conftest.py). The product's real database is
  PostgreSQL.

The invariant

A job_fired event exists only for a command whose row is committed, and at most one emission
ever happens per command identity — however many callers fire or retry it, and however they
interleave.

Enforce it in one named place and make every path reach it. The two known violations:

1. Emit precedes commit. _emit() publishes inside the still-open request transaction. If the
   commit then fails, the behaviour has been executed and no command records it, so the next retry
   executes it again — and each further failed commit repeats that.
2. Concurrent retries of an undelivered command. Both callers read the same non-terminal row,
   neither meets the unique constraint (the row already exists), and both emit.

Acceptance criteria

1. No job_fired is published for a command whose row is not durably committed. Ordering, not
   hope: the command is committed first, and the emission happens after, so a failure between them
   leaves a recorded command that a retry can complete rather than an execution nobody recorded.
2. At most one emission per identity under concurrency. Choose the mechanism and justify it in the
   report — a row lock taken on the emit path, or a conditional terminal transition whose winner is
   decided by the database — and say which hazard your choice accepts: a double emission is what the
   card exists to remove, but a command marked terminal that never emitted would be a lost
   execution wearing trustworthy evidence, which is worse. Name which one your design can still
   produce, in what window, and why that is the safe side.
3. The guarantee is real on the product's actual database. The backend suite runs on SQLite, which
   silently makes some locking a no-op, so state plainly in the report what the suite proves and
   what it cannot: if the mechanism depends on PostgreSQL semantics, show that the statement the
   ORM emits is the one you intend (compiled SQL, dialect-aware), and do not present a green SQLite
   test as proof of mutual exclusion.
4. Everything the released

… (truncated; the full text is on the card)

What the worker reports

service-template-897 — jobs: emit only what is committed, and only once

Branch pipeline/service-template-897, one commit 6dc680e on top of d4b07a5. Working tree
clean. No contract, schema, route, manifest, migration or contract_version changed.

What changed

template/services/backend/src/controllers/jobs.py

• JobsController.fire now: validates → repository.record(...) → session.commit() → hands
  off to one named place, _emit_once(repository, fired_by_product, command_id).
• _emit_once is the only place a job_fired is published and the only place dispatch becomes
  terminal. It takes the committed row's lock, decides from the row that statement returns, emits,
  marks dispatched on a reported delivery, builds the contract, and commits (releasing the lock).

template/services/backend/src/app/repositories/job_command.py

• New module function dispatch_lock_statement(fired_by_product, command_id):
  select(JobCommand).where(identity).with_for_update().execution_options(populate_existing=True),
  and JobCommandRepository.lock_for_dispatch(...) which executes it. It is a module-level function
  precisely so a test can compile it dialect-aware without a database.
• populate_existing is there for the real deployment: a waiting request has its own session, and
  without it SQLAlchemy would hand back the copy that session had already loaded instead of the row
  the winner committed.

template/services/backend/tests/conftest.py

• The test sessionmaker now joins the outer transaction with join_transaction_mode="create_savepoint",
  and the get_async_db override yields the session directly instead of wrapping the request in a
  savepoint. That is the documented SQLAlchemy 2.0 recipe for code that commits inside the request;
  the outer transaction still rolls every test's writes back. No existing assertion was changed or
  weakened — this is the fixture learning to host a controller that owns a commit.

Docs: docs/CONTRACTS.md gains "Committed before emitted, emitted once" inside Core jobs v1;
CHANGELOG.md (English) carries the change under Unreleased/Changed;
template/services/backend/AGENTS.md had a flat rule "Controllers must not call session.commit()" —
it now names the jobs core as the one deliberate exception and says why, because leaving it would
have made the shipped guidance false.

AC2 — the mechanism, and the hazard it accepts

Chosen: a row lock taken on the emit path (SELECT ... FOR UPDATE on the committed row), not a
conditional terminal transition.

Why: the ordering must stay emit, then mark terminal. A conditional UPDATE ... WHERE status = 'undelivered' that claims the command before publishing would let the database pick a unique
winner, but the winner could then fail to deliver, or die, leaving a row marked dispatched with
dispatched_at set and no event ever published — a lost execution wearing trustworthy evidence,
which the card names as the worse outcome and which central QA would read as proof the behaviour
ran. The row lock gets mutual exclusion without ever inverting that order.

Hazard this design can still produce: a second emission, in exactly one window — after
publish_job_fired has returned and before the commit that records dispatched succeeds (process
death, connection loss, a failing commit). The command stays undelivered, so a later retry emits
again. That is at-least-once on a bounded, self-healing path: each retry still records no second
command, and the number of executions is bounded by the number of retries, not unbounded. The
impossible direction is the important one — the row is never marked dispatched before delivery has
been reported, so a command carrying terminal evidence always had its event published.

Cost accepted knowingly: the lock is held across the publish, so a slow broker holds one row lock
per command identity. It serialises only retries of the same identity; different commands never
contend.

… (truncated; the full text is on the card)

---

Opened by the secretary CI gate so that the pull_request workflow runs. Title and body are built from the card and the worker's done report.