Skip to content

Headless route acks a delivery on spawn, before the child has read anything (seam rule 4) #1831

Description

@khaliqgant

Summary

crates/broker/src/runtime/headless.rs sends delivery_ack immediately after
the child process spawns, before the child has read anything. A successful
spawn is not evidence that the message was consumed.

This violates the delivery seam's rule 4 — "never claim an acknowledgement you
did not observe"
(docs/native-delivery-migration.md) — at the source. It
pre-dates the seam; phase 0 (relay#1825) documented it in place rather than
changing it.

Why it matters

The broker treats delivery_ack as proof of observation. On receiving it the
runtime confirms the pending delivery, releases the withheld engine-facing
fleet ack, emits message_delivery_confirmed and marks the message read. For
a headless delivery, every one of those consequences currently fires on
spawn, so a child that crashes before reading stdin — or one whose input
never arrives — has already been reported as delivered.

The genuine observation already exists a few lines below: on a clean exit the
route emits delivery_verified with verification: "process_exit", which
is_observed() correctly counts as an observation. Non-clean exits emit
delivery_failed. So the accurate signal is present; the premature one just
arrives first and wins.

Why this is not a one-line removal

Deleting the early ack would withhold the fleet ack until the child exits. A
long-running headless child would then sit unacked past the broker's Steer ack
timeout, and the retry would re-inject a message whose first copy is still
running — exactly the double delivery STEER_ACK_SLACK
(crates/broker/src/broker/delivery_verification.rs) exists to prevent.

What a fix needs

A confirmation path for routes whose observation arrives late:

  1. A hand-over state that releases the sender without claiming observation, so
    the fleet ack is not blocked on a signal that legitimately takes minutes.
    SendStatus::HandedOver(HandoverState::HandedOver) is already this shape at
    the seam; the runtime's confirmation path has no equivalent.
  2. Retry suppression keyed on "this route has accepted responsibility and is
    still working", distinct from "no ack yet".
  3. Confirmation driven by delivery_verified{process_exit} / delivery_failed
    rather than by the spawn.

Test shape: a headless child that sleeps past the Steer ack timeout and then
exits cleanly must produce exactly one injection and exactly one confirmation.

Evidence

  • Premature ack: crates/broker/src/runtime/headless.rs (annotated in place).
  • Real observation: same file, clean-exit branch, verification: "process_exit".
  • Observation predicate: is_observed() in
    crates/broker/src/broker/delivery_verification.rs.
  • Found by adversarial review of relay#1825 (finding F7); every deterministic
    gate was green at the time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions