Dinner Rush

An event-driven food-delivery ordering system: seven asynchronous Python/FastAPI services integrating over RabbitMQ, with Postgres as the durable authority and Redis as a non-authoritative fast path. Orders are driven through an explicit lifecycle (placed -> confirmed -> preparing -> ready -> out_for_delivery -> delivered) while fulfillment is handed to deliberately unreliable payment, restaurant, and courier downstreams. A React ops dashboard shows the pipeline live, a Next.js storefront places real customer orders through the same APIs, and a load generator produces controllable traffic.

Why it exists

The project demonstrates correctness under failure in distributed order processing. Downstreams here are slow, rate-limited, randomly failing, and able to fall over entirely - and the pipeline must still guarantee that no accepted order is lost, no effect (charge, kitchen ticket, courier dispatch) is applied twice, and parked work resumes on its own once the failure clears. Every guarantee is enforced by a mechanism you can point at in the code - a transaction, a unique constraint, a queue - not by hoping messages arrive exactly once. scripts/check.sh verifies the invariants against the live databases after a run.

Architecture

Each service is its own container with its own Postgres database (one Postgres instance, five databases: ingestion, orchestrator, payment, restaurant, courier). Services share no tables; the integration contract is locked in docs/CONTRACT.md and the shared library libs/drcommon.

flowchart LR
    subgraph clients [Traffic]
        LG[load-gen :8006]
        CW[customer-web :8090<br/>Next.js storefront]
    end

    subgraph broker [RabbitMQ]
        EV{{dinner.events<br/>topic}}
        WK{{dinner.work<br/>direct}}
        RT{{dinner.retry<br/>direct}}
        DLX{{dinner.dlx<br/>fanout}}
        QP[q.orch.placed]
        QA[q.order.advance]
        QR[q.order.advance.retry<br/>TTL bounce]
        QD[q.order.dlq]
        QDASH[q.dashboard.events]
    end

    ING[ingestion-svc :8001]
    ORCH[orchestrator-svc :8002<br/>state machine]

    subgraph downstreams [Flaky downstreams]
        PAY[payment-svc :8007<br/>POST /v1/payment_intents]
        REST[restaurant-svc :8003<br/>POST /fulfill]
        COUR[courier-svc :8004<br/>POST /fulfill]
    end

    PG[(Postgres<br/>orders, outbox,<br/>operations, dispatches)]
    RD[(Redis<br/>first-line dedup)]

    subgraph observability [Observability]
        DAPI[dashboard-api :8005]
        DWEB[dashboard-web :8080]
        PROM[Prometheus :9090]
        GRAF[Grafana :3000]
    end

    LG -- "POST /orders" --> ING
    CW -- "POST /orders" --> ING
    CW -- "GET /orders/{id}" --> ORCH
    ING -- "outbox: order.placed" --> EV
    EV -- "order.placed" --> QP --> ORCH
    ORCH -- "outbox: order.advance" --> WK
    WK -- "order.advance" --> QA --> ORCH
    ORCH -- "transient failure" --> RT --> QR
    QR -- "dead-letter after TTL" --> WK
    ORCH -- "attempts exhausted" --> DLX --> QD
    QD -- "auto-replay when healthy" --> WK
    ORCH -- "idempotent HTTP" --> PAY & REST & COUR
    ORCH -- "outbox: order.transition" --> EV
    EV -- "order.#" --> QDASH --> DAPI
    DAPI -- SSE --> DWEB
    ING & ORCH & PAY & REST & COUR --- PG
    ORCH --- RD
    PROM -- "scrape /metrics" --> ING & ORCH & downstreams
    GRAF --- PROM

Loading

Flow: ingestion accepts an order, commits it, and publishes order.placed (exchange dinner.events). The orchestrator seeds an order.advance command onto dinner.work and then drives the order forward one step per command. Entering confirmed authorizes payment, entering preparing calls the restaurant, entering out_for_delivery dispatches the courier (the STAGE_DOWNSTREAM map in libs/drcommon/drcommon/states.py); the other steps are internal think-time. Every transition is published as order.transition, which dashboard-api consumes (via q.dashboard.events) and streams to the browser over SSE.

Design decisions

Transactional outbox: state change and publish are atomic

A service never publishes directly from a request handler. The state change and the outgoing event are written to an outbox table in the same Postgres transaction; a background relay (libs/drcommon/drcommon/outbox.py) moves committed rows to RabbitMQ on a publisher-confirmed channel and stamps them published_at. This closes the publish-then-crash hole: the event cannot exist without the state change, and the state change cannot commit without the event eventually going out. Ingestion returns 2xx only after the order row and its order.placed event are committed together, so "accepted" means "durable". The relay claims rows with FOR UPDATE SKIP LOCKED, so multiple instances would not double-relay. Delivery is at-least-once by construction (a crash between publish and stamp re-sends), which is exactly why every consumer is idempotent.

Deterministic operation IDs: exactly-once effects over at-least-once delivery

Every lifecycle step has a deterministic operation id, "{order_id}:{from}->{to}" (op_id_for in drcommon), identical on any redelivery or retry. The durable dedup authority is a primary key on that id: the operations table in the orchestrator and a dispatches table in each downstream. A second attempt to apply the same step hits ON CONFLICT (op_id) DO NOTHING, loses the race, and becomes a no-op - so a courier is dispatched once even when the command that triggered it is delivered three times. The payment service takes the same op id as a Stripe-style Idempotency-Key header on POST /v1/payment_intents, so a retried authorization returns the original charge instead of billing twice. Redis adds a first-line SET NX check (drcommon/idempotency.py) to skip known-done work cheaply, but it is never authoritative: a wiped Redis degrades performance, not correctness. The claim is exactly-once effects, never exactly-once delivery.

Two retry tiers, circuit breakers, and a self-draining DLQ

Failure handling is layered so that transient blips are absorbed fast and real outages park work instead of losing it:

• In-process (fast): ResilientClient (drcommon/http_client.py) wraps every downstream call with a timeout, bounded retries with exponential backoff plus full jitter, and a per-downstream circuit breaker (closed -> open -> half-open with a single probe). 5xx and timeouts count toward the breaker; 429 backs off (honoring Retry-After) without tripping it, because rate limiting is flow control, not an outage; other 4xx (e.g. a declined card, 402) are permanent - never retried, the order moves to failed.
• Broker (durable): when in-process retries are exhausted or the breaker is open, the command is republished to dinner.retry, sits in q.order.advance.retry for a fixed x-message-ttl (default 5 s), and dead-letters back onto dinner.work. Fixed delay at this tier is deliberate: RabbitMQ per-message TTLs cause head-of-line blocking, so the exponential character lives in the HTTP tier. After BROKER_MAX_ATTEMPTS (default 6) the message parks in q.order.dlq via the dinner.dlx fanout exchange.
• Recovery: the orchestrator probes each downstream's /control endpoint every DLQ_REPLAY_INTERVAL_SECONDS (default 15) and, once all report healthy, replays the DLQ back onto the work queue automatically. Replayed traffic re-probes the half-open breaker, which closes on its own - recovery after an outage requires no operator action (there is also a manual POST /admin/replay-dlq). Retry and DLQ messages are republished with publisher confirms before the original is acked, so nothing is lost in the gap; unparseable poison messages go straight to the DLQ instead of hot-looping.

An explicit order state machine

The lifecycle is a real state machine (drcommon/states.py), not ad-hoc status strings: eight states (placed, confirmed, preparing, ready, out_for_delivery, delivered, cancelled, failed), an allowed-transition table, and IllegalTransition raised on anything else - jumping confirmed -> out_for_delivery is rejected in one place, server-side. Each order.advance command names the state it expects (data.from); if the order already moved past it, the command is acked as stale, which makes duplicate delivery harmless. The transition commit itself is guarded twice: the op-id claim and a compare-and-swap UPDATE ... WHERE state = $expected run in one transaction, so a concurrent delivery either applies the step or rolls back cleanly. Terminal states accept no further transitions, and every transition is recorded in an append-only transitions table for audit and the customer-facing tracker.

How to run

docker compose up --build

That brings up all seven backend services, both web UIs, RabbitMQ, Postgres, Redis, Prometheus, and Grafana (docker-compose.yml). Schema migrations run on boot and the load generator starts a baseline drip (3 orders/sec) automatically. No .env needed; copy .env.example to .env to override any tunable.

What	URL	Notes
Ops dashboard	http://localhost:8080	live pipeline view (SSE)
Customer storefront	http://localhost:8090	place and track a real order
Grafana	http://localhost:3000	anonymous viewer, admin/admin
Prometheus	http://localhost:9090	raw metrics
RabbitMQ management	http://localhost:15672	dinner / dinner
ingestion API	http://localhost:8001	POST /orders
orchestrator API	http://localhost:8002	/orders/{id}, /downstreams
restaurant / courier	http://localhost:8003 / 8004	/control to tune flakiness
load-gen	http://localhost:8006	/rush, /baseline, /stop
payment-svc	http://localhost:8007	Stripe-style, /control

Every service exposes /health and /metrics.

Exercise the failure paths (all also available as dashboard buttons):

curl -X POST http://localhost:8006/rush                 # ramp load to the rush rate (default 80/s target)
curl -X POST http://localhost:8003/control/down         # hard-kill the restaurant downstream
curl -X POST http://localhost:8003/control/up           # restore it; parked orders resume on their own
curl -X POST http://localhost:8002/admin/replay-dlq     # force an immediate DLQ replay

Verify the invariants (no lost orders, no double charge, no double dispatch) against the live databases:

./scripts/check.sh

Unit tests for the pure logic (state machine, breaker/retry, outbox, downstream simulator):

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e libs/drcommon pytest
.venv/bin/python -m pytest libs/drcommon/tests -q

Repo layout

libs/drcommon/      shared infra: broker topology, outbox, idempotency,
                    resilient HTTP client, state machine, downstream simulator
services/           ingestion, orchestrator, payment, restaurant, courier,
                    dashboard_api, load_gen (one container each)
web/                React ops dashboard (SSE, served by nginx)
customer-web/       Next.js storefront; stateless, talks only to ingestion
                    and the orchestrator
infra/              postgres init, prometheus config, grafana provisioning
docs/CONTRACT.md    the locked cross-service integration contract
scripts/check.sh    correctness invariants, run against the live system