📦 Spring Kafka Orders

Two Spring Boot microservices that talk to each other only through Apache Kafka. An order is placed over REST, the inventory service reserves stock asynchronously, and the result flows back to update the order.

The project is intentionally small, but it covers the parts of Kafka that matter in real systems: message keys and partition ordering, consumer groups consuming partitions in parallel, retries, a dead letter topic, poison-pill messages and idempotent consumers — each one triggered by hand and covered by a test.

🛠 Tech Stack

Layer	Technology
Language	Java 17
Framework	Spring Boot 3.5, Spring for Apache Kafka
Message broker	Apache Kafka 4.3 (KRaft mode, no ZooKeeper)
Database	H2 (in-memory, one per service)
Monitoring	Kafka UI (kafbat)
Build / tests	Maven Wrapper, JUnit 5, Mockito, EmbeddedKafka
CI / runtime	GitHub Actions, Docker Compose

🏗 Architecture

flowchart LR
    client([curl / client])
    subgraph order-service :8081
        api[REST API] --> odb[(H2 orders)]
        srl[stock-results listener] --> odb
    end
    subgraph Kafka
        orders[[orders]]
        results[[stock-results]]
        dlt[[orders.DLT]]
    end
    subgraph inventory-service :8082
        ol[orders listener] --> idb[(H2 products +<br/>processed events)]
    end

    client -->|POST /orders| api
    api -->|OrderCreated<br/>key = orderId| orders
    orders --> ol
    ol -->|StockResult<br/>key = orderId| results
    ol -. failed after retries .-> dlt
    results --> srl

Loading

sequenceDiagram
    participant C as Client
    participant O as order-service
    participant K as Kafka
    participant I as inventory-service
    C->>O: POST /orders {productId, quantity}
    O->>O: save order (PENDING)
    O->>K: OrderCreated → "orders"
    O-->>C: 201 {status: PENDING}
    K->>I: OrderCreated
    I->>I: check eventId, lock product row, reserve stock (one transaction)
    I->>K: StockResult → "stock-results"
    K->>O: StockResult
    O->>O: status = CONFIRMED / REJECTED
    C->>O: GET /orders/{id}
    O-->>C: {status: CONFIRMED}

Loading

Topics

Topic	Producer	Consumer (group)	Payload
orders	order-service	inventory-service (inventory-service)	{eventId, orderId, productId, quantity}
stock-results	inventory-service	order-service (order-service)	{orderId, approved, reason}
orders.DLT	inventory-service	— (inspect in Kafka UI)	original record + kafka_dlt-* error headers

All topics have 3 partitions and are created on startup by the services (NewTopic beans). Both services consume with concurrency: 3 — one listener thread per partition. Events are plain JSON without Java type headers, so the services share a contract, not a library.

🚀 Running it

Option A — everything in Docker (only Docker needed):

docker compose --profile app up -d --build

Option B — services on the host (Java 17 + Docker; handy while changing code). Maven is not needed — the Maven Wrapper downloads it. On Windows, use mvnw.cmd instead of ./mvnw.

# 1. Kafka + Kafka UI
docker compose up -d

# 2. Each service in its own terminal
./mvnw -pl order-service spring-boot:run       # http://localhost:8081
./mvnw -pl inventory-service spring-boot:run   # http://localhost:8082

Kafka UI is at http://localhost:8080 — topics, partitions, messages and consumer group lag.

The inventory starts with KEYBOARD: 10, MOUSE: 5, MONITOR: 1.

Happy path

curl -X POST localhost:8081/orders -H "Content-Type: application/json" \
     -d '{"productId":"KEYBOARD","quantity":2}'
# {"id":1,"productId":"KEYBOARD","quantity":2,"status":"PENDING","reason":null}

curl localhost:8081/orders/1
# {"id":1,"productId":"KEYBOARD","quantity":2,"status":"CONFIRMED","reason":null}

curl localhost:8082/products
# [{"id":"KEYBOARD","stock":8},{"id":"MOUSE","stock":5},{"id":"MONITOR","stock":1}]

Rejections: {"productId":"MOUSE","quantity":50} → REJECTED / Insufficient stock, {"productId":"LAPTOP","quantity":1} → REJECTED / Unknown product.

Failure scenarios

These write straight to the orders topic with the Kafka CLI inside the container, the way a buggy or misbehaving producer would. The part before | is the message key.

produce() { docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders \
  --reader-property parse.key=true --reader-property key.separator='|'; }

Duplicate delivery (idempotency) — send an event whose eventId was already processed. The inventory logs Duplicate event same-id, returning the stored result: the result is published again, but the stock does not change a second time.

echo '7|{"eventId":"same-id","orderId":7,"productId":"KEYBOARD","quantity":1}' | produce
echo '7|{"eventId":"same-id","orderId":7,"productId":"KEYBOARD","quantity":1}' | produce
curl localhost:8082/products   # KEYBOARD went down by 1, not 2

Retry, then dead letter topic — an event that fails during processing is attempted 3 times, 1 second apart, and then published to orders.DLT. The consumer carries on with the next message.

echo '8|{"eventId":"bad-qty","orderId":8,"productId":"KEYBOARD","quantity":-1}' | produce

Poison pill — a payload that is not valid JSON cannot be fixed by retrying, so it goes to orders.DLT immediately, byte for byte as it was sent.

echo '9|this is not json' | produce

Open orders.DLT in Kafka UI to see both records and their kafka_dlt-exception-message headers.

🧪 Tests

./mvnw test

Test	Type	What it proves
OrderServiceTest	Unit	Order is saved as PENDING and published with key = orderId; results confirm/reject it
OrderKafkaIntegrationTest	EmbeddedKafka	POST /orders publishes plain JSON keyed by orderId; a result on stock-results updates the status
InventoryServiceTest	Unit	Stock is reserved or rejected correctly; a known eventId gets its stored result back; bad quantity throws
InventoryKafkaIntegrationTest	EmbeddedKafka	Duplicate event reserves stock once; 12 parallel orders for 5 items approve exactly 5; malformed JSON lands in the DLT; a failing event is retried and dead-lettered

The integration tests run the real listeners, error handler and database against an in-memory broker, so they need no Docker. The same ./mvnw verify runs on every push in GitHub Actions.

📁 Project Structure

spring-kafka-orders/
├── docker-compose.yml          Kafka (KRaft) + Kafka UI; both services under the "app" profile
├── Dockerfile                  multi-stage build, shared by both services (SERVICE build arg)
├── .github/workflows/ci.yml    runs all tests on push
├── pom.xml                     parent: Spring Boot, shared dependencies
├── order-service/
│   └── src/main/java/com/ardazeybek/order/
│       ├── OrderController     POST /orders, GET /orders/{id}
│       ├── OrderService        save + publish, apply stock results
│       ├── StockResultListener consumes "stock-results"
│       ├── KafkaConfig         topic definitions
│       └── event/              OrderCreatedEvent, StockResultEvent
└── inventory-service/
    └── src/main/java/com/ardazeybek/inventory/
        ├── OrderEventListener  consumes "orders", publishes "stock-results"
        ├── InventoryService    idempotency check + stock reservation
        ├── ProductRepository   row lock (SELECT ... FOR UPDATE)
        ├── KafkaConfig         topics, DefaultErrorHandler + DeadLetterPublishingRecoverer
        ├── Product, ProcessedEvent
        └── event/

No environment variables are required; everything runs on the defaults in each application.yml.

📚 Concepts in this project

• Topic & partition — a topic is split into partitions; partitions are the unit of parallelism. A consumer group can have at most as many active consumers as there are partitions.
• Keys and ordering — Kafka picks the partition from the message key, so all events of one order (key = orderId) go to the same partition. Ordering is guaranteed only within a partition.
• Consumer group & offsets — consumers sharing a group-id split the partitions between them; each group tracks its own committed offset per partition, so a restarted service resumes where it stopped.
• Parallel consumers need database locking — with concurrency: 3, orders for the same product can arrive on different partitions at the same moment. Kafka orders messages per key, not per database row, so the product row is locked with SELECT ... FOR UPDATE. Without it the race test sold 11 items out of 5.
• At-least-once delivery — a message can be delivered again (e.g. the service crashed after processing but before committing the offset). Consumers must be designed for that.
• Idempotent consumer — inventory stores every processed eventId, with the result it decided, in the same database transaction as the stock change. A redelivered event gets the stored result again without touching the stock, so a crash between committing and publishing heals itself. The order side needs no such table: setting a final status twice has the same effect as setting it once.
• Retry and dead letter topic — DefaultErrorHandler retries a failing record with a FixedBackOff, then DeadLetterPublishingRecoverer moves it to orders.DLT so one bad record cannot block a partition.
• Poison pills — ErrorHandlingDeserializer turns a payload that cannot be deserialized into an error the handler can route to the DLT, instead of the consumer failing on the same record forever.

⚖️ Known limitations

Kept out on purpose to keep the code small; each is a natural next step.

• Dual write: order-service saves to its database and then publishes to Kafka. If the process dies between the two, the order stays PENDING. The standard fix is the transactional outbox pattern.
• Retrying non-transient errors: the negative-quantity example is retried to make retries easy to see; in production, validation errors would be marked non-retryable and sent to the DLT directly.
• In-memory H2: data is lost on restart. Fine for a demo, not for anything else.

⚠️ Pitfalls

• Embedded Kafka fails with No enum constant ... GroupType.CLASSİC → the JVM runs with a Turkish locale and the broker upper-cases classic → surefire runs tests with -Duser.language=en (see pom.xml).
• Old events update the wrong order after a restart → H2 is wiped but Kafka keeps its topics and the new order ids start at 1 again → restart Kafka too: docker compose down && docker compose up -d.
• Dead-lettered records go to orders-dlt instead of orders.DLT → the recoverer's default topic name differs between versions → pass an explicit destination resolver (see inventory-service KafkaConfig).
• ./mvnw: Permission denied on Linux/macOS or in CI → a wrapper added from Windows is committed without the executable bit → git update-index --chmod=+x mvnw.
• docker exec kafka /opt/kafka/bin/... fails with C:/Program Files/Git/opt/... in Git Bash → Git Bash rewrites Unix paths → prefix the command with MSYS_NO_PATHCONV=1 (only for that command, it breaks mvnw).