KIOU

KIOU is an Apache Kafka-inspired single-broker experiment focused on io_uring.

Motivation

I wanted to learn about io_uring, then generated two words like an clanker: "Kafka io_uring", deep-dived into Kafka's codebase, and this project is what came out of it.

Current Scope

Today KIOU has:

• a single broker runtime with separate broker and controller listeners
• disk-backed partition logs with rolled segments and sparse .index files
• io_uring on the append/write path (async writes)
• topic creation and metadata
• producer acks 0 and 1
• fetch with long polling
• consumer groups (currently limited to 1 consumer per group)
• committed offsets persisted by the broker
• small binaries for topics, producers, and consumers

Warning

Tests are currently AI-generated only. They have not been reviewed deeply yet, so treat them as smoke/regression coverage, not as strong proof that the system is correct.

Core Write Path

flowchart LR
  Producer[Produce request] --> BrokerThread

  subgraph BrokerThread["broker/client thread"]
    Append[PartitionLog::append]
    Reserve[prepare_append<br/>reserve offset + log position]
    Publish[publish in order from<br/>pending_appends_]
    Append --> Reserve
  end

  subgraph Ring["io_uring worker thread"]
    Submit[submit SQE]
    CQE[handle CQE]
    Fsync[handle roll fsync CQE]
    Submit --> CQE --> Fsync
  end

  subgraph Tails["segment tails"]
    ReservedTail[reserved]
    CompletedTail[completed]
    DurableTail[durable]
    ReservedTail --> CompletedTail --> DurableTail
  end

  Reserve -->|advance| ReservedTail
  Reserve -->|submit async write| Submit
  CQE -->|write finished| Publish
  Publish -->|advance| CompletedTail
  Fsync -->|advance| DurableTail
  CompletedTail --> Fetch[fetch reads up to here]

Loading

• One worker thread owns the ring. It does both submission and completion handling.
• reserved moves when the append is allocated.
• completed moves after the write CQE comes back and the append can be published in order.
• durable moves later, after roll-time fsync completes.
• Fetch reads from completed, not durable.

Quickstart

Build everything:

nix develop
meson setup build
meson compile -C build
meson test -C build

Start the broker:

./build/kiou

By default that starts:

• broker listener on 127.0.0.1:9092
• controller listener on 127.0.0.1:9093
• data dir at ./data

Create a topic from another terminal:

./bin/topics.sh --bootstrap-server 127.0.0.1:9093 --create --topic demo --partitions 1

Produce a few records:

printf 'one\ntwo\nthree\n' | ./bin/producer.sh --topic demo

Consume them with a group:

./bin/consumer.sh \
  --topic demo \
  --group demo-group \
  --session-timeout-ms 5000 \
  --from-beginning \
  --follow

Client Commands

Topic admin:

./bin/topics.sh --bootstrap-server 127.0.0.1:9093 --create --topic demo --partitions 3
./bin/topics.sh --bootstrap-server 127.0.0.1:9093 --describe --topic demo

Producer:

printf 'hello\nworld\n' | ./bin/producer.sh --topic demo --acks 1
printf 'fire-and-forget\n' | ./bin/producer.sh --topic demo --acks 0

Consumer:

./bin/consumer.sh --topic demo --group demo-group --session-timeout-ms 5000 --from-beginning
./bin/consumer.sh --topic demo --group demo-group --session-timeout-ms 5000 --follow

Runtime Configuration

Optional environment variables:

• KIOU_PORT
• KIOU_CONTROLLER_HOST
• KIOU_CONTROLLER_PORT
• KIOU_DATA_DIR

Dev Notes

Formatting:

./scripts/format.sh --fix
./scripts/format.sh --check

clangd / LSP:

nix develop
meson setup build
ln -sf build/compile_commands.json compile_commands.json