Kafka Load Test

Load testing tool for Google Managed Kafka using the java-pubsublite library.

Features

• Publisher Load Test: Measure publish throughput and latency
• Subscriber Load Test: Measure receive throughput
• End-to-End Load Test: Measure end-to-end latency and verify message durability

Prerequisites

1. Java 11+ installed
2. Maven 3.6+ installed
3. Google Cloud authentication configured:

   gcloud auth application-default login

4. IAM permissions to access your Managed Kafka cluster:

   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member="user:YOUR_EMAIL" \
     --role="roles/managedkafka.client"

Building

mvn clean package -DskipTests

Usage

Publisher Load Test

Measures how fast messages can be published:

./run-publisher-test.sh \
  --bootstrap bootstrap.mycluster.us-central1.managedkafka.myproject.cloud.goog:9092 \
  --topic test-topic \
  --messages 100000 \
  --size 1024 \
  --threads 4

Options:

• -b, --bootstrap (required): Kafka bootstrap servers
• -t, --topic (required): Topic name
• -n, --messages: Number of messages (default: 10000)
• -s, --size: Message size in bytes (default: 1024)
• --threads: Publisher threads (default: 1)
• --ordering-key: Enable ordering with specified key
• --warmup: Warmup seconds (default: 5)
• --json: Output results in JSON format

Subscriber Load Test

Measures how fast messages can be consumed:

./run-subscriber-test.sh \
  --bootstrap bootstrap.mycluster.us-central1.managedkafka.myproject.cloud.goog:9092 \
  --topic test-topic \
  --duration 60 \
  --group-id my-consumer-group

Options:

• -b, --bootstrap (required): Kafka bootstrap servers
• -t, --topic (required): Topic name
• -d, --duration: Test duration in seconds (default: 60)
• -g, --group-id: Consumer group ID (default: auto-generated)
• --warmup: Warmup seconds (default: 5)
• --json: Output results in JSON format

End-to-End Load Test

Measures full pipeline latency and verifies message durability:

./run-e2e-test.sh \
  --bootstrap bootstrap.mycluster.us-central1.managedkafka.myproject.cloud.goog:9092 \
  --topic test-topic \
  --messages 10000 \
  --warmup 10

Options:

• -b, --bootstrap (required): Kafka bootstrap servers
• -t, --topic (required): Topic name
• -n, --messages: Number of messages (default: 10000)
• -s, --size: Message size in bytes (default: 1024)
• --threads: Publisher threads (default: 1)
• -g, --group-id: Consumer group ID (default: auto-generated)
• --ordering-key: Enable ordering with specified key
• --warmup: Warmup seconds (default: 5)
• --json: Output results in JSON format

Sample Output

============================================================
                    LOAD TEST RESULTS
============================================================

Duration:           60.0 s
Messages Sent:      100,000
Messages Received:  100,000
Message Loss:       0 (0.00%)

Throughput:
  Publisher:        1,667 msg/sec (1.63 MB/sec)
  Subscriber:       1,654 msg/sec (1.62 MB/sec)

Publish Latency:
  p50:              12.30 ms
  p95:              45.20 ms
  p99:              89.10 ms
  max:              234.50 ms

End-to-End Latency:
  p50:              18.70 ms
  p95:              67.30 ms
  p99:              142.80 ms
  max:              312.40 ms

============================================================

DURABILITY CHECK: PASSED - All messages received!

JSON Output

Use --json flag for machine-readable output:

./run-e2e-test.sh --bootstrap ... --topic ... --json

Output:

{
  "duration_seconds": 60.0,
  "messages_sent": 100000,
  "messages_received": 100000,
  "messages_lost": 0,
  "message_loss_rate": 0.0,
  "throughput": {
    "publish_msg_per_sec": 1667.0,
    "publish_mb_per_sec": 1.63,
    "receive_msg_per_sec": 1654.0,
    "receive_mb_per_sec": 1.62
  },
  "publish_latency": {
    "p50_ms": 12.3,
    "p95_ms": 45.2,
    "p99_ms": 89.1,
    "max_ms": 234.5
  },
  "end_to_end_latency": {
    "p50_ms": 18.7,
    "p95_ms": 67.3,
    "p99_ms": 142.8,
    "max_ms": 312.4
  }
}

Dependencies

This project uses the java-pubsublite library with the Managed Kafka backend.

License

Apache License 2.0