Kafka Debezium Postgres CDC Demo

Get started - Architecture - Development - Validation - Contributing - Runbook

---

This repository is a local, end-to-end change data capture sandbox. It sends transaction events through a Spring Boot producer, consumes and stores audit records in Postgres, then uses Debezium PostgreSQL CDC to publish database changes back into Kafka for inspection.

Project status

This is a development/demo repository for learning, validating, and evolving a Kafka + Debezium + Postgres CDC pipeline. It is configured for local infrastructure and non-secret demo credentials. It is not production hardened.

What it does

• Accepts transaction events over HTTP in transaction-service.
• Publishes transaction events to Kafka with JSON Schema serialization.
• Consumes transaction events in transaction-audit-service.
• Persists audit records to Postgres through JPA.
• Manages the audit table and CDC publication with Flyway migrations.
• Runs local Kafka, Schema Registry, Kafka Connect, Debezium, Postgres, and Redpanda Console with Docker Compose.
• Registers a Debezium PostgreSQL source connector for the audit table.
• Publishes CDC records to cdc.public.transaction_audit.
• Provides scripts for startup, sample event production, CDC verification, and bulk snapshot testing.

How it works

HTTP client
  |
  v
transaction-service
  |
  v
Kafka topic: transaction.audit.events.v1
  |
  v
transaction-audit-service
  |
  v
Postgres table: transaction_audit
  |
  v
Debezium PostgreSQL connector
  |
  v
Kafka topic: cdc.public.transaction_audit
  |
  v
Redpanda Console / kafka-console-consumer

The application event pipeline and the database CDC pipeline are deliberately separate. The producer and consumer own the business event flow, while Debezium observes committed database state through Postgres logical replication.

Get started

Prerequisites:

• Docker with Docker Compose
• Java 25
• Bash and curl

Start the local infrastructure:

./scripts/start-infra.sh

In a second terminal, start the audit consumer:

./scripts/run-audit-service.sh

In a third terminal, start the transaction API:

./scripts/run-transaction-service.sh

Generate demo transactions and verify that Debezium emits matching CDC records:

./scripts/produce-cdc-demo.sh

Useful local endpoints:

Service	URL
Transaction API	http://localhost:9081/api/transactions
Transaction API OpenAPI	http://localhost:9081/swagger-ui.html
Audit service OpenAPI	http://localhost:8080/swagger-ui.html
Schema Registry	http://localhost:8081
Kafka Connect	http://localhost:8083
Redpanda Console	http://localhost:8993

Stop the local infrastructure:

docker compose -f infra/docker-compose.yaml down

Remove local infrastructure state:

docker compose -f infra/docker-compose.yaml down -v

Local development

Copy the local environment template when you need shell-managed configuration:

cp .env.example .env

Run service tests:

cd transaction-service
./mvnw test

cd ../transaction-audit-service
./mvnw test

Run full Maven verification, including integration tests matched by each service POM:

cd transaction-service
./mvnw verify

cd ../transaction-audit-service
./mvnw verify

Validate Docker Compose:

docker compose -f infra/docker-compose.yaml config

For operational commands, troubleshooting, connector checks, and the bulk snapshot flow, see docs/RUNBOOK.md.

Configuration

Runtime defaults are intentionally local-only. Public defaults live in .env.example; service defaults live in each service's src/main/resources/application.yaml.

Variable	Default	Purpose
KAFKA_BOOTSTRAP_SERVERS	localhost:9092	Kafka bootstrap servers used by both services
KAFKA_SCHEMA_REGISTRY_URL	http://localhost:8081	Schema Registry URL
KAFKA_AUTO_REGISTER_SCHEMAS	true	Allows local schema registration during development
DB_HOST	localhost	Audit Postgres host
DB_PORT	5432	Audit Postgres port
DB_USERNAME	postgres	Local audit database user
DB_PASSWORD	postgres	Local audit database password

Project layout

transaction-service/        Spring Boot HTTP API and Kafka producer
transaction-audit-service/  Spring Boot Kafka consumer, JPA persistence, Flyway migrations
infra/                      Docker Compose stack and Debezium connector configuration
scripts/                    Local startup, demo, and data loading workflows
docs/                       Runbooks and maintainer documentation
.github/                    GitHub Actions, issue templates, and PR template

Key modules

Area	Description
Transaction API	Accepts and validates transaction event requests
Event publishing	Resolves Kafka topics and keys, then publishes transaction event envelopes
Audit consumer	Reads transaction events and persists audit records
Persistence	Stores audit records in Postgres through JPA
Migrations	Creates transaction_audit and transaction_audit_publication
Local Kafka	Provides broker, Schema Registry, topic initialization, and console inspection
Debezium CDC	Streams committed audit table changes to Kafka

Validation

The default validation path for pull requests is:

cd transaction-service && ./mvnw test
cd ../transaction-audit-service && ./mvnw test
docker compose -f ../infra/docker-compose.yaml config

Before merging changes that touch Kafka, persistence, Flyway, Docker Compose, or Debezium configuration, also run:

./scripts/start-infra.sh
./scripts/run-audit-service.sh
./scripts/run-transaction-service.sh
./scripts/produce-cdc-demo.sh

Documentation

Topic	Link
Runbook	docs/RUNBOOK.md
Contributing	CONTRIBUTING.md
Changelog	CHANGELOG.md
Security	SECURITY.md
Infrastructure	infra/README.md
Transaction service	transaction-service/README.md
Transaction service agent guide	transaction-service/AGENTS.md
Audit service agent guide	transaction-audit-service/AGENTS.md
Docker Compose	infra/docker-compose.yaml

Security notes

The checked-in credentials are local demo defaults, including postgres/postgres and debezium/debezium. Do not commit real .env files, cloud API keys, Kafka credentials, private keys, database dumps, or production configuration.