How to stream database changes with Debezium CDC

What and why

Change data capture (CDC) turns every insert, update, and delete in a database into an event stream. Debezium reads the database's transaction log and publishes ordered change events to Kafka, enabling cache updates, search indexing, replication, and analytics without polling. This tutorial captures changes from PostgreSQL.

Prerequisites

• A PostgreSQL instance you can reconfigure and restart.
• A Kafka cluster and a Kafka Connect runtime with the Debezium PostgreSQL connector installed.
• curl and a Kafka consumer for testing.

Steps

1. Enable logical replication

In postgresql.conf:

wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Restart PostgreSQL and grant the connector user REPLICATION privileges.

2. Run Kafka Connect with Debezium

Use the Debezium Connect image so the plugin is present:

docker run -d --name connect -p 8083:8083 \
  -e BOOTSTRAP_SERVERS=kafka:9092 \
  -e GROUP_ID=cdc -e CONFIG_STORAGE_TOPIC=connect_configs \
  -e OFFSET_STORAGE_TOPIC=connect_offsets \
  -e STATUS_STORAGE_TOPIC=connect_status \
  debezium/connect:2.6

3. Register the connector

POST a connector config to the Connect REST API:

curl -X POST localhost:8083/connectors -H 'Content-Type: application/json' -d '{
  "name": "pg-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres", "database.port": "5432",
    "database.user": "debezium", "database.password": "secret",
    "database.dbname": "app", "topic.prefix": "app",
    "table.include.list": "public.orders",
    "plugin.name": "pgoutput"
  }
}'

4. Inspect change topics

Debezium creates a topic per table, here app.public.orders. List topics to confirm it exists.

5. Read a change event

Consume the topic and update a row; each event has a before, after, and op field (c, u, d):

kafka-console-consumer --bootstrap-server kafka:9092 \
  --topic app.public.orders --from-beginning

6. Handle snapshots and offsets

On first start, Debezium snapshots existing rows, then streams new changes from the WAL. Connect stores offsets so restarts resume without loss. Monitor the replication slot so it does not retain WAL indefinitely if the connector stops.

Verification

Insert, update, and delete a row in orders; you should see matching c, u, and d events in the topic in commit order. curl localhost:8083/connectors/pg-cdc/status should report RUNNING.

Next Steps

Add a sink connector to land changes in a warehouse, use the outbox pattern for reliable event publishing, and configure schema history and SMTs to shape events.