How to build a Kafka producer and consumer

What and why

Apache Kafka is a distributed log for streaming events between systems. Producers append records to topics; consumers read them at their own pace. Topics are split into partitions for parallelism, and consumer groups divide partitions among members. This tutorial builds both sides and explains the core mechanics.

Prerequisites

• A Kafka broker reachable on a known address.
• A Kafka client library for your language (this example uses Python's confluent-kafka).
• The Kafka CLI tools for topic management.

Steps

1. Start Kafka

For local development:

docker run -d --name kafka -p 9092:9092 apache/kafka:3.7.0

2. Create a topic

kafka-topics.sh --create --topic orders \
  --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

Partitions control parallelism; pick a count that matches expected consumer concurrency.

3. Produce messages

Use a key so related records land on the same partition and keep order:

from confluent_kafka import Producer
p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce("orders", key="customer-42", value='{"id":1}')
p.flush()

4. Consume in a group

from confluent_kafka import Consumer
c = Consumer({
  "bootstrap.servers": "localhost:9092",
  "group.id": "order-workers",
  "auto.offset.reset": "earliest"
})
c.subscribe(["orders"])
while True:
    msg = c.poll(1.0)
    if msg and not msg.error():
        print(msg.key(), msg.value())
        c.commit(msg)

5. Manage offsets

Offsets mark how far a group has read each partition. Committing after processing gives at-least-once delivery. Disable auto-commit and commit explicitly for control over when progress is recorded.

6. Scale with partitions

Add consumers to the same group to share partitions; throughput scales up to the partition count. More consumers than partitions leaves some idle.

Verification

Produce a few keyed messages and confirm the consumer prints them. Start a second consumer in the same group and observe partitions rebalance across both. Restart a consumer and confirm it resumes from the committed offset.

Next Steps

Add a schema registry with Avro or Protobuf, tune acks and retries for durability, and handle reprocessing with idempotent consumers or transactions for exactly-once semantics.