How to use Avro and Schema Registry with Kafka

What and why

Kafka records are just bytes, so producers and consumers must agree on structure. Avro is a compact binary format with a schema, and a Schema Registry stores those schemas centrally and enforces compatibility as they evolve. Together they give typed, versioned messages that survive change. This tutorial wires them up.

Prerequisites

• A running Kafka cluster.
• A Schema Registry instance (Confluent or compatible).
• A Kafka client with Avro support, such as confluent-kafka[avro].

Steps

1. Run Schema Registry

Start the registry pointing at your cluster; it exposes a REST API, typically on port 8081. Confirm with curl localhost:8081/subjects.

2. Define an Avro schema

Describe the record in order.avsc:

{
  "type": "record", "name": "Order",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "amount", "type": "double"}
  ]
}

3. Produce serialized records

The Avro serializer registers the schema on first use and prefixes each message with its schema id:

from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
# configure serializer with the schema string, then produce
producer.produce(topic="orders", value=serializer({"id": 1, "amount": 9.99}, ctx))

4. Consume and deserialize

The consumer reads the schema id, fetches the schema from the registry, and decodes the bytes into a typed object. No schema is hard-coded in the consumer.

5. Evolve the schema

Add a field with a default so old and new readers coexist:

{"name": "currency", "type": "string", "default": "USD"}

Defaults let consumers on the old schema read new data and vice versa.

6. Set compatibility rules

Configure the subject's compatibility (e.g. BACKWARD) so the registry rejects breaking changes:

curl -X PUT localhost:8081/config/orders-value \
  -H 'Content-Type: application/json' -d '{"compatibility": "BACKWARD"}'

A non-defaulted required field addition will then be rejected.

Verification

Produce and consume a record and confirm the consumer receives a typed object. Register an evolved schema with a default and confirm it is accepted; attempt an incompatible change and confirm the registry rejects it under your compatibility setting.

Next Steps

Manage schemas as code in version control, use the Maven/Gradle plugins or CI to register schemas, and consider Protobuf or JSON Schema with the same registry if they fit your ecosystem better.