Transactional Outbox

The Transactional Outbox solves the dual-write problem: a service must both update its database and publish a message, but doing them as two separate operations is not atomic. If the database commit succeeds and the message broker call fails (or vice versa), the system is left inconsistent. The outbox makes the two happen reliably together.

How It Works

Instead of publishing to the broker directly, the service writes the message as a row in an outbox table within the same local database transaction as the business change. Because it is one transaction, either both the data change and the outbox row are committed or neither is. A separate process then reads new outbox rows and publishes them to the message broker, marking them sent.

The relay can poll the table or, better, use change data capture to tail the database transaction log and emit each new outbox row. Delivery is at-least-once, so consumers should be idempotent.

A message in the outbox usually carries a unique identifier, the destination topic, a payload, and a status column. The relay marks each row as sent only after the broker acknowledges it, so a crash mid-publish results in a retry rather than a loss. Because the same row may be published more than once, including the identifier in the message lets consumers deduplicate and remain effectively exactly-once at the application level.

When to Use It

Use the outbox whenever a service must update state and emit an event or command atomically, which is the common case in event-driven microservices and saga participants. It is the standard remedy for the dual-write problem when you cannot use a distributed transaction.

If losing or duplicating the occasional message is acceptable, or if your store and broker support a shared transaction, you may not need it.

It is the default choice for saga participants, where each local step must both change state and reliably emit the event that advances the saga to the next step.

Trade-offs

The pattern guarantees that no message is lost when the data change commits, but it introduces latency between commit and publish and requires a relay process to run and be monitored. Consumers must tolerate duplicates because delivery is at-least-once. The outbox table needs cleanup or archiving to avoid unbounded growth. Operationally, the outbox table needs an index on status and a periodic purge of delivered rows, and the relay should be monitored for lag so a stalled publisher is noticed before downstream consumers fall behind.

Related Patterns

Change Data Capture is the preferred mechanism for reading the outbox. The outbox is a building block for the Saga pattern, ensuring each step's events are published reliably. It complements Event Sourcing and the Database per Service model.

Example

An order service must save an order and publish OrderPlaced. In one transaction it inserts the order row and an outbox row containing the event. The transaction commits atomically. A Debezium connector tails the database log, sees the new outbox row, and publishes the event to Kafka, then the row is marked processed. If the service crashes between commit and publish, the relay still delivers the event on recovery.