Transactional Outbox for Reliable Events
The transactional outbox writes each event into the same database transaction as its state change, eliminating the dual-write problem. A relay reliably forwards committed outbox rows to Kafka, and idempotent consumers handle at-least-once delivery on AWS.
Overview
A common bug in event-driven systems is the dual-write problem: a service updates its database and then publishes an event, but the two are not atomic, so a crash between them leaves state and events inconsistent. The transactional outbox pattern fixes this by writing the event into an outbox table within the same database transaction as the state change. A separate relay then reliably forwards outbox rows to the message broker.
Use this whenever a service must both change its own state and emit an event, and you cannot tolerate lost or phantom events.
Components
- Aurora PostgreSQL: the service database holding both business tables and the outbox table.
- Lambda: the relay that reads new outbox rows and publishes them.
- MSK (Kafka): the durable event log consumers subscribe to.
- DynamoDB Streams: an alternative outbox source for DynamoDB-backed services.
- EventBridge: routes published events to downstream targets.
- SQS: buffers delivery and provides dead-lettering.
- CloudWatch: monitors relay lag and publish failures.
Data Flow
Within one transaction, the service writes its business change and an outbox row describing the event. Because both commit or roll back together, there is no inconsistency. The relay (driven by CDC or polling the outbox) reads committed rows and publishes them to Kafka, marking each as sent. Consumers receive the event, and idempotent processing handles the at-least-once delivery the relay guarantees.
Scaling and Resilience
The relay scales with outbox volume and resumes from its last committed position after a failure, ensuring no event is lost. Publishing is at-least-once, so consumers dedupe with idempotency keys. SQS dead-letter queues capture poison messages. Because the outbox lives in the source transaction, correctness is preserved even under crashes and retries.
Security
The relay uses a least-privilege role scoped to read the outbox and publish to the broker. Kafka enforces TLS, authentication, and per-topic ACLs. Outbox payloads avoid embedding secrets. Encryption protects data at rest in Aurora and in the broker. Data contracts govern event schemas so consumers are not broken by changes.
Trade-offs and Alternatives
The outbox adds a table, a relay process, and cleanup of sent rows, plus the same eventual-consistency and idempotency concerns as any event system. For systems that can tolerate occasional inconsistency, simple post-commit publishing may be acceptable, though risky. Event sourcing subsumes the outbox by making the event log itself the source of truth. The outbox is the pragmatic choice for adding reliable eventing to a state-centric service.