The Transactional Outbox Pattern

The Dual-Write Problem

In a microservices architecture, a common requirement is: "When Service A saves an order to its database, it must notify Service B (Shipping) that the order exists." The naive implementation is a dual write:

# Anti-pattern: dual write
db.execute("INSERT INTO orders ...")   # Step 1
requests.post("http://shipping/notify", ...)  # Step 2

This is a distributed systems trap. If Step 1 succeeds and Step 2 fails (network timeout, service crash), the order exists in the database but Shipping never received the notification. The system is now inconsistent, and there is no automatic recovery path. Retrying Step 2 risks duplicate notifications. This is the dual-write problem.

The alternative — publishing to a message queue first, then writing to the database — has the same problem in reverse: the message is sent but the database write fails.

The Outbox Pattern: Atomic Writes via CDC

The Transactional Outbox Pattern solves this by exploiting the atomicity of a single database transaction. Instead of writing to the database AND sending a message, Service A writes to the database AND writes to an outbox table — both in the same transaction. Either both succeed or neither does.

BEGIN;

INSERT INTO orders (id, customer_id, status, total_cents)
VALUES (1001, 42, 'pending', 4999);

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (
  gen_random_uuid(),
  'Order',
  1001,
  'OrderPlaced',
  '{"order_id": 1001, "customer_id": 42, "total_cents": 4999}'
);

COMMIT;

The outbox table is just a regular database table. The transaction is local — no distributed coordination required.

Debezium monitors the outbox table via CDC. When a new row appears, Debezium reads it from the WAL and publishes the payload to the appropriate Kafka topic (derived from aggregate_type or event_type). Debezium even has a dedicated Outbox Event Router transformation that handles this routing automatically.

Guarantees and Trade-offs

Guarantee: At-least-once delivery. If Debezium crashes after reading the WAL but before committing the Kafka offset, it will re-read and re-publish the same event. Consumers must be idempotent — processing the same event twice must produce the same result as processing it once (e.g., use INSERT ... ON CONFLICT DO NOTHING keyed on the outbox id).

No guarantee of exactly-once without additional coordination (Kafka transactions + idempotent consumers). For most business events, at-least-once with idempotent consumers is sufficient and far simpler.

Trade-off: The outbox table grows indefinitely unless you periodically delete processed rows. Debezium's Outbox Event Router can be configured to delete rows after they are published, or you can run a scheduled cleanup job.