Outbox Pattern for Engineers: Poll First, Add CDC Only When Needed

The outbox pattern ensures your business update and the event intent commit together, by writing both inside one database transaction; a separate relay later publishes the event. This collapses the dual-write problem into a single local commit, and the resulting delivery model is at-least-once. Start with outbox plus a polling relay, make your consumers idempotent, and only add complexity once you measure a real limit.

TL;DR:

• Consumers must deduplicate event IDs because a relay can crash after publishing but before marking a row sent; use an inbox when processing guarantees matter.
• Polling is simpler and suits modest throughput; switch to CDC only when measured load or latency becomes a constraint, since connectors add operational work.
• Prune or archive confirmed published rows on a schedule, retaining them only for replay or audit needs, and alert on table depth and relay lag.
• When event order matters, partition by entity key and attach a monotonically increasing sequence number so each aggregate’s events remain ordered downstream.

Table of Contents

• What the outbox pattern actually is
• Why use the outbox pattern
• How the outbox flow works end to end
• Choosing a relay: polling, CDC or managed pipes
• Operational hardening for production outbox systems
• What enterprise teams get right and wrong with outbox
• The real decision isn’t pattern choice, it’s discipline
• How we help you implement and scale the outbox pattern
• FAQ
• Sources

What the outbox pattern actually is

The outbox pattern defines a small append-only table that lives in the same database as your business data. When a service changes state, it writes the business row and an event row into that outbox table inside one transaction. Because both writes share a commit, there is no window where the database succeeds but the event is lost, which is exactly the failure that a separate publish call to a broker cannot rule out, as described in this dual-write explainer.

An outbox row typically carries:

• A unique event identifier for deduplication downstream.
• The aggregate or entity identifier the event relates to.
• A payload (often JSON) describing what changed.
• A status or published flag, and a created timestamp.

A relay process reads unpublished rows, sends them to the broker, then marks them as sent. The database transaction never waits on the broker.

Why use the outbox pattern

Dual writes fail silently: a service commits a database change, then crashes or times out before it can publish the matching event, and nobody notices until data drifts out of sync. The outbox pattern removes that scenario by making the event write part of the same commit as the business change. It also improves perceived API latency, because the request handler returns as soon as the local transaction commits, rather than waiting on a broker round trip.

The trade-offs are real rather than theoretical:

• Delivery becomes at-least-once, so duplicates are expected, not exceptional.
• You now operate an extra table, a relay process and its failure modes.
• Outbox rows accumulate and need active pruning once published.

Pro Tip: Treat the outbox table like a queue, not an audit log: prune published rows on a schedule rather than letting the table grow unbounded.

How the outbox flow works end to end

The canonical sequence runs in four steps:

1. The producer service opens a database transaction, updates its business tables, and inserts a matching row into the outbox table.
2. The transaction commits. At this point the business change and the event intent are durable together, or neither is.
3. A relay reads unpublished outbox rows, publishes each to the message broker, and marks it as sent once the broker acknowledges.
4. Consumers process the event, using the event identifier to detect and discard duplicates.

A minimal producer transaction looks like: begin transaction, insert or update the order row, insert an outbox row with the order’s event payload, commit. Nothing about the broker appears in that transaction at all.

The relay’s job is strictly read, publish, mark. If it crashes after publishing but before marking a row as sent, it will republish that row on restart. This is why the relay pattern yields at-least-once delivery rather than exactly-once, and why consumers must be built to tolerate duplicate messages through idempotent processing, as set out in this transactional outbox overview. At-least-once delivery requires idempotent consumer processing to tolerate duplicates, which is the one architectural fact every team adopting this pattern needs to design around from day one, according to the same outbox analysis.

Choosing a relay: polling, CDC or managed pipes

Two relay styles dominate production systems, and the choice between them is mostly about latency and operational appetite. A polling publisher queries the outbox table on an interval, claims a batch of rows, publishes them, and marks them sent. It is simple to build and reason about, and many implementations use a FOR UPDATE SKIP LOCKED clause so multiple relay instances can run in parallel without claiming the same rows twice, which supports horizontal scaling without duplicate publishing, as documented in this outbox pattern breakdown.

Change Data Capture tools such as Debezium tail the database’s transaction log directly and emit events for committed rows without any application-level polling, which removes query load from the database and lowers publish latency, though it adds a connector and replication slots to operate, per this polling versus CDC comparison.

• Start with polling when your throughput and latency needs are modest.
• Move to CDC once polling load or latency becomes a measured constraint, not a guess.
• Managed options, such as cloud stream pipes tied to a database’s native change stream, suit teams who want CDC’s latency without running connector infrastructure themselves.

Operational hardening for production outbox systems

Running an outbox pattern reliably at scale takes more than the happy path. A few practices separate stable implementations from fragile ones:

1. Make every consumer idempotent, using the event identifier as a deduplication key, and pair this with an inbox table on the consumer side when exactly-once-processing semantics matter to the business.
2. Preserve ordering where it matters by partitioning on an entity key, so events for the same aggregate land in the same partition or queue, and attach a monotonically increasing sequence number per entity.
3. Prune or archive published rows on a schedule, with a retention window sized to your replay and audit needs, rather than letting the outbox table grow indefinitely.
4. Monitor outbox depth, relay lag, and republish rates, and alert when unpublished rows age past an expected threshold.
5. Use row-locking patterns like FOR UPDATE SKIP LOCKED to let multiple relay workers run safely in parallel without a leader election step.

Pro Tip: Alert on outbox table growth, not just relay errors. A silently stalled relay looks healthy until the table triples in size.

What enterprise teams get right and wrong with outbox

Across regulated, high-volume environments in finance, healthcare and telecommunications, we have seen the outbox pattern solve the same recurring problem: a service updates a system of record, then fails to notify downstream systems, and nobody catches the drift until a reconciliation job flags it weeks later. Drawing on extensive engineering experience, teams typically introduce outbox plus a polling relay first, instrument publish latency and outbox depth, and only migrate to CDC once polling genuinely becomes the bottleneck.

The pattern earns its keep fastest in:

• Order and payment flows where a lost event means a financial reconciliation problem.
• Multi-system integrations where several downstream services depend on one source of truth staying consistent.
• Regulated environments where audit trails of every state change and every published event matter as much as the change itself.

The real decision isn’t pattern choice, it’s discipline

Most outbox write-ups treat the polling-versus-CDC decision as the hard part. It isn’t. The genuinely difficult part is the discipline to keep consumers idempotent, prune the outbox table, and monitor relay lag long after the initial build is done, and that is where most implementations quietly degrade.

Teams tend to over-invest in choosing the “right” relay upfront, when the pattern performs identically at modest throughput whether you poll every second or tail the transaction log. The conventional advice to “just use CDC for reliability” skips the fact that CDC introduces its own operational dependency: a connector that needs monitoring as closely as the outbox table it replaces.

If we had to name one priority, it would be this: build the pruning and monitoring story before you touch CDC. An outbox table nobody watches is a slow leak, not a safety net. Get that right with a boring polling relay first, and the migration decision becomes a measurement exercise rather than a guess.

— Polarising Team

How we help you implement and scale the outbox pattern

We design and build outbox implementations as part of our event-driven architecture work, starting with an architecture review that maps your dual-write risk before a single line of relay code gets written.

For teams already running a polling relay and weighing a move to CDC or managed stream pipes, our enterprise integration services cover that migration end to end, from connector setup through monitoring. If you are comparing prompt tooling for the engineering teams building these pipelines, Claude Opus 4.7’s benchmarks are worth a look alongside your platform choices. Where legacy systems sit underneath the dual-write problem in the first place, our legacy modernisation services address the root cause rather than just the symptom. Get in touch to scope a pilot through our AI Pods delivery model.

FAQ

What is the difference between the inbox and outbox patterns?

The outbox pattern guarantees a producer commits its business change and event intent together, then relays that event out at least once. The inbox pattern sits on the consumer side, recording processed event identifiers so duplicate deliveries from an at-least-once relay are detected and discarded rather than processed twice.

How does the transactional outbox pattern work in Spring Boot?

In a Spring Boot service, the application writes its business entity and an outbox row to the same database within one @Transactional method, so both commit atomically or neither does. A separate scheduled job or CDC connector then reads unpublished outbox rows and publishes them to a broker such as Kafka.

Does the outbox pattern guarantee exactly-once delivery?

No, the outbox pattern guarantees at-least-once delivery, because a relay can crash after publishing but before marking a row as sent, causing a republish. Consumers need to be idempotent, using the event identifier for deduplication, to achieve effectively-once processing from an at-least-once feed, as explained in this outbox and dual-write analysis.

Should I start with polling or CDC for my relay?

Start with a polling publisher: it is simpler to build, easier to debug, and sufficient for most moderate-throughput systems. Move to Change Data Capture only once polling introduces measurable database load or latency you can point to, as recommended in this dual-write and migration guide.

How do I stop the outbox table from growing forever?

Prune or archive outbox rows on a schedule once the relay confirms they are published, sized to whatever audit or replay window your system needs. Treat the table as an operational queue that gets cleaned continuously, not as a permanent log.

Sources

• Dual-write and the transactional outbox pattern — Igor Grieder
• Transactional outbox pattern — Software Craftsperson
• The dual-write problem and the outbox pattern — Richard Cooper

Recommended

• Custom Application Development Services
• AI-Assisted Legacy System Analysis
• Enterprise Data Integration Services