How Things Work

Event-Driven Architecture

Decoupling services by reacting to events instead of calling each other — fan-out, eventual consistency, and the transactional outbox + idempotency that make it reliable.

How It Works

Event-driven architecture has services communicate by emitting and reacting to events rather than calling each other directly. It delivers strong decoupling, easy fan-out, and resilience — but introduces eventual consistency and the dual-write problem. The transactional outbox pattern (plus idempotent consumers) is what makes event publishing reliable in practice.

React to Events, Don't Call

In event-driven architecture, a service announces that something happened ('OrderPlaced') by publishing an event to a bus, rather than calling other services directly. Interested services subscribe and react on their own. The producer doesn't know or care who consumes — extreme decoupling.

Fan-out & Extensibility

One event can trigger many independent reactions: email, analytics, shipping, loyalty. Adding a new reaction means adding a new subscriber — the producer never changes. This makes the system open to extension and keeps each consumer focused on its own concern.

The Dual-Write Problem

A service usually must both update its database AND publish an event. These are two separate systems, so there's no atomicity: a crash after the DB commit but before publishing loses the event, leaving the rest of the system unaware — silent inconsistency that's hard to detect.

Transactional Outbox + Idempotency

The fix: write the event into an 'outbox' table inside the same database transaction as the business change. A relay process then reads the outbox and publishes reliably, retrying until it succeeds. Because delivery is at-least-once, consumers must be idempotent — dedupe on the event id so a redelivery is a harmless no-op.

Key Concepts

⚡Event

An immutable fact that something happened ('OrderPlaced'). Producers emit; consumers react.

🔌Event Bus / Broker

The infrastructure (Kafka, RabbitMQ, Service Bus) that routes events from producers to subscribers.

📡Fan-out

One event delivered to many independent subscribers, each handling its own concern.

✂️Dual-Write Problem

Updating a DB and publishing an event aren't atomic; a crash between them loses the event or data.

📤Transactional Outbox

Persist the event in the same DB transaction as the data, then a relay publishes it reliably.

♻️Idempotency

Designing consumers so processing the same event twice has the same effect as once (dedupe on id).

Outbox pattern + idempotent consumer

tsx

1// ❌ Dual write: two systems, no atomicity — a crash loses the event.
2await db.orders.insert(order);          // committed
3await bus.publish("OrderPlaced", order); // 💥 crash here → event lost forever
4
5// ✅ Transactional Outbox: write data + event in ONE local transaction.
6await db.transaction(async (tx) => {
7  await tx.orders.insert(order);
8  await tx.outbox.insert({ type: "OrderPlaced", payload: order }); // same tx
9});  // both commit together — or neither does
10
11// A separate relay polls the outbox and publishes reliably (at-least-once):
12async function relay() {
13  for (const row of await db.outbox.findUnsent()) {
14    await bus.publish(row.type, row.payload);
15    await db.outbox.markSent(row.id);
16  }
17}
18
19// Consumers must be IDEMPOTENT — at-least-once means an event may arrive twice:
20bus.subscribe("OrderPlaced", async (e) => {
21  if (await alreadyProcessed(e.id)) return;   // dedupe on event id
22  await sendConfirmationEmail(e.payload);
23});

💡

Why This Matters

Event-driven design underpins most large microservice and streaming systems, and the dual-write/outbox problem is a favorite senior-interview question because so many teams get it wrong. Knowing how to publish events reliably (outbox), handle at-least-once delivery (idempotency), and reason about eventual consistency is essential for building correct distributed systems.

Common Pitfalls

⚠The dual-write problem — committing to the DB then publishing as separate steps loses events on a crash.

⚠Non-idempotent consumers — at-least-once delivery means duplicate processing without dedup.

⚠Treating eventual consistency as instant — UIs/queries that expect immediate effects show stale data.

⚠Event schema sprawl with no versioning — old consumers break when an event's shape changes.

⚠Hidden coupling through events (consumers depending on a producer's internal fields) — design events as contracts.

⚠No dead-letter handling or monitoring of consumer lag, so failures and backlogs go unnoticed.

Real-World Use Cases

1Adding Features Without Touching Checkout

Scenario

Every time the business wants a new post-order action — a loyalty point, a partner webhook, a fraud check — engineers have to modify and redeploy the already-risky checkout service, slowing delivery and increasing risk.

Problem

Checkout was directly calling each downstream concern, so it was tightly coupled to all of them. New requirements meant editing critical payment code, and a failing downstream call could break checkout itself.

Solution

Make checkout publish a single OrderPlaced event. Loyalty, partner-webhook, fraud, and analytics each become independent subscribers. New behavior is a new subscriber — checkout never changes, and a slow or failing subscriber can't break order placement.

💡

Takeaway: Event-driven fan-out makes a system open for extension: emit events and let new consumers subscribe. Core flows stay stable and decoupled while the business adds reactions freely.

2The Silently Lost Event

Scenario

Occasionally an order is created but no confirmation email is sent and analytics never records it. Logs show the order committed fine, but downstream services have no record of it.

Problem

The service committed the order to its database and then published the event as a separate step. Rare crashes (or broker hiccups) between the two lost the event — the classic dual-write inconsistency, invisible until customers complain.

Solution

Adopt the transactional outbox: insert the event into an outbox table within the same transaction as the order. A relay publishes outbox rows and marks them sent, retrying on failure. Consumers dedupe by event id. Now an event is published if and only if the order was saved — no loss, no phantom events.

💡

Takeaway: Never rely on a non-atomic DB-write-then-publish. The transactional outbox guarantees the event and the data commit together, and idempotent consumers make the unavoidable redeliveries safe.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.