How Things Work

Message Queues & Pub/Sub

How asynchronous messaging decouples services, levels load, and survives failures — acknowledgements, at-least-once delivery, idempotency, and the dead-letter queue.

How It Works

A message queue is a durable buffer between services. Producers write messages and move on; consumers read them at their own pace. This decoupling smooths traffic spikes, lets services fail and recover independently, and is the backbone of event-driven architecture — but it trades synchronous certainty for at-least-once delivery and eventual consistency.

Decouple Producer from Consumer

Instead of calling a service directly, the producer drops a message on a queue and moves on. The consumer reads at its own pace. Producer and consumer no longer need to be online at the same time or run at the same speed — the queue absorbs the difference.

Buffer Bursts (Load Leveling)

When traffic spikes, messages pile up in the queue instead of crashing the consumer. The consumer drains the backlog steadily. This 'load leveling' turns a spiky workload into a smooth one and protects fragile downstream systems.

Acknowledge or Redeliver

A consumer must ACK a message after successfully processing it. If it crashes or fails before acking, the broker assumes the work didn't happen and redelivers — guaranteeing at-least-once delivery. This is why handlers must be idempotent: the same message may arrive twice.

Dead-Letter Queue

A 'poison' message that fails every time would loop forever. After a retry limit, the broker routes it to a Dead-Letter Queue — a side channel for failed messages that humans or alerts can inspect — so one bad message never blocks the rest.

Key Concepts

📬Queue vs Pub/Sub

A queue delivers each message to ONE consumer (work distribution). Pub/sub broadcasts each message to ALL subscribers (fan-out).

🔁At-Least-Once

The default guarantee: every message is delivered, but possibly more than once after a failure. Handlers must be idempotent.

🎯Exactly-Once

Each message effectively processed once — achieved with idempotency + dedup, not magic. True end-to-end exactly-once is expensive.

✅Acknowledgement

The consumer's signal that a message was handled. No ack → the broker redelivers it.

☠️Dead-Letter Queue

Where messages go after exhausting retries, so poison messages don't block the queue and can be inspected.

📈Backpressure

When consumers can't keep up, the queue depth grows — a signal to scale consumers or shed load.

Producer, consumer & idempotency

tsx

1// Producer — fire-and-forget, returns immediately
2await queue.send("orders", { orderId: 1234, total: 99.0 });
3return res.status(202).json({ status: "accepted" }); // don't block the user
4
5// Consumer — pull, process, then ACK
6queue.subscribe("orders", async (msg) => {
7  try {
8    await chargePayment(msg.body);       // the slow work
9    await msg.ack();                     // ✅ remove from queue
10  } catch (err) {
11    // No ack → broker redelivers later (AT-LEAST-ONCE).
12    // So the handler MUST be idempotent:
13    await msg.nack({ requeue: msg.deliveryCount < 3 });
14    // after N attempts, the broker routes it to the DLQ
15  }
16});
17
18// Idempotency: dedupe by a key so a redelivery is a no-op
19if (await seen(msg.id)) return msg.ack();

💡

Why This Matters

Queues (Kafka, RabbitMQ, SQS, Azure Service Bus) are everywhere in distributed systems — they enable async processing, load leveling, retries, and fan-out. Knowing delivery semantics, idempotency, ordering, and DLQs is essential for designing reliable backends and is a staple of system-design interviews.

Common Pitfalls

⚠Assuming exactly-once delivery — the default is at-least-once, so non-idempotent handlers cause duplicate side effects.

⚠No dead-letter queue: a single poison message retries forever and blocks the consumer.

⚠Expecting global ordering — most brokers only guarantee order within a partition/key, not across the whole topic.

⚠Ignoring queue depth (backpressure) — a growing backlog is an early warning to scale consumers.

⚠Doing the ack before the work, not after — a crash then silently loses the message.

Real-World Use Cases

1The Slow Checkout That Times Out

Scenario

Checkout calls payment, inventory, email, and analytics services synchronously. When the email provider is slow, the whole checkout request hangs and users see timeouts.

Problem

Synchronous chaining means the user's request is only as fast as the slowest downstream call, and any one failure fails the whole checkout. Coupling all four services to the request path is fragile.

Solution

Persist the order, then publish an 'OrderPlaced' event to a queue and return 202 immediately. Separate consumers handle payment, inventory, email, and analytics asynchronously. A slow email provider now just means a slightly delayed email, not a failed checkout.

💡

Takeaway: Queues let you return fast and do the slow, failure-prone work asynchronously. The user's critical path shrinks to 'save + enqueue', and downstream slowness no longer cascades into the request.

2The Double-Charged Customer

Scenario

A payment consumer crashes right after charging the card but before acking the message. The broker redelivers it, and the customer is charged twice.

Problem

At-least-once delivery means the same message can be processed more than once. A non-idempotent handler (charge on every delivery) turns a normal redelivery into a real-money bug.

Solution

Make the handler idempotent: record a unique idempotency key (e.g. the message/order id) inside the same transaction as the charge. On redelivery, detect the key already exists and skip straight to ack — the charge happens at most once even though delivery is at-least-once.

💡

Takeaway: At-least-once delivery is the norm, so idempotency is not optional. Dedup on a stable key and the inevitable redelivery becomes a harmless no-op instead of a duplicate side effect.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.