How Things Work

Circuit Breaker & Resilience

Keeping a distributed system up when dependencies fail — composing timeouts, retries with backoff, circuit breakers, bulkheads, and fallbacks into one resilience pipeline (Polly-style).

How It Works

In microservices, the network is unreliable and dependencies fail constantly, so resilience is built from layered policies rather than a single mechanism. Timeouts bound waits, retries absorb transient blips, circuit breakers stop cascades, bulkheads isolate resources, and fallbacks degrade gracefully. Libraries like Polly and Resilience4j let you compose these into one pipeline around every remote call.

Timeouts Bound the Wait

Every inter-service call needs a timeout. Without one, a slow dependency holds your request threads hostage indefinitely, and the pressure backs up until your own service falls over. A tight timeout converts 'hangs forever' into a fast, handleable failure.

Retries Handle Transient Failures

Many failures are momentary — a brief network blip or a node restarting. Retrying often succeeds. But retries must use exponential backoff with jitter; retrying immediately and in lockstep across all callers creates a 'retry storm' that hammers an already-struggling service.

Circuit Breaker Stops the Cascade

When a dependency keeps failing, the circuit breaker trips OPEN and fast-fails further calls without even trying — sparing the sick service and freeing your resources. After a cooldown it goes HALF-OPEN to probe recovery, then CLOSES when healthy. This is what prevents one failure from cascading across services.

Bulkheads & Fallbacks

A bulkhead isolates each dependency in its own resource pool (threads/connections), so one overloaded dependency can't exhaust the capacity needed for the others — like watertight compartments in a ship. And a fallback provides a graceful degraded response (cached or default data) when all else fails, so the user sees less, not an error.

Key Concepts

🔌Circuit Breaker

Trips OPEN after repeated failures to fast-fail and protect a struggling dependency, then probes (HALF-OPEN) to recover.

⏱️Timeout

An upper bound on how long to wait for a call, so a slow dependency can't tie up resources indefinitely.

🔁Retry + Backoff

Re-attempting transient failures with increasing, jittered delays to avoid synchronized retry storms.

🚢Bulkhead

Isolating each dependency in its own resource pool so one failure can't starve the rest.

🪂Fallback

A degraded but valid response (cached/default) returned when the real call can't succeed.

🧰Polly / Resilience4j

Libraries that compose these policies into a single pipeline so you don't hand-roll resilience.

Polly resilience pipeline

tsx

1// Polly (.NET) — compose resilience policies into one pipeline.
2var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
3    // Bulkhead: cap concurrent calls so this dependency can't starve others
4    .AddConcurrencyLimiter(permitLimit: 4)
5    // Timeout: never wait longer than 2s
6    .AddTimeout(TimeSpan.FromSeconds(2))
7    // Retry transient failures with exponential backoff + jitter
8    .AddRetry(new RetryStrategyOptions<HttpResponseMessage> {
9        MaxRetryAttempts = 3,
10        BackoffType = DelayBackoffType.Exponential,
11        UseJitter = true,
12    })
13    // Circuit breaker: trip after 50% failures, then fast-fail for 30s
14    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage> {
15        FailureRatio = 0.5, SamplingDuration = TimeSpan.FromSeconds(10),
16        BreakDuration = TimeSpan.FromSeconds(30),
17    })
18    // Fallback: degrade gracefully instead of throwing
19    .AddFallback(new FallbackStrategyOptions<HttpResponseMessage> {
20        FallbackAction = _ => Outcome.FromResultAsValueTask(CachedRecommendations()),
21    })
22    .Build();
23
24var result = await pipeline.ExecuteAsync(ct => recommendClient.GetAsync(ct));

💡

Why This Matters

Cascading failures and retry storms cause the worst distributed-systems outages, and the patterns that prevent them — circuit breakers, bulkheads, backoff, fallbacks — are core SRE practice and senior-interview staples. In microservices, where every call can fail, layering these policies is the difference between graceful degradation and a system-wide collapse.

Common Pitfalls

⚠No timeouts — a slow dependency exhausts your thread pool and takes the service down.

⚠Retries without backoff/jitter or a breaker — synchronized retries become a self-inflicted DDoS.

⚠Retrying non-idempotent operations — duplicates real side effects (double charges, double sends).

⚠A circuit breaker with no fallback — it opens, but users still get errors.

⚠Sharing one resource pool across all dependencies (no bulkhead), so one failure starves everything.

⚠Tuning breaker thresholds blindly — too sensitive trips on noise, too lax never protects.

Real-World Use Cases

1One Slow Dependency Takes Down the Whole Service

Scenario

A product page calls a recommendation service. When recommendations slow to a crawl, every product-page request blocks waiting on it, the request thread pool fills, and the entire product service stops responding — even for requests that don't need recommendations.

Problem

Without a timeout and bulkhead, a single non-critical, slow dependency consumed all shared request threads. There was no isolation and no fast-fail, so localized slowness became a full outage.

Solution

Wrap the recommendation call in a bulkhead (isolated thread pool), a tight timeout, a circuit breaker, and a fallback to cached/popular items. When recommendations degrade, the breaker opens, the page instantly falls back, and unrelated requests keep flowing.

💡

Takeaway: Isolate and time-bound every dependency, especially non-critical ones. Bulkhead + timeout + breaker + fallback turns 'the whole service is down' into 'one widget shows generic data'.

2Retries Amplified an Outage

Scenario

A downstream service had a brief hiccup. Every caller immediately retried three times in lockstep, tripling the load on the recovering service and knocking it back down — repeatedly.

Problem

Naive immediate retries (no backoff, no jitter, no breaker) turned a 2-second blip into a sustained outage. The retries became a self-inflicted DDoS just as the dependency tried to recover.

Solution

Add exponential backoff with jitter so retries spread out, cap attempts, and put a circuit breaker in front so callers stop retrying entirely once failures accumulate — giving the dependency breathing room to recover before traffic resumes.

💡

Takeaway: Retries without backoff, jitter, and a circuit breaker make outages worse. Combine them so a struggling dependency can actually recover instead of being repeatedly re-overwhelmed.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.