Microservices · Intermediate
Timeouts, retries and circuit breakers
Why it matters
In a system of services, something is always failing somewhere. Without protection, a slow payment service makes every order request wait, threads and connections run out, and the whole site goes down: a cascading failure. Timeouts, careful retries and circuit breakers are the standard defences, and they come up in almost every backend system design interview.
The concept
Timeouts on every network call. Without one, a hung dependency holds your resources forever. Set them from measured latency (for example a little above the 99th percentile), not defaults.
Retries only for transient failures, only for idempotent operations, with exponential backoff and jitter, and a small maximum. Retries multiply load on a service that is already struggling; three layers each retrying three times turn one request into 27.
Circuit breakers track failures. After a threshold the circuit opens and calls fail immediately without touching the dependency. After a cool-down it goes half-open and lets one trial call through: success closes the circuit, failure opens it again.
Fallbacks: what to do when a call fails fast. Show cached data, queue the work for later, or return a clear error.
Bulkheads: separate connection pools per dependency, so one slow dependency cannot use up all of them.
Worked example: a breaker in front of a payment service
The payment service is down for its first four calls, and each failure is a slow timeout of 200 ms.
import time
class CircuitOpen(Exception):
pass
class CircuitBreaker:
"""Closed: calls pass. Open: calls fail fast. Half-open: one trial call decides."""
def __init__(self, failure_threshold: int = 3, reset_after: float = 1.0):
self.failure_threshold = failure_threshold
self.reset_after = reset_after
self.failures = 0
self.opened_at: float | None = None
@property
def state(self) -> str:
if self.opened_at is None:
return "closed"
if time.monotonic() - self.opened_at >= self.reset_after:
return "half-open"
return "open"
def call(self, fn, *args):
if self.state == "open":
raise CircuitOpen("failing fast")
try:
result = fn(*args)
except Exception:
self.failures += 1
if self.state == "half-open" or self.failures >= self.failure_threshold:
self.opened_at = time.monotonic()
raise
self.failures = 0
self.opened_at = None
return result
# A downstream payment service that is down for the first 4 calls, then recovers.
calls = {"n": 0}
def charge(amount: int) -> str:
calls["n"] += 1
time.sleep(0.2) # a slow failure: each timeout costs the caller 200 ms
if calls["n"] <= 4:
raise TimeoutError("payment service timed out")
return f"charged {amount}"
breaker = CircuitBreaker(failure_threshold=3, reset_after=1.0)
start = time.monotonic()
for i in range(1, 18):
t0 = time.monotonic()
try:
outcome = breaker.call(charge, 499)
except CircuitOpen:
outcome = "fast fail (circuit open)"
except TimeoutError as e:
outcome = f"error: {e}"
print(f"request {i:2}: {outcome:32} {1000 * (time.monotonic() - t0):4.0f} ms state={breaker.state}")
time.sleep(0.15)
print(f"downstream calls made: {calls['n']} of 17 requests")$ python3 breaker.py
request 1: error: payment service timed out 201 ms state=closed
request 2: error: payment service timed out 200 ms state=closed
request 3: error: payment service timed out 200 ms state=open
request 4: fast fail (circuit open) 0 ms state=open
request 5: fast fail (circuit open) 0 ms state=open
request 6: fast fail (circuit open) 0 ms state=open
request 7: fast fail (circuit open) 0 ms state=open
request 8: fast fail (circuit open) 0 ms state=open
request 9: fast fail (circuit open) 0 ms state=open
request 10: error: payment service timed out 206 ms state=open
request 11: fast fail (circuit open) 0 ms state=open
request 12: fast fail (circuit open) 0 ms state=open
request 13: fast fail (circuit open) 0 ms state=open
request 14: fast fail (circuit open) 0 ms state=open
request 15: fast fail (circuit open) 0 ms state=open
request 16: fast fail (circuit open) 0 ms state=open
request 17: charged 499 200 ms state=closed
downstream calls made: 5 of 17 requestsRead the timeline:
Requests 1 to 3 each wait 200 ms and fail. After the third failure the circuit opens.
Requests 4 to 9 fail in 0 ms without calling the payment service. Users get a fast answer (for example "payments are delayed, we will retry your order") and the struggling service gets breathing room.
After one second, request 10 is the half-open trial. The service is still down, so the circuit opens again.
Request 17 is the next trial. The service has recovered, the call succeeds and the circuit closes.
Only 5 of the 17 requests reached the payment service, so only 5 waited 200 ms; the other 12 were answered at once.
In production, use a maintained library rather than your own class: Resilience4j for Java, Polly for .NET, or the retry and breaker features of a service mesh such as Istio. The behaviour is the same.
Common mistakes
No timeout, or a 30-second default. Your service is only as fast as its slowest dependency.
Retrying everything. Retrying a 400 or a non-idempotent payment can make things worse.
Retries without backoff and jitter, which cause synchronised waves of load.
A breaker with no fallback. Failing fast is only useful if the caller knows what to do next.
Interview angle
"Service B is slow and service A's latency has tripled. What do you do?" Talk about timeouts, circuit breakers, bulkheads, fallbacks and monitoring. "How many retries would you configure, and where?" Retry at one layer only, with backoff and jitter, and an overall deadline propagated through the call chain.
Practice task
Add retries with exponential backoff (for example 50 ms, 100 ms, 200 ms) inside the breaker for timeouts only, and count how many calls reach the payment service.
Add a fallback that writes failed charges to a queue table and a job that retries them once the circuit closes.