Designing for partial failure

Retries feel like a safety net. A downstream call times out, you try again, and everything is fine. Simple. But in a distributed system at scale, naive retry logic is one of the most reliable ways to turn a five-minute degradation into a full-blown incident. I've watched it happen more than once.

Here's the situation: your service calls a payments API. The API is slow — not down, just slow — because it's under load. Your service times out after 30 seconds and retries. So does every other instance of your service. The payments API, which was at 80% capacity, is now receiving 2× the traffic it was before you started "helping." It tips over. Your retry logic caused the outage it was meant to prevent.

The retry storm problem

This failure mode is called a retry storm, and it's a textbook positive feedback loop. The more your upstream degrades, the more retries you generate; the more retries you generate, the more it degrades. The only way out is for something to shed load — which usually means your load balancer starts returning 502s across the board.

The standard prescription is exponential backoff with jitter. Instead of retrying immediately or at a fixed interval, you wait 2^n seconds (plus a random jitter term) between attempts. This spreads your retries across time, reducing the thundering herd effect.

The jitter part matters as much as the backoff. Without it, all your service instances — which failed at roughly the same time — will retry at roughly the same time. Full jitter randomises the retry window; decorrelated jitter does the same but is less prone to bimodal distributions. AWS's architecture blog has a good breakdown of both.

But backoff and jitter are table stakes. They reduce the blast radius of a retry storm; they don't make your retry logic correct.

Idempotency is not optional

Before you can safely retry anything, you need to answer one question: is this operation idempotent? That is, can you call it multiple times and get the same result as calling it once?

Read operations are trivially idempotent. Write operations rarely are, unless you design them to be. Charging a card twice is not idempotent. Creating a database row twice is not idempotent. Sending a webhook notification twice is not idempotent — at least not from the recipient's perspective.

The solution is idempotency keys: a unique identifier the client generates per logical operation and sends with every request. The server stores the key and, on a duplicate, returns the cached result without re-executing the side effects. Stripe's API does this well. Yours probably should too.

POST /v1/charges
Idempotency-Key: 7f3d9c1a-4b2e-4f8a-9d0c-1e2f3a4b5c6d
{
  "amount": 2000,
  "currency": "inr",
  "source": "tok_visa"
}

The key insight: idempotency keys shift the responsibility of "did this already happen?" from the network layer to the application layer, where you have the context to answer correctly.

Know what you're retrying

Not all errors are retryable. Categorising them upfront saves debugging time later:

Transient errors (503, 429, timeouts) — safe to retry with backoff
Client errors (400, 401, 403, 422) — retrying is pointless; the request is wrong
Server errors (500) — depends; a 500 on a non-idempotent endpoint is dangerous to retry

In practice, I only auto-retry on network timeouts, 503s, and 429s (with respect for the Retry-After header). Everything else surfaces to the caller.

Circuit breakers: retries at the system level

Even perfect retry logic fails if the downstream is completely unavailable for an extended period. This is where circuit breakers come in. The idea is simple: after a threshold of consecutive failures, stop sending requests entirely and fast-fail until a probe succeeds.

A circuit breaker has three states:

Closed — requests flow normally; failures are counted
Open — all requests fail immediately; a timer runs
Half-open — a single probe request is allowed through to test recovery

The half-open state is the subtlest part to implement correctly. You want exactly one probe, not a burst of them — which means you need coordination across instances if you're running horizontally. A shared Redis key with a TTL works well for this.

Timeouts are part of the contract

One thing I see omitted surprisingly often: every call that can be retried needs a tight timeout. Without one, a single slow request occupies a thread (or a connection) while the retry window extends indefinitely. In a high-concurrency service, this saturates your connection pool and makes the failure look much larger than it is.

My defaults: connect timeout 2s, read timeout equal to your p99 latency × 1.5. Tune these per dependency, not globally.

The goal of retry logic is not to hide failures. It's to tolerate transient ones gracefully while surfacing persistent ones quickly. If your retries are masking a real problem, that's a monitoring gap, not a success.

When I was building the diagnostic chatbot at Nielsen, we had a chain of five services, each with their own retry logic. The cumulative effect was a theoretical worst-case latency of several minutes before an error surfaced. We fixed it by introducing a deadline propagated through the call chain — one global timeout, not five independent ones. It's the distributed systems equivalent of keeping your total elapsed time below a threshold regardless of how many hops you make.

The boring truth about retry logic: done right, it's invisible. Done wrong, it's the first thing in the post-mortem.