Retries fell 30% when I stopped retrying the wrong things

The background queue was retrying a lot. The dashboards were noisy, the same jobs reappeared for days, and the working assumption was that the system was flaky.

It was not especially flaky. It was retrying failures that were never going to succeed, and it was doing so in a way that made some of them worse.

A retry is a claim about the future

Retrying says: this failed for a reason that will stop being true. That is a real category — a connection reset, a lock timeout, a rate limit, a node restarting. Wait, try again, succeed.

It is not the only category. A malformed payload will be malformed on the tenth attempt. A record deleted between enqueue and execution stays deleted. A bug raising NoMethodError raises it identically forever.

Default configuration treats every exception the same, so both categories get the full retry schedule. The permanent failures occupy the queue for days, generate alerts that teach everyone to ignore alerts, and eventually fail anyway.

Splitting them was most of the win. Errors known to be transient keep their retries. Errors known to be permanent go straight to the dead set, where they are visible as something to fix rather than something to wait out.

Idempotency, or: the retry that already succeeded

The more interesting failures were jobs that did their work and *then* failed.

A job charges a card, sends a confirmation, and raises while writing a log line. The work happened. The job failed. It retries. Now the card is charged twice.

This is not a rare edge case, it is the normal shape of a job with more than one side effect, and no amount of retry tuning fixes it. The job has to be safe to run twice.

In practice that means a few habits. Do the idempotency check first — has this already been done? Use a natural key the caller supplies rather than one generated inside the job, so a second attempt computes the same key. Order side effects so the riskiest and least reversible happens last, after everything that could fail cheaply already has. And where the external service supports an idempotency key, use it, because that is exactly what it is for.

Assume every job runs at least twice. If that assumption is frightening, the job is the problem, not the retry policy.

Backoff that does not synchronise

A dependency wobbles and a thousand jobs fail together. With a fixed delay they all retry together, hit the recovering service simultaneously, knock it over again, and repeat. The retry policy has become a load generator.

Exponential backoff spreads them out over time. Jitter — a random component in the delay — spreads them out relative to each other, which is the part that actually breaks the synchronisation. Backoff without jitter still produces a thundering herd, just a slightly later one.

Poison messages and the queue behind them

One job that always fails and always retries will, given a large enough retry budget, consume a meaningful share of your workers doing nothing. A handful of them can noticeably degrade throughput for everything else while every individual metric looks fine.

A retry limit is not an admission of defeat, it is a circuit breaker. After N attempts, stop, park the job somewhere inspectable, and let a human look. The dead set is a feature.

Queues are a priority system, not a filing cabinet

A related problem surfaced while fixing the retries: everything shared one queue.

That means a burst of low-value work — a bulk export, a nightly recalculation — sits in front of something a user is actively waiting for. The system is not overloaded; it is badly ordered. Throughput looks fine and the experience is poor.

Separating by latency expectation rather than by feature fixed it. Work a person is waiting on goes in one queue. Work that must happen soon goes in another. Work that must happen eventually goes in a third. Workers are then weighted so the first is drained aggressively and the last is drained when there is room.

The useful question when placing a job is not "what feature is this part of" but "who notices if this is ten minutes late". Those two groupings almost never match.

Make the failure legible

The last piece was observability, and it was what made the rest stick.

Three numbers turned out to be worth watching. Queue latency — how long the oldest job has been waiting — is the one that actually correlates with user pain, far more than queue depth, because depth can be large and moving quickly or small and stuck. The size of the dead set, which should be small and should be triaged rather than periodically cleared. And retry counts broken down by job class, which is where a newly-broken integration shows up first.

Before this, "the queue is backed up" was a feeling. Afterwards it was a number with a threshold, and the threshold was set from what users could actually tolerate rather than what looked tidy on a graph.

What changed

Retries dropped about 30%, and — more usefully — the ones that remained were mostly real. The queue stopped being a place where known-broken work circulated, and alerts became worth reading again.

None of this made the system faster in a way a benchmark would show. It made it legible, which turned out to matter more: when the retry count went up afterwards, it meant something, and somebody investigated.