A trap I see people fall into when implementing resiliency is assuming a timeout means something failed.
Let’s imagine you’re buying some very expensive concert tickets online. Obviously, you’re trying to catch the latest Taylor Swift tour.
You’ve picked your seats, entered your information, and clicked checkout. Then the page times out…
Do you immediately go back and buy the tickets again? Probably not.
First, you check whether your card was charged. You look for a confirmation email. Maybe you check your account to see if the tickets are there.
Only after confirming nothing happened do you decide whether to try again.
Now imagine the same scenario, but you’re just looking at tour dates. When the page times out, do you refresh? Absolutely.
So why is one safe to unthinkingly retry and the other isn’t? Because one scenario has an operation that could be painful if performed twice.
A timeout on a checkout page doesn’t mean the operation failed; it just means you don’t know the outcome. And when you don’t know the outcome, a blind retry is a terrible idea.
The Blind Retry Resiliency Trap
Blind retries are a trap I see teams fall into when trying to make their systems more resilient. They treat all requests the same.
Maybe they do so by adopting a retry library or configuring automatic retries in their service mesh. Or maybe they ask an AI coding agent to add retry logic to every outbound request.
But some requests need more than a retry.
If you’re querying for data and the request times out, retrying might be perfectly fine. But what if you're posting a financial transaction? Just because your client timed out doesn’t mean the transaction failed.
Before implementing retries, ask yourself what happens if the first attempt actually succeeded.
Sometimes You Need to Know What Happened
For requests that perform operations with side effects, you need a way to manage that uncertainty. Maybe you query the downstream system to determine whether the transaction completed before retrying.
Or maybe the operation is idempotent. If every attempt includes a transaction ID and the downstream system recognizes duplicates before performing the operation again, great—retry.
The key is, the right approach depends on the use case and the implementation at hand.
Doing Nothing Can Be Just As Bad
It might be tempting to read my warnings above and think, “ok I just won’t retry operations that make changes.”
But that can also cause problems.
Imagine a point-of-sale system initiating a payment. Your request to the payment system times out, so you don’t know whether the payment succeeded. Meanwhile, the point-of-sale system also times out waiting for you and tells the end-user the transaction failed.
But what if the payment actually succeeded? The end-user may be charged for something they were told they didn’t purchase.
Doing nothing doesn’t always restore the system to a correct state.
Resiliency Requires Correctness
Resiliency isn’t just about making sure something eventually completes. It’s also about maintaining correctness when things fail in unexpected ways.
Sometimes that means retrying. Sometimes it means checking the current state before deciding what to do. And sometimes it requires a compensating action.
- If a payment fails, maybe you need to cancel a reservation.
- If a payment outcome is uncertain, maybe you need to put a shipment on hold.
- If fulfillment fails after the customer was charged, maybe you need to refund the transaction.
These decisions require something a generic retry library, and often a simple AI prompt, doesn’t understand: business context.
Final Thoughts
Retry libraries, exponential backoff, circuit breakers, and infrastructure-level resiliency features are incredibly useful.
But they don’t inherently understand what “correct” means for your business operations. That often requires business logic specifically designed to reconcile or compensate when an outcome is uncertain.
A timeout doesn’t mean an operation failed. It means you don’t know the outcome. Resilient systems need to handle uncertainty.