Sometimes the biggest contributor to latency isn’t processing or network time. It’s waiting for work to start.
I’ve got a talk coming up at P99 Conf (Oct 21-22) about managing correctness in asynchronous systems. While thinking about how to explain the differences between synchronous and asynchronous protocols, I found a useful way to describe one performance benefit of asynchronous protocols: concurrency.
Today I want to walk through how sometimes making something faster has nothing to do with making the actual request faster. It can also be about simply reducing how long the request waits before it can even be sent.
The Hidden Latency of Waiting
Imagine you’re building a client that makes synchronous requests to another service as part of its request processing.
The client has a connection pool that allows it to make multiple requests concurrently. As long as the connection pool has enough capacity, everything works great.
Some work needs to make a request. The handler grabs an available connection from the pool. The request is sent immediately. The server processes it, responds, and completes the request.
Let’s say that took 10ms.
Great, your processing time is 10ms, right?
Maybe, but what happens if you’ve got a lot of work to do and those connections are all busy? When the next piece of work tries to grab an available connection from the pool, it has to wait. With synchronous protocols, the connection won’t return to the pool until the previous request finishes.
Meaning, our new work has to wait; maybe it’s a short wait, maybe it’s not.
Let’s say it waits for 5ms.
The request itself may only take 10ms, but end-to-end, that work took 15ms.
And if you don’t start measuring until after you’ve acquired a connection, your request metrics may still show it took only 10ms.
Sometimes You Don’t Need to Make the Request Faster
As engineers, when we start optimizing latency, we naturally look at the work being performed.
- Can we make the database query faster?
- Can we reduce serialization overhead?
- Can we improve the network path?
Those are all meaningful tuning options, but sometimes the simplest answer is to address the work that is sitting around waiting.
More Work In Flight
In the example above, we could eliminate 5ms just by giving work more paths to execute.
With a traditional synchronous request/response model, each worker may spend its time waiting for a response before it can move on to more work. Depending on the protocol, increasing concurrency might mean increasing the number of workers, connections, or available streams. An asynchronous model makes it easy to keep multiple operations in flight without tying each request to a worker/thread waiting for a response.
Asynchronous protocols can allow multiple requests to remain in flight concurrently without waiting for each previous response before sending the next request.
Even if an individual request still takes 10ms, multiple in-flight requests can eliminate the 5ms of waiting in our example.
Turning 15ms processing time back into 10ms.
Concurrency Doesn’t Eliminate Queuing Fully
Of course, increasing concurrency doesn’t give infinite capacity. Eventually, you’ll hit another bottleneck.
Increase client concurrency, and now the server worker pool becomes the bottleneck. Increase server capacity, and maybe the database connection pool becomes the bottleneck.
At sufficient load, removing one bottleneck often exposes the next.
That’s why the answer isn’t always “make everything asynchronous” or “increase the connection pool”. It’s important to understand where the work is waiting, and why. Only then do you know which bottleneck to address to meet your requirements.
Final Thoughts
I often post about the differences between synchronous and asynchronous protocols, and this is yet another post about how concurrency can improve performance.
But I think it’s important to call out that embracing asynchronous protocols has some significant downsides. It introduces many complexities around correlation, timeouts, retries, ordering, and correctness. Which is exactly what I’ll be talking about at P99 Conf.
But from a performance perspective, the lesson is simple: Sometimes the easiest way to speed up processing is to stop making it wait.