#Bengineering 🧐

Microservices can work for low-latency workloads if you design them that way.

Over the years, many people have told me, “Microservices don’t work when you need low latency; use a monolith.” I get it; for many, this advice comes from negative experiences.

When you break a monolith into multiple services, you’re replacing function calls with network calls. Those network calls introduce latency, and the more services you depend on, the slower the overall request becomes.

But does that really mean microservices can’t be used for low-latency systems? No, and I say that from experience.

I’ve spent a lot of time working on low-latency distributed systems, and I think the key is designing your architecture to compensate for the added overhead.

Caveat: Low-latency is subjective; designing a system for single-digit milliseconds vs. double- or triple-digit milliseconds will differ, since every additional digit changes your tolerance.

Locality Matters

One thing I really like about cell-based architecture is that it encourages locality, and locality is good for latency. We often talk about cells in the context of resiliency. They contain failures, reduce the blast radius, and help keep one problem from impacting the entire platform.

But they also provide a pattern for keeping communication between services local.

Suppose you are building a microservices-based platform with four or five services that each request must pass through. If you spread those services across multiple availability zones, clusters, or regions, you'll create significant latency.

Depending on the network topology, that traffic may cross additional switches, routers, firewalls, and other infrastructure. Crossing regions also introduces significant physical distance, which adds even more latency.

Now, let's imagine those same services running close together within the same Kubernetes cluster and availability zone.

Keeping those services local can significantly reduce the network infrastructure each request has to traverse. In the best case, the services might even be running on the same physical or virtual server.

Of course, Kubernetes clusters can span multiple availability zones, so simply being in the same cluster doesn't necessarily guarantee locality. If you're optimizing purely for network latency, keeping communicating services within the same availability zone is generally preferable. But that doesn't mean your entire platform should depend on a single availability zone.

Multiple cells that each independently run in an availability zone is one option. Another option is to leverage locality-based routing. You can still design for multi-AZ resiliency while preferring local communication.

Applying a low-latency lens to cell-based architecture lets you deliberately optimize how services communicate within each cell.

But locality comes with trade-offs. You need to manage how traffic is distributed, ensure each environment has enough capacity, and think carefully about data consistency. All of this adds operational complexity.

If low latency is important enough, those trade-offs may be worth it.

Stop Rebuilding Connections

Another common source of unnecessary latency is frequent connections.

When two microservices communicate hundreds or thousands of times per second, closing and re-establishing a new connection for every request is highly inefficient.

Every new TCP connection requires a three-way handshake: SYN, SYN-ACK, and ACK.

If you're using TLS, you've also got the additional negotiation required to establish a secure connection.

Each of these steps adds network round trips, and when you're trying to squeeze milliseconds out of a system, those costs matter.

In contrast, a persistent connection allows you to establish a connection once and reuse it for multiple requests.

While this is absolutely possible with HTTP/1.1, protocols like HTTP/2 take this further by allowing multiple concurrent requests over the same connection.

Instead of constantly opening and closing connections, keeping a pool of reusable connections open and using them efficiently can significantly reduce request latency.

But this performance improvement has a trade-off: load distribution.

Connection Reuse Requires Rethinking Load Distribution

Let’s imagine you’ve got ten instances of a service and your client microservice is configured to establish reusable connections based on what it needs.

With HTTP/2, your client might only need a single connection to handle a significant amount of traffic. But if that connection is pinned to one backend instance, all those requests would end up going to the same instance.

Meanwhile, the other nine instances are sitting idle.

When clients frequently establish new connections, layer 4 (network-level) load balancers have more opportunities to distribute those connections across your microservice instances. But when clients reuse a small number of long-lived connections, those same load balancers have fewer opportunities to redistribute traffic. Balancing connections isn't necessarily the same as balancing requests.

There are multiple options to approach load-distribution with reusable connections, such as having clients open a pool of connections and round-robin sending traffic across that pool. You could also leverage something like a service mesh, which does this for you.

This is a lesson I learned the hard way: early on, when embracing HTTP/2 in a microservices platform, I was unpleasantly surprised to find two or three microservice instances being hammered with traffic while seven others sat idle. But once we fixed it (we used a service mesh), our latency suddenly dropped as we distributed traffic better, letting each microservice run with less stress.

It was a great lesson, I just wish I didn’t have to learn it the hard way…

Avoid Unnecessary Waiting

This is a topic of a recent talk I gave at P99 Conf 2026.

HTTP/1.1 supports persistent connections, but it doesn't support multiplexing independent requests over a single connection the way HTTP/2 does.

In practice, an HTTP/1.1 connection typically handles one request at a time. If your client has more requests than available connections, some requests have to wait for a connection to become available.

HTTP/2 changes that by allowing multiple requests to be in flight concurrently over the same connection. This can significantly reduce the time requests spend waiting for available connections.

While HTTP/2 clients make asynchronous communication feel synchronous, there are always trade-offs with asynchronous communication. Managing timeouts, knowing when it’s safe to retry, etc. All topics I covered in the talk.

Managing asynchronous communication introduces complexity, but when you're trying to squeeze every millisecond out of a system, that complexity can be worth it.

Final Thoughts

Microservices will always introduce some communication overhead compared to the internal function calls of a monolith. But that doesn't necessarily make them a bad choice for low-latency workloads.

When I talk to folks who prefer monoliths over microservices, many have had bad experiences because their platforms weren't designed for low latency.

Maybe their services communicated across regions unnecessarily, they weren't reusing connections efficiently, or their load distribution created hot spots.

It's easy to understand why a monolith might seem like a better option in those situations. To play devil's advocate, solving these problems introduces complexity.

Microservices aren't free, and not every organization needs the independent scaling, ownership boundaries, and operational flexibility they provide. For many organizations, though, those benefits are valuable enough to justify the additional complexity.

My advice for anyone building a low-latency platform is simple: don't immediately discount microservices.

You can absolutely build low-latency systems with microservices. You just have to design for it, and make sure the complexity is worth it.