I’ve shared a lot about the benefits of gRPC, and most of that focuses on what makes it fast and reliable. One of the biggest benefits has nothing to do with performance. It’s maintainability and consistency.

But that maintainability and consistency rely heavily on how well you manage the Protobuf contract.

Protobuf Contracts Can Be Mismanaged

Protobuf and gRPC naturally gravitate toward well-managed contracts, but like anything, we can still overcomplicate or mismanage things.

The most common mistake I’ve seen is every application keeping its own copy of the .proto file and generating code from it independently.

When changes happen, they have to be made across every copy of the .proto file. Those copies can drift very quickly.

If nobody introduces a conflicting change, it might work for a while. But conflicts happen eventually.

Imagine two teams independently modify their copies of the .proto file.

One adds:

string email = 3;

Another adds:

string phone_number = 3;

Now both have assigned different meanings to the same field number. And because both assigned a string to field 3, this doesn’t fail in predictable ways.

One application could send an email address, while another is expecting a phone number. Hopefully, you find that drift in testing, not in production.

The easiest way to avoid this scenario is to maintain a single source of truth for the contract.

One Contract, One Source of Truth

At sufficient scale, you may need a schema registry or platform that manages Protobuf/gRPC contracts. But if you only have a few gRPC contracts to manage, I have a pretty simple solution that works well.

First, keep the .proto files in one source-controlled home. That’s the source of truth.

Make any updates there via pull requests. Changes get reviewed and tracked, and because everyone is modifying the same .proto file, it’s much easier to catch conflicts.

Second, generate code from that contract. Bonus points if you automate this as part of the CI process. Then commit and publish the generated code as versioned libraries for the languages you support.

With this approach, applications don’t maintain their own copies of the .proto definition or generate code independently. They import a library/package.

You don’t have to worry about which version of the .proto file was used. You don’t have to worry about the protoc version or which plugin options were used. It becomes a normal software dependency.

All you need to worry about is which version you are using.

Contract changes follow a pretty simple process:

Pull Request -> Review -> Merge -> Code Generation -> Publish -> Dependency Update -> Profit.

Consistency Becomes the Easy Path

The best part of this approach is that consistency becomes the easy path.

If five services consume the same gRPC service, five teams don’t need to understand how to generate the client correctly.

They don’t need to manage a .proto file. They don’t need to install the right version of protoc. They import a library/package.

This approach also makes contract adoption visible.

If version v1.8.0 adds a new capability, you can easily see which applications have upgraded and which are still using older versions. This approach turns contract management into dependency management. And every team has to manage software dependencies anyway.

You Still Need to Manage Compatibility

While this approach makes contract and implementation management much easier, it’s not foolproof.

Breaking changes can still be introduced. Consumers can still sit on old versions.

Adding new fields is generally straightforward. But removing or changing existing fields requires coordination (and should be avoided if possible).

It’s also important to recognize that producers/servers and consumers/clients evolve independently. If you only added a field or two, and the consumer doesn’t need it, running an older version might be fine. But over time, letting consumers run significantly different versions can cause functionality to drift across applications.

Good dependency management and automation can make this much easier to manage, though.

Overall, if you keep backward compatibility in mind, this approach works well. Still, if you let those practices slip, you’ll find yourself coordinating version changes across multiple applications, which is time-consuming and often challenging.

Final Thoughts

I’ve seen entire projects lose months because two teams understood the same contract differently.

Protobuf/gRPC doesn’t make that impossible. But if you structure things well, it can make it much harder. And that is one of my favorite things about Protobuf/gRPC.

It’s fast and efficient, which is great. But a well-managed .proto contract makes integration between systems consistent and much easier to maintain. Don't treat a contract as documentation; treat it as code. Code that can be versioned, distributed, and managed like any other dependency.

Message queues and brokers are great solutions for many use cases. They can provide excellent resiliency and strong guarantees that a message gets where it needs to go.

But I find those delivery guarantees can sometimes create a false sense of resiliency. Because delivering a message and completing the work that message represents are two different things.

Peeling Back the Everything Breaks Onion

A message broker might reliably deliver a message to a consumer. Great. But what if that consumer receives the message and fails while processing it?

What if it processes the message but fails while calling a downstream service? What if the consumer succeeds, but the overall business process doesn’t?

Message delivery isn’t the same thing as business-process completion. For best-effort messages, maybe that’s fine. For mission-critical work, I generally want some mechanism to know whether that work actually completed.

Tracking Completion

One way is to track the operation and expect an application-level acknowledgment.

Let’s explore a simplified approach:

A producer publishes:

ProcessPayment:123456789

At the same time, it records in a local DB:

123456789 -> PENDING

The consumer processes the message and, once the work completes, publishes:

PaymentProcessed:123456789

When the producer receives that message, it updates the local DB:

123456789 -> COMPLETE

Now instead of simply knowing the original message was delivered, we have a record of the work actually completing.

What If the Completion Never Arrives?

If operation 123456789 stays PENDING, it’s tempting to assume the work failed. But that’s not necessarily true.

Maybe the consumer failed. Maybe a downstream dependency failed. Or maybe the producer failed to process the acknowledgment. Either way, we haven’t confirmed operation completion, and what happens next depends on the use case.

You may be able to retry, or you may need to query the downstream system and validate the outcome. You might need to wake someone up, or you do nothing because the message was best-effort anyway.

The important part is that the uncertainty is detectable and visible, which means you can decide what to do about it.

Final Thoughts

Of course, many message brokers already have acknowledgment mechanisms. Those are useful, and depending on how they are used, might provide similar assurances.

But many implementations only tell you a message has been delivered or consumed. If the consumer hands the work off to another system, or starts a larger business process, the broker doesn’t understand whether the work ultimately completed.

That uncertainty can be problematic for some use cases.

Of course, adding this additional layer of certainty comes with its own complexities. It adds state management, more components in the flow, and more room for failure.

But for mission-critical work, guarantees are essential. A broker can provide strong guarantees around message delivery, but your application is ultimately responsible for determining and exposing whether the work actually happened.

Some should be rules. Others should be guidelines.

Explicitly calling out the difference gives teams enough structure to maintain the architectural vision, while still giving engineers room to make good decisions.

Why Architecture Principles Matter

I’ve helped envision, modernize, transform, and fix many platforms throughout my career. One of the most valuable things you can do early in that process is to define a set of architectural principles.

Large transformations take time. Along the way, different engineers and teams will make tons of technical decisions.

A good set of principles helps ground those decisions and ensures the platform you end up with still resembles the platform you originally envisioned.

But the platform doesn’t stop evolving when the project ends.

New requirements appear, new technologies emerge, constraints change, and individual engineers/architects come and go. Your principles need to provide structure without preventing the architecture from evolving.

That’s why I like to separate them into two categories: Rules and Guidelines.

Rules: These Must Remain True

An architecture rule defines something that must remain true about the platform.

For example:

“Data must be encrypted at rest and in transit.”

Or:

“Services within one cell cannot directly communicate with services in another cell.”

These aren’t preferences; they are architectural constraints that engineers must design with rather than decide whether to follow.

Guidelines: This Is How We Prefer to Build

Architecture guidelines provide direction while leaving room for engineers to apply contextual judgment.

For example:

“Services should be stateless whenever practical.”

There may be good reasons for a service to maintain state, and that’s ok.

The guideline establishes the preferred approach, but engineers can deviate when the problem context justifies it.

Why You Need Both

I’ve run into problems when architecture principles are treated entirely as guidelines or entirely as rules.

If everything is a guideline, eventually nothing is a rule.

Someone may have a legitimate reason to deviate. Another team will see that implementation and copy it.

Now the exception becomes precedent, and over time the original principle slowly disappears.

But making everything a rule creates the opposite problem.

Eventually, you’ll encounter a scenario you didn’t anticipate when the principles were written. Now you need an exception.

Once you establish a process for exceptions, that process can quickly become the path of least resistance. The rule slowly becomes a guideline anyway.

Or your platform becomes so rigid with rules that engineers can't meet changing needs.

Explicitly defining which principles are rules and which are guidelines gives you the best of both worlds. Rules protect the architectural characteristics you aren’t willing to compromise. Guidelines give engineers room to make contextual decisions when circumstances change.

This Matters Even More With Coding Agents

Smaller engineering teams have sometimes gotten away without explicitly documenting principles. Engineers learn them through code reviews, architecture discussions, and tribal knowledge.

Coding agents don’t have years of tribal knowledge. They only understand the design constraints they can access.

If we want coding agents to make increasingly larger implementation decisions, we need to explicitly tell them where they have engineering freedom and where they don’t.

Writing down your architecture principles and making them easily accessible to engineers and agents alike is becoming even more important.

Tell an engineer who understands your architecture to retrieve account data. That engineer will instinctively know they should leverage the owning service's API rather than accessing its database directly.

Ask a coding agent to fetch the same data; it might try to connect to the database directly, breaking the “microservices must not share databases” architecture rule.

Ask a coding agent to do something without context, and you’ll get exactly what you asked for, whether it violates your architecture principles or not.

If the agent doesn’t know the rules, it can’t follow them.

Final Thoughts

Architecture principles have always been valuable. Writing them down is more important now than ever.

But don’t just define how you want your platform to look at the end. Be explicit about which principles are rules and which are guidelines.

Good architecture principles should tell engineers and agents where they have freedom to make decisions and where they don’t.

Sharing objects without proper synchronization across threads in Java. Timing issues across multiple workers in Python. Race conditions and deadlocked goroutines in Go.

Concurrency can be difficult to get right regardless of the programming language.

But for high-performance and high-scale systems, it’s often a necessary complexity.

Concurrency lets our software process multiple tasks at the same time. But the moment you do, you introduce an entirely new set of problems.

Some of those problems can hide for years, like landmines waiting for the right time.

Shared Data Gets Complicated

Let’s take a simple example: a shared map.

If multiple threads or goroutines are reading from the map concurrently, things may be perfectly fine. But what happens when one starts modifying the map while another is reading or also writing? Now you need synchronization.

In Go, for example, unsynchronized concurrent access to a map where at least one goroutine is writing can result in runtime failures or data races. So you need something to coordinate access: a mutex, channels, or a design that avoids sharing objects altogether.

It might sound obvious to use something like a mutex to solve this problem, but sometimes, these bugs don’t exist when the code is originally written.

Maybe the map was initially scoped to a single goroutine. Six months later, someone else adds a background process that updates it periodically. Suddenly, something that was perfectly safe is not.

The code didn’t fundamentally change; the way it was accessed did.

Timing Creates Another Problem

Shared data isn’t the only concurrency challenge. Sometimes it’s ordering.

Imagine you need to process a sequence of related messages. Message 1 arrives, then message 2.

If you distribute those messages across multiple threads, goroutines, or even servers, arrival order no longer guarantees processing order.

Maybe the worker processing Message 2 has less of a work backlog. Maybe the server it lands on is slightly faster.

Maybe Message 1 gets delayed by a database call.

What if Message 2 finishes first? Does it matter? Maybe, maybe not.

But if Message 2 depends on something Message 1 was supposed to do, you now have a correctness problem.

In cases like this, related work needs to be serialized or partitioned so that operations that require ordering follow the same funnel.

It Works Until It Doesn’t

What makes concurrency bugs nasty is that your code can be wrong, but still work.

Take the shared map example. Run one request at a time during testing, and you’ll never see this problem. Run a few requests concurrently, and you still might not see it.

The timing has to be just right.

You could have a concurrency bug sitting in production for years without knowing it exists. Then traffic increases, or CPU utilization changes, or a dependency gets just a bit slower. Something that has worked millions of times before starts failing.

Try again, and everything is fine. Not because you fixed something; timing was just different.

Concurrency bugs are notoriously hard to reproduce and diagnose for this reason.

So How Do You Find Them?

Functional tests aren't enough; you need tests and tooling that deliberately create or detect concurrency behavior.

In Go, one of the easiest things you can do is run your tests with the race detector by adding the -race flag.

It can identify concurrent memory access that might trigger data races. Linters can also help catch other concurrency mistakes.

But one technique I’ve found surprisingly effective is performance testing.

Normally we think of performance tests as a way to answer how many requests a service can handle, or latency at a certain throughput. But performance tests are also a great way to find correctness problems.

Why? Because they create concurrency.

Thousands or millions of operations increase the chances of those rare timing conditions occurring.

Running functional performance (go run -race) or benchmark (go test -race) tests with Go’s data race detector won’t produce trustworthy performance numbers. Still, it can be incredibly effective at exposing concurrency bugs.

Final Thoughts

Concurrency is difficult to get right, even in languages like Go that give us excellent tools to work with it.

But for many systems, it’s a necessary complexity. You can’t always avoid concurrency. You need to build guardrails around it.

Use data race detection, static analysis, and test systems with many requests at once. Put these checks into your build pipelines so engineers don’t have to remember to run them.

Because concurrency bugs aren't always obvious at first, they can sit there quietly for years, waiting for just the right timing to create havoc.

When thinking about performance, especially for low-latency and high-scale applications, I don’t believe in “good is good enough.” I think it’s important to leave yourself room to grow and keep looking for improvements.

But there is a point where gaining performance requires significantly more complexity. And sometimes that tradeoff isn’t worth it.

I recently had a conversation with a team considering gRPC to send events from one system to another.

We are talking about very high volumes of relatively small requests.

They had two important requirements:

• Events needed to move quickly
• The sender needed to know each event was successfully received

Given their other constraints, traditional HTTP APIs and message brokers weren’t a great fit.

But I posed an interesting question: should they use unary gRPC calls or bidirectional streams?

🚀 The Faster Option

Their initial plan was to implement bidirectional streaming.

The client could continuously send events across a stream, and the server could asynchronously send acknowledgments as it accepted events. It’s not a bad design.

From a performance perspective, it’s a great design.

But there is a catch.

With this approach, the application needs to track what was sent, what is still in-flight, what was acknowledged, and what needs to be retried. And that’s if all goes well. What happens if the stream disconnects? Which events were processed, and which were lost? How do you avoid duplicates?

All solvable problems, but they all add complexity to the application.

🤔 What If You Don’t Need The Fastest Option?

Now let’s compare with unary gRPC.

From an application perspective, you send a request, and you get a response. Success or failure is tied directly to that request.

You don’t need an in-flight cache mapping acknowledgments back to events because the request/response lifecycle already does that for you.

It’s a much simpler implementation. And importantly, unary gRPC is still fast.

You still get the benefits of gRPC: Protocol Buffers, persistent HTTP/2 connections, and multiplexing requests across those connections. But you do give up some performance compared with a well-designed streaming implementation.

But the real question is, did they need that additional performance?

⚖️ There Are Also Operational Tradeoffs

Streaming changes how traffic is distributed. A long-lived stream is established with a particular backend and remains there for the life of that connection.

If you have three clients, six servers, and each client opens one stream, you could end up actively sending traffic to only three servers or less.

It’s solvable: open multiple streams, build a stream pool, balance the work across them, and manage the connection lifecycle during failures. Solvable, but more complexity for the team to implement.

In contrast, with unary requests, infrastructure like a service mesh can distribute requests across available backends without the application needing to manage pools, connection lifecycles, etc.

Send a request to the service mesh; it will distribute it across the six servers. Scaling and failure handling become far less complex.

📈 Performance Has Diminishing Returns

The team will test unary operations for their application.

If unary operations meet their throughput and latency requirements with plenty of headroom for growth, they face an interesting choice.

Bidirectional streaming could give them even more performance. But what are they buying with that additional performance?

If they need it, the complexity is justified. If they don’t, they are taking on additional development, testing, and operational complexity for performance they won't use.

That’s a bad tradeoff.

🧐 Final Thoughts

I dislike the “good is good enough” mentality. Continuously improving your platforms is an essential practice for any platform that cares about performance.

But performance has a cost. Few of us are building systems just to be fast. We’re also building them to be reliable and maintainable. And eventually, someone has to operate that system in the middle of the night when something unexpected happens.

If the simpler solution meets your needs while leaving enough headroom to scale for tomorrow, it may be the best option. The question to ask yourself is: what is the right balance of simplicity and performance?

I was recently putting together a talk that walks through some architecture and engineering decisions I’ve made over the last eight-plus years when something stood out to me.

Without context, many of those decisions probably look like over-engineering.

🤔 An Example: Operational Toggles

In several mission-critical payment applications, we follow a simple philosophy:

If a dependency isn’t required to process customer traffic, we should be able to turn it off.

For example, metrics.

Metrics are important, but they aren’t more important than processing transactions. If something goes wrong with our metrics implementation, we want to be able to turn off metrics while continuing to process traffic.

The same applies to out-of-band management messages and data updates. We typically use message brokers for this, and if there is an issue with that integration or a bug in the message handlers, we want the ability to stop consuming those messages without taking down the application.

Why?

Because we want fine-grained operational control over our platform. A non-critical capability shouldn’t be able to take down the critical path.

🤯 Isn’t That Over-Engineering?

Maybe.

These operational toggles add complexity to the system: flags that need to be managed, tests that need to be written and executed, and operational procedures for failures that might never happen.

For many applications, it’s probably not worth it.

If a metrics library causes a memory leak in an application that does batch processing, restarting to release the memory is perfectly acceptable.

If a message handler crashes an application that is automatically restarted, it might be ok depending on the application. Not great, but if requests are retried and nothing truly fails, no harm, no foul.

Not every application needs the complexity of these operational toggles.

But when an application crash could cause impact, when the system is mission-critical, and every failure is potentially impactful, the equation changes.

🧠 Context Determines Everything

For the batch and non-critical application examples, these operational toggles could easily be considered over-engineering.

But for a mission-critical, always-on, large-scale platform, when a situation occurs and these operational toggles are used, they become incredibly valuable.

The above examples are not hypothetical.

A change in metrics causes a spike in memory usage. An unsafe type assertion in a data sync handler from a message broker causes an application to crash.

These are real scenarios, scenarios that I’ve seen in mission-critical systems. Because these operational toggles existed, these scenarios were not disastrous or even inconvenient.

These operational toggles let us turn off the cause of an issue. More importantly, they gave us time. Time to understand what happened, build and properly test a fix, and validate that the fix actually solved the root problem.

All without rushing a change to production while the platform was unstable.

🧐 Final Thoughts

If you look at the way we use operational toggles, it could easily feel like over-engineering. But building critical systems means planning for failure scenarios even if they are unlikely.

Sometimes that additional complexity might look unnecessary. But when something goes wrong, you’re thankful it’s there.

Before coding agents, creating more code cost time and effort because engineers had to write it.

That naturally encouraged us to reuse code as much as possible: libraries, frameworks, keeping implementations concise, and thinking twice before rebuilding functionality that already existed.

Coding agents have dramatically reduced that natural cost.

Today, generating another implementation costs almost nothing. But you still have to maintain it.

🤖 Just Let the Agent Build It

There’s a growing mindset that because agents can generate code quickly, there’s less reason to worry about reuse.

Why spend time finding an existing library when an agent can recreate the functionality in seconds? And initially, that might feel faster, better.

But let’s say you have six applications that all need to validate email addresses against your internal requirements.

Instead of creating a common package, you let your coding agent implement email validation directly in each application. As long as the prompt is the same, it’s fine, right? Not necessarily. You can easily end up with six implementations, all slightly different.

Then you find a bug.

🪳 Creation is Cheap, Maintenance Isn’t

Fixing the application with the bug is easy.

Tell the agent about the bug; it fixes the implementation, maybe it adds some tests, done.

What about those five other implementations? Do they also have this bug?

Maybe, maybe not. Because each application has a different implementation, they may have the same bug or different bugs.

You’ve essentially found yourself in a situation where, depending on the application, an email might work, or it might not. Maybe it works in three of the six, or two. What happens if that email works for a registration page, but not the login page?

Not a great user experience.

🧱 Reuse Still Matters

Compare that to using a shared package. You write it once, everyone uses the same package.

You find a bug, fix it once, add tests, and release a new version.

Every instance using that new version has that bug fixed.

Now your suite of applications is consistent, and consistency matters most. Rejecting a valid email isn’t great, but if you’re consistent about it, that's a much better experience than it sometimes working and sometimes not.

That consistency is much easier to manage if the implementation is in one reusable library.

⚖️ Don’t Go DRY Crazy

Don’t use the above example to go to the other extreme, where every similar three lines of code turns into a shared library. Breaking everything into its own little shared library creates overhead costs, too.

It’s important to strike a balance between reusable and duplicative.

If you’re writing something with complex or nuanced behavior that’s needed across multiple applications, or more importantly, needs to behave consistently across them, a shared library is probably a good idea.

If it’s simple, situationally unique, and unlikely to change, don’t bother.

🧐 Final Thoughts

The cost of creating code has changed significantly. But the cost of managing it hasn’t.

Testing it, patching it, securing it, and keeping implementations consistent all become more expensive as the amount of code you own grows.

AI has made it incredibly cheap to create more code. Don’t confuse cheap to generate with cheap to own.

I’m a believer that, as an engineer, you should be able to run your software locally. But I hear it often: “We can’t run locally because of some reason.”

Sometimes it’s valid. There are architectures and platforms out there that prevent running locally. But more often than not, when it comes to backend distributed systems, it’s a design or implementation decision that nobody has challenged.

Why This Matters

Everyone these days is focused on speeding up software delivery. Today, that generally means coding agents, AI tooling, and code generation.

But writing code faster doesn’t matter much if validating a one-line change takes an hour.

The speed of software delivery is often dictated by validation, not code generation.

The Problem with Shared Dev Environments

Many teams still rely on shared development environments as their primary way of validating changes.

That process usually looks something like this:

• Make a change
• Commit the change
• Push the branch
• Wait for a build
• Wait for a deployment
• Run tests

If everything goes well, you can validate your change. If a mistake was made, or something doesn’t work right, you have to fix it and start over.

The result is a slow feedback loop, which can sometimes influence behavior.

The more friction there is to validate a change, the larger that change becomes. If testing takes a long time, you will naturally focus on making sure everything is perfect before spending time validating.

Local Validation Influences Good Behavior

Compare the above process to one that uses a locally running service.

• Make a change
• Build
• Start the service
• Run tests

The process is dramatically shorter and, more importantly, faster. This encourages engineers to make smaller changes, test more frequently, iterate their implementations, and find mistakes earlier.

This all results in better software.

The Excuses

“My service depends on too many other services.”

Run the services you own locally. Mock the services you don’t.

“I need a database or message broker.”

Run those locally too. There’s very likely a Docker container for each of them.

“We use cloud services.”

This one can be harder, but emulators exist for some services. Other services might have a Dockerized open-source alternative. For those that don’t, you could mock them.

There will always be exceptions: mainframes, specialized hardware, or unique managed services.

But I find many teams jump straight to “we can’t run locally” before they’ve seriously explored how they could.

Running Local Doesn’t Mean Everything

You don’t need to recreate your entire production environment on a laptop, though if you can, go for it.

The goal is to validate your change without depending on a shared or non-local environment that takes forever to test against.

Local development isn’t about recreating production. It’s about creating enough of the environment to validate your change. If that means turning off some functionality or creating mock services, it might be worth it.

Why This Matters Even More Now

Validating changes quickly is becoming more important as teams adopt coding agents. An agent can generate code in seconds.

But if every iteration with your coding agent requires committing, pushing, building, deploying, and waiting, the feedback loop becomes the bottleneck.

Final Thoughts

Agents make mistakes. Humans make mistakes.

The faster you can validate a change, the faster you can correct those mistakes. That’s why local execution isn’t just a convenience.

It’s one of the most effective ways to shorten the feedback loop.

One of the simplest reliability rules I’ve learned is this: Every dependency is another way for your service to fail.

Why Dependencies Matter

Every service has dependencies.

• Databases
• Caches
• Configuration services
• Secrets managers
• Logging pipelines
• Tracing backends

All of these dependencies can fail, and when they do, the typical service will fail with them.

The more dependencies a service has, the more failures it inherits. Every dependency adds features, but it also adds failure modes.

Reliability is often about deciding which dependencies are actually worth the tradeoff.

Why Edge Systems Tend to Be Dependency-Light

I recently wrote about how systems closest to the customer carry the greatest responsibility for availability.

This is one reason edge systems tend to be dependency-light.

Load balancers, API gateways, and routers are responsible for availability.

Every dependency added to these systems creates another opportunity to take down the entire platform. So they tend to avoid dependencies whenever possible.

Eliminating Dependencies Isn’t Always Necessary

Sometimes removing a dependency entirely isn’t practical.

A better question is:

• What happens if the dependency disappears?
• Can the service continue operating?
• Can it use cached data?
• Can it fall back to a last-known-good configuration?
• Can it degrade gracefully?

If the answer is yes, you’ve significantly improved reliability even though the dependency still exists.

A Real-World Example

Take Envoy Proxy. Envoy can receive configuration from an xDS service. But it doesn’t call xDS for every request.

Instead:

• Configuration is fetched periodically
• Stored in memory
• Used locally during request processing

If the xDS service becomes unavailable, Envoy continues routing traffic using its last known configuration. The dependency still exists, but request processing no longer depends on its availability.

That’s a very different reliability model.

Don’t Make Observability a Hard Dependency

One of the most common mistakes I see is making observability a required dependency.

If your logging or tracing backend becomes unavailable, should customer traffic stop flowing? No.

In most cases, observability should be a best effort.

Use asynchronous logging, buffering, and truncation policies so customer traffic continues to flow even when your observability platform is experiencing issues.

Operational visibility is important, but customer availability is more important.

Final Thoughts

You’ll never eliminate every dependency. But you can eliminate unnecessary ones.

And for the remaining dependencies, you can design your service to survive failures.

The most reliable services aren’t dependency-free. They’re designed to survive dependency failures.

You’ve all heard the advice: “Avoid caching because caching is difficult to get right.”

I agree with part of that statement. Caching can absolutely be difficult. But I think the reality is more nuanced.

The difficulty level of caching depends heavily on the type of data you are caching.

Not All Data Is Equal

When engineers talk about caching complexity, they’re usually thinking about data that changes frequently and requires strong consistency.

That’s the hardest kind of data to cache. Because you need to answer some difficult questions.

• How do you invalidate the cache?
• How do you handle updates?
• What happens when the cache population fails?
• How stale is too stale?

These problems are real, but they don’t apply to all data.

Frequently Updated Data

Some data changes constantly and has strict accuracy requirements. Account balances, inventory counts, active orders, and similar records. This is where caching is difficult.

Every stale read has a consequence, and in some cases, it’s not worth the complexity.

Infrequently Updated Data

Some data changes occasionally, once an hour, once a day, or even longer. Configuration data, product catalogs, country codes, and similar records.

In these cases, a small amount of staleness might be fine. If data changes hourly and your cache refreshes every few minutes, that’s often a reasonable tradeoff. The less frequently data changes, the less sensitive you become to cache staleness.

Caching is much easier with infrequently updated data.

Immutable Data

Immutable data is the easiest to cache. Because it never changes. Ledger entries, event records, receipts, and other immutable records.

Once immutable data is loaded into a cache, cache invalidation largely disappears as a problem. Cache complexity for immutable data is more about balancing performance and hit-miss ratios.

Most of the time, immutable data is an ideal candidate for caching.

The Real Question

Instead of asking, “Should I cache?” Ask yourself, “How often does this data change? How accurate does it need to be?”

The answers will give you an idea of how complex caching will be.

Final Thoughts

Caching isn’t inherently hard.

Maintaining correctness for frequently changing data is hard.

The more stable the data, the less complex the caching needs to be.

Like most things in system design, you can’t just take all-or-none answers like “always cache” or “never cache.” It’s important to understand the trade-offs, constraints, and consistency requirements of the data you’re working with.

The closer a component is to the customer, the greater its responsibility for keeping the entire platform available, even when everything behind it is having a bad day.

Not All Services Carry the Same Reliability Burden

Let’s consider a typical platform.

Customer -> Load Balancer -> API Gateway -> Orchestrator -> Microservices -> Database

Every layer has a different job. But every layer also has a different level of responsibility for resiliency.

As you move toward the customer, that responsibility increases.

Deep Services Focus on Business Logic

At the deepest layers of the platform, services are usually focused on business capabilities.

They process orders, transfer money, manage inventory, and store data.

These services often have databases, business rules, stateful operations, and multiple dependencies.

Resiliency matters, but it’s often focused on correctness.

If a database call fails:

• Should the transaction roll back?
• Should the service fail over?
• Should a compensating transaction occur?

These services are primarily concerned with business outcomes.

The Middle Layers Absorb Failures

Move up a layer, and you often find orchestrators and workflow services. These components coordinate work across multiple services.

If one service fails, the orchestrator may retry, execute fallback logic, trigger compensating actions, or roll back a workflow. Their job is not just executing business logic, it’s ensuring execution succeeds despite failures.

The Edge Exists to Protect Everything Behind It

At the edge, things change.

Load balancers and API gateways are often stateless, dependency-light, highly available, and extremely fast.

Why?

Because their primary responsibility is availability. Everything behind them is allowed to fail, and they absorb as much of that failure as possible.

They:

• Route around failures
• Shed load
• Fail over traffic
• Enforce timeouts
• Apply retries
• Protect backend systems

The edge isn’t just resilient for itself. It’s resilient on behalf of everything behind it.

Final Thoughts

The deepest services in a platform should be focused on business logic. The edge should be focused on availability.

The more failures your edge can absorb, the less every downstream service needs to care. That’s why the closer you get to the customer, the more stable the platform must become.

Most resiliency discussions focus on retries, timeouts, and circuit breakers.

But some of the most important resiliency patterns happen after the failure.

That’s where compensating transactions come in.

Resiliency Is About Recovery

A common mistake is thinking resiliency means preventing failures.

Failures are inevitable.

Networks fail. Services crash. Messages get lost. Requests time out.

True resiliency is accepting that failures will occur and having a plan to recover when they do.

Sometimes recovery means correcting actions that may already have occurred.

A Real-World Example

Have you ever swiped your card at a store, received a charge notification, only to have the terminal report an error and later discover that the charge was nowhere to be found?

What happened?

The point-of-sale terminal sent an authorization request.

But before it received a response, something failed: a network issue, a timeout, or a problem somewhere in the payment flow.

Now the terminal doesn’t know whether the transaction succeeded or failed.

The authorization request may have reached the card issuer and been processed, or it may not have.

Rather than risk leaving the customer incorrectly charged, the terminal sends a second transaction: a reversal.

The Compensating Transaction

That reversal is a real-world example of a compensating transaction.

Its purpose is simple:

Undo the effects of a previous action if that action completed successfully. If the original authorization never happened, nothing changes.

If it did happen, the reversal corrects it. Instead of determining exactly what happened, the system performs a corrective action.

Beyond Payments

Compensating transactions show up in many real-world distributed systems:

• Releasing inventory after a failed order
• Refunding a payment after a fulfillment failure
• Canceling a reservation when verification times out

The pattern is always the same: something failed, and the system takes a corrective action.

Final Thoughts

When engineers think about resiliency, they often focus on preventing failures. But distributed systems fail in unexpected ways.

Sometimes the most resilient thing a system can do isn’t retry. It’s correct the mistake and move forward.

This debate comes up constantly.

Should resiliency live in the platform components, or should the application own it?

Like most things in distributed systems, the answer is: It depends.

Infrastructure Understands Traffic

Service meshes, API gateways, and load balancers are great at handling generic resiliency concerns.

Things like:

• Connection timeouts
• Automatic retries
• Circuit breakers
• Request-level failover

The advantage is obvious. You remove complexity from the application and apply resiliency consistently across services.

For many scenarios, this is the right answer.

Infrastructure Doesn’t Understand Intent

The challenge is that platform services only understand traffic. They don’t understand why the request is made.

They don’t know whether a request is reading customer profile data, reserving inventory, transferring money, or uploading cat videos.

To the service mesh, they’re all just requests. Some requests might even have custom timeouts and retries configured. Most of the time, that’s fine.

Sometimes, it’s not.

Context-Aware Resiliency

Some resiliency decisions require application context.

Consider a financial transaction. Should a timeout trigger a retry? Maybe.

Does the request have an idempotency key? If yes, blindly retrying might be safe. If not, things become more complicated.

Did the request reach the downstream system? Was it partially processed?

Do you need a compensating transaction before retrying?

At this point, the retry is no longer a networking decision. It’s a business decision. And business decisions belong in the application.

The Real Answer

The best architectures usually use both approaches.

Use platform resiliency whenever the decision can be made without application context. But when correctness depends on understanding the request itself, move that logic into the application.

There should be a strong preference toward offloading complexity when possible. Just don’t offload decisions that require business context.

Final Thoughts

I often see teams over-index on platform-level resiliency. And while reducing application complexity is valuable, it isn’t free.

Infrastructure understands traffic and request characteristics. Applications understand business intent.

When resiliency decisions depend on business intent, they belong in the application.

Dual writes are one approach that deserves more attention.

Most database migrations fall into one of a few buckets:

• Export and import
• Replication between two databases
• Services specifically built to synchronize data

All of those approaches can work well. But sometimes you need both databases active while gradually migrating traffic from one to the other.

That’s where dual writes become interesting.

What Are Dual Writes?

The idea is exactly what it sounds like. Every write is sent to both databases.

Insert a row? Write it twice.

Update a record? Update it twice.

Delete something? Yup, delete it twice.

The goal is to keep the old and new databases synchronized while both are active.

How It Works

The most common approach is implementing dual writes directly in the application.

Instead of maintaining one database connection pool, the application maintains two and executes write operations against both systems.

In some cases, infrastructure can help as well.

For example, Envoy supports request mirroring patterns that can be useful when migrating certain technologies, such as Redis.

The implementation will vary, but the concept remains the same: each write is performed twice.

The Hard Part: Failure Handling

The hard part is not writing twice, but rather handling partial success.

What happens when Database A succeeds, but Database B fails?

Now the two databases disagree.

Do you retry? Can the operation be safely retried?

Can you roll back the successful write?

Do you have locking or reconciliation mechanisms?

This is where dual writes become significantly more complex than they initially sound.

Writing twice is easy. Maintaining correctness is the hard part.

Where Dual Writes Work Best

Dual writes tend to work best when:

• Updates occur frequently
• Eventual consistency is acceptable
• Reconciliation processes exist
• Future updates naturally correct drift

They become much harder in systems that require strict consistency guarantees for every operation.

Why Teams Use Them

Despite the complexity, dual writes enable a powerful migration approach.

You can:

• Introduce a new database platform or table
• Keep it synchronized with the old platform
• Gradually migrate read traffic
• Eventually retire the old database

All without requiring a large downtime window.

Final Thoughts

Like most migration strategies, dual writes are a tradeoff.

They add complexity.

But they also enable something extremely valuable: migrating between database platforms while the system remains live.

When downtime is not an option, dual writes can be one of the most powerful migration techniques available.

I recently talked about using glue services (Anti-Corruption Layers) to isolate modern platforms from legacy integrations.

Today I want to talk about another type of glue service: data synchronization services.

🗃️ The Real Modernization Problem

One of the hardest parts of replacing a legacy platform is usually not the application itself.

It’s the data.

In a perfect world, you could: take downtime, export the database, import it into the new platform, and switch traffic.

Simple.

But large downtime windows are increasingly rare. And big-bang migrations are risky enough that many organizations actively avoid them.

That means old and new systems often need to run side-by-side for a while.

Both platforms stay active.

Both serve customers.

Both need access to the same data. This is where modernization becomes difficult.

🔄 Keeping Two Systems in Sync

When you are changing both the application and the underlying database, keeping data synchronized becomes difficult very quickly.

Especially when the two systems use different schemas, database technologies, data models, and update patterns.

You can’t simply export/import anymore. You need continuous synchronization.

👨🏻‍🤝‍👨🏾 Enter the Synchronization Service

One useful pattern is to build a dedicated synchronization service whose sole responsibility is to keep data aligned across systems.

How synchronization works depends entirely on the platform.

I’ve used several approaches over the years.

⚙️ Database Triggers

One approach is to use database triggers or change-capture mechanisms in the legacy system.

When data changes, it’s detected, the synchronization service processes it, and the new platform is updated.

📩 Event-Based Synchronization

Another approach is to have the legacy platform emit events via a message broker.

The synchronization service consumes those events and updates the new database/platform accordingly. This tends to work especially well when modernizing toward event-driven systems.

⏳ Temporary Infrastructure

The important thing to remember is that these synchronization services are usually temporary. Their job is not to become a permanent platform.

Their purpose is to enable safe migration while reducing downtime and risk.

Once the migration is complete, the glue service disappears.

⚠️ One Important Warning

Bi-directional synchronization becomes extremely difficult very quickly.

With bi-directional synchronization, you have to solve:

• Conflict resolution
• Consistency problems
• Update ordering
• Partial failures

Whenever possible, avoid bi-directional synchronization.

In many cases, it is simpler to migrate the application layer first, then the database layer.

🧐 Final Thought

Modernization is rarely just replacing code. A major challenge is safely bridging old and new systems during transition periods.

Synchronization glue services are often what make zero-downtime migrations possible.

One of the biggest mistakes I see during modernization efforts is letting legacy integrations dictate the design of the new platform.

That usually leads to putting fresh paint on the same old house. Rebuilding the same architecture with a newer tech stack.

😴 The Dream vs. Reality

The dream project is building a brand-new platform with no existing users, integrations, or constraints.

You can design everything “the right way” from day one. But most real-world projects are not like that.

Most projects are modernization efforts. And most modernization efforts are weighed down by existing integrations, legacy protocols, and operational dependencies.

Changing customer behavior and expectations is often harder than rebuilding the platform itself.

🥲 The Common Mistake

Many teams respond to this by centering their new platform on the old integration model.

If customers use XML over Message Brokers, the new platform may speak JSON, but it still inherits the event-driven constraints—even when they no longer make sense.

If the legacy system exchanges files, the new platform is usually heavily batch-based.

The problem is:

Your modernization effort becomes constrained by the past.

🤯 A Better Approach: Glue Services

The formal term for this pattern is an Anti-Corruption Layer.

Personally, I think “glue service” explains it better. It makes the concept easier to understand.

Build the internal platform the way you actually want it designed.

Then build lightweight edge services that translate between legacy integrations and your modern platform.

• XML over Message Brokers? Use gRPC internally.
• Files? Break them into APIs.
• Long-lived ISO8583 TCP connections? Terminate them at the edge and use gRPC + microservices behind them.

The glue service has one responsibility: to translate between old and new worlds.

🤔 Why This Matters

These glue services give you two major advantages.

First, your internal architecture stays modern and optimized for current engineering practices. Second, your customers and integrations do not need to migrate immediately.

That dramatically reduces modernization risk. Temporary glue services can be huge modernization enablers.

But sometimes these glue services live forever, which is ok.

The important thing is that legacy integrations no longer constrain your modern platform.

🔁 It Works at Both Ends

This pattern applies both to inbound (clients calling your platform) and outbound (your platform calling legacy systems) integrations.

In many large systems, you’ll find glue services on both sides of the platform.

At the edge entering the system, and again leaving it.

🧐 Final Thoughts

Modernization is rarely just rewriting code. A major challenge is safely bridging old and new systems during transition periods.

I’ve used this pattern many times.

• Breaking files into APIs
• Translating ISO8583 and ISO20022 messages into Protobuf
• Terminating long-lived TCP sessions at the edge of a gRPC-based microservices platform

Glue services may not be glamorous, but they are one of the safest ways to modernize systems without dragging the past into the future.

I’ve been talking a lot about architecture documentation and how it helps both humans and agents.

From my experience, most teams spend 90% of their effort on diagrams and 10% on text.

Why? Because they are the fastest way to communicate a system:

Diagrams show how components interact, how data flows, and what depends on what. They help humans understand complex systems quickly.

That’s why most architecture documentation leans heavily on diagrams.

🤔 The Problem

Most diagrams are images. Images are great for humans, but not for agents.

Some agents can interpret images, but not reliably or consistently. And even when an agent can interpret an image, it can’t reliably reason about it or keep it up to date.

🧠 Make Diagrams Understandable

You don’t need to move away from diagrams to embrace agents. Just make them understandable to agents.

If diagrams are the most valuable part of your architecture documentation, make them readable as code.

📝 Use Code-Based Diagrams

Tools like Mermaid turn text into diagrams.

That means agents can read them, reason about them, and even keep them up to date.

A Mermaid diagram isn’t just documentation—it’s structured context.

💡 Why This Matters

When your diagrams are code:

• They live with your system
• They evolve with changes (as long as you update them)
• They can be versioned
• They can be generated or modified by agents

And most importantly, they become a living, usable context.

🔄 Keep Them Up to Date

Keeping architecture documentation up to date is always a pain. Images make that even harder.

Code-based diagrams are much easier to keep up to date. Especially when you use agents and the architecture sits next to your code.

When you make a change, direct the agent to update the architecture documentation as well. Even if you don’t fully trust agents with code changes, architecture documentation is low risk.

Instead of letting it drift over time, let agents keep it up to date.

I recently wrote about keeping architecture documentation in Git and making it available to coding agents.

A common pattern I see is teams maintaining a central architecture repository that spans multiple services and platforms. It’s an approach I use as well—but there’s a nuance.

🏯 Architecture Has Layers

Good architecture documentation spans multiple levels.

At a high level, you have:

Business Architecture 👨‍💼

Capabilities, domains, and high-level concepts.

Solution Architecture 👩‍🎨

Mapping capabilities to systems.

Defining which systems own what.

What gets replaced.

What gets invested in.

System Architecture 👷

Services, databases, jobs, infrastructure, and how they interact.

These three layers belong in a central architecture repository.

But there is one more layer that many teams forget.

🔍 The Missing Layer

Component Architecture 🧱

This is where architecture stops being abstract and starts becoming actionable.

Things like:

• Why middleware X was chosen over Y
• How a request flows through a handler
• What interfaces exist and why
• What metrics and tracing are required
• How a feature is expected to behave

These decisions are too detailed for a central architecture repo—and too important to leave undocumented.

📦 Put It in the Codebase

Component-level architecture belongs with the code.

Not in a separate repo, not in a wiki, and certainly not only in tribal knowledge.

Why does this matter? Because this is where architecture stops being abstract and becomes a specification.

A specification for engineers, reviewers, and now, coding agents.

🤖 Why This Matters for Agents

Agents don’t understand your system by osmosis the way human engineers do.

They only know what you give them. If your architecture decisions live far away from your code, your agent:

• Misses constraints: Uses technologies that aren’t allowed in your environment
• Guesses at design: Assumes REST when you expect event-driven messaging
• Makes inconsistent choices: Pulls in the wrong libraries or dependencies

But when that context lives in the repo, architecture:

• Evolves with the code
• Is versioned
• Is available at the moment it’s needed

🧠 Final Thought

I’ve always believed the best engineers don’t just understand one codebase; they learn the end-to-end system.

The same applies to agents.

If you want better output from an agent, give it the same level of detail you’d give a senior engineer.

I’ve been a big believer in Architecture Decision Records (ADRs) long before coding agents came along.

Documenting decisions gives engineers context:

Why is the system designed a certain way? What constraints existed at the time? What tradeoffs were made?

That context matters. It also matters for agents.

🤖 Agents Need Context Too

Unlike human engineers, agents don’t get context from hallway conversations, shadowing others, or tribal knowledge.

They only know what you capture. The best way to capture architectural context? Write it down as a decision record—and make it accessible to agents.

The only question is, what’s the best way to make decision records accessible?

🏗️ Option 1: MCP Server

If your ADRs live in a wiki or documentation system, you can expose them through an MCP server.

This works well when documentation is spread across teams or multiple systems that need to be aggregated.

You want a unified interface for agents. MCP is a good approach, but it comes with some infrastructure overhead.

🧱 Option 2: Keep ADRs in Git

I’ve long preferred storing ADRs in Git.

It provides versioning, review workflows, automated validation, and is where engineering work happens. Storing ADRs in Git, ideally alongside your code, is the fastest way to give agents usable context.

The challenge is that architecture often spans multiple services and repositories. So many centralize their architecture into a single repository, which is not where your code lives.

🌉 Bridging the Gap

Most modern coding agents let you include additional directories or sources at runtime, either through slash commands or CLI options.

That means you can: open your codebase, include your architecture repository, and run the agent with context.

Just adding another directory gives your agent an understanding of system constraints, architecture decisions, technology choices, and surrounding systems. These are not things an agent can reliably infer from code alone.

💡 Final Thought: Why Context Matters

With architectural context, agents produce code that aligns with your system.

When engineers understand the system end to end, they make better decisions. The same applies to agents.

If you want better results, give better context.

A common issue I see teams run into when adopting gRPC is leaving readiness checks pointed at their HTTP listener while production traffic actually flows through gRPC.

Everything looks fine until it suddenly doesn’t.

🤔 The Problem

Many gRPC services run two listeners: one for HTTP and one for gRPC.

The HTTP listener often exists for metrics, liveness checks, and management APIs. Teams moving to gRPC often reuse the HTTP health checks they set up for their REST-based services.

It’s generally a good idea to reuse what you already have, but in this case, it can be misleading.

⚠️ Health-Check What Serves Traffic

If customers connect through gRPC, your first readiness check should too.

Your HTTP listener can be perfectly healthy while the gRPC listener is misconfigured, hung, or otherwise failing.

Meanwhile, Kubernetes, load balancers, and dashboards might all show green. ✅

This happens more often than people think.

🩺 Better Ways to Monitor gRPC

There are better ways to monitor your gRPC service.

gRPC Health Probe ✅

Use a real gRPC health check request against the listener.

This validates the actual serving path and confirms the service can respond over gRPC.

A strong default option.

Build a Status gRPC Service 📋

Expose an internal status method in your gRPC API.

This gives you flexibility to check deeper dependencies, such as database readiness, downstream systems, internal state, and maintenance toggles.

It’s more work, but more control.

Use a Single Shared Listener ☝️

Because gRPC runs on top of HTTP/2, many languages and frameworks can serve HTTP and gRPC traffic on the same listener.

That means an HTTP health endpoint may be acceptable because it checks the same network path. It still does not fully validate gRPC behavior, but it is better than checking an entirely separate listener.

🧠 Final Thoughts

gRPC is awesome.

But making a service production-ready means revisiting configurations inherited from REST services.

• Health checks
• Load balancing behavior
• Connection management
• Contracts
• Operational tooling

None of these changes are difficult. They’re just easy to miss.

Many engineers think of load balancers as simple traffic distributors.

Send requests across servers. Keep systems available. Move on.

But one of their most valuable capabilities is often overlooked. Weighted load balancing.

🤨 What Is Weighted Load Balancing?

From enterprise hardware appliances to software load balancers like HAProxy and Envoy Proxy, nearly all modern load balancers support some form of weighted routing.

Start with a standard balancing algorithm, such as round-robin. Then apply weights so some targets receive more traffic than others.

For example, if two targets are weighted at 90 and 10, roughly 90% of traffic goes to one target and 10% to the other. If targets have equal weights, traffic is typically distributed evenly.

Simple idea, critical feature.

🤔 Why It Matters

Weighted load balancing turns migrations from risky big-bang cutovers into small, adjustable dials.

Instead of flipping traffic all at once, you can gradually shift production traffic while observing behavior in real time.

That means a smaller blast radius, easier rollbacks, and safer production migrations.

🧰 What I Actually Use It For

I’ve rarely used weighted load balancing because one server had more capacity than another.

What I’ve used it for repeatedly is change management.

Ten years ago, to migrate from a legacy file transfer platform to newer platforms. We used weighted load balancing to introduce the new platform gradually.

Six years ago, to control which transactions were routed to our old card payments platform versus the new platform, we introduced weighted load balancing in our global transaction router.

Last night, to run a canary deployment, we adjusted our service mesh using weighted routing via xDS.

Different eras, different platforms, same core concept.

🕰️ Standing the Test of Time

Weighted load balancing is not new. It has existed for a long time.

Foundational patterns often become the enablers for newer platform practices. Canary releases, blue/green deployments, service mesh traffic shifting.

Many of those ideas rely on the same underlying capability: controlling traffic through percentages and weighting.

Good patterns tend to survive generations of technology.

🧠 Final Thoughts

Many software engineers will never build a load balancer. They will never configure one themselves.

But understanding what these systems can do is still an advantage.

Because migrations are often less about code and more about traffic control.

Writing good software isn’t enough. Knowing when and how to shift 1% of traffic can make or break a migration.

How many times have you seen it?

Your platform is running smoothly. No alerts, no issues. Then suddenly, something breaks.

After digging in, you discover the cause: another system you depend on made a change, and that change broke your platform.

They didn’t notice it broke. You did, much too late…

How many times have you been the cause of another platform breaking?

🥶 Cold Reality

I wish the above scenario were rare, but it happens constantly across the technology industry.

It happens between internal teams, third-party integrations, and shared infrastructure teams.

These scenarios make you wonder, “How was that change validated?”

Maybe they tested it, and their validation had gaps. Maybe they did little validation at all. If any.

Either way, the result is the same: they validated their change with 100% of production traffic. Bad plan.

💡 Better Ways to Validate Changes

There are many ways teams can reduce production risk when rolling out changes, and the best teams combine the following approaches.

Canary Releases 🐤

I talk about canary deployments often.

Instead of moving 100% of traffic at once, move small percentages gradually and observe behavior closely.

That observed part matters. Look at error rates, latency changes (beyond normal platform warmup), resource spikes, and unexpected retries. All of these indicate customer impact.

Canary deployments are one of the best ways to reduce the blast radius of changes, identify problems quickly, and self-correct.

Shadow Traffic 🪞

Traffic mirroring sends production traffic to a new version before routing live traffic there.

Responses are ignored, but you observe behavior and monitor the same signals you would with a canary release without sacrificing a customer request.

Synthetic Traffic 🤖

Synthetic traffic simulates user behavior continuously. It’s great for monitoring customer experience, but also a great way to validate new deployments.

Route synthetic traffic to upgraded instances first and verify behavior before moving real traffic. If it fails with synthetic traffic, it likely won’t survive real traffic.

Smoke Tests 😶‍🌫️

The classic approach. After deployment, run a small set of fast tests to confirm the platform is fundamentally working.

Smoke tests don’t need to be fancy; they can be shell scripts, API calls, read-only requests, a test file, or full end-to-end validation.

Their purpose is simple: to quickly catch obvious breakage.

🧠 Final Thoughts

Don’t think of the above methods as mutually exclusive choices. Combine them.

Some platforms I work on combine canary releases, shadow traffic, and synthetic traffic. Others use smoke tests plus canary releases.

The more layers of validation you have, the more likely you are to catch issues before your customers do. Because having your customers validate changes for you is a poor strategy.

It is a foundational technique used by many modern databases, caches, and large-scale platforms. Understand how it works and you can apply the same pattern in your own systems.

🤔 Understanding the Problem

At some point, every successful system hits the limits of a single database instance.

A single server can only handle so many connections, queries, writes, storage capacity, or CPU/memory demands. Even with the best hardware, performance eventually degrades. So systems scale horizontally.

Instead of sending all traffic to a single database server, requests are distributed across multiple nodes.

At the same time, resiliency matters. If one server fails and all data resides there, the outage can be severe.

So modern databases spread data across multiple nodes, availability zones, and regions.

Distributing load and data solves both capacity and resiliency problems. But it introduces another challenge.

How do you keep request behavior consistent when data is distributed across multiple systems?

⚠️ Why Replication Is Not Enough

Replication helps, but it does not solve every consistency problem.

Imagine a write lands on Server 1. Immediately after, a read request for the same data lands on Server 67. Will Server 67 have the latest version? Maybe, but often not.

Asynchronous Replication

With asynchronous replication, Server 1 will accept the write and replicate the data to other servers in the background. That means a follow-up read on any other node may return stale data.

Synchronous Replication

With synchronous replication, the write on Server 1 will wait for an acknowledgment from all replicas before returning a success. While this improves consistency guarantees, it increases latency.

The farther apart a replica is, the worse this gets. Local writes may be fast, but cross-region writes will be slow. Plus, is it really feasible to replicate data across every single node?

So the question becomes: How do you preserve consistency, without paying latency taxes?

🔀 Route Requests to the Data

A highly effective answer is deterministic routing.

Instead of moving data to where requests might land, move requests to where the data already exists.

If requests for the same key can go to the same node, you gain predictable ownership, reduced stale reads, lower coordination overhead, and easier horizontal scaling.

👨‍🏫 How Deterministic Routing Works

At a high level, the system needs a repeatable way to decide where requests should go.

A common approach is hashing.

• A hash of user123 always goes to Node 7
• A hash of user456 always goes to Node 42

As long as the same key produces the same result, requests can be consistently routed to the same owner. Many modern databases implement deterministic routing through techniques like consistent hashing, partition maps, and shard ranges.

🗺️ Where Routing Logic Lives

Different systems solve routing in different places.

Client-side Routing

The client library knows the partition map and sends requests directly to the correct node. Used by many distributed caches and databases.

Proxy / Router Tier

A small router sits in front of nodes and forwards traffic appropriately. Useful when client behavior cannot be influenced.

Server-side Forwarding

Requests land anywhere, and the receiving node forwards internally to the owning node. Simple for clients, doesn’t introduce a proxy failure point, but introduces complex cluster discovery/health monitoring.

Each model has tradeoffs.

🧰 Routing Does Not Replace Replication

Deterministic routing is powerful, but not magic. What happens when the owning node is down? You still need replication.

Modern databases combine both: deterministic routing for performance and ownership, plus replication for durability and failover.

🧠 Why This Matters Beyond Databases

Distributed databases use this approach, but it is not unique to them.

Deterministic routing can be used to solve: session ownership, user affinity, in-memory workflow coordination, work queue partitioning, and more.

I’ve used deterministic routing many times to solve load distribution and consistency problems.

At scale, the answer is not always more/better hardware. Consistency and availability problems are not always solved with replication alone.

Sometimes the best answer is simply to send the request to the right place.

🧩 How It Usually Works

Most microservice designs follow the same model:

• Break systems into capabilities, teams, or functions
• Deploy one shared service for each capability
• Any platform that needs it calls that centralized service

That works well for many cases, but it’s not the only model.

🏗️ How We Got Here

Before microservices, many organizations used Service-Oriented Architecture (SOA).

Despite being labeled as antiquated, SOA and microservices are not that different. Both break down systems into capabilities that communicate with each other. The biggest difference is scope.

In SOA, a “Payments Service” might own:

• Message parsing
• Validation
• Balance checks
• Currency conversion
• Settlement logic

While other SOA services would own “Users” or “Accounting”. Today, that payment service would be considered an entire platform, with each of those capabilities implemented as microservices within that domain.

Microservices are often the same idea as SOA, just at a more granular level.

🎯 Why Centralization Became the Default

One reason microservices gained traction was the need to avoid duplication. Capabilities were often rebuilt across multiple systems. For example, Currency Conversion is needed in Payments, Accounting, and many other platforms.

Duplication is not just wasteful, it creates real problems: logic drift, coordination overhead, and inconsistent outcomes across systems. Packaging that capability as a standalone service solved real problems: build once, reuse everywhere.

⚠️ The Downside of Centralization

In cell-based architectures, platforms are usually designed to be self-contained and failure-isolated. That means a mission-critical platform depending on a centralized service shared by other platforms can become a design smell.

• Cross-cell dependencies
• Added latency
• Shared failure domains
• Complex failover scenarios

So teams, once again, solve these problems by rebuilding the same capability locally.

🔁 Another Option

Instead of rebuilding the capability each time, deploy the same microservice codebase inside multiple platforms. If both Payments and Accounting need a currency conversion service, deploy the same service within each platform.

It’s the same codebase and capability, but with local ownership and resilience. You get reuse without forced centralization.

🧪 Caveats from Experience

This pattern works when applied carefully.

1️⃣ Strong Ownership

A shared codebase still needs a clear owning team. Others can contribute, but someone must own quality, roadmap, and releases.

2️⃣ Pick the Right Capabilities

Not everything is a great fit. Something like currency conversion is well-scoped, relatively stateless, and doesn’t have unique business logic based on which platform is calling it. It’s a strong example.

But other services that have unique logic for each platform domain or require consistency across different platforms are less of a fit.

3️⃣ Operational Discipline

Using the same codebase doesn’t automatically solve all problems; you can still run into drift across platforms if each is running a different version. Changes in behavior still sometimes need coordination.

But with a single codebase, these issues are far easier to address.

💭 Final Thoughts

Microservices gave us reusable building blocks. Sometimes the best use of a microservice is not one centralized deployment. Sometimes it’s many local deployments of the same capability.

Just reuse the software while maintaining autonomy.

🚦 What is Traffic Mirroring?

Traffic mirroring in Istio or Envoy Proxy lets you send a copy of live traffic to a secondary target.

When enabled, traffic to /service routes to cluster1 as normal, and a mirrored copy is sent to cluster2.

The key: mirrored traffic is fire-and-forget. Responses are ignored and never impact the primary request.

🧪 Why It’s Powerful

1️⃣ Shadow Traffic for Safe Testing

The most common use case is shadow traffic.

When migrating platforms or deploying a new version of an application, you can send real traffic to the new system, observe behavior, and validate responses.

All without impacting users. No risky cutovers. You see exactly how the new system behaves under real load.

2️⃣ Out-of-Band Traffic Inspection

Another powerful use case is traffic inspection.

Inline inspection is risky. It adds latency, introduces new failure points, and becomes part of the critical path.

With traffic mirroring, you can inspect traffic, analyze requests, and detect anomalies.

All without impacting the primary path.

😶‍🌫️ Reality Check

It’s not perfect. There is some overhead.

Mirroring adds load to the sidecar, which may or may not be acceptable for your system. In my experience, it’s negligible, but it’s something you should measure in your own environment before deploying to production.

🧠 Final Thoughts

Traffic mirroring is one of the safest ways to validate migrations, test new systems, and observe real production behavior.

The hard part isn’t mirroring traffic. It’s running two production systems in parallel. That’s the real cost, and the real tradeoff.

But if you can afford that cost, traffic mirroring is an incredibly powerful tool.

If you want to dig deeper:

• Istio traffic mirroring docs explain the workflow.
• Envoy request mirror policy docs cover the lower-level routing behavior.