I’m a believer that, as an engineer, you should be able to run your software locally. But I hear it often: “We can’t run locally because of some reason.”

Sometimes it’s valid. There are architectures and platforms out there that prevent running locally. But more often than not, when it comes to backend distributed systems, it’s a design or implementation decision that nobody has challenged.

Why This Matters

Everyone these days is focused on speeding up software delivery. Today, that generally means coding agents, AI tooling, and code generation.

But writing code faster doesn’t matter much if validating a one-line change takes an hour.

The speed of software delivery is often dictated by validation, not code generation.

The Problem with Shared Dev Environments

Many teams still rely on shared development environments as their primary way of validating changes.

That process usually looks something like this:

• Make a change
• Commit the change
• Push the branch
• Wait for a build
• Wait for a deployment
• Run tests

If everything goes well, you can validate your change. If a mistake was made, or something doesn’t work right, you have to fix it and start over.

The result is a slow feedback loop, which can sometimes influence behavior.

The more friction there is to validate a change, the larger that change becomes. If testing takes a long time, you will naturally focus on making sure everything is perfect before spending time validating.

Local Validation Influences Good Behavior

Compare the above process to one that uses a locally running service.

• Make a change
• Build
• Start the service
• Run tests

The process is dramatically shorter and, more importantly, faster. This encourages engineers to make smaller changes, test more frequently, iterate their implementations, and find mistakes earlier.

This all results in better software.

The Excuses

“My service depends on too many other services.”

Run the services you own locally. Mock the services you don’t.

“I need a database or message broker.”

Run those locally too. There’s very likely a Docker container for each of them.

“We use cloud services.”

This one can be harder, but emulators exist for some services. Other services might have a Dockerized open-source alternative. For those that don’t, you could mock them.

There will always be exceptions: mainframes, specialized hardware, or unique managed services.

But I find many teams jump straight to “we can’t run locally” before they’ve seriously explored how they could.

Running Local Doesn’t Mean Everything

You don’t need to recreate your entire production environment on a laptop, though if you can, go for it.

The goal is to validate your change without depending on a shared or non-local environment that takes forever to test against.

Local development isn’t about recreating production. It’s about creating enough of the environment to validate your change. If that means turning off some functionality or creating mock services, it might be worth it.

Why This Matters Even More Now

Validating changes quickly is becoming more important as teams adopt coding agents. An agent can generate code in seconds.

But if every iteration with your coding agent requires committing, pushing, building, deploying, and waiting, the feedback loop becomes the bottleneck.

Final Thoughts

Agents make mistakes. Humans make mistakes.

The faster you can validate a change, the faster you can correct those mistakes. That’s why local execution isn’t just a convenience.

It’s one of the most effective ways to shorten the feedback loop.

One of the simplest reliability rules I’ve learned is this: Every dependency is another way for your service to fail.

Why Dependencies Matter

Every service has dependencies.

• Databases
• Caches
• Configuration services
• Secrets managers
• Logging pipelines
• Tracing backends

All of these dependencies can fail, and when they do, the typical service will fail with them.

The more dependencies a service has, the more failures it inherits. Every dependency adds features, but it also adds failure modes.

Reliability is often about deciding which dependencies are actually worth the tradeoff.

Why Edge Systems Tend to Be Dependency-Light

I recently wrote about how systems closest to the customer carry the greatest responsibility for availability.

This is one reason edge systems tend to be dependency-light.

Load balancers, API gateways, and routers are responsible for availability.

Every dependency added to these systems creates another opportunity to take down the entire platform. So they tend to avoid dependencies whenever possible.

Eliminating Dependencies Isn’t Always Necessary

Sometimes removing a dependency entirely isn’t practical.

A better question is:

• What happens if the dependency disappears?
• Can the service continue operating?
• Can it use cached data?
• Can it fall back to a last-known-good configuration?
• Can it degrade gracefully?

If the answer is yes, you’ve significantly improved reliability even though the dependency still exists.

A Real-World Example

Take Envoy Proxy. Envoy can receive configuration from an xDS service. But it doesn’t call xDS for every request.

Instead:

• Configuration is fetched periodically
• Stored in memory
• Used locally during request processing

If the xDS service becomes unavailable, Envoy continues routing traffic using its last known configuration. The dependency still exists, but request processing no longer depends on its availability.

That’s a very different reliability model.

Don’t Make Observability a Hard Dependency

One of the most common mistakes I see is making observability a required dependency.

If your logging or tracing backend becomes unavailable, should customer traffic stop flowing? No.

In most cases, observability should be a best effort.

Use asynchronous logging, buffering, and truncation policies so customer traffic continues to flow even when your observability platform is experiencing issues.

Operational visibility is important, but customer availability is more important.

Final Thoughts

You’ll never eliminate every dependency. But you can eliminate unnecessary ones.

And for the remaining dependencies, you can design your service to survive failures.

The most reliable services aren’t dependency-free. They’re designed to survive dependency failures.

You’ve all heard the advice: “Avoid caching because caching is difficult to get right.”

I agree with part of that statement. Caching can absolutely be difficult. But I think the reality is more nuanced.

The difficulty level of caching depends heavily on the type of data you are caching.

Not All Data Is Equal

When engineers talk about caching complexity, they’re usually thinking about data that changes frequently and requires strong consistency.

That’s the hardest kind of data to cache. Because you need to answer some difficult questions.

• How do you invalidate the cache?
• How do you handle updates?
• What happens when the cache population fails?
• How stale is too stale?

These problems are real, but they don’t apply to all data.

Frequently Updated Data

Some data changes constantly and has strict accuracy requirements. Account balances, inventory counts, active orders, and similar records. This is where caching is difficult.

Every stale read has a consequence, and in some cases, it’s not worth the complexity.

Infrequently Updated Data

Some data changes occasionally, once an hour, once a day, or even longer. Configuration data, product catalogs, country codes, and similar records.

In these cases, a small amount of staleness might be fine. If data changes hourly and your cache refreshes every few minutes, that’s often a reasonable tradeoff. The less frequently data changes, the less sensitive you become to cache staleness.

Caching is much easier with infrequently updated data.

Immutable Data

Immutable data is the easiest to cache. Because it never changes. Ledger entries, event records, receipts, and other immutable records.

Once immutable data is loaded into a cache, cache invalidation largely disappears as a problem. Cache complexity for immutable data is more about balancing performance and hit-miss ratios.

Most of the time, immutable data is an ideal candidate for caching.

The Real Question

Instead of asking, “Should I cache?” Ask yourself, “How often does this data change? How accurate does it need to be?”

The answers will give you an idea of how complex caching will be.

Final Thoughts

Caching isn’t inherently hard.

Maintaining correctness for frequently changing data is hard.

The more stable the data, the less complex the caching needs to be.

Like most things in system design, you can’t just take all-or-none answers like “always cache” or “never cache.” It’s important to understand the trade-offs, constraints, and consistency requirements of the data you’re working with.

The closer a component is to the customer, the greater its responsibility for keeping the entire platform available, even when everything behind it is having a bad day.

Not All Services Carry the Same Reliability Burden

Let’s consider a typical platform.

Customer -> Load Balancer -> API Gateway -> Orchestrator -> Microservices -> Database

Every layer has a different job. But every layer also has a different level of responsibility for resiliency.

As you move toward the customer, that responsibility increases.

Deep Services Focus on Business Logic

At the deepest layers of the platform, services are usually focused on business capabilities.

They process orders, transfer money, manage inventory, and store data.

These services often have databases, business rules, stateful operations, and multiple dependencies.

Resiliency matters, but it’s often focused on correctness.

If a database call fails:

• Should the transaction roll back?
• Should the service fail over?
• Should a compensating transaction occur?

These services are primarily concerned with business outcomes.

The Middle Layers Absorb Failures

Move up a layer, and you often find orchestrators and workflow services. These components coordinate work across multiple services.

If one service fails, the orchestrator may retry, execute fallback logic, trigger compensating actions, or roll back a workflow. Their job is not just executing business logic, it’s ensuring execution succeeds despite failures.

The Edge Exists to Protect Everything Behind It

At the edge, things change.

Load balancers and API gateways are often stateless, dependency-light, highly available, and extremely fast.

Why?

Because their primary responsibility is availability. Everything behind them is allowed to fail, and they absorb as much of that failure as possible.

They:

• Route around failures
• Shed load
• Fail over traffic
• Enforce timeouts
• Apply retries
• Protect backend systems

The edge isn’t just resilient for itself. It’s resilient on behalf of everything behind it.

Final Thoughts

The deepest services in a platform should be focused on business logic. The edge should be focused on availability.

The more failures your edge can absorb, the less every downstream service needs to care. That’s why the closer you get to the customer, the more stable the platform must become.

Most resiliency discussions focus on retries, timeouts, and circuit breakers.

But some of the most important resiliency patterns happen after the failure.

That’s where compensating transactions come in.

Resiliency Is About Recovery

A common mistake is thinking resiliency means preventing failures.

Failures are inevitable.

Networks fail. Services crash. Messages get lost. Requests time out.

True resiliency is accepting that failures will occur and having a plan to recover when they do.

Sometimes recovery means correcting actions that may already have occurred.

A Real-World Example

Have you ever swiped your card at a store, received a charge notification, only to have the terminal report an error and later discover that the charge was nowhere to be found?

What happened?

The point-of-sale terminal sent an authorization request.

But before it received a response, something failed: a network issue, a timeout, or a problem somewhere in the payment flow.

Now the terminal doesn’t know whether the transaction succeeded or failed.

The authorization request may have reached the card issuer and been processed, or it may not have.

Rather than risk leaving the customer incorrectly charged, the terminal sends a second transaction: a reversal.

The Compensating Transaction

That reversal is a real-world example of a compensating transaction.

Its purpose is simple:

Undo the effects of a previous action if that action completed successfully. If the original authorization never happened, nothing changes.

If it did happen, the reversal corrects it. Instead of determining exactly what happened, the system performs a corrective action.

Beyond Payments

Compensating transactions show up in many real-world distributed systems:

• Releasing inventory after a failed order
• Refunding a payment after a fulfillment failure
• Canceling a reservation when verification times out

The pattern is always the same: something failed, and the system takes a corrective action.

Final Thoughts

When engineers think about resiliency, they often focus on preventing failures. But distributed systems fail in unexpected ways.

Sometimes the most resilient thing a system can do isn’t retry. It’s correct the mistake and move forward.

This debate comes up constantly.

Should resiliency live in the platform components, or should the application own it?

Like most things in distributed systems, the answer is: It depends.

Infrastructure Understands Traffic

Service meshes, API gateways, and load balancers are great at handling generic resiliency concerns.

Things like:

• Connection timeouts
• Automatic retries
• Circuit breakers
• Request-level failover

The advantage is obvious. You remove complexity from the application and apply resiliency consistently across services.

For many scenarios, this is the right answer.

Infrastructure Doesn’t Understand Intent

The challenge is that platform services only understand traffic. They don’t understand why the request is made.

They don’t know whether a request is reading customer profile data, reserving inventory, transferring money, or uploading cat videos.

To the service mesh, they’re all just requests. Some requests might even have custom timeouts and retries configured. Most of the time, that’s fine.

Sometimes, it’s not.

Context-Aware Resiliency

Some resiliency decisions require application context.

Consider a financial transaction. Should a timeout trigger a retry? Maybe.

Does the request have an idempotency key? If yes, blindly retrying might be safe. If not, things become more complicated.

Did the request reach the downstream system? Was it partially processed?

Do you need a compensating transaction before retrying?

At this point, the retry is no longer a networking decision. It’s a business decision. And business decisions belong in the application.

The Real Answer

The best architectures usually use both approaches.

Use platform resiliency whenever the decision can be made without application context. But when correctness depends on understanding the request itself, move that logic into the application.

There should be a strong preference toward offloading complexity when possible. Just don’t offload decisions that require business context.

Final Thoughts

I often see teams over-index on platform-level resiliency. And while reducing application complexity is valuable, it isn’t free.

Infrastructure understands traffic and request characteristics. Applications understand business intent.

When resiliency decisions depend on business intent, they belong in the application.

Dual writes are one approach that deserves more attention.

Most database migrations fall into one of a few buckets:

• Export and import
• Replication between two databases
• Services specifically built to synchronize data

All of those approaches can work well. But sometimes you need both databases active while gradually migrating traffic from one to the other.

That’s where dual writes become interesting.

What Are Dual Writes?

The idea is exactly what it sounds like. Every write is sent to both databases.

Insert a row? Write it twice.

Update a record? Update it twice.

Delete something? Yup, delete it twice.

The goal is to keep the old and new databases synchronized while both are active.

How It Works

The most common approach is implementing dual writes directly in the application.

Instead of maintaining one database connection pool, the application maintains two and executes write operations against both systems.

In some cases, infrastructure can help as well.

For example, Envoy supports request mirroring patterns that can be useful when migrating certain technologies, such as Redis.

The implementation will vary, but the concept remains the same: each write is performed twice.

The Hard Part: Failure Handling

The hard part is not writing twice, but rather handling partial success.

What happens when Database A succeeds, but Database B fails?

Now the two databases disagree.

Do you retry? Can the operation be safely retried?

Can you roll back the successful write?

Do you have locking or reconciliation mechanisms?

This is where dual writes become significantly more complex than they initially sound.

Writing twice is easy. Maintaining correctness is the hard part.

Where Dual Writes Work Best

Dual writes tend to work best when:

• Updates occur frequently
• Eventual consistency is acceptable
• Reconciliation processes exist
• Future updates naturally correct drift

They become much harder in systems that require strict consistency guarantees for every operation.

Why Teams Use Them

Despite the complexity, dual writes enable a powerful migration approach.

You can:

• Introduce a new database platform or table
• Keep it synchronized with the old platform
• Gradually migrate read traffic
• Eventually retire the old database

All without requiring a large downtime window.

Final Thoughts

Like most migration strategies, dual writes are a tradeoff.

They add complexity.

But they also enable something extremely valuable: migrating between database platforms while the system remains live.

When downtime is not an option, dual writes can be one of the most powerful migration techniques available.

I recently talked about using glue services (Anti-Corruption Layers) to isolate modern platforms from legacy integrations.

Today I want to talk about another type of glue service: data synchronization services.

🗃️ The Real Modernization Problem

One of the hardest parts of replacing a legacy platform is usually not the application itself.

It’s the data.

In a perfect world, you could: take downtime, export the database, import it into the new platform, and switch traffic.

Simple.

But large downtime windows are increasingly rare. And big-bang migrations are risky enough that many organizations actively avoid them.

That means old and new systems often need to run side-by-side for a while.

Both platforms stay active.

Both serve customers.

Both need access to the same data. This is where modernization becomes difficult.

🔄 Keeping Two Systems in Sync

When you are changing both the application and the underlying database, keeping data synchronized becomes difficult very quickly.

Especially when the two systems use different schemas, database technologies, data models, and update patterns.

You can’t simply export/import anymore. You need continuous synchronization.

👨🏻‍🤝‍👨🏾 Enter the Synchronization Service

One useful pattern is to build a dedicated synchronization service whose sole responsibility is to keep data aligned across systems.

How synchronization works depends entirely on the platform.

I’ve used several approaches over the years.

⚙️ Database Triggers

One approach is to use database triggers or change-capture mechanisms in the legacy system.

When data changes, it’s detected, the synchronization service processes it, and the new platform is updated.

📩 Event-Based Synchronization

Another approach is to have the legacy platform emit events via a message broker.

The synchronization service consumes those events and updates the new database/platform accordingly. This tends to work especially well when modernizing toward event-driven systems.

⏳ Temporary Infrastructure

The important thing to remember is that these synchronization services are usually temporary. Their job is not to become a permanent platform.

Their purpose is to enable safe migration while reducing downtime and risk.

Once the migration is complete, the glue service disappears.

⚠️ One Important Warning

Bi-directional synchronization becomes extremely difficult very quickly.

With bi-directional synchronization, you have to solve:

• Conflict resolution
• Consistency problems
• Update ordering
• Partial failures

Whenever possible, avoid bi-directional synchronization.

In many cases, it is simpler to migrate the application layer first, then the database layer.

🧐 Final Thought

Modernization is rarely just replacing code. A major challenge is safely bridging old and new systems during transition periods.

Synchronization glue services are often what make zero-downtime migrations possible.

One of the biggest mistakes I see during modernization efforts is letting legacy integrations dictate the design of the new platform.

That usually leads to putting fresh paint on the same old house. Rebuilding the same architecture with a newer tech stack.

😴 The Dream vs. Reality

The dream project is building a brand-new platform with no existing users, integrations, or constraints.

You can design everything “the right way” from day one. But most real-world projects are not like that.

Most projects are modernization efforts. And most modernization efforts are weighed down by existing integrations, legacy protocols, and operational dependencies.

Changing customer behavior and expectations is often harder than rebuilding the platform itself.

🥲 The Common Mistake

Many teams respond to this by centering their new platform on the old integration model.

If customers use XML over Message Brokers, the new platform may speak JSON, but it still inherits the event-driven constraints—even when they no longer make sense.

If the legacy system exchanges files, the new platform is usually heavily batch-based.

The problem is:

Your modernization effort becomes constrained by the past.

🤯 A Better Approach: Glue Services

The formal term for this pattern is an Anti-Corruption Layer.

Personally, I think “glue service” explains it better. It makes the concept easier to understand.

Build the internal platform the way you actually want it designed.

Then build lightweight edge services that translate between legacy integrations and your modern platform.

• XML over Message Brokers? Use gRPC internally.
• Files? Break them into APIs.
• Long-lived ISO8583 TCP connections? Terminate them at the edge and use gRPC + microservices behind them.

The glue service has one responsibility: to translate between old and new worlds.

🤔 Why This Matters

These glue services give you two major advantages.

First, your internal architecture stays modern and optimized for current engineering practices. Second, your customers and integrations do not need to migrate immediately.

That dramatically reduces modernization risk. Temporary glue services can be huge modernization enablers.

But sometimes these glue services live forever, which is ok.

The important thing is that legacy integrations no longer constrain your modern platform.

🔁 It Works at Both Ends

This pattern applies both to inbound (clients calling your platform) and outbound (your platform calling legacy systems) integrations.

In many large systems, you’ll find glue services on both sides of the platform.

At the edge entering the system, and again leaving it.

🧐 Final Thoughts

Modernization is rarely just rewriting code. A major challenge is safely bridging old and new systems during transition periods.

I’ve used this pattern many times.

• Breaking files into APIs
• Translating ISO8583 and ISO20022 messages into Protobuf
• Terminating long-lived TCP sessions at the edge of a gRPC-based microservices platform

Glue services may not be glamorous, but they are one of the safest ways to modernize systems without dragging the past into the future.

I’ve been talking a lot about architecture documentation and how it helps both humans and agents.

From my experience, most teams spend 90% of their effort on diagrams and 10% on text.

Why? Because they are the fastest way to communicate a system:

Diagrams show how components interact, how data flows, and what depends on what. They help humans understand complex systems quickly.

That’s why most architecture documentation leans heavily on diagrams.

🤔 The Problem

Most diagrams are images. Images are great for humans, but not for agents.

Some agents can interpret images, but not reliably or consistently. And even when an agent can interpret an image, it can’t reliably reason about it or keep it up to date.

🧠 Make Diagrams Understandable

You don’t need to move away from diagrams to embrace agents. Just make them understandable to agents.

If diagrams are the most valuable part of your architecture documentation, make them readable as code.

📝 Use Code-Based Diagrams

Tools like Mermaid turn text into diagrams.

That means agents can read them, reason about them, and even keep them up to date.

A Mermaid diagram isn’t just documentation—it’s structured context.

💡 Why This Matters

When your diagrams are code:

• They live with your system
• They evolve with changes (as long as you update them)
• They can be versioned
• They can be generated or modified by agents

And most importantly, they become a living, usable context.

🔄 Keep Them Up to Date

Keeping architecture documentation up to date is always a pain. Images make that even harder.

Code-based diagrams are much easier to keep up to date. Especially when you use agents and the architecture sits next to your code.

When you make a change, direct the agent to update the architecture documentation as well. Even if you don’t fully trust agents with code changes, architecture documentation is low risk.

Instead of letting it drift over time, let agents keep it up to date.

I recently wrote about keeping architecture documentation in Git and making it available to coding agents.

A common pattern I see is teams maintaining a central architecture repository that spans multiple services and platforms. It’s an approach I use as well—but there’s a nuance.

🏯 Architecture Has Layers

Good architecture documentation spans multiple levels.

At a high level, you have:

Business Architecture 👨‍💼

Capabilities, domains, and high-level concepts.

Solution Architecture 👩‍🎨

Mapping capabilities to systems.

Defining which systems own what.

What gets replaced.

What gets invested in.

System Architecture 👷

Services, databases, jobs, infrastructure, and how they interact.

These three layers belong in a central architecture repository.

But there is one more layer that many teams forget.

🔍 The Missing Layer

Component Architecture 🧱

This is where architecture stops being abstract and starts becoming actionable.

Things like:

• Why middleware X was chosen over Y
• How a request flows through a handler
• What interfaces exist and why
• What metrics and tracing are required
• How a feature is expected to behave

These decisions are too detailed for a central architecture repo—and too important to leave undocumented.

📦 Put It in the Codebase

Component-level architecture belongs with the code.

Not in a separate repo, not in a wiki, and certainly not only in tribal knowledge.

Why does this matter? Because this is where architecture stops being abstract and becomes a specification.

A specification for engineers, reviewers, and now, coding agents.

🤖 Why This Matters for Agents

Agents don’t understand your system by osmosis the way human engineers do.

They only know what you give them. If your architecture decisions live far away from your code, your agent:

• Misses constraints: Uses technologies that aren’t allowed in your environment
• Guesses at design: Assumes REST when you expect event-driven messaging
• Makes inconsistent choices: Pulls in the wrong libraries or dependencies

But when that context lives in the repo, architecture:

• Evolves with the code
• Is versioned
• Is available at the moment it’s needed

🧠 Final Thought

I’ve always believed the best engineers don’t just understand one codebase; they learn the end-to-end system.

The same applies to agents.

If you want better output from an agent, give it the same level of detail you’d give a senior engineer.

I’ve been a big believer in Architecture Decision Records (ADRs) long before coding agents came along.

Documenting decisions gives engineers context:

Why is the system designed a certain way? What constraints existed at the time? What tradeoffs were made?

That context matters. It also matters for agents.

🤖 Agents Need Context Too

Unlike human engineers, agents don’t get context from hallway conversations, shadowing others, or tribal knowledge.

They only know what you capture. The best way to capture architectural context? Write it down as a decision record—and make it accessible to agents.

The only question is, what’s the best way to make decision records accessible?

🏗️ Option 1: MCP Server

If your ADRs live in a wiki or documentation system, you can expose them through an MCP server.

This works well when documentation is spread across teams or multiple systems that need to be aggregated.

You want a unified interface for agents. MCP is a good approach, but it comes with some infrastructure overhead.

🧱 Option 2: Keep ADRs in Git

I’ve long preferred storing ADRs in Git.

It provides versioning, review workflows, automated validation, and is where engineering work happens. Storing ADRs in Git, ideally alongside your code, is the fastest way to give agents usable context.

The challenge is that architecture often spans multiple services and repositories. So many centralize their architecture into a single repository, which is not where your code lives.

🌉 Bridging the Gap

Most modern coding agents let you include additional directories or sources at runtime, either through slash commands or CLI options.

That means you can: open your codebase, include your architecture repository, and run the agent with context.

Just adding another directory gives your agent an understanding of system constraints, architecture decisions, technology choices, and surrounding systems. These are not things an agent can reliably infer from code alone.

💡 Final Thought: Why Context Matters

With architectural context, agents produce code that aligns with your system.

When engineers understand the system end to end, they make better decisions. The same applies to agents.

If you want better results, give better context.

A common issue I see teams run into when adopting gRPC is leaving readiness checks pointed at their HTTP listener while production traffic actually flows through gRPC.

Everything looks fine until it suddenly doesn’t.

🤔 The Problem

Many gRPC services run two listeners: one for HTTP and one for gRPC.

The HTTP listener often exists for metrics, liveness checks, and management APIs. Teams moving to gRPC often reuse the HTTP health checks they set up for their REST-based services.

It’s generally a good idea to reuse what you already have, but in this case, it can be misleading.

⚠️ Health-Check What Serves Traffic

If customers connect through gRPC, your first readiness check should too.

Your HTTP listener can be perfectly healthy while the gRPC listener is misconfigured, hung, or otherwise failing.

Meanwhile, Kubernetes, load balancers, and dashboards might all show green. ✅

This happens more often than people think.

🩺 Better Ways to Monitor gRPC

There are better ways to monitor your gRPC service.

gRPC Health Probe ✅

Use a real gRPC health check request against the listener.

This validates the actual serving path and confirms the service can respond over gRPC.

A strong default option.

Build a Status gRPC Service 📋

Expose an internal status method in your gRPC API.

This gives you flexibility to check deeper dependencies, such as database readiness, downstream systems, internal state, and maintenance toggles.

It’s more work, but more control.

Use a Single Shared Listener ☝️

Because gRPC runs on top of HTTP/2, many languages and frameworks can serve HTTP and gRPC traffic on the same listener.

That means an HTTP health endpoint may be acceptable because it checks the same network path. It still does not fully validate gRPC behavior, but it is better than checking an entirely separate listener.

🧠 Final Thoughts

gRPC is awesome.

But making a service production-ready means revisiting configurations inherited from REST services.

• Health checks
• Load balancing behavior
• Connection management
• Contracts
• Operational tooling

None of these changes are difficult. They’re just easy to miss.

Many engineers think of load balancers as simple traffic distributors.

Send requests across servers. Keep systems available. Move on.

But one of their most valuable capabilities is often overlooked. Weighted load balancing.

🤨 What Is Weighted Load Balancing?

From enterprise hardware appliances to software load balancers like HAProxy and Envoy Proxy, nearly all modern load balancers support some form of weighted routing.

Start with a standard balancing algorithm, such as round-robin. Then apply weights so some targets receive more traffic than others.

For example, if two targets are weighted at 90 and 10, roughly 90% of traffic goes to one target and 10% to the other. If targets have equal weights, traffic is typically distributed evenly.

Simple idea, critical feature.

🤔 Why It Matters

Weighted load balancing turns migrations from risky big-bang cutovers into small, adjustable dials.

Instead of flipping traffic all at once, you can gradually shift production traffic while observing behavior in real time.

That means a smaller blast radius, easier rollbacks, and safer production migrations.

🧰 What I Actually Use It For

I’ve rarely used weighted load balancing because one server had more capacity than another.

What I’ve used it for repeatedly is change management.

Ten years ago, to migrate from a legacy file transfer platform to newer platforms. We used weighted load balancing to introduce the new platform gradually.

Six years ago, to control which transactions were routed to our old card payments platform versus the new platform, we introduced weighted load balancing in our global transaction router.

Last night, to run a canary deployment, we adjusted our service mesh using weighted routing via xDS.

Different eras, different platforms, same core concept.

🕰️ Standing the Test of Time

Weighted load balancing is not new. It has existed for a long time.

Foundational patterns often become the enablers for newer platform practices. Canary releases, blue/green deployments, service mesh traffic shifting.

Many of those ideas rely on the same underlying capability: controlling traffic through percentages and weighting.

Good patterns tend to survive generations of technology.

🧠 Final Thoughts

Many software engineers will never build a load balancer. They will never configure one themselves.

But understanding what these systems can do is still an advantage.

Because migrations are often less about code and more about traffic control.

Writing good software isn’t enough. Knowing when and how to shift 1% of traffic can make or break a migration.

How many times have you seen it?

Your platform is running smoothly. No alerts, no issues. Then suddenly, something breaks.

After digging in, you discover the cause: another system you depend on made a change, and that change broke your platform.

They didn’t notice it broke. You did, much too late…

How many times have you been the cause of another platform breaking?

🥶 Cold Reality

I wish the above scenario were rare, but it happens constantly across the technology industry.

It happens between internal teams, third-party integrations, and shared infrastructure teams.

These scenarios make you wonder, “How was that change validated?”

Maybe they tested it, and their validation had gaps. Maybe they did little validation at all. If any.

Either way, the result is the same: they validated their change with 100% of production traffic. Bad plan.

💡 Better Ways to Validate Changes

There are many ways teams can reduce production risk when rolling out changes, and the best teams combine the following approaches.

Canary Releases 🐤

I talk about canary deployments often.

Instead of moving 100% of traffic at once, move small percentages gradually and observe behavior closely.

That observed part matters. Look at error rates, latency changes (beyond normal platform warmup), resource spikes, and unexpected retries. All of these indicate customer impact.

Canary deployments are one of the best ways to reduce the blast radius of changes, identify problems quickly, and self-correct.

Shadow Traffic 🪞

Traffic mirroring sends production traffic to a new version before routing live traffic there.

Responses are ignored, but you observe behavior and monitor the same signals you would with a canary release without sacrificing a customer request.

Synthetic Traffic 🤖

Synthetic traffic simulates user behavior continuously. It’s great for monitoring customer experience, but also a great way to validate new deployments.

Route synthetic traffic to upgraded instances first and verify behavior before moving real traffic. If it fails with synthetic traffic, it likely won’t survive real traffic.

Smoke Tests 😶‍🌫️

The classic approach. After deployment, run a small set of fast tests to confirm the platform is fundamentally working.

Smoke tests don’t need to be fancy; they can be shell scripts, API calls, read-only requests, a test file, or full end-to-end validation.

Their purpose is simple: to quickly catch obvious breakage.

🧠 Final Thoughts

Don’t think of the above methods as mutually exclusive choices. Combine them.

Some platforms I work on combine canary releases, shadow traffic, and synthetic traffic. Others use smoke tests plus canary releases.

The more layers of validation you have, the more likely you are to catch issues before your customers do. Because having your customers validate changes for you is a poor strategy.

It is a foundational technique used by many modern databases, caches, and large-scale platforms. Understand how it works and you can apply the same pattern in your own systems.

🤔 Understanding the Problem

At some point, every successful system hits the limits of a single database instance.

A single server can only handle so many connections, queries, writes, storage capacity, or CPU/memory demands. Even with the best hardware, performance eventually degrades. So systems scale horizontally.

Instead of sending all traffic to a single database server, requests are distributed across multiple nodes.

At the same time, resiliency matters. If one server fails and all data resides there, the outage can be severe.

So modern databases spread data across multiple nodes, availability zones, and regions.

Distributing load and data solves both capacity and resiliency problems. But it introduces another challenge.

How do you keep request behavior consistent when data is distributed across multiple systems?

⚠️ Why Replication Is Not Enough

Replication helps, but it does not solve every consistency problem.

Imagine a write lands on Server 1. Immediately after, a read request for the same data lands on Server 67. Will Server 67 have the latest version? Maybe, but often not.

Asynchronous Replication

With asynchronous replication, Server 1 will accept the write and replicate the data to other servers in the background. That means a follow-up read on any other node may return stale data.

Synchronous Replication

With synchronous replication, the write on Server 1 will wait for an acknowledgment from all replicas before returning a success. While this improves consistency guarantees, it increases latency.

The farther apart a replica is, the worse this gets. Local writes may be fast, but cross-region writes will be slow. Plus, is it really feasible to replicate data across every single node?

So the question becomes: How do you preserve consistency, without paying latency taxes?

🔀 Route Requests to the Data

A highly effective answer is deterministic routing.

Instead of moving data to where requests might land, move requests to where the data already exists.

If requests for the same key can go to the same node, you gain predictable ownership, reduced stale reads, lower coordination overhead, and easier horizontal scaling.

👨‍🏫 How Deterministic Routing Works

At a high level, the system needs a repeatable way to decide where requests should go.

A common approach is hashing.

• A hash of user123 always goes to Node 7
• A hash of user456 always goes to Node 42

As long as the same key produces the same result, requests can be consistently routed to the same owner. Many modern databases implement deterministic routing through techniques like consistent hashing, partition maps, and shard ranges.

🗺️ Where Routing Logic Lives

Different systems solve routing in different places.

Client-side Routing

The client library knows the partition map and sends requests directly to the correct node. Used by many distributed caches and databases.

Proxy / Router Tier

A small router sits in front of nodes and forwards traffic appropriately. Useful when client behavior cannot be influenced.

Server-side Forwarding

Requests land anywhere, and the receiving node forwards internally to the owning node. Simple for clients, doesn’t introduce a proxy failure point, but introduces complex cluster discovery/health monitoring.

Each model has tradeoffs.

🧰 Routing Does Not Replace Replication

Deterministic routing is powerful, but not magic. What happens when the owning node is down? You still need replication.

Modern databases combine both: deterministic routing for performance and ownership, plus replication for durability and failover.

🧠 Why This Matters Beyond Databases

Distributed databases use this approach, but it is not unique to them.

Deterministic routing can be used to solve: session ownership, user affinity, in-memory workflow coordination, work queue partitioning, and more.

I’ve used deterministic routing many times to solve load distribution and consistency problems.

At scale, the answer is not always more/better hardware. Consistency and availability problems are not always solved with replication alone.

Sometimes the best answer is simply to send the request to the right place.

🧩 How It Usually Works

Most microservice designs follow the same model:

• Break systems into capabilities, teams, or functions
• Deploy one shared service for each capability
• Any platform that needs it calls that centralized service

That works well for many cases, but it’s not the only model.

🏗️ How We Got Here

Before microservices, many organizations used Service-Oriented Architecture (SOA).

Despite being labeled as antiquated, SOA and microservices are not that different. Both break down systems into capabilities that communicate with each other. The biggest difference is scope.

In SOA, a “Payments Service” might own:

• Message parsing
• Validation
• Balance checks
• Currency conversion
• Settlement logic

While other SOA services would own “Users” or “Accounting”. Today, that payment service would be considered an entire platform, with each of those capabilities implemented as microservices within that domain.

Microservices are often the same idea as SOA, just at a more granular level.

🎯 Why Centralization Became the Default

One reason microservices gained traction was the need to avoid duplication. Capabilities were often rebuilt across multiple systems. For example, Currency Conversion is needed in Payments, Accounting, and many other platforms.

Duplication is not just wasteful, it creates real problems: logic drift, coordination overhead, and inconsistent outcomes across systems. Packaging that capability as a standalone service solved real problems: build once, reuse everywhere.

⚠️ The Downside of Centralization

In cell-based architectures, platforms are usually designed to be self-contained and failure-isolated. That means a mission-critical platform depending on a centralized service shared by other platforms can become a design smell.

• Cross-cell dependencies
• Added latency
• Shared failure domains
• Complex failover scenarios

So teams, once again, solve these problems by rebuilding the same capability locally.

🔁 Another Option

Instead of rebuilding the capability each time, deploy the same microservice codebase inside multiple platforms. If both Payments and Accounting need a currency conversion service, deploy the same service within each platform.

It’s the same codebase and capability, but with local ownership and resilience. You get reuse without forced centralization.

🧪 Caveats from Experience

This pattern works when applied carefully.

1️⃣ Strong Ownership

A shared codebase still needs a clear owning team. Others can contribute, but someone must own quality, roadmap, and releases.

2️⃣ Pick the Right Capabilities

Not everything is a great fit. Something like currency conversion is well-scoped, relatively stateless, and doesn’t have unique business logic based on which platform is calling it. It’s a strong example.

But other services that have unique logic for each platform domain or require consistency across different platforms are less of a fit.

3️⃣ Operational Discipline

Using the same codebase doesn’t automatically solve all problems; you can still run into drift across platforms if each is running a different version. Changes in behavior still sometimes need coordination.

But with a single codebase, these issues are far easier to address.

💭 Final Thoughts

Microservices gave us reusable building blocks. Sometimes the best use of a microservice is not one centralized deployment. Sometimes it’s many local deployments of the same capability.

Just reuse the software while maintaining autonomy.

🚦 What is Traffic Mirroring?

Traffic mirroring in Istio or Envoy Proxy lets you send a copy of live traffic to a secondary target.

When enabled, traffic to /service routes to cluster1 as normal, and a mirrored copy is sent to cluster2.

The key: mirrored traffic is fire-and-forget. Responses are ignored and never impact the primary request.

🧪 Why It’s Powerful

1️⃣ Shadow Traffic for Safe Testing

The most common use case is shadow traffic.

When migrating platforms or deploying a new version of an application, you can send real traffic to the new system, observe behavior, and validate responses.

All without impacting users. No risky cutovers. You see exactly how the new system behaves under real load.

2️⃣ Out-of-Band Traffic Inspection

Another powerful use case is traffic inspection.

Inline inspection is risky. It adds latency, introduces new failure points, and becomes part of the critical path.

With traffic mirroring, you can inspect traffic, analyze requests, and detect anomalies.

All without impacting the primary path.

😶‍🌫️ Reality Check

It’s not perfect. There is some overhead.

Mirroring adds load to the sidecar, which may or may not be acceptable for your system. In my experience, it’s negligible, but it’s something you should measure in your own environment before deploying to production.

🧠 Final Thoughts

Traffic mirroring is one of the safest ways to validate migrations, test new systems, and observe real production behavior.

The hard part isn’t mirroring traffic. It’s running two production systems in parallel. That’s the real cost, and the real tradeoff.

But if you can afford that cost, traffic mirroring is an incredibly powerful tool.

If you want to dig deeper:

• Istio traffic mirroring docs explain the workflow.
• Envoy request mirror policy docs cover the lower-level routing behavior.

Every organization has institutional knowledge.

• Internal frameworks
• Preferred practices
• Platform-specific capabilities

It exists everywhere. But it’s often undocumented… or buried in a wiki no one reads.

As coding agents take on more work, this problem gets worse.

If you ask an agent to build a new service, you want it to use your internal framework, follow your patterns, and respect your organizational constraints.

A human engineer would ask questions. An agent won’t, unless you give it that context.

📚 Agent Skills as Knowledge Distribution

Most people think about Agent Skills as actions:

• Convert markdown to PDF
• Review this pull request
• Commit my changes

But the more interesting use case is guidance.

Skills aren’t just for doing things. They’re for shaping agent output.

Agents discover and use skills based on intent.

If a user asks: “Create a new Python service.”

The agent looks for relevant skills:

• Language conventions (PEP 8, etc.)
• Internal frameworks
• Organizational standards

That’s where institutional knowledge belongs.

Instead of hoping engineers remember to tell the agent:

• “We use Flask, not Django.”
• “Stick to the standard library.”
• “Follow this service layout.”

You capture that into a skill. The agent applies it automatically.

🧠 Why This Matters

Institutional knowledge only works if it's:

• Discoverable
• Applied consistently

Agent Skills give you both.

They turn tribal knowledge into something agents can find, understand, and use.

⚠️ The Tradeoff (For Now)

Right now, this introduces duplication.

Most teams already have internal docs, style guides, & wikis.

And now you’re putting the same information into skills. Which feels like extra work.

But it poses an interesting question:

As agents become the primary interface… Will engineers read the wiki? Or ask the agent?

🧠 Final Thoughts

As agents take on more of the implementation work, where you store knowledge becomes more important. Making that knowledge accessible to agents becomes essential.

Agent Skills aren’t just automation tools.

They are becoming the interface for standards, practices, and institutional knowledge.

And teams that embrace that early will see more consistent output from both humans and agents.

If you’ve been around for a while, you probably have a file full of bash one-liners.

Small scripts or commands you saved because they solved a problem you didn’t want to automate properly.

When coding agents arrived, prompts became the new one-liners.

Useful prompts were saved, reused, and eventually turned into “prompt files”, then slash commands like /do-something.

But that model has already evolved.

⚙️ Agent Skills

Agent Skills are the next iteration.

At a basic level, a skill looks a lot like a saved prompt: a directory with a markdown file.

What makes it different is how it’s used.

Skills include metadata like name and description, allowing agents to discover them.

Instead of explicitly calling a prompt every time, the agent can determine when to use a skill based on intent.

This is referred to as progressive disclosure:

• Agent loads skill metadata
• Matches it to your task
• Then loads and executes the full skill when needed

You can still call skills directly (/, $, @), but you don’t always have to.

🧠 More Than Just Prompts

The real differentiator is that skills aren’t just prompts.

They can include reference documentation, templates, and scripts.

This means you’re no longer just telling the agent what to do.

You’re giving it tools and context to execute and validate tasks.

For more complex workflows, it’s often easier to write a script and teach the agent how to use it than to encode everything in a prompt.

⚠️ A Word of Caution

This power comes with risk.

Skills can include executable logic and tell agents to perform tasks.

That means a shared skill can contain malicious or unsafe behavior.

Treat them like any script you install:

• Understand what they do
• Know where they come from
• Review before using (watch out for hidden text or obfuscated instructions)

🧠 Final Thoughts

Agent skills are a meaningful step forward.

They let you codify workflows, preferences, and repeatable agent tasks in a way that agents can discover.

They’re a strong productivity accelerator and a powerful way to capture institutional knowledge in a form agents can actually use.

(More on that in the next post.)

Software engineering has never really been about writing code. Coding is often the easy part.

Testing is harder, and many teams struggle with it.

As tools make it easier to generate code quickly, that gap widens. If you can produce changes faster than you can validate them, you eventually create more code than you can safely operate.

Which begs the question: What does good testing actually look like?

🔍 What Good Looks Like

One of the biggest challenges I see is that teams struggle to understand what “good” testing means and never define it.

Pipelines are often built early in a project, when the team is small, and they rarely keep pace with the system and organization as they grow.

My starting principle is simple:

• At pull request time, you should have strong confidence that the change will not break the service or platform being modified.

• Within a day of merging, you should have strong confidence that the change hasn’t broken the full customer journey that the platform supports.

🔁 On Pull Request

For backend platforms, I like to see three levels of automated testing before merging.

Code Tests (Unit Tests)

This level is the foundation. Unit tests validate internal logic, error handling, and edge cases. Techniques such as fuzz testing and benchmarking also reveal issues early. As the test pyramid tells us, this is where the majority of testing and logic validation should take place.

Service-Level Functional Tests

Too many teams stop at unit tests for pull requests. Functional tests should also be run in CI for every pull request.

Services should be tested in isolation with functional tests. Dependencies can be mocked, but things like databases should ideally run for real (Dockerized).

This is where API contracts are validated and regressions can be identified without wondering whether the issue came from this change or another service.

Platform-Level Functional Tests

Testing a service alone isn’t enough. Changes can break upstream or downstream dependencies. Platform-level tests spin up the entire platform in CI and validate that services interact correctly.

These tests ensure the platform continues to work as a system.

For platforms with strict latency or resiliency requirements, I recommend introducing light stress tests at both the service and platform levels. These aren’t full performance tests, but they act as early indicators of performance regressions.

If these three layers pass, you should have high confidence in the change. But not complete confidence.

🌙 Nightly Testing

Some failures take time to appear.

Memory leaks, performance degradation, and cross-platform integration issues may not show up immediately.

That’s why I like to run a nightly build (or every few hours).

This environment runs end-to-end customer journey tests, performance tests, and chaos tests.

These are typically the same tests used during release validation, but running them continuously accelerates feedback. If something breaks, you learn about it early, before the pressure of a release.

🧠 Final Thoughts

There is no universal approach everyone can follow.

Different systems have different needs; mission-critical systems may focus heavily on correctness and resilience. Non-mission-critical systems may focus more on validating core functionality.

Your testing strategy depends heavily on architecture, dependencies, and operational constraints. But if your organization is increasing its ability to generate code quickly, your testing capabilities must evolve at the same pace.

AI-generated code becomes much easier to review when you already have high confidence in your testing.

I was recently talking with another engineer who is rolling out gRPC into production. He asked what the biggest gotchas were.

My first answer: Load Distribution.

🚦 HTTP/1 vs. HTTP/2

Most teams first implement services using REST over HTTP/1 and then migrate to gRPC as they seek its performance benefits.

That shift introduces a subtle but important change in how traffic gets distributed across instances.

With HTTP/1, requests are generally tied closely to connections. A client opens a connection, sends a request, waits for the response, and then sends another (if connection re-use is enabled).

HTTP/2 (which underpins gRPC) works differently.

HTTP/2 multiplexes requests over persistent connections. A client can send many requests over the same connection without waiting for responses.

This is one of the reasons gRPC provides a performance boost, but it can create unexpected load distribution issues.

If your infrastructure isn’t built for an HTTP/2 world, you’ll quickly find traffic becoming unevenly distributed.

🏗️ Infrastructure Support

In an HTTP/1 world, load balancing at the connection (Layer 4) level often works well enough. But with HTTP/2, connections live much longer and carry far more concurrent traffic.

If your load balancer distributes traffic based only on connections, a busy client may hammer a single instance while others sit idle.

Unfortunately, much of the infrastructure still doesn’t fully support HTTP/2-aware load balancing.

Depending on your environment, your load balancers or ingress controllers may operate primarily at Layer 4. That works fine for HTTP/1, but once you introduce HTTP/2 via gRPC, the effectiveness changes significantly.

⚙️ Supporting gRPC

To get the most out of gRPC, the best approach is to use infrastructure that understands HTTP/2 and load-balances requests rather than just connections.

If that’s not possible, another option is client-side load balancing.

Many gRPC clients support opening a pool of connections and distributing requests across them. You still benefit from HTTP/2’s persistent connections, but you avoid concentrating all traffic on a single backend instance.

🧠 Final Thoughts

gRPC offers many advantages, including performance, strongly typed contracts, and efficient communication. But it also introduces different networking behavior.

If you’re rolling out gRPC into production, make sure your load balancing infrastructure is ready for an HTTP/2 world.

A concept that took me a while to really grasp is that building highly available systems and highly resilient systems is not the same thing.

The difference is how the system reacts to failure.

🚄 High Availability

When you build for high availability, the goal is simple: ensure there is always another path.

If something fails, traffic can be redirected somewhere else.

For example, a service might run across multiple availability zones or regions. If one fails, traffic is routed to another.

Detecting failures and redirecting traffic are core elements of building for high availability.

Availability is about rerouting traffic when something fails.

🚂 High Resiliency

Building for resiliency is different.

The solution to failure isn’t another path; it’s how the system handles the error.

When a dependency fails, the decision becomes:

Do we retry? Do we continue without that dependency? Do we degrade functionality? Do we stop processing altogether?

Resiliency is about defining what happens when things go wrong.

Sometimes you can continue processing. Sometimes you can defer work and fix it later.

Resiliency is absorbing failure instead of avoiding it.

🧩 A Simple Example

When you design systems with resiliency in mind, you tend to treat dependencies differently.

A simple example is configuration.

Many systems use distributed configuration services so that runtime behavior can change without redeployment.

But that configuration service then becomes a dependency. To avoid turning it into a hard dependency, many systems cache the configuration in memory.

When updates occur, the system fetches the new configuration and switches only after it’s fully loaded into memory.

If configuration refresh fails, the system continues operating with the last known configuration. Transient failures don’t bring the system down.

That’s resiliency.

🧠 Final Thoughts

When I talk about non-functional requirements, you’ll hear me say:

“Highly available and resilient systems”

I separate them intentionally because the approaches are different.

Availability ensures there is always another path. Resiliency ensures the system can continue operating when failures occur.

Availability routes around failure. Resiliency survives failure. You need both.

Junk in, junk out.

One of the best ways to get more from agentic coding tools is to give the agent context.

The more an agent understands your project, the better its work will be.

If you ask an agent to add a method to a class, it will. It might read the file. It might infer some structure. But it won’t understand the project's intent.

If you asked a human engineer to make the same change, they would have questions.

What is the purpose of this project? How is it used? What constraints exist?

If they skipped that step, you’d get exactly what you asked for, even if it was wrong.

That’s the same challenge many face with coding agents. A lack of context means it only does what it’s told — which isn’t always what you actually need.

But when it understands a project, it operates with far more clarity.

🧙‍♂️ My “Old School” Method

Before I start serious work with an agent, I have it learn the project.

Read the docs 📚 Review the codebase ⚙️ Understand the architecture 🏙️ Learn how to build, test, and run the project locally 👩‍🔧

I even ask the agent to summarize its understanding back to me.

This started as a saved prompt, turned into a slash command, and is now a skill.

This step is a huge productivity boost.

🤖 Agents Files (AGENTS.md)

Over the past year, an open standard for providing agents with structured context has emerged.

Instead of prompting the agent to rediscover your project every time, document that context once — and the agent will reference it going forward.

Most modern agents support an Agents.md file and reference it during each interaction.

💽 What Goes in an Agents File?

Think of the Agents file as onboarding documentation, but for an agent.

Project context:

• Purpose
• Architecture
• Layout
• CI/CD instructions

Team context:

• Code style preferences
• Testing philosophy (TDD or YOLO)
• Tech stack constraints

Any tribal knowledge you’d expect a new team member to learn belongs in an Agents file.

👨‍💻 Personal Agent Files

Many tools also support a personal Agents file in your home directory.

That’s where your workflow preferences live. Are you a two-space tabs person? Do you want your agent to prefer table tests?

If you have preferences you want to apply to every project, but are unique to you, they go in the personal Agents file.

🧠 Final Thoughts

Using an Agents file dramatically improves agent quality.

Even then, I still use my “learn-this” slash command — sometimes that extra context makes a difference.

If you wouldn’t drop a new engineer into a project without context, don’t do it to your agents.

I was listening to a team recently, and an engineer was discussing how a coding agent added additional tests to a project that already had 100% code coverage.

The conversation reminded me that coverage is directional and often mistaken for quality. Just because your coverage shows 100% doesn’t mean your software is fully tested.

👨‍🏫 Understanding How Coverage Is Measured

Code Coverage measures the percentage of executable lines that run during code tests. Executed doesn’t mean well-tested.

Just because every function runs doesn’t mean it’s free of logic errors or safe.

😃 Happy Path Testing

A common challenge teams face with testing is focusing too much on the happy path.

Suppose you have a function that accepts an array. In your tests, you always pass 5 elements — because that’s the expected usage. Coverage shows all branches executed. You’re good, right?

What happens if you pass 4 elements? Or 0?

If you never test fewer than 5, how do you know? You may say: “But wait, it’s only ever called with 5 elements.” That may be true, for now.

⚠️ Protecting Against Your Future Self

Code is rarely static; someone will come along and change things. That might be you, it might be someone else.

Eventually someone changes that function. Will they add tests for new edge cases? Maybe. Assume they won’t.

When you write tests, don’t just focus on how you know a function is going to be used; also include tests that misuse the function.

Rather than sending an array with 5 elements, send one with 4, 0, and send a nil value.

Rather than sending strings that match an expected pattern, send junk that doesn’t.

Does the function still behave correctly? Should it?

The more you test outside the happy path, the more resilient your code becomes — and the less likely it is to break later.

🧠 Final Thoughts

Code coverage is a guide, don’t let it give you false confidence. Test the happy path, and the unexpected ones. Validate function outputs against the input you provide.

100% Coverage is easy. Writing reliable code is not.