Junk in, junk out.

One of the best ways to get more from agentic coding tools is to give the agent context.

The more an agent understands your project, the better its work will be.

If you ask an agent to add a method to a class, it will. It might read the file. It might infer some structure. But it won’t understand the project's intent.

If you asked a human engineer to make the same change, they would have questions.

What is the purpose of this project? How is it used? What constraints exist?

If they skipped that step, you’d get exactly what you asked for, even if it was wrong.

That’s the same challenge many face with coding agents. A lack of context means it only does what it’s told — which isn’t always what you actually need.

But when it understands a project, it operates with far more clarity.

🧙‍♂️ My “Old School” Method

Before I start serious work with an agent, I have it learn the project.

Read the docs 📚 Review the codebase ⚙️ Understand the architecture 🏙️ Learn how to build, test, and run the project locally 👩‍🔧

I even ask the agent to summarize its understanding back to me.

This started as a saved prompt, turned into a slash command, and is now a skill.

This step is a huge productivity boost.

🤖 Agents Files (AGENTS.md)

Over the past year, an open standard for providing agents with structured context has emerged.

Instead of prompting the agent to rediscover your project every time, document that context once — and the agent will reference it going forward.

Most modern agents support an Agents.md file and reference it during each interaction.

💽 What Goes in an Agents File?

Think of the Agents file as onboarding documentation, but for an agent.

Project context:

• Purpose
• Architecture
• Layout
• CI/CD instructions

Team context:

• Code style preferences
• Testing philosophy (TDD or YOLO)
• Tech stack constraints

Any tribal knowledge you’d expect a new team member to learn belongs in an Agents file.

👨‍💻 Personal Agent Files

Many tools also support a personal Agents file in your home directory.

That’s where your workflow preferences live. Are you a two-space tabs person? Do you want your agent to prefer table tests?

If you have preferences you want to apply to every project, but are unique to you, they go in the personal Agents file.

🧠 Final Thoughts

Using an Agents file dramatically improves agent quality.

Even then, I still use my “learn-this” slash command — sometimes that extra context makes a difference.

If you wouldn’t drop a new engineer into a project without context, don’t do it to your agents.

I was listening to a team recently, and an engineer was discussing how a coding agent added additional tests to a project that already had 100% code coverage.

The conversation reminded me that coverage is directional and often mistaken for quality. Just because your coverage shows 100% doesn’t mean your software is fully tested.

👨‍🏫 Understanding How Coverage Is Measured

Code Coverage measures the percentage of executable lines that run during code tests. Executed doesn’t mean well-tested.

Just because every function runs doesn’t mean it’s free of logic errors or safe.

😃 Happy Path Testing

A common challenge teams face with testing is focusing too much on the happy path.

Suppose you have a function that accepts an array. In your tests, you always pass 5 elements — because that’s the expected usage. Coverage shows all branches executed. You’re good, right?

What happens if you pass 4 elements? Or 0?

If you never test fewer than 5, how do you know? You may say: “But wait, it’s only ever called with 5 elements.” That may be true, for now.

⚠️ Protecting Against Your Future Self

Code is rarely static; someone will come along and change things. That might be you, it might be someone else.

Eventually someone changes that function. Will they add tests for new edge cases? Maybe. Assume they won’t.

When you write tests, don’t just focus on how you know a function is going to be used; also include tests that misuse the function.

Rather than sending an array with 5 elements, send one with 4, 0, and send a nil value.

Rather than sending strings that match an expected pattern, send junk that doesn’t.

Does the function still behave correctly? Should it?

The more you test outside the happy path, the more resilient your code becomes — and the less likely it is to break later.

🧠 Final Thoughts

Code coverage is a guide, don’t let it give you false confidence. Test the happy path, and the unexpected ones. Validate function outputs against the input you provide.

100% Coverage is easy. Writing reliable code is not.

Software engineering is changing fast.

Agentic coding tools became widely available last year, and if you’re not using them today, you’re already behind. But many still struggle to move beyond the “fancy chat” experience.

Just like any tool in our engineering tool belts, knowing how to use it effectively matters.

🤖 Agents Are More Than A Better Chat

Last year, most were using tab-complete with a useful chat interface where you could ask questions, get suggestions, and maybe copy/paste into your code.

But agents can do much more than make suggestions — they can understand your codebase and act.

Instead of asking an agent:

“Can you suggest additional tests?”

Tell your agent:

“Create additional test cases, then run make tests and validate they pass.”

An agent can create tests, run them, inspect failures, adjust the implementation, and re-run the suite until it passes.

This isn’t about suggestions anymore; agents have more autonomy.

I think of coding agents as assistants working toward a shared goal. They do some work, you do some, and you iterate together.

🏆 Moving from Direction to Outcomes

A big mental shift is moving away from simple directions to defining an outcome with guidance & guardrails.

Agents don’t just perform a single task; they can execute multiple steps (and even parallelize them). You don’t need to spoon-feed each directive one by one.

Instead, define the outcome you want, along with guidance and guardrails.

The clearer you are on the outcomes, constraints, and context around what you are trying to do, the better the agent will perform.

📋 Examples: Real-world tasks I’ve asked Agents to handle

“Using the existing DB Driver X as a reference, create a set of table tests for driver Y. The tests should be structured similarly to the existing driver, surface any logic issues, concurrency issues, and act as a clear insurance against the defined interface.”

“Update CI workflows to Go 1.26.0, find and update any references to 1.25.6, then run tests to ensure everything still builds and passes”

I also use agents for mundane work like git commits and opening pull requests. They consistently produce better commit messages and PR descriptions than I would.

Agents don’t always get it exactly right, but with a bit of feedback and occasional adjustment, you can get a lot done quickly.

Avoid going down the rabbit hole of endless refinement, sometimes it’s better to reset with a clearer prompt.

👨‍🏫 Context is Key

If you want the best results from agents, you need to give them context.

Before I do serious work on a project, I have the agent:

• Read the Docs 📚
• Review the Architecture 🏙️
• Understand the Project Structure 📐
• Understand how to build, test, and run the application locally 👩‍🔧

The same steps that a human would take. Agents are no different.

(I’ll dive deeper into Agent files, skills, and effective ways to provide more context in a future post)

🧠 Final Thoughts

Engineers are doing amazing things with agents, and new capabilities are being added daily. But you don’t need to be at the bleeding edge to get more out of them (I certainly am not).

Don’t worry about the hype. Understand what these tools can do, making small adjustments in how you use them can drastically change what you get back.

I’ve worked on many systems in my career, and one thing that I’ve noticed is that those that leverage infrastructure-as-code tend to be more stable than those that don’t.

🤔 But wait, isn’t everyone using IaC these days?

You may be thinking, "Why am I talking about IaC in 2026? Isn’t this just the de facto standard at this point?"

My hope is yes, everyone does this, but I’m sure many don’t invest the time into it.

I’m not here to tell you to use IaC; I’m here to tell you why it’s important, and it’s not necessarily about the speed of deployment.

🏎️ Fast is great, but it’s not the biggest benefit

A very clear and correct reason people leverage IaC is the speed of infrastructure provisioning.

It’s much faster to provision infrastructure with IaC; it takes less time, enabling you to scale faster, and it lets you do cool things like ephemeral environments.

But the biggest benefit of IaC, in my mind, is correctness.

⚠️ IaC reduces human error

Humans make mistakes. When you ask humans to click the same buttons in the same sequence every time, you’ll get mixed results.

Steps get missed — especially when time passes or people rely on memory instead of process.

Documentation helps, but there are those of us who think, “I’ve done this a million times, I don’t need instructions.”

This attitude is the same reason one of my kid’s desks wobbles and the other one doesn’t…

IaC is a contract. Once defined, every environment is created from the same source of truth.

✅ Consistency is essential to production stability

The consistency of IaC is what brings production stability.

When your performance testing environment matches production, your tests become more accurate.

If one service has a larger memory footprint in testing than it does in production, you might find yourself surprised by out-of-memory errors, especially if heap sizes are configured based on your test environment and not your production environment (because, of course, they would be the same, right?).

When I come across platforms that use IaC, I see fewer mistakes and fewer incorrect assumptions. And production tends to be more stable, at least with respect to infrastructure and capacity-related issues.

🧠 Final Thoughts

So, to answer the question, why is IaC so important? It’s not the speed of provisioning; it’s the correctness of the environments.

In production systems, correctness beats speed every time.

I work with and talk to a lot of engineers, and to explain my point, I’ll describe two engineers on the same team.

💪 Engineer 1

The first engineer churns out a lot of code and user stories. They’re focused, consistently finishing on time, and often doing more than they’re assigned.

When it comes to shipping business features, this person does a great job.

But this person is also more than happy to let their build run for 3 hours.

🦾 Engineer 2

The second engineer completes their assigned user stories, but when they encounter inefficiencies, they spend time fixing them. Sometimes it’s improving the build pipeline, fixing flaky tests, making code more maintainable, etc.

While this engineer may finish fewer user stories because they are distracted by these “side quests,” they make a bigger impact.

🏋️ Enabling Others

While avoiding the 10x engineer trope, Engineer 2 has a bigger impact by resolving issues affecting the whole team.

A slow pipeline slows everyone’s work.

Open a single change, then wait 3 hours. A test fails—wait another 3 hours. Feedback comes in—wait 3 more.

Broken workflows turn simple changes into long, inefficient endeavors.

So fixing these not just for themselves but for everyone means the whole team can ship code faster.

📈 Invest in Workflows

Investing time in optimizing your workflow and the team’s workflow usually pays dividends.

Sometimes it’s hard to quantify, but the smallest optimizations can be huge.

Someone on the team who gets frustrated with inefficiencies and decides to fix them is incredibly valuable.

👩‍🔧 Do you take ownership of your codebase?

If you want to make a greater impact, look at how you work.

When you fix a bug, do you search the codebase for the same bug elsewhere?

When your build pipeline is slow, or you have flaky tests, do you fix them or live with them, complaining while nothing changes?

Over time, everyone (including yourself) forgets the original intention of the system's design as new requirements emerge.

This law applies at all levels, from system design to microservices, or even libraries.

🧬 Systems Evolve (and Intent Fades)

When building new platforms/services/whatever, we create a system design that follows a structure.

Different components have distinct responsibilities; they interact clearly with the rest of the system, and there is a plan.

But as time progresses, new people may not understand the original intentions of the design.

As new requirements come in, the pressure to deliver may push you or others down a path that doesn't align with the original plan.

When the architecture’s intent is understood, additions can be beneficial. When it’s forgotten, they start to feel duct-taped on.

Duct-taped solutions turn into technical debt or operational/management complexity that starts to weigh the system down.

📠 How Good Systems Become Legacy Nightmares

We've all seen the legacy platform that feels brittle, does too much, and is daunting to refactor.

It didn't start that way.

At the time, it was probably a great design, but over time, new features and capabilities turned it into Frankenstein's monster.

👮 How to Defend Architecture from Collective Amnesia

While it may not be possible to prevent the system from devolving forever, you can reduce the need for duct tape solutions by designing for change.

📜 Roles and Responsibilities

An important—but not always effective—step is to document and define the roles and responsibilities of components within the system.

When a system is broken down into components with distinct roles and responsibilities, it becomes easier for people to make informed decisions about where new capabilities should reside.

The documentation “should” influence how change is implemented.

But it relies on people following that documentation, which is the fundamental flaw.

🚧 Architectural Guardrails: Make the Right Path the Easy Path

When I say "architectural guardrails," you probably think of review boards and ADRs. These processes are essential, but they don't always work as a prevention.

Instead, I mean designing the system so that the correct placement of functionality is the path of least resistance.

🔏 Contracts as Constraints, Not Convenience

In general, I feel like back-end APIs should provide as much data as possible, and it should be up to the clients to use what's relevant.

But sometimes contracts can be used to enforce design behaviors.

Systems can't act unless they receive the data required to act.

🚪 Control Ingress and Egress to Control Evolution

Ensuring that only specific systems serve as entry and exit points helps direct future design decisions.

It's often easier to add a new endpoint than to add a new platform that serves as an entry point.

Knowing this can allow you to put in place processing at those entry and exit points that ensure future capabilities follow specific patterns.

🧩 Design for Change, Not Today’s Requirements

When you are first building a system, it's easy to want to make it quickly based on the requirements in front of you.

But when you know a platform will evolve, it's beneficial to take time and implement interfaces that make the system more modular.

Within a microservice, this can be how you structure the application, how you create packages that can be extended even though you don't need them day one.

At a platform level, it could be the decision between monolith and microservices. If you know there will be a rapid change, it may make sense to leverage microservices. If you know there won't be a fast change, start with a monolith.

🧠 Final Thoughts: Assume Intent Will Be Forgotten

The above examples are just a subset of the ways you can enforce a design that aligns with your intentions.

The key lesson: don't build a plan that relies on people to follow your intentions. They won't.

You have to assume the next person won't design systems the way you do, they won't understand the reasons behind your design, and they'll be under pressure to deliver.

Did you win or did you stop early?

I previously shared my thoughts on benchmark and endurance tests, but before ever running a test, a target must be defined.

🎯 Why Set Targets?

Without a target, how do you know what good looks like?

I've often come across teams that have incorporated performance testing into their releases (which is excellent). But they had no targets defined.

No production baseline.

No service-level objectives from the business.

How did they know whether the system was meeting expectations? They didn't.

In some cases, after targets were defined, the system was performing as needed.

In others, it clearly wasn't, and the team had no idea until targets were defined and compared with production.

🏆 Defining Targets

It's easier to define targets for existing systems (and modernization projects) than for a brand-new system.

Existing platforms have production numbers you can reference, user expectations, and service-level objectives that can be translated into performance targets.

New systems rarely have much to baseline from.

For a brand-new system, I like to work with the product/business team and understand their goals.

- 📈 What is the expected growth? Slow and steady, or fast and unpredictable?

- 🚨 What is the criticality of the platform? If it fails to respond, is it a problem or an inconvenience?

- 🌟 What unique constraints or features of the platform might influence performance requirements?

Once defined, targets should not be treated as static.

As traffic starts, you can adjust targets accordingly. Maybe it's higher, perhaps it's lower.

🪫 Leave Some Buffer

Once a target is agreed upon, I like to add a bit of buffer.

If the requirement is 100ms, I’ll target closer to 75ms, or lower, depending on the system and its purpose.

Why? Adding capacity or tuning the system takes time.

Things change, sometimes in unexpected ways.

Sometimes unexpected changes can be handled by automatic/manual scaling, but not always.

It's important to give yourself a bit of buffer to respond to those changes.

🧠 Final Thoughts

I've talked a lot about setting targets and their importance. But one of the most important aspects of having targets is monitoring and measuring production.

Having visibility in production helps validate that your targets are realistic.

Maybe they are too high, and you have wasted infrastructure reserved.

Perhaps they are too low, and you won't be able to survive the next traffic spike.

Traffic changes over time, and application performance naturally drifts as new capabilities are added.

Clear visibility into traffic and latency patterns is essential for anyone operating mission-critical, large-scale systems.

But also a foundational practice for most platforms.

Do you have performance targets for your platform? Is it grounded in production measurements? Should you?

The approach I’ve found most effective is to split performance testing into two distinct categories:

• 🏋️‍♀️ Benchmark testing
• 🚣‍♀️ Endurance testing

Both stress the system, but they answer different questions.

🏋️‍♀️ Benchmark Testing:

Benchmark tests are where most teams start: increasing load until the system fails.

Failure might mean:

• ⏱️ Latency SLAs are exceeded
• ⚠️ Error rates cross acceptable thresholds

Sometimes failure is measured by when the system stops responding entirely. This is known as breakpoint testing.

Even when SLAs are the target, I recommend running breakpoint tests after thresholds are exceeded.

Knowing how the system breaks under load is useful when dealing with the uncertainties of production.

🚣‍♀️ Endurance Testing:

Endurance tests answer a different question:

Can the system sustain high load over time?

Running at high but realistic levels (often near production max) over extended periods exposes different problems:

• 🪣 Queues, file systems, and databases slowly fill
• 🧹 Garbage collection and thread pools behave differently
• 🧵 Memory or thread leaks become visible

These issues rarely show up in short spikes of traffic. If you only run benchmarks, you’ll discover them for the first time in production.

⌛️ Testing Thoroughly vs Deployment Speed:

Benchmarks run fast; Endurance testing takes time.

A 24-hour endurance test can slow down releases, especially when you want to release the same service multiple times a day.

It's a trade-off between the system's criticality and the need for rapid deployments.

How tolerant is the system to minor performance regressions?

If performance truly matters, slowing releases down to run endurance tests might be the right call.

🧠 Final Thoughts:

Effective performance testing isn’t just about surviving spikes.

Spikes matter, but so does answering:

• 📈 Can the system withstand peak load for extended periods?
• 🔎 If not, how does it fail, and why?

All too often, I see the system's capacity become the breaking point during unexpected traffic patterns.

While an application might handle spikes, the overall platform often can't sustain them. That's where endurance tests deliver their real value.

It works, but it adds complexity, which is why most teams avoid it.

📖 Context

For context, in this post, I’m talking about scenarios where one system requires data from another system, I.E., the source of record (SOR). The data is needed frequently, and the decision to cache has already been made.

A good traditional approach is the cache-aside pattern, which maintains a local cache of data.

That cache is populated organically by checking for records as needed, finding that the data is not cached, fetching it from the SOR, and storing the result.

A pro of this approach is that the cache is transient. If it's dropped, it's ok because you can always go back to the SOR, albeit with a performance penalty.

But slow is better than broken.

🤔 Why?

Calls to the SOR are problematic for low-latency or random-access workloads.

When 9 out of 10 requests all want the same data, you’ll have infrequent cache misses. But when 9 out of 10 requests all require different data, you’ll have more cache misses, which reduces the effectiveness of caching.

Pre-populating caches is a way to avoid those cache misses by trading off latency for complexity.

⚙️ How?

Caveat: I use pre-population purely as a bolt-on optimization, not a core dependency.

Typically, I keep the cache-aside path as the primary mechanism. If anything goes wrong (and it will), there is always the option to go to the SOR for data (slow > broken).

A key decision is whether to pull the data or listen for it.

I prefer the SOR publishes updates as they occur, but platform constraints or circumstances may require you to pull the data.

Pub/sub works great when the SOR publishes, but other options exist as well (webhooks, files) with their own trade-offs.

Use whatever makes sense for your environment.

⚠️ Why Not?

Implementing pre-populating a cache can be easier said than done, as a lot can go wrong.

What happens if you lose a message or two?

What happens when you’re rebuilding the cache (errors or new instances)? How do you repopulate?

The cache-aside will cover any dropped messages, but implementing republish mechanisms is complicated.

You can’t rely solely on deltas; at some point, you'll need to republish the entire dataset.

Building all of these systems is complicated; there's more to monitor, patch, and manage.

If the latency hit and traffic volume to the SOR are not a concern, then that complexity is not worth it.

🧠 Final Thoughts

Pre-populating caches can be a significant performance win, but it can also be an operational overhead.

If your data is primarily static (changing infrequently), the overhead can be worthwhile.

If your data changes frequently, stick with cache-aside (and aggressive TTLs), or no cache at all.

When managing infrastructure or building and testing software, you'll inevitably find repetitive tasks that feel like a tool should exist to make them easier or faster.

Maybe one exists, perhaps it doesn't. Either way, your team doesn’t know.

So inevitably, the tasks stay manual, inefficient, and frustrating.

Everyone wishes there was a tool—so why not build one?

😱 Why People are Afraid to Build Tools:

1️⃣ “What if something already exists?”

You build something, then someone says, “Why didn't you just use X?”

That moment sucks, especially if you didn't know X even existed.

Don't defend it, own it, and say: “I didn't know X existed, let me take a look at it.”

No one expects you to know every tool out there.

Building something and later discovering a better tool is not a failure.

It's a learning experience; what you learn while building the tool is priceless.

2️⃣ “I don't want to maintain another system.”

This is a genuine concern.

But maintaining a small set of tools that save hours of repetitive work or enable something that couldn't be done otherwise is often worth it.

The key is to reduce overhead early on:

• Prefer CLI tools vs services

• Minimize dependencies

The less complexity, the easier it is to manage.

3️⃣ “We are too busy building features.”

This one is hard, not because it's a technical challenge. But because it's a mindset change.

Teams get so focused on what they’re building that they forget to look at how they are building it.

If a process is slow, manual, or error-prone, that's a signal that it could be improved.

The secret is taking a step back and re-evaluating how.

👨‍🔬 How I Approach Building Tools:

1️⃣ Look for existing tools

I prefer open source, so that's where I start. There is a lot out there, and you have to search for it.

2️⃣ Look for “close enough” projects

You might not find a project that does exactly what you want. But maybe it's 80-90% of the way there.

If it's close, extend it, contribute upstream. Or fork it (license permitting).

3️⃣ If nothing fits, build new

If nothing meets your needs, then build it.

Start small, use it yourself, share it with your team and solicit feedback.

😍 Don't Become Too Attached:

When someone says, “Why not use X?” Evaluate X objectively.

Is it the best tool now? If so, use it.

Using a well-known, widely adopted tool is often more efficient.

🧠 Final Thoughts:

Custom tools can be lifesavers.

They only become a problem when someone gets too attached and refuses to replace them with more standard solutions.

Build tools when needed, replace them when something better appears.

A Kenjutsu Instructor (Japanese Swordsmanship) I knew would always warn us. He’d say Americans always want to treat the sword like it’s a precious gem. It's a tool; treat it like you would a hammer.

I think that applies here as well.

When you treat engineering as a craft, it’s easy to get passionate about solutions. Strong opinions are a good thing — many great engineers have them.

But you also need to know when to challenge a decision and when to accept it.

🎯 The Inflection Point

Every architecture review eventually narrows down to a few viable options. Maybe it’s captured in an ADR, maybe through discussion, maybe through a decision-maker.

If your preferred option isn’t chosen, you have two paths:

1. Keep challenging the decision
2. Accept it and support it fully

Knowing which path to take is a critical engineering skill.

🔥 When to Keep Challenging

My rule: Will this decision cause me to lose sleep — figuratively or literally?

If the decision risks:

• Breaking production
• Waking you up at 2 a.m.
• Introducing significant operational or security risks

It’s worth continuing the conversation.

And the best way to challenge is respectfully — usually in a 1:1 with the decision-maker(s). This gives space for deeper context, trade-offs, and clearer alignment.

🤝 When to Support a Decision You Disagree With

If the decision isn’t dangerous — just not your preferred option— it’s time to commit. Many architectural choices have multiple valid options; one may be your preference.

In these cases, being a good engineer means supporting the direction chosen.

You can still improve the solution by suggesting micro-adjustments that reduce risk or enhance reliability without reopening the whole debate.

Sometimes, you will find that the chosen path was actually right. Don’t worry, no one cares if you were right or wrong in the debate if you supported the implementation.

🧠 Final Thoughts

Sometimes decisions are mistakes. That’s normal.

What matters is catching them early and being willing to revisit them once real-world data reveals new information. Implementation often teaches us things the whiteboard never could.

Just be careful not to over-index on finding every minor issue as a fundamental flaw in the solution.

Good architecture isn’t about being right all the time. It’s about making informed decisions, supporting the team, and knowing when to push and when to commit.

Let’s break it down.

🎞️ A Quick Recap

Both Blue/Green and Canary start the same way:

Take two instances (or clusters) of a service & deploy the new code version to the idle one.

Where they differ is how traffic shifts.

🐤 Canary

Canary deployments gradually shift traffic from old to new.

Both versions serve live traffic during the transition.

🔵|🟢 Blue/Green

A Blue/Green traffic shift is an all-or-nothing shift.

Only one instance is serving traffic; there is no gradual ramp-up.

⚙️ Why Canary Is More Complex

Running two versions at the same time (with both taking traffic) introduces challenges:

• Backward compatibility
• Shared (or replicated) databases
• Sticky sessions
• Context-aware routing
• Event ordering across versions
• Consistency of state

Blue/Green avoids most of this. You still need a rollback plan, but you don’t have to worry about parallel operations.

So if Canary is so complicated… why use it?

🏅 Why Canary Is Worth It (Sometimes)

Canary shines when:

• The system is highly critical
• It must run 24/7 with no interruption
• You cannot accept even a brief outage
• You want to reduce the blast radius of regressions
• You release often and need tight control/quick fallback

Canary lets you validate a new version with a small percentage of traffic before gradually increasing it further. If something breaks, roll traffic back instantly.

More importantly, when it breaks, only a portion of traffic is impacted.

For high-risk and mission-critical systems, the complexity is worth it.

🧠 Final Thoughts

Blue/Green is a great default deployment strategy, and in many cases, the optimal one.

A perfect example is file-based batch workloads. Batch systems usually have flexibility in timing. You can:

• Pause traffic
• Cut over to the new version
• Resume processing
• And if it fails… reprocess the files

Yes, easier said than done, but still far simpler than Canary.

Both approaches have their place. The key is matching the deployment strategy to the system’s criticality and level of acceptable risk.

Ever been in a technical debate where the other side seems way too attached to their solution?

Ever notice others feel the same way about you?

Sometimes your solution is the right solution. But sometimes… It’s just bias.

🧠 Understanding Bias

Bias gets a bad rep. Bias doesn’t always come from a negative space.

In the context of technical solutions, bias usually forms from experience.

Throughout our careers, we see countless architectures, patterns, outages, and wins.

We remember what worked.

We remember what didn’t.

Over time, we build a gut sense of solutions that are safe based on our experiences.

That gut sense is bias, and it’s often well intentioned.

💪 Working Through Bias (Without Ignoring It)

Accept that everyone has bias — including you.

This is the hardest part.

You need to assume that other people's biases stem from good intentions and real experience, just like yours do.

With this assumption, you can have more objective conversations and begin hearing other perspectives.

Ask yourself: Is this solution based on reality or comfort?

Why do you have a preference for your solution?

Are you pushing a strategy?

Are you avoiding something unfamiliar?

Are you sticking to what has worked in the past?

Understanding why you hold bias is key to making the case for your solution.

Use data to guide the decision, but make sure it’s objective.

Data makes decisions easier, but be careful. Bias can influence what data you choose to look at.

Sometimes we subconsciously cherry-pick data that supports our views.

It’s essential to take an objective look at the data, even if it challenges your case.

Bring in a trusted third party — but present data carefully.

An impartial opinion can help, but only if you give the whole picture.

When bringing in a third party, it’s crucial to present solutions and data objectively; that way, you get their honest opinion, not your opinion echoed back to you.

🧩 Final Thoughts

The most important part of technical decision-making is accepting the possibility that you might be wrong.

On many occasions, I've had to step back, evaluate my own bias, review the data objectively, and listen to opposing views.

Bias isn’t something you can eliminate; it’s something you recognize and manage.

🙋 What is an ADR?

An Architecture Decision Record (ADR) is a lightweight document that captures architectural decisions.

A good ADR typically consists of:

• The context behind the problem
• The options considered
• The decision made, including the why

Different companies/teams will add their own spin, but these are the core elements.

🤔 Why ADRs Matter

The ADR itself is helpful; it gives product, architecture, and engineering teams a shared reference point. Clear documentation reduces ambiguity, enabling teams to align and build effectively.

But the real value is the process.

Writing an ADR forces you to explore alternatives, consider trade-offs, and debate options objectively. If done well, ADRs capture everyone’s input and clearly document why a path was chosen.

This keeps architectural decisions grounded in logic rather than bias or preferences.

🧠 Final Thoughts

The documentation and process are valuable, but they only work with the right culture.

Teams need a culture where:

• Everyone is free to contribute to architectural decisions
• Diverse options are encouraged
• Decisions are made objectively
• ADRs are accessible and visible to everyone

Without the culture, the process becomes a formality and a burden of red tape.

With the right culture, ADRs become a powerful tool for making well-balanced & transparent decisions.

Of course, that culture and the process need to be embraced by all levels of the team. ADRs are only as useful as the effort you put into them.

A simple answer to your problem might be to add a bit of jitter (random timing delay).

When you schedule recurring tasks or loops, adding a bit of jitter can significantly improve how your application behaves.

📖 What is Jitter?

In simple terms, jitter is a small bit of randomness added to the time between two events.

While there are many types of jitter in computing, for this post, we will keep the scope to randomness, adding delay between events.

⚙️ Why it Matters:

When tasks don’t implement jitter, they can accidentally synchronize, running at the same time.

Which leads to:

• CPU and Memory spikes
• Thread contention
• Request storms to downstream systems

🧩 A Simple Example

Imagine an API Gateway that caches responses. You decide to invalidate responses every 30 minutes with a scheduled thread.

No problem for a handful of APIs, but scale this to 1,000 APIs. Suddenly, every 30 minutes, 1,000 threads fire up at once.

The periodic spikes could cause performance issues or even crash the gateway.

Now add random Jitter: instead of running every 30 minutes, add or subtract a few random seconds for each task.

You’ve just spread out the load, making utilization smoother and more predictable.

⚠️ Caveats

Jitter isn’t perfect; this approach spreads out the load, but with random jitter, small spikes could still occur.

Still, for many scenarios, it’s a simple approach.

🧠 Final Thoughts

If you find your application slowing down periodically with spikes in resource utilization. You might be dealing with synchronized tasks, and adding random jitter might be a good solution.

It’s simple, it’s easy, and it usually works well.

🛑 The Common Mistake:

I've seen many shutdown implementations that stop the listener as soon as the shutdown signal is received.

The assumption is usually:

“Stopping the listener will fail readiness probes, and traffic will be redirected.”

That's half right…

It will trigger traffic redirection, but not immediately.

⏱️ Probe Intervals Matter:

Readiness probes (Kubernetes), Load Balancer health checks, & service mesh probes all run at fixed intervals.

In Kubernetes, the default is 10 seconds.

That means it can take up to 10 seconds for the platform to detect an unhealthy status and adjust traffic.

Longer if the failure threshold is greater than 1.

💥 What Happens During Those 10 Seconds?

New traffic still goes to the unhealthy instance.

And because you stopped the listener, every request to that instance fails for 10 seconds.

Some clients retry and land on another instance.

Some will not.

Either way, every rollout will result in failed requests that could have been avoided.

✅ What You Should Do Instead

When shutting down an instance:

1️⃣ Keep the listener running; Don’t slam the door shut.

2️⃣ Fail readiness probes; Report failures from the readiness endpoint, but allow new requests to other endpoints.

3️⃣ Wait for traffic to drain; Let in-flight requests finish, and let the platform stop routing new requests.

4️⃣ Then stop the listener; Only when it's safe.

This is a graceful shutdown.

🧠 Final Thoughts

Resiliency isn't only about surviving failures, it's also about preventing them.

Handle shutdown properly, and you can roll out new code without ever failing a request.

🤔 The Problem:

gRPC is fast thanks to protobuf and how it handles connections, mainly:

• Persistent connections that avoid repeated TCP handshakes
• Sending multiple requests over a single HTTP/2 connection.

However, these performance optimizations are also a source of failover challenges.

😫 Challenges with Failover:

Let’s say you’ve just implemented gRPC and want to trigger a manual failover for your service.

For many, failover typically happens at the load-balancer level, which works fine for HTTP/1.

When you take an instance down, new requests go to another instance.

However, with gRPC over HTTP/2, connections stay open and are reused, which means existing connections continue to send requests to the old instances even during failover.

Unless your load-balancer understands HTTP/2 and gRPC, failover will not work as it used to.

🛠️ Failover with gRPC

For proper failover, you’ve got two main options:

1. Use a load balancer that understands HTTP/2 and gRPC, such as an AWS Application Load Balancer vs. a Network Load Balancer, Envoy vs. HAProxy.
2. Cycle connections periodically—force clients to reconnect and redistribute the load.

Both options get the job done, but the first is overall cleaner.

💡 Final Thoughts:

There is a lot to love about gRPC: strong contracts, outstanding performance, and client-server simplicity.

But it takes work to operationalize it. Nobody tells you that upfront, though.

Or when it changes, the next person will understand the original context.

Reality check, they wont.

The next person (future you) has no idea what you were thinking.

🔎 A Simple Example:

You are building a service that receives JSON requests.

You write a method that takes in an array from the request and accesses index 2.

The request handler has already validated the array length and content.

So you don't need to recheck it before accessing index two, right?

It's just less efficient to check it twice, right?

Wrong.

Your original implementation might work fine, but fast forward to years later, when someone else (perhaps yourself) uses that method.

Will they always ensure the array has the right length? 🤷‍♂️

If they don't, your method is a ticking time bomb.

🧠 Fix the Mindset:

Embrace defensive programming, where you expect that your methods will be misused.

Recheck the array's length before you use it, even if something else has previously checked.

Expect bad inputs, expect errors to occur, and have a path to do something about it.

💡 Expect the Unexpected

If you assume the following person:

• Won't read your docs or code comments
• Will reuse your code in a different context
• Will misuse your code for things you've never designed it for

You will write safer, more resilient code.

🧩 Context:

I recently shared performance tuning results where we reduced Microservice-to-Microservice latency from 1.3 ms to 0.3 ms in a new platform.

That’s a huge performance win, but it doesn’t sound like much.

In card payments, where every millisecond counts, it’s easy to see the value. But for an average backend system, does 1 ms matter?

A honeybee can flap its wings in 5 ms, so who is going to notice 1 ms?

🧘‍♂️ Perspective:

It’s not just 1 ms.

Modern distributed systems are built from many microservices and layers. A single customer journey typically touches dozens of components.

If you shave off 1 ms from every call, the gains compound quickly.

End-to-end, that can add up to tens or even hundreds of milliseconds for every incoming request.

💡Final Thoughts

Does saving 100 ms even matter?

Kind of.

Even if your platform isn’t latency-sensitive, throughput and latency are closely related.

Faster requests mean more available capacity.

That 100 ms may allow you to scale better or reduce infrastructure costs.

A 1 ms improvement doesn’t sound like much on the surface, but the compounding effect is massive, even for systems that “don’t care” about latency.

The concept is that rather than failing a request when a dependency is down, store it, and send the request when it is back up.

🤔 How it works:

We have two services, Service A, which is highly dependent on Service B to process requests.

Traditionally, when Service B is down, Service A would have no choice but to reject requests with a failure.

With the Store and Forward design, when Service B is unavailable, Service A will reply to the request with a “degraded processing” (rejecting it, accepting it, or saying, “We’ll let you know later”).

But before replying to the request, it is “stored” somewhere that can be accessed quickly, such as a cache, a queue, a database, etc.

When Service B is back up, Service A will “forward” the stored requests to Service B.

🥹 What I like about this design pattern:

It accepts that failures are going to occur because everything fails.

Rather than creating “retry storms,” it adds more intelligence to the process, only sending requests to Service B when it’s back online.

It ensures that no request is lost, even in significant outages.

But this design pattern isn’t without complexity.

Blind retries are easy; you keep retrying.

But with Store and Forward, you need to:

🛑 Know when Service B is unavailable

🧠 Add logic around degraded processing

✅ Detect when Service B has recovered

🤹 Figure out the best way to dequeue the stored requests

While more complex than blind retries, store and forward is a great resiliency design for when every request matters.

In payments, where reliability and fast responses are critical, as is accuracy, the complexity of store-and-forward designs is a worthwhile trade-off.

“Push as much processing as possible to later.”

Why It Matters? 🤔

In many systems—checkout, login, trade execution—latency matters because someone (or something) is waiting:

• A customer at a point of sale

• A user at a login screen

• A system waiting on a transaction confirmation

Platforms that support these scenarios must respond in milliseconds. If not, requests will fail, and user experiences will suffer.

My Approach 🧠

I typically divide these platforms into two sub-platforms to optimize for speed and scale.

🏎️ Real-Time Platform: Optimized for scale and speed, only performing what is essential before responding to the request.

📥 Event-Driven Platform (sometimes Batch): Handles processing deferred from the real-time platform. It is still built for scale, but seconds, not milliseconds.

Deciding What Belongs Where 🗃

I try to break down processing into steps, and for each step I ask:

“Does this step need to happen before we respond or after?”

✅ If it MUST be performed before the response, use a real-time path.

⏭ If it can wait until after, event-driven path.

Things that tend to follow the event-driven path are:

• Audit logging

• Downstream asynchronous notifications

• Enrichment and Transformations

• Checks that trigger out-of-band tasks

These are not slow but don’t need to be “blocking.”

Final Thoughts ✍️

The key message is that the more you do on the real-time path, the slower it is.

This pattern is a good way to reduce the real-time workload.

But the trick is to find a reliable and fast way to move work from a real-time to an event-driven system.

Pub/Sub and gRPC streams are two of my go-to options.

With AI Coding Assistants and Autonomous Agents being all the rage lately, I feel like this is something folks—especially those early in their careers—need to hear.

Coding is essential, but only a portion of what is required to build production systems.

Writing software can take a lot of effort, but just as much is spent before anyone starts to code.

Let's think about some of those tasks:

• Defining API contracts
• Designing a database schema
• Choosing the right database
• Configuring build & deployment pipelines
• Selecting a runtime environment
• Packaging software (Dockerfiles?)
• Integrating observability tools
• Writing runbooks and service manuals

And that's all before you even consider architectural trade-offs, system design, or cross-team alignment.

A lot of software engineering involves understanding what needs to be done and why—and then figuring out how to do it well.

Even with how fast AI is evolving, we are still a long way from AI completely taking over software engineering, but it's revolutionizing how we attack the process.

Are you embracing our robot overlords or resisting change? 🤖

My honest answer:

> Which motivates you more?

It sounds simple, but it's an essential question for many.

🧰 If you enjoy building systems more than building people, then the IC track is probably right for you.

👔 If you enjoy growing and leading others, then people leadership might be the right path for you.

The good news is that you can switch later.

A lot of people change between IC and people leadership roles. Some love both equally and switch multiple times throughout their career.

Just keep those leadership and technical skills sharp.

A Word of Warning ⚠️

The higher you go, the fewer IC roles you’ll find—especially at companies without a strong IC path.

Some companies top out early, and some treat the IC track as a first-class path.

So, choosing the IC track might limit your options, but it's worth it if that's what you genuinely want to do.

Have you chosen your path?

For me, the IC path was the clear winner, but there were several times I considered people leadership opportunities.

Before I became a Staff Engineer, I applied for a Director of Engineering role. I didn't get it, but the process helped me figure out what I wanted.

While the idea is simple, designing a system around this concept is anything but.

🤔 Why it’s Effective:

The core idea is straightforward:

Keeping traffic local is better for performance and resilience.

🚄 Performance:

Performance is easy to understand:

• In-region traffic (including cross availability zone) usually sees single-digit millisecond latency (or less)

• Cross-region traffic introduces latency, double-digit milliseconds or more, depending on the region

Latency adds up fast when you cross-regions multiple times in a microservices architecture.

🚀 Resilience:

Resilience is a bit more nuanced.

Every cross-region call passes through more network hops, such as firewalls, routers, switches, load balancers, etc.

More hops == More failure points

Keeping traffic local means fewer packet loss chances and less impact when things break.

🧙‍♂️ Complexities:

Designing for regional isolation (core concept of cell-based architecture) means:

1️⃣ Having active instances of critical services in each region (active-passive doesn't work with this approach)

2️⃣ Figuring out data replication and consistency across regions

3️⃣ Building robust routing and failover capabilities

4️⃣ Establishing management processes and capabilities that let you manage each region independently

Yes, the design is much more complex, and the operational overhead is much higher, but the blast radius of failure is smaller.

A failure with a critical service in one region only impacts that region.

🧠 Final Thoughts:

Perfect isolation isn't always possible; you might need to cross-region for data consistency or as a fallback.

When you need to cross-region:

✅ Reduce the number of cross-region hops as much as possible

✅ Do it up front, ideally before the request lands in your system.

The more cross-region routing you perform at the edge, the more you can avoid regional isolation complexities in the underlying systems.

If you’ve run applications in Kubernetes, you’ve probably heard of Kube-proxy, the service responsible for routing traffic to Services.

But the interesting twist is that Kube-proxy doesn’t perform the routing, and iptables does (or IPVS, or nftables).

⚙️ How it works:

When you define a Service, Kubernetes will assign it an IP address.

Kube-proxy watches for these events and creates iptables rules that handle routing.

The iptables rules will:

• Forward new connections with a destination of the Service IP to a Pod IP
• Use the statistics module to select which Pod IP to forward the connection to

I like to think of it as follows: Kube-proxy identifies the need for routing, and iptables does the work.

🤔 Why it’s important:

If you plan to use gRPC, this is critical to understand.

gRPC uses HTTP/2 as its underlying protocol, which sends multiple requests down a single connection.

Since iptables forwards traffic at a connection level (layer 4), multiple requests down a single connection will all land on the same pod, even if more are available.

You might assume traffic will be balanced across pods, and be surprised to find it is not.

You're fine if you use ``HTTP/1.1 (without connection reuse). But anything that keeps long-lived connections open or sends multiple requests down a single connection, Kube-proxy won’t cut it.

🔭 What’s Next:

Scaling has been a challenge for iptables, as having lots of rules and connection tracking are known bottlenecks.

IPVS and nftables (iptables successor) have been introduced as new options for routing and load-balancing.

Both are still layer 4.

If you need layer 7 (request level routing), that’s where Istio comes in.

🧠 Final Thoughts:

Understanding Kube-proxy, iptables, and gRPC & HTTP/2 work is essential for anyone building fast, scalable backend systems on Kubernetes.

You can’t optimize what you don’t understand.

🔗 References:

Here are some reference links for those looking for a deeper dive.

• https://kubernetes.io/docs/reference/networking/virtual-ips/

• https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/

• https://en.m.wikipedia.org/wiki/IP_Virtual_Server