Sometimes good engineering looks like over-engineering.

I was recently putting together a talk that walks through some architecture and engineering decisions I’ve made over the last eight-plus years when something stood out to me.

Without context, many of those decisions probably look like over-engineering.

🤔 An Example: Operational Toggles

In several mission-critical payment applications, we follow a simple philosophy:

If a dependency isn’t required to process customer traffic, we should be able to turn it off.

For example, metrics.

Metrics are important, but they aren’t more important than processing transactions. If something goes wrong with our metrics implementation, we want to be able to turn off metrics while continuing to process traffic.

The same applies to out-of-band management messages and data updates. We typically use message brokers for this, and if there is an issue with that integration or a bug in the message handlers, we want the ability to stop consuming those messages without taking down the application.

Why?

Because we want fine-grained operational control over our platform. A non-critical capability shouldn’t be able to take down the critical path.

🤯 Isn’t That Over-Engineering?

Maybe.

These operational toggles add complexity to the system: flags that need to be managed, tests that need to be written and executed, and operational procedures for failures that might never happen.

For many applications, it’s probably not worth it.

If a metrics library causes a memory leak in an application that does batch processing, restarting to release the memory is perfectly acceptable.

If a message handler crashes an application that is automatically restarted, it might be ok depending on the application. Not great, but if requests are retried and nothing truly fails, no harm, no foul.

Not every application needs the complexity of these operational toggles.

But when an application crash could cause impact, when the system is mission-critical, and every failure is potentially impactful, the equation changes.

🧠 Context Determines Everything

For the batch and non-critical application examples, these operational toggles could easily be considered over-engineering.

But for a mission-critical, always-on, large-scale platform, when a situation occurs and these operational toggles are used, they become incredibly valuable.

The above examples are not hypothetical.

A change in metrics causes a spike in memory usage. An unsafe type assertion in a data sync handler from a message broker causes an application to crash.

These are real scenarios, scenarios that I’ve seen in mission-critical systems. Because these operational toggles existed, these scenarios were not disastrous or even inconvenient.

These operational toggles let us turn off the cause of an issue. More importantly, they gave us time. Time to understand what happened, build and properly test a fix, and validate that the fix actually solved the root problem.

All without rushing a change to production while the platform was unstable.

🧐 Final Thoughts

If you look at the way we use operational toggles, it could easily feel like over-engineering. But building critical systems means planning for failure scenarios even if they are unlikely.

Sometimes that additional complexity might look unnecessary. But when something goes wrong, you’re thankful it’s there.