Software bugs happen; how you handle them is what matters.
You’ve probably heard the saying, “If you build it, you should be willing to support it.” But I think real engineering ownership goes beyond supporting what you’ve built. It’s about taking responsibility when things go wrong. I don’t mean taking blame; I mean taking action.
It doesn’t matter how experienced you are; we all make mistakes. More experience doesn’t necessarily mean fewer mistakes; with more experience, you learn to build the right guardrails to catch them before they reach production.
But sometimes bugs slip through. What matters is how you respond when they do.
If You Build It, Be Willing to Support It at 3 a.m.
When something breaks in production, sometimes the engineer who built it is the best person to help resolve the issue.
If you’ve built something business- or mission-critical, I think you should be willing to jump in and help when things go wrong.
Of course, work-life balance matters.
That’s why on-call rotations, team ownership, and avoiding single-person dependencies are important.
Being involved in production support creates an incredibly valuable feedback loop. You quickly discover things about your system that aren’t always obvious during development.
- Maybe troubleshooting an issue takes longer than it should because you’re missing observability.
- Maybe recovering from a failure requires restarting an entire service because you don’t have the right operational controls.
- Maybe a minor dependency failure becomes a major incident because the system wasn’t designed to handle it gracefully.
You learn these things by operating software, not just building it.
And when you’re the one responding to those late-night incidents, you have a pretty strong incentive to make the system easier to operate and more resilient.
But Fixing the Issue Isn’t Enough
Showing up to resolve production incidents matters. But I think real ownership goes further.
Personally, when I discover that a bug I introduced caused a production issue, I don’t focus only on fixing it.
I want to understand why we didn’t catch it before it reached production.
- What testing gaps allowed this bug to slip through?
- Did we miss a scenario?
- Did the system design make this failure more likely or more impactful?
- Could we have detected the problem earlier?
And most importantly, how do we prevent it from happening again?
I don’t expect myself to write perfect code. But I do hold myself accountable for learning from my mistakes and preventing them from happening again.
Don’t Just Fix the Bug. Fix the Gap.
When a production issue occurs, the immediate priority is restoring service. But once the incident is resolved, the work shouldn’t stop there.
Next, recreate that scenario in your tests to ensure they catch the bug if it comes back.
You may even need to reevaluate your system design to account for failure scenarios you’ve never considered.
And sometimes, incidents expose weaknesses that have nothing to do with the original bug. Maybe the incident revealed that troubleshooting was unnecessarily difficult, or that recovering the system required too much manual intervention.
Those problems are worth fixing too.
Your goal isn’t just to prevent the same bug from happening again; it’s to learn from the incident and improve the system as a whole.
Final Thoughts
Production issues happen.
No amount of experience, testing, or good system design will eliminate every possible failure. But experience should change how we respond to those failures. Taking responsibility doesn’t mean assigning blame when something goes wrong. It means learning from those failures and improving the system rather than simply restoring service and moving on.
If an issue happens once, it’s something to fix; if it keeps happening, it’s a system flaw.