Sometimes when I tell people that logging can impact a microservices response time, I get strange looks. 🤨
But I get it. Logging doesn’t feel significant enough to create issues, yet I’ve seen it repeatedly make a considerable difference.
Recently, the team was benchmarking a new microservice that performed well until the request volume passed a few thousand transactions per second.
Once the request volume increased, the p99 latency skyrocketed.
The fix? Two major actions.
1️⃣ Changed a log message in the hot path of every request from Info (logged always) to Debug (only logged when enabled).
2️⃣ Enabling Asynchronous logging puts log messages into a buffer where they can be written to stdout in a different goroutine instead of blocking requests.
These two changes improved the p99 latency by nearly 200%.
This post isn't about not logging.
It's about being intentional:
-
Are you logging in the hot path for every request?
-
Is the log level correct?
-
Do you need every request logged at this point in the process?
Because logging does matter.
