Message queues are great for handing off work, but how do you know the work actually happens?
Message queues and brokers are great solutions for many use cases. They can provide excellent resiliency and strong guarantees that a message gets where it needs to go.
But I find those delivery guarantees can sometimes create a false sense of resiliency. Because delivering a message and completing the work that message represents are two different things.
Peeling Back the Everything Breaks Onion
A message broker might reliably deliver a message to a consumer. Great. But what if that consumer receives the message and fails while processing it?
What if it processes the message but fails while calling a downstream service? What if the consumer succeeds, but the overall business process doesn’t?
Message delivery isn’t the same thing as business-process completion. For best-effort messages, maybe that’s fine. For mission-critical work, I generally want some mechanism to know whether that work actually completed.
Tracking Completion
One way is to track the operation and expect an application-level acknowledgment.
Let’s explore a simplified approach:
A producer publishes:
ProcessPayment:123456789
At the same time, it records in a local DB:
123456789 -> PENDING
The consumer processes the message and, once the work completes, publishes:
PaymentProcessed:123456789
When the producer receives that message, it updates the local DB:
123456789 -> COMPLETE
Now instead of simply knowing the original message was delivered, we have a record of the work actually completing.
What If the Completion Never Arrives?
If operation 123456789 stays PENDING, it’s tempting to assume the work failed.
But that’s not necessarily true.
Maybe the consumer failed. Maybe a downstream dependency failed. Or maybe the producer failed to process the acknowledgment. Either way, we haven’t confirmed operation completion, and what happens next depends on the use case.
You may be able to retry, or you may need to query the downstream system and validate the outcome. You might need to wake someone up, or you do nothing because the message was best-effort anyway.
The important part is that the uncertainty is detectable and visible, which means you can decide what to do about it.
Final Thoughts
Of course, many message brokers already have acknowledgment mechanisms. Those are useful, and depending on how they are used, might provide similar assurances.
But many implementations only tell you a message has been delivered or consumed. If the consumer hands the work off to another system, or starts a larger business process, the broker doesn’t understand whether the work ultimately completed.
That uncertainty can be problematic for some use cases.
Of course, adding this additional layer of certainty comes with its own complexities. It adds state management, more components in the flow, and more room for failure.
But for mission-critical work, guarantees are essential. A broker can provide strong guarantees around message delivery, but your application is ultimately responsible for determining and exposing whether the work actually happened.