Case study: the login emails stopped and nobody was told
Login emails were sent by a background worker. The worker failed and logged every failure. The application kept answering 200, and the business found out a week later from a customer. What the audit after it looked for, and what else it found.
Situation. Customers signed in with a code sent by email. Sending was handed to a background worker, so the request that triggered it answered 200 as soon as the job was queued. That response was accurate about queuing and said nothing about delivery, which is the normal trade of doing work asynchronously.
Then the worker stopped being able to send. It recorded every failure it hit. The application went on answering 200, the frontend went on showing the same confirmation screen, and every person trying to sign in waited for a code that was never going to arrive.
Risk. Nothing looked broken from any angle a person was actually watching. No error page, no failed request, no alarm. The failures existed as log lines, and nobody had a reason to open the logs, because nothing was pointing at them.
It ran for a week. It surfaced because one customer complained. Reading the logs after that showed the size of it: a week of people who had tried to sign in, could not, and left. Those signups were not recoverable.
Decision. The failure was already detected and already recorded. What was missing was any path from a recorded failure to a person, and there was a structural reason for that: an asynchronous send has no request left to fail. By the time the worker gives up, the HTTP call it belongs to is long finished and the user is gone. Nothing in the request path can report this, so the signal has to leave the system by another route.
So the failure was pushed to a channel the business owner already had open, as a message saying login emails have stopped going out. An alert delivered to a place nobody reads is the same outage with extra steps.
Audit. Fixing the sender would have closed this one case and left the shape of it everywhere else, so the rest of the system was read with one question in mind: where can something fail with nobody left to hear it. Three answers came out of that pass, and closing them mattered more than the original fix.
Payment creation at the provider. A payment that is never created looks, from inside the application, indistinguishable from a customer who changed their mind. Nothing errors, and the only party who knows something went wrong is the person who wanted to pay and gave up. A failure to create a payment now reports itself when it happens.
OAuth2 authorization. When it stops going through, the people affected cannot get in and the application keeps running normally. It is the email failure again, in a different subsystem.
Every internal worker. Scheduled and queued jobs all carry the property that made this case expensive. There is no request, no user watching the result, and a log line is the entire record of the failure. They were put under one rule instead of being fixed one at a time, each after its own incident: a job nobody is waiting on has to report its own failure to a person.
Result. The distance between the system knowing and a person knowing went from a week to the moment it happens. The owner is told while the failure is running, in the place they already look, instead of learning it from a customer and confirming it in a log file afterwards.
What this case is about. The bug was ordinary and the logging was working. The outage was expensive because recording a failure and telling someone about it are two different jobs, and only one of them had been built. That gap is the problem I later built AlertLoop around.