Case study: the login emails stopped and nobody was told
Login emails were sent by a background worker. The worker failed and logged every failure. The application kept answering 200, and the business found out a week later from a customer.
Situation. Customers signed in with a code sent by email. Sending was handed to a background worker, so the request that triggered it answered 200 as soon as the job was queued. That response was accurate about queuing and said nothing about delivery, which is the normal trade of doing work asynchronously.
Then the worker stopped being able to send. It recorded every failure it hit. The application went on answering 200, the frontend went on showing the same confirmation screen, and every person trying to sign in waited for a code that was never going to arrive.
Risk. Nothing looked broken from any angle a person was actually watching. No error page, no failed request, no alarm. The failures existed as log lines, and nobody had a reason to open the logs, because nothing was pointing at them.
It ran for a week. It surfaced because one customer complained. Reading the logs after that showed the size of it: a week of people who had tried to sign in, could not, and left. Those signups were not recoverable.
Decision. The failure was already detected and already recorded. What was missing was any path from a recorded failure to a person, and there was a structural reason for that: an asynchronous send has no request left to fail. By the time the worker gives up, the HTTP call it belongs to is long finished and the user is gone. Nothing in the request path can report this, so the signal has to leave the system by another route.
So the failure was pushed to a channel the business owner already had open, as a message saying login emails have stopped going out. Then the same treatment was applied to the other places where a silent failure would be equally expensive, rather than only to the one that had already cost a week of signups.
Result. The distance between the system knowing and a person knowing went from a week to the moment it happens. The owner is told while the failure is running, in the place they already look, instead of learning it from a customer and confirming it in a log file afterwards.
What this case is about. The bug was ordinary and the logging was working. The outage was expensive because recording a failure and telling someone about it are two different jobs, and only one of them had been built. That gap is the problem I later built AlertLoop around.