Case study: two requests, one payment
A checkout that could not tell a customer's second attempt from a network retry, and a balance debit where the same ambiguity moved real money. What the fix actually was.
Situation. A checkout flow created the order and charged the customer in one operation. The same request could arrive twice: a customer pressing the button again on a slow connection, or the payment provider retrying a call it had already accepted. Nothing in the system could tell those two cases apart. Both looked like a second, valid request to pay.
The sharper version of the same defect sat one layer deeper, on debits from an internal balance. There the duplicate did not create a confusing extra row, it moved real money twice.
Risk. A customer could be charged for one purchase twice, and support had no way to distinguish a duplicate from a genuine repeat order: in the database the two are identical. On the balance side there is nothing left to interpret, because the money has already moved. Putting it back was a manual correction, made after the customer complained. The system held no opinion about whether the second debit was legitimate, so the only detector was the person who lost the money.
Decision. Order preparation and payment confirmation were split into two steps with an explicit state between them, so that "we are about to charge" and "we have charged" stopped being the same moment. Every write coming from the outside became idempotent, keyed on an identifier the caller supplies and the database enforces as unique, so a repeat of the same request updates the existing record instead of creating a second one. A retry is therefore not a judgement call about intent: it either matches an existing key or it does not. For balance debits the same key protects the money operation itself, and the balance is re-read at the moment of confirmation rather than trusted from the earlier step.
Result. A provider retry and a second click are both safe by construction rather than by luck. A duplicate now shows up as a repeated key on one record, which is something you can look at, instead of as a second charge that has to be discovered and reversed. Payment state is readable from the record itself, without reconstructing the sequence from logs.
What this case is about. The decision that mattered was noticing that the system had no way to represent the difference between "the customer wants to pay again" and "the network said this twice", and that no amount of careful code helps until that distinction exists in the data model.