Finding a $90,000 Remote Code Execution Vulnerability in Google
Concept of the day · distributed systems
Idempotency Keys
The client picks a unique key so a retried request runs at most once.
The mechanism. The caller generates a unique key per logical operation and sends it with the request. The server stores the key alongside the result of the first execution. A second request carrying the same key does not re-execute: it returns the stored response. The key must be recorded in the same transaction as the side effect, or a crash between the two leaves you with a charge and no key.
Where it bites. Networks fail in the one place that matters: after the server committed but before the client saw the response. The client cannot distinguish "never arrived" from "arrived and succeeded", so it retries, and the user is charged twice. Idempotency moves the decision to the server, which is the only party that knows.
The hard parts. Key scope (per user? per endpoint?), retention (Stripe keeps keys 24 hours), and what to do when the same key arrives with a different body — that is a client bug, and returning the original result silently hides it. Return a 422 instead.
Takeaway. Any endpoint that moves money, sends a message, or creates a resource needs this. "The client will only call it once" is not a design.