The most important skill for building reliable distributed systems.
Discover why at-least-once delivery requires idempotency to prevent duplicate charges, double emails, and silent data corruption.
Why "Idempotency" is the most important—and ignored—skill for new software engineers.
In the past, code ran on one single computer. If something went wrong, the whole program just crashed. It was simple. You either succeeded, or you failed completely.
Schools still teach this way.
Today, apps use hundreds of computers connected by the internet. The internet breaks constantly. Messages get lost, servers crash, and wires get cut. Failure is normal.
Welcome to the "Unknown State."
Because schools only teach the "Happy Path" (where everything works perfectly), new coders assume the internet is reliable. They build systems that break down the moment a network fails.
If a user pays for a product and the internet drops for one second, the system has to guess: do we cancel the order, or do we try again? If we try again blindly, we might charge the user's credit card twice.
In simple English: It is absolutely safe to retry.
If you push a button once, it does the work. If you mash that same button 100 times, the final result stays exactly the same as if you only pushed it once. No duplicate emails. No double charges.
When two computers talk, engineers have to choose a rule for how messages are delivered. You can't cheat math. There are three main rules, and one of them is a lie.
The system sends the message once and never tries again. If the internet breaks, the data is lost forever. Good for: Simple logs or weather sensors where missing one second of data doesn't matter.
The system will keep sending the message until it gets a confirmed reply. Data is never lost, but it might arrive twice. This is the safest real-world rule, but your code must handle the duplicates.
New coders think computers can guarantee a message arrives perfectly once. This is mostly a fairy tale. True exactly-once is incredibly slow and hard to build across different systems.
When websites ask a server to do something (using HTTP), some requests are naturally safe to repeat, and some will cause a disaster if repeated.
To stop POST requests from making duplicates, smart systems use a "ticket number" called an Idempotency Key. Here is exactly how it works:
CLIENT MAKES A TICKET
The user's phone generates a random ID for the checkout click.
Key: X-Order-ID:
abc-123
SERVER CHECKS THE MEMORY
The server receives the request and checks its database.
"Have I seen ticket abc-123
before?"
DECISION TIME
IF SEEN BEFORE: Stop. Do no work. Return the saved receipt.
IF NEW: Process the payment, save the receipt, and save the ticket number.
In data engineering, a job might copy millions of rows of data every night. If the job crashes halfway through, an automated system will just restart it.
If the code is not idempotent, the restart will copy the exact same rows again. The system won't alert you. It will just report a "Success". Suddenly, your company's reports show you made double the money you actually did. This is called silent data corruption.
Using random IDs for every row, and just adding them to the bottom of the list (INSERT). Retries equal duplicates.
Use fixed IDs based on the data (like joining the user's name + date). Or, completely delete yesterday's data before you write yesterday's data again.
A junior coder writes code and hopes the internet works perfectly. A senior coder writes code assuming everything will break. Here is the checklist senior coders use to review work:
| Check | Why it matters |
|---|---|
| Are there Idempotency Keys? | If we are making or changing data (POST/PATCH), we must block duplicate tickets from entering. |
| Do the keys expire? | Keys can't be kept in memory forever. They need an expiration time (like 24 hours) to save server space. |
| Are external actions safe? | If the code sends a "Welcome" email, and the database crashes, will a retry send a second email? It shouldn't. |
| Can this run twice at the same time? | If a user double-clicks fast, two requests might hit two different servers at the exact same millisecond. The database needs a lock. |
Continue your research with other reports from the Engineering series.