Idempotency keys for email: preventing duplicate sends

A customer clicks "Pay," your server calls the email API to send a receipt, and the connection times out. Did the send happen?

Your code cannot know, so it retries, and the customer gets two receipts. Idempotency keys exist to make that retry harmless.

The problem in one diagram#

client ──POST /emails──▶ API ── queues message ──▶ (sent)
client ◀──── timeout ────X   (response lost)
client ──POST /emails──▶ API ── queues message ──▶ (sent again)

The first request succeeded, but the response never arrived. From the client's side, a timeout looks exactly like a failure. Retrying is the right instinct; without idempotency, it is also how duplicates happen.

For most API calls, a duplicate is an annoyance. For email, it is visible to your customer, it erodes trust in billing and security mail, and at scale it inflates volume and complaints.

How idempotency keys work#

The client generates a unique value for each logical operation and sends it in a header:

POST /v1/emails
Idempotency-Key: receipt-order-48213

The server's contract is:

1. First request with a new key: perform the operation, store the response against the key, return it.
2. Repeat with the same key after completion: do not perform the operation again; return the stored response.
3. Repeat while the first is still running: do not start a second operation; tell the client to wait and retry.

The IETF HTTP API working group has a draft specification for the Idempotency-Key header field, and many payment and messaging APIs follow this pattern. As one example, Koltrix's send endpoint accepts an Idempotency-Key and keeps the stored response for 24 hours.

The race condition most implementations miss#

The naive implementation checks the key at the start and saves it at the end:

# BROKEN: check-then-act race
cached = store.get(key)
if cached:
    return cached
response = send_email(payload)      # takes time: DB writes, enqueue
store.set(key, response, ttl=86400)
return response

Between get and set, a second request with the same key, often a client retry fired because the first one was slow, sees nothing cached and sends again. This window is precisely where retries land, so the bug shows up in exactly the situation the feature was meant to handle.

The fix is to claim the key atomically before doing any work:

IN_PROGRESS = "__in_progress__"

def handle(key, payload):
    if store.set_if_absent(key, IN_PROGRESS, ttl=86400):   # atomic, e.g. Redis SET NX
        try:
            response = send_email(payload)
        except Exception:
            store.delete(key)            # nothing was sent; allow an immediate retry
            raise
        store.set(key, serialize(response), ttl=86400)
        return response, 202

    cached = store.get(key)
    if cached is None or cached == IN_PROGRESS:
        return {"error": "request with this key is still in progress"}, 409
    return deserialize(cached), 200

Three details in that sketch matter:

• The claim is atomic. Only one request can win set_if_absent.
• Failures before anything is queued release the key, so the client is not locked out for the whole retention window by an attempt that sent nothing. Be careful to release only when you are sure nothing was queued.
• If the store is unreachable, fail closed. Return a 503 and send nothing. Sending without the ability to deduplicate risks exactly the duplicate you are trying to prevent, and the client can safely retry with the same key once the store is back.