Idempotency: Safe Retry Operations
An operation is idempotent if executing it multiple times produces the same result as executing it once. In mathematics, `abs(abs(x)) = abs(x)`. In distributed systems, `createPayment(id=123)` called...
Key Insights
- Idempotency transforms unreliable networks into reliable systems by ensuring that duplicate requests produce the same result as a single request, protecting data integrity during retries.
- The idempotency key pattern—client-generated unique identifiers stored server-side—is the most flexible approach for complex operations like payments, but simpler patterns like PUT semantics and database upserts often suffice.
- Race conditions between concurrent duplicate requests are the hardest edge case; solve them with distributed locks or database-level constraints, not application logic alone.
What Is Idempotency and Why It Matters
An operation is idempotent if executing it multiple times produces the same result as executing it once. In mathematics, abs(abs(x)) = abs(x). In distributed systems, createPayment(id=123) called five times should create exactly one payment.
This matters because networks lie. A client sends a request, the server processes it, but the response never arrives. Did the operation succeed? The client doesn’t know. The safe choice is to retry—but without idempotency, that retry might duplicate the operation.
Consider a payment system. A customer clicks “Pay” and the request times out. They click again. Without idempotency, you’ve charged them twice. With idempotency, the second request recognizes the duplicate and returns the original result.
The problem compounds in distributed architectures. Message queues deliver at-least-once. Service meshes retry failed requests automatically. Kubernetes restarts crashed pods mid-operation. Every layer adds retry potential. Your operations must handle this reality.
Common Idempotency Patterns
Three patterns cover most idempotency needs, each with different tradeoffs.
Natural Idempotency leverages HTTP semantics. PUT replaces a resource entirely—calling it twice with the same payload yields the same state. DELETE removes a resource—deleting an already-deleted resource is a no-op. POST creates new resources and is inherently non-idempotent.
from flask import Flask, request, jsonify
app = Flask(__name__)
users = {}
# Non-idempotent: Each call creates a new user
@app.route('/users', methods=['POST'])
def create_user():
user_id = generate_uuid() # New ID each time
users[user_id] = request.json
return jsonify({'id': user_id}), 201
# Idempotent: Same payload always yields same state
@app.route('/users/<user_id>', methods=['PUT'])
def replace_user(user_id):
users[user_id] = request.json # Overwrites or creates
return jsonify({'id': user_id}), 200
Conditional Operations use ETags or version numbers to prevent conflicting updates. The server rejects requests with stale conditions.
Idempotency Keys are client-generated unique identifiers that the server uses to detect duplicates. This is the most flexible pattern and works for any operation type.
Implementing Idempotency Keys
The idempotency key pattern requires three components: a client-provided unique key, server-side storage for processed keys, and logic to detect and handle duplicates.
Here’s a complete implementation for a payment API:
import redis
import json
import hashlib
from functools import wraps
from flask import Flask, request, jsonify, g
app = Flask(__name__)
redis_client = redis.Redis(host='localhost', port=6379, db=0)
IDEMPOTENCY_TTL = 86400 # 24 hours
def idempotent(f):
@wraps(f)
def decorated(*args, **kwargs):
idempotency_key = request.headers.get('Idempotency-Key')
if not idempotency_key:
return jsonify({'error': 'Idempotency-Key header required'}), 400
# Create a composite key including the endpoint and method
cache_key = f"idempotency:{request.method}:{request.path}:{idempotency_key}"
# Check for existing response
cached = redis_client.get(cache_key)
if cached:
cached_data = json.loads(cached)
# Verify request payload matches (prevent key reuse with different data)
request_hash = hashlib.sha256(request.get_data()).hexdigest()
if cached_data['request_hash'] != request_hash:
return jsonify({'error': 'Idempotency key already used with different request'}), 422
# Return cached response
return jsonify(cached_data['response']), cached_data['status_code']
# Process the request
response, status_code = f(*args, **kwargs)
# Cache the response
cache_data = {
'request_hash': hashlib.sha256(request.get_data()).hexdigest(),
'response': response.get_json(),
'status_code': status_code
}
redis_client.setex(cache_key, IDEMPOTENCY_TTL, json.dumps(cache_data))
return response, status_code
return decorated
@app.route('/payments', methods=['POST'])
@idempotent
def create_payment():
data = request.json
# Process payment (charge card, update balances, etc.)
payment = process_payment(
amount=data['amount'],
currency=data['currency'],
customer_id=data['customer_id']
)
return jsonify({
'payment_id': payment.id,
'status': payment.status,
'amount': payment.amount
}), 201
The client generates a unique key (typically a UUID) and includes it with every request attempt:
import requests
import uuid
def create_payment_with_retry(amount, customer_id, max_retries=3):
idempotency_key = str(uuid.uuid4())
for attempt in range(max_retries):
try:
response = requests.post(
'https://api.example.com/payments',
json={'amount': amount, 'customer_id': customer_id, 'currency': 'USD'},
headers={'Idempotency-Key': idempotency_key},
timeout=30
)
return response.json()
except requests.exceptions.Timeout:
if attempt == max_retries - 1:
raise
continue # Retry with same idempotency key
Database-Level Idempotency
Sometimes the simplest solution is pushing idempotency enforcement to the database. Unique constraints and upserts handle many scenarios without application-level complexity.
-- Create table with natural business key constraint
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
external_order_id VARCHAR(255) NOT NULL,
customer_id INTEGER NOT NULL,
amount DECIMAL(10, 2) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(external_order_id) -- Prevents duplicate orders
);
-- Upsert: Insert or update if exists
INSERT INTO orders (external_order_id, customer_id, amount)
VALUES ('ORD-12345', 42, 99.99)
ON CONFLICT (external_order_id)
DO UPDATE SET
amount = EXCLUDED.amount,
customer_id = EXCLUDED.customer_id
RETURNING *;
In application code, handle the constraint violation gracefully:
from sqlalchemy.exc import IntegrityError
def create_order(external_id: str, customer_id: int, amount: float) -> Order:
try:
order = Order(
external_order_id=external_id,
customer_id=customer_id,
amount=amount
)
db.session.add(order)
db.session.commit()
return order
except IntegrityError:
db.session.rollback()
# Duplicate detected, return existing order
return Order.query.filter_by(external_order_id=external_id).first()
For optimistic locking, add a version column:
UPDATE accounts
SET balance = balance - 100, version = version + 1
WHERE id = 42 AND version = 5;
-- If no rows affected, concurrent modification occurred
Handling Edge Cases
The hardest problem isn’t detecting duplicates—it’s handling concurrent duplicates. Two identical requests arrive milliseconds apart, both check for existing records, both find none, both proceed.
Distributed locks solve this:
import redis
from contextlib import contextmanager
class IdempotencyLock:
def __init__(self, redis_client):
self.redis = redis_client
@contextmanager
def acquire(self, idempotency_key: str, timeout: int = 30):
lock_key = f"lock:idempotency:{idempotency_key}"
lock_value = str(uuid.uuid4())
# Try to acquire lock
acquired = self.redis.set(lock_key, lock_value, nx=True, ex=timeout)
if not acquired:
# Another request is processing this key
# Wait and return cached result
raise ConcurrentRequestError("Request already in progress")
try:
yield
finally:
# Release lock only if we still own it
if self.redis.get(lock_key) == lock_value.encode():
self.redis.delete(lock_key)
# Usage in endpoint
@app.route('/payments', methods=['POST'])
def create_payment():
idempotency_key = request.headers.get('Idempotency-Key')
lock = IdempotencyLock(redis_client)
try:
with lock.acquire(idempotency_key):
# Check cache, process request, store result
return process_payment_request()
except ConcurrentRequestError:
# Wait briefly and fetch cached result
time.sleep(0.5)
return get_cached_response(idempotency_key)
TTL considerations matter too. Keys must live long enough for retry windows but not forever. 24 hours works for most payment scenarios. For message queue consumers, match the message retention period.
Partial failures are trickier. If your operation involves multiple steps and fails midway, the idempotency record might indicate “in progress” indefinitely. Implement status tracking:
cache_data = {
'status': 'processing', # or 'completed', 'failed'
'started_at': datetime.utcnow().isoformat(),
'response': None
}
Testing Idempotent Operations
Idempotency bugs are insidious—they only manifest under specific timing conditions. Explicit testing is essential.
import pytest
from unittest.mock import patch, MagicMock
class TestPaymentIdempotency:
def test_duplicate_request_returns_same_response(self, client):
idempotency_key = 'test-key-123'
payment_data = {'amount': 100, 'customer_id': 42, 'currency': 'USD'}
# First request
response1 = client.post(
'/payments',
json=payment_data,
headers={'Idempotency-Key': idempotency_key}
)
# Second identical request
response2 = client.post(
'/payments',
json=payment_data,
headers={'Idempotency-Key': idempotency_key}
)
assert response1.status_code == 201
assert response2.status_code == 201
assert response1.json['payment_id'] == response2.json['payment_id']
def test_duplicate_request_does_not_charge_twice(self, client):
idempotency_key = 'test-key-456'
with patch('payments.charge_card') as mock_charge:
mock_charge.return_value = {'id': 'ch_123', 'status': 'succeeded'}
# Submit same request 5 times
for _ in range(5):
client.post(
'/payments',
json={'amount': 100, 'customer_id': 42, 'currency': 'USD'},
headers={'Idempotency-Key': idempotency_key}
)
# Charge should only be called once
assert mock_charge.call_count == 1
def test_different_payload_with_same_key_rejected(self, client):
idempotency_key = 'test-key-789'
client.post(
'/payments',
json={'amount': 100, 'customer_id': 42, 'currency': 'USD'},
headers={'Idempotency-Key': idempotency_key}
)
# Same key, different amount
response = client.post(
'/payments',
json={'amount': 200, 'customer_id': 42, 'currency': 'USD'},
headers={'Idempotency-Key': idempotency_key}
)
assert response.status_code == 422
Real-World Implementation Checklist
Before shipping idempotent endpoints, verify these items:
Storage: Use Redis for high-throughput scenarios (payments, orders). Use your primary database for lower-volume operations where you need transactional guarantees with the main operation.
Key Generation: Clients should generate UUIDs. Document this requirement in your API docs. Reject requests without idempotency keys on sensitive endpoints.
TTL Policy: Set TTL based on your retry window. 24 hours is standard for synchronous APIs. For async/webhook scenarios, extend to match your message retention.
Request Fingerprinting: Hash the request body and store it with the idempotency record. Reject reuse of keys with different payloads—this catches client bugs.
Monitoring: Track idempotency cache hit rates. High hit rates might indicate client issues or network problems. Alert on sudden spikes.
Client Guidance: Document retry behavior. Specify which errors are safe to retry, recommended backoff strategies, and idempotency key requirements.
Idempotency isn’t optional in distributed systems—it’s a requirement for correctness. The patterns are well-established. The implementation details matter. Get them right, and your system handles the chaos of real networks gracefully.