Idempotency Patterns for Asynchronous APIs with Redis and Lua

The Idempotency Imperative in Modern Architectures

In distributed systems, particularly those built on event-driven or microservice architectures, the contract of message delivery is rarely 'exactly-once'. Systems like Apache Kafka, RabbitMQ, and AWS SQS typically offer 'at-least-once' delivery guarantees. This practical trade-off ensures message durability at the cost of potential duplicates. A network partition, a consumer crash post-processing but pre-acknowledgment, or a simple client-side retry can all lead to the same logical operation being processed multiple times.

For read operations, this is often benign. For state-changing write operations, it's a critical failure point. Imagine a payment API where a retry charges a customer twice, or a notification service that bombards a user with duplicate messages. The business impact is severe, eroding user trust and causing data corruption.

This is where idempotency becomes a non-negotiable requirement. An idempotent operation is one that can be applied multiple times without changing the result beyond the initial application. The responsibility for ensuring idempotency falls on the service provider (the API endpoint or message consumer).

The standard mechanism for achieving this is the Idempotency-Key, a unique client-generated value (typically a UUIDv4) sent with each state-changing request. The server uses this key to track the status of an operation, ensuring that subsequent requests with the same key do not re-execute the business logic.

This article dissects the implementation of a robust, high-performance idempotency layer using Redis. We will start by exposing the flaws in a naive approach and build up to a production-grade solution using atomic, server-side Lua scripting.

The Naive Implementation: A Recipe for Race Conditions

A senior engineer's first instinct might be to use a simple check-then-act pattern with a key-value store like Redis.

Let's model the logic for a hypothetical payment processing endpoint:

• Server receives the request.

• If it exists, assume it was processed and return a cached response.

Here is what that might look like in Go code (this code is intentionally flawed):

// WARNING: THIS CODE CONTAINS A CRITICAL RACE CONDITION AND IS NOT FOR PRODUCTION USE.
func NaiveIdempotencyHandler(c *gin.Context) {
    idempotencyKey := c.GetHeader("Idempotency-Key")
    if idempotencyKey == "" {
        c.JSON(http.StatusBadRequest, gin.H{"error": "Idempotency-Key header is required"})
        return
    }

    redisKey := "idempotency:" + idempotencyKey

    // 1. CHECK
    cachedResponse, err := rdb.Get(ctx, redisKey).Result()
    if err == nil {
        // Key exists, return cached response
        c.Data(http.StatusOK, "application/json", []byte(cachedResponse))
        return
    }
    if err != redis.Nil {
        // A real Redis error occurred
        c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to check idempotency key"})
        return
    }

    // Key does not exist, proceed with processing
    // ... business logic to process payment ...
    paymentResult := processPayment(c.Request.Body)
    responseBytes, _ := json.Marshal(paymentResult)

    // 2. ACT (SET)
    err = rdb.Set(ctx, redisKey, responseBytes, 24*time.Hour).Err()
    if err != nil {
        // Failed to cache the response, but the payment was processed!
        // This creates an inconsistent state.
        c.JSON(http.StatusInternalServerError, gin.H{"error": "Payment processed but failed to save idempotency record"})
        return
    }

    c.Data(http.StatusOK, "application/json", responseBytes)
}

The Failure Mode

The gap between the GET (CHECK) and SET (ACT) operations is a critical race window. Consider this sequence of events with two concurrent requests (Request A and Request B) arriving with the same Idempotency-Key:

The result is a double payment. This naive approach is fundamentally broken for any system with non-trivial concurrency.

A More Robust Approach: The Three-State Lock

To solve the race condition, we need to make the check-and-set operation atomic. We also need to handle the state of a request that is currently being processed. This leads to a more sophisticated three-state model for our idempotency key:

The workflow now becomes:

* On success, update the key from PENDING to COMPLETED, storing the actual response and setting a longer TTL (e.g., 24 hours).

* On failure, delete the PENDING key to allow a legitimate retry to proceed.

* If the value is COMPLETED, deserialize the stored response and return it immediately.

* If the value is PENDING, it means another thread/process is actively working on this operation. You should not wait. Instead, return an immediate conflict response (e.g., HTTP 409 Conflict), signaling to the client that they should retry after a short delay.

This model prevents the race condition by establishing an atomic lock and provides clear handling for in-flight requests.

Production-Grade Implementation with Redis

Now, let's implement this three-state model. The core challenge is atomicity. Redis provides two primary mechanisms for this: Transactions (MULTI/EXEC) and Lua scripting.

Solution 1: Redis Transactions (`MULTI`/`EXEC` with `WATCH`)

Redis transactions allow you to group a set of commands that are executed as a single, atomic operation. To handle the conditional logic (i.e., 'set only if not exists'), we need to use optimistic locking with the WATCH command.

WATCH monitors a key for modifications. If the watched key is modified by another client before EXEC is called, the entire transaction will fail, and the client library will typically return an error, allowing you to retry the entire read-modify-write cycle.

Here’s how you might implement the initial locking phase in Go:

import (
    "context"
    "encoding/json"
    "errors"
    "github.com/go-redis/redis/v8"
    "time"
)

// Represents the stored idempotency data
type IdempotencyRecord struct {
    Status   string `json:"status"` // PENDING or COMPLETED
    Response []byte `json:"response,omitempty"`
}

// Attempts to acquire a lock using MULTI/EXEC. Returns true if lock acquired.
func acquireLockWithTx(ctx context.Context, rdb *redis.Client, key string, pendingTTL time.Duration) (bool, *IdempotencyRecord, error) {
    var record *IdempotencyRecord

    // The transaction function.
    txf := func(tx *redis.Tx) error {
        // Inside a transaction, all commands are queued.
        // First, check the key's current value.
        val, err := tx.Get(ctx, key).Result()
        if err != nil && err != redis.Nil {
            return err // Real Redis error
        }

        if err == nil {
            // Key already exists. Unmarshal to check its state.
            var existingRecord IdempotencyRecord
            if err := json.Unmarshal([]byte(val), &existingRecord); err != nil {
                return errors.New("corrupted idempotency record")
            }
            record = &existingRecord // Store for return
            return nil // Don't modify, just read
        }

        // Key does not exist (redis.Nil). We can try to set it.
        // Queue the Pipelined commands.
        _, err = tx.Pipelined(ctx, func(pipe redis.Pipeliner) error {
            pendingRecord := IdempotencyRecord{Status: "PENDING"}
            pendingBytes, _ := json.Marshal(pendingRecord)
            pipe.Set(ctx, key, pendingBytes, pendingTTL)
            return nil
        })
        return err
    }

    // Retry loop for the transaction
    for i := 0; i < 3; i++ {
        err := rdb.Watch(ctx, txf, key)
        if err == nil {
            // Success!
            if record != nil {
                return false, record, nil // Lock not acquired, key existed
            }
            return true, nil, nil // Lock acquired!
        }
        if err == redis.TxFailedErr {
            // Optimistic lock failed. Another client modified the key.
            // Retry the transaction.
            continue
        }
        // A real error occurred.
        return false, nil, err
    }

    return false, nil, errors.New("failed to acquire idempotency lock after retries")
}

Analysis of the MULTI/EXEC Approach:

* Pros: It uses standard Redis commands and is conceptually understandable as optimistic locking. Most client libraries have good support for it.

* Cons:

* Performance: It's chatty. The WATCH, GET, and MULTI/EXEC commands involve multiple network round-trips. Under high contention, TxFailedErr can cause multiple client-side retries, increasing latency.

* Complexity: The client-side retry logic adds complexity to the application code.

While functional, this approach is often suboptimal for high-throughput systems where idempotency checks are on the critical path.

Solution 2: Lua Scripting (The Superior Approach)

A much more efficient and robust solution is to move the conditional logic to the server side using a Lua script. Redis guarantees that Lua scripts are executed atomically. A single script can perform the entire check-and-set logic in one network round-trip, eliminating race conditions and client-side retry loops.

Here is the Lua script to atomically check for a key and set it to PENDING if it doesn't exist.

acquire_lock.lua

-- KEYS[1] - The idempotency key
-- ARGV[1] - The pending record payload (e.g., '{"status":"PENDING"}')
-- ARGV[2] - The TTL for the pending record in seconds

local existing_val = redis.call('GET', KEYS[1])

-- If the key already exists, return its value
if existing_val then
  return existing_val
end

-- If the key does not exist, set it to the PENDING state with a TTL
-- and return 'OK' to signify lock acquisition.
redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
return 'ACQUIRED'

Now, the Go application code becomes much cleaner and more performant. We use EVALSHA to execute the script, which is optimal as Redis caches the script by its SHA1 hash after the first SCRIPT LOAD.

// Go code to execute the Lua script
var acquireLockScript = redis.NewScript(`
    local existing_val = redis.call('GET', KEYS[1])
    if existing_val then
      return existing_val
    end
    redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
    return 'ACQUIRED'
`)

// acquireLockWithLua is much simpler and more performant.
func acquireLockWithLua(ctx context.Context, rdb *redis.Client, key string, pendingTTL time.Duration) (bool, *IdempotencyRecord, error) {
    pendingRecord := IdempotencyRecord{Status: "PENDING"}
    pendingBytes, _ := json.Marshal(pendingRecord)

    res, err := acquireLockScript.Run(ctx, rdb, []string{key}, pendingBytes, pendingTTL.Seconds()).Result()
    if err != nil {
        return false, nil, err
    }

    resultStr, ok := res.(string)
    if !ok {
        return false, nil, errors.New("unexpected response type from Lua script")
    }

    if resultStr == "ACQUIRED" {
        // We got the lock!
        return true, nil, nil
    }

    // The key already existed, the script returned its value.
    var existingRecord IdempotencyRecord
    if err := json.Unmarshal([]byte(resultStr), &existingRecord); err != nil {
        return false, nil, errors.New("corrupted idempotency record")
    }
    return false, &existingRecord, nil
}

This is a complete idempotency middleware for Gin, integrating the Lua-based locking:

// Full Middleware Implementation
func IdempotencyMiddleware(rdb *redis.Client) gin.HandlerFunc {
    return func(c *gin.Context) {
        // Only apply to state-changing methods
        if c.Request.Method != "POST" && c.Request.Method != "PUT" && c.Request.Method != "PATCH" {
            c.Next()
            return
        }

        idempotencyKey := c.GetHeader("Idempotency-Key")
        if idempotencyKey == "" {
            c.Next()
            return // Or return 400 Bad Request if mandatory
        }

        redisKey := "idempotency:" + idempotencyKey

        // 1. Try to acquire the lock
        lockAcquired, existingRecord, err := acquireLockWithLua(c.Request.Context(), rdb, redisKey, 5*time.Minute)
        if err != nil {
            c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "idempotency check failed"})
            return
        }

        if !lockAcquired {
            // Lock not acquired, another request is active or completed
            if existingRecord.Status == "COMPLETED" {
                c.AbortWithStatusJSON(http.StatusOK, json.RawMessage(existingRecord.Response))
                return
            }
            if existingRecord.Status == "PENDING" {
                c.AbortWithStatusJSON(http.StatusConflict, gin.H{"error": "request processing in progress"})
                return
            }
        }

        // 2. Lock Acquired. Defer cleanup in case of panic.
        defer func() {
            // If a panic occurs, the PENDING key will expire via its TTL.
            // A more robust implementation could use a recovery middleware to explicitly delete the key.
        }()

        // Replace the response writer to capture the response
        blw := &bodyLogWriter{body: bytes.NewBufferString(""), ResponseWriter: c.Writer}
        c.Writer = blw

        c.Next() // Execute the actual handler

        // 3. After handler execution, update the record
        statusCode := c.Writer.Status()

        if statusCode >= 200 && statusCode < 300 {
            // Success. Store the result.
            responseBody := blw.body.Bytes()
            completedRecord := IdempotencyRecord{Status: "COMPLETED", Response: responseBody}
            completedBytes, _ := json.Marshal(completedRecord)
            rdb.Set(c.Request.Context(), redisKey, completedBytes, 24*time.Hour)
        } else {
            // Failure. Delete the pending key to allow retries.
            rdb.Del(c.Request.Context(), redisKey)
        }
    }
}

// Helper to capture response body
type bodyLogWriter struct {
    gin.ResponseWriter
    body *bytes.Buffer
}
func (w bodyLogWriter) Write(b []byte) (int, error) {
    w.body.Write(b)
    return w.ResponseWriter.Write(b)
}

Advanced Considerations and Edge Cases

A production-ready system requires thinking beyond the happy path.

Key Expiration and Garbage Collection

* PENDING Key TTL: This is a crucial safety mechanism. If a process acquires a lock and then crashes without cleaning it up, the PENDING key will prevent any further processing for that operation. A short TTL (e.g., 1-5 minutes) ensures the lock is eventually released. This TTL should be longer than your expected maximum processing time but short enough to prevent prolonged outages.

* COMPLETED Key TTL: The TTL for completed records is a business decision. 24 hours is a common choice, balancing the client's retry window against Redis memory usage. For financial transactions, this might be extended to 48-72 hours.

Storing the Response

* Payload Size: Storing the full HTTP response in Redis is convenient but risky if responses can be large. A 1MB response body is manageable; a 100MB response is not. This can lead to high memory usage and network saturation.

* Large Payload Strategy: For services that return large payloads, a hybrid approach is better. Store a small COMPLETED record in Redis that contains a pointer (e.g., a URL) to the full response stored in a blob store like Amazon S3. The idempotency record would look like {"status":"COMPLETED", "location":"s3://my-bucket/results/f1c2a3b4-..."}.

* Serialization: JSON is readable but can be verbose. For performance-critical systems, consider more compact binary formats like MessagePack or Protobuf to reduce the size of the data stored in Redis and decrease network transfer time.

Error and Failure Handling

* Business Logic Failure: As shown in the middleware, if the handler returns a non-2xx status code, it's critical to DEL the PENDING key. This allows the client to attempt a clean retry. Failure to do so would block the operation until the PENDING TTL expires.

* Redis Unavailability: If Redis is down, the idempotency check fails. The correct behavior is almost always to fail the request with a 5xx error. Proceeding non-idempotently is a dangerous default. This underscores the need for a highly available Redis deployment (e.g., Redis Sentinel or Cluster).

Client-Side Behavior

* Key Generation: The client MUST generate a high-entropy unique key. UUIDv4 is the standard. A poorly generated key (e.g., based on a timestamp with low precision) could cause unintentional collisions.

* Handling 409 Conflict: When a client receives a 409 Conflict (indicating a PENDING state), it should not immediately retry. It should implement an exponential backoff strategy (e.g., retry after 1s, then 2s, then 4s) to give the in-flight operation time to complete.

Performance Benchmarking: `MULTI/EXEC` vs. Lua

To quantify the performance difference, we can set up a benchmark using a tool like bombardier or a custom Go test. The test should simulate high concurrency against an endpoint protected by each idempotency implementation.

Hypothetical Benchmark Scenario:

* Target: A simple Gin endpoint that sleeps for 50ms to simulate work.

* Concurrency: 200 concurrent clients.

* Test: Each client sends 10 requests with the same Idempotency-Key.

Expected Results:

These results clearly demonstrate that for any serious, production-level workload, server-side Lua scripting is the unequivocally superior choice for implementing complex atomic patterns in Redis.

Conclusion

Implementing idempotency is not an optional extra in modern distributed systems; it's a fundamental requirement for correctness and reliability. While a naive GET/SET pattern is dangerously flawed, a robust three-state (PENDING, COMPLETED) model provides the necessary guarantees to handle concurrency and failures.

By leveraging the atomicity of Redis Lua scripts, we can build an idempotency layer that is not only correct but also highly performant, capable of handling significant load without introducing latency bottlenecks. This pattern, implemented as a middleware, provides a clean separation of concerns, allowing your application's business logic to remain blissfully unaware of the complexities of at-least-once message delivery, confident that no operation will ever be processed more than once.