Atomic Idempotency Keys for Microservices using Redis and Lua Scripts

The Inevitability of Duplicate Requests in Distributed Systems

In any non-trivial distributed system, the specter of duplicate requests is not a matter of if, but when. Client-side retries after network timeouts, misconfigured webhooks, or complex service-to-service communication chains can all lead to the same logical operation being processed multiple times. For read operations, this is often benign. For state-changing mutations—processing a payment, creating an order, or transferring funds—it can be catastrophic.

Senior engineers understand that simply telling clients "don't send duplicate requests" is not a viable strategy. The responsibility for ensuring an operation is executed exactly once lies with the service provider. This property is known as idempotency.

This article eschews introductory concepts. We assume you're familiar with the Idempotency-Key HTTP header pattern and have likely considered or implemented naive solutions. We will dissect why those simple approaches fail under concurrent load and in the face of partial system failures, and then construct a production-ready, atomic solution using the unique capabilities of Redis and Lua scripting.

Why Naive Approaches Are Insufficient for Production

Let's briefly examine common but flawed patterns to establish our baseline for a superior solution.

* No In-Progress State: It cannot distinguish between a request that has completed and one that is currently in progress. If Request B arrives while Request A is still processing, it cannot find the key and will attempt an INSERT, resulting in a unique constraint violation. The service would typically return a generic 500 or 400 error, forcing the client to guess whether to retry or not.

No Response Caching: The pattern doesn't naturally lend itself to storing the result of the original successful operation. If a client retries a completed* request, the ideal response is not an error, but the original success response.

* Orphaned Keys: If the server crashes after inserting the key but before completing the business logic, the key is now permanently in the database, blocking all future retries for that operation.

* Crash State: Like the database unique constraint, if the server crashes after the SETNX but before completing the operation, the key is stuck in the IN_PROGRESS state indefinitely. Adding a TTL (SET key value EX seconds NX) helps, but forces the client to wait for the TTL to expire before a safe retry, which might be unacceptably long.

* Lack of Response Caching: This pattern doesn't provide a mechanism to store the final response.

A Robust Solution: State Machine with Atomic Redis Operations

To build a truly resilient system, we must model the lifecycle of an idempotent request as a state machine and ensure our state transitions are atomic. The states are:

To manage these states without race conditions, we cannot rely on multiple, separate Redis commands. The latency between a GET and a SET is a window for another process to intervene. The solution is to bundle our logic into a server-side Lua script, which Redis guarantees will execute atomically.

The Core Logic: The 'Start Processing' Lua Script

This script is the entry gate for any request carrying an Idempotency-Key. It decides whether to proceed, reject due to a conflict, or return a cached response.

Design Goals for the Script:

* Atomically check for the key's existence and its state.

* If UNSEEN, transition it to IN_PROGRESS with a short TTL (the lock timeout). This TTL is crucial for crash recovery.

* If IN_PROGRESS, signal a conflict.

* If COMPLETED, return the cached response data.

Here is the Lua script (start_processing.lua):

--KEYS[1]: The idempotency key (e.g., 'idem:uuid-v4-goes-here')
--ARGV[1]: The lock timeout in seconds (e.g., 30)

-- Check if the key exists
local existing_value = redis.call('GET', KEYS[1])

if existing_value == false then
  -- Key does not exist. This is the first time we've seen it.
  -- Set the key to a 'IN_PROGRESS' state with a lock TTL.
  -- The value is a simple JSON object indicating the state.
  redis.call('SET', KEYS[1], '{"status":"IN_PROGRESS"}', 'EX', ARGV[1])
  return 'PROCEED'
else
  -- Key exists. We need to inspect its state.
  local data = cjson.decode(existing_value)
  if data.status == 'IN_PROGRESS' then
    -- Another request is currently processing. Return a conflict.
    return 'CONFLICT'
  elseif data.status == 'COMPLETED' then
    -- The request was already completed. Return the cached response.
    return existing_value
  else
    -- Should not happen in a well-behaved system, but handle it.
    -- Could be a FAILED state or some other unexpected value.
    return 'ERROR_UNKNOWN_STATE'
  end
end

Analysis of the Script:

* KEYS[1] and ARGV[1] are the standard ways to pass arguments to Redis Lua scripts.

* redis.call() is used to execute Redis commands from within the script.

* cjson.decode() is used to parse the JSON string stored as the key's value. Redis includes a built-in cjson library for Lua scripts.

* The Return Values are Critical: We return simple strings ('PROCEED', 'CONFLICT') or the full cached data. Our application code will interpret these return values to drive its logic.

* The Lock TTL (ARGV[1]): This is a critical parameter. It should be longer than the expected P99 response time of your API endpoint but short enough that a genuine server crash doesn't lock out retries for an unreasonable period. A value between 15 and 60 seconds is often a reasonable starting point.

The 'Complete Processing' Lua Script

Once the business logic in your application succeeds, you must atomically update the key's state from IN_PROGRESS to COMPLETED and store the response.

Design Goals for the Script:

* Ensure the key still exists and is in the IN_PROGRESS state before updating (to prevent overwriting a key that has timed out and been re-used).

* Store the HTTP status code, headers, and body of the response.

* Set a new, longer TTL for the cached response.

Here is the Lua script (complete_processing.lua):

--KEYS[1]: The idempotency key
--ARGV[1]: The response data (JSON string: {status, headers, body})
--ARGV[2]: The cache TTL in seconds (e.g., 86400 for 24 hours)

local existing_value = redis.call('GET', KEYS[1])

if existing_value == false then
  -- The lock expired before we could complete. This is an edge case.
  -- The client may have already retried. We should not store the result.
  return 'LOCK_EXPIRED'
end

-- We could add a check here to ensure the state was 'IN_PROGRESS',
-- but for simplicity, we'll just overwrite. A more robust implementation might verify.
-- local data = cjson.decode(existing_value)
-- if data.status ~= 'IN_PROGRESS' then
--   return 'STATE_MISMATCH'
-- end

-- Atomically set the new value and the final TTL.
redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
return 'OK'

Analysis of the Script:

The LOCK_EXPIRED return value handles a significant edge case: what if your business logic takes longer than the lock TTL? The original lock expires, a client retries, and a new* IN_PROGRESS lock is acquired. When the original, slow operation finally finishes, it should not overwrite the state of the new attempt. This check prevents that.

* The final TTL (ARGV[2]) should be determined by your business requirements. How long do you need to protect against duplicate requests for the same operation? 24 hours is a common choice.

Production Implementation: Node.js and Express.js Middleware

Let's translate this powerful Redis/Lua pattern into a reusable piece of infrastructure for a Node.js application using express and the ioredis library.

1. Setting up the Redis Client and Loading Scripts

ioredis has a convenient way to define and load Lua scripts, automatically handling the EVALSHA optimization (sending the SHA1 hash of the script instead of the full script on subsequent calls).

// redisClient.js
const Redis = require('ioredis');
const fs = require('fs');
const path = require('path');

// In a real app, use environment variables
const redis = new Redis({
  port: 6379,
  host: '127.0.0.1',
});

// Define the custom commands by loading the Lua scripts
redis.defineCommand('startIdempotentRequest', {
  numberOfKeys: 1,
  lua: fs.readFileSync(path.join(__dirname, 'lua/start_processing.lua'), 'utf8'),
});

redis.defineCommand('completeIdempotentRequest', {
  numberOfKeys: 1,
  lua: fs.readFileSync(path.join(__dirname, 'lua/complete_processing.lua'), 'utf8'),
});

redis.on('error', (err) => console.error('Redis Client Error', err));

module.exports = redis;

2. The Idempotency Middleware

This middleware will encapsulate the entire logic flow.

// idempotencyMiddleware.js
const redis = require('./redisClient');

const IDEMPOTENCY_KEY_HEADER = 'Idempotency-Key';
const LOCK_TTL_SECONDS = 30;
const CACHE_TTL_SECONDS = 24 * 60 * 60; // 24 hours

const idempotencyMiddleware = async (req, res, next) => {
  const idempotencyKey = req.get(IDEMPOTENCY_KEY_HEADER);

  // If no key is provided, proceed without idempotency guarantees.
  // Alternatively, you could reject the request.
  if (!idempotencyKey) {
    return next();
  }

  const key = `idem:${idempotencyKey}`;

  try {
    const result = await redis.startIdempotentRequest(key, LOCK_TTL_SECONDS);

    if (result === 'PROCEED') {
      // This is the first time we see this key. Attach response handlers.
      attachResponseListeners(res, key);
      return next();
    } else if (result === 'CONFLICT') {
      // A request with the same key is already in progress.
      return res.status(409).json({ message: 'Request in progress. Please try again later.' });
    } else {
      // The result is the cached response data.
      try {
        const cachedData = JSON.parse(result);
        // Important: Replay the headers as well!
        Object.entries(cachedData.headers).forEach(([header, value]) => {
          res.setHeader(header, value);
        });
        return res.status(cachedData.statusCode).send(cachedData.body);
      } catch (e) {
        console.error('Failed to parse cached idempotency data:', e);
        // Fallback to a generic error if cached data is corrupt
        return res.status(500).json({ message: 'Internal server error retrieving cached response.' });
      }
    }
  } catch (error) {
    console.error('Redis error during idempotency check:', error);
    // If Redis is down, we can choose to fail open or fail closed.
    // Failing closed (503) is safer for mutations.
    return res.status(503).json({ message: 'Service temporarily unavailable.' });
  }
};

// Helper to capture the response and store it in Redis
const attachResponseListeners = (res, key) => {
  const originalSend = res.send;
  const originalJson = res.json;
  const chunks = [];

  // We need to capture the response body. This is a simplified way.
  // For full support (streams, etc.), a more robust library might be needed.
  res.send = function (body) {
    if (res.statusCode >= 200 && res.statusCode < 300) {
        const responseData = {
            status: 'COMPLETED',
            statusCode: res.statusCode,
            headers: res.getHeaders(),
            body: body,
        };
        redis.completeIdempotentRequest(key, JSON.stringify(responseData), CACHE_TTL_SECONDS)
            .catch(err => console.error('Failed to save idempotent response:', err));
    }
    return originalSend.apply(res, arguments);
  };

  res.json = function (body) {
    if (res.statusCode >= 200 && res.statusCode < 300) {
        const responseData = {
            status: 'COMPLETED',
            statusCode: res.statusCode,
            headers: res.getHeaders(),
            body: body,
        };
        // Use stringify twice for JSON body to embed it correctly in the outer JSON
        redis.completeIdempotentRequest(key, JSON.stringify(responseData), CACHE_TTL_SECONDS)
            .catch(err => console.error('Failed to save idempotent response:', err));
    }
    return originalJson.apply(res, arguments);
  }

  // Also handle failures. If the request fails, we should not cache the response,
  // and we should release the lock to allow for retries.
  res.on('finish', () => {
    if (res.statusCode >= 400) {
        // On client or server error, delete the 'IN_PROGRESS' key to allow retries.
        redis.del(key).catch(err => console.error('Failed to delete failed idempotency key:', err));
    }
  });

   res.on('close', () => {
    // Handle cases where the client closes the connection prematurely
    // Check if the response was sent. If not, the request was aborted.
    if (!res.headersSent) {
      redis.del(key).catch(err => console.error('Failed to delete aborted idempotency key:', err));
    }
  });
};

module.exports = idempotencyMiddleware;

Key Implementation Details:

* Monkey-Patching res.send / res.json: This is a common pattern in Express middleware to intercept the response just before it's sent to the client. It's the ideal moment to capture the final status, headers, and body for caching.

* Handling Failures: Crucially, if the request results in an error (4xx or 5xx status code), we explicitly DEL the IN_PROGRESS key. This releases the lock immediately, allowing a client to correct their request and retry without waiting for the lock TTL to expire.

* Handling Aborted Requests: The res.on('close', ...) handler is an important edge case. If a client disconnects before the server can respond, we should clean up the lock to prevent that operation from being blocked for the full TTL.

* Redis Unavailability: The top-level try...catch block handles Redis being down. In this scenario, for a critical mutation, it is safer to fail closed by returning a 503 Service Unavailable. This prevents any possibility of duplicate processing. Failing open (letting the request proceed) would temporarily disable idempotency guarantees.

3. Using the Middleware

Applying the middleware to a protected route is now trivial.

const express = require('express');
const idempotencyMiddleware = require('./idempotencyMiddleware');

const app = express();
app.use(express.json());

// A mock payment processing function
const processPayment = async (amount, currency, reference) => {
  console.log(`Processing payment of ${amount} ${currency} for ${reference}...`);
  // Simulate network latency and work
  await new Promise(resolve => setTimeout(resolve, 1000));
  console.log('Payment successful.');
  return { transactionId: `txn_${Date.now()}` };
};

app.post('/api/payments', idempotencyMiddleware, async (req, res) => {
  try {
    const { amount, currency, reference } = req.body;
    if (!amount || !currency || !reference) {
      return res.status(400).json({ message: 'Missing required payment details.' });
    }

    const result = await processPayment(amount, currency, reference);
    res.status(201).json({ status: 'SUCCESS', ...result });

  } catch (error) {
    console.error('Payment processing failed:', error);
    res.status(500).json({ message: 'An internal error occurred during payment processing.' });
  }
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

Advanced Considerations and Performance Tuning

While the above implementation is robust, senior engineers must consider the impact on a large-scale, high-performance system.

1. Redis Cluster and Hash Tags

In a standard Redis Cluster setup, keys are distributed across different shards based on a hash of the key name. This poses a problem for Lua scripts, as a script can only operate on keys that reside on the same shard. Our scripts only use one key, so they are safe by default.

However, if you were to expand this pattern (e.g., to also update a counter atomically with the idempotency key), you would need to ensure both keys land on the same shard. This is achieved using hash tags. Any part of a key enclosed in {} is used for the hash calculation. For example, {idem:user123}:key and {idem:user123}:counter are guaranteed to be on the same shard.

2. Memory Footprint of Cached Responses

Storing the full HTTP response body in Redis can consume significant memory, especially if responses are large. Consider these strategies:

* Store Only What's Necessary: Instead of the full body, perhaps you only need to store the transactionId or a success message. Modify the complete_processing.lua script and middleware to store a more compact object.

* External Blob Storage: For very large responses (e.g., generated PDF reports), store the response in a dedicated blob store like Amazon S3. The Redis cache entry would then just contain the S3 object URL, keeping the Redis memory footprint minimal.

* Response Compression: Before JSON.stringify and storing in Redis, you could compress the response body using a library like zlib to trade CPU cycles for memory.

3. Garbage Collection and Key Eviction

Our TTL-based approach is a form of garbage collection. However, ensure your Redis instance is configured with an appropriate eviction policy (e.g., volatile-lru or volatile-ttl) to gracefully handle memory pressure. If Redis runs out of memory, it needs to know which keys to evict first. By setting TTLs on all our idempotency keys, we make them candidates for eviction under policies that target keys with an expire set.

4. Client-Side Responsibilities

While this is a server-side pattern, its effectiveness depends on well-behaved clients.

Key Generation: Clients should be instructed to generate a high-entropy key, like a UUIDv4, for each distinct operation they wish to perform. The same key must* be used for retries of the same operation.

* Retry Logic: Clients should implement an exponential backoff strategy for retries, especially when receiving a 409 Conflict or 503 Service Unavailable response. They should not hammer the API.

Conclusion

Implementing idempotency is a non-negotiable requirement for reliable mutable APIs in a distributed environment. By moving beyond simple, race-condition-prone patterns and embracing an atomic, state-machine-based approach with Redis and Lua, we can build a system that is resilient to network failures, client retries, and server crashes. The combination of a short-lived IN_PROGRESS lock and a long-lived COMPLETED cached response provides both safety during processing and efficiency for subsequent retries. This pattern, encapsulated in reusable middleware, provides a powerful tool in the arsenal of any senior engineer tasked with building robust, production-grade microservices.