Idempotent DynamoDB Writes with Condition Expressions in EDA

The Idempotency Challenge in Asynchronous Systems

In any non-trivial event-driven architecture (EDA), services communicate asynchronously via message brokers like Amazon SQS, Kafka, or EventBridge. A fundamental contract of these systems is at-least-once delivery. This guarantee ensures message durability but introduces a critical challenge for consumers: the same message may be delivered, and therefore processed, more than once. This can occur due to network partitions, consumer crashes after processing but before acknowledgement, or broker-side redelivery mechanisms.

Processing a CreateOrder event twice results in duplicate orders. Processing a DebitAccount event twice leads to incorrect financial state. The responsibility for handling these duplicates falls squarely on the consumer application. This is the principle of idempotency: an operation, when performed multiple times, has the same effect as if it were performed only once.

Traditional solutions to enforce idempotency often introduce significant complexity and external dependencies:

This article presents a superior pattern for services using DynamoDB as their primary data store. By leveraging DynamoDB's ConditionExpression parameter, we can achieve atomic, high-performance, and cost-effective idempotency directly at the database layer. This offloads the complexity of state management, eliminates the need for external dependencies, and simplifies our application code, allowing it to focus purely on business logic.

The Core Pattern: Conditional Writes and Idempotency Keys

The foundation of this pattern lies in the ConditionExpression parameter available in DynamoDB's data manipulation operations (PutItem, UpdateItem, DeleteItem). A ConditionExpression is a string containing a condition that must evaluate to true for the operation to proceed. If the condition is false, the operation fails with a ConditionalCheckFailedException.

For idempotency, the most powerful function within these expressions is attribute_not_exists(path). It evaluates to true only if the item being written does not already have the specified attribute. This gives us an atomic "check-and-set" capability.

The Strategy:

• The consumer extracts this key from the event.

Let's implement this for a user creation service. Our event from SQS might look like this:

{
  "eventType": "UserCreated",
  "payload": {
    "userId": "user-123",
    "email": "[email protected]",
    "name": "Jane Doe"
  },
  "metadata": {
    "idempotencyKey": "a1b2c3d4-e5f6-7890-1234-567890abcdef"
  }
}

Our goal is to write a new item to our Users table but only if we haven't processed this idempotencyKey before. We can use the idempotency key itself as the primary key of a dedicated idempotency record, but a more efficient pattern for create operations is to use a dedicated attribute on the primary business item.

However, for a Create operation, the simplest condition is to ensure the item itself doesn't exist. The idempotencyKey is a logical concept ensuring the operation is unique, and the ConditionExpression enforces the state change is unique.

Code Example 1: Idempotent `PutItem` for User Creation

Here's a TypeScript example using the AWS SDK v3 within an AWS Lambda function. The primary key for our Users table is userId.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({});
const docClient = DynamoDBDocumentClient.from(client);

const USERS_TABLE_NAME = process.env.USERS_TABLE_NAME!;

interface UserCreatedEvent {
  payload: {
    userId: string;
    email: string;
    name: string;
  };
  metadata: {
    idempotencyKey: string;
  };
}

export const handler = async (event: { Records: { body: string }[] }): Promise<void> => {
  for (const record of event.Records) {
    const body: UserCreatedEvent = JSON.parse(record.body);
    const { userId, email, name } = body.payload;
    const { idempotencyKey } = body.metadata;

    console.log(`Processing user creation for userId: ${userId} with idempotencyKey: ${idempotencyKey}`);

    const command = new PutCommand({
      TableName: USERS_TABLE_NAME,
      Item: {
        userId: userId,
        email: email,
        name: name,
        createdAt: new Date().toISOString(),
      },
      // This is the core of the idempotency pattern for creation
      ConditionExpression: "attribute_not_exists(userId)",
    });

    try {
      await docClient.send(command);
      console.log(`Successfully created user ${userId}.`);
      // Proceed with any subsequent logic (e.g., emitting another event)

    } catch (error: any) {
      if (error.name === 'ConditionalCheckFailedException') {
        // This is not a failure. It's a successful detection of a duplicate request.
        console.log(`Idempotency check failed: User ${userId} already exists. This is a duplicate event and will be ignored.`);
        // Acknowledge the message from the queue by simply returning successfully.
      } else {
        // Any other error is a real failure and should be retried.
        console.error(`Failed to create user ${userId}. Error:`, error);
        throw error; // This will cause the Lambda to fail, and SQS will retry the message.
      }
    }
  }
};

In this scenario, if the same UserCreated event is delivered twice, the first PutCommand will succeed. The second PutCommand will attempt to write an item with the same userId primary key. The ConditionExpression: "attribute_not_exists(userId)" will evaluate to false, DynamoDB will reject the write, and the SDK will throw a ConditionalCheckFailedException. Our catch block correctly interprets this exception not as a system failure, but as a successful idempotency check, and exits gracefully.

Advanced Scenario: Idempotent Updates and State Transitions

The attribute_not_exists pattern works perfectly for creation events. But what about events that modify existing items? Consider an OrderUpdated event that adds a new item to an order's items list. If we process this event twice, the item will be added twice.

Here, we can't check for the existence of the primary key, as the order item already exists. We need a way to track which events have been processed for a given item.

The solution is to maintain a set of processed idempotency keys within the DynamoDB item itself. A DynamoDB String Set (SS) is perfect for this.

The Strategy:

* It executes the required business logic (e.g., adding an item to a list).

* It adds the event's idempotencyKey to a processedEvents attribute (a String Set).

Because both the state change and the idempotency key tracking happen in a single, atomic UpdateItem call, we eliminate race conditions.

Code Example 2: Idempotent `UpdateItem` for Order Processing

Let's model an Order item that looks like this:

{
  "orderId": "order-abc-123",
  "status": "PENDING",
  "items": [
    { "sku": "SKU001", "quantity": 1 }
  ],
  "processedEvents": ["key-for-create-event"]
}

Now, we receive an event to add a new item to this order:

{
  "eventType": "OrderItemAdded",
  "payload": {
    "orderId": "order-abc-123",
    "item": { "sku": "SKU002", "quantity": 2 }
  },
  "metadata": {
    "idempotencyKey": "fghij-67890-klmno-12345"
  }
}

Here's the Lambda handler to process this idempotently:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, UpdateCommand } from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({});
const docClient = DynamoDBDocumentClient.from(client);

const ORDERS_TABLE_NAME = process.env.ORDERS_TABLE_NAME!;

interface OrderItemAddedEvent {
  payload: {
    orderId: string;
    item: { sku: string; quantity: number };
  };
  metadata: {
    idempotencyKey: string;
  };
}

export const handler = async (event: { Records: { body: string }[] }): Promise<void> => {
  for (const record of event.Records) {
    const body: OrderItemAddedEvent = JSON.parse(record.body);
    const { orderId, item } = body.payload;
    const { idempotencyKey } = body.metadata;

    console.log(`Processing OrderItemAdded for orderId: ${orderId} with key: ${idempotencyKey}`);

    const command = new UpdateCommand({
      TableName: ORDERS_TABLE_NAME,
      Key: {
        orderId: orderId,
      },
      // Atomically update the items list and add the idempotency key to the processed set
      UpdateExpression: "SET #items = list_append(if_not_exists(#items, :empty_list), :new_item) ADD #processedEvents :idempotencyKey",
      
      // Only perform the update if the idempotency key is NOT in the processedEvents set
      ConditionExpression: "attribute_exists(orderId) AND NOT contains(#processedEvents, :idempotencyKeyValue)",
      
      ExpressionAttributeNames: {
        "#items": "items",
        "#processedEvents": "processedEvents",
      },
      ExpressionAttributeValues: {
        ":new_item": [item],
        ":empty_list": [],
        ":idempotencyKey": new Set([idempotencyKey]), // ADD action requires a Set
        ":idempotencyKeyValue": idempotencyKey // contains() function requires a String
      },
      ReturnValues: "UPDATED_NEW",
    });

    try {
      const result = await docClient.send(command);
      console.log(`Successfully updated order ${orderId}.`, result.Attributes);

    } catch (error: any) {
      if (error.name === 'ConditionalCheckFailedException') {
        console.log(`Idempotency check failed for order ${orderId} with key ${idempotencyKey}. Duplicate event ignored.`);
      } else {
        console.error(`Failed to update order ${orderId}. Error:`, error);
        throw error;
      }
    }
  }
};

Dissecting the UpdateCommand:

* UpdateExpression: SET #items = list_append(if_not_exists(#items, :empty_list), :new_item) safely appends to the items list, initializing it as an empty list if it doesn't exist. ADD #processedEvents :idempotencyKey adds our new key to the String Set. The ADD action is idempotent by nature for sets; adding an existing value has no effect.

* ConditionExpression: attribute_exists(orderId) AND NOT contains(#processedEvents, :idempotencyKeyValue) is the crucial part. It ensures the order item actually exists AND that the processedEvents set does not contain the key from our current event.

* ExpressionAttributeValues: Note the distinction between :idempotencyKey (a Set, for the ADD action) and :idempotencyKeyValue (a String, for the contains function). This is a nuance of DynamoDB's expression syntax.

This pattern is incredibly robust. It handles the business logic and idempotency check in a single atomic, server-side operation, making it immune to client-side race conditions.

Handling Race Conditions and `ConditionalCheckFailedException`

A common question from engineers new to this pattern is: "What if two Lambda invocations process the exact same message at the exact same time?" This is where the beauty of DynamoDB's atomicity shines.

Let's trace the concurrent execution for our OrderItemAdded example:

• DynamoDB receives both requests. Internally, it serializes operations on a single item. One request will be processed first—let's say it's Invocation A.

Your application code receives these distinct outcomes:

* Invocation A proceeds down the try path, logs success, and exits. The SQS message is successfully deleted.

* Invocation B jumps to the catch block, identifies the ConditionalCheckFailedException, logs it as a harmless duplicate, and also exits successfully.

This guarantees exactly-once processing despite at-least-once delivery. The ConditionalCheckFailedException is not an error to be retried; it is the success signal that your idempotency logic is working correctly.

Code Example 3: A Production-Ready Handler Structure

This example shows a more robust structure with clear logging and error handling, suitable for production.

// ... imports and client setup ...

const processRecord = async (record: { body: string }): Promise<void> => {
    const body: OrderItemAddedEvent = JSON.parse(record.body);
    const { orderId, item } = body.payload;
    const { idempotencyKey } = body.metadata;

    const logger = { // A structured logger would be better in a real app
        info: (message: string, data: object) => console.log(JSON.stringify({ level: 'INFO', message, ...data })),
        warn: (message: string, data: object) => console.warn(JSON.stringify({ level: 'WARN', message, ...data })),
        error: (message: string, data: object) => console.error(JSON.stringify({ level: 'ERROR', message, ...data }))
    };

    const logContext = { orderId, idempotencyKey };
    logger.info("Starting event processing", logContext);

    const command = new UpdateCommand({ /* ... command from Example 2 ... */ });

    try {
        await docClient.send(command);
        logger.info("Event processed successfully", logContext);
    } catch (error: any) {
        if (error.name === 'ConditionalCheckFailedException') {
            logger.warn("Duplicate event detected and ignored", logContext);
            // This is a successful outcome from an idempotency perspective.
            // We return normally to allow the message to be deleted from the queue.
            return;
        } else {
            logger.error("Unhandled error during event processing", { ...logContext, error: error.message, stack: error.stack });
            // Re-throw the error to signal failure to the Lambda runtime.
            // SQS will then follow its redrive policy (retry or send to DLQ).
            throw error;
        }
    }
};

export const handler = async (event: { Records: { body: string }[] }): Promise<void> => {
    // In a real-world scenario, you might use Promise.allSettled for partial batch failure handling
    const processingPromises = event.Records.map(processRecord);
    await Promise.all(processingPromises);
};

Performance and Cost Optimization

While this pattern is highly effective, it's not free. Conditional writes have different performance and cost characteristics than standard writes.

Write Capacity Unit (WCU) Consumption:

* Standard PutItem/UpdateItem: Consumes WCUs based on the item size (1 WCU per 1 KB, rounded up).

* Conditional PutItem/UpdateItem:

* If the condition succeeds: Consumes the standard WCUs for the write.

* If the condition fails (ConditionalCheckFailedException): Consumes half the WCUs that would have been required for the write. For a 1KB item, a failed conditional write consumes 0.5 WCUs.

For provisioned capacity mode, this means you must account for the WCU consumption of duplicate messages. For on-demand mode, you will be billed for these failed checks.

Benchmark Comparison (Illustrative):

As the table shows, the latency overhead of a conditional check is minimal, often less than a millisecond. The cost of failed checks is lower than successful writes. Compared to an external system like Redis, which requires a separate network hop and infrastructure cost, the DynamoDB-native pattern is almost always more performant and cost-effective.

TTL for Idempotency Keys:

The processedEvents set can grow indefinitely, which can be a problem. DynamoDB items have a 400 KB size limit. A large set can also increase the cost of reads and writes for the item.

The solution is Time-to-Live (TTL).

• When adding a new event, add an entry to this map instead of a simple set. The TTL value should be set to a duration safely longer than your message processing window (e.g., SQS visibility timeout + a buffer).
• Enable TTL on the DynamoDB table, targeting an attribute that holds the overall expiry for the idempotency data.

This is more complex to manage within a single UpdateExpression. A more pragmatic approach is to set a TTL on the entire business item if it's ephemeral (like a session). If the item is long-lived, you may need a periodic cleanup process or accept the storage growth, which is often negligible in practice unless an item is updated thousands of times.

For most systems, a simple processedEvents set is sufficient. Only consider TTL-based cleanup if you have long-lived items that are updated with extremely high frequency.

Production Patterns and Anti-Patterns

Pattern: The Idempotency Layer

Don't sprinkle ConditionExpression logic throughout your codebase. Abstract it into a dedicated data access layer (DAL) or repository class. Your business logic should be unaware of the implementation details.

// Example of an abstracted method
class OrderRepository {
  public async addItem(orderId: string, item: any, idempotencyKey: string): Promise<void> {
    // All the UpdateCommand logic from Example 2 is encapsulated here
  }
}

// Business logic becomes clean:
await orderRepository.addItem(orderId, newItem, idempotencyKey);

Pattern: Combining with `TransactWriteItems`

What if an event must atomically update an Order and decrement Inventory for a product? TransactWriteItems allows you to group up to 100 write actions into a single all-or-nothing transaction.

You can include your idempotency check as one of the actions in the transaction. The best way is to use a dedicated idempotency record.

* Put to the Idempotency table with ConditionExpression: "attribute_not_exists(idempotencyKey)".

* Update to the Orders table.

* Update to the Inventory table.

If the idempotency key already exists, the first operation fails, and the entire transaction is rolled back. No partial state is written.

Anti-Pattern: Using a Separate Idempotency Table without Transactions

Never perform a separate check-then-act sequence. This is a classic race condition.

// ANTI-PATTERN: DO NOT DO THIS
const isProcessed = await checkIdempotencyTable(idempotencyKey);
if (isProcessed) {
  return; // Duplicate
}

// *** A crash or concurrent process can interfere here ***

await updateOrderTable(order);
await writeToIdempotencyTable(idempotencyKey);

If the process crashes after updating the order but before writing the idempotency key, the operation is no longer idempotent. The atomicity provided by ConditionExpression or TransactWriteItems is non-negotiable.

Anti-Pattern: Reusing Non-Unique Idempotency Keys

The idempotency key must be unique to the specific operation you want to de-duplicate. Using an orderId is incorrect. If you receive two different valid updates for the same order (e.g., AddItem and then ChangeAddress), using orderId as the key would cause the second valid operation to be incorrectly rejected as a duplicate. The key must be generated by the producer for each unique event instance, typically as a UUID.

Conclusion

By embedding idempotency logic directly into your DynamoDB write operations using ConditionExpression, you build systems that are inherently more resilient, simpler, and more performant. This pattern elegantly solves the at-least-once delivery problem in event-driven architectures without introducing external dependencies or complex application-level state management. It shifts the responsibility for atomic state checking to the database, which is designed to handle it with high concurrency and low latency. For senior engineers building scalable services on AWS, mastering conditional expressions is not just a useful trick—it is a fundamental pattern for robust system design.