Resilient Sagas: Choreographing Microservices with Kafka & Debezium

The Inherent Challenge: Atomicity in a Distributed World

In monolithic architectures, maintaining data consistency is a solved problem. We wrap multiple database operations in a single ACID transaction, and the database guarantees atomicity. If any step fails, the entire operation is rolled back, leaving the system in a consistent state. When we decompose a monolith into microservices, we trade this simplicity for scalability, resilience, and organizational autonomy. However, we lose the safety net of a single, overarching database transaction.

Consider a classic e-commerce scenario: placing an order. This single business action might involve three distinct services:

In a distributed system, these services have their own private databases. A two-phase commit (2PC) protocol is often too complex, brittle, and introduces tight coupling, making it an anti-pattern for most microservice architectures. The alternative is a Saga: a sequence of local transactions where each transaction updates the database of a single service and triggers the next step in the process. If a local transaction fails, the saga executes a series of compensating transactions to undo the preceding operations.

There are two primary ways to coordinate a saga:

* Orchestration: A central coordinator (the orchestrator) tells each participant what to do and when. It's a command-and-control model.

* Choreography: There is no central coordinator. Each service publishes events when it completes its local transaction. Other services subscribe to these events and react accordingly.

This article focuses exclusively on choreography. Its primary benefit is extreme decoupling. Services don't need to know about each other; they only need to know about the events they produce and consume. This fosters independent development and deployment. However, this decoupling comes at the cost of complexity in implementation and monitoring. We will tackle this complexity head-on.

Core Architecture: The Kafka & Debezium Synergy

The fundamental challenge in an event-driven choreography is guaranteeing that a service's state change and the publication of its corresponding event happen atomically. A common but flawed approach is the "dual-write":

// ANTI-PATTERN: DO NOT DO THIS
func (s *OrderService) CreateOrder(ctx context.Context, order *Order) error {
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback() // Rollback on error

    // 1. Write to the database
    if err := s.orderRepo.Create(ctx, tx, order); err != nil {
        return err
    }

    // 2. Publish to message broker
    // WHAT HAPPENS IF THIS FAILS? THE DB IS COMMITTED, BUT NO EVENT IS SENT.
    if err := s.eventPublisher.Publish(ctx, "OrderCreated", order); err != nil {
        // We can't reliably roll back the DB commit here.
        return err
    }

    return tx.Commit()
}

If the database commit succeeds but the event publication fails (e.g., the message broker is down), the system is left in an inconsistent state. The order exists, but no downstream service will ever know about it. The solution to this is the Transactional Outbox Pattern.

Instead of publishing an event directly, the service writes the event to a dedicated outbox table within its own database, as part of the same local transaction as the business state change. Because this is a single, atomic ACID transaction, we guarantee that the business data and the event message are saved together or not at all.

This is where our core technologies come into play:

This combination provides the atomicity we need. The event is considered "published" the moment the local database transaction commits. Debezium and Kafka handle the rest, ensuring at-least-once delivery to downstream consumers.

Here is a diagram of our target architecture for an order placement saga:

graph TD
    subgraph Order Service
        A[API Request] --> B{Create Order TXN}
        B --> C[Insert into 'orders' table]
        B --> D[Insert into 'outbox' table]
    end

    subgraph Database
        E[Order DB] -- WAL --> F[Debezium Connector]
    end

    C --> E
    D --> E

    F -- Publishes Event --> G[Kafka Topic: 'order.events']

    subgraph Payment Service
        H[Consumer] -- Subscribes --> G
        H --> I{Process Payment TXN}
        I --> J[Update 'payments' table]
        I --> K[Insert into 'outbox' table]
    end

    subgraph Inventory Service
        L[Consumer] -- Subscribes --> M[Kafka Topic: 'payment.events']
        L --> N{Reserve Inventory TXN}
        N --> O[Update 'inventory' table]
        N --> P[Insert into 'outbox' table]
    end

    K -- Debezium --> M

Production Implementation: A Multi-Service Order Placement Saga

Let's build this system. We'll use PostgreSQL as our database and Go for our service implementations, but the principles are language-agnostic.

Step 1: The Order Service - Initiating the Saga

First, we define the database schema for the Order Service. Note the outbox table.

-- Order Service Schema
CREATE TABLE orders (
    id UUID PRIMARY KEY,
    customer_id UUID NOT NULL,
    total_amount DECIMAL(10, 2) NOT NULL,
    status VARCHAR(50) NOT NULL, -- e.g., 'PENDING', 'PAID', 'CANCELLED'
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL, -- e.g., 'order'
    aggregate_id UUID NOT NULL,         -- e.g., the order ID
    event_type VARCHAR(255) NOT NULL,    -- e.g., 'OrderCreated'
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The createOrder function now performs a single transaction to write to both tables.

// internal/order/service.go
package order

import (
	"context"
	"database/sql"
	"encoding/json"

	"github.com/google/uuid"
)

// ... (Order struct, Repository interface definitions)

// OrderCreatedEvent defines the payload for our event
type OrderCreatedEvent struct {
	OrderID      uuid.UUID `json:"orderId"`
	CustomerID   uuid.UUID `json:"customerId"`
	TotalAmount  float64   `json:"totalAmount"`
}

func (s *Service) CreateOrder(ctx context.Context, customerID uuid.UUID, totalAmount float64) (*Order, error) {
	tx, err := s.db.BeginTx(ctx, nil)
	if err != nil {
		return nil, fmt.Errorf("failed to begin transaction: %w", err)
	}
	defer tx.Rollback() // Guarantees rollback on any error path

	// 1. Create the order entity
	order := &Order{
		ID:          uuid.New(),
		CustomerID:  customerID,
		TotalAmount: totalAmount,
		Status:      "PENDING",
	}

	if err := s.orderRepo.Create(ctx, tx, order); err != nil {
		return nil, fmt.Errorf("failed to create order: %w", err)
	}

	// 2. Create the outbox event within the same transaction
	eventPayload, err := json.Marshal(OrderCreatedEvent{
		OrderID:     order.ID,
		CustomerID:  order.CustomerID,
		TotalAmount: order.TotalAmount,
	})
	if err != nil {
		return nil, fmt.Errorf("failed to marshal event payload: %w", err)
	}

	outboxEvent := &OutboxEvent{
		ID:            uuid.New(),
		AggregateType: "order",
		AggregateID:   order.ID,
		EventType:     "OrderCreated",
		Payload:       eventPayload,
	}

	if err := s.outboxRepo.Create(ctx, tx, outboxEvent); err != nil {
		return nil, fmt.Errorf("failed to create outbox event: %w", err)
	}

	// 3. Commit the transaction
	if err := tx.Commit(); err != nil {
		return nil, fmt.Errorf("failed to commit transaction: %w", err)
	}

	return order, nil
}

This code is the foundation of our resilient system. The order and the OrderCreated event are now an atomic unit.

Step 2: Configuring the Debezium Connector

Now we configure Debezium to watch the outbox table. This is typically done via a POST request to the Kafka Connect REST API.

{
  "name": "order-service-outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "postgres-orders",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "password",
    "database.dbname": "order_db",
    "database.server.name": "orderserver",
    "table.include.list": "public.outbox",
    "tombstones.on.delete": "false",

    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregate_type",
    "transforms.outbox.route.topic.replacement": "${routedByValue}.events",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload"
  }
}

Let's break down the critical transforms configuration. This is where the magic happens:

* transforms.outbox.type: We're using Debezium's built-in EventRouter Single Message Transform (SMT).

* transforms.outbox.route.by.field: This tells the SMT to look at the aggregate_type column in our outbox table to decide the destination topic.

* transforms.outbox.route.topic.replacement: This is a powerful expression. It takes the value from the aggregate_type column (which is "order" in our case) and constructs the topic name ${routedByValue}.events, resulting in order.events.

* transforms.outbox.table.field.event.key: Sets the Kafka message key to the value of the aggregate_id column. This is essential for ordering. All events for the same order will go to the same Kafka partition.

* transforms.outbox.table.field.event.payload: Instructs the SMT to extract the actual business event from the payload column of the outbox table, discarding the rest of the CDC envelope.

This SMT transforms a raw, noisy CDC event into a clean, domain-specific event, ready for consumption by other services.

Step 3: The Payment Service - Consuming and Reacting with Idempotency

The Payment Service listens to the order.events topic. Its consumer must be idempotent. Kafka provides at-least-once delivery semantics, meaning a message could be delivered more than once (e.g., during a consumer rebalance). Our business logic must be able to handle duplicate messages without causing incorrect side effects.

A robust pattern for idempotency is to track processed event IDs.

-- Payment Service Schema
CREATE TABLE payments (
    id UUID PRIMARY KEY,
    order_id UUID NOT NULL UNIQUE,
    amount DECIMAL(10, 2) NOT NULL,
    status VARCHAR(50) NOT NULL, -- 'SUCCESS', 'FAILED'
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- For idempotency
CREATE TABLE processed_events (
    event_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Payment service also needs an outbox
CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id UUID NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Here is a conceptual Go implementation for an idempotent consumer handler:

// internal/payment/consumer.go

// We assume the incoming Kafka message has been deserialized
// and includes a unique ID for the event itself.
// This ID could come from the outbox table's primary key.
type KafkaMessage struct {
    EventID     uuid.UUID
    EventType   string
    Payload     json.RawMessage
}

func (c *Consumer) HandleOrderCreated(ctx context.Context, msg KafkaMessage) error {
    // Begin transaction for idempotency check and business logic
    tx, err := c.db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    // IDEMPOTENCY CHECK
    isProcessed, err := c.eventRepo.IsEventProcessed(ctx, tx, msg.EventID)
    if err != nil {
        return fmt.Errorf("failed to check for processed event: %w", err)
    }
    if isProcessed {
        log.Printf("Event %s already processed, skipping.", msg.EventID)
        return nil // Not an error, just a duplicate
    }

    // Business Logic
    var eventPayload OrderCreatedEvent
    if err := json.Unmarshal(msg.Payload, &eventPayload); err != nil {
        return fmt.Errorf("failed to unmarshal payload: %w", err) // This could be a poison pill
    }

    // Simulate calling a payment gateway
    paymentStatus := c.paymentGateway.Process(eventPayload.OrderID, eventPayload.TotalAmount)

    // Create payment record
    payment := &Payment{ ... }
    if err := c.paymentRepo.Create(ctx, tx, payment); err != nil {
        return err
    }

    // Publish the outcome to its OWN outbox
    var nextEventType string
    var nextEventPayload []byte
    if paymentStatus == "SUCCESS" {
        nextEventType = "PaymentSucceeded"
        // ... marshal success payload
    } else {
        nextEventType = "PaymentFailed"
        // ... marshal failure payload
    }
    
    outboxEvent := &OutboxEvent{
        AggregateType: "payment",
        AggregateID:   payment.ID,
        EventType:     nextEventType,
        Payload:       nextEventPayload,
    }
    if err := c.outboxRepo.Create(ctx, tx, outboxEvent); err != nil {
        return err
    }

    // Mark event as processed
    if err := c.eventRepo.MarkEventProcessed(ctx, tx, msg.EventID); err != nil {
        return fmt.Errorf("failed to mark event as processed: %w", err)
    }

    return tx.Commit()
}

By wrapping the idempotency check, business logic, outbox write, and processed event tracking in a single transaction, we ensure the entire operation is atomic.

Advanced Patterns & Edge Case Handling

Real-world systems are messy. Here's how to handle the inevitable failures.

Compensating Transactions: Rolling Back the Saga

What happens if the Payment Service publishes a PaymentFailed event? The saga needs to be rolled back. The Order Service, which initiated the saga, must listen for this outcome and perform a compensating transaction.

// internal/order/consumer.go

func (c *Consumer) HandlePaymentFailed(ctx context.Context, msg KafkaMessage) error {
    var eventPayload PaymentFailedEvent
    // ... unmarshal payload

    // The compensating transaction is to update the order status.
    // This operation should also be idempotent.
    err := c.orderRepo.UpdateStatus(ctx, eventPayload.OrderID, "CANCELLED")
    if err != nil {
        // Handle error, perhaps retry or alert
        return err
    }

    log.Printf("Order %s cancelled due to payment failure.", eventPayload.OrderID)
    return nil
}

Key Insight: A compensating transaction is not a true database rollback. It's a new business transaction that semantically reverses a previous action. It's possible for a compensating transaction to fail, which requires its own retry or escalation strategy. This highlights the complexity of eventual consistency.

Handling Consumer Poison Pills with a Dead Letter Queue (DLQ)

A "poison pill" is a message that a consumer cannot process, causing it to crash, restart, re-read the message, and crash again in a loop. A common cause is a malformed JSON payload that the unmarshaler cannot handle.

This can bring your entire service to a halt. The solution is a Dead Letter Queue (DLQ) pattern. After a configured number of failed processing attempts, the message is moved to a separate DLQ topic for later inspection, allowing the consumer to move on to the next message.

If you're using a framework like Spring Kafka (Java), this is declarative. With a manual implementation in Go using a library like segmentio/kafka-go, you need to build this logic yourself:

• Commit the original message's offset to move on.

// Simplified DLQ logic
func (c *Consumer) ConsumeLoop(ctx context.Context) {
    const maxRetries = 3

    for {
        msg, err := c.reader.FetchMessage(ctx)
        // ... handle fetch error

        var success bool
        for i := 0; i < maxRetries; i++ {
            err = c.handler(ctx, msg)
            if err == nil {
                success = true
                break
            }
            log.Printf("Attempt %d failed for message %s: %v", i+1, msg.Offset, err)
            time.Sleep(time.Second * time.Duration(i+1)) // Exponential backoff
        }

        if !success {
            log.Printf("Message failed after %d retries. Sending to DLQ.", maxRetries)
            // Add error info to message headers
            c.dlqWriter.WriteMessages(ctx, createDLQMessage(msg, err))
        }

        // Commit offset regardless of success or failure (after DLQ)
        c.reader.CommitMessages(ctx, msg)
    }
}

Out-of-Order Events and Concurrency

Kafka only guarantees message order within a single partition. We ensured this for events related to a single order by using the order_id as the Kafka message key. This is a critical design decision.

However, you can still face concurrency issues if your system allows for multiple, conflicting sagas to operate on the same aggregate simultaneously (e.g., an UpdateOrder saga and a CancelOrder saga). A common solution is to include a version number in your event payloads and database tables. The consumer can then use optimistic locking:

• The event carries the version of the aggregate when the event was created.
• The consumer, in its transaction, reads the aggregate from its database.

• If they match, it proceeds with the update and increments the version number in the database.
• If they don't match, it means another event was processed first. The consumer can then choose to abort, retry, or apply a different logic.

Performance and Scalability Considerations

* Debezium Connector Performance: The connector reads from the database's transaction log. In PostgreSQL, this can increase WAL size and put a minor load on the DB. Monitor this closely. You can tune Debezium's max.batch.size and poll.interval.ms to balance between latency and throughput.

* Kafka Topic Partitions: The number of partitions on a topic is the maximum level of parallelism for your consumer group. If you have a topic with 4 partitions, you can run at most 4 instances of your consumer service to process messages in parallel. Choose a partition count that allows for future scaling, as increasing it later can be complex. A good starting point is often 12 or 24.

* Consumer Lag Monitoring: This is the most critical metric for an event-driven system. It measures the difference in offsets between the last message written to a partition and the last message processed by a consumer group. A consistently growing lag indicates your consumers cannot keep up with the producers. Use tools like Prometheus with the Kafka Lag Exporter or Burrow to monitor this and set up alerts.

* Outbox Table Growth: The outbox table will grow indefinitely. Implement a separate, asynchronous cleanup job that periodically deletes rows that have been successfully processed by Debezium. Debezium commits offsets back to Kafka Connect, but it doesn't know when to delete the source row. Your cleanup job can safely delete rows older than a certain threshold (e.g., 24 hours), assuming Debezium is healthy.

Conclusion: The Trade-offs of Choreography

The choreographed saga pattern, powered by the transactional outbox, Debezium, and Kafka, is an exceptionally powerful tool for building resilient, scalable, and highly decoupled microservice architectures. It solves the critical problem of atomic state change and event publication, forming the bedrock of reliable eventual consistency.

However, this power comes with significant complexity. End-to-end business flow monitoring is more difficult, as there is no central place to see the state of a saga. This requires robust distributed tracing (e.g., using OpenTelemetry) and correlation IDs passed through all events. Debugging becomes a process of piecing together a story from multiple service logs and Kafka topics.

This pattern is not a silver bullet. For simpler request/reply interactions, synchronous communication is often sufficient. But for complex, multi-step business processes that must survive partial failures and service outages, the choreographed saga is an indispensable pattern in the modern senior engineer's toolkit.