Saga Pattern Mastery: Orchestrating Microservices with Temporal in Go

The Inescapable Problem: Atomic Transactions in a Distributed World

In a monolithic architecture, the database transaction is our safety net. A series of operations—debiting one account, crediting another, writing an audit log—can be wrapped in BEGIN TRANSACTION and COMMIT. If any step fails, ROLLBACK restores a consistent state. This ACID guarantee is a foundational pillar of reliable software.

Microservices shatter this pillar. When your OrderService, PaymentService, and InventoryService are separate applications, each with its own database, a single, all-encompassing ACID transaction is no longer feasible. Two-phase commit (2PC) protocols exist but are often dismissed in modern systems due to their complexity, performance overhead, and reliance on synchronous communication, which introduces tight coupling and reduces availability.

This leaves us with a critical challenge: how do we maintain data consistency across service boundaries when an operation involves multiple, independent services? A classic example is an e-commerce order:

What happens if Step 3 fails because an item just went out of stock? We've already charged the customer. We must reliably undo the payment. This is the essence of the Saga pattern.

A Saga is a sequence of local transactions where each transaction updates data within a single service. If a local transaction fails, the Saga executes a series of compensating transactions to reverse the effects of the preceding successful transactions.

There are two primary Saga implementation styles:

* Choreography: Each service publishes events that trigger subsequent services. This is decentralized but can become incredibly difficult to track, debug, and reason about as the number of services grows. The business logic becomes scattered across multiple event handlers.

* Orchestration: A central coordinator (the orchestrator) tells the participant services what to do. The business logic is centralized, making it easier to understand, manage, and modify the workflow. The state of the entire transaction is explicitly managed in one place.

While choreography has its uses, orchestration provides superior visibility and control for complex, mission-critical workflows. However, building a robust orchestrator from scratch is a monumental task. You need to manage state, handle retries with exponential backoff, deal with worker crashes, and ensure that compensation logic is executed reliably. This is precisely the problem Temporal was designed to solve.

Temporal is a durable execution system. It allows you to write orchestration logic as straightforward code, while the Temporal platform handles the persistence, retries, and state management, guaranteeing that your code will execute to completion, even in the face of process failures or server restarts. It's a perfect fit for implementing the Saga orchestration pattern.

This post will guide you through building a production-grade Saga orchestrator using Temporal and Go, focusing on the advanced patterns and edge cases you'll encounter in a real-world system.

System Architecture: The E-Commerce Order Saga

Let's define our scenario. We'll build an order processing workflow with the following steps and corresponding compensations:

Our orchestrator, a Temporal Workflow, will execute these steps sequentially. If any step fails, it will execute the necessary compensations in reverse order.

Project Setup

First, ensure you have a local Temporal server running. The simplest way is with docker-compose:

# docker-compose.yml
version: '3.5'
services:
  temporal:
    image: temporalio/auto-setup:1.21.3
    ports:
      - '7233:7233'
      - '8233:8233'
    environment:
      - DB=sqlite3
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamic_config.yaml

Run docker-compose up.

Our Go project structure will look like this:

/saga-temporal-go
|-- /activities
|   |-- activities.go
|-- /shared
|   |-- types.go
|-- /workflow
|   |-- workflow.go
|-- go.mod
|-- go.sum
|-- main.go  # Worker and Starter

Section 1: Implementing the Forward Path with Durable Execution

The foundation of our Saga is the "happy path" or forward execution flow. In Temporal, this is our main workflow function. The orchestrator's logic is just Go code, making it incredibly expressive.

Here's the core OrderWorkflow definition. Notice how it reads like a standard function, yet each ExecuteActivity call is a durable, retriable step managed by the Temporal cluster.

// workflow/workflow.go
package workflow

import (
	"fmt"
	"time"

	"go.temporal.io/sdk/workflow"

	"saga-temporal-go/activities"
	"saga-temporal-go/shared"
)

func OrderWorkflow(ctx workflow.Context, orderDetails shared.OrderDetails) (string, error) {
	wfLogger := workflow.GetLogger(ctx)
	wfLogger.Info("Order workflow started", "OrderID", orderDetails.OrderID)

	ao := workflow.ActivityOptions{
		StartToCloseTimeout: time.Minute * 1,
		RetryPolicy: &workflow.RetryPolicy{
			InitialInterval:    time.Second * 1,
			BackoffCoefficient: 2.0,
			MaximumInterval:    time.Minute * 1,
			MaximumAttempts:    3,
		},
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var a *activities.Activities

	// 1. Create Order record
	err := workflow.ExecuteActivity(ctx, a.CreateOrder, orderDetails).Get(ctx, nil)
	if err != nil {
		wfLogger.Error("Failed to create order record", "Error", err)
		return "", fmt.Errorf("CreateOrder failed: %w", err)
	}

	// 2. Process Payment
	var paymentID string
	err = workflow.ExecuteActivity(ctx, a.ProcessPayment, orderDetails).Get(ctx, &paymentID)
	if err != nil {
		wfLogger.Error("Failed to process payment", "Error", err)
		return "", fmt.Errorf("ProcessPayment failed: %w", err)
	}

	// 3. Update Inventory
	err = workflow.ExecuteActivity(ctx, a.UpdateInventory, orderDetails.Items).Get(ctx, nil)
	if err != nil {
		wfLogger.Error("Failed to update inventory", "Error", err)
		return "", fmt.Errorf("UpdateInventory failed: %w", err)
	}

	// 4. Ship Order
	var shippingID string
	err = workflow.ExecuteActivity(ctx, a.ShipOrder, orderDetails).Get(ctx, &shippingID)
	if err != nil {
		wfLogger.Error("Failed to ship order", "Error", err)
		return "", fmt.Errorf("ShipOrder failed: %w", err)
	}

	// 5. Mark Order as Confirmed
	err = workflow.ExecuteActivity(ctx, a.MarkOrderAsConfirmed, orderDetails.OrderID).Get(ctx, nil)
	if err != nil {
		wfLogger.Error("Failed to mark order as confirmed", "Error", err)
		return "", fmt.Errorf("MarkOrderAsConfirmed failed: %w", err)
	}

	wfLogger.Info("Workflow completed successfully!")
	return "SUCCESS", nil
}

This code is simple, but its power lies in what's happening behind the scenes. If the worker process running UpdateInventory crashes after the payment is processed, Temporal knows exactly where the workflow left off. When a worker comes back online, it will resume execution from UpdateInventory, preserving the paymentID and other local variables. This durability is the key to building reliable Sagas.

Our activities are simple functions that would, in a real system, make API calls to other services.

// activities/activities.go
package activities

import (
	"context"
	"errors"
	"fmt"
	"log"
	"math/rand"
	"time"

	"saga-temporal-go/shared"
	"github.com/google/uuid"
)

// Activities struct can hold dependencies like DB connections or HTTP clients.
type Activities struct{}

func (a *Activities) CreateOrder(ctx context.Context, order shared.OrderDetails) error {
	log.Printf("Creating order record for OrderID: %s", order.OrderID)
	// Simulate DB write
	time.Sleep(100 * time.Millisecond)
	return nil
}

// Forcing a failure for demonstration purposes
var inventoryShouldFail = true

func (a *Activities) UpdateInventory(ctx context.Context, items []shared.Item) error {
	log.Printf("Updating inventory for %d items", len(items))
	// Simulate a failure scenario
	if inventoryShouldFail {
		log.Println("Simulating inventory failure!")
		return errors.New("inventory service unavailable: out of stock")
	}
	// Simulate DB write
	time.Sleep(100 * time.Millisecond)
	return nil
}

// ... other activity stubs for Payment, Shipping, etc.
func (a *Activities) ProcessPayment(ctx context.Context, order shared.OrderDetails) (string, error) {
	paymentID := uuid.New().String()
	log.Printf("Processing payment for OrderID: %s. PaymentID: %s", order.OrderID, paymentID)
	time.Sleep(100 * time.Millisecond)
	return paymentID, nil
}

// ... stubs for MarkOrderAsConfirmed, ShipOrder

This code defines the happy path. But the real value of the Saga pattern—and Temporal's implementation of it—lies in how it handles the unhappy path.

Section 2: Implementing Durable Compensation Logic

When UpdateInventory fails, our current workflow simply fails and stops. We need to actively undo the previous steps. This is where compensation logic comes in.

The most robust way to implement this in a Temporal workflow is by using Go's defer statement. A function deferred inside a Temporal workflow is durable. It's guaranteed to be executed when the workflow function exits, either by returning successfully or by failing. This is a perfect mechanism for our cleanup/compensation logic.

We will refactor our workflow to register a compensation function immediately after an activity successfully completes.

// workflow/workflow.go
package workflow

import (
	// ... imports
)

func OrderWorkflow(ctx workflow.Context, orderDetails shared.OrderDetails) (string, error) {
	// ... (logger, activity options setup)

	var a *activities.Activities
	var compensations []func(workflow.Context)

	// Use a deferred function to run all compensations on workflow exit (failure or success).
	// The 'err' variable will be captured from the parent scope at the time of exit.
	var err error
	defer func() {
		if err != nil {
			wfLogger.Info("Workflow failed. Starting compensation logic.")
			// Execute compensations in reverse order
			for i := len(compensations) - 1; i >= 0; i-- {
				compensations[i](ctx)
			}
		}
	}()

	// 1. Create Order
	err = workflow.ExecuteActivity(ctx, a.CreateOrder, orderDetails).Get(ctx, nil)
	if err != nil {
		return "", err
	}
	compensations = append(compensations, func(ctx workflow.Context) {
		_ = workflow.ExecuteActivity(ctx, a.MarkOrderAsFailed, orderDetails.OrderID).Get(ctx, nil)
	})

	// 2. Process Payment
	var paymentID string
	err = workflow.ExecuteActivity(ctx, a.ProcessPayment, orderDetails).Get(ctx, &paymentID)
	if err != nil {
		return "", err
	}
	compensations = append(compensations, func(ctx workflow.Context) {
		_ = workflow.ExecuteActivity(ctx, a.RefundPayment, paymentID).Get(ctx, nil)
	})

	// 3. Update Inventory
	err = workflow.ExecuteActivity(ctx, a.UpdateInventory, orderDetails.Items).Get(ctx, nil)
	if err != nil {
		return "", err
	}
	compensations = append(compensations, func(ctx workflow.Context) {
		_ = workflow.ExecuteActivity(ctx, a.RestoreInventory, orderDetails.Items).Get(ctx, nil)
	})

	// ... (Shipping and Confirmation steps would follow the same pattern)

	wfLogger.Info("Workflow completed successfully!")
	// Clear compensations if workflow is successful
	compensations = nil 
	return "SUCCESS", nil
}

Let's break down this powerful pattern:

Now, when we run this workflow and UpdateInventory fails, the Temporal logs will show the following sequence:

• Workflow function exits with an error.
• Deferred compensation logic triggers.

This orchestration is now resilient to worker failures at any stage. The state of the compensations stack is durably persisted by Temporal along with the rest of the workflow state.

Section 3: Advanced Edge Cases and Production Patterns

The defer pattern is robust, but real-world systems present further challenges. Let's address some critical edge cases.

3.1 Idempotency in Compensation Activities

What happens if our RefundPayment activity executes, but before it can report completion back to the Temporal server, the worker crashes? Temporal's at-least-once execution guarantee means it will reschedule the activity on another worker. If our RefundPayment activity is not idempotent, we might refund the customer twice.

This is a classic distributed systems problem. The solution is to make our compensation activities idempotent. The orchestrator must provide a unique key for each invocation.

We can enhance our workflow to generate a unique idempotency key for each compensation call.

// workflow/workflow.go (inside OrderWorkflow)

// ... inside the ProcessPayment block
compensations = append(compensations, func(ctx workflow.Context) {
    // Generate a unique token for this specific compensation attempt.
    // Using a random UUID from the workflow is deterministic.
    sideEffectID := "refund-" + orderDetails.OrderID
    var refundIdempotencyKey string
    
    // workflow.SideEffect is used for non-deterministic operations within a workflow
    // that need to be recorded in the history for replay safety.
    // A better approach for idempotency keys is to use a deterministic one.
    // Let's use workflow.NewUUID() for a deterministic, replay-safe UUID.
    refundIdempotencyKey = workflow.NewUUID().String()

    refundArgs := shared.RefundArgs{
        PaymentID:      paymentID,
        IdempotencyKey: refundIdempotencyKey,
    }
    
    // We can use a separate, more aggressive retry policy for compensations.
    compensationAo := workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute * 5, // Give more time for compensations
        RetryPolicy: &workflow.RetryPolicy{MaximumAttempts: 10},
    }
    compensationCtx := workflow.WithActivityOptions(ctx, compensationAo)

    _ = workflow.ExecuteActivity(compensationCtx, a.RefundPayment, refundArgs).Get(ctx, nil)
})

The RefundPayment activity must then be implemented to honor this key. This typically involves a check-and-set operation in a database or a key-value store like Redis.

// activities/activities.go

// In a real app, this would be a database connection or Redis client.
var processedIdempotencyKeys = make(map[string]bool)

func (a *Activities) RefundPayment(ctx context.Context, args shared.RefundArgs) error {
	log.Printf("Attempting to refund PaymentID: %s with IdempotencyKey: %s", args.PaymentID, args.IdempotencyKey)

	// PSEUDOCODE: This logic needs to be atomic in a real system.
	// For example, using a database transaction or a Redis SETNX command.
	// lock.Lock()
	if processed, exists := processedIdempotencyKeys[args.IdempotencyKey]; exists && processed {
		log.Printf("Refund for key %s already processed. Skipping.", args.IdempotencyKey)
		// lock.Unlock()
		return nil // Return success to not block the Saga
	}
	processedIdempotencyKeys[args.IdempotencyKey] = true
	// lock.Unlock()

	log.Printf("Processing refund for PaymentID: %s...", args.PaymentID)
	// Actual call to payment gateway API would go here.
	time.Sleep(200 * time.Millisecond)

	return nil
}

This pattern ensures that even if the RefundPayment activity is retried multiple times, the actual refund operation is performed exactly once.

3.2 Non-Compensatable Failures

What if a compensation itself fails? For example, RefundPayment fails because the payment gateway is down and all retries are exhausted. This is a non-compensatable failure, and the Saga is now in a critical, inconsistent state. The system cannot automatically resolve this.

This is where human intervention is required. We must design our workflow to handle this scenario gracefully by alerting an operator.

Temporal's Signal feature is an excellent tool for this. A Signal is an external, asynchronous message that can be sent to a running workflow. We can modify our compensation logic to send a signal to an "admin dashboard" workflow or call an alerting service if a compensation fails permanently.

// workflow/workflow.go (inside the compensation defer block)

for i := len(compensations) - 1; i >= 0; i-- {
    future := workflow.ExecuteActivity(ctx, compensations[i]) // Note: simplified call for clarity
    err := future.Get(ctx, nil)
    if err != nil {
        // The compensation has failed after all retries.
        wfLogger.Error("CRITICAL: Compensation failed. Manual intervention required.", "Error", err)
        
        // Alerting logic
        alertDetails := shared.AlertDetails{
            OrderID: orderDetails.OrderID,
            FailedActivity: "[Compensation for X]", // Need to track this
            ErrorMessage: err.Error(),
        }
        
        // This could be an activity that sends an email, a PagerDuty alert, etc.
        _ = workflow.ExecuteActivity(ctx, a.SendAlert, alertDetails).Get(ctx, nil)
        
        // The workflow is now stuck. It will remain in a 'Running' state
        // until manually resolved. This is often the desired behavior.
        // We can wait on a signal that indicates manual resolution.
        workflow.NewSelector(ctx).AddReceive(workflow.GetSignalChannel(ctx, "manual_resolution_signal"), func(c workflow.Channel, more bool) {}).Select(ctx)
    }
}

This makes the failure explicit and auditable. An SRE or support engineer can investigate the issue, resolve it manually (e.g., by issuing a refund through the payment gateway's dashboard), and then send a signal to the workflow to either retry the compensation or mark it as complete, allowing the Saga to finish its rollback.

3.3 Asynchronous Activity Completion

Some activities don't complete immediately. For example, ShipOrder might initiate a process in a Warehouse Management System (WMS) and then need to wait for a webhook callback to confirm the shipment has actually been dispatched.

Temporal supports this pattern out of the box. An activity can report that it will not complete synchronously and then be completed later by an external process.

// activities/activities.go
func (a *Activities) ShipOrder(ctx context.Context, order shared.OrderDetails) (string, error) {
	// Get the unique task token for this activity execution
	taskToken := activity.GetInfo(ctx).TaskToken

	shippingID := "ship-" + uuid.New().String()

	// In a real app, you'd store this mapping in a DB:
	// db.StoreTokenForShippingID(shippingID, taskToken)
	log.Printf("Storing task token for shipping ID %s", shippingID)

	// Make the async API call to the WMS
	log.Printf("Calling WMS API for OrderID %s. ShippingID: %s", order.OrderID, shippingID)
	// ... http.Post(...)

	// Tell Temporal that this activity will be completed externally
	return "", activity.ErrResultPending
}

// In your API server (e.g., an HTTP handler for the webhook)
func (server *APIServer) WMSWebhookHandler(w http.ResponseWriter, r *http.Request) {
	// ... parse webhook payload to get shippingID and status ...

	// Retrieve the task token from your DB
	taskToken, err := server.db.GetTokenForShippingID(shippingID)
	if err != nil { /* handle error */ }

	if status == "SHIPPED" {
		server.temporalClient.CompleteActivity(context.Background(), taskToken, "SHIPPED_SUCCESSFULLY", nil)
	} else {
		server.temporalClient.CompleteActivity(context.Background(), taskToken, nil, errors.New("WMS failed to ship"))
	}

	// ... respond to webhook ...
}

This pattern allows Sagas to orchestrate processes that span long periods and involve asynchronous callbacks, without resorting to complex polling mechanisms.

Section 4: Performance and Scalability

As you move to production, the performance of your Temporal workers and the design of your workflows become critical.

4.1 Worker Tuning

A Temporal Worker is a process that polls a specific Task Queue for work. You can and should run many worker processes in parallel for scalability.

* Task Queues: Use different task queues for different types of workflows or activities. For example, have a high-priority-orders task queue for critical workflows and a batch-processing task queue for less time-sensitive work. This allows you to scale the worker pools independently.

* Worker Options: Tune worker options like MaxConcurrentActivityExecutionSize and MaxConcurrentWorkflowTaskExecutionSize based on the resource consumption of your activities and the expected load. IO-bound activities can often have higher concurrency than CPU-bound ones.

* Sticky Workflows: By default, Temporal tries to route tasks for a specific workflow execution to the same worker that last processed it. This is called a sticky execution and it improves performance by keeping the workflow's state cached in the worker's memory. Ensure your deployment strategy doesn't frequently kill and restart workers, as this negates the benefit of the cache.

4.2 Managing Workflow History Size

Every event in a workflow's life (start, activity scheduled, activity completed, timer fired, etc.) is recorded in its event history. Temporal has a hard limit on history size (typically 50,000 events or 50 MB). A Saga that involves many steps or large data payloads can approach this limit.

For very long-running or infinitely-running Sagas (e.g., a subscription management workflow), you must use the Continue-As-New pattern. This allows a workflow to complete its execution and immediately start a new execution with a fresh history, carrying over any necessary state.

// Example of a workflow that processes items in batches to avoid large history
func BatchProcessingWorkflow(ctx workflow.Context, items []shared.Item) error {
    const batchSize = 100
    if len(items) <= batchSize {
        // Process the final batch
        // ... execute activities for these items ...
        return nil
    }

    // Process the current batch
    currentBatch := items[:batchSize]
    // ... execute activities for currentBatch ...

    // Continue as a new workflow with the remaining items
    remainingItems := items[batchSize:]
    return workflow.NewContinueAsNewError(ctx, BatchProcessingWorkflow, remainingItems)
}

This is a critical pattern for ensuring the long-term health and performance of your Temporal application.

Conclusion: From Fragile Choreography to Resilient Orchestration

The Saga pattern is a powerful solution for maintaining data consistency in a microservices environment. However, implementing it manually with message queues and databases is fraught with peril. It forces developers to build a complex, distributed state machine from scratch, handling retries, state persistence, and failure recovery—all of which is undifferentiated heavy lifting.

By leveraging Temporal, we shift our focus from the plumbing of distributed systems to the business logic of our workflow. The orchestration logic becomes simple, testable Go code. The Saga's state, its progress, and its compensation stack are all managed durably by the Temporal platform.

We have explored how to:

* Implement a forward-path Saga using sequential, retriable activities.

* Build a robust, durable compensation mechanism using defer.

* Handle critical production edge cases like idempotency and non-compensatable failures.

* Orchestrate asynchronous processes using external activity completion.

* Consider performance and scale through worker tuning and history management.

This approach provides a system that is not only resilient but also highly observable. The entire history of every Saga is available for inspection in the Temporal UI, making debugging complex distributed failures an order of magnitude simpler than trying to piece together a story from scattered logs across a dozen services. By adopting a durable execution model for Saga orchestration, you can build complex, fault-tolerant distributed systems with confidence.