Advanced Kubernetes Operator Reconciliation Loops with Finalizers

October 7, 2025

15 min read

Goh Ling Yong

Technology enthusiast and software architect specializing in AI-driven development tools and modern software engineering practices. Passionate about the intersection of artificial intelligence and human creativity in building tomorrow's digital solutions.

The Inevitable Problem: Orphaned External Resources

As a senior engineer building on Kubernetes, you've likely moved past simple stateless applications and are now responsible for systems that manage stateful, external resources. Whether it's provisioning an S3 bucket, a CloudSQL instance, a Stripe subscription, or a DNS record, the Kubernetes Operator pattern provides a powerful abstraction. However, a naive implementation of the reconciliation loop hides a critical flaw: the deletion lifecycle.

When a user executes kubectl delete my-crd, the Kubernetes API server marks the object for deletion. A simple controller might receive a reconciliation request for a non-existent object and simply do nothing, assuming its work is done. The CR is gone from etcd, but the S3 bucket it provisioned remains, silently accruing costs and becoming a difficult-to-track piece of infrastructure debt. This is the path to orphaned resources.

The Kubernetes-native solution to this is the Finalizer. A finalizer is not a piece of code, but a declarative lock. It's a string entry in an object's metadata.finalizers array that tells the Kubernetes garbage collector, "Do not delete this object from etcd until I, the controller, have explicitly removed this entry." This mechanism transforms the deletion process from a fire-and-forget operation into a graceful, stateful shutdown sequence, giving your operator the chance to perform necessary cleanup.

This article is a deep dive into the practical, production-grade implementation of finalizers within a Go-based operator using controller-runtime. We will dissect the reconciliation loop as a state machine, implement idempotent cleanup logic, handle complex edge cases, and provide a complete, working example for managing an external resource.

Dissecting the Reconcile Loop as a State Machine

A robust reconciliation loop is not a linear script; it's a state machine that reacts to the observed state of a Custom Resource (CR). The presence or absence of a deletionTimestamp on the object's metadata is the primary fork in this state machine.

Let's define our Custom Resource for this example: a ManagedDatabase that provisions a hypothetical database instance via an external API.

// api/v1/manageddatabase_types.go
package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ManagedDatabaseSpec defines the desired state of ManagedDatabase
type ManagedDatabaseSpec struct {
	// DBName is the name of the database to be provisioned.
	DBName string `json:"dbName"`
	// OwnerEmail is the email of the user responsible for this database.
	OwnerEmail string `json:"ownerEmail"`
}

// ManagedDatabaseStatus defines the observed state of ManagedDatabase
type ManagedDatabaseStatus struct {
	// Conditions represent the latest available observations of the ManagedDatabase's state.
	Conditions []metav1.Condition `json:"conditions,omitempty"`
	// DatabaseID is the unique identifier of the provisioned database in the external system.
	DatabaseID string `json:"databaseId,omitempty"`
	// Endpoint is the connection endpoint for the database.
	Endpoint string `json:"endpoint,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

type ManagedDatabase struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   ManagedDatabaseSpec   `json:"spec,omitempty"`
	Status ManagedDatabaseStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

type ManagedDatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []ManagedDatabase `json:"items"`
}

func init() {
	SchemeBuilder.Register(&ManagedDatabase{}, &ManagedDatabaseList{})
}

The core logic resides in the Reconcile function of our controller. Here is the high-level structure we will build:

// controllers/manageddatabase_controller.go

// ... imports

const myFinalizerName = "database.example.com/finalizer"

func (r *ManagedDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	// 1. Fetch the ManagedDatabase instance
	instance := &databasev1.ManagedDatabase{}
	err := r.Get(ctx, req.NamespacedName, instance)
	// ... handle not found error

	// 2. Examine the deletion timestamp to determine if the object is under deletion.
	if instance.ObjectMeta.DeletionTimestamp.IsZero() {
		// The object is NOT being deleted. This is the CREATE/UPDATE path.

		// 3. Ensure our finalizer is present.
		if !controllerutil.ContainsFinalizer(instance, myFinalizerName) {
			log.Info("Adding Finalizer")
			controllerutil.AddFinalizer(instance, myFinalizerName)
			if err := r.Update(ctx, instance); err != nil {
				return ctrl.Result{}, err
			}
		}

		// 4. Execute the main reconciliation logic (create/update external resource)
		// ...

	} else {
		// The object IS being deleted. This is the DELETE path.

		// 5. Check if our finalizer is still present.
		if controllerutil.ContainsFinalizer(instance, myFinalizerName) {
			// 6. Execute the cleanup logic.
			if err := r.cleanupExternalResources(ctx, instance); err != nil {
				// If cleanup fails, return error to retry.
				// The finalizer is NOT removed, so deletion is blocked.
				return ctrl.Result{}, err
			}

			// 7. Cleanup was successful, remove the finalizer.
			log.Info("Removing Finalizer")
			controllerutil.RemoveFinalizer(instance, myFinalizerName)
			if err := r.Update(ctx, instance); err != nil {
				return ctrl.Result{}, err
			}
		}

		// Stop reconciliation as the item is being deleted
		return ctrl.Result{}, nil
	}

	return ctrl.Result{}, nil
}

This structure explicitly separates the create/update path from the deletion path, forming the two primary branches of our state machine.

Production-Grade Finalizer Implementation

Let's flesh out the skeleton above with production-ready code. We'll use a mock DatabaseAPIClient to simulate interactions with an external service.

The Create/Update Path: Adding the Finalizer

The most critical step in the create/update path is ensuring the finalizer is attached before any external resources are created. This prevents a race condition where a user could create and delete a CR in rapid succession, before the operator has a chance to add the finalizer, leading to an orphaned resource.

// ... inside Reconcile, in the DeletionTimestamp.IsZero() block

if !controllerutil.ContainsFinalizer(instance, myFinalizerName) {
    log.Info("Adding finalizer for ManagedDatabase")
    // Using Patch is generally safer than Update to avoid conflicts.
    patch := client.MergeFrom(instance.DeepCopy())
    controllerutil.AddFinalizer(instance, myFinalizerName)
    if err := r.Patch(ctx, instance, patch); err != nil {
        log.Error(err, "Failed to add finalizer")
        return ctrl.Result{}, err
    }
    // Requeue immediately after adding the finalizer to ensure the next reconcile
    // has the updated object. This is a good practice.
    return ctrl.Result{Requeue: true}, nil
}

// --- Main reconciliation logic begins here ---
// Check if the external database exists. We use the CR's UID for a unique ID.
externalDB, err := r.DBClient.GetDatabase(ctx, string(instance.GetUID()))
if err != nil {
    if errors.Is(err, db_api.ErrNotFound) {
        // Database does not exist, let's create it.
        log.Info("Creating external database")
        newDBID, endpoint, err := r.DBClient.CreateDatabase(ctx, instance.Spec.DBName, string(instance.GetUID()))
        if err != nil {
            log.Error(err, "Failed to create external database")
            // Update status with a failure condition
            meta.SetStatusCondition(&instance.Status.Conditions, metav1.Condition{
                Type:    "Ready",
                Status:  metav1.ConditionFalse,
                Reason:  "ProvisioningFailed",
                Message: err.Error(),
            })
            if updateErr := r.Status().Update(ctx, instance); updateErr != nil {
                return ctrl.Result{}, updateErr
            }
            return ctrl.Result{}, err
        }

        // Database created successfully. Update status.
        instance.Status.DatabaseID = newDBID
        instance.Status.Endpoint = endpoint
        meta.SetStatusCondition(&instance.Status.Conditions, metav1.Condition{
            Type:   "Ready",
            Status: metav1.ConditionTrue,
            Reason: "Provisioned",
            Message: "External database provisioned successfully.",
        })
        if err := r.Status().Update(ctx, instance); err != nil {
            log.Error(err, "Failed to update ManagedDatabase status")
            return ctrl.Result{}, err
        }

        log.Info("Successfully created and reconciled ManagedDatabase")
        return ctrl.Result{}, nil

    } else {
        // Some other API error occurred.
        log.Error(err, "Failed to get external database state")
        return ctrl.Result{}, err
    }
}

// Database exists, ensure it's in the desired state (omitted for brevity, but would involve comparing spec to external state)
log.Info("External database already exists, reconciliation complete.")

Key Points:

Use Patch over Update: When adding the finalizer, using r.Patch is preferable to r.Update. Update requires a full object replacement and can fail if another controller (or the user) modifies the object between your Get and Update calls. Patch is more targeted and less prone to conflicts.

Requeue After Adding: Requeuing immediately after patching the finalizer ensures the next reconciliation pass operates on the object state as it exists in etcd, preventing stale reads.

Unique External ID: We use the CR's metadata.uid as the unique identifier for the external resource. This is critical. A user might delete and recreate a CR with the same name/namespace. The UID is immutable and guarantees we're always operating on the correct instance.

The Deletion Path: Idempotent Cleanup

This is where the finalizer's power is realized. The logic here must be idempotent. An operation is idempotent if running it multiple times has the same effect as running it once. Why is this critical? Because your operator could crash or be restarted at any point during the cleanup process. The reconciliation will be re-triggered, and your cleanup code will run again.

Consider this sequence:

Controller starts deleting the external database.
The external API call succeeds; the database is deleted.
The operator process crashes before it can remove the finalizer from the CR.

The operator restarts, sees the CR with the deletion timestamp and finalizer, and tries to delete the external database again.

Your code must handle this gracefully.

// This function is called from the 'else' block of the DeletionTimestamp check.
func (r *ManagedDatabaseReconciler) cleanupExternalResources(ctx context.Context, db *databasev1.ManagedDatabase) error {
	log := log.FromContext(ctx)

	// We use the UID as the unique identifier.
	// The DatabaseID in the status might not be populated if creation failed midway.
	// The UID is always available.
	externalID := string(db.GetUID())

	log.Info("Starting cleanup for external database", "externalID", externalID)

	err := r.DBClient.DeleteDatabase(ctx, externalID)
	if err != nil {
		// CRITICAL: Check if the error is a 'NotFound' error.
		// If it is, the resource is already gone, which is our desired state.
		// We can consider this a success and proceed with finalizer removal.
		if errors.Is(err, db_api.ErrNotFound) {
			log.Info("External database already deleted. Cleanup is successful.")
			return nil
		}

		// Any other error means we should retry.
		log.Error(err, "Failed to delete external database during cleanup")
		return err
	}

	log.Info("Successfully deleted external database")
	return nil
}

// Inside the Reconcile function's deletion block:
if controllerutil.ContainsFinalizer(instance, myFinalizerName) {
    if err := r.cleanupExternalResources(ctx, instance); err != nil {
        log.Error(err, "Cleanup failed, will retry")
        // Update status to reflect cleanup failure
        meta.SetStatusCondition(&instance.Status.Conditions, metav1.Condition{
            Type:    "Ready",
            Status:  metav1.ConditionFalse,
            Reason:  "TerminationFailed",
            Message: "Failed to clean up external resources: " + err.Error(),
        })
        _ = r.Status().Update(ctx, instance) // Use best-effort status update

        return ctrl.Result{}, err
    }

    log.Info("Cleanup successful, removing finalizer")
    patch := client.MergeFrom(instance.DeepCopy())
    controllerutil.RemoveFinalizer(instance, myFinalizerName)
    if err := r.Patch(ctx, instance, patch); err != nil {
        log.Error(err, "Failed to remove finalizer")
        return ctrl.Result{}, err
    }
    log.Info("Finalizer removed")
}

The most important piece of code here is if errors.Is(err, db_api.ErrNotFound). By treating a "not found" error during deletion as a success, we achieve idempotency. The system self-heals from the crash scenario described earlier.

Advanced Edge Cases and Performance Considerations

Stuck in `Terminating`: The Finalizer's Curse

A common production issue is a CR getting stuck in the Terminating state. This happens when the controller is unable to remove its finalizer, usually because the cleanup logic is failing repeatedly.

Causes:

Bug in Cleanup Logic: The code has a bug that prevents it from ever succeeding.

External System Unavailability: The external database API is down, and the controller cannot reach it to perform the deletion.

Permissions Issues: The operator's service account has lost the necessary IAM permissions to delete the external resource.

Orphaned Finalizer: The operator was uninstalled without a proper cleanup procedure, leaving behind CRs with finalizers that no controller is watching.

Debugging and Resolution:

Check Operator Logs: This is the first step. The logs for the controller responsible for the CR will tell you why the cleanup is failing.

Inspect CR Status: A well-written operator will update the CR's status conditions even during termination attempts. kubectl describe manageddatabase might reveal a TerminationFailed condition with a descriptive message.

Manual Intervention (The Escape Hatch): If the external resource has been confirmed to be deleted out-of-band, or if you accept it will be orphaned, you can manually remove the finalizer to allow Kubernetes to garbage collect the CR. This should be a last resort.

bash

    # Get the current object YAML
    kubectl get manageddatabase my-db -o yaml > my-db.yaml

    # Edit my-db.yaml and remove the finalizer from the metadata.finalizers list.

    # Replace the object
    kubectl replace --raw "/apis/database.example.com/v1/namespaces/default/manageddatabases/my-db" -f ./my-db.yaml

    # A more direct, but riskier, approach is using patch:
    kubectl patch manageddatabase my-db --type='json' -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

Performance: The Cost of an Extra Write

Adding a finalizer requires an additional API server write (Patch or Update) at the beginning of the resource's lifecycle. In operators that manage thousands of CRs with high churn, this can contribute to API server load.

Is it worth it? Absolutely. The cost of a single extra write is negligible compared to the operational cost and financial impact of untracked, orphaned cloud resources. Correctness trumps micro-optimization here.

However, you can be efficient:

* Use Patch instead of Update to reduce the payload size and chance of conflicts.

* Ensure your reconciliation loop doesn't perform unnecessary updates. Only call r.Update or r.Status().Update when the spec or status has actually changed.

Handling Multiple Controllers and Finalizers

It's possible for a single resource to be managed by multiple controllers. For example, one operator might manage the database provisioning, while another manages backup policies for that same database object. Each controller should add its own uniquely named finalizer.

* database.example.com/provisioner-finalizer

* backup.example.com/backup-finalizer

When kubectl delete is called, both controllers will see the deletionTimestamp. Each one will perform its own cleanup logic and then remove only its own finalizer. The Kubernetes object will only be deleted after the metadata.finalizers list is completely empty. This allows for clean, decoupled, and cooperative lifecycle management.

Complete, Runnable Example Structure

To put it all together, here is a simplified but complete controller implementation.

// controllers/manageddatabase_controller.go
package controllers

import (
	"context"
	"errors"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/api/meta"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	databasev1 "my-operator/api/v1"
	// Assume this package exists and provides a client for our DB service
	// and defines a specific error type for not found resources.
	"my-operator/internal/db_api"
)

const myFinalizerName = "database.example.com/finalizer"

type ManagedDatabaseReconciler struct {
	client.Client
	Scheme   *runtime.Scheme
	DBClient db_api.Client // Our mock or real client
}

//+kubebuilder:rbac:groups=database.example.com,resources=manageddatabases,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=database.example.com,resources=manageddatabases/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=database.example.com,resources=manageddatabases/finalizers,verbs=update

func (r *ManagedDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	instance := &databasev1.ManagedDatabase{}
	if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
		if client.IgnoreNotFound(err) != nil {
			logger.Error(err, "Failed to get ManagedDatabase")
			return ctrl.Result{}, err
		}
		logger.Info("ManagedDatabase resource not found. Ignoring since object must be deleted.")
		return ctrl.Result{}, nil
	}

	if instance.ObjectMeta.DeletionTimestamp.IsZero() {
		// CREATE/UPDATE path
		if !controllerutil.ContainsFinalizer(instance, myFinalizerName) {
			logger.Info("Adding Finalizer")
			patch := client.MergeFrom(instance.DeepCopy())
			controllerutil.AddFinalizer(instance, myFinalizerName)
			if err := r.Patch(ctx, instance, patch); err != nil {
				logger.Error(err, "Failed to add finalizer")
				return ctrl.Result{}, err
			}
			return ctrl.Result{Requeue: true}, nil
		}

		// Main reconciliation logic
		return r.reconcileNormal(ctx, instance)
	} else {
		// DELETE path
		if controllerutil.ContainsFinalizer(instance, myFinalizerName) {
			if err := r.cleanupExternalResources(ctx, instance); err != nil {
				logger.Error(err, "External resource cleanup failed")
				return ctrl.Result{}, err
			}

			logger.Info("Removing Finalizer")
			patch := client.MergeFrom(instance.DeepCopy())
			controllerutil.RemoveFinalizer(instance, myFinalizerName)
			if err := r.Patch(ctx, instance, patch); err != nil {
				logger.Error(err, "Failed to remove finalizer")
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, nil
	}
}

func (r *ManagedDatabaseReconciler) reconcileNormal(ctx context.Context, db *databasev1.ManagedDatabase) (ctrl.Result, error) {
	// Implementation from the earlier section
	// 1. Check if external DB exists using db.GetUID()
	// 2. If not, create it.
	// 3. Update status with ID, endpoint, and Ready=True condition.
	// 4. If exists, ensure state matches spec (omitted).
	// ...
	return ctrl.Result{}, nil
}

func (r *ManagedDatabaseReconciler) cleanupExternalResources(ctx context.Context, db *databasev1.ManagedDatabase) error {
	// Implementation from the earlier section
	// 1. Call DBClient.DeleteDatabase(db.GetUID())
	// 2. If error is db_api.ErrNotFound, return nil.
	// 3. If other error, return the error.
	// 4. If no error, return nil.
	// ...
	return nil
}

func (r *ManagedDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&databasev1.ManagedDatabase{}).
		Complete(r)
}

Conclusion

Finalizers are not an optional enhancement for Kubernetes operators that manage external resources; they are a fundamental requirement for correctness. By treating the reconciliation loop as a state machine governed by the deletionTimestamp, you can build robust, fault-tolerant controllers that prevent resource leakage and provide a clean, predictable lifecycle for your Custom Resources.

The key takeaways for senior engineers are:

Always Add a Finalizer First: Add your finalizer before creating any external resources to prevent race conditions.

Implement Idempotent Cleanup: Your deletion logic must be able to run multiple times without adverse effects. Handling "Not Found" errors as a success state is the primary pattern.

Use the CR UID: Always use the metadata.uid as the unique key for external resources, not the name/namespace, which are mutable.

Monitor for Stuck Terminations: Have alerts in place for CRs that remain in a Terminating state for an extended period, as this indicates a persistent failure in your cleanup logic or its dependencies.

By internalizing these patterns, you can elevate your operators from simple automation scripts to truly production-grade, reliable infrastructure components.