Idempotent Kubernetes Finalizers for Stateful Resource Deletion

The Declarative Deletion Dilemma

In the Kubernetes ecosystem, we operate within a declarative paradigm. We define the desired state in a manifest, and controllers work tirelessly to make reality match that state. This works beautifully for stateless resources contained within the cluster. But what happens when a Kubernetes Custom Resource (CR) represents a tangible, stateful entity outside the cluster, like an S3 bucket, a Cloud SQL database, or a DNS record?

Consider a simple Database CR:

apiVersion: db.example.com/v1alpha1
kind: PostgresDatabase
metadata:
  name: my-prod-db
spec:
  storageGB: 20
  version: "14"
  region: "us-east-1"

A controller might react to this by provisioning an actual PostgreSQL instance in AWS RDS. The problem arises when an engineer runs kubectl delete postgresdatabase my-prod-db. From Kubernetes' perspective, the desired state is now the absence of this object. The API server obliges by removing the PostgresDatabase object from etcd. The controller's reconciliation loop for my-prod-db will no longer be triggered.

The result? The Kubernetes object is gone, but a costly AWS RDS instance is now orphaned, silently accruing charges. This is a critical resource leak, and it stems from a fundamental mismatch: Kubernetes's declarative, fire-and-forget deletion model doesn't account for the imperative, often multi-step, cleanup required by external systems.

This is the problem that Finalizers solve. They are a core Kubernetes mechanism that allows controllers to insert themselves into the deletion process, pausing the removal of an object from etcd until the controller signals that all associated external cleanup is complete.

Anatomy of a Finalizer-Driven Deletion

A finalizer is simply a string key added to the metadata.finalizers array of an object. Its presence is a signal to the Kubernetes API server. When a user requests to delete an object that has one or more finalizers:

This two-phase process transforms deletion from a simple removal into a robust, observable, and stateful workflow, giving controllers the power to guarantee graceful shutdowns.

Production Implementation in a Go Controller

Let's build a practical implementation using Go and the controller-runtime library, the foundation for Kubebuilder and the Operator SDK. We'll focus on the core Reconcile function for our PostgresDatabase controller.

First, we define our finalizer name. It's best practice to use a domain-prefixed name to avoid collisions with other controllers that might operate on the same object.

// controllers/postgresdatabase_controller.go

const postgresFinalizer = "db.example.com/finalizer"

Our Reconcile function becomes a state machine, branching based on the deletionTimestamp.

// controllers/postgresdatabase_controller.go

import (
	// ... other imports
	"context"

	"github.com/go-logr/logr"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

	dbv1alpha1 "github.com/your-org/your-repo/api/v1alpha1"
)

// ... Reconciler struct definition

func (r *PostgresDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := r.Log.WithValues("postgresdatabase", req.NamespacedName)

	// 1. Fetch the PostgresDatabase instance
	db := &dbv1alpha1.PostgresDatabase{}
	if err := r.Get(ctx, req.NamespacedName, db); err != nil {
		// Handle not-found errors, which can happen after deletion.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Examine DeletionTimestamp to determine if the object is under deletion.
	if db.ObjectMeta.DeletionTimestamp.IsZero() {
		// The object is NOT being deleted. Let's add our finalizer if it doesn't exist.

		if !controllerutil.ContainsFinalizer(db, postgresFinalizer) {
			log.Info("Adding finalizer for PostgresDatabase")
			controllerutil.AddFinalizer(db, postgresFinalizer)
			if err := r.Update(ctx, db); err != nil {
				log.Error(err, "Failed to add finalizer")
				return ctrl.Result{}, err
			}
		}

		// *** NORMAL RECONCILIATION LOGIC GOES HERE ***
		// (e.g., check if RDS instance exists, if not, create it)
		log.Info("Reconciling PostgresDatabase")
		// ...

	} else {
		// The object IS being deleted.

		if controllerutil.ContainsFinalizer(db, postgresFinalizer) {
			log.Info("Handling deletion for PostgresDatabase")
			// Our finalizer is present, so we must handle external dependency cleanup.
			if err := r.cleanupExternalResources(ctx, log, db); err != nil {
				// If cleanup fails, we don't remove the finalizer. 
				// This ensures we retry on the next reconciliation.
				log.Error(err, "External resource cleanup failed")
				return ctrl.Result{}, err
			}

			// Cleanup was successful. Remove the finalizer.
			log.Info("External resources cleaned up, removing finalizer")
			controllerutil.RemoveFinalizer(db, postgresFinalizer)
			if err := r.Update(ctx, db); err != nil {
				log.Error(err, "Failed to remove finalizer")
				return ctrl.Result{}, err
			}
		}

		// Stop reconciliation as the object is being deleted.
		return ctrl.Result{}, nil
	}

	return ctrl.Result{}, nil
}

This structure correctly separates the creation/update path from the deletion path. The key takeaways are:

Advanced Edge Cases and Idempotency

The previous example is a solid foundation, but production environments are messy. A robust controller must be resilient to crashes, network failures, and API errors. The most critical property of your cleanup logic is idempotency.

Scenario: Your controller calls the AWS API to delete an RDS instance. The API call succeeds, and the database begins terminating. Before the controller can remove the finalizer from the PostgresDatabase object, the controller pod crashes and restarts. When it reconciles the object again, the deletionTimestamp is still set, and the finalizer is still present. It will call cleanupExternalResources a second time.

If your cleanup function is not idempotent, this could cause an error. The function must handle the case where the resource it's trying to delete is already gone or is in the process of being deleted.

Here is a more robust implementation of cleanupExternalResources:

// A mock AWS SDK client for demonstration
type MockRDSClient struct{}

// In a real implementation, this would be the AWS SDK's DBInstanceNotFoundFault error code.
var ErrDBInstanceNotFound = errors.New("DBInstanceNotFound")

func (c *MockRDSClient) DeleteDBInstance(instanceID string) error {
    // MOCK LOGIC: In a real scenario, this would call the AWS API.
    // Here we simulate the case where it might already be deleted.
    fmt.Printf("Attempting to delete RDS instance: %s\n", instanceID)
    // if instance doesn't exist in our mock state, return not found error.
    // return ErrDBInstanceNotFound
    return nil // Simulate success
}

func (r *PostgresDatabaseReconciler) cleanupExternalResources(ctx context.Context, log logr.Logger, db *dbv1alpha1.PostgresDatabase) error {
	// This name should be derived from the CR's spec or status in a real controller,
	// ensuring a unique and predictable identifier for the external resource.
	externalDBInstanceID := "rds-" + db.Name

	log.Info("Deleting external RDS instance", "InstanceID", externalDBInstanceID)

	// Assume r.RDSClient is an interface to the AWS RDS API
	err := r.RDSClient.DeleteDBInstance(externalDBInstanceID)
	if err != nil {
		// CRITICAL: Check for the specific "Not Found" error from the cloud provider.
		// If the resource is already gone, we consider the cleanup a success.
		// The actual error code/type will depend on the cloud provider's SDK.
		if errors.Is(err, ErrDBInstanceNotFound) { // Use awserr.Is(err, rds.ErrCodeDBInstanceNotFoundFault) in real code
			log.Info("External RDS instance already deleted. Cleanup successful.")
			return nil
		}

		// For any other error, we must retry.
		log.Error(err, "Failed to delete external RDS instance")
		return err
	}

	log.Info("Successfully initiated deletion of external RDS instance")
	// Note: Some cloud deletions are asynchronous. You might need to poll for completion here.
	// For simplicity, we'll assume the Delete call is sufficient.
	return nil
}

The key logic is if errors.Is(err, ErrDBInstanceNotFound). We explicitly check if the deletion failed because the resource was already gone. If so, we treat it as a success and return nil, allowing the finalizer to be removed. Any other error is treated as a genuine failure and is propagated up, causing a requeue.

Handling Asynchronous Deletion

Many cloud resources are not deleted synchronously. An API call to delete a database might return a 200 OK immediately, but the resource enters a deleting state for several minutes. If you remove the finalizer immediately, another user could create a new Kubernetes object with the same name, and your controller might try to provision a new RDS instance with a name that is still in the process of being torn down by AWS, leading to a naming conflict.

A more robust pattern is to poll for the completion of the deletion process.

func (r *PostgresDatabaseReconciler) cleanupExternalResourcesWithPolling(ctx context.Context, log logr.Logger, db *dbv1alpha1.PostgresDatabase) error {
	externalDBInstanceID := "rds-" + db.Name

	// 1. Check current status of the instance
	instance, err := r.RDSClient.DescribeDBInstance(externalDBInstanceID)
	if err != nil {
		if errors.Is(err, ErrDBInstanceNotFound) {
			log.Info("External RDS instance not found. Cleanup complete.")
			return nil
		}
		return err // Other API error, retry
	}

	// 2. If instance is not in a deleting state, initiate deletion.
	if *instance.DBInstanceStatus != "deleting" {
		log.Info("Initiating deletion for RDS instance", "InstanceID", externalDBInstanceID)
		if err := r.RDSClient.DeleteDBInstance(externalDBInstanceID); err != nil {
			// Again, check for already-deleted race condition
			if errors.Is(err, ErrDBInstanceNotFound) {
				return nil 
			}
			return err
		}
		// Deletion initiated, but not complete. Requeue to check status later.
		log.Info("Deletion initiated, requeuing to poll for status.")
		return errors.New("RDS instance is still deleting") // Return an error to force requeue
	}

	// 3. If instance is already in a 'deleting' state, we just need to wait.
	log.Info("RDS instance is still in 'deleting' state. Requeuing.")
    // Returning an error ensures we reconcile again. The controller-runtime's default
    // exponential backoff is perfect for this kind of polling.
	return errors.New("RDS instance is still deleting")
}

In this improved version, we first check the status. If it's already gone, we're done. If it exists and isn't deleting, we start the deletion. Crucially, if the instance is in any state other than non-existent (e.g., available, deleting), we return an error. This forces the Reconcile loop to retry. The controller-runtime manager will automatically use an exponential backoff for these retries, which is exactly the polling behavior we want, preventing us from hammering the AWS API.

Performance and Stability Considerations

The "Stuck Finalizer" Problem

The biggest operational risk with this pattern is a stuck finalizer. This happens when a controller has a bug, is misconfigured, or is uninstalled, and it can no longer process the deletion logic to remove its own finalizer. The result is an object that cannot be deleted. kubectl delete will hang indefinitely.

This is a feature, not a bug—it's preventing resource leaks. But it requires manual intervention.

To fix a stuck finalizer, an administrator must manually patch the object to remove the finalizer from the list:

kubectl patch postgresdatabase my-prod-db -p '{"metadata":{"finalizers":[]}}' --type='merge'

Warning: This is a dangerous operation. Before running this command, the administrator must manually ensure that the external resource has been cleaned up. Otherwise, this command will orphan the resource, which is the very problem finalizers were designed to prevent.

Requeue-After for Transient Errors

When cleanupExternalResources fails due to a transient issue like API rate limiting from the cloud provider, returning a raw error triggers an immediate requeue with exponential backoff. This is generally good, but sometimes you want more control. For example, if you get a specific rate-limiting error, you might want to wait a longer, fixed period.

// ... inside Reconcile's deletion block

if err := r.cleanupExternalResources(ctx, log, db); err != nil {
    if IsRateLimitError(err) { // Hypothetical error checker
        log.Info("Hit API rate limit, requeueing after 1 minute")
        return ctrl.Result{RequeueAfter: time.Minute}, nil // Note: nil error
    }
    log.Error(err, "External resource cleanup failed")
    return ctrl.Result{}, err // Requeue with exponential backoff for other errors
}

By returning a ctrl.Result{RequeueAfter: ...} with a nil error, you are telling the controller manager that the reconciliation did not fail, but you would like to be triggered again after a specific duration. This can be a useful tool for managing interactions with fragile or rate-limited external APIs.

Controller Concurrency

By default, a controller manager can reconcile multiple resources concurrently (MaxConcurrentReconciles option). If a hundred PostgresDatabase objects are deleted at once, your controller might fire off a hundred simultaneous deletion requests to the cloud provider API, potentially triggering rate limits or causing other issues. When designing your cleanup logic, be aware of the downstream impact of concurrent operations and implement appropriate client-side rate limiting or adjust the controller's concurrency settings if necessary.

Conclusion

Finalizers are not just a feature; they are a fundamental building block for writing production-grade Kubernetes operators that manage anything beyond the cluster's edge. They elevate a controller from a simple state synchronizer to a true lifecycle manager.

By embracing the two-phase deletion process and meticulously implementing idempotent, error-aware cleanup logic, you can build controllers that are resilient, predictable, and prevent the costly resource leaks that occur when the declarative world of Kubernetes collides with the imperative reality of external systems. Mastering this pattern is a critical step for any engineer moving from building simple controllers to authoring robust, production-ready operators.