Back to Resources

Kubernetes Finalizers: Stateful Resource Cleanup in Custom Operators

12 min read
Goh Ling Yong
Technology enthusiast and software architect specializing in AI-driven development tools and modern software engineering practices. Passionate about the intersection of artificial intelligence and human creativity in building tomorrow's digital solutions.

The Deletion Lifecycle Gap: Why `kubectl delete` Isn't Enough

In a mature Kubernetes environment, operators are the cornerstone of automation, extending the Kubernetes API to manage complex, often stateful, applications. A common pattern is for a Custom Resource (CR) to represent an entity that exists outside the Kubernetes cluster—a managed database in a cloud provider, a DNS record, or an object in a storage bucket. The operator's reconciliation loop ensures the state of this external resource matches the spec of the CR.

However, a critical gap emerges during deletion. When a user executes kubectl delete my-cr, the Kubernetes API server marks the object for deletion. The default garbage collection process is swift and simple: it removes the object from etcd. This process has no awareness of the external resources your operator provisioned. The result is an orphaned resource—a database instance racking up costs, a DNS record pointing to a non-existent service, or a dangling storage bucket. This is not just a resource leak; it's a critical reliability and cost-management failure.

This is the problem that Finalizers solve. A finalizer is a list of strings in an object's metadata that acts as a locking mechanism. When a finalizer is present on an object, a delete request will only set a deletionTimestamp on the object. The object remains visible to the API and to controllers, but it is in a read-only, deleting state. It will not be fully removed from etcd by the garbage collector until its metadata.finalizers list is empty.

This provides a hook for your operator. The reconciliation loop can detect the deletionTimestamp, perform the necessary external cleanup tasks, and, only upon successful completion, remove its own finalizer from the list. This transforms the deletion process from a fire-and-forget operation into a robust, stateful, and verifiable workflow.

This article will walk through the implementation of a production-grade finalizer pattern in a Go-based operator built with Kubebuilder. We'll manage a hypothetical ManagedDatabase CR that provisions a user in an external PostgreSQL instance.

Scaffolding the `ManagedDatabase` Operator

We assume you have a working Go environment and have installed Kubebuilder. We'll start by scaffolding the project and API.

bash
# Initialize the project
kubebuilder init --domain my.domain --repo my.domain/managed-database-operator

# Create the API for our ManagedDatabase
kubebuilder create api --group db --version v1alpha1 --kind ManagedDatabase

Now, let's define the ManagedDatabase spec and status in api/v1alpha1/manageddatabase_types.go. The spec will define the desired state (the database and username), and the status will reflect the provisioned state.

go
// api/v1alpha1/manageddatabase_types.go

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ManagedDatabaseSpec defines the desired state of ManagedDatabase
type ManagedDatabaseSpec struct {
	// SecretName containing the connection details for the target PostgreSQL instance.
	// Must contain keys: 'host', 'port', 'user', 'password', 'dbname'.
	// +kubebuilder:validation:Required
	ConnectionSecretName string `json:"connectionSecretName"`

	// Username for the database user to be created.
	// +kubebuilder:validation:Required
	// +kubebuilder:validation:MinLength=4
	Username string `json:"username"`
}

// ManagedDatabaseStatus defines the observed state of ManagedDatabase
type ManagedDatabaseStatus struct {
	// Conditions represent the latest available observations of the ManagedDatabase's state.
	// +optional
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// Provisioned indicates whether the database user has been successfully created.
	// +optional
	Provisioned bool `json:"provisioned,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:name="Provisioned",type=boolean,JSONPath=".status.provisioned"
//+kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"

// ManagedDatabase is the Schema for the manageddatabases API
type ManagedDatabase struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:",inline"`

	Spec   ManagedDatabaseSpec   `json:"spec,omitempty"`
	Status ManagedDatabaseStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// ManagedDatabaseList contains a list of ManagedDatabase
type ManagedDatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:",inline"`
	Items           []ManagedDatabase `json:"items"`
}

func init() {
	SchemeBuilder.Register(&ManagedDatabase{}, &ManagedDatabaseList{})
}

After updating the types, run make manifests generate to update the CRD and generated code.

The Core Reconciliation Loop: Implementing Finalizer Logic

The heart of the operator is the Reconcile method in internal/controller/manageddatabase_controller.go. This is where we'll implement the complete lifecycle management, including the finalizer logic.

First, let's define our finalizer name as a constant.

go
// internal/controller/manageddatabase_controller.go

const managedDatabaseFinalizer = "db.my.domain/finalizer"

Now, we'll structure the Reconcile function. The core logic branches based on whether the deletionTimestamp is set.

go
// internal/controller/manageddatabase_controller.go

import (
	// ... other imports
	"context"
	"time"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	dbv1alpha1 "my.domain/managed-database-operator/api/v1alpha1"
)

// ... (Reconciler struct definition)

func (r *ManagedDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the ManagedDatabase instance
	dbInstance := &dbv1alpha1.ManagedDatabase{}
	if err := r.Get(ctx, req.NamespacedName, dbInstance); err != nil {
		logger.Error(err, "unable to fetch ManagedDatabase")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Examine if the object is under deletion
	isMarkedForDeletion := dbInstance.GetDeletionTimestamp() != nil
	if isMarkedForDeletion {
		if controllerutil.ContainsFinalizer(dbInstance, managedDatabaseFinalizer) {
			// Run finalization logic. If it fails, we'll retry.
			if err := r.finalizeManagedDatabase(ctx, dbInstance); err != nil {
				logger.Error(err, "failed to finalize managed database")
				// Return error to trigger requeue. Use exponential backoff.
				return ctrl.Result{}, err
			}

			// Cleanup was successful, remove our finalizer.
			logger.Info("External database user deleted, removing finalizer")
			controllerutil.RemoveFinalizer(dbInstance, managedDatabaseFinalizer)
			if err := r.Update(ctx, dbInstance); err != nil {
				return ctrl.Result{}, err
			}
		}
		// Stop reconciliation as the item is being deleted
		return ctrl.Result{}, nil
	}

	// 3. The object is not being deleted, so let's ensure our finalizer is present.
	if !controllerutil.ContainsFinalizer(dbInstance, managedDatabaseFinalizer) {
		logger.Info("Adding finalizer for ManagedDatabase")
		controllerutil.AddFinalizer(dbInstance, managedDatabaseFinalizer)
		if err := r.Update(ctx, dbInstance); err != nil {
			return ctrl.Result{}, err
		}
	}

	// 4. This is the main reconciliation logic: provision the database user.
	// We'll implement this as a separate function for clarity.
	if err := r.reconcileExternalDatabase(ctx, dbInstance); err != nil {
		logger.Error(err, "failed to reconcile external database")
		// Update status to reflect the error (not shown for brevity)
		return ctrl.Result{RequeueAfter: 30 * time.Second}, err
	}

	// Update status to reflect success
	dbInstance.Status.Provisioned = true
	if err := r.Status().Update(ctx, dbInstance); err != nil {
		logger.Error(err, "failed to update ManagedDatabase status")
		return ctrl.Result{}, err
	}

	logger.Info("Successfully reconciled ManagedDatabase")
	return ctrl.Result{}, nil
}

// (SetupWithManager function remains the same)

This structure is the canonical pattern for finalizer-aware controllers:

  • Fetch the CR: Standard boilerplate.
  • Check deletionTimestamp: This is the critical branch. If it's non-nil, the object is being deleted.
  • Handle Deletion: If deleting and our finalizer is present, we execute our cleanup logic (finalizeManagedDatabase). If cleanup succeeds, we remove the finalizer and update the object. The Kubernetes garbage collector will then see the empty finalizer list and complete the deletion. If cleanup fails, we return an error, causing the controller-runtime to requeue the request, typically with exponential backoff.
  • Handle Creation/Update: If the object is not being deleted, we first ensure our finalizer is present. This is an idempotent operation. We add it if it's missing and update the object. This is crucial because it ensures that from the moment our operator successfully acts on a resource, it is protected by the finalizer.
  • Reconcile External State: After the finalizer is in place, we proceed with the normal reconciliation logic (reconcileExternalDatabase).

    • Production-Grade Cleanup: Idempotency and Error Handling

    The finalizeManagedDatabase function is where we interact with the external system to perform cleanup. This function must be idempotent. That is, it must be safe to call multiple times. The reconciliation loop offers no guarantee of single execution; if a previous attempt failed or the operator restarted, the finalization logic will run again.

    Here is a robust implementation of the finalization and reconciliation logic. For this example, we'll simulate a database client.

    go
    // internal/controller/manageddatabase_controller.go

// NOTE: In a real application, this would be a proper database client.
// For demonstration, we'll use a mock client.

type MockDBClient struct {}

func (c *MockDBClient) CreateUser(username, password string) error {
	// Simulate creating a user. This should be idempotent.
	log.Log.Info("Simulating: CREATE USER", "username", username)
	// In a real DB, this might be 'CREATE USER ... IF NOT EXISTS'
	return nil
}

func (c *MockDBClient) DeleteUser(username string) error {
	// Simulate deleting a user. This should also be idempotent.
	log.Log.Info("Simulating: DROP USER", "username", username)
	// A real DB might return an error if the user doesn't exist.
	// The logic should handle 'user not found' as a success case for deletion.
	return nil
}

func (c *MockDBClient) UserExists(username string) (bool, error) {
    log.Log.Info("Simulating: CHECK IF USER EXISTS", "username", username)
    // This logic would query the database's system tables.
    return true, nil // Assume exists for now
}

// newMockDBClient would normally parse connection details from the secret.
func newMockDBClient(ctx context.Context, k8sClient client.Client, instance *dbv1alpha1.ManagedDatabase) (*MockDBClient, error) {
    // In a real implementation:
    // 1. Fetch the secret specified in instance.Spec.ConnectionSecretName
    // 2. Extract host, port, user, password, dbname
    // 3. Establish and verify a connection
    // 4. Return the client
    return &MockDBClient{}, nil
}

func (r *ManagedDatabaseReconciler) reconcileExternalDatabase(ctx context.Context, dbInstance *dbv1alpha1.ManagedDatabase) error {
	logger := log.FromContext(ctx)
	dbClient, err := newMockDBClient(ctx, r.Client, dbInstance)
	if err != nil {
		logger.Error(err, "failed to create database client")
		return err
	}

	// For simplicity, we assume a password would be generated and stored in a separate Secret.
	// Here we just use a placeholder.
	if err := dbClient.CreateUser(dbInstance.Spec.Username, "some-generated-password"); err != nil {
		logger.Error(err, "failed to create database user", "username", dbInstance.Spec.Username)
		return err
	}

	logger.Info("Successfully provisioned database user", "username", dbInstance.Spec.Username)
	return nil
}

func (r *ManagedDatabaseReconciler) finalizeManagedDatabase(ctx context.Context, dbInstance *dbv1alpha1.ManagedDatabase) error {
	logger := log.FromContext(ctx)
	dbClient, err := newMockDBClient(ctx, r.Client, dbInstance)
	if err != nil {
		// If we can't even connect to the DB, we can't clean up. Retry.
		logger.Error(err, "failed to create database client for finalization")
		return err
	}

	// Idempotency check: does the user still exist?
	exists, err := dbClient.UserExists(dbInstance.Spec.Username)
	if err != nil {
		logger.Error(err, "failed to check for user existence during finalization")
		return err
	}

	if !exists {
		logger.Info("Database user no longer exists. Cleanup is considered complete.")
		return nil
	}

	// The user exists, so we proceed with deletion.
	if err := dbClient.DeleteUser(dbInstance.Spec.Username); err != nil {
		logger.Error(err, "failed to delete database user", "username", dbInstance.Spec.Username)
		// This is a real failure; return the error to trigger a requeue.
		return err
	}

	logger.Info("Successfully deleted database user", "username", dbInstance.Spec.Username)
	return nil
}

    Key points for the finalizeManagedDatabase function:

    * Error Handling: If creating the database client fails (e.g., the database is down), we must return an error. The operator will retry, and the finalizer will prevent the CR from being deleted until the connection can be re-established and cleanup can be performed.

    * Idempotency: The logic first checks if the resource to be deleted still exists. If dbClient.UserExists returns false, it means a previous finalization attempt may have succeeded but the operator crashed before removing the finalizer. In this case, we consider the job done and return nil, allowing the finalizer to be removed.

    * Success on 'Not Found': A robust DeleteUser implementation should treat a "user not found" error from the database as a success condition for the purposes of cleanup.

    Advanced Edge Cases and Considerations

    While the above pattern covers 80% of use cases, senior engineers must consider the edge cases that arise in complex production systems.

    1. Stuck Finalizers

    A finalizer can become "stuck" if the operator's finalization logic continually fails or if the operator is uninstalled or misconfigured and can no longer process the CR. This leaves the CR in a terminating state indefinitely, and it cannot be deleted.

    Problem: A user runs kubectl delete my-cr, but it hangs forever.

    Diagnosis: kubectl get manageddatabase my-cr -o yaml will show a deletionTimestamp and the operator's finalizer still in metadata.finalizers.

    Manual Intervention (The Break-Glass Procedure):

    An administrator with sufficient permissions may need to manually patch the CR to remove the finalizer.

    bash
    kubectl patch manageddatabase my-cr --type json --patch='[{"op": "remove", "path": "/metadata/finalizers"}]'

    WARNING: This is a dangerous operation. It should only be performed after manually verifying that the external resource has been cleaned up. Bypassing the finalizer tells Kubernetes to complete the deletion, which will orphan the external resource if the operator's cleanup was genuinely failing for a valid reason (e.g., the external API was down).

    Prevention: Robust monitoring and alerting are key. Operators should expose metrics (e.g., via Prometheus) on the number of failing reconciliations and the duration objects spend in a terminating state. An alert can fire if a CR has been terminating for more than a reasonable period (e.g., one hour).

    2. Controller Concurrency and Race Conditions

    By default, controller-runtime can run multiple reconciliation loops in parallel (configurable via MaxConcurrentReconciles). While this improves throughput, it introduces potential race conditions, especially with finalizer logic.

    Scenario: Two reconcilers, R1 and R2, process the same deletion event for my-cr.

  • R1 calls finalizeManagedDatabase, which successfully deletes the external user.
    • Before R1 can remove the finalizer, R2 starts its loop.
  • R2 calls finalizeManagedDatabase. Because our logic is idempotent, it correctly sees the user is gone and returns nil.
  • R1 proceeds to remove the finalizer and calls r.Update(ctx, dbInstance).
  • R2 also proceeds to remove the finalizer and calls r.Update(ctx, dbInstance).

    • One of the Update calls will likely fail with a conflict error because the resource version has changed. Controller-runtime's client is designed to handle this, and the failing reconciler will simply requeue the request. On the next attempt, it will see the finalizer is already gone and do nothing. While the system self-heals, it's important to understand this behavior and ensure all state-modifying operations on the external system are idempotent.

    The critical race condition to avoid is between cleanup and finalizer removal. The order must always be:

    • Successfully confirm external resource deletion.
    • Remove the finalizer from the CR object in memory.
    • Persist the CR update to the API server.

    If the operator crashes between steps 1 and 3, the next reconciliation loop will safely re-run the idempotent cleanup logic.

    3. Long-Running Cleanup Tasks

    What if deleting an external resource takes minutes? A synchronous call in the finalizeManagedDatabase function would block the reconciler, consuming a worker goroutine and reducing the operator's ability to service other CRs.

    Solution: For long-running tasks, adopt an asynchronous pattern.

  • Initiate Deletion: The finalizeManagedDatabase function makes a non-blocking API call to the external system to start the deletion process. This external system should return a job ID or task token.
  • Update Status: Store this job ID in the CR's .status field and update the status.
  • Requeue: Return ctrl.Result{RequeueAfter: 30 * time.Second} to instruct the controller to check back later.
  • Poll for Completion: In subsequent reconciliation loops for this terminating CR, the logic checks the status. If a job ID is present, it polls the external system for the job's status.
  • Finalize: Once the external job completes successfully, the operator can then remove its finalizer.

    • This approach keeps the operator responsive, but adds significant complexity to the status management and reconciliation logic.

    Conclusion: Finalizers as a Pillar of Reliable Automation

    Finalizers are not an optional feature for operators that manage external state; they are a fundamental requirement for building reliable, production-grade automation on Kubernetes. By providing a contract between your controller and the Kubernetes garbage collector, they ensure that your cleanup logic is always executed before a resource is removed from the cluster's view.

    Mastering the finalizer pattern involves more than just adding and removing a string from a list. It requires a deep understanding of idempotency, robust error handling for external systems, and awareness of edge cases like stuck finalizers and long-running tasks. The patterns discussed here—branching on deletionTimestamp, ensuring finalizer presence before action, and implementing idempotent cleanup—form the bedrock of any operator that safely bridges the gap between the Kubernetes API and the outside world.

    Found this article helpful?

    Share it with others who might benefit from it.

    More Articles