Kubernetes Operators: Implementing Finalizers for Stateful Resource Deletion
The Deletion Lifecycle Fallacy in Stateful Systems
In a stateless world, kubectl delete is a fire-and-forget operation. The Kubernetes garbage collector efficiently reaps associated objects like Pods, ReplicaSets, and Services. However, when an operator manages resources outside the Kubernetes cluster—a managed database, a DNS record, a cloud storage bucket—this model breaks down. Deleting a Custom Resource (CR) instance without a proper cleanup mechanism results in orphaned, and often costly, external resources.
This is where the Kubernetes finalizer pattern becomes indispensable. A finalizer is not a piece of code; it's a declarative lock. It's a key in an object's metadata.finalizers array that tells the Kubernetes API server, "Do not fully delete this object until my controller says it's okay." This mechanism transforms the deletion process from an abrupt removal into a two-phase, controller-managed shutdown sequence.
When a user requests deletion of an object with a finalizer, the API server simply sets the metadata.deletionTimestamp field. The object enters a Terminating state but remains visible via the API. This is the signal for the operator's reconciliation loop to execute its pre-defined cleanup logic. Only after the controller verifies successful cleanup and removes its finalizer key from the array does the API server proceed with garbage collection.
This article bypasses the introductory concepts and dives directly into the production-level implementation of a finalizer within a Go-based operator built with Kubebuilder/Operator-SDK. We will focus on a realistic scenario: managing a hypothetical ManagedDatabase CR that requires a final backup before its underlying cloud resource is de-provisioned.
Core Implementation Pattern in Controller-Runtime
The Reconcile function is the heart of any operator. When implementing a finalizer, we must partition its logic to handle two primary states: the object is alive, or the object is terminating.
Our Reconcile function's control flow will look like this:
- Fetch the CR instance.
metadata.deletionTimestamp is zero. * If zero (Object is alive): Ensure our finalizer is present. If not, add it and update the object. This is a critical self-healing step to ensure that even pre-existing resources become managed by the finalizer logic. Then, proceed with normal reconciliation (create/update logic).
* If non-zero (Object is terminating): Check if our finalizer is still present. If so, execute the cleanup logic. Upon successful cleanup, remove the finalizer and update the object. If the finalizer is absent, the work is done; do nothing.
Let's translate this into Go code within a typical Reconcile method.
package controllers
import (
"context"
"time"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
dboperatorv1alpha1 "github.com/your-org/db-operator/api/v1alpha1"
)
// ManagedDatabaseReconciler reconciles a ManagedDatabase object
type ManagedDatabaseReconciler struct {
client.Client
Scheme *runtime.Scheme
// A stub for an external service client
CloudDBManager CloudDatabaseManager
}
// A unique name for our finalizer
const managedDatabaseFinalizer = "db.operator.example.com/finalizer"
func (r *ManagedDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// 1. Fetch the ManagedDatabase instance
instance := &dboperatorv1alpha1.ManagedDatabase{}
err := r.Get(ctx, req.NamespacedName, instance)
if err != nil {
// Request object not found, could have been deleted after reconcile request.
// Return and don't requeue
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// 2. Examine if the object is under deletion
if instance.ObjectMeta.DeletionTimestamp.IsZero() {
// The object is not being deleted, so if it does not have our finalizer,
// then lets add the finalizer and update the object. This is equivalent
// to registering our finalizer.
if !controllerutil.ContainsFinalizer(instance, managedDatabaseFinalizer) {
logger.Info("Adding finalizer for ManagedDatabase")
controllerutil.AddFinalizer(instance, managedDatabaseFinalizer)
if err := r.Update(ctx, instance); err != nil {
logger.Error(err, "Failed to update ManagedDatabase with finalizer")
return ctrl.Result{}, err
}
}
} else {
// The object is being deleted
if controllerutil.ContainsFinalizer(instance, managedDatabaseFinalizer) {
// Our finalizer is present, so lets handle any external dependency
if err := r.handleFinalization(ctx, instance); err != nil {
// If fail to delete the external dependency here, return with error
// so that it can be retried.
logger.Error(err, "Finalization failed. Retrying...")
return ctrl.Result{}, err
}
// Once finalization is complete, remove the finalizer.
logger.Info("External resources cleaned up, removing finalizer")
controllerutil.RemoveFinalizer(instance, managedDatabaseFinalizer)
if err := r.Update(ctx, instance); err != nil {
return ctrl.Result{}, err
}
}
// Stop reconciliation as the item is being deleted
return ctrl.Result{}, nil
}
// --- Your normal reconciliation logic for create/update events goes here ---
logger.Info("Reconciling ManagedDatabase normally")
// ...
return ctrl.Result{}, nil
}
// SetupWithManager sets up the controller with the Manager.
func (r *ManagedDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&dboperatorv1alpha1.ManagedDatabase{}).
Complete(r)
}This structure correctly separates the creation/update path from the deletion path, using the DeletionTimestamp as the switch.
Building an Idempotent, Multi-Step Finalizer
Real-world cleanup is rarely a single API call. It's often a sequence of operations that must be idempotent. If the operator pod crashes and restarts mid-cleanup, the Reconcile function will be called again for the terminating object. The cleanup logic must be able to resume gracefully without causing errors or duplicate actions.
Our handleFinalization function will orchestrate the following stateful cleanup for a ManagedDatabase:
Condition on the CR to Terminating with a reason BackupInProgress to provide visibility to users.Completed.Here is a detailed implementation of handleFinalization, demonstrating idempotency and status updates.
// A mock client for interacting with a cloud DB provider
type CloudDatabaseManager interface {
InitiateBackup(ctx context.Context, dbID string) (string, error)
GetBackupStatus(ctx context.Context, backupID string) (string, error) // Returns "InProgress", "Completed", "Failed"
DeleteDatabase(ctx context.Context, dbID string) error
GetDatabaseStatus(ctx context.Context, dbID string) (string, error) // Returns "Available", "Deleting", "NotFound"
}
func (r *ManagedDatabaseReconciler) handleFinalization(ctx context.Context, instance *dboperatorv1alpha1.ManagedDatabase) error {
logger := log.FromContext(ctx)
// The external resource ID is typically stored in the CR's status after creation.
dbID := instance.Status.DatabaseID
if dbID == "" {
logger.Info("Database ID not found in status, assuming external resource is already gone.")
return nil
}
// --- Step 1: Handle Final Backup ---
backupID := instance.Status.FinalBackupID
if backupID == "" {
logger.Info("Initiating final backup for database", "dbID", dbID)
newBackupID, err := r.CloudDBManager.InitiateBackup(ctx, dbID)
if err != nil {
// Update status to reflect failure
// meta.SetStatusCondition(&instance.Status.Conditions, metav1.Condition{...})
// r.Status().Update(ctx, instance)
return fmt.Errorf("failed to initiate final backup: %w", err)
}
instance.Status.FinalBackupID = newBackupID
// Update status immediately and requeue
if err := r.Status().Update(ctx, instance); err != nil {
return err
}
// Requeue to start polling
return fmt.Errorf("backup initiated, requeueing to check status") // Use error to force requeue with backoff
}
// --- Step 2: Poll Backup Status (Idempotent Check) ---
backupStatus, err := r.CloudDBManager.GetBackupStatus(ctx, backupID)
if err != nil {
return fmt.Errorf("failed to get backup status for ID %s: %w", backupID, err)
}
switch backupStatus {
case "Completed":
logger.Info("Final backup completed successfully", "backupID", backupID)
// Proceed to next step
case "InProgress":
logger.Info("Final backup is still in progress", "backupID", backupID)
// This is where intelligent requeueing is critical. Returning an error causes exponential backoff,
// which might be too aggressive. We will refine this later.
return fmt.Errorf("backup in progress, will re-check")
case "Failed":
logger.Error(nil, "Final backup failed. Manual intervention may be required.", "backupID", backupID)
// Update status to reflect failure. Don't remove the finalizer.
// The object will be stuck in 'Terminating' until an admin intervenes.
return fmt.Errorf("final backup failed with ID %s", backupID)
default:
return fmt.Errorf("unknown backup status: %s", backupStatus)
}
// --- Step 3: De-provision the Database ---
dbStatus, err := r.CloudDBManager.GetDatabaseStatus(ctx, dbID)
if err != nil && dbStatus != "NotFound" {
return fmt.Errorf("failed to get database status for ID %s: %w", dbID, err)
}
if dbStatus == "NotFound" {
logger.Info("Database already de-provisioned.", "dbID", dbID)
return nil // Cleanup is complete
}
// Only call delete if it's not already in a 'deleting' state
if dbStatus != "Deleting" {
logger.Info("De-provisioning database instance", "dbID", dbID)
if err := r.CloudDBManager.DeleteDatabase(ctx, dbID); err != nil {
return fmt.Errorf("failed to de-provision database %s: %w", dbID, err)
}
}
logger.Info("Database de-provisioning in progress. Requeuing for verification.")
return fmt.Errorf("waiting for database deletion confirmation")
}Notice the idempotency checks: we don't initiate a backup if FinalBackupID already exists. We don't issue a DeleteDatabase call if the database is already NotFound or Deleting. This resilience is paramount for production operators.
Advanced Edge Cases and Performance Tuning
Simple finalizer logic works in ideal conditions. Production environments are never ideal. Here's how to handle the complex realities.
1. The "Stuck in Terminating" Problem
A finalizer is a double-edged sword. If your handleFinalization logic can never complete—due to a bug, a persistent external API failure, or a logical impossibility—the CR will be stuck in the Terminating state forever. Kubernetes will refuse to delete it.
Mitigation Strategies:
* Timeouts and Status Conditions: Implement a timeout within your finalization logic. If a backup is InProgress for more than a reasonable period (e.g., 12 hours), update the CR's status to a Degraded or FinalizationFailed condition. This provides observability for platform administrators.
* Alerting: Configure monitoring to alert when a CR has been in a Terminating state for an extended period.
* Manual Intervention (The Last Resort): An administrator can forcibly remove the finalizer via kubectl patch:
kubectl patch manageddatabase my-db -p '{"metadata":{"finalizers":[]}}' --type=mergeThis is a dangerous operation. It breaks the operator's contract and will almost certainly lead to orphaned external resources. It should only be performed when the operator is confirmed to be non-functional and the external resources have been cleaned up manually.
2. Intelligent Requeueing vs. Exponential Backoff
In our handleFinalization example, we returned an error fmt.Errorf("backup in progress...") to trigger a requeue. By default, controller-runtime uses an exponential backoff strategy for retries on error. This is often desirable for transient network errors, but it's inefficient for polling.
If a backup takes 30 minutes, we don't want to requeue after 1s, 2s, 4s, 8s... We want to requeue after a fixed, reasonable interval, like 1 minute.
To achieve this, we must return a ctrl.Result with RequeueAfter set, instead of an error.
Refined Polling Logic:
// Inside the Reconcile function's deletion block
if controllerutil.ContainsFinalizer(instance, managedDatabaseFinalizer) {
// handleFinalization now returns a ctrl.Result and an error
result, err := r.handleFinalization(ctx, instance)
if err != nil {
logger.Error(err, "Finalization failed. Retrying with backoff...")
return ctrl.Result{}, err // Use exponential backoff for true errors
}
if result.Requeue || result.RequeueAfter > 0 {
logger.Info("Finalization in progress. Requeuing...", "after", result.RequeueAfter)
return result, nil // Requeue as requested
}
// Finalization is complete, remove the finalizer
logger.Info("External resources cleaned up, removing finalizer")
controllerutil.RemoveFinalizer(instance, managedDatabaseFinalizer)
if err := r.Update(ctx, instance); err != nil {
return ctrl.Result{}, err
}
}And the handleFinalization function signature and logic must be updated:
// handleFinalization now returns a result for requeueing control
func (r *ManagedDatabaseReconciler) handleFinalization(ctx context.Context, instance *dboperatorv1alpha1.ManagedDatabase) (ctrl.Result, error) {
// ... (backup initiation logic as before, returning error on failure)
// --- Step 2: Poll Backup Status (Idempotent Check) ---
backupStatus, err := r.CloudDBManager.GetBackupStatus(ctx, backupID)
if err != nil {
// This is a real error, not a state to poll. Use backoff.
return ctrl.Result{}, fmt.Errorf("failed to get backup status: %w", err)
}
switch backupStatus {
case "Completed":
logger.Info("Final backup completed successfully")
case "InProgress":
logger.Info("Final backup is still in progress")
// Intelligent requeue: check again in 30 seconds
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
case "Failed":
logger.Error(nil, "Final backup failed. Stopping reconciliation.")
// A terminal failure. Don't requeue, let it get stuck for manual review.
return ctrl.Result{}, nil
}
// ... (database de-provisioning logic)
// When polling for DB deletion confirmation:
if dbStatus == "Deleting" {
logger.Info("Database de-provisioning in progress.")
return ctrl.Result{RequeueAfter: 1 * time.Minute}, nil
}
// If we reach here, all steps are complete.
// Return empty result and no error to signal completion.
return ctrl.Result{}, nil
}This refined approach gives us fine-grained control over the reconciliation loop, preventing API throttling and ensuring efficient polling for long-running asynchronous tasks.
3. Concurrency and Optimistic Locking
By default, a controller can reconcile multiple objects concurrently (MaxConcurrentReconciles). What happens if a user deletes a CR at the exact moment the controller is performing a regular update? The controller-runtime's client and the Kubernetes API server protect us through optimistic locking.
When you Get() an object, you retrieve it at a specific resourceVersion. When you Update() or Status().Update(), the API server will reject the request if the resourceVersion you're providing doesn't match the one currently stored in etcd. The controller-runtime client will then return an error. The Reconcile function will fail, and the request will be automatically requeued. On the next attempt, the controller will Get() the fresh version of the object, which will now have the deletionTimestamp set, and the logic will correctly proceed down the finalization path.
While this is largely handled for you, it's crucial to understand this mechanism. Your reconciliation logic should be stateless and idempotent, assuming that any Reconcile call could be aborted and retried with a newer version of the object at any time.
Complete Production-Grade Example Snippet
Here is a more complete view of the files involved.
api/v1alpha1/manageddatabase_types.go
package v1alpha1
import (
metav1 "k8s.ioio/apimachinery/pkg/apis/meta/v1"
)
// ManagedDatabaseSpec defines the desired state of ManagedDatabase
type ManagedDatabaseSpec struct {
DBName string `json:"dbName"`
Engine string `json:"engine"`
Size string `json:"size"`
}
// ManagedDatabaseStatus defines the observed state of ManagedDatabase
type ManagedDatabaseStatus struct {
// Represents the observations of a ManagedDatabase's current state.
Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
// The ID of the database instance in the cloud provider
DatabaseID string `json:"databaseId,omitempty"`
// The ID of the final backup job triggered during deletion
FinalBackupID string `json:"finalBackupId,omitempty"`
}
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
type ManagedDatabase struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ManagedDatabaseSpec `json:"spec,omitempty"`
Status ManagedDatabaseStatus `json:"status,omitempty"`
}
// ... List types ...controllers/manageddatabase_controller.go
(Combines the logic discussed above into a cohesive file. See previous sections for the detailed implementation of Reconcile and handleFinalization)
Conclusion: Finalizers as a Contract for Reliability
Implementing a finalizer is more than just adding a string to an object's metadata; it's establishing a contract between your operator and the Kubernetes control plane. This contract guarantees that your controller gets the final say before a resource it manages is removed from the cluster. For operators managing any stateful or external resource, this is not an optional feature—it is a fundamental requirement for building a reliable, production-ready system.
By mastering idempotent cleanup logic, intelligent requeue strategies, and robust handling of edge cases like external API failures, you can build operators that prevent resource leakage, ensure data integrity, and provide the seamless, declarative experience that Kubernetes promises.