Idempotent K8s Operator Reconcilers with Finalizers and Caching

October 6, 2025

15 min read

Goh Ling Yong

Technology enthusiast and software architect specializing in AI-driven development tools and modern software engineering practices. Passionate about the intersection of artificial intelligence and human creativity in building tomorrow's digital solutions.

The Fallacy of the Simple Reconciliation Loop

In the world of Kubernetes operators, the reconciliation loop is the heart of the controller. The canonical goal is to make the state of the world match the desired state declared in a Custom Resource (CR). A junior developer's first pass at a reconciler for an ExternalDatabase CR might look deceptively simple: fetch the CR, check if the database exists via a cloud provider's API, and if not, create it.

This approach is fundamentally flawed and dangerous in a production environment. It's a direct path to resource leaks, inconsistent states, and API throttling. The core issues stem from a lack of idempotency and lifecycle management.

Consider this naive reconciliation flow:

Reconcile is triggered for a new ExternalDatabase CR.

Controller checks the cloud provider API. No database found.

Controller calls cloud.CreateDatabase().

The API call succeeds, and the database starts provisioning.

The controller attempts to update the CR's .status field to Provisioning.

This status update API call to the Kubernetes API server fails (e.g., transient network issue, etcd contention).

The reconciler will be triggered again. It will re-run the same logic, see no database (because its internal state wasn't updated), and attempt to call cloud.CreateDatabase() a second time. Depending on the external API's behavior, this could either fail with a 'resource already exists' error (the best case) or, in worse scenarios, create a second, orphaned database.

Furthermore, what happens when a user runs kubectl delete externaldatabase my-db? The CR is removed from etcd, but the controller is never notified in a way that allows it to clean up the actual database in the cloud. The result is an orphaned, billable resource.

This article dissects the patterns required to build a robust, idempotent, and performant reconciliation loop that gracefully handles creation, updates, and, most critically, deletion of external resources. We will focus on two primary mechanisms: Finalizers for lifecycle management and Controller-Side Caching for performance and rate-limit avoidance.

Section 1: Achieving True Idempotency with State-Aware Reconciliation

Idempotency means that an operation can be applied multiple times without changing the result beyond the initial application. In our context, a reconciliation loop should be able to run 100 times in a row and, assuming the CR spec doesn't change, result in the exact same external resource state without causing errors or side effects.

The key is to shift from a command-based approach (Create!, Update!) to a state-based one (Is the current state the desired state? If not, make it so.).

The State-Driven Reconciler Pattern

A robust reconciler always follows this sequence:

Fetch the Custom Resource instance.

Fetch the actual state of the corresponding external resource.

Compare the desired state (from the CR spec) with the actual state.

If they differ, calculate the necessary delta and perform the minimal action (Create, Update, or Delete) to converge the states.

Update the CR status with the observed actual state.

Let's implement this for our ExternalDatabase controller using Go and the controller-runtime library.

First, our ExternalDatabase CRD definition (api/v1/externaldatabase_types.go):

package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ExternalDatabaseSpec defines the desired state of ExternalDatabase
type ExternalDatabaseSpec struct {
	Engine  string `json:"engine"`         // e.g., "postgres" or "mysql"
	Version string `json:"version"`       // e.g., "14.2"
	SizeGB  int    `json:"sizeGb"`          // e.g., 20
}

// ExternalDatabaseStatus defines the observed state of ExternalDatabase
type ExternalDatabaseStatus struct {
	// +optional
	State string `json:"state,omitempty"` // e.g., "Provisioning", "Available", "Failed"
	// +optional
	Endpoint string `json:"endpoint,omitempty"`
	// +optional
	ProviderID string `json:"providerId,omitempty"` // The ID of the resource in the external system
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// ExternalDatabase is the Schema for the externaldatabases API
type ExternalDatabase struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   ExternalDatabaseSpec   `json:"spec,omitempty"`
	Status ExternalDatabaseStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// ExternalDatabaseList contains a list of ExternalDatabase
type ExternalDatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []ExternalDatabase `json:"items"`
}

func init() {
	SchemeBuilder.Register(&ExternalDatabase{}, &ExternalDatabaseList{})
}

Now, the improved Reconcile function. Note how it explicitly fetches the external state before acting.

// controllers/externaldatabase_controller.go

import (
	// ... other imports
	"context"
	"fmt"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	apiv1 "my.domain/externaldb/api/v1"
	"my.domain/externaldb/internal/cloudprovider"
)

type ExternalDatabaseReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	Cloud  *cloudprovider.Client // Our mock cloud provider client
}

func (r *ExternalDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the ExternalDatabase CR instance
	var db crd.ExternalDatabase
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		logger.Error(err, "unable to fetch ExternalDatabase")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Fetch the actual state of the external resource
	// We use the CR's status to store the provider ID.
	externalDB, err := r.Cloud.GetDatabase(ctx, db.Status.ProviderID)
	if err != nil && !cloudprovider.IsNotFound(err) {
		logger.Error(err, "failed to get database from cloud provider")
		// Requeue with backoff if it's a transient error
		return ctrl.Result{RequeueAfter: 30 * time.Second}, err
	}

	// 3. Compare desired state with actual state

	// Case 1: External resource does not exist, so we must create it.
	if cloudprovider.IsNotFound(err) || externalDB == nil {
		logger.Info("External database not found, creating a new one.")
		newDB, err := r.Cloud.CreateDatabase(ctx, &cloudprovider.DatabaseSpec{
			Name:    db.Name,
			Engine:  db.Spec.Engine,
			Version: db.Spec.Version,
			SizeGB:  db.Spec.SizeGB,
		})
		if err != nil {
			logger.Error(err, "failed to create external database")
			db.Status.State = "Failed"
			_ = r.Status().Update(ctx, &db) // Best effort status update
			return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
		}

		// IMPORTANT: Update status immediately with the new ProviderID
		db.Status.ProviderID = newDB.ID
		db.Status.State = newDB.State
		db.Status.Endpoint = newDB.Endpoint
		if err := r.Status().Update(ctx, &db); err != nil {
			logger.Error(err, "failed to update ExternalDatabase status after creation")
			// If this fails, the next reconcile will fix it because we can still find the DB by name/tags
			return ctrl.Result{Requeue: true}, err
		}
		logger.Info("Successfully created external database and updated status", "ProviderID", newDB.ID)
		return ctrl.Result{}, nil
	}

	// Case 2: External resource exists, check for drift.
	if externalDB.SizeGB != db.Spec.SizeGB || externalDB.Version != db.Spec.Version {
		logger.Info("Drift detected. Updating external database.", "currentSize", externalDB.SizeGB, "desiredSize", db.Spec.SizeGB)
		updatedDB, err := r.Cloud.UpdateDatabase(ctx, externalDB.ID, &cloudprovider.DatabaseSpec{
			// Assume only size can be updated for simplicity
			SizeGB: db.Spec.SizeGB,
		})
		if err != nil {
			logger.Error(err, "failed to update external database")
			return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
		}

		// Update status with the latest observed state
		db.Status.State = updatedDB.State
		if err := r.Status().Update(ctx, &db); err != nil {
			logger.Error(err, "failed to update status after modification")
			return ctrl.Result{Requeue: true}, err
		}
		return ctrl.Result{}, nil
	}

	// Case 3: No drift, just update status if needed.
	if db.Status.State != externalDB.State {
		db.Status.State = externalDB.State
		db.Status.Endpoint = externalDB.Endpoint // Endpoint might change
		if err := r.Status().Update(ctx, &db); err != nil {
			logger.Error(err, "failed to sync status")
			return ctrl.Result{Requeue: true}, err
		}
	}

	logger.Info("Reconciliation complete, no action needed.")
	return ctrl.Result{}, nil
}

This is a significant improvement. If the status update fails after creation, the next reconciliation will call r.Cloud.GetDatabase(ctx, db.Status.ProviderID). If the ProviderID wasn't saved, this call will fail. A robust implementation of GetDatabase should be able to look up the resource by tags (e.g., kubernetes.io/cr-name: my-db) to find the ProviderID and self-heal the status. This self-healing capability is a hallmark of an idempotent operator.

However, this code still has a critical flaw: it doesn't handle deletion.

Section 2: Graceful Deletion with Finalizers

When a user deletes a Kubernetes object, it is immediately removed from etcd. The reconciliation loop is not triggered for the deletion event itself. To solve the orphaned resource problem, we must intercept the deletion process. This is the role of finalizers.

A finalizer is a key in the metadata.finalizers list of an object. When a user requests to delete an object that has finalizers, the API server does two things:

It sets the metadata.deletionTimestamp field to the current time.

It leaves the object in etcd.

The object is now in a 'terminating' state. The presence of deletionTimestamp is a signal to our controller that it must perform cleanup logic. The controller's reconciliation loop will continue to be called for this terminating object. Once our controller finishes its cleanup (e.g., deleting the external database), it must remove its finalizer from the metadata.finalizers list. Only when this list is empty will the Kubernetes garbage collector finally delete the CR object from etcd.

Implementing the Finalizer Pattern

Step 1: Define and Add the Finalizer

First, we define a unique finalizer string for our controller.

// controllers/externaldatabase_controller.go
const externalDatabaseFinalizer = "db.my.domain/finalizer"

Next, we modify our reconciler to add this finalizer to any new CR instance that doesn't have it.

// Inside the Reconcile function, after fetching the 'db' object

// ... (fetch db object as before)

// Examine if the object is under deletion
if db.ObjectMeta.DeletionTimestamp.IsZero() {
	// The object is not being deleted, so if it does not have our finalizer,
	// then lets add the finalizer and update the object. This is equivalent
	// to registering our finalizer.
	if !controllerutil.ContainsFinalizer(&db, externalDatabaseFinalizer) {
		controllerutil.AddFinalizer(&db, externalDatabaseFinalizer)
		if err := r.Update(ctx, &db); err != nil {
			logger.Error(err, "failed to add finalizer")
			return ctrl.Result{}, err
		}
		// Requeue immediately after adding the finalizer to process the rest
		return ctrl.Result{Requeue: true}, nil
	}
} else {
	// The object is being deleted. Handle finalizer logic.
	// (This part will be implemented next)
}

// ... (rest of the reconciliation logic for create/update)

Step 2: Implement the Deletion Logic

Now, we add the else block to handle the case where deletionTimestamp is set.

// ... (inside the Reconcile function)

if db.ObjectMeta.DeletionTimestamp.IsZero() {
	// ... (logic to add finalizer, create/update external resource)
} else {
	// The object is being deleted
	if controllerutil.ContainsFinalizer(&db, externalDatabaseFinalizer) {
		logger.Info("Performing cleanup for ExternalDatabase.")
		// Our finalizer is present, so lets handle any external dependency
		if err := r.deleteExternalResources(ctx, &db); err != nil {
			// if fail to delete the external dependency here, return with error
			// so that it can be retried
			logger.Error(err, "failed to delete external resources")
			// Use RequeueAfter to avoid tight loop on persistent failure
			return ctrl.Result{RequeueAfter: 30 * time.Second}, err
		}

		// Once external resources are gone, remove the finalizer.
		logger.Info("External resources deleted. Removing finalizer.")
		controllerutil.RemoveFinalizer(&db, externalDatabaseFinalizer)
		if err := r.Update(ctx, &db); err != nil {
			logger.Error(err, "failed to remove finalizer")
			return ctrl.Result{}, err
		}
	}

	// Stop reconciliation as the item is being deleted
	return ctrl.Result{}, nil
}

// ... (rest of the reconciliation logic)

We need a helper function to perform the actual deletion.

func (r *ExternalDatabaseReconciler) deleteExternalResources(ctx context.Context, db *crd.ExternalDatabase) error {
	logger := log.FromContext(ctx)

	if db.Status.ProviderID == "" {
		logger.Info("ProviderID is empty, assuming external resource was never created.")
		return nil
	}

	logger.Info("Deleting external database", "ProviderID", db.Status.ProviderID)
	err := r.Cloud.DeleteDatabase(ctx, db.Status.ProviderID)
	if err != nil && !cloudprovider.IsNotFound(err) {
		return fmt.Errorf("failed to delete external database %s: %w", db.Status.ProviderID, err)
	}

	logger.Info("Successfully deleted external database.")
	return nil
}

This pattern ensures that our cleanup logic is executed before Kubernetes deletes the CR. If deleteExternalResources fails, the reconciler returns an error, and the loop will be retried. The finalizer remains, preventing the CR's deletion until the external dependency is successfully removed. This is the foundation of a production-grade, leak-proof operator.

Section 3: Performance Optimization with External Resource Caching

Our reconciler is now robust, but it's not efficient. Every reconciliation, even for a CR that hasn't changed, results in at least one API call to the cloud provider (r.Cloud.GetDatabase). In a cluster with thousands of CRs, this can lead to significant problems:

* API Rate Throttling: Cloud providers enforce strict rate limits. A busy operator can easily exceed these limits, causing reconciliation failures across all its managed resources.

* Increased Latency: Network calls to external systems are slow. High reconciliation latency means the system is slow to respond to changes.

* Cost: Many cloud APIs have a cost associated with them.

We can mitigate this by implementing a controller-side cache for the state of our external resources.

Important Distinction: This is not the built-in controller-runtime cache for Kubernetes objects. That cache, populated by informers, keeps a local copy of objects from the Kubernetes API server. We need a separate, custom cache for the objects managed in the external system.

Caching Strategy and Implementation

We will use a simple in-memory, time-based cache. The reconciler will first check the cache for the external resource's state. If a fresh entry exists, it will use that instead of making an API call. If the entry is missing or stale, it will fetch from the source of truth (the cloud provider), perform the reconciliation, and update the cache with the new state.

Let's use a library like patrickmn/go-cache for simplicity.

Step 1: Integrate the Cache into the Reconciler

// main.go

import (
	// ...
	"time"
	"github.com/patrickmn/go-cache"
)

func main() {
	// ... (setup code)

	// Create a cache with a 5-minute default expiration and 10-minute purge interval.
	externalResourceCache := cache.New(5*time.Minute, 10*time.Minute)

	if err = (&controllers.ExternalDatabaseReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
		Cloud:  cloudprovider.NewClient(), // Your cloud client
		Cache:  externalResourceCache,
	}).SetupWithManager(mgr); err != nil {
		// ...
	}

	// ...
}

Update the reconciler struct:

// controllers/externaldatabase_controller.go

type ExternalDatabaseReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	Cloud  *cloudprovider.Client
	Cache  *cache.Cache
}

Step 2: Create a Cached Getter Function

We'll wrap our r.Cloud.GetDatabase call in a function that interacts with the cache.

// controllers/externaldatabase_controller.go

func (r *ExternalDatabaseReconciler) getCachedExternalDatabase(ctx context.Context, cr *crd.ExternalDatabase) (*cloudprovider.Database, error) {
	logger := log.FromContext(ctx)
	cacheKey := client.ObjectKeyFromObject(cr).String()

	// 1. Check cache first
	if cached, found := r.Cache.Get(cacheKey); found {
		if db, ok := cached.(*cloudprovider.Database); ok {
			logger.Info("Cache hit for external database state")
			return db, nil
		}
	}

	logger.Info("Cache miss. Fetching from cloud provider.")
	// 2. Cache miss, fetch from the API
	if cr.Status.ProviderID == "" {
		// Cannot fetch if we don't have an ID. The main reconcile loop handles creation.
		return nil, nil
	}

	externalDB, err := r.Cloud.GetDatabase(ctx, cr.Status.ProviderID)
	if err != nil {
		// Do not cache errors, as they might be transient
		return nil, err
	}

	// 3. Populate cache on successful fetch
	if externalDB != nil {
		r.Cache.Set(cacheKey, externalDB, cache.DefaultExpiration)
		logger.Info("Cache populated for external database.")
	}

	return externalDB, nil
}

Step 3: Update the Reconcile Loop to Use the Cache

Now, we replace the direct API call with our new cached function.

// Inside the Reconcile function

// ... (after fetching the CR instance)

// 2. Fetch the actual state of the external resource using the cache
externalDB, err := r.getCachedExternalDatabase(ctx, &db)
if err != nil && !cloudprovider.IsNotFound(err) {
	logger.Error(err, "failed to get database from cloud provider")
	return ctrl.Result{RequeueAfter: 30 * time.Second}, err
}

// ... (rest of the logic remains the same)

We also need to be strategic about invalidating the cache. When we perform an action that changes the external state (CreateDatabase or UpdateDatabase), we should immediately invalidate or update the cache entry to avoid serving stale data.

// In the 'Create' case
// ... after successfully calling r.Cloud.CreateDatabase
		cacheKey := client.ObjectKeyFromObject(&db).String()
		r.Cache.Set(cacheKey, newDB, cache.DefaultExpiration)
		logger.Info("Cache updated after database creation.")

// In the 'Update' case
// ... after successfully calling r.Cloud.UpdateDatabase
		cacheKey := client.ObjectKeyFromObject(&db).String()
		r.Cache.Set(cacheKey, updatedDB, cache.DefaultExpiration)
		logger.Info("Cache updated after database modification.")

// In the 'Delete' case
// ... after successfully calling r.Cloud.DeleteDatabase
		cacheKey := client.ObjectKeyFromObject(db).String()
		r.Cache.Delete(cacheKey)
		logger.Info("Cache entry deleted.")

Performance Impact and Considerations

With this caching layer, for a stable CR, the reconciliation loop becomes a near-instantaneous, local operation. The controller only makes an external API call once every cache TTL (e.g., 5 minutes) to re-verify the state and check for external drift.

Benchmark: In a cluster with 1,000 ExternalDatabase objects, without caching, the operator would perform 1,000 GetDatabase calls every reconciliation cycle (which can be frequent due to unrelated status updates). With a 5-minute cache, it performs 1,000 calls over 5 minutes, averaging ~3.3 calls per second, a massive reduction in API load.

Edge Case: What if a user modifies the external resource directly via the cloud console? Our cache will be stale until the TTL expires. This is a fundamental trade-off. The cache TTL must be chosen based on how critical it is to detect out-of-band changes. For most use cases, a 1-5 minute TTL is a reasonable balance between performance and consistency.

Section 4: The Complete Production-Grade Reconciler

Let's assemble all these concepts into a single, robust Reconcile function. This version incorporates:

Finalizer registration and handling.
State-driven, idempotent logic for create/update.
The external resource cache.

Proper error handling with exponential backoff (RequeueAfter).

// controllers/externaldatabase_controller.go

package controllers

import (
	"context"
	"fmt"
	"time"

	"github.com/patrickmn/go-cache"
	crd "my.domain/externaldb/api/v1"
	"my.domain/externaldb/internal/cloudprovider"

	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
)

const externalDatabaseFinalizer = "db.my.domain/finalizer"

type ExternalDatabaseReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	Cloud  *cloudprovider.Client
	Cache  *cache.Cache
}

//+kubebuilder:rbac:groups=crd.my.domain,resources=externaldatabases,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=crd.my.domain,resources=externaldatabases/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=crd.my.domain,resources=externaldatabases/finalizers,verbs=update

func (r *ExternalDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the ExternalDatabase CR instance
	var db crd.ExternalDatabase
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		if errors.IsNotFound(err) {
			logger.Info("ExternalDatabase resource not found. Ignoring since object must be deleted")
			return ctrl.Result{}, nil
		}
		logger.Error(err, "unable to fetch ExternalDatabase")
		return ctrl.Result{}, err
	}

	// 2. Handle Finalizer Logic
	if db.ObjectMeta.DeletionTimestamp.IsZero() {
		if !controllerutil.ContainsFinalizer(&db, externalDatabaseFinalizer) {
			logger.Info("Adding finalizer to the ExternalDatabase")
			controllerutil.AddFinalizer(&db, externalDatabaseFinalizer)
			if err := r.Update(ctx, &db); err != nil {
				return ctrl.Result{}, err
			}
			return ctrl.Result{Requeue: true}, nil
		}
	} else {
		if controllerutil.ContainsFinalizer(&db, externalDatabaseFinalizer) {
			if err := r.handleDelete(ctx, &db); err != nil {
				return ctrl.Result{RequeueAfter: 30 * time.Second}, err
			}
			controllerutil.RemoveFinalizer(&db, externalDatabaseFinalizer)
			if err := r.Update(ctx, &db); err != nil {
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, nil
	}

	// 3. Main Reconciliation Logic
	externalDB, err := r.getCachedExternalDatabase(ctx, &db)
	if err != nil {
		logger.Error(err, "failed to get database from cloud provider")
		return ctrl.Result{RequeueAfter: 30 * time.Second}, err
	}

	if externalDB == nil {
		// Create case
		return r.handleCreate(ctx, &db)
	}

	// Update/Sync case
	return r.handleUpdate(ctx, &db, externalDB)
}

func (r *ExternalDatabaseReconciler) handleCreate(ctx context.Context, db *crd.ExternalDatabase) (ctrl.Result, error) {
	logger := log.FromContext(ctx, "action", "create")
	logger.Info("Creating a new external database.")

	newDB, err := r.Cloud.CreateDatabase(ctx, &cloudprovider.DatabaseSpec{
		Name:    db.Name,
		Engine:  db.Spec.Engine,
		Version: db.Spec.Version,
		SizeGB:  db.Spec.SizeGB,
	})
	if err != nil {
		logger.Error(err, "failed to create external database")
		db.Status.State = "Failed"
		_ = r.Status().Update(ctx, db)
		return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
	}

	db.Status.ProviderID = newDB.ID
	db.Status.State = newDB.State
	db.Status.Endpoint = newDB.Endpoint
	if err := r.Status().Update(ctx, db); err != nil {
		return ctrl.Result{Requeue: true}, err
	}

	r.Cache.Set(client.ObjectKeyFromObject(db).String(), newDB, cache.DefaultExpiration)
	return ctrl.Result{}, nil
}

func (r *ExternalDatabaseReconciler) handleUpdate(ctx context.Context, db *crd.ExternalDatabase, externalDB *cloudprovider.Database) (ctrl.Result, error) {
	logger := log.FromContext(ctx, "action", "update/sync")

	// Check for spec drift
	if externalDB.SizeGB != db.Spec.SizeGB || externalDB.Version != db.Spec.Version {
		logger.Info("Drift detected. Updating external database.")
		updatedDB, err := r.Cloud.UpdateDatabase(ctx, externalDB.ID, &cloudprovider.DatabaseSpec{
			SizeGB:  db.Spec.SizeGB,
			Version: db.Spec.Version,
		})
		if err != nil {
			return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
		}
		externalDB = updatedDB // Use the updated state for status sync
		r.Cache.Set(client.ObjectKeyFromObject(db).String(), externalDB, cache.DefaultExpiration)
	}

	// Sync status regardless of drift
	if db.Status.State != externalDB.State || db.Status.Endpoint != externalDB.Endpoint {
		db.Status.State = externalDB.State
		db.Status.Endpoint = externalDB.Endpoint
		if err := r.Status().Update(ctx, db); err != nil {
			return ctrl.Result{Requeue: true}, err
		}
	}

	logger.Info("Reconciliation complete.")
	return ctrl.Result{}, nil
}

func (r *ExternalDatabaseReconciler) handleDelete(ctx context.Context, db *crd.ExternalDatabase) error {
	logger := log.FromContext(ctx, "action", "delete")

	if db.Status.ProviderID == "" {
		logger.Info("ProviderID is empty, nothing to clean up.")
		return nil
	}

	logger.Info("Deleting external database", "ProviderID", db.Status.ProviderID)
	if err := r.Cloud.DeleteDatabase(ctx, db.Status.ProviderID); err != nil && !cloudprovider.IsNotFound(err) {
		return fmt.Errorf("failed to delete external database %s: %w", db.Status.ProviderID, err)
	}

	r.Cache.Delete(client.ObjectKeyFromObject(db).String())
	logger.Info("Successfully deleted external database.")
	return nil
}

// ... SetupWithManager function remains the same

Final Thoughts

Building a Kubernetes operator that manages external, stateful resources is a complex task that extends far beyond the boilerplate generated by tools like Kubebuilder. The patterns discussed here—state-driven idempotency, finalizers for lifecycle management, and external resource caching—are not optional extras; they are fundamental requirements for creating an operator that is reliable, efficient, and safe to run in a production environment. By moving from a simple, command-oriented reconciler to a sophisticated, state-aware system, you build controllers that truly embody the robust, self-healing principles of Kubernetes itself.