Kubernetes CRD Controllers: Finalizers & Webhook Validation

Beyond the Basic Reconcile Loop: Production-Grade CRD Controllers

If you're reading this, you've likely moved past the "Hello, World" of Kubernetes operators. You understand the basic reconciliation loop: watch a Custom Resource (CR), check its state, and take action to move the world towards the desired state described in the CR's spec. This is the foundation, but it's dangerously insufficient for production systems that manage stateful or external resources.

Two critical questions quickly emerge when a controller graduates from managing simple, ephemeral resources:

Attempting to solve these problems within the core reconciliation loop leads to brittle, complex, and error-prone code. The Kubernetes API machinery provides two powerful, purpose-built mechanisms to solve these problems elegantly: Finalizers and Validating Admission Webhooks. This article is a deep dive into the advanced implementation of both, focusing on production patterns, edge cases, and performance considerations for senior engineers building mission-critical operators.

Part 1: The Deletion Problem & Finalizers Deep Dive

A standard Kubernetes delete operation is asynchronous. When you issue a kubectl delete, the API server first performs a "soft delete." It sets the metadata.deletionTimestamp field on the object to the current time. The object is now in a terminating state but is not yet removed from etcd. It's the controller's responsibility to notice this timestamp, perform any necessary cleanup, and then signal to the API server that the cleanup is complete.

This signal is the Finalizer. A finalizer is simply a string key in the metadata.finalizers array. As long as this array is not empty, the Kubernetes garbage collector will not fully delete the object, even if its deletionTimestamp is set. This gives your controller a hook to execute pre-delete logic.

The Finalizer Reconciliation Pattern

The pattern is a critical modification to your standard Reconcile function:

Code Example: Implementing a Finalizer in Go

Let's consider a ManagedDatabase CRD. When a ManagedDatabase object is deleted, we must ensure the corresponding database in an external cloud service is de-provisioned.

We'll use the controller-runtime library, standard in Kubebuilder and Operator-SDK projects. Our finalizer will be a unique string, typically in the format my-domain.com/finalizer.

// controllers/manageddatabase_controller.go

package controllers

import (
	"context"
	"fmt"

	"github.com/go-logr/logr"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

	appv1alpha1 "my.domain/api/v1alpha1"
)

const managedDatabaseFinalizer = "db.my.domain/finalizer"

// A mock external database client
type ExternalDBClient struct {}
func (c *ExternalDBClient) DeleteDatabase(id string) error {
    // In a real implementation, this would make an API call
    // to a cloud provider like AWS RDS, GCP Cloud SQL, etc.
    fmt.Printf("DELETING external database with ID: %s\n", id)
    // Simulate a potentially failing operation
    // if id == "fail-cleanup" { return fmt.Errorf("API call failed") }
    return nil
}

// ManagedDatabaseReconciler reconciles a ManagedDatabase object
type ManagedDatabaseReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
    ExternalClient *ExternalDBClient // Our mock client
}

func (r *ManagedDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := r.Log.WithValues("manageddatabase", req.NamespacedName)

	// 1. Fetch the ManagedDatabase instance
	instance := &appv1alpha1.ManagedDatabase{}
	if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
		// Ignore not-found errors, as they can't be fixed by an immediate requeue.
		// They could be transient or the object may have been deleted.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Examine if the object is under deletion
	isMarkedForDeletion := instance.GetDeletionTimestamp() != nil
	if isMarkedForDeletion {
		if controllerutil.ContainsFinalizer(instance, managedDatabaseFinalizer) {
			// Run our finalization logic. If it fails, we return the error
			// which will cause the reconciliation to be re-queued.
			if err := r.finalizeManagedDatabase(log, instance); err != nil {
				log.Error(err, "Failed to finalize ManagedDatabase")
				return ctrl.Result{}, err
			}

			// Cleanup was successful. Remove our finalizer from the list and update it.
			log.Info("External database deleted, removing finalizer")
			controllerutil.RemoveFinalizer(instance, managedDatabaseFinalizer)
			if err := r.Update(ctx, instance); err != nil {
				return ctrl.Result{}, err
			}
		}
		// Stop reconciliation as the item is being deleted
		return ctrl.Result{}, nil
	}

	// 3. Add finalizer for a new object if it doesn't exist
	if !controllerutil.ContainsFinalizer(instance, managedDatabaseFinalizer) {
		log.Info("Adding finalizer for ManagedDatabase")
		controllerutil.AddFinalizer(instance, managedDatabaseFinalizer)
		if err := r.Update(ctx, instance); err != nil {
			return ctrl.Result{}, err
		}
	}

	// ... your normal reconciliation logic to create/update the external database ...
	log.Info("Reconciling ManagedDatabase normally")

	return ctrl.Result{}, nil
}

func (r *ManagedDatabaseReconciler) finalizeManagedDatabase(log logr.Logger, db *appv1alpha1.ManagedDatabase) error {
	// This is where you put your cleanup logic.
	// The database ID might be stored in the CR's status field.
	dbID := db.Status.DatabaseID
	if dbID == "" {
		log.Info("External database ID not found in status, assuming it was never created.")
		return nil
	}

	log.Info("Starting finalization logic: deleting external database", "DatabaseID", dbID)
	if err := r.ExternalClient.DeleteDatabase(dbID); err != nil {
		// Important: If the external deletion fails, we return an error.
		// This prevents the finalizer from being removed, and the reconciliation
		// will be retried.
		return fmt.Errorf("failed to delete external database %s: %w", dbID, err)
	}

	log.Info("Successfully deleted external database", "DatabaseID", dbID)
	return nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *ManagedDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&appv1alpha1.ManagedDatabase{}).
		Complete(r)
}

Advanced Edge Cases and Production Considerations

Idempotency is Non-Negotiable: Your finalize function will* be called multiple times if it returns an error or if the controller restarts. The cleanup logic must be idempotent. If you call a cloud API to delete a resource, that API should gracefully handle being called on an already-deleted resource. If it doesn't, your code must check for the resource's existence before attempting deletion.

Controller Crash During Finalization: The most critical race condition is a crash after successful cleanup but before* the r.Update(ctx, instance) call that removes the finalizer. On restart, the controller will see the object is still terminating and will re-run the finalize function. This is another reason for strict idempotency. Without it, you could attempt to delete a non-existent resource, causing an error and getting the object stuck in a Terminating state forever.

Handling Unrecoverable Cleanup Failure: What if the external API is permanently down or the credentials have expired? The object will be stuck terminating. This is often the desired* behavior, as it forces an operator to investigate the failure instead of leaving orphaned resources. The fix requires manual intervention: fix the external issue or, in a true disaster recovery scenario, manually patch the object to remove the finalizer: kubectl patch manageddatabase my-db --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'.

Finalizer Ownership: If multiple controllers act on the same object, they should each use their own uniquely named finalizer. A controller should never* remove a finalizer it does not own.

Part 2: The Validation Problem & Admission Webhooks

OpenAPI schema validation is powerful for static checks, but it's fundamentally limited. It cannot:

* Validate a field's value against another field (e.g., endDate must be after startDate).

* Enforce immutability (e.g., spec.storageClass cannot be changed after creation).

* Validate against external state (e.g., spec.userName must not already exist in an external directory).

* Provide user-friendly, context-aware error messages.

Validating Admission Webhooks solve this by intercepting requests to the Kubernetes API server before they are persisted to etcd. Your webhook is a simple HTTPS endpoint that receives an AdmissionReview object, performs its validation logic, and returns a response indicating whether the request should be allowed or denied.

The Admission Webhook Flow

• The API server receives the response. If denied, it rejects the original request with the error message you provided.

Code Example: A Validating Webhook Server in Go

Let's add validation to our ManagedDatabase CRD. We want to enforce two rules:

We'll build a minimal webhook server. In a real project, Kubebuilder can scaffold this for you.

// webhooks/database_validator.go

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"

	admissionv1 "k8s.io/api/admission/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/serializer"

	appv1alpha1 "my.domain/api/v1alpha1"
)

var (
	universalDeserializer = serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer()
)

// admitFunc is the type for our validation logic
type admitFunc func(admissionv1.AdmissionReview) *admissionv1.AdmissionResponse

// admitHandler is a helper to wrap our validation logic
func admitHandler(admit admitFunc) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		body, err := ioutil.ReadAll(r.Body)
		defer r.Body.Close()
		if err != nil {
			http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)
			return
		}

		var admissionReviewReq admissionv1.AdmissionReview
		if _, _, err := universalDeserializer.Decode(body, nil, &admissionReviewReq); err != nil {
			http.Error(w, fmt.Sprintf("could not deserialize request: %v", err), http.StatusBadRequest)
			return
		}

		admissionReviewResponse := admit(admissionReviewReq)

		response := admissionv1.AdmissionReview{
			TypeMeta: metav1.TypeMeta{
				Kind:       "AdmissionReview",
				APIVersion: "admission.k8s.io/v1",
			},
			Response: admissionReviewResponse,
		}
		response.Response.UID = admissionReviewReq.Request.UID

		respBytes, err := json.Marshal(response)
		if err != nil {
			http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError)
			return
		}

		w.Header().Set("Content-Type", "application/json")
		w.Write(respBytes)
	})
}

// validateManagedDatabase is our core logic
func validateManagedDatabase(ar admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
	raw := ar.Request.Object.Raw
	db := appv1alpha1.ManagedDatabase{}
	if _, _, err := universalDeserializer.Decode(raw, nil, &db); err != nil {
		return &admissionv1.AdmissionResponse{
			Result: &metav1.Status{
				Message: err.Error(),
			},
		}
	}

	// Rule 1: Validate engine type
	if db.Spec.Engine != "postgres" && db.Spec.Engine != "mysql" {
		return &admissionv1.AdmissionResponse{
			Allowed: false,
			Result: &metav1.Status{
				Message: fmt.Sprintf("invalid engine '%s'. Must be 'postgres' or 'mysql'", db.Spec.Engine),
			},
		}
	}

	// Rule 2: Prevent decreasing storage size on UPDATE
	if ar.Request.Operation == admissionv1.Update {
		oldRaw := ar.Request.OldObject.Raw
		oldDb := appv1alpha1.ManagedDatabase{}
		if _, _, err := universalDeserializer.Decode(oldRaw, nil, &oldDb); err != nil {
			return &admissionv1.AdmissionResponse{
				Result: &metav1.Status{Message: err.Error()},
			}
		}

		if db.Spec.StorageGB < oldDb.Spec.StorageGB {
			return &admissionv1.AdmissionResponse{
				Allowed: false,
				Result: &metav1.Status{
					Message: fmt.Sprintf("storageGB cannot be decreased. Current: %d, Requested: %d", oldDb.Spec.StorageGB, db.Spec.StorageGB),
				},
			}
		}
	}

	// If all checks pass
	return &admissionv1.AdmissionResponse{Allowed: true}
}

func main() {
	appv1alpha1.AddToScheme(runtime.NewScheme()) // Needed for deserializer

	http.Handle("/validate", admitHandler(validateManagedDatabase))
	// IMPORTANT: In production, you must use TLS
	// http.ListenAndServeTLS(":8443", "/certs/tls.crt", "/certs/tls.key", nil)
    fmt.Println("Listening on :8080...")
	http.ListenAndServe(":8080", nil) // For simplicity, no TLS in this example
}

Deployment and Configuration

To make this work, you need two more Kubernetes objects:

# webhook-config.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: manageddatabase-validation-webhook
webhooks:
  - name: validation.db.my.domain
    clientConfig:
      # In a real cluster, this would point to a service.
      # For local testing with kubectl proxy, you can use a direct URL.
      # service:
      #   name: manageddatabase-webhook-svc
      #   namespace: my-operator-system
      #   path: "/validate"
      #   port: 443
      # IMPORTANT: caBundle must be the CA that signed the webhook server's cert.
      # caBundle: LS0t...=
      url: "http://127.0.0.1:8080/validate" # ONLY FOR LOCAL DEV
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["db.my.domain"]
        apiVersions: ["v1alpha1"]
        resources: ["manageddatabases"]
    # CRITICAL production setting
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1"]

Performance and Reliability Considerations

* failurePolicy is Critical: failurePolicy: Fail (the default) means that if the API server cannot reach your webhook (e.g., your webhook pod is down, network issues), the API call will be rejected. This guarantees validation but can impact cluster availability if your webhook is not highly available. failurePolicy: Ignore bypasses the webhook on failure, which preserves availability but sacrifices validation guarantees. For critical validation, always use Fail and ensure your webhook is deployed with multiple replicas.

* Latency Impact: Every webhook adds latency to the API request path. Your validation logic must be fast. Avoid slow network calls to external services. If you must validate against external state, use aggressive caching. Set a reasonable timeoutSeconds in your webhook configuration (e.g., 2-5 seconds) to prevent hung API requests.

Certificate Management: The API server must* trust the TLS certificate presented by your webhook server. Manually managing these certificates is a common source of production outages. The standard production pattern is to use a tool like cert-manager to automatically provision and rotate the certificates for your webhook service and inject the corresponding CA bundle into the ValidatingWebhookConfiguration.

* Avoid Re-entrancy: A webhook should never trigger an API call that would, in turn, invoke the same webhook. For example, don't have your ManagedDatabase webhook update a ConfigMap if you also have a webhook that validates ConfigMap updates. This can lead to infinite loops.

Part 3: Tying It All Together: A Resilient Controller

Finalizers and webhooks are not independent gadgets; they are two pillars of a robust resource lifecycle management strategy. They work in tandem to ensure a resource is both valid upon creation and cleaned up gracefully upon deletion.

Consider our ManagedDatabase CR:

* A user submits a ManagedDatabase manifest.

* The Validating Webhook intercepts it. It checks that spec.engine is valid and spec.storageGB is above a minimum threshold. If not, the request is rejected immediately. The controller never even sees it.

* The request is approved and persisted to etcd.

* The Controller's Reconcile loop is triggered. It sees a new object with no deletionTimestamp.

* It adds the db.my.domain/finalizer to the object and updates it.

* On the next reconcile, it sees the finalizer is present and proceeds to provision the external database.

* A user runs kubectl delete manageddatabase/my-db.

* The API server sets the deletionTimestamp.

* The Controller's Reconcile loop is triggered. It sees the deletionTimestamp is set and its finalizer is present.

* It executes its finalizeManagedDatabase logic, calling the external API to de-provision the database.

* Upon successful de-provisioning, it removes its finalizer and updates the object.

* The API server, seeing the finalizer list is now empty, garbage collects the object from etcd.

This combination ensures that invalid resource specifications are rejected upfront, and for every valid resource that successfully gets created, its external counterpart is guaranteed to be cleaned up. This prevents both invalid states and orphaned resources, forming the core of a production-grade, reliable Kubernetes operator.