Idempotent Mutating Webhooks in K8s for Sidecar Injection

October 8, 2025

14 min read

Goh Ling Yong

Technology enthusiast and software architect specializing in AI-driven development tools and modern software engineering practices. Passionate about the intersection of artificial intelligence and human creativity in building tomorrow's digital solutions.

The Idempotency Imperative in Mutating Admission Webhooks

In a mature Kubernetes environment, mutating admission webhooks are a powerful mechanism for enforcing policy, injecting configurations, and automating operational concerns. A common use case is the automatic injection of a sidecar container—for service mesh proxies, logging agents, or security scanners. However, a naive implementation that simply checks for the absence of a sidecar and adds it will inevitably fail in a production cluster. The core challenge is idempotency.

An operation is idempotent if applying it multiple times produces the same result as applying it once. A mutating webhook is not invoked just once on CREATE. It can be invoked on UPDATE operations triggered by other controllers (e.g., a scheduler updating a node name, or a Horizontal Pod Autoscaler updating an annotation). If your webhook re-applies its logic on every UPDATE, you can trigger a reconciliation loop where your webhook's change triggers another controller, which in turn triggers your webhook again. This leads to API server overload and deployment failures.

This article details the design and implementation of a production-grade, idempotent mutating webhook in Go. We will focus on a state-tracking mechanism using annotations and the correct use of JSON Patch (RFC 6902) to communicate mutations, ensuring our webhook behaves predictably and safely, even when re-invoked.

Why Simple Logic Fails

Consider this naive logic:

Receive AdmissionReview for a Pod.

Iterate through pod.Spec.Containers.

If my-sidecar container is not found, add it.

Return the modified Pod object.

This fails for several reasons:

* Reconciliation Loops: An UPDATE on the Pod for an unrelated reason (e.g., a label change by a CI/CD pipeline) triggers the webhook. The webhook sees the sidecar is already there, but if the logic isn't perfectly clean, it might try to re-add or modify it, creating a new UPDATE event.

* Patch Conflicts: On an UPDATE, the API server expects a patch. If you return the entire modified Pod object, you risk overwriting changes made by another actor between the time you received the object and the time your modification is applied.

* Re-invocation Policy: The MutatingWebhookConfiguration has a reinvocationPolicy. If set to IfNeeded, the API server may re-invoke your webhook after other webhooks have run. Your logic must be resilient to being called multiple times within the same admission request lifecycle.

Our solution will address these issues head-on.

Core Strategy: Annotation-Driven State Tracking

The most robust pattern for achieving idempotency is to use the object's own metadata to track the state of our mutation. We will use annotations to serve as a control flag.

Status Annotation: We'll define an annotation like injector.my-company.com/status: injected. When our webhook successfully injects the sidecar, it will also add this annotation to the Pod.

Version Annotation: To handle future upgrades of the sidecar, we'll use a version annotation, e.g., injector.my-company.com/version: "1.2.1".

Our webhook's core logic becomes a simple state machine:

text

func handleAdmission(pod *corev1.Pod) -> patch {
  annotations := pod.GetAnnotations()

  if annotations["injector.my-company.com/status"] == "injected" {
    // Already injected. Check if an upgrade is needed.
    if annotations["injector.my-company.com/version"] == currentSidecarVersion {
      // Correct version is injected. Do nothing.
      return nil
    } else {
      // Version mismatch. Generate a patch to upgrade the sidecar.
      return createUpgradePatch(pod)
    }
  } else {
    // Not injected yet. Generate a patch to add the sidecar and annotations.
    return createInitialInjectionPatch(pod)
  }
}

This approach is inherently idempotent. On subsequent invocations for an already-injected Pod, the webhook immediately sees the injected status and exits, returning no patch and causing no UPDATE.

Building the Webhook Server in Go

Let's implement the webhook server. We will use the standard net/http library and Kubernetes Go client libraries for type definitions.

Project Structure:

text

/mutating-webhook
├── go.mod
├── go.sum
├── main.go
├── Dockerfile
├── /pkg
│   └── webhook
│       └── webhook.go
└── /deployment
    ├── deployment.yaml
    ├── service.yaml
    ├── webhook-config.yaml
    └── cert-manager.yaml

main.go:

package main

import (
	"crypto/tls"
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/your-org/mutating-webhook/pkg/webhook"
)

func main() {
	certPath := os.Getenv("TLS_CERT_PATH")
	keyPath := os.Getenv("TLS_KEY_PATH")
	if certPath == "" || keyPath == "" {
		log.Fatal("TLS_CERT_PATH and TLS_KEY_PATH must be set")
	}

	// Load TLS certificates
	cert, err := tls.LoadX509KeyPair(certPath, keyPath)
	if err != nil {
		log.Fatalf("Failed to load key pair: %v", err)
	}

	whServer := webhook.NewServer()

	http.HandleFunc("/mutate", whServer.HandleMutate)

	server := &http.Server{
		Addr:      ":8443",
		TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
	}

	log.Println("Starting webhook server on :8443")
	if err := server.ListenAndServeTLS("", ""); err != nil {
		log.Fatalf("Failed to start server: %v", err)
	}
}

This sets up a basic HTTPS server, which is a requirement for admission webhooks. The certificate paths will be mounted from a Kubernetes Secret, which we'll manage later with cert-manager.

Advanced Patch Generation with `jsonpatch`

Returning the entire modified object in the AdmissionResponse is bad practice. It's inefficient and can lead to race conditions. The correct method is to return a jsonpatch array, which describes the precise changes.

We will use the mattbaird/jsonpatch library to generate these patches.

pkg/webhook/webhook.go (Initial Structure):

package webhook

import (
	"encoding/json"
	"io/ioutil"
	"log"
	"net/http"

	jsonpatch "github.com/evanphx/json-patch"
	"k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/serializer"
)

const (
	annStatus  = "injector.my-company.com/status"
	annVersion = "injector.my-company.com/version"

	sidecarVersion = "1.2.1"
)

var (
	universalDeserializer = serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer()
)

type Server struct{}

func NewServer() *Server {
	return &Server{}
}

// HandleMutate is the main entry point for the webhook
func (s *Server) HandleMutate(w http.ResponseWriter, r *http.Request) {
	// 1. Read and decode the AdmissionReview request
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		log.Printf("Error reading request body: %v", err)
		http.Error(w, "bad request", http.StatusBadRequest)
		return
	}

	var admissionReview v1.AdmissionReview
	if _, _, err := universalDeserializer.Decode(body, nil, &admissionReview); err != nil {
		log.Printf("Error decoding admission review: %v", err)
		http.Error(w, "invalid request body", http.StatusBadRequest)
		return
	}

	if admissionReview.Request == nil {
		http.Error(w, "invalid request: missing request body", http.StatusBadRequest)
		return
	}

	// 2. Unmarshal the Pod from the AdmissionRequest
	pod := &corev1.Pod{}
	if err := json.Unmarshal(admissionReview.Request.Object.Raw, pod); err != nil {
		log.Printf("Error unmarshaling pod: %v", err)
		http.Error(w, "failed to unmarshal pod", http.StatusBadRequest)
		return
	}

	// 3. Generate the patch (logic to be implemented)
	patchBytes, err := s.createPatch(pod)
	if err != nil {
		log.Printf("Error creating patch: %v", err)
		http.Error(w, "internal server error", http.StatusInternalServerError)
		return
	}

	// 4. Create the AdmissionResponse
	admissionResponse := &v1.AdmissionResponse{
		UID:     admissionReview.Request.UID,
		Allowed: true,
	}

	if patchBytes != nil {
		admissionResponse.Patch = patchBytes
		patchType := v1.PatchTypeJSONPatch
		admissionResponse.PatchType = &patchType
	}

	// 5. Construct the final AdmissionReview and send response
	responseReview := v1.AdmissionReview{
		TypeMeta: metav1.TypeMeta{
			APIVersion: "admission.k8s.io/v1",
			Kind:       "AdmissionReview",
		},
		Response: admissionResponse,
	}

	respBytes, err := json.Marshal(responseReview)
	if err != nil {
		log.Printf("Error marshalling response: %v", err)
		http.Error(w, "internal server error", http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	w.Write(respBytes)
}

// createPatch is where the core idempotent logic resides
func (s *Server) createPatch(pod *corev1.Pod) ([]byte, error) {
	// Implementation in the next section
	return nil, nil
}

The Idempotency Logic in Detail

Now we implement the createPatch function. This function will contain our core state machine logic.

// pkg/webhook/webhook.go (continued)

func (s *Server) createPatch(pod *corev1.Pod) ([]byte, error) {
	originalData, err := json.Marshal(pod)
	if err != nil {
		return nil, fmt.Errorf("failed to marshal original pod: %w", err)
	}

	// Create a deep copy to modify
	modifiedPod := pod.DeepCopy()

	annotations := modifiedPod.GetAnnotations()
	if annotations == nil {
		annotations = make(map[string]string)
	}

	// The core idempotency check
	if annotations[annStatus] == "injected" && annotations[annVersion] == sidecarVersion {
		log.Printf("Pod %s/%s already has the correct sidecar version. Skipping.", pod.Namespace, pod.Name)
		return nil, nil // Return empty patch, no changes needed
	}

	// Perform the mutation
	log.Printf("Injecting/updating sidecar for Pod %s/%s", pod.Namespace, pod.Name)
	
	// Add or update annotations
	annotations[annStatus] = "injected"
	annotations[annVersion] = sidecarVersion
	modifiedPod.SetAnnotations(annotations)

	// Add the sidecar container
	sidecar := corev1.Container{
		Name:  "my-sidecar",
		Image: fmt.Sprintf("my-org/my-sidecar:%s", sidecarVersion),
		Ports: []corev1.ContainerPort{{
			ContainerPort: 8080,
			Name:          "http",
		}},
		Resources: corev1.ResourceRequirements{
			Limits:   corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("64Mi")},
			Requests: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("50m"), corev1.ResourceMemory: resource.MustParse("32Mi")},
		},
	}

	// If the container already exists (upgrade scenario), replace it. Otherwise, add it.
	found := false
	for i, container := range modifiedPod.Spec.Containers {
		if container.Name == "my-sidecar" {
			modifiedPod.Spec.Containers[i] = sidecar
			found = true
			break
		}
	}
	if !found {
		modifiedPod.Spec.Containers = append(modifiedPod.Spec.Containers, sidecar)
	}

	modifiedData, err := json.Marshal(modifiedPod)
	if err != nil {
		return nil, fmt.Errorf("failed to marshal modified pod: %w", err)
	}

	// Generate the JSON patch
	patch, err := jsonpatch.CreateMergePatch(originalData, modifiedData)
	if err != nil {
		return nil, fmt.Errorf("failed to create json merge patch: %w", err)
	}

	return patch, nil
}

_Note: For simplicity, I've used CreateMergePatch. In a real-world scenario with complex list manipulations, you might need to construct a more explicit RFC 6902 patch with add, replace, remove operations for finer control, especially when dealing with multiple sidecars or complex volume mounts._

Edge Cases and Production Hardening

Senior engineering is about handling the edge cases.

API Server Retries & Re-invocation: Our annotation-based approach handles this seamlessly. If the API server invokes our webhook, experiences a network blip, and re-invokes, the second invocation will see the state from the original object. Our logic will produce the exact same patch. If the first invocation did* succeed and this is a re-invocation due to another webhook, our annStatus check prevents any further mutation.

* Webhook Versioning and Sidecar Upgrades: The annVersion check is crucial. When you need to roll out a new sidecar version (e.g., 1.2.2), you update the sidecarVersion constant in your webhook code and deploy it. The next time any Pod with an older version (1.2.1) is updated for any reason, the webhook will trigger. The logic annotations[annVersion] == currentSidecarVersion will fail, and the webhook will generate a patch that replaces the existing sidecar container with the new image and updates the version annotation. This provides a powerful, rolling upgrade mechanism.

* Object Deletion: Our current code doesn't check the admissionReview.Request.Operation. For a DELETE operation, we should short-circuit and do nothing. Add this at the beginning of HandleMutate:

    if admissionReview.Request.Operation == v1.Delete {
        // ... create and send an empty allowed response ...
        return
    }

* Concurrency: Our Go HTTP handler is inherently concurrent. However, because our logic is stateless—it only depends on the input AdmissionReview and has no global state—it is safe for concurrent execution without locks or other synchronization primitives.

Deployment and TLS Certificate Management

Manually managing TLS certificates for webhooks is a common source of production outages. We will use cert-manager to automate this entirely.

1. Install cert-manager:

bash

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

2. Create an Issuer:

This Issuer will create a self-signed CA and issue certificates from it. The API server will be configured to trust this CA.

deployment/cert-manager.yaml:

yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sidecar-injector-certs
  namespace: default
spec:
  secretName: sidecar-injector-tls
  dnsNames:
  - sidecar-injector-svc.default.svc
  - sidecar-injector-svc.default.svc.cluster.local
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer

Applying this will create a Secret named sidecar-injector-tls containing tls.crt, tls.key, and ca.crt.

3. Webhook Deployment and Service:

deployment/deployment.yaml:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidecar-injector-deployment
  labels:
    app: sidecar-injector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sidecar-injector
  template:
    metadata:
      labels:
        app: sidecar-injector
    spec:
      containers:
      - name: webhook
        image: your-org/mutating-webhook:latest
        ports:
        - containerPort: 8443
          name: webhook-tls
        env:
        - name: TLS_CERT_PATH
          value: /etc/webhook/certs/tls.crt
        - name: TLS_KEY_PATH
          value: /etc/webhook/certs/tls.key
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
          readOnly: true
      volumes:
      - name: webhook-certs
        secret:
          secretName: sidecar-injector-tls

deployment/service.yaml:

yaml

apiVersion: v1
kind: Service
metadata:
  name: sidecar-injector-svc
  namespace: default
spec:
  selector:
    app: sidecar-injector
  ports:
    - port: 443
      targetPort: webhook-tls

4. The MutatingWebhookConfiguration:

This is the critical resource that tells the API server to call our webhook. cert-manager helps us again by automatically injecting the caBundle.

deployment/webhook-config.yaml:

yaml

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector-webhook-config
  annotations:
    cert-manager.io/inject-ca-from: default/sidecar-injector-certs # This is key!
webhooks:
  - name: sidecar-injector.my-company.com
    clientConfig:
      service:
        name: sidecar-injector-svc
        namespace: default
        path: "/mutate"
      # caBundle will be populated by cert-manager
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    sideEffects: None
    admissionReviewVersions: ["v1"]
    failurePolicy: Fail # In production, consider 'Ignore' during initial rollout
    reinvocationPolicy: IfNeeded
    # Performance Optimization: Only select pods that opt-in
    objectSelector:
      matchLabels:
        sidecar-injection: "enabled"

Key Production Considerations in this manifest:

* cert-manager.io/inject-ca-from: This annotation instructs cert-manager to watch the sidecar-injector-certs Certificate and inject its CA public key into the caBundle field. This automates the entire trust relationship.

* failurePolicy: Fail: This is the safest option, as it prevents potentially misconfigured Pods from being created if the webhook is down. However, it also means a webhook outage can block all Pod creations in the cluster. Monitor your webhook's availability closely. Start with Ignore in development.

* reinvocationPolicy: IfNeeded: We explicitly support re-invocation, which is why our idempotent design is so important. It allows our webhook to play nicely with others.

* objectSelector: This is a critical performance optimization. It tells the API server to only send admission requests for Pods that have the label sidecar-injection: "enabled". This dramatically reduces the load on your webhook, as it won't be invoked for every single Pod created in the cluster (e.g., system components).

Performance and Scalability

A mutating webhook is in the critical path of resource creation. It must be fast and scalable.

* Metrics: Instrument the Go HandleMutate function with Prometheus metrics. Track the latency of patch creation (prometheus.NewHistogramVec) and the count of successful vs. failed requests (prometheus.NewCounterVec). This is non-negotiable for production monitoring.

* Resource Limits: Profile your webhook under load to determine appropriate CPU and memory requests/limits. A webhook that is constantly CPU-throttled or OOMKilled will bring down your cluster's control plane.

* Horizontal Pod Autoscaler (HPA): Deploy an HPA targeting the webhook deployment. Scale on CPU utilization (e.g., targetAverageUtilization: 75). This ensures you can handle bursts of activity, such as a large-scale deployment or cluster upgrade.

yaml

    apiVersion: autoscaling/v2
    kind: HorizontalPodAutoscaler
    metadata:
      name: sidecar-injector-hpa
    spec:
      scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: sidecar-injector-deployment
      minReplicas: 2
      maxReplicas: 5
      metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 75

Conclusion: A Resilient Pattern

Building a simple sidecar injector is easy. Building one that survives the chaos of a production Kubernetes cluster requires a disciplined approach to state management and API interaction. The annotation-driven, idempotent pattern detailed here is a robust and scalable solution.

By tracking mutation state directly on the object, generating precise JSON patches, and automating TLS certificate management, you create a system that is predictable, resilient to re-invocation, and safe to run in the critical path of your cluster's API server. This pattern extends far beyond sidecar injection and can be adapted for any scenario requiring default settings, policy enforcement, or automated resource modification, forming a cornerstone of a mature platform engineering strategy.

The Idempotency Imperative in Mutating Admission Webhooks

Why Simple Logic Fails

Core Strategy: Annotation-Driven State Tracking

Building the Webhook Server in Go

Advanced Patch Generation with `jsonpatch`

The Idempotency Logic in Detail

Edge Cases and Production Hardening

Deployment and TLS Certificate Management

Performance and Scalability

Conclusion: A Resilient Pattern

Found this article helpful?