eBPF for Sidecar-less Service Mesh Observability in Kubernetes

The Inherent Performance Tax of the Sidecar Pattern

For years, the sidecar proxy—epitomized by Envoy in service meshes like Istio and Linkerd—has been the de facto standard for injecting observability, security, and traffic management capabilities into microservices architectures. The pattern is powerful: it decouples application logic from network logic, enabling language-agnostic feature injection without modifying service code. However, this elegance comes at a non-trivial performance cost, a cost that becomes increasingly burdensome at scale.

Every packet destined for or originating from an application pod must traverse the user-space/kernel-space boundary multiple times and be processed by the sidecar proxy. Let's trace the path for a simple ingress request in a sidecar-based mesh:

The return path mirrors this journey. This round trip adds two extra user-space hops and multiple context switches, introducing measurable latency and consuming significant CPU and memory resources. For a high-throughput, low-latency service, this overhead can become the primary performance bottleneck, negating the benefits of a highly optimized application runtime.

This resource consumption isn't just theoretical. A typical Envoy sidecar can add 50-100m CPU and 50-100MB of RAM per pod replica. In a cluster with thousands of pods, this translates to dozens or even hundreds of dedicated nodes just to run the mesh infrastructure, representing a substantial operational cost.

This is the problem space where eBPF (extended Berkeley Packet Filter) emerges not as an incremental improvement, but as a paradigm shift for service mesh data planes.

eBPF: Programmable Kernel-Level Observability

eBPF allows us to run sandboxed programs within the Linux kernel itself, triggered by various events like system calls, network events, or kernel tracepoints. For service mesh observability, the most relevant hooks are:

* Traffic Control (TC) Hooks: eBPF programs can be attached to the TC ingress and egress hooks on a network interface (physical or virtual, like a veth pair). This allows for packet inspection, modification, and redirection at the earliest possible point in the networking stack.

* Socket Hooks (sock_ops): Programs can be attached to TCP events for a specific socket, enabling visibility into connection state and even accelerating socket-level operations.

* Kernel Probes (kprobes/kretprobes): These can be attached to almost any function in the kernel, allowing us to trace system calls like sendmsg, recvmsg, connect, and accept. This is the key to transparently understanding application-level network communication.

By leveraging these hooks, an eBPF-based agent running on each Kubernetes node can achieve the core observability functions of a service mesh without redirecting traffic through a user-space proxy. The data path is radically simplified.

Architectural Deep Dive: Sidecar vs. eBPF Agent

Let's contrast the data paths visually.

Sidecar Architecture:

          Pod
+-------------------------+
|   App    <-- localhost -->  Envoy   |
| Container     Proxy     |         
+----^--------------------+         
     |                              
     | Kernel TCP/IP Stack          
     | (veth pair)                  
+----v--------------------+         
|      Node Kernel        |
+-------------------------+

Traffic Flow: App <-> Kernel (Loopback) <-> Envoy <-> Kernel (veth) <-> External

eBPF Agent Architecture (e.g., Cilium):

          Pod
+-------------------------+
|      App Container      |
+----^--------------------+         
     |                              
     | Kernel TCP/IP Stack          
     | (veth pair with eBPF hooks)  
+----|--------------------+         
|    | Node Kernel        |         
|    v                    |
| eBPF Program <-- Cilium Agent   |
| (on veth)       (User Space)    |
+-------------------------+

Traffic Flow: App <-> Kernel (veth) <-> External

In the eBPF model, the Cilium agent (a user-space daemon) loads eBPF programs into the kernel. These programs are attached to the veth pair associated with the pod. When the application sends or receives data, the packet flows through the standard kernel TCP/IP stack. The attached eBPF program executes in-kernel, inspecting the packet headers and socket metadata to gather metrics, enforce network policies, and provide observability. The packet never leaves the kernel for proxying. The Cilium agent simply orchestrates these eBPF programs and collects aggregated data from them via eBPF maps.

The result is a dramatic reduction in per-packet overhead, leading to lower latency and reduced CPU consumption.

Production Implementation: Cilium and Hubble

Cilium is a CNI (Container Network Interface) plugin for Kubernetes that uses eBPF to provide networking, observability, and security. Let's walk through a production-grade setup to demonstrate these capabilities.

Step 1: Setting up a Kind Cluster with Cilium

For a reproducible local environment, we'll use kind. First, create a kind-config.yaml to disable the default CNI and allow for a custom pod CIDR.

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "10.1.0.0/16"
  serviceSubnet: "10.2.0.0/16"
nodes:
- role: control-plane
- role: worker
- role: worker

Create the cluster:

kind create cluster --config kind-config.yaml

Now, install the Cilium CLI and deploy Cilium to the cluster with Hubble (its observability component) enabled.

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

# Install Cilium and Hubble
cilium install --version 1.15.1 \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true

Verify the installation:

cilium status --wait
# Expected output shows Cilium and Hubble are healthy

Step 2: Deploying Sample Microservices

Let's deploy two simple Go microservices, api-service and db-service, to simulate a realistic interaction.

api-service exposes an endpoint /users/{id} and calls db-service to fetch data.

// api-service/main.go
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

func main() {
	dbServiceURL := os.Getenv("DB_SERVICE_URL")
	if dbServiceURL == "" {
		dbServiceURL = "http://db-service:8080"
	}

	http.HandleFunc("/users/", func(w http.ResponseWriter, r *http.Request) {
		userID := r.URL.Path[len("/users/"):]
		log.Printf("API service received request for user %s", userID)

		resp, err := http.Get(fmt.Sprintf("%s/data/%s", dbServiceURL, userID))
		if err != nil {
			http.Error(w, "Failed to call db-service", http.StatusInternalServerError)
			log.Printf("Error calling db-service: %v", err)
			return
		}
		defer resp.Body.Close()

		body, _ := io.ReadAll(resp.Body)
		log.Printf("API service received response from db-service: %s", string(body))
		w.Header().Set("Content-Type", "application/json")
		w.Write(body)
	})

	log.Println("API service starting on port 8080")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalf("Failed to start server: %v", err)
	}
}

db-service simulates a database lookup.

// db-service/main.go
package main

import (
	"encoding/json"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/data/", func(w http.ResponseWriter, r *http.Request) {
		userID := r.URL.Path[len("/data/"):]
		log.Printf("DB service processing request for user %s", userID)

		data := map[string]string{"id": userID, "name": fmt.Sprintf("User-%s", userID), "source": "database"}
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(data)
	})

	log.Println("DB service starting on port 8080")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalf("Failed to start server: %v", err)
	}
}

Now, package these into containers and deploy them to your Kubernetes cluster. Here is the Kubernetes manifest:

# microservices.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: demo
  labels:
    app: api-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
      - name: api-service
        image: your-repo/api-service:v1 # Replace with your image
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: demo
spec:
  selector:
    app: api-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: db-service
  namespace: demo
  labels:
    app: db-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db-service
  template:
    metadata:
      labels:
        app: db-service
    spec:
      containers:
      - name: db-service
        image: your-repo/db-service:v1 # Replace with your image
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: db-service
  namespace: demo
spec:
  selector:
    app: db-service
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

Apply it: kubectl apply -f microservices.yaml

Step 3: Observing Traffic with Hubble

With our services running, let's use Hubble to see the magic. Port-forward the Hubble UI service:

cilium hubble ui

Navigate to http://localhost:12000. You will see a blank canvas. Now, generate some traffic. Exec into a temporary pod and curl the api-service:

kubectl run -it --rm --image=curlimages/curl:latest curl-client -n demo -- sh
# Inside the pod:
curl http://api-service.demo.svc.cluster.local/users/123

Instantly, the Hubble UI will populate with a service map, showing the flow from the curl-client to api-service and then to db-service. Clicking on the arrow between services reveals detailed L4 and L7 information, including HTTP method, path, and response codes. All of this is captured by eBPF programs at the kernel level, with zero changes to our application code or pod manifests.

Advanced L7 Protocol Parsing with eBPF

Hubble's L7 visibility for HTTP, Kafka, and gRPC is powerful, but how does it work without a full user-space proxy? It relies on eBPF's ability to trace syscalls and perform limited in-kernel parsing.

When api-service calls db-service, it ultimately uses syscalls like sendmsg. Cilium's eBPF programs, attached via kprobes to these syscalls, can access the memory buffers containing the raw HTTP request data. The eBPF program then performs a bounded parse—it doesn't need to be a fully compliant HTTP/1.1 parser. It just needs to read enough of the buffer to extract the method (GET), path (/data/123), and headers of interest.

Here is a simplified conceptual snippet of what such an eBPF C program might look like (this is illustrative, not complete Cilium code):

// Simplified eBPF program to trace sendmsg and parse HTTP
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct http_event {
    char method[8];
    char path[128];
    u32 status_code;
};

// BPF ring buffer to send events to user space
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("kprobe/tcp_sendmsg")
int BPF_KPROBE(trace_tcp_sendmsg, struct sock *sk, struct msghdr *msg, size_t size) {
    // In a real implementation, we'd check if this is a connection
    // we care about by inspecting socket info (ports, IPs).
    
    struct iov_iter *iter = &msg->msg_iter;
    if (iter->type != ITER_IOVEC) {
        return 0;
    }

    // Read the first part of the user-space buffer
    char buffer[256];
    bpf_probe_read_user(&buffer, sizeof(buffer), iter->iov->iov_base);

    // Very basic HTTP GET parsing
    if (buffer[0] == 'G' && buffer[1] == 'E' && buffer[2] == 'T') {
        struct http_event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
        if (!e) {
            return 0;
        }

        // Copy method
        bpf_probe_read_user_str(&e->method, sizeof(e->method), buffer);

        // Find and copy path (highly simplified)
        // ... parsing logic to find the start and end of the path ...

        bpf_ringbuf_submit(e, 0);
    }

    return 0;
}

char LICENSE[] SEC("license") = "GPL";

This demonstrates the core principle: the eBPF program directly reads application memory during the syscall, performs a lightweight parse, and pushes structured data to a user-space agent via a high-performance ring buffer. This is fundamentally more efficient than copying the entire payload to a sidecar proxy.

Performance Benchmarking and Analysis

The most compelling reason to adopt an eBPF-based data plane is performance. Let's outline a methodology for quantifying this.

Test Setup:

* Cluster A: Kubernetes cluster with Istio installed, automatic sidecar injection enabled.

* Cluster B: Identical Kubernetes cluster with Cilium installed as the CNI.

* Workload: A simple request-response gRPC service (which is highly sensitive to latency) and a load generator like ghz.

Metrics to Capture:

Hypothetical Results:

These hypothetical numbers reflect a common pattern seen in real-world benchmarks. The eBPF data plane significantly reduces the latency tail (P99) because it eliminates the variable scheduling delays and processing overhead of user-space proxies. The reduction in aggregate CPU and memory is a direct result of centralizing the mesh logic in a single, efficient per-node agent instead of a per-pod sidecar.

Critical Edge Cases and Security Considerations

While powerful, the eBPF approach is not without its complexities and trade-offs.

1. The TLS Challenge

This is the most significant challenge. When traffic is encrypted with TLS/mTLS, an eBPF program attached to sendmsg at the kernel level sees only encrypted gibberish. It cannot parse HTTP headers or gRPC payloads.

Sidecar proxies solve this by terminating the TLS session (acting as the TLS server for ingress and client for egress). This gives them access to plaintext data but also requires them to manage certificates and keys, adding complexity.

Cilium's approach to this is evolving but generally involves a hybrid model:

uprobes (User-space Probes): eBPF can also trace user-space library calls. By attaching probes to the read/write functions in common SSL libraries (like OpenSSL's SSL_read/SSL_write), it's possible to intercept the data before encryption or after* decryption. This is highly complex, requires support for specific library versions, and can be fragile.

* Socket-level Abstraction: For some use cases, understanding the plaintext is less important than simply enforcing that traffic between service A and B is encrypted. eBPF can validate this at the socket level.

Sidecar as a Fallback: For workloads requiring deep L7 processing on encrypted traffic (e.g., gRPC method-level authorization), the recommended pattern is often to use Cilium's eBPF data plane for most of the mesh and selectively enable an Envoy sidecar only* for those specific pods. This provides a pragmatic best-of-both-worlds solution.

2. Kernel Version Dependency

eBPF is a rapidly developing kernel feature. Advanced capabilities, like certain program types or helper functions, require modern kernel versions (typically 5.10+ for mature features). Deploying an eBPF-based solution in an environment with legacy enterprise Linux distributions can be a non-starter. This is a critical prerequisite to validate before considering adoption.

3. Security and the eBPF Verifier

Running custom code in the kernel sounds inherently dangerous. The Linux kernel's primary defense is the eBPF verifier. Before any eBPF program is loaded, the verifier performs a static analysis of its code. It checks for:

* No Unbounded Loops: Prevents kernel hangs.

* Valid Memory Access: Ensures the program only accesses allowed memory regions and cannot cause a kernel panic.

* Finite Execution Path: The program must be guaranteed to complete.

This makes eBPF significantly safer than traditional kernel modules. However, the verifier is complex, and writing eBPF code that satisfies it can be challenging. Furthermore, a user with the CAP_BPF capability can load programs that could potentially leak sensitive kernel information, making cluster security and RBAC paramount.

Conclusion: The Future is a Hybrid, Optimized Mesh

The sidecar pattern is not obsolete. It offers a mature, feature-rich, and battle-tested solution for L7 traffic management, especially with encrypted traffic. However, the performance and resource overhead are undeniable.

eBPF represents the next frontier in service mesh data planes, offering a path to near-zero overhead for L4 and basic L7 observability and policy enforcement. For most workloads, an eBPF-based CNI like Cilium can provide the majority of required service mesh functionality with superior performance and a fraction of the resource cost.

The future of the service mesh is not a binary choice between sidecars and eBPF. It is a hybrid model where senior engineers make deliberate architectural decisions based on workload requirements. The default should be a highly efficient eBPF data plane, with the powerful but resource-intensive sidecar proxy enabled surgically, only where its advanced L7 capabilities on encrypted traffic are strictly necessary. This pragmatic approach allows teams to build scalable, observable, and secure microservices without paying the universal performance tax of a one-size-fits-all sidecar architecture.