eBPF for Sidecar-less Service Mesh Telemetry in Kubernetes

The Sidecar Proxy Dilemma in Production

For years, the sidecar proxy—epitomized by Envoy in Istio—has been the de facto standard for implementing service meshes in Kubernetes. It provides critical features like traffic management, security, and observability by intercepting all network traffic to and from a pod. While powerful, this pattern introduces significant, non-trivial overhead in production environments that platform and senior engineers constantly battle.

These are not theoretical concerns; they are daily operational realities:

* Injection: Managing mutating webhooks for sidecar injection can be fragile.

* Upgrades: Rolling out a new version of the service mesh requires restarting every application pod in the cluster to inject the new sidecar version, a high-risk and disruptive operation in large-scale environments.

* Resource Management: Fine-tuning CPU/memory requests and limits for hundreds or thousands of sidecars is a constant battle.

To quantify this, consider a typical high-throughput service under load:

These are representative figures; actual impact varies with workload and configuration.

The core issue is that the sidecar model forces network-level logic into a user-space process co-located with every application instance. The alternative is to push this logic down the stack, into a shared, highly efficient layer: the Linux kernel. This is where eBPF (extended Berkeley Packet Filter) fundamentally changes the game.

The eBPF Alternative: Kernel-Level Transparency

eBPF allows us to run sandboxed programs directly within the Linux kernel, triggered by various events like system calls, network events, or function entries/exits. For a service mesh, this means we can achieve the same goals of observability, security, and traffic management without a per-pod proxy.

The mechanism is fundamentally more efficient:

* Transparent Interception: Instead of redirecting traffic with iptables to a user-space proxy, we attach eBPF programs to kernel hooks on the TCP/IP stack, such as Traffic Control (TC) hooks (cls_act) or socket-level hooks (connect, sendmsg, recvmsg).

* Kernel-Native Execution: These eBPF programs execute in the kernel's context. They can inspect, filter, modify, and redirect packets at line rate, orders of magnitude faster than context-switching to a user-space process.

* Shared Resource Model: A single eBPF-enabled agent (like Cilium) runs per node, managing the eBPF programs for all pods on that node. The resource cost is fixed per-node, not per-pod, leading to massive efficiency gains.

Data Path Comparison:

Sidecar Model:

App -> veth -> Pod NetNS -> iptables -> Envoy Proxy (User Space) -> Pod NetNS -> veth -> Node

eBPF Model:

App -> veth -> Pod NetNS -> eBPF Program (Kernel Space) -> Node

The iptables redirection and the entire user-space proxy hop are eliminated. For L7 policies, the eBPF program can parse protocol headers (HTTP, gRPC, etc.) directly in the kernel, make a decision, and forward the packet without ever leaving the kernel.

This approach isn't just a theoretical improvement; it's a paradigm shift in how we build cloud-native infrastructure. Let's move to a practical, production-focused implementation using Cilium.

Practical Implementation with Cilium Service Mesh

Cilium is a CNI (Container Network Interface) that leverages eBPF for networking, observability, and security. Its built-in service mesh capabilities allow us to realize the sidecar-less vision.

Prerequisites:

* A running Kubernetes cluster (v1.23+ recommended).

* Linux kernel v5.10+ on all nodes. This is a critical production requirement. While some features work on older kernels, modern eBPF capabilities for a service mesh depend on recent kernel developments.

Step 1: Install and Configure Cilium

We will use Helm to install Cilium, enabling the necessary features for a sidecar-less service mesh.

# Add the Cilium Helm repository
helm repo add cilium https://helm.cilium.io/

# Create a values.yaml file for our configuration
cat <<EOF > cilium-values.yaml
# Enable Hubble for observability
hubble:
  relay:
    enabled: true
  ui:
    enabled: true

# Enable service mesh features
# This uses eBPF to power L7 visibility and policy
serviceMesh:
  enabled: true

# Use kube-proxy replacement for maximum efficiency
# This replaces iptables/ipvs with eBPF for service routing
kubeProxyReplacement: strict

# Enable BPF-based host routing for pod traffic
bpf:
  masquerade: true

# Recommended for performance
# Reduces CPU overhead for routing
endpointRoutes:
  enabled: true
EOF

# Install Cilium
helm install cilium cilium/cilium --version 1.15.5 \
  --namespace kube-system \
  -f cilium-values.yaml

This configuration does several key things:

Step 2: Deploy Sample Microservices

Let's deploy a classic bookinfo-style application to test our mesh. We'll use a simplified version with a productpage service calling a details service.

# bookinfo.yaml
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      labels:
        app: productpage
        version: v1
    spec:
      containers:
      - name: productpage
        image: docker.io/istio/examples-bookinfo-productpage-v1:1.17.0
        ports:
        - containerPort: 9080
---
apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.17.0
        ports:
        - containerPort: 9080

Apply this manifest: kubectl apply -f bookinfo.yaml.

Notice there is no sidecar injection annotation. The pods are standard, unmodified Kubernetes deployments. The observability and policy enforcement will be applied transparently by Cilium at the node level.

Step 3: Enforce L7 Traffic Policies with eBPF

Now, let's create a policy that only allows the productpage service to call the details service on GET /details/* paths.

# details-l7-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "details-l7-access-policy"
spec:
  endpointSelector:
    matchLabels:
      app: details
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: productpage
    toPorts:
    - ports:
      - port: "9080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/details/.*"

Apply the policy: kubectl apply -f details-l7-policy.yaml.

How this works:

When a packet from productpage destined for details:9080 arrives at the TC hook on the details pod's virtual ethernet device (veth), Cilium's eBPF program is triggered. The program:

• Sees that an L7 HTTP rule is attached.
• Instead of immediately forwarding the packet, it begins parsing the TCP stream for HTTP headers.

This entire decision happens within the kernel context, without any user-space proxy involved.

Deep Dive into Observability and Telemetry

With our services running and policy in place, let's explore the telemetry we get for free.

Using Hubble for Real-time Visibility

Hubble is Cilium's observability component. Let's forward its port and access the UI.

# Forward the Hubble Relay port
kubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Use the Hubble CLI to check status
cilium status

# Open the Hubble UI
cilium hubble ui

This will open a web browser showing a live service map of your applications. Generate some traffic by exec-ing into the productpage pod and calling the details service.

PRODUCTPAGE_POD=$(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}')

# Successful call
kubectl exec -it $PRODUCTPAGE_POD -- curl -s http://details:9080/details/1

# This call would be blocked by our policy, but the app doesn't make it.
# If it did, Hubble would show the traffic as dropped.

Querying L7 Metrics with the Hubble CLI

The Hubble CLI is a powerful tool for inspecting traffic flows captured by eBPF.

# See all recent flows in the default namespace
hubble observe --namespace default -f

# Filter for HTTP requests from productpage to details
hubble observe --namespace default --from-pod default/productpage-v1 --to-pod default/details-v1 --protocol http

# Sample Output:
# TIMESTAMP          SOURCE -> DESTINATION                                   VERDICT     SUMMARY
# Apr 23 15:30:01.123  default/productpage-v1-.. -> default/details-v1-..:9080  FORWARDED   HTTP/1.1 200 GET /details/1

This output is generated directly from data collected by eBPF programs in the kernel and aggregated by the Cilium agent. It includes HTTP method, path, and response code.

Exporting Metrics to Prometheus

Hubble can expose these metrics in a Prometheus-compatible format. This is typically enabled by default in the Helm chart. You just need to configure Prometheus to scrape the hubble-relay service.

Prometheus Scrape Configuration:

- job_name: 'hubble'
  scrape_interval: 10s
  static_configs:
  - targets: ['hubble-relay.kube-system.svc.cluster.local:4245']

Once scraped, you can run powerful PromQL queries in Grafana:

* HTTP Request Rate:

sum(rate(hubble_flows_processed_total{verdict="FORWARDED", l7_protocol="http"}[5m])) by (source_service, destination_service)

* HTTP Error Rate (5xx):

sum(rate(hubble_http_responses_total{status_code=~"5.."}[5m])) by (source_service, destination_service)

* p99 Latency (from Cilium's experimental latency metrics):

histogram_quantile(0.99, sum(rate(hubble_tcp_latency_seconds_bucket[5m])) by (le, source_service, destination_service))

Edge Case: Handling Encrypted (TLS) Traffic

This is a critical production question: How can eBPF provide L7 visibility into TLS-encrypted traffic without terminating TLS?

The sidecar model solves this with mTLS, where the sidecar terminates the client-side TLS, inspects the plaintext, and then re-encrypts it for the server-side proxy. This is effective but complex.

eBPF offers a more clever solution by using Kernel Probes (kprobes). Cilium can attach eBPF programs to the read/write system calls within the kernel, specifically targeting common SSL libraries like OpenSSL or Go's crypto/tls library.

The process:

• An application uses a library like OpenSSL to handle TLS.

This provides L7 visibility without terminating TLS or requiring private keys. However, it comes with significant caveats:

* Fragility: It depends on the specific implementation details and function signatures of the SSL library being used. An update to the library could break the probes.

* Security: The Cilium agent needs elevated privileges to inspect application memory, which has security implications.

* Setup: It requires careful configuration to point Cilium to the correct library binaries within the pod. This feature is still evolving but shows the power and flexibility of eBPF.

Performance Benchmarking and Analysis

Let's revisit the performance claims with a more structured benchmark. We'll use the fortio load testing tool to compare three scenarios:

Test Setup:

* Workload: A simple gRPC service.

* Load: 1000 QPS for 5 minutes.

* Cluster: 3-node GKE cluster (e2-standard-4 nodes).

Benchmark Results (Representative):

Analysis:

The results are stark. The eBPF-based mesh (Cilium) adds minimal latency over the baseline, while the sidecar model (Istio) adds significant latency, especially at the tail (p99). The resource savings are even more dramatic. The CPU cost is nearly halved, and the memory overhead model shifts from a costly per-pod tax to a fixed, low per-node cost.

For platforms running thousands of pods, this difference translates directly into millions of dollars in infrastructure savings and improved application performance.

Advanced Considerations and Production Caveats

Adopting an eBPF-based service mesh is not a silver bullet. Senior engineers must be aware of the following trade-offs and complexities.

Production Strategy: Standardize your node OS images on a distribution with a modern kernel (e.g., Ubuntu 22.04+, RHEL 9+). Actively manage kernel versions as part of your infrastructure lifecycle.

Production Strategy: Audit your application protocols. For services requiring L7 policy on unsupported protocols, you may need a hybrid approach, selectively using a traditional proxy gateway for those specific workloads.

* cilium status: Your first port of call. It provides a detailed health check.

* cilium monitor: A powerful tool to see packet drop events and policy verdicts in real-time.

* bpftool: A low-level utility for inspecting loaded eBPF programs and maps. For example, bpftool map dump name cilium_policy_... can show you the kernel-level representation of a network policy.

* Hubble: Remains the best high-level tool for visualizing and understanding traffic flows and drops.

Production Strategy: Harden the Cilium agent configuration. Use Kubernetes RBAC to restrict who can modify Cilium's CRDs and DaemonSet. Ensure the agent's container image is scanned and comes from a trusted source. The security trade-off is moving from a distributed risk (a vulnerable proxy in every pod) to a centralized one (a privileged agent on every node).

The shift from sidecar proxies to eBPF-native service meshes represents a major evolution in cloud-native architecture. By moving network intelligence from user-space into the kernel, we can build platforms that are not only faster and more efficient but also simpler to operate at scale. While it requires a deeper understanding of the underlying Linux kernel, the performance and resource benefits are too significant for senior engineers to ignore.