eBPF Observability: What It Replaces and Where Sidecars Stay

eBPF observability means watching what software does by running small, checked programs inside the Linux kernel instead of placing a monitoring helper next to every application. It has real adoption and removes some of the cost of sidecars, though not all of it.

eBPF handles basic network and system visibility well, and a large Alibaba paper reports very low overhead when tracing AI training jobs with it. Anything that needs deep knowledge of application protocols, such as retries or header rewriting, still tends to run in a user-space proxy like Envoy.

What is eBPF, and what does it do for observability?

eBPF is a Linux kernel technology. According to ebpf.io, it can run sandboxed programs in a privileged context such as the operating system kernel, extending the kernel without changing its source code or loading kernel modules.

The kernel is the core of the operating system, the layer that handles files, memory and network traffic. An eBPF program attaches to a hook, which is a point where something happens, such as a system call, a function entry or a network event. When that event fires, the program runs and can record what took place.

For observability, this means a tool can see which process opened a connection, how long a request waited or where packets were dropped, without editing the application. The same page lists observability, networking and security enforcement as the main uses.

Because the program runs outside the application, nothing has to be added to the application's code or its container. That is why teams describe eBPF tools as zero-code instrumentation, and it is the main reason the approach appeals to platform teams who do not own every service they run.

What is a sidecar, and why do people want to replace it?

A sidecar is a helper program that runs in the same Kubernetes pod as the application. A pod is a small group of containers scheduled together. A service mesh, the layer that manages traffic between services, injects a proxy sidecar so it can encrypt, route and measure every request without changing the app.

Every pod gets its own proxy, so the resource cost grows with the number of pods. That cost is the motivation for sidecar-free designs. Istio's ambient mode, described in the Istio documentation, is informally called a sidecar-less mesh because workload pods no longer need proxies in sidecars to join the mesh.

Istio's ambient mode relies on a per-node Layer 4 proxy written in Rust rather than on eBPF. Sidecar-free and eBPF-based are two overlapping ideas, and mixing them up is common.

Do eBPF tools actually replace sidecars?

Partly. The Cilium documentation describes a Kubernetes networking project built on eBPF. It offers flow visibility through a component called Hubble, identity-based network policy from Layer 3 to Layer 7, and a service mesh that it says avoids the cost and complexity of traditional proxy designs.

Layers here describe how deep a tool reads a message. Layer 4 covers connections and ports. Layer 7 covers application protocols such as HTTP and gRPC.

Solo.io sells Envoy-based products and so has a stake in the answer. It argues that eBPF can shorten network paths, offload encryption and collect data, but cannot replace the Layer 7 proxy. Its reasoning is that eBPF programs must be fully known and checked before they run, and arbitrary loops are not allowed. That makes protocol parsing, retries and header changes hard to build in the kernel.

Istio's design lines up with that split. Its optional waypoint proxies are Envoy, they handle the Layer 7 features, and Layer 7 telemetry needs them.

How does eBPF observability work on AI clusters?

AI training jobs run on many GPUs that exchange data constantly through collective communication, where every worker waits for the slowest one. A single slow node can hold back the whole job, so finding it matters.

A paper on arXiv, SysOM-AI, describes a system deployed at Alibaba. It combines CPU profiling, GPU kernel tracing and NCCL event tracing, using eBPF-based tracing. NCCL is NVIDIA's library for GPU-to-GPU communication.

The authors report less than 0.4% overhead, a deployment across more than 80,000 GPUs for over a year, and a median diagnosis time cut from days to about 10 minutes. They also say existing tools cover only one layer, add 10 to 30% overhead, or cannot run continuously.

Those are the authors' own figures from their own deployment. The paper does not compare eBPF with sidecars, because traffic between training GPUs does not pass through a service mesh in the first place. It shows that eBPF tracing can run continuously on a very large fleet. For related infrastructure reading, see how co-packaged optics address the GPU bandwidth wall.

What are the limits of eBPF observability?

The requirements are concrete. Grafana's Beyla documentation says the tool needs Linux kernel 5.8 or higher with a feature called BTF enabled, and kernel 5.17 or higher for HTTP header-based trace propagation. It also needs elevated privileges, though specific capabilities can be granted in place of full root access.

Beyla captures generic metrics and transaction-level trace spans for HTTP and gRPC services. Its docs still recommend language agents or manual instrumentation when a team wants fine-grained detail on critical code. Tracing support varies by language, and the docs advise validating before relying on it for reactive Java.

An eBPF tool sits outside the application, which is convenient, and sees only what the kernel and the network can show. Details such as which customer a request belongs to live inside the application. A platform team can get service maps and request rates quickly, but a product team asking business questions will still need instrumentation in the code.

Kernel age is the other practical limit. Clusters running older kernels cannot use tools that require newer features, and upgrading nodes is a project of its own.

Is eBPF safe to run in production?

The kernel checks every eBPF program before it attaches. Per ebpf.io, the verifier confirms that the loader has the right privileges, that the program will not crash the system, that it runs to completion and that it has no out-of-bounds memory access. After that, a just-in-time compiler turns it into machine code.

The same page adds a limit. The verifier is not a security tool that inspects what programs are doing. A verified program is safe to run, yet it can still collect data it should not, so permission to load eBPF programs deserves the same care as root access.

Running untrusted code is a separate problem with its own trade-offs, covered in our look at micro-VMs and WebAssembly sandboxes.

What to watch next

Solo.io says kernel-side Layer 7 support may be years away. How widely sidecar-less meshes spread is also worth following, as are independent, published comparisons of eBPF and sidecar overhead on the same workload, since the figures available today come from vendors or from single deployments.

For a team choosing now, a mixed setup looks sensible. Use eBPF for baseline network and kernel visibility, a proxy where Layer 7 features are needed, and in-application instrumentation for business detail. A small pilot on one node pool will show what the tool captures on your own workloads before any wider rollout. Check the kernel version of your nodes first, because that is the requirement the documentation states most plainly.