Skip to content
Published on

eBPF Is Eating Observability

🇰🇷KO🇺🇸EN🇯🇵JA
필사 모드
Authors

Introduction — The Idea of Running Code Inside the Kernel

Anyone who has ever set up observability tooling knows the fatigue. You add an agent to every application, a instrumentation library for every language, a sidecar to every pod — and then all of that starts eating CPU and memory of its own. It is the paradox of making a system heavier and more complex just to observe it.

eBPF attacks this problem from a completely different angle. Instead of touching the application, it drops down into the kernel and watches everything from below. Network packets, system calls, function calls, disk I/O — they all pass through the kernel, so if you observe once at the kernel, you can instrument whatever runs on top of it. The application does not even know it is being watched.

This post covers what eBPF actually is, how it can safely run arbitrary code inside the kernel, and why it is swallowing observability tools one by one. It is also honest about the fact that eBPF is not a silver bullet, and where its limits lie.

What eBPF Is — A Tiny Virtual Machine Inside the Kernel

eBPF (extended Berkeley Packet Filter) is, despite the name, no longer confined to packet filtering. Today's eBPF is closer to a tiny virtual machine embedded inside the Linux kernel. You write a small program, the kernel attaches it to a particular event, and every time that event fires, it runs your program inside the kernel context.

Here is the crux. Traditionally, to change or inspect kernel behavior you had to write a kernel module. Kernel modules are powerful but dangerous. A single bug can send the whole kernel into a panic, and bad code can freeze the system. Loading a module onto a production kernel was therefore a decision you did not take lightly.

eBPF removes that danger. An eBPF program runs inside the kernel, but it is strictly constrained so that it cannot bring the kernel down. What enforces that constraint is the verifier, which we will cover in detail below. As a result, we get something close to the power of a kernel module, but without the risk of freezing the system, and we can attach and detach programs in production at will.

The big picture looks like this.

  User space
  +------------------------------+
  | 1. Write eBPF in C/Rust/etc. |
  | 2. LLVM compiles it into     |
  |    eBPF bytecode             |
  +--------------+---------------+
                 | load via the bpf() syscall
                 v
  Kernel space
  +------------------------------+
  | 3. Verifier checks safety    |
  | 4. JIT compiles to native    |
  | 5. Attach to hook points     |
  +--------------+---------------+
                 | runs when the event fires
                 v
  +------------------------------+
  | 6. Write results to a map    |
  |    User space reads the map  |
  +------------------------------+

The program is written and compiled in user space, then loaded into the kernel via the bpf() syscall. If it passes the verifier, the JIT compiler turns it into native machine code and attaches it to the hook points you specified. Data the program collects is passed back to user space through shared data structures called maps.

Where You Can Hook — kprobes, uprobes, tracepoints, XDP

The power of eBPF comes from where you can attach it. The kinds of hook points define the scope of what you can observe. Let's look at the four main ones.

  • kprobe (kernel probe): Attaches a program at the entry or return of a kernel function. For example, you can hook it to run every time tcp_connect is called. You can hook essentially any kernel function, which makes it the most flexible option — but it depends on kernel-internal function names, so it can break when the kernel version changes.
  • uprobe (user probe): Attaches to a function in a user-space program. It runs when a specific function in an application binary is called. This is the key mechanism for peering inside an application "without changing its code." For instance, you can put a uprobe on an SSL library's encryption function to observe plaintext traffic.
  • tracepoint: A stable instrumentation point placed in advance by kernel developers. Unlike kprobes, it is an API the kernel officially maintains, so it rarely breaks across kernel versions. When stability matters, tracepoints are preferred over kprobes.
  • XDP (eXpress Data Path): Runs at the very front of the network stack, the moment a packet arrives at the driver — even before it passes through the kernel network stack, which makes it extremely fast. DDoS defense and load balancing that handle millions of packets per second happen here.

There is one insight that runs through this list: almost anything interesting in a system passes through the kernel. Whether you open a file, create a socket, fork a process, or send a packet, some kernel function or tracepoint fires at that moment. eBPF sits right at those points and watches the activity of the entire system from a single vantage point.

The Verifier — Why the Kernel Does Not Crash

A natural question arises here. If you run arbitrary code inside the kernel, can it not get stuck in an infinite loop or touch bad memory and bring the kernel down? The reason eBPF is trusted in production is exactly the answer to that question: the verifier.

The verifier statically analyzes a program the moment it is loaded into the kernel, before it ever runs. It walks every execution path the program can take and tries to prove the following.

  • Guaranteed termination: The program must finish. Unbounded loops were traditionally forbidden, and today only bounded loops whose upper limit the verifier can prove are allowed. A program that never halts will not even load.
  • Memory safety: The program may only read and write permitted memory regions. The verifier tracks whether a pointer is null-checked before it is dereferenced and whether array accesses stay within bounds.
  • Bounded instruction count: There is an upper limit on program complexity so that the verifier can analyze the whole thing within a realistic amount of time.

If a program fails this process, the kernel refuses to load it. In other words, an unsafe eBPF program never even gets the chance to run. This is the decisive difference from kernel modules. A kernel module is something you "load on trust," whereas eBPF is something that "must be proven before it loads."

Of course, the verifier is not perfect. Sometimes it rejects a program that is in fact safe, simply because it cannot prove the safety. That is why eBPF developers joke about "fighting the verifier." Anyone who has worked with eBPF seriously has experienced twisting perfectly good code around just to satisfy the verifier. But that strictness is precisely the price of a kernel that does not crash.

Zero-Instrumentation Tracing — You Don't Change the Code

If you had to summarize in one phrase why eBPF is revolutionary for observability, it is "zero-instrumentation." Traditional observability starts by planting instrumentation into application code. To trace, you add code that opens a span for each request; to extract metrics, you add code that increments a counter. The SDK differs per language, and you have to redo the work every time you update a library.

eBPF flips this premise. It leaves the application untouched and observes the information it needs from kernel hooks. Want to trace HTTP requests? Hook socket-related kernel functions. Want to measure gRPC latency? Hook the relevant uprobes. The application does not need to be recompiled, does not need to restart, and does not even need to know it is being observed.

The implications in practice are large.

  • Language-agnostic: Go, Python, Rust, Java — from the kernel's point of view, they are all the same system calls and network events. No need to maintain per-language SDKs.
  • Covers legacy: Third-party binaries with no source code, or that you cannot modify, are observable too. This is especially powerful when you cannot change the code.
  • Minimal performance overhead: The instrumentation code does not enter the application's hot path; it runs JIT-compiled in the kernel, so overhead is small.

That said, the phrase "zero-instrumentation" can be misleading, so let's be precise. What eBPF sees automatically at the kernel level are low-level signals like system calls, network flows, and function calls. Application-specific semantics like "which business transaction does this request belong to" are invisible to the kernel. So things like context propagation for high-level distributed tracing often still require cooperation from the application. eBPF removes a large chunk of instrumentation, but it does not remove all of it.

The Tools — Cilium, Falco, Pixie, bpftrace

Beyond theory, eBPF is already the engine behind tools widely used in production. Let's look at four notable ones.

  • Cilium: Implements Kubernetes networking and security with eBPF. Traditional Kubernetes networking relied on iptables rules, and when pods grew into the thousands, the number of iptables rules exploded and performance collapsed. Cilium replaces this with eBPF programs to handle routing, policy enforcement, and load balancing far more efficiently. It is also moving toward implementing service mesh features with eBPF, without sidecar proxies.
  • Falco: A runtime security observation tool. It watches system calls in real time and raises alerts when it detects suspicious behavior — for example, a shell spawning inside a container, a sensitive file being read, or an unexpected network connection appearing. Because it observes directly at the kernel, it is hard for an attacker to bypass.
  • Pixie: A Kubernetes observability platform that puts eBPF's zero-instrumentation front and center. Without any instrumentation code, it automatically captures a cluster's HTTP, gRPC, and database traffic and shows inter-service latency and error rates. The "install it and you can see everything" experience is a flagship example of what eBPF makes possible.
  • bpftrace: The Swiss Army knife of eBPF. It is a high-level tracing language that lets you observe the kernel on the spot with a one-line script. It is used to dive straight into "what is happening right now" in production.

For instance, counting how many of which system calls a process makes with bpftrace is this simple.

# Tally system call counts per process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

This one line hooks a kernel tracepoint, increments a counter per process name (comm) every time a syscall fires, and prints the tally on exit. No separate agent, no application changes. That immediacy is why bpftrace is beloved for production debugging.

If you want to explore these ideas conceptually in the browser, the eBPF playground lets you learn the eBPF instruction set and how the verifier behaves through a simulation, and you can look at the Kubernetes networking context in the Kubernetes network lab.

Why It Beats Sidecars

For the last several years, the default pattern of the cloud-native world was the sidecar. Service meshes attached a proxy container next to every pod to intercept and observe traffic. This model worked well, but the cost was high.

Let's spell out the problems with sidecars.

  • Resource multiplication: One proxy per pod means 1,000 proxies for 1,000 pods. Each proxy consumes CPU and memory, and the sum becomes a non-trivial amount.
  • Added latency: Every request goes application → sidecar → network → peer sidecar → peer application. Latency piles up at every hop.
  • Operational complexity: Injecting sidecars, keeping versions aligned, and upgrading them is itself a management burden.

eBPF fundamentally changes this structure. Instead of placing a proxy next to each pod, it observes once inside the kernel and applies policy there. Rather than 1,000 proxies per node, everything is handled at one shared point: the kernel. Traffic does not have to detour out to a user-space proxy and back, so latency drops too.

Of course, this does not mean "the sidecar is dead." Sidecars are still advantageous for handling complex application-level logic (advanced routing rules, sophisticated protocol-specific manipulation). eBPF has its strength on the low-level fast path, and sidecars on the high-level flexible path. In fact, the industry is converging on a hybrid: a "sidecarless data plane plus proxies only where needed."

What eBPF Cannot Do — Limits and Pitfalls

eBPF is powerful but not a silver bullet. There are clear limits to know before adopting it.

  • Linux-centric: eBPF is fundamentally a Linux kernel technology. A Windows port of eBPF is underway, but its maturity and ecosystem lag far behind Linux. On non-Linux environments the story is completely different.
  • Kernel-version dependence: Hooks that depend on kernel internals, like kprobes, can break when the kernel version changes. Technologies like CO-RE (Compile Once, Run Everywhere) have greatly eased this, but there are still constraints where you cannot use certain features on older kernels.
  • The verifier wall: As noted, the verifier is safe but finicky. It is common to hit the verifier while trying to implement complex logic, and working around it makes the code awkward. The upper limits on program size and complexity are real constraints too.
  • Requires high privilege: Loading an eBPF program generally requires powerful privileges (CAP_BPF and friends). This means eBPF itself can become an attack surface. An attacker who gains the privilege to load a malicious or defective eBPF program can deeply observe or manipulate the system.
  • Semantic limits: As emphasized, the kernel only sees low-level events. The application's business meaning (which user's which order this request is) is invisible to the kernel. Observation that needs high-level context still requires cooperation from the application.

Taken together, these limits mean eBPF is optimal for "low-level, high-performance, system-wide observation and networking on top of Linux." When you need cross-platform support, when deep application semantics are the core, or when you need logic too complex for the verifier to handle, another approach is better.

Conclusion — The Center of Gravity Descends

The essence of the change eBPF drives is that the center of gravity of observability descends from the application to the kernel. Previously, to observe, you planted instrumentation in every application and attached a sidecar to every pod. Now you hook once at a shared point, the kernel, and watch whatever runs on top of it.

The reasons this shift is attractive are clear. It is language-agnostic, requires no code changes, has small overhead, and is hard to bypass. That is exactly why Cilium is redefining the service mesh, Falco runtime security, Pixie automatic observation, and bpftrace on-the-spot debugging, each in its own way.

At the same time, sobriety is warranted. eBPF is tied to Linux, the verifier is finicky, and there are semantic limits to what the kernel can see. "eBPF is eating observability" is not an exaggeration, but it does not mean it eats everything. In the low-level world the kernel sees well, eBPF is overwhelming; in the high-level world above it, it still coexists with other tools.

The next time you design an observability stack, before you attach one more thing to the application, ask this: "Can't I already see this from the kernel?" Surprisingly often, the answer is yes.

References

Discuss on Twitter • View on GitHub