eBPF Runtime Security — Tetragon, Falco, and BPF LSM

• Introduction
• Why Runtime Security
• Detection and Enforcement Are Different Problems
• Falco: The Standard for Rule-Based Runtime Detection
  • Architecture
  • Rule syntax and practical examples
• Tetragon: From Observation to Enforcement
  • Architecture and the TracingPolicy CRD
  • Falco vs Tetragon
• BPF LSM: Plugging eBPF into Kernel Security Hooks
• The Relationship with seccomp and AppArmor
• Kubernetes Deployment and the Event Pipeline
• Rule Operations: The War on Noise
• Bypass Potential and Limits — From the Defender Point of View
• Measuring Performance Impact
• Adoption Roadmap: Observe, Detect, Enforce
• Adoption Checklist
• Closing Thoughts
• References

Introduction

You pass image scans, manage secrets safely, and enforce network policies. Done? The security incidents of the 2020s answer with a firm no. Supply chain attacks where a seemingly benign dependency sails through the build pipeline and turns malicious at runtime; container escapes through vulnerable kernel features or misconfigured privileges; lateral movement by attackers who walked in with stolen credentials — none of these can be caught by static checks performed before deployment, because they happen during execution.

Runtime security is the layer that observes "what is this running workload doing right now" and detects or blocks behavior that violates policy. The most compelling technology for building that layer today is eBPF. The eBPF fundamentals and observability techniques from the previous posts now turn toward security. We will walk through rule-based detection with Falco, kernel-level enforcement with Tetragon, and the foundation underneath: BPF LSM.

Why Runtime Security

Plot a typical container compromise on a timeline and the coverage split between static and runtime security becomes obvious.

Attack chain and defense layers
            Build time                    Runtime
+---------------------------+ +------------------------------------+
| Image scan, SBOM, signing | | From here on, runtime security     |
+---------------------------+ +------------------------------------+
                                  |
  exploit / malicious dependency  |
        v                         v
  [Initial access] -> [Recon] -> [Privilege esc.] -> [Persistence] -> [Objective]
   webshell exec      ls, id     container           cron entry       data exfil
   reverse shell      read /etc  escape               backdoor        cryptomining
        ^                ^            ^                  ^               ^
        |                |            |                  |               |
   exec events      file access   kernel syscalls    file writes     network
   detectable       detectable    detect/block       detectable      detectable

The key observation: every stage of an attack eventually surfaces as system calls. Process execution is execve, file access is openat, network connection is connect. If you can observe the syscall boundary, attacker behavior has nowhere to hide — and eBPF is precisely the technology that observes that boundary at low overhead (and, since kernel 5.7, can block at it too).

Detection and Enforcement Are Different Problems

This is the first axis to separate when evaluating runtime security tools.

Asynchronous detection has one fundamental limitation: between the event reaching the user-space agent and the rule being evaluated, the malicious act may already have completed (the TOCTOU problem). The mature operating model is therefore a combination of "broad detection plus narrow, certain enforcement." We return to this in the adoption roadmap at the end.

Falco: The Standard for Rule-Based Runtime Detection

Falco is a CNCF graduated project that matches the system call stream against rules to detect anomalous behavior.

Architecture

+--------------------------------------------------------------+
|                      Falco architecture                       |
|                                                              |
|  Event sources                                               |
|   ├── Syscalls: modern eBPF probe (CO-RE, default)           |
|   │             legacy: kernel module / classic eBPF         |
|   └── Plugins: k8s audit log, cloud logs, etc.               |
|        |                                                     |
|        v                                                     |
|  libs (libscap/libsinsp): event capture, container metadata  |
|        |                                                     |
|        v                                                     |
|  Rule engine: matches YAML rules and condition expressions   |
|        |                                                     |
|        v                                                     |
|  Outputs: stdout / gRPC / file --> Falcosidekick             |
|                                  ├── Slack / PagerDuty       |
|                                  ├── Elasticsearch / Loki    |
|                                  └── SIEM / webhooks         |
+--------------------------------------------------------------+

The default driver in modern Falco is the CO-RE-based modern eBPF probe. It needs no kernel module, survives node OS upgrades gracefully, and works out of the box on BTF-enabled kernels (5.8 or later recommended).

Rule syntax and practical examples

A Falco rule consists of a condition (expression), an output (alert message), and a priority (severity); macros and lists provide reusable building blocks.

Example 1 — detect a shell spawned inside a container (the classic rule):

- macro: container
  condition: (container.id != host)

- macro: spawned_process
  condition: (evt.type in (execve, execveat) and evt.dir = <)

- list: shell_binaries
  items: [bash, sh, zsh, dash, ksh]

- rule: Shell Spawned in Container
  desc: A shell was spawned inside a container, possible interactive access
  condition: >
    spawned_process and container
    and proc.name in (shell_binaries)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name image=%container.image.repository
     parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Example 2 — detect access to sensitive files:

- list: sensitive_files
  items: [/etc/shadow, /etc/sudoers, /root/.ssh/authorized_keys]

- rule: Sensitive File Opened for Reading
  desc: Detect attempts to read sensitive files by non-trusted programs
  condition: >
    evt.type in (open, openat, openat2) and evt.dir = <
    and fd.name in (sensitive_files)
    and not proc.name in (sshd, systemd, sudo)
  output: >
    Sensitive file opened (file=%fd.name user=%user.name
    proc=%proc.name cmdline=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [filesystem, secrets, mitre_credential_access]

Example 3 — detect unexpected outbound connections (a mining/exfiltration signal):

- rule: Unexpected Outbound Connection from Web Container
  desc: Web tier should only talk to the app tier and DNS
  condition: >
    evt.type = connect and evt.dir = <
    and container.image.repository = "myorg/web-frontend"
    and fd.type = ipv4
    and not fd.sip in ("10.0.20.0/24")
    and not fd.sport = 53
  output: >
    Unexpected outbound connection (dest=%fd.rip:%fd.rport
    container=%container.name image=%container.image.repository)
  priority: NOTICE
  tags: [network, mitre_exfiltration]

Three practical rule-writing tips. First, the negative clauses (not ...) in a condition become your exception list, so factor them into macros and lists from day one. Second, put enough container, image, and command-line fields into the output — triage speed depends on it. Third, bind each priority level one-to-one with a response procedure so that alerts translate into action.

Tetragon: From Observation to Enforcement

Tetragon is the eBPF-based runtime security tool from the Cilium ecosystem (Isovalent, now part of Cisco) and a CNCF project. Its biggest difference from Falco is in-kernel inline filtering and enforcement. Instead of shipping every event to user space for evaluation, policies are compiled down into eBPF programs in the kernel, and on a match the kernel can immediately send a signal (SIGKILL) or override a return value.

Architecture and the TracingPolicy CRD

+---------------------------------------------------------------+
|                     Tetragon architecture                      |
|                                                               |
|  TracingPolicy CRD (YAML, cluster resource)                   |
|        |  interpreted by the Tetragon operator/agent          |
|        v                                                      |
|  Per-node Tetragon agent (DaemonSet)                          |
|        |  compiles/loads policies as eBPF programs            |
|        v                                                      |
|  Kernel: attached to kprobe / tracepoint / LSM hooks          |
|        ├── matched events --> ringbuf --> agent --> JSON/gRPC |
|        └── matchActions: Sigkill / Override (in-kernel)       |
+---------------------------------------------------------------+

Being Kubernetes-native matters too: events are automatically enriched with Pod, namespace, and container metadata, and policies are CRDs, so they are managed through GitOps.

Example 1 — block (sigkill) package manager execution inside containers:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: deny-package-managers
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/usr/bin/apt"
                - "/usr/bin/apk"
                - "/usr/bin/yum"
                - "/usr/bin/dnf"
          matchActions:
            - action: Sigkill

A container installing packages at runtime is almost always a bad sign (a violation of the immutable image principle, or a compromise), which makes this one of the safest candidates for enforcement.

Example 2 — deny writes to a specific file at an LSM hook (return value override):

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: protect-ssh-authorized-keys
spec:
  lsmhooks:
    - hook: "file_open"
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Postfix"
              values:
                - ".ssh/authorized_keys"
          matchActions:
            - action: Override
              argError: -1

LSM-hook-based enforcement is cleaner than sigkill: rather than killing the process after the act, the kernel rejects the operation itself with a permission error (requires a kernel with BPF LSM enabled).

Falco vs Tetragon

In practice this is not an either/or choice: "detect broadly with Falco, enforce only the narrow, proven policies with Tetragon" is a perfectly reasonable combined deployment.

BPF LSM: Plugging eBPF into Kernel Security Hooks

LSM (Linux Security Modules) is the set of security decision points used by SELinux and AppArmor. For operations like opening a file, executing a program, or creating a socket, a security hook is invoked — and if the hook returns nonzero, the operation is denied. Since kernel 5.7, eBPF programs can attach directly to these hooks. In other words, you can now write security policy in C instead of the SELinux policy language.

Check whether it is enabled by looking for bpf in the kernel lsm list:

cat /sys/kernel/security/lsm
# e.g.: lockdown,capability,landlock,yama,apparmor,bpf

Mini example — a BPF LSM program that denies executing binaries from a specific path:

// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#define EPERM 1

/* bprm_check_security: LSM hook invoked right before program execution */
SEC("lsm/bprm_check_security")
int BPF_PROG(deny_tmp_exec, struct linux_binprm *bprm)
{
    const char *path = bprm->filename;
    char buf[16];

    bpf_probe_read_kernel_str(buf, sizeof(buf), path);

    /* deny executing binaries under /tmp/ */
    if (buf[0] == '/' && buf[1] == 't' && buf[2] == 'm' &&
        buf[3] == 'p' && buf[4] == '/')
        return -EPERM;

    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Returning a negative error code makes the kernel deny the execution. A production policy would replace the path comparison with map-based allow/deny lists and add namespace/cgroup conditions, but the skeleton — "LSM hook plus eBPF equals programmable enforcement" — is exactly this mini example. Tetragon lsmhooks and the mechanism once called KRSI (Kernel Runtime Security Instrumentation) share this same foundation.

The Relationship with seccomp and AppArmor

eBPF runtime security does not replace the existing mechanisms; they live at different layers.

The recommended posture is to stack "basic hygiene plus precision policy." Use a seccomp default profile and AppArmor to cut away clearly unneeded capabilities and shrink the attack surface, then add per-workload behavioral policy and detection with the eBPF layer on top. If one layer is breached, the next one holds — defense in depth.

Kubernetes Deployment and the Event Pipeline

Runtime security agents must exist on every node, so they are deployed as DaemonSets. A typical Falco deployment via Helm:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl=https://hooks.slack.com/services/XXX

Detection events must leave the node. An attacker who owns the node will wipe local logs first. The standard collection pipeline:

[Node] Falco/Tetragon (DaemonSet)
   |  JSON events (stdout or gRPC)
   v
[Collect] Falcosidekick / Fluent Bit / Vector
   |  filtering, routing, buffering
   v
[Store/analyze] Elasticsearch / Loki / S3  --->  SIEM (Splunk etc.)
   |                                          correlation, retention
   v
[Respond] Slack, PagerDuty alerts  /  SOAR automation (Pod isolation etc.)

Tetragon exposes events as JSON logs and a gRPC stream, joining the same pipeline naturally.

# Watch Tetragon events live (tetra CLI)
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o compact

Rule Operations: The War on Noise

The most common reason runtime security adoption fails is not the technology — it is alert fatigue. Operating principles:

1. Baseline first. Before enabling rules, run in detect-only (audit) mode for one to two weeks and collect what "normal" looks like in your environment: which jobs spawn shells, which agents read /etc.
2. Manage exceptions as code. Change Falco macros/lists and Tetragon selectors through a Git repository with reviews. An "ignore for now" button becomes a black hole of exceptions.
3. Separate response channels by severity. CRITICAL pages someone; WARNING goes into a daily digest. Otherwise real incidents drown.
4. Put plenty of context into alerts. Pod, image, command line, and parent process in the alert body dramatically cut triage time.
5. Version the ruleset and run regression tests. Verify in CI, with simulated events (such as an intentional shell exec), that rule changes do not break existing detections.
6. Keep metrics. Track weekly alert volume, false positive rate, and mean triage time to measure the effect of rule tuning.

Bypass Potential and Limits — From the Defender Point of View

eBPF runtime security is not a silver bullet either. Limits to bake into your defensive design (this is a defense-oriented discussion, not a reproduction of attack techniques):

• The asynchronous detection window: in user-space rule evaluation models, the act may complete before detection. Protect truly critical assets with synchronous LSM-based enforcement.
• Visibility gaps: syscall-based detection cannot see behavior that does not surface as syscalls (for example, computation within already-mapped memory). This is why other layers — memory protections, image signing — remain necessary.
• The agent itself is a target: a root-privileged security agent is an attractive target. You need tamper detection for the agent process and configuration (self-protection rules) and least-privilege configuration.
• After kernel compromise, game over: an attacker with kernel privileges can unload eBPF programs. Audit bpf() calls themselves, and keep kernel patch cycles short as a precondition.
• Drops under event floods: if event buffers overflow, detections are missed. Always include drop counters in your monitoring.

In short, runtime security does not replace the other layers (image signing, least privilege, network policy, patching) — it adds the final layer of visibility and control.

Measuring Performance Impact

What to check in load tests before adoption:

# 1. Throughput comparison for syscall-heavy workloads (agent on/off)
#    e.g., a file IO benchmark
fio --name=randrw --rw=randrw --size=1g --runtime=60 --time_based

# 2. p99 comparison for network-heavy workloads
wrk -t8 -c256 -d60s http://service.example/api

# 3. Resource usage of the agent itself
kubectl top pod -n falco
kubectl top pod -n kube-system -l app.kubernetes.io/name=tetragon

# 4. Event drops (Falco internal metrics)
#    enable falco_metrics and watch the n_drops gauge

Empirically, overhead shows up most clearly on workloads with very high syscall rates (databases issuing many small IOs, high-QPS proxies). For typical web workloads it usually stays within single-digit percent, but absolute numbers depend on rule count and event volume — measuring with your own workload is the only trustworthy method.

Adoption Roadmap: Observe, Detect, Enforce

Starting with enforcement always boomerangs into a service incident. Walking the stages is the fast path.

1. Stage 1 — observe (2 to 4 weeks): deploy the agent detect-only and complete the event pipeline (collection, retention, dashboards). Send no alerts yet. This is the baseline collection period.
2. Stage 2 — detect (4 to 8 weeks): enable the default ruleset and tune it to your environment. Manage false positive rate and triage time as metrics, and write severity-tiered response runbooks.
3. Stage 3 — selective enforcement: convert to blocking only the narrow policies proven to have effectively zero false positives. Good first candidates are acts with no legitimate reason to occur in normal workloads: "package manager execution inside containers," "runtime modification of authorized_keys," "connections to known mining pool domains."
4. Stage 4 — automated response: integrate with SOAR or operators to automate responses like Pod isolation (applying a network policy) or node cordoning. Give automation its own circuit breaker (maximum executions per hour).

Adoption Checklist

• Node kernels meet the required versions (BTF present; for BPF LSM, bpf included in the lsm list)
• A baseline collection period was completed in detect-only mode
• Events are shipped off-node (SIEM/log store) with a retention policy
• Rules/policies are version-controlled in Git with mandatory reviews
• Severity-tiered alert channels are wired to response runbooks
• Event drop counters and agent resource usage are monitored
• Enforcement policies are limited to items verified to have zero false positives
• Tamper-detection rules exist for the agent itself
• Performance impact was measured and recorded on representative workloads
• The stack layers on top of lower defenses such as seccomp/AppArmor default profiles

Closing Thoughts

Across this series we have seen one technology wear three faces: in the fundamentals post, as a way to program the kernel safely; in the observability post, as the tool that opens the black box of a system; and here, as the last line of defense for running workloads. The shared foundation is identical — verified programs watching events at kernel hooks and sharing data through maps.

Adopting runtime security is less about tool choice and more about operational maturity. Start with observation, refine detection, and enforce only where confidence has accumulated. Follow that order, and eBPF becomes — without exaggeration — the strongest runtime defense available today.

References

• Falco official documentation
• Falco rules syntax reference
• Tetragon official documentation
• Tetragon TracingPolicy guide
• Linux kernel LSM documentation (kernel.org)
• BPF LSM documentation (kernel.org)
• seccomp documentation (kernel.org)
• Kubernetes security docs — Pod Security Standards
• Cilium official documentation
• ebpf.io — eBPF applications landscape
• Falcosidekick (GitHub)
• MITRE ATT&CK — Containers matrix