Cilium Network Policy Engine: L3/L4/L7 Filtering Implementation

Overview

Cilium's network policy engine extends Kubernetes NetworkPolicy to provide fine-grained traffic control at L3/L4/L7 levels. It combines eBPF with Envoy proxy in a hybrid architecture that achieves both high performance and flexibility.

1. Policy Repository and Rule Compilation

1.1 Policy Sources

Cilium collects network policies from multiple sources:

Policy sources:
1. Kubernetes NetworkPolicy
   - Standard L3/L4 policies
   - Automatically translated and applied by Cilium

2. CiliumNetworkPolicy (CNP)
   - Namespace-scoped Cilium extended policy
   - Advanced features: L7, FQDN, Identity-based

3. CiliumClusterwideNetworkPolicy (CCNP)
   - Cluster-wide policies
   - Default deny policies, common security rules

4. CiliumEnvoyConfig (CEC)
   - Envoy proxy L7 traffic management rules

1.2 Policy Repository

The policy repository within the Agent manages all policies in a unified manner:

Policy collection flow:

K8s API Server
    |
    v
[Watcher: NetworkPolicy]  ->  Policy Repository
[Watcher: CNP]             ->  Policy Repository
[Watcher: CCNP]            ->  Policy Repository
                                    |
                                    v
                               [Policy Computation Engine]
                                    |
                                    v
                               [Per-Endpoint Policy Set]
                                    |
                                    v
                               [BPF Program Compilation]

1.3 Translation from Policy to BPF Maps

CiliumNetworkPolicy YAML
    |
    v
Parse and normalize policy
    |
    v
Selector matching: determine affected endpoints
    |
    v
Generate Identity-based allow list
    |
    v
Update BPF policy maps:
  Add entries to cilium_policy_<endpoint_id> map
  Key: (Identity, port, protocol)
  Value: (allow/deny, proxy port)
    |
    v
Regenerate endpoint BPF programs (if needed)

2. Identity-Based vs IP-Based Enforcement

2.1 Identity-Based Enforcement

This is Cilium's default policy enforcement method, using numeric Identities instead of IP addresses:

Traditional IP-based firewall:
  Allow: 10.244.1.5 -> 10.244.2.10:8080
  Problem: IP changes when Pod restarts

Cilium Identity-based:
  Allow: Identity 48291 (app=frontend) -> Identity 52103 (app=backend):8080
  Advantage: Policy persists regardless of IP changes

# Check policy map contents (by endpoint ID)
cilium bpf policy get 1234

# Example output:
# POLICY   DIRECTION   IDENTITY   PORT/PROTO   PROXY PORT   ENTRY TYPE
# Allow    Ingress     48291      8080/TCP     0            allow
# Allow    Ingress     52103      443/TCP      0            allow
# Allow    Egress      0          53/UDP       0            allow (DNS)

2.2 Policy Map Structure

// Policy map key (conceptual)
struct policy_key {
    __u32 identity;      // Source/destination Identity
    __u16 dport;         // Destination port (0 = all ports)
    __u8  protocol;      // TCP, UDP, ICMP
    __u8  direction;     // ingress(1), egress(2)
};

// Policy map value
struct policy_entry {
    __u8  action;        // allow(1), deny(2), audit(3)
    __u16 proxy_port;    // L7 proxy port (0 = no proxy)
    __u8  auth_type;     // Auth type (mTLS, etc.)
    __u64 packets;       // Matched packet counter
    __u64 bytes;         // Matched byte counter
};

2.3 Policy Lookup Algorithm

Policy lookup on packet receive:

1. Look up Identity from source IP (ipcache)
2. Search in policy map for (Identity, Port, Protocol) combination
3. Exact match attempt:
   - Check (src_identity, dst_port, protocol)
4. Wildcard match:
   - Check (src_identity, ANY_PORT, protocol)
   - Check (ANY_IDENTITY, dst_port, protocol)
5. Check Deny rules first:
   - If Deny exists, unconditionally deny
6. Result: Allow / Deny / Audit

3. L3/L4 Enforcement: High-Speed Filtering in eBPF

3.1 L3 Policy (Network Layer)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l3-policy-example
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
    - fromCIDR:
        - 10.0.0.0/8
    - fromEntities:
        - world
  egress:
    - toEndpoints:
        - matchLabels:
            app: database
    - toCIDR:
        - 0.0.0.0/0
      toCIDRSet:
        - cidr: 0.0.0.0/0
          except:
            - 169.254.169.254/32

3.2 L4 Policy (Transport Layer)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l4-policy-example
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: '8080'
              protocol: TCP
            - port: '8443'
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            app: database
      toPorts:
        - ports:
            - port: '5432'
              protocol: TCP

3.3 L3/L4 Enforcement Flow in eBPF

Packet: src=10.244.1.5:34567 dst=10.244.2.10:8080 TCP

1. [from-container or to-container BPF program]

2. Source Identity lookup:
   ipcache[10.244.1.5] -> Identity: 48291

3. Policy map lookup:
   Search in policy_map[endpoint_id]:
   key = (identity=48291, port=8080, proto=TCP, dir=ingress)

4. Result:
   - Allow: Continue packet delivery
   - Deny: Drop packet + monitor event
   - L7 Redirect: Redirect to Envoy proxy port

4. L7 Enforcement: Envoy Proxy Integration

4.1 L7 Policy Overview

L7 policies provide application protocol-level filtering:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-http-policy
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: '8080'
              protocol: TCP
          rules:
            http:
              - method: GET
                path: '/api/v1/users'
              - method: GET
                path: '/api/v1/products'
              - method: POST
                path: '/api/v1/orders'
                headers:
                  - 'Content-Type: application/json'

4.2 L7 Redirect Mechanism

Traffic flow with L7 policy applied:

Pod A -> [from-container BPF]
    |
    v (policy map: L7 proxy redirect needed)
    |
    v
[tc redirect to Envoy proxy port]
    |
    v
[Envoy Proxy (1 per node)]
  - HTTP/gRPC/Kafka parsing
  - L7 rule matching
  - Allow/deny decision
    |
    v (if allowed)
    |
    v
[Forward to destination via BPF]
    |
    v
Pod B

4.3 Supported Protocols

Protocol	Filterable Fields
HTTP	method, path, headers, host
gRPC	service, method
Kafka	topic, clientID, apiKey, apiVersion
DNS	query name pattern (matchName, matchPattern)

4.4 Kafka L7 Policy Example

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kafka-policy
spec:
  endpointSelector:
    matchLabels:
      app: kafka-consumer
  egress:
    - toEndpoints:
        - matchLabels:
            app: kafka-broker
      toPorts:
        - ports:
            - port: '9092'
              protocol: TCP
          rules:
            kafka:
              - role: consume
                topic: 'orders'
              - role: consume
                topic: 'events'

5. FQDN Policies: DNS Proxy Integration

5.1 How FQDN Policies Work

FQDN policy flow:

1. Policy definition:
   toFQDNs:
     - matchName: "api.example.com"

2. Cilium DNS proxy activation:
   - Transparently intercept Pod DNS traffic
   - Learn IP addresses from DNS responses

3. IP cache update:
   DNS response: api.example.com -> 203.0.113.10, 203.0.113.11
   |
   v
   Add IP -> FQDN mapping to ipcache
   Add allow rules for those IPs to BPF policy map

4. Traffic allow/block:
   - Only allow traffic to learned IPs
   - Mapping expires and refreshes based on DNS TTL

5.2 FQDN Policy Example

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: fqdn-policy
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
    - toFQDNs:
        - matchName: 'api.external-service.com'
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
    - toFQDNs:
        - matchPattern: '*.storage.googleapis.com'
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: UDP
          rules:
            dns:
              - matchPattern: '*'

6. CiliumNetworkPolicy vs CiliumClusterwideNetworkPolicy

6.1 Scope Differences

CiliumNetworkPolicy (CNP):
  - Namespace scope
  - endpointSelector selects Pods in the same namespace
  - Cross-namespace via namespaceSelector in fromEndpoints/toEndpoints

CiliumClusterwideNetworkPolicy (CCNP):
  - Cluster scope
  - Can select specific nodes with nodeSelector
  - Applies across all namespaces
  - Suitable for cluster-level default policies

6.2 CCNP Example

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  endpointSelector: {}
  ingress:
    - fromEntities:
        - cluster

6.3 Host Policy

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: host-firewall
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ''
  ingress:
    - fromEntities:
        - remote-node
        - health
      toPorts:
        - ports:
            - port: '10250'
              protocol: TCP
    - fromCIDR:
        - 10.0.0.0/8
      toPorts:
        - ports:
            - port: '22'
              protocol: TCP

7. Policy Audit Mode

7.1 Audit Mode Operation

In audit mode, policy-violating traffic is not blocked but only logged:

# Set endpoint to audit mode
cilium endpoint config 1234 PolicyAuditMode=Enabled

# Global audit mode
cilium config PolicyAuditMode=true

# Check audit logs
cilium monitor --type policy-verdict

# Example output:
# Policy verdict log: flow ... action audit
# -> Would be DROPPED by policy (ingress)
# Identity: 48291 -> 52103, port 8080/TCP

7.2 Transitioning from Audit to Enforcement

Audit mode workflow:

Step 1: Enable audit mode
  -> All traffic allowed, violations logged

Step 2: Analyze traffic patterns with Hubble
  -> Identify actual required communication paths

Step 3: Write policies and compare with audit results
  -> Verify no unintended blocks

Step 4: Disable audit mode
  -> Actual policy enforcement begins

8. Endpoint Regeneration and Policy Changes

8.1 Regeneration Process on Policy Change

Policy change detected:
    |
    v
Determine affected endpoints:
  - All endpoints matching the selector
  - Including indirect impact via Identity references
    |
    v
Recompute policy maps:
  - Generate new allow/deny rule set
  - Compare with existing policy map (apply diff only)
    |
    v
Update BPF maps:
  - Add/delete entries in cilium_policy map
  - Atomic updates to minimize packet loss
    |
    v
Recompile BPF programs (if needed):
  - When L7 redirect is added/removed
  - When program logic changes are required
    |
    v
Update endpoint state: Ready

8.2 Regeneration Performance Optimization

Optimization techniques:

1. Incremental updates:
   - Update only BPF maps instead of full recompilation
   - Most policy changes only need map updates

2. Batch regeneration:
   - Batch multiple policy changes in a short timeframe
   - Prevent unnecessary duplicate regenerations

3. Selective regeneration:
   - Only regenerate affected endpoints
   - Precisely calculate impact scope based on Identity

9. Deny Policies

9.1 Deny Rule Behavior

Cilium supports explicit Deny rules, which always take precedence over Allow:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: deny-specific-cidr
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egressDeny:
    - toCIDR:
        - 169.254.169.254/32
      toPorts:
        - ports:
            - port: '80'
              protocol: TCP
  egress:
    - toCIDR:
        - 0.0.0.0/0
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP

9.2 Policy Priority

Policy evaluation order:

1. Check Deny rules
   - If matching Deny rule exists, immediately deny
   - Ignore Allow rules

2. Check Allow rules
   - If matching Allow rule exists, allow

3. Default behavior
   - If policies exist for that direction (ingress/egress): default deny
   - If no policies at all: default allow

10. Policy Troubleshooting

10.1 Debugging Commands

# Check applied policies
cilium policy get

# Per-endpoint policy state
cilium endpoint get 1234

# Policy map contents
cilium bpf policy get 1234

# Look up labels from Identity
cilium identity get 48291

# Policy verdict monitoring
cilium monitor --type policy-verdict

# Check dropped packets
cilium monitor --type drop

10.2 Common Issues and Solutions

Issue: Pod-to-Pod communication blocked
Possible causes:
1. Identity not correctly assigned
   -> Verify with cilium identity list
2. Policy selector is incorrect
   -> Check applied policies with cilium policy get
3. DNS egress policy missing
   -> DNS allow is required when using FQDN policies

Issue: L7 policy not applied
Possible causes:
1. Envoy proxy not functioning properly
   -> Check Proxy status in cilium status
2. Syntax error in rules under toPorts
   -> Check parsed result with cilium policy get

Summary

Cilium's network policy engine provides robust traffic control through a multi-layered security model:

Identity-Based Enforcement: Applies policies based on labels, not IP addresses, ensuring stability in dynamic environments
eBPF High-Speed Filtering: L3/L4 policies applied directly in the kernel for minimal latency
Envoy L7 Integration: Application-level filtering for HTTP, gRPC, Kafka
FQDN Policies: Domain-based egress control through DNS proxy
Deny Precedence: Explicit deny rules always take priority for stronger security
Audit Mode: Assess impact before enforcing policies
Incremental Updates: Apply policy changes with minimal regeneration