Skip to content
Published on

Cilium Network Policy in Practice — Zero Trust from L3 to L7 and DNS

🇰🇷KO🇺🇸EN🇯🇵JA
Split View필사 모드
Authors

Introduction

The default state of a Kubernetes cluster is that every pod can talk to every pod. A payment service pod can reach the internal wiki pod, and a compromised frontend pod can connect straight to the database. The starting point of zero trust is flipping this default: only explicitly allowed communication is possible.

Standard NetworkPolicy gets you part of the way, but in practice you hit walls quickly. You cannot control HTTP paths, you cannot allow external APIs by domain name, and there is no way to see denied traffic. Cilium fills these gaps with CiliumNetworkPolicy (CNP) and CiliumClusterwideNetworkPolicy (CCNP). This article walks through the policy model, L3/L4/L7/DNS policy YAML, the default-deny rollout strategy, the Hubble-driven authoring workflow, and the common pitfalls — in the order you would tackle them in production.

Limits of Standard NetworkPolicy and What CNP Adds

Capabilityk8s NetworkPolicyCiliumNetworkPolicy
L3/L4 (pod selectors, ports)YesYes
L7 HTTP (method, path)NoYes
Kafka topics, gRPC methodsNoYes
DNS-name based egressNoYes (toFQDNs)
Explicit deny rulesNo (allowlist only)Yes (ingressDeny/egressDeny)
Cluster-wide policyNo (namespace scoped)Yes (CCNP)
Host (node) policyNoYes (nodeSelector)
Visibility into denied trafficImplementation dependentImmediate via Hubble
Entity concepts (world, host, etc.)NoYes

An important premise: both CNP and standard NetworkPolicy are evaluated identity-based on the same eBPF datapath. When you mix the two kinds, the result is additive — if either allows, traffic is allowed — so deciding as a team which resource type is your standard reduces operational confusion.

The Policy Model — Identity and Direction

The mental model for Cilium policy evaluation is as follows.

        ingress policy                     egress policy
  "who may come to me"                "where may I go"

  [src identity] ----> (endpoint) ----> [dst identity/CIDR/FQDN]
       |                    |
       |   O(1) verdict in the per-endpoint policy map
       |   key: (identity, port, proto, direction)
       v
  Verdict: ALLOW / DENY / (no policy) default allow*

  * BUT: as soon as any policy selects an endpoint in a
    given direction, that direction flips to default deny (!)

That last line is the rule that causes the most incidents in practice. The moment you apply even one ingress policy to a pod, that pod ingress becomes allowlist mode. Its egress remains fully open until an egress policy is applied. Remember that the flip happens independently per direction.

L3/L4 Policy — Practical YAML

Namespace isolation (allow only the same namespace)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: ns-isolation
  namespace: payments
spec:
  endpointSelector: {}          # applies to all pods in the namespace
  ingress:
    - fromEndpoints:
        - {}                    # allow all pods in the same namespace

An empty endpointSelector means all endpoints in this namespace, and an empty fromEndpoints entry means all endpoints in the same namespace. This single policy blocks all traffic arriving from outside the namespace.

Allowing specific services (frontend → backend 8080)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

Allowing pods from another namespace

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-from-monitoring
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: monitoring
            app: prometheus
      toPorts:
        - ports:
            - port: "9090"
              protocol: TCP

To select across namespaces, use the k8s:io.kubernetes.pod.namespace label. It is more direct than the namespaceSelector of standard NetworkPolicy.

Explicit deny — egressDeny

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: deny-metadata-endpoint
  namespace: shop
spec:
  endpointSelector: {}
  egressDeny:
    - toCIDR:
        - 169.254.169.254/32   # block the cloud metadata endpoint

Deny rules always take precedence over allow rules. They are suited to items that must be blocked no matter what, such as the cloud metadata server.

L7 Policy — HTTP, Kafka, gRPC

How it works: Envoy integration

To operate L7 policy you must understand how the traffic path changes.

Up to L4:   client pod --eBPF--> server pod   (handled in kernel)

L7 policy:  client pod --eBPF--> [Envoy proxy] --> server pod
                                    ^
                       embedded in cilium-agent (or dedicated pod)
                       eBPF redirects only the affected flows
                       parses HTTP, matches rules, 403 on violation

eBPF hands only the flows carrying L7 rules to Envoy, so traffic without L7 policy is still processed entirely in the kernel. For the traffic that is L7-inspected, you must accept the proxy cost: added latency and connection termination.

Restricting HTTP methods and paths

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l7-allow
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: order-api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /api/v1/orders.*
              - method: POST
                path: /api/v1/orders
              - method: GET
                path: /healthz

Requests that do not match (for example DELETE, or the /admin path) still establish a connection but are rejected with HTTP 403. Unlike an L4 block, from the application log perspective this looks like "connection works but 403" — keep that difference in mind when troubleshooting.

Restricting Kafka topics

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kafka-topic-policy
  namespace: streaming
spec:
  endpointSelector:
    matchLabels:
      app: kafka
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: order-service
      toPorts:
        - ports:
            - port: "9092"
              protocol: TCP
          rules:
            kafka:
              - role: produce
                topic: orders
    - fromEndpoints:
        - matchLabels:
            app: settlement-service
      toPorts:
        - ports:
            - port: "9092"
              protocol: TCP
          rules:
            kafka:
              - role: consume
                topic: orders

This example lets the order service only produce to the orders topic, and the settlement service only consume from it. In multi-tenant environments sharing a message broker, you can enforce topic-level isolation at the network layer.

Restricting gRPC methods

gRPC calls run over HTTP/2 in the form "POST /package.Service/Method", so they are expressed with HTTP rules.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: grpc-method-policy
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: inventory-grpc
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: order-api
      toPorts:
        - ports:
            - port: "50051"
              protocol: TCP
          rules:
            http:
              - method: POST
                path: /inventory.InventoryService/CheckStock
              - method: POST
                path: /inventory.InventoryService/ReserveStock

DNS-Based Egress Policy — toFQDNs

Allowing external SaaS APIs by IP is unmaintainable (the IPs change constantly). toFQDNs allows egress by domain name.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-external-apis
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: pg-gateway
  egress:
    # 1) Allow DNS itself and make it observable via the DNS proxy
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
    # 2) The external domains to allow
    - toFQDNs:
        - matchName: api.stripe.com
        - matchPattern: "*.tosspayments.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

How the DNS proxy works

pod --(DNS query)--> [Cilium DNS proxy] --> CoreDNS/external DNS
                          |
                          | intercepts A/AAAA records in the response
                          | "this pod learned api.stripe.com = 54.187.x.x"
                          v
                  registers that IP under the toFQDNs identity
                  in the ipcache/policy maps
                          |
pod --(TCP 443 to 54.187.x.x)--> eBPF allows based on the IP

The key insight: toFQDNs does not inspect the SNI of packets. It remembers the pairing of the name this pod resolved via DNS and the answer IPs, then allows by IP. Therefore, without the DNS rule (block 1 in the YAML above), toFQDNs never works. This is the single most common configuration mistake.

The Default-Deny Rollout Strategy — A Four-Stage Roadmap

Turning on default deny all at once in a live cluster causes outages. The proven incremental sequence is:

Stage 1: Observe      Stage 2: Allow core    Stage 3: Audit-mode    Stage 4: Enforce
Watch all current     paths. Turn findings   deny. Deploy default   Disable audit,
traffic via Hubble →  into policies, apply → deny + policy-audit  → actually block
(2-4 weeks)           without any deny       mode (violations       (namespace by
                      (per-service PRs)      only logged)           namespace)
  1. Observe: collect the real communication matrix per namespace from Hubble metrics and flow logs. Allow at least a month so you do not miss rarely-running traffic such as cron batch jobs.
  2. Author allow policies for core paths: deploy allow policies first, with no deny policies. Nothing is blocked at this stage, so it is safe.
  3. Audit mode: switching endpoints to policy-audit-mode means that even with a default deny policy in place, nothing is actually blocked — traffic that would have been blocked is recorded as a verdict.
# Agent-wide audit mode (helm: policyAuditMode=true)
# or for a specific endpoint only
kubectl -n kube-system exec ds/cilium -- cilium endpoint config 1234 PolicyAuditMode=Enabled

# Observe audit verdicts: find traffic that would have been blocked
hubble observe --verdict AUDIT --namespace payments
  1. Enforce: once audit shows no violations over a set period (e.g. two weeks), disable audit namespace by namespace. Never flip the entire cluster at once.

Deploy default deny itself explicitly, like this:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints: []        # matches nothing = explicit default deny
  egress:
    - toEndpoints: []

The Policy Authoring Workflow — From Hubble to Policy

In practice, policies are not designed in your head; they are derived from observation.

# 1) Observe what the target service actually exchanges
hubble observe --namespace shop --pod shop/order-api --last 1000

# 2) Summarize which identities it talks to
hubble observe --namespace shop --pod shop/order-api \
  --output json | jq -r '.flow.destination.labels | join(",")' | sort | uniq -c

# 3) Confirm nothing would be denied after applying (in audit mode)
hubble observe --verdict AUDIT --namespace shop

# 4) Monitor drops after enforcing
hubble observe --verdict DROPPED --namespace shop --since 1h

For drafting, the network policy editor (editor.networkpolicy.io) is useful. Upload Hubble flows and it visually generates a policy draft based on observed traffic. However, a human must always review the generated draft: legitimate traffic that did not occur during the observation window (failover paths, monthly batches) is missing from it.

Host Policy and External Entities

The entities concept

Reserved identifiers for dealing with the world outside the cluster.

EntityMeaning
worldEverything outside the cluster (including the internet)
clusterAll endpoints inside the cluster
hostThe local node itself
remote-nodeThe other nodes
kube-apiserverThe API server
healthCilium health-check endpoints
# Egress example allowing only in-cluster traffic plus the API server
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cluster-only-egress
  namespace: internal-tools
spec:
  endpointSelector: {}
  egress:
    - toEntities:
        - cluster
        - kube-apiserver

Host policy (CCNP + nodeSelector)

The traffic of the node itself can also be policed. An example allowing only SSH and the kubelet port:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: host-fw-control-plane
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: ""
  ingress:
    - fromEntities:
        - cluster
    - fromCIDR:
        - 10.50.0.0/24          # management network
      toPorts:
        - ports:
            - port: "22"
              protocol: TCP
            - port: "6443"
              protocol: TCP

A badly written host policy can lock you out of the entire node, so it must be validated thoroughly in policy-audit-mode before enforcing. The host firewall feature (hostFirewall.enabled) must also be enabled in helm.

Automating Policy Testing and Verification

Policies are code too. Verification steps you can put into CI:

# 1) Schema/syntax validation (CI stage)
kubectl apply --dry-run=server -f policies/

# 2) Simulation: pre-verdict whether specific traffic would be allowed
kubectl -n kube-system exec ds/cilium -- \
  cilium policy trace --src-k8s-pod shop:frontend-abc --dst-k8s-pod shop:backend-xyz --dport 8080

# 3) Real cluster integration test (staging)
cilium connectivity test --test pod-to-pod,pod-to-world

# 4) Regression test: compare drop counts after deployment
hubble observe --verdict DROPPED --since 10m --output json | jq length

In a GitOps setup, enforce steps 1 and 2 in the pipeline for PRs against the policy directory, and automatically run steps 3 and 4 in staging after merge.

Common Pitfalls and Anti-Patterns

  1. toFQDNs without a DNS rule: as shown above, without the DNS proxy rule, toFQDNs never matches. The symptom is "DNS works but the connection is refused", or the reverse.
  2. DNS TTL and IP churn: toFQDNs is based on DNS answers, so an application holding a stale DNS cache and connecting to an expired IP can get denied. For targets with fast IP rotation like CDNs, use broader matchPattern entries and review the agent FQDN TTL settings (tofqdns-idle-connection-grace-period and friends).
  3. Locking down system namespaces along with the rest: hastily applying default deny to kube-system, monitoring, or ingress controller namespaces kills cluster functionality itself. Treat them as a separate track — last, and most carefully.
  4. Blocking health checks/probes: kubelet liveness/readiness probes come from the node (host identity). Forget the host entity in an ingress policy and pods fall into endless restarts. Cilium auto-allows probe traffic by default, but combinations with host policy can break this.
  5. The mystery of only new connections breaking: right after applying a policy, existing connections remain in conntrack and keep working while only new connections are blocked (or vice versa). "It works right now" is not evidence that "the policy allows it".
  6. Label typos and the empty-selector misunderstanding: a typo in matchLabels fails silently as "selects nothing". After applying, always check the number of endpoints in enforcing state with cilium endpoint list.
  7. Spraying L7 policy on all traffic: sending everything through Envoy raises latency and CPU together. The standard practice is selective application only at boundaries that truly need L7 control (externally exposed APIs, sensitive data access).

A Real-World Scenario — PCI DSS Style Isolation

A pattern for isolating workloads handling card payment data (the CDE) inside a cluster. The regulatory phrasing is a generalized example; confirm actual assessment requirements with your QSA.

# 1) CDE namespace: default deny + explicit allows only
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cde-lockdown
  namespace: cde-payments
spec:
  endpointSelector: {}
  ingress:
    # Only payment requests from the API gateway, with L7 path restrictions
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: gateway
            app: api-gateway
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
          rules:
            http:
              - method: POST
                path: /v1/payments
              - method: GET
                path: /v1/payments/[0-9a-f-]+
  egress:
    # DNS (observed via the proxy)
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.internal.example.com"
              - matchName: api.pgprovider.com
    # The internal ledger DB (dedicated namespace)
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: cde-db
            app: ledger-db
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP
    # Only the external payment provider API
    - toFQDNs:
        - matchName: api.pgprovider.com
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

Add long-term retention of Hubble flow logs to submit as audit evidence that isolation was actually maintained (covered in the next article), and you have a complete technical answer to network segmentation requirements.

Adoption Checklist

  • Has the team standardized on either standard NetworkPolicy or CNP
  • Does the whole team understand that one policy flips that direction to default deny
  • Have you secured a Hubble observation window (at least 2-4 weeks, covering batch cycles)
  • Does every toFQDNs policy have a paired DNS proxy rule
  • Is a policy-audit-mode validation stage part of the deployment procedure
  • Are system namespaces split into a separate track
  • Are dry-run and policy trace enforced via CI on policy PRs
  • Are DROPPED verdict alerts wired into monitoring after enforcement
  • Are host policies enforced only after audit validation
  • Is the policy change history traceable via GitOps

Closing

The power of Cilium policy lies not only in its expressiveness (L7, FQDN) but in the fact that observation and policy come from the same datapath. The flows Hubble shows you and the flows the policy engine judges are identical, so the loop of observe → codify → audit → enforce closes without guesswork. Zero trust is not a single big bang; it is the practice of running this loop one namespace at a time. In the next article we cover Hubble, the observation axis of this loop, and ClusterMesh, which extends it to multiple clusters.

References

Discuss on Twitter • View on GitHub