+-----------------------------------------------------------+
|                   Hubble Architecture                      |
+-----------------------------------------------------------+
|                                                           |
|  Node 1             Node 2             Node 3             |
|  +-----------+     +-----------+     +-----------+        |
|  | Cilium    |     | Cilium    |     | Cilium    |        |
|  | Agent     |     | Agent     |     | Agent     |        |
|  |  +------+ |     |  +------+ |     |  +------+ |        |
|  |  |Hubble| |     |  |Hubble| |     |  |Hubble| |        |
|  |  |Server| |     |  |Server| |     |  |Server| |        |
|  |  +------+ |     |  +------+ |     |  +------+ |        |
|  +-----------+     +-----------+     +-----------+        |
|       |                 |                 |               |
|       +--------+--------+--------+--------+               |
|                |                 |                         |
|          +-----v-----+    +-----v-----+                   |
|          |  Hubble   |    | Hubble UI |                   |
|          |  Relay    |    |           |                   |
|          +-----------+    +-----------+                   |
|                |                                          |
|          +-----v-----+                                    |
|          | Prometheus|                                    |
|          | Grafana   |                                    |
|          +-----------+                                    |
+-----------------------------------------------------------+

eBPF datapath events:
  [Packet processing] -> [Policy verdict] -> [Conntrack events]
       |                      |                    |
       v                      v                    v
  [Perf Event Ring Buffer]
       |
       v
  [Hubble Server: Event parser]
       |
       v
  [Flow ring buffer (in-memory)]
       |
       v
  [gRPC server: Stream flows to clients]

# Ring buffer size configuration
# Default: 4095 flows
# Config: --hubble-buffer-size=16383

# Check current ring buffer status
hubble status
# Nodes:
#   node-1: Connected, Flows: 4095/4095 (100.00%), ...
#   node-2: Connected, Flows: 3821/4095 (93.31%), ...

Key fields in a flow record:

- time: Event timestamp
- source:
    identity: Source Identity
    namespace: Source namespace
    labels: Source labels
    pod_name: Source Pod name
- destination:
    identity: Destination Identity
    namespace: Destination namespace
    labels: Destination labels
    pod_name: Destination Pod name
- IP:
    source: Source IP
    destination: Destination IP
- l4:
    TCP/UDP:
      source_port: Source port
      destination_port: Destination port
- l7:
    type: HTTP/DNS/Kafka
    http:
      method: GET/POST/...
      url: Request URL
      code: Response code
- verdict: FORWARDED/DROPPED/AUDIT/REDIRECTED
- drop_reason: Drop reason (when verdict is DROPPED)
- Type: L3_L4/L7/TRACE/DROP

Relay connection management:

1. Discover all Cilium Agents (Hubble Servers) in the cluster
2. Connect to each node's Hubble gRPC service
3. Distribute flow requests to all nodes
4. Aggregate responses and deliver to clients
5. Automatic reconnection on node add/remove

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hubble-relay
  namespace: kube-system
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: hubble-relay
          image: quay.io/cilium/hubble-relay:v1.16.0
          ports:
            - containerPort: 4245
              name: grpc
          args:
            - serve
            - --peer-service=unix:///var/run/cilium/hubble.sock
            - --listen-address=:4245

# Local port forwarding to Hubble Relay
cilium hubble port-forward &

# Then access via hubble CLI
hubble observe
hubble status

Key features:
1. Service Map
   - Visualize service-to-service communication as topology graph
   - Real-time traffic flow display
   - Color-coded normal/abnormal connections

2. Flow Table
   - Real-time network flow listing
   - Filtering (namespace, labels, verdict)
   - Detailed information for each flow

3. Policy Visualization
   - Display allow/deny status from policies
   - Highlight dropped traffic

# Observe L3/L4 flows
hubble observe --type l3/l4

# Example output:
# Mar 20 10:15:32.123 default/frontend-abc -> default/backend-xyz
#   TCP SYN 10.244.1.5:34567 -> 10.244.2.10:8080
#   verdict: FORWARDED

# Track TCP connections
hubble observe --protocol tcp --to-port 8080

# UDP traffic
hubble observe --protocol udp --to-port 53

# Observe HTTP flows
hubble observe --type l7 --protocol http

# Example output:
# Mar 20 10:15:32.456 default/frontend-abc -> default/api-server-xyz
#   HTTP GET /api/v1/users
#   Response: 200 OK (23ms)

# DNS flows
hubble observe --type l7 --protocol dns

# Kafka flows
hubble observe --type l7 --protocol kafka

# Observe only dropped flows
hubble observe --verdict DROPPED

# Example output:
# Mar 20 10:15:34.012 default/untrusted-app -> default/backend-xyz
#   TCP 10.244.3.5:45678 -> 10.244.2.10:8080
#   verdict: DROPPED (Policy denied)

# Filter by specific drop reason
hubble observe --verdict DROPPED --drop-reason POLICY_DENIED

# Observe all flows (real-time streaming)
hubble observe -f

# Last 100 flows
hubble observe --last 100

# Specific time range
hubble observe --since 5m
hubble observe --since "2026-03-20T10:00:00Z" --until "2026-03-20T10:30:00Z"

# Namespace filtering
hubble observe --namespace production

# Pod name filtering
hubble observe --from-pod production/frontend-abc
hubble observe --to-pod production/backend-xyz

# Label filtering
hubble observe --from-label "app=frontend"
hubble observe --to-label "app=backend"

# IP filtering
hubble observe --from-ip 10.244.1.5
hubble observe --to-ip 10.244.2.10

# Port filtering
hubble observe --to-port 8080

# Verdict filtering
hubble observe --verdict FORWARDED
hubble observe --verdict DROPPED

# Combined filters
hubble observe \
  --namespace production \
  --from-label "app=frontend" \
  --to-label "app=backend" \
  --to-port 8080 \
  --verdict FORWARDED

# Default output (human-readable)
hubble observe

# JSON output
hubble observe -o json

# Compact output
hubble observe -o compact

# Dictionary output
hubble observe -o dict

# jsonpb (Protocol Buffers JSON format)
hubble observe -o jsonpb

# Overall Hubble status
hubble status

# Example output:
# Healthcheck (via localhost:4245): Ok
# Current/Max Flows: 16383/16383 (100.00%)
# Flows/s: 245.32
# Connected Nodes: 3/3

# Per-node status
hubble list nodes

groups:
  - name: hubble-alerts
    rules:
      - alert: HighDropRate
        expr: rate(hubble_drop_total[5m]) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High packet drop rate detected'

      - alert: HTTPErrorRate
        expr: |
          rate(hubble_http_responses_total{status=~"5.."}[5m])
          / rate(hubble_http_responses_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'HTTP 5xx error rate exceeds 5%'

      - alert: DNSLatencyHigh
        expr: |
          histogram_quantile(0.99, rate(hubble_dns_response_time_seconds_bucket[5m]))
          > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'DNS response latency p99 exceeds 500ms'

// Hubble Observer API (simplified)
service Observer {
    // Stream flows
    rpc GetFlows(GetFlowsRequest) returns (stream GetFlowsResponse);

    // Hubble server status
    rpc ServerStatus(ServerStatusRequest) returns (ServerStatusResponse);

    // Node list
    rpc GetNodes(GetNodesRequest) returns (GetNodesResponse);
}

# Direct API call with gRPCurl
grpcurl -plaintext localhost:4245 observer.Observer/ServerStatus

# Stream flows
grpcurl -plaintext -d '{}' localhost:4245 observer.Observer/GetFlows

Hubble gRPC API use cases:

1. Custom Dashboards
   - Collect specific business metrics
   - Custom visualizations

2. Automated Security Analysis
   - Detect abnormal traffic patterns
   - Automatic policy violation alerts

3. Audit Logging
   - Record network activity for compliance
   - Export to external systems for long-term storage

4. Service Mesh Integration
   - Monitor service-to-service latency
   - Error tracking and analysis

Overhead from enabling Hubble:

CPU:
  - Basic L3/L4 observation: Minimal (~1-2%)
  - L7 observation (HTTP, etc.): Depends on Envoy proxy overhead
  - High-traffic environments: Flow parsing and ring buffer management

Memory:
  - Proportional to ring buffer size
  - Default 4095 flows x ~500 bytes per flow = ~2MB
  - 16383 flows: ~8MB

Network:
  - gRPC streaming traffic to Relay
  - Proportional to number of observing clients

# Ring buffer size adjustment
--hubble-buffer-size=16383

# Event queue size
--hubble-event-queue-size=0  # 0 = auto

# Enable/disable specific metrics
--hubble-metrics=dns,drop,tcp,flow

# Monitor specific events
--hubble-monitor-events="drop:true;trace:true;l7:true"

Considerations for large clusters:

1. Relay resource limits
   - Set appropriate CPU/memory requests/limits
   - Relay load increases with node count

2. Metric cardinality management
   - Minimize label contexts to control metric count
   - Disable unnecessary metrics

3. Flow retention strategy
   - Balance ring buffer size with traffic volume
   - Export to external systems for long-term retention

4. Network bandwidth
   - Consider gRPC traffic between Relay and Agents
   - Limit impact of large observation queries

# Diagnose Pod-to-Pod connectivity issues
hubble observe --from-pod default/app-a --to-pod default/app-b

# Analyze dropped traffic causes
hubble observe --verdict DROPPED --from-pod default/app-a

# Check DNS resolution issues
hubble observe --type l7 --protocol dns --from-pod default/app-a

# Check traffic to specific service
hubble observe --to-label "app=database" --to-port 5432

# Monitor policy violation traffic
hubble observe --verdict DROPPED --drop-reason POLICY_DENIED

# Track egress traffic to external
hubble observe --to-identity reserved:world

# All ingress traffic to sensitive namespace
hubble observe --namespace sensitive-ns --traffic-direction ingress