Fluent Bit 완벽 가이드: 경량 로그 프로세서의 아키텍처, 설정, Kubernetes 연동까지 총정리

+------------------------------------------------------------------+
|                        Fluent Bit Engine                         |
|                                                                  |
|  +--------+   +--------+   +--------+   +--------+   +--------+ |
|  | Input  |-->| Parser |-->| Filter |-->| Buffer |-->| Output | |
|  +--------+   +--------+   +--------+   +--------+   +--------+ |
|  | tail   |   | json   |   | k8s    |   | memory |   | es     | |
|  | systemd|   | regex  |   | grep   |   | filesys|   | loki   | |
|  | forward|   | logfmt |   | modify |   |        |   | s3     | |
|  | http   |   | cri    |   | lua    |   |        |   | kafka  | |
|  | tcp    |   | docker |   | nest   |   |        |   | stdout | |
|  +--------+   +--------+   +--------+   +--------+   +--------+ |
|                                                                  |
|  [Scheduler]  [Router / Tag Matching]  [HTTP Server / Monitoring]|
+------------------------------------------------------------------+

  [Data Source]
       |
       v
  +---------+     +---------+     +---------+     +---------+     +---------+
  |  INPUT  | --> | PARSER  | --> | FILTER  | --> | BUFFER  | --> | OUTPUT  |
  |         |     |         |     |         |     |         |     |         |
  | 데이터   |     | 비정형   |     | 데이터   |     | 메모리/  |     | 최종    |
  | 수집     |     | -> 정형  |     | 가공     |     | 디스크   |     | 전달    |
  +---------+     +---------+     +---------+     +---------+     +---------+
       |               |               |               |               |
    Tag 부여       구조화 변환      enrichment      안정성 보장    목적지 전송
                                   필터링                          (Tag 매칭)

Pipeline A: [tail: app-*.log] --tag:app--> [filter:k8s] --> [output:elasticsearch]
Pipeline B: [tail: sys-*.log] --tag:sys--> [filter:grep] --> [output:loki]
Pipeline C: [forward:24224]   --tag:fwd--> [filter:lua]  --> [output:s3]

# GPG 키 및 저장소 추가
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# 또는 수동 설치
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
echo "deb https://packages.fluentbit.io/ubuntu/$(lsb_release -cs) $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/fluent-bit.list

sudo apt-get update
sudo apt-get install -y fluent-bit

# 서비스 시작
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit

cat > /etc/yum.repos.d/fluent-bit.repo << 'EOF'
[fluent-bit]
name=Fluent Bit
baseurl=https://packages.fluentbit.io/centos/$releasever/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
EOF

sudo yum install -y fluent-bit
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit

# 최신 버전 실행
docker run -ti cr.fluentbit.io/fluent/fluent-bit:latest

# 설정 파일 마운트
docker run -ti \
  -v /path/to/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
  -v /var/log:/var/log \
  cr.fluentbit.io/fluent/fluent-bit:latest \
  /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.yaml

# Docker Compose 예시
cat > docker-compose.yaml << 'EOF'
version: '3.8'
services:
  fluent-bit:
    image: cr.fluentbit.io/fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml
      - /var/log:/var/log:ro
    ports:
      - "2020:2020"   # HTTP 모니터링
      - "24224:24224"  # Forward 프로토콜
EOF

brew install fluent-bit

# 실행
fluent-bit -c /opt/homebrew/etc/fluent-bit/fluent-bit.conf

# 또는 YAML 설정 파일로 실행
fluent-bit -c /path/to/fluent-bit.yaml

# Helm 저장소 추가
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

# 기본 설치
helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace

# values.yaml 커스터마이징으로 설치
helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace \
  -f custom-values.yaml

# 업그레이드
helm upgrade fluent-bit fluent/fluent-bit \
  --namespace logging \
  -f custom-values.yaml

# GitHub Releases에서 바이너리 다운로드
FLUENT_BIT_VERSION=3.2.2
wget https://github.com/fluent/fluent-bit/releases/download/v${FLUENT_BIT_VERSION}/fluent-bit-${FLUENT_BIT_VERSION}-linux-x86_64.tar.gz

tar xzf fluent-bit-${FLUENT_BIT_VERSION}-linux-x86_64.tar.gz
cd fluent-bit-${FLUENT_BIT_VERSION}-linux-x86_64/

# 실행
./bin/fluent-bit -c conf/fluent-bit.yaml

# 버전 확인
./bin/fluent-bit --version

# fluent-bit.conf - Classic INI 형식

[SERVICE]
    Flush         5
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    Hot_Reload    On

[INPUT]
    Name          tail
    Path          /var/log/containers/*.log
    Parser        cri
    Tag           kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Refresh_Interval 10
    DB            /var/log/flb_kube.db

[FILTER]
    Name          kubernetes
    Match         kube.*
    Kube_URL      https://kubernetes.default.svc:443
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log     On
    K8S-Logging.Parser On
    K8S-Logging.Exclude On

[OUTPUT]
    Name          es
    Match         kube.*
    Host          elasticsearch.logging.svc.cluster.local
    Port          9200
    Logstash_Format On
    Logstash_Prefix kube
    Retry_Limit   False

# fluent-bit.yaml - YAML 형식 (v3.2+ 표준)
service:
  flush: 5
  daemon: off
  log_level: info
  parsers_file: parsers.conf
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      parser: cri
      tag: kube.*
      mem_buf_limit: 5MB
      skip_long_lines: on
      refresh_interval: 10
      db: /var/log/flb_kube.db

  filters:
    - name: kubernetes
      match: kube.*
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      merge_log: on
      k8s-logging.parser: on
      k8s-logging.exclude: on

  outputs:
    - name: es
      match: kube.*
      host: elasticsearch.logging.svc.cluster.local
      port: 9200
      logstash_format: on
      logstash_prefix: kube
      retry_limit: false

# YAML 환경변수 예시
pipeline:
  outputs:
    - name: es
      match: '*'
      host: ${ELASTICSEARCH_HOST}
      port: ${ELASTICSEARCH_PORT}
      http_user: ${ES_USER}
      http_passwd: ${ES_PASSWORD}
      tls: ${ES_TLS_ENABLED}

# Classic 환경변수 예시
[OUTPUT]
    Name          es
    Match         *
    Host          ${ELASTICSEARCH_HOST}
    Port          ${ELASTICSEARCH_PORT}
    HTTP_User     ${ES_USER}
    HTTP_Passwd   ${ES_PASSWORD}

pipeline:
  outputs:
    - name: es
      http_passwd: file:///run/secrets/es-password

# fluent-bit.conf (Classic)
[SERVICE]
    Flush 5

@INCLUDE inputs.conf
@INCLUDE filters.conf
@INCLUDE outputs.conf

# fluent-bit.yaml
includes:
  - inputs.yaml
  - filters.yaml
  - outputs.yaml

service:
  flush: 5

pipeline:
  inputs:
    - name: tail
      tag: app.logs
      path: /var/log/app/*.log
      path_key: filename # 파일 경로를 레코드에 포함
      exclude_path: /var/log/app/debug.log # 특정 파일 제외
      parser: json # 기본 파서
      db: /var/log/flb_app.db # offset 저장 DB (재시작 시 이어서 수집)
      db.sync: normal # DB 동기화 모드
      refresh_interval: 10 # 파일 목록 갱신 주기 (초)
      read_from_head: false # true: 파일 처음부터 읽기
      skip_long_lines: on # 매우 긴 라인 스킵
      mem_buf_limit: 5MB # 메모리 버퍼 한도
      rotate_wait: 5 # 로테이션 대기 시간 (초)
      multiline.parser: docker, cri # 멀티라인 파서

pipeline:
  inputs:
    - name: systemd
      tag: host.systemd
      systemd_filter: _SYSTEMD_UNIT=docker.service
      systemd_filter: _SYSTEMD_UNIT=kubelet.service
      read_from_tail: on
      strip_underscores: on
      db: /var/log/flb_systemd.db

pipeline:
  inputs:
    - name: forward
      tag: forward.incoming
      listen: 0.0.0.0
      port: 24224
      buffer_chunk_size: 1M
      buffer_max_size: 6M

pipeline:
  inputs:
    # HTTP 수신
    - name: http
      tag: http.logs
      listen: 0.0.0.0
      port: 9880
      successful_response_code: 201

    # TCP 수신
    - name: tcp
      tag: tcp.logs
      listen: 0.0.0.0
      port: 5170
      format: json

    # UDP 수신
    - name: udp
      tag: udp.logs
      listen: 0.0.0.0
      port: 5170
      format: json

pipeline:
  inputs:
    - name: kubernetes_events
      tag: kube.events
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval_sec: 1
      retention_time: 1h

pipeline:
  inputs:
    # 노드 메트릭 수집
    - name: node_exporter_metrics
      tag: node.metrics
      scrape_interval: 30

    # Prometheus 엔드포인트 스크레이핑
    - name: prometheus_scrape
      tag: prom.metrics
      host: 127.0.0.1
      port: 9090
      metrics_path: /metrics
      scrape_interval: 10s

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fb.metrics
      scrape_interval: 30
      scrape_on_start: true

# parsers.conf

# Nginx 에러 로그 파서
[PARSER]
    Name        nginx_error
    Format      regex
    Regex       ^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)$
    Time_Key    time
    Time_Format %Y/%m/%d %H:%M:%S

# Spring Boot 로그 파서
[PARSER]
    Name        spring_boot
    Format      regex
    Regex       ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+)\s+(?<level>\w+)\s+(?<pid>\d+)\s+---\s+\[(?<thread>[^\]]+)\]\s+(?<logger>\S+)\s+:\s+(?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L

# Apache Combined 로그 파서
[PARSER]
    Name        apache_combined
    Format      regex
    Regex       ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

# parsers.yaml
parsers:
  - name: nginx_error
    format: regex
    regex: '^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)$'
    time_key: time
    time_format: '%Y/%m/%d %H:%M:%S'

  - name: spring_boot
    format: regex
    regex: '^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+)\s+(?<level>\w+)\s+(?<pid>\d+)\s+---\s+\[(?<thread>[^\]]+)\]\s+(?<logger>\S+)\s+:\s+(?<message>.*)$'
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'

# 멀티라인 파서 정의
[MULTILINE_PARSER]
    Name          java_stacktrace
    Type          regex
    Flush_Timeout 1000
    # 첫 줄 패턴: 타임스탬프로 시작
    Rule          "start_state"  "/^\d{4}-\d{2}-\d{2}/"  "cont"
    # 연속 줄 패턴: 공백이나 Caused by로 시작
    Rule          "cont"         "/^\s+|^Caused by:/"      "cont"

[MULTILINE_PARSER]
    Name          python_traceback
    Type          regex
    Flush_Timeout 1000
    Rule          "start_state"  "/^Traceback/"            "python_tb"
    Rule          "python_tb"    "/^\s+/"                   "python_tb"
    Rule          "python_tb"    "/^\w+Error/"              "end"

pipeline:
  inputs:
    - name: tail
      tag: app.java
      path: /var/log/app/application.log
      multiline.parser: java_stacktrace
      read_from_head: true

[PARSER]
    Name        custom_time
    Format      regex
    Regex       ^(?<time>[^ ]+) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep   On    # 변환 후에도 time 필드 유지
    Time_Offset +0900  # KST 타임존

# 파서 테스트용 설정
service:
  flush: 1
  log_level: debug

pipeline:
  inputs:
    - name: tail
      tag: test
      path: /tmp/test.log
      parser: my_custom_parser

  outputs:
    - name: stdout
      match: test
      format: json_lines

# 테스트 로그 생성 및 확인
echo '2026-03-01T12:00:00.000Z ERROR [main] App - Connection failed' >> /tmp/test.log
fluent-bit -c test.yaml

pipeline:
  filters:
    - name: kubernetes
      match: kube.*
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      merge_log: on # JSON 로그를 최상위로 병합
      merge_log_key: log_parsed # 병합 키 이름
      keep_log: off # 원본 log 필드 제거
      k8s-logging.parser: on # Pod 어노테이션의 파서 설정 사용
      k8s-logging.exclude: on # Pod 어노테이션으로 로그 제외
      labels: on # Pod Labels 포함
      annotations: off # Pod Annotations 포함 여부
      buffer_size: 0 # API 응답 버퍼 (0=무제한)
      kube_meta_cache_ttl: 300 # 메타데이터 캐시 TTL (초)
      use_kubelet: false # kubelet API 사용 여부

{
  "log": "Connection established to database",
  "kubernetes": {
    "pod_name": "api-server-7d4b8c6f5-x2k9j",
    "namespace_name": "production",
    "pod_id": "abc-123-def",
    "container_name": "api-server",
    "container_image": "myregistry/api-server:v2.1.0",
    "labels": {
      "app": "api-server",
      "version": "v2.1.0",
      "team": "backend"
    },
    "host": "node-01"
  }
}

pipeline:
  filters:
    - name: modify
      match: "*"
      # 필드 추가
      add: environment production
      add: cluster_name main-cluster
      # 필드 이름 변경
      rename: log message
      rename: stream source
      # 필드 삭제
      remove: unwanted_field
      # 조건부 필드 추가 (키가 없을 때만)
      set: default_level INFO
      # 하드카피
      copy: source source_backup

pipeline:
  filters:
    # ERROR 또는 WARN 레벨만 통과
    - name: grep
      match: app.*
      regex: level (ERROR|WARN)

    # healthcheck 경로 제외
    - name: grep
      match: access.*
      exclude: path /health

    # 특정 네임스페이스만 통과
    - name: grep
      match: kube.*
      regex: $kubernetes['namespace_name'] ^(production|staging)$

    # 여러 조건 조합 (AND)
    - name: grep
      match: app.*
      regex: level ERROR
      regex: message .*timeout.*

pipeline:
  filters:
    - name: record_modifier
      match: "*"
      record: hostname ${HOSTNAME}
      record: service_name my-application
      remove_key: unnecessary_field
      allowlist_key: timestamp
      allowlist_key: level
      allowlist_key: message

pipeline:
  filters:
    # 중첩(Nest): 플랫 -> 중첩
    - name: nest
      match: '*'
      operation: nest
      wildcard: 'app_*'
      nest_under: application
      # app_name, app_version -> application: { name, version }

    # 해제(Lift): 중첩 -> 플랫
    - name: nest
      match: '*'
      operation: lift
      nested_under: kubernetes
      # kubernetes: { pod_name, namespace } -> pod_name, namespace

pipeline:
  filters:
    - name: lua
      match: app.*
      script: /fluent-bit/scripts/transform.lua
      call: process_log

-- /fluent-bit/scripts/transform.lua

function process_log(tag, timestamp, record)
    -- 로그 레벨 정규화
    if record["level"] then
        record["level"] = string.upper(record["level"])
    end

    -- 민감 정보 마스킹
    if record["message"] then
        record["message"] = string.gsub(
            record["message"],
            "%d%d%d%d%-%d%d%d%d%-%d%d%d%d%-%d%d%d%d",
            "****-****-****-****"
        )
    end

    -- 응답 시간에 따른 등급 추가
    if record["response_time"] then
        local rt = tonumber(record["response_time"])
        if rt > 5000 then
            record["performance"] = "critical"
        elseif rt > 1000 then
            record["performance"] = "slow"
        else
            record["performance"] = "normal"
        end
    end

    -- 타임스탬프 필드 추가
    record["processed_at"] = os.date("!%Y-%m-%dT%H:%M:%SZ")

    -- 2 = MODIFIED
    return 2, timestamp, record
end

pipeline:
  filters:
    - name: rewrite_tag
      match: kube.*
      rule: $kubernetes['namespace_name'] ^(production)$ prod.$TAG false
      rule: $kubernetes['namespace_name'] ^(staging)$    stg.$TAG  false
      rule: $level                        ^(ERROR)$      alert.$TAG false

pipeline:
  filters:
    - name: throttle
      match: app.*
      rate: 1000 # 윈도우당 허용량
      window: 5 # 윈도우 크기 (초)
      interval: 1s # 평가 주기
      print_status: true # 상태 출력

pipeline:
  filters:
    - name: multiline
      match: app.*
      multiline.parser: java_stacktrace
      multiline.key_content: log

pipeline:
  outputs:
    # Elasticsearch
    - name: es
      match: kube.*
      host: ${ES_HOST}
      port: 9200
      index: logs
      type: _doc
      http_user: ${ES_USER}
      http_passwd: ${ES_PASSWORD}
      logstash_format: on
      logstash_prefix: kube-logs
      logstash_dateformat: %Y.%m.%d
      time_key: '@timestamp'
      include_tag_key: true
      tag_key: fluentbit_tag
      generate_id: on # 중복 방지 ID 생성
      buffer_size: 512KB
      tls: on
      tls.verify: on
      tls.ca_file: /certs/ca.pem
      retry_limit: 5
      workers: 2
      suppress_type_name: on # ES 8.x 호환

    # OpenSearch
    - name: opensearch
      match: app.*
      host: opensearch.logging.svc
      port: 9200
      index: app-logs
      http_user: admin
      http_passwd: ${OPENSEARCH_PASSWORD}
      tls: on
      suppress_type_name: on
      trace_output: off

pipeline:
  outputs:
    - name: loki
      match: kube.*
      host: loki-gateway.logging.svc
      port: 3100
      uri: /loki/api/v1/push
      tenant_id: my-tenant
      labels: job=fluent-bit
      label_keys: $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name']
      label_map_path: /fluent-bit/etc/loki-labelmap.json
      remove_keys: kubernetes,stream
      auto_kubernetes_labels: on
      line_format: json
      drop_single_key: on
      http_user: ${LOKI_USER}
      http_passwd: ${LOKI_PASSWORD}
      tls: on
      tls.verify: on
      workers: 2

{
  "kubernetes": {
    "namespace_name": "namespace",
    "pod_name": "pod",
    "container_name": "container",
    "labels": {
      "app": "app"
    }
  },
  "stream": "stream"
}

pipeline:
  outputs:
    - name: s3
      match: archive.*
      region: ap-northeast-2
      bucket: my-log-bucket
      s3_key_format: /logs/$TAG/%Y/%m/%d/%H/$UUID.gz
      s3_key_format_tag_delimiters: .
      total_file_size: 50M # 이 크기에 도달하면 업로드
      upload_timeout: 10m # 크기 미달이라도 이 시간 후 업로드
      use_put_object: on
      compression: gzip
      content_type: application/gzip
      store_dir: /tmp/fluent-bit-s3 # 로컬 버퍼 디렉터리
      store_dir_limit_size: 512M
      retry_limit: 5
      # IAM 역할 사용 시 (IRSA / EKS Pod Identity)
      role_arn: arn:aws:iam::123456789012:role/fluent-bit-s3
      # 정적 자격증명 (비권장)
      # endpoint:
      # sts_endpoint:

pipeline:
  outputs:
    - name: kafka
      match: app.*
      brokers: kafka-0:9092,kafka-1:9092,kafka-2:9092
      topics: application-logs
      timestamp_key: '@timestamp'
      timestamp_format: iso8601
      format: json
      message_key: log
      queue_full_retries: 10
      rdkafka.request.required.acks: 1
      rdkafka.log.connection.close: false
      rdkafka.compression.codec: snappy

pipeline:
  outputs:
    - name: cloudwatch_logs
      match: kube.*
      region: ap-northeast-2
      log_group_name: /eks/my-cluster/containers
      log_group_template: /eks/my-cluster/$kubernetes['namespace_name']
      log_stream_prefix: from-fluent-bit-
      log_stream_template: $kubernetes['pod_name']
      auto_create_group: on
      extra_user_agent: fluent-bit
      retry_limit: 5
      workers: 1

pipeline:
  outputs:
    - name: stdout
      match: '*'
      format: json_lines # json_lines | msgpack

pipeline:
  outputs:
    - name: forward
      match: '*'
      host: fluentd-aggregator.logging.svc
      port: 24224
      time_as_integer: off
      send_options: true
      require_ack_response: true
      # TLS
      tls: on
      tls.verify: on
      tls.ca_file: /certs/ca.pem
      # 공유 키 인증
      shared_key: my-shared-secret
      self_hostname: fluent-bit-node01

pipeline:
  outputs:
    - name: prometheus_exporter
      match: metrics.*
      host: 0.0.0.0
      port: 2021
      add_label: app fluent-bit

service:
  flush: 1
  log_level: info
  # 파일시스템 버퍼 활성화
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal # normal | full
  storage.checksum: off # 데이터 무결성 검증
  storage.max_chunks_up: 128 # 메모리에 올릴 최대 청크 수
  storage.backlog.mem_limit: 5M # 백로그 메모리 한도
  storage.metrics: on # 스토리지 메트릭 활성화

pipeline:
  inputs:
    # 메모리 버퍼 사용 (기본)
    - name: tail
      tag: app.mem
      path: /var/log/app/*.log
      mem_buf_limit: 10MB # 메모리 버퍼 한도

    # 파일시스템 버퍼 사용
    - name: tail
      tag: app.fs
      path: /var/log/app/*.log
      storage.type: filesystem # filesystem | memory
      storage.pause_on_chunks_overlimit: off # 한도 초과 시 디스크에 계속 기록

pipeline:
  outputs:
    - name: es
      match: '*'
      host: elasticsearch
      port: 9200
      storage.total_limit_size: 1G # Output별 파일시스템 버퍼 총량 제한
      retry_limit: 10 # 재시도 횟수
      # retry_limit: false          # 무한 재시도

  [Input: 데이터 수집]
         |
         v
  [메모리 버퍼에 저장]
         |
    메모리 한도 도달?
    /              \
   No              Yes
   |                |
   v                v
 [계속 수집]   storage.type=filesystem?
                /              \
              No               Yes
              |                 |
              v                 v
       [Input 일시 정지]   [디스크에 기록]
       (데이터 손실 위험)        |
                          디스크 한도 도달?
                          /              \
                        No              Yes
                        |                |
                        v                v
                   [계속 기록]     [Input 일시 정지]

service:
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal
  storage.max_chunks_up: 128
  storage.backlog.mem_limit: 10M

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      storage.type: filesystem
      storage.pause_on_chunks_overlimit: off

  outputs:
    - name: es
      match: kube.*
      host: elasticsearch
      storage.total_limit_size: 5G
      retry_limit: false
      net.keepalive: on
      net.keepalive_idle_timeout: 30

+---------------------------------------------------------------------+
|                      Kubernetes Cluster                             |
|                                                                     |
|  +-------------------+  +-------------------+  +-------------------+|
|  |     Node 1        |  |     Node 2        |  |     Node 3        ||
|  |                    |  |                    |  |                    ||
|  | +------+ +------+ |  | +------+ +------+ |  | +------+ +------+ ||
|  | |Pod A | |Pod B | |  | |Pod C | |Pod D | |  | |Pod E | |Pod F | ||
|  | +--+---+ +--+---+ |  | +--+---+ +--+---+ |  | +--+---+ +--+---+ ||
|  |    |        |      |  |    |        |      |  |    |        |      ||
|  |    v        v      |  |    v        v      |  |    v        v      ||
|  | /var/log/containers|  | /var/log/containers|  | /var/log/containers||
|  |         |          |  |         |          |  |         |          ||
|  |    +----+-----+    |  |    +----+-----+    |  |    +----+-----+    ||
|  |    |Fluent Bit|    |  |    |Fluent Bit|    |  |    |Fluent Bit|    ||
|  |    |(DaemonSet)|   |  |    |(DaemonSet)|   |  |    |(DaemonSet)|   ||
|  |    +----+-----+    |  |    +----+-----+    |  |    +----+-----+    ||
|  +---------|----------+  +---------|----------+  +---------|----------+|
|            |                       |                       |           |
|            +----------+------------+----------+------------+           |
|                       |                       |                        |
|               +-------v--------+     +--------v--------+              |
|               | Elasticsearch  |     |  Grafana Loki   |              |
|               +----------------+     +-----------------+              |
+---------------------------------------------------------------------+

# 저장소 추가
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

# 기본 설치
helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace

# 설치 확인
kubectl get pods -n logging -l app.kubernetes.io/name=fluent-bit
kubectl get ds -n logging

# custom-values.yaml
kind: DaemonSet

image:
  repository: cr.fluentbit.io/fluent/fluent-bit
  tag: '3.2.2'
  pullPolicy: IfNotPresent

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

tolerations:
  - operator: Exists # 모든 노드에 배포 (master 포함)

serviceAccount:
  create: true
  annotations:
    # IRSA (EKS)
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fluent-bit

# 볼륨 마운트
volumeMounts:
  - name: varlog
    mountPath: /var/log
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: etcmachineid
    mountPath: /etc/machine-id
    readOnly: true

volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: etcmachineid
    hostPath:
      path: /etc/machine-id

# 환경변수
env:
  - name: ELASTICSEARCH_HOST
    value: 'elasticsearch-master.logging.svc.cluster.local'
  - name: ELASTICSEARCH_PORT
    value: '9200'

# Fluent Bit 설정
config:
  service: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  /fluent-bit/etc/parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
        storage.path  /var/log/fluent-bit/buffer/
        storage.sync  normal
        storage.max_chunks_up 128
        Hot_Reload    On

  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        multiline.parser  docker, cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        storage.type      filesystem

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Labels              On
        Annotations         Off

  outputs: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            ${ELASTICSEARCH_HOST}
        Port            ${ELASTICSEARCH_PORT}
        Logstash_Format On
        Logstash_Prefix kube
        Retry_Limit     False
        Suppress_Type_Name On

# Prometheus 메트릭 수집을 위한 ServiceMonitor
serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s

# Liveness/Readiness Probe
livenessProbe:
  httpGet:
    path: /
    port: http
readinessProbe:
  httpGet:
    path: /api/v1/health
    port: http

/var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log
  -> symlink -> /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/0.log

2026-03-01T12:00:00.123456789Z stdout F This is a log message
2026-03-01T12:00:00.123456789Z stderr P This is a partial log line

# Namespace에 따라 다른 Output으로 라우팅
pipeline:
  filters:
    - name: kubernetes
      match: kube.*
      merge_log: on
      labels: on

    # 프로덕션 네임스페이스 태그 변경
    - name: rewrite_tag
      match: kube.*
      rule: $kubernetes['namespace_name'] ^(production)$ prod.$TAG false
      rule: $kubernetes['namespace_name'] ^(staging)$    stg.$TAG  false
      rule: $kubernetes['namespace_name'] ^(monitoring)$ mon.$TAG  false

  outputs:
    # 프로덕션 로그 -> Elasticsearch
    - name: es
      match: prod.*
      host: es-production.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: prod-logs

    # 스테이징 로그 -> Loki
    - name: loki
      match: stg.*
      host: loki.logging.svc
      port: 3100
      labels:
        env=staging

    # 모니터링 로그 -> S3 아카이빙
    - name: s3
      match: mon.*
      bucket: monitoring-logs-archive
      region: ap-northeast-2
      total_file_size: 100M
      upload_timeout: 10m
      compression: gzip

    # 기타 로그 -> 기본 Elasticsearch
    - name: es
      match: kube.*
      host: es-default.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: default-logs

pipeline:
  filters:
    - name: kubernetes
      match: kube.*
      merge_log: on
      labels: on

    # 팀 Label 기반 태그 재작성
    - name: rewrite_tag
      match: kube.*
      rule: $kubernetes['labels']['team'] ^(backend)$  team.backend.$TAG  false
      rule: $kubernetes['labels']['team'] ^(frontend)$ team.frontend.$TAG false
      rule: $kubernetes['labels']['team'] ^(data)$     team.data.$TAG     false

  outputs:
    # 백엔드 팀 -> 전용 Elasticsearch 인덱스
    - name: es
      match: team.backend.*
      host: elasticsearch.logging.svc
      logstash_format: on
      logstash_prefix: team-backend

    # 프론트엔드 팀 -> 전용 Loki tenant
    - name: loki
      match: team.frontend.*
      host: loki.logging.svc
      tenant_id: frontend-team
      labels:
        team=frontend

    # 데이터 팀 -> S3 + Elasticsearch 듀얼
    - name: es
      match: team.data.*
      host: elasticsearch.logging.svc
      logstash_format: on
      logstash_prefix: team-data

    - name: s3
      match: team.data.*
      bucket: data-team-logs
      region: ap-northeast-2
      compression: gzip

# fluent-bit-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: ['']
    resources:
      - namespaces
      - pods
      - pods/logs
      - nodes
      - nodes/proxy
    verbs: ['get', 'list', 'watch']
  - apiGroups: ['']
    resources:
      - events
    verbs: ['get', 'list', 'watch']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging

# fluent-bit-efk.yaml
service:
  flush: 5
  log_level: info
  parsers_file: /fluent-bit/etc/parsers.conf
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal
  storage.max_chunks_up: 128

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      multiline.parser: docker, cri
      db: /var/log/flb_kube.db
      mem_buf_limit: 5MB
      skip_long_lines: on
      refresh_interval: 10
      storage.type: filesystem

  filters:
    - name: kubernetes
      match: kube.*
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      merge_log: on
      keep_log: off
      k8s-logging.parser: on
      k8s-logging.exclude: on
      labels: on

    # kube-system, logging 네임스페이스 로그 제외
    - name: grep
      match: kube.*
      exclude: $kubernetes['namespace_name'] ^(kube-system|logging)$

    # 환경 정보 추가
    - name: modify
      match: kube.*
      add: cluster_name my-eks-cluster
      add: environment production

  outputs:
    - name: es
      match: kube.*
      host: elasticsearch-master.logging.svc.cluster.local
      port: 9200
      http_user: elastic
      http_passwd: ${ES_PASSWORD}
      logstash_format: on
      logstash_prefix: kube-logs
      logstash_dateformat: "%Y.%m.%d"
      time_key: "@timestamp"
      include_tag_key: true
      tag_key: fluentbit_tag
      suppress_type_name: on
      generate_id: on
      retry_limit: false
      workers: 2
      tls: on
      tls.verify: on
      tls.ca_file: /certs/es-ca.pem

# fluent-bit-loki.yaml
service:
  flush: 5
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      multiline.parser: docker, cri
      db: /var/log/flb_kube.db
      mem_buf_limit: 5MB
      skip_long_lines: on

  filters:
    - name: kubernetes
      match: kube.*
      merge_log: on
      keep_log: off
      labels: on
      annotations: off

  outputs:
    - name: loki
      match: kube.*
      host: loki-gateway.logging.svc.cluster.local
      port: 3100
      uri: /loki/api/v1/push
      labels: job=fluent-bit,cluster=my-cluster
      label_keys: $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name']
      auto_kubernetes_labels: on
      line_format: json
      workers: 2
      retry_limit: false

# fluent-bit-dual.yaml
service:
  flush: 5
  log_level: info
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal
  storage.max_chunks_up: 128

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      multiline.parser: docker, cri
      db: /var/log/flb_kube.db
      storage.type: filesystem

  filters:
    - name: kubernetes
      match: kube.*
      merge_log: on
      keep_log: off
      labels: on

  outputs:
    # 실시간 검색/분석용 Elasticsearch
    - name: es
      match: kube.*
      host: elasticsearch.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: kube-logs
      suppress_type_name: on
      retry_limit: false
      workers: 2

    # 장기 보관용 S3 아카이빙
    - name: s3
      match: kube.*
      region: ap-northeast-2
      bucket: log-archive-bucket
      s3_key_format: /kubernetes/$TAG/%Y/%m/%d/%H/$UUID.gz
      s3_key_format_tag_delimiters: .
      total_file_size: 100M
      upload_timeout: 10m
      compression: gzip
      content_type: application/gzip
      store_dir: /tmp/fluent-bit-s3
      store_dir_limit_size: 1G
      use_put_object: on
      retry_limit: 5
      workers: 1

# fluent-bit-routing.yaml
service:
  flush: 5
  log_level: info
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      multiline.parser: docker, cri
      db: /var/log/flb_kube.db
      storage.type: filesystem

  filters:
    - name: kubernetes
      match: kube.*
      merge_log: on
      labels: on

    # Namespace별 태그 재작성
    - name: rewrite_tag
      match: kube.*
      rule: $kubernetes['namespace_name'] ^(production)$  route.prod.$TAG  false
      rule: $kubernetes['namespace_name'] ^(staging)$     route.stg.$TAG   false
      rule: $kubernetes['namespace_name'] ^(kube-system)$ route.sys.$TAG   false

    # 프로덕션에서 ERROR만 추출
    - name: rewrite_tag
      match: route.prod.*
      rule: $log ^.*ERROR.*$ alert.prod.$TAG true

  outputs:
    # 프로덕션 전체 로그 -> Elasticsearch
    - name: es
      match: route.prod.*
      host: es-prod.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: prod-all
      suppress_type_name: on
      workers: 2

    # 프로덕션 ERROR 알림 -> 별도 인덱스
    - name: es
      match: alert.prod.*
      host: es-prod.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: prod-alerts
      suppress_type_name: on

    # 스테이징 로그 -> Loki (비용 효율)
    - name: loki
      match: route.stg.*
      host: loki.logging.svc
      port: 3100
      labels:
        env=staging
      auto_kubernetes_labels: on
      line_format: json

    # 시스템 로그 -> S3 아카이빙 (장기 보관)
    - name: s3
      match: route.sys.*
      bucket: system-logs-archive
      region: ap-northeast-2
      total_file_size: 50M
      upload_timeout: 15m
      compression: gzip
      s3_key_format: /kube-system/%Y/%m/%d/$UUID.gz

    # 미분류 로그 -> 기본 Elasticsearch
    - name: es
      match: kube.*
      host: es-default.logging.svc
      port: 9200
      logstash_format: on
      logstash_prefix: default-logs
      suppress_type_name: on

pipeline:
  outputs:
    - name: es
      match: '*'
      host: elasticsearch
      port: 9200
      workers: 4 # 4개 병렬 Worker
      net.keepalive: on
      net.keepalive_idle_timeout: 30

service:
  flush: 1 # 1초마다 플러시 (실시간성 중시)
  # flush: 5  # 5초마다 플러시 (처리량 중시)

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      # 높은 처리량 환경
      buffer_chunk_size: 512KB # 청크 단위 크기 (기본 32KB)
      buffer_max_size: 5MB # 최대 버퍼 크기 (기본 32KB)
      mem_buf_limit: 50MB # 메모리 버퍼 한도

  outputs:
    - name: es
      match: '*'
      buffer_size: 512KB # HTTP 버퍼 크기

pipeline:
  inputs:
    # 애플리케이션 로그 파이프라인
    - name: tail
      tag: app.*
      path: /var/log/containers/app-*.log
      multiline.parser: docker, cri
      threaded: on # 별도 스레드에서 실행

    # 시스템 로그 파이프라인
    - name: tail
      tag: sys.*
      path: /var/log/containers/kube-*.log
      multiline.parser: docker, cri
      threaded: on

  outputs:
    - name: es
      match: app.*
      host: elasticsearch
      workers: 4

    - name: es
      match: sys.*
      host: elasticsearch
      workers: 2

# Hot Reload 활성화
service:
  hot_reload: on

# 방법 1: SIGHUP 시그널
kill -SIGHUP $(pidof fluent-bit)

# 방법 2: HTTP API (v4.0+)
curl -X POST http://localhost:2020/api/v2/reload

# 방법 3: 커맨드라인 옵션
fluent-bit -c fluent-bit.yaml -Y   # --enable-hot-reload

# Fluent Bit 프로세스 메모리 확인
ps aux | grep fluent-bit

# HTTP API로 내부 메트릭 확인
curl -s http://localhost:2020/api/v1/storage | jq .

# Prometheus 메트릭으로 확인
curl -s http://localhost:2020/api/v2/metrics/prometheus | grep fluentbit_input

service:
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  health_check: on
  hc_errors_count: 5 # 이 수 이상 에러 발생 시 unhealthy
  hc_retry_failure_count: 5 # 이 수 이상 재시도 실패 시 unhealthy
  hc_period: 60 # 헬스체크 주기 (초)

# 헬스 체크
curl -s http://localhost:2020/api/v1/health
# 응답: ok (HTTP 200) 또는 error (HTTP 500)

# 스토리지 상태 확인
curl -s http://localhost:2020/api/v1/storage | jq .
# {
#   "storage_layer": {
#     "chunks": {
#       "total_chunks": 15,
#       "mem_chunks": 10,
#       "fs_chunks": 5,
#       "fs_chunks_up": 3,
#       "fs_chunks_down": 2
#     }
#   }
# }

# Prometheus 메트릭 확인
curl -s http://localhost:2020/api/v2/metrics/prometheus

# Fluent Bit 자체 메트릭을 Prometheus로 노출
pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fb.metrics
      scrape_interval: 30

  outputs:
    - name: prometheus_exporter
      match: fb.metrics
      host: 0.0.0.0
      port: 2021

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  endpoints:
    - port: http
      path: /api/v2/metrics/prometheus
      interval: 30s
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
      - logging

# 1. Fluent Bit 프로세스 상태 확인
kubectl get pods -n logging -l app.kubernetes.io/name=fluent-bit
kubectl logs -n logging <fluent-bit-pod> --tail=50

# 2. 설정 파일 문법 검증
fluent-bit -c fluent-bit.yaml --dry-run

# 3. 로그 파일 경로 확인
kubectl exec -n logging <fluent-bit-pod> -- ls -la /var/log/containers/

# 4. DB 파일(offset) 확인
kubectl exec -n logging <fluent-bit-pod> -- ls -la /var/log/flb_kube.db

# 5. 권한 확인
kubectl exec -n logging <fluent-bit-pod> -- cat /var/log/containers/<target-log>

# 메모리 제한 설정
pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      mem_buf_limit: 5MB # Input 메모리 한도
      skip_long_lines: on # 긴 라인 스킵
      buffer_chunk_size: 32KB # 청크 크기 제한
      buffer_max_size: 32KB # 최대 버퍼

service:
  storage.path: /var/log/fluent-bit/buffer/
  storage.max_chunks_up: 64 # 메모리 청크 수 제한 (기본 128)

# Backpressure 발생 확인
curl -s http://localhost:2020/api/v1/storage | jq .

# 메트릭으로 확인
curl -s http://localhost:2020/api/v2/metrics/prometheus | grep -E "retries|errors|dropped"

# Backpressure 대응 설정 예시
service:
  flush: 1
  storage.path: /var/log/fluent-bit/buffer/
  storage.sync: normal
  storage.max_chunks_up: 256

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      storage.type: filesystem
      storage.pause_on_chunks_overlimit: off

  outputs:
    - name: es
      match: kube.*
      host: elasticsearch
      port: 9200
      workers: 4
      storage.total_limit_size: 10G
      retry_limit: false
      net.keepalive: on
      net.keepalive_idle_timeout: 15

# TLS 연결 테스트
kubectl exec -n logging <fluent-bit-pod> -- \
  curl -v --cacert /certs/ca.pem https://elasticsearch:9200

# 인증서 만료 확인
kubectl exec -n logging <fluent-bit-pod> -- \
  openssl x509 -in /certs/ca.pem -noout -enddate

service:
  log_level: debug # error, warn, info, debug, trace

pipeline:
  outputs:
    # 디버깅용: 모든 레코드를 표준 출력으로
    - name: stdout
      match: '*'
      format: json_lines

    # 실제 Output
    - name: es
      match: '*'
      host: elasticsearch

# Input만 테스트
fluent-bit -i tail -p path=/var/log/test.log -o stdout

# Parser 테스트
fluent-bit -i tail -p path=/var/log/test.log -p parser=json -o stdout

# 전체 설정 테스트 (dry-run)
fluent-bit -c fluent-bit.yaml --dry-run

# Fluent Bit 로그 스트리밍
kubectl logs -n logging -l app.kubernetes.io/name=fluent-bit -f --tail=100

# 특정 Pod의 로그
kubectl logs -n logging fluent-bit-xxxxx -f

# 이전 컨테이너 로그 (crash 시)
kubectl logs -n logging fluent-bit-xxxxx --previous

# Kubernetes Pod Security
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  capabilities:
    drop: ['ALL']

# 필요한 볼륨만 마운트
volumes:
  - name: varlog
    hostPath:
      path: /var/log
      type: ''
  - name: buffer
    emptyDir:
      sizeLimit: 2Gi

# TLS 강제 (Output)
pipeline:
  outputs:
    - name: es
      tls: on
      tls.verify: on
      tls.ca_file: /certs/ca.pem

pipeline:
  inputs:
    - name: tail
      path: /var/log/app/*.log
      db: /var/log/flb_app.db # 필수: offset 추적
      rotate_wait: 5 # 로테이션 후 대기 시간 (초)
      refresh_interval: 10 # 파일 목록 갱신 주기