Skip to content

Split View: GitOps 완전 가이드 2025: ArgoCD vs Flux, ApplicationSet, Image Updater, 멀티 클러스터

✨ Learn with Quiz
|

GitOps 완전 가이드 2025: ArgoCD vs Flux, ApplicationSet, Image Updater, 멀티 클러스터

목차

1. 들어가며: GitOps란 무엇인가

GitOps는 Git 저장소를 인프라와 애플리케이션의 단일 진실의 원천(Single Source of Truth)으로 사용하는 운영 방법론이다. 선언적 설정 파일을 Git에 저장하고, 자동화된 프로세스가 클러스터의 실제 상태를 Git의 원하는 상태와 동기화한다.

GitOps의 4가지 원칙:

  1. 선언적(Declarative) - 시스템의 원하는 상태를 선언적으로 기술
  2. 버전 관리(Versioned) - 원하는 상태가 Git에 저장되어 완전한 이력 추적 가능
  3. 자동 적용(Pulled) - 에이전트가 원하는 상태 변경을 자동으로 감지하고 적용 (Pull 기반)
  4. 지속 조정(Continuously Reconciled) - 실제 상태와 원하는 상태의 차이를 지속적으로 감지하고 수정

2. ArgoCD 심화

2.1 아키텍처

ArgoCD는 다음 핵심 컴포넌트로 구성된다:

  • API Server: UI, CLI, CI/CD 시스템과의 인터페이스
  • Repository Server: Git 리포지토리의 매니페스트를 생성하는 내부 서비스
  • Application Controller: 실행 중인 애플리케이션 상태를 모니터링하고 동기화
# ArgoCD 설치
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# CLI 로그인
argocd login argocd.example.com --grpc-web

# 클러스터 등록
argocd cluster add production-context --name production

2.2 Application CRD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
    kustomize:
      namePrefix: prod-
      commonLabels:
        env: production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jqPathExpressions:
        - .spec.metrics[].resource.target

2.3 Sync Hooks

# Pre-sync: DB 마이그레이션
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp/db-migrate:latest
          command: ["./migrate", "up"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url
      restartPolicy: Never
  backoffLimit: 3
---
# Post-sync: Slack 알림
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-deploy
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"API Server deployed successfully to production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never
---
# SyncFail: 실패 시 알림
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-fail
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"ALERT: API Server deployment FAILED in production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never

2.4 Health Checks 커스터마이징

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.argoproj.io_Rollout: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.currentStepIndex ~= nil then
        hs.status = "Progressing"
        hs.message = "Rollout in progress"
      end
      if obj.status.phase == "Healthy" then
        hs.status = "Healthy"
        hs.message = "Rollout is healthy"
      end
      if obj.status.phase == "Degraded" then
        hs.status = "Degraded"
        hs.message = "Rollout is degraded"
      end
    end
    return hs
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

2.5 RBAC 설정

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # 프로젝트 관리자
    p, role:project-admin, applications, *, production/*, allow
    p, role:project-admin, applications, sync, production/*, allow
    p, role:project-admin, logs, get, production/*, allow

    # 개발자: 읽기 + sync만
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, staging/*, allow
    p, role:developer, logs, get, */*, allow

    # CI/CD 시스템
    p, role:ci-cd, applications, sync, */*, allow
    p, role:ci-cd, applications, get, */*, allow

    # 그룹 매핑
    g, platform-team, role:admin
    g, backend-team, role:project-admin
    g, frontend-team, role:developer
    g, github-actions, role:ci-cd
  scopes: '[groups, email]'

3. Flux 심화

3.1 핵심 리소스

# GitRepository: Git 소스 정의
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-repo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/platform-config
  ref:
    branch: main
  secretRef:
    name: git-credentials
  ignore: |
    # exclude all
    /*
    # include deploy dir
    !/deploy
---
# Kustomization: 무엇을 어디에 배포할지 정의
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  retryInterval: 2m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server/production
  prune: true
  wait: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api-server
      namespace: production
  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: api-server
        spec:
          replicas: 5
      target:
        kind: Deployment
        name: api-server
---
# HelmRelease: Helm 차트 배포
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: production
spec:
  interval: 10m
  chart:
    spec:
      chart: redis
      version: "18.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    architecture: replication
    auth:
      enabled: true
      existingSecret: redis-auth
    replica:
      replicaCount: 3
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
  valuesFrom:
    - kind: ConfigMap
      name: redis-values
      valuesKey: values.yaml
  upgrade:
    remediation:
      retries: 3
  rollback:
    cleanupOnFail: true

3.2 Flux 알림 설정

# Provider: Slack 알림
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-webhook
---
# Alert: 특정 이벤트에 대한 알림
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: deployment-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"
  exclusionList:
    - ".*upgrade.*retries exhausted.*"
  summary: "Flux deployment issue detected"

4. ArgoCD vs Flux 비교

항목ArgoCDFlux
UI풍부한 웹 UI 내장별도 UI 필요 (Weave GitOps 등)
아키텍처중앙 집중형분산형, 네임스페이스 단위
CRD 수적음 (Application, AppProject)많음 (GitRepo, Kustomization, HelmRelease 등)
Helm 지원Template 렌더링 후 적용Native HelmRelease CRD
Kustomize내장 지원내장 지원
멀티테넌시AppProject로 격리네임스페이스 기반 격리
Image 자동 업데이트ArgoCD Image UpdaterFlux Image Automation
SSOOIDC/SAML/LDAP 지원UI 도구에 따라 다름
RBAC세밀한 역할 기반 접근 제어Kubernetes RBAC 활용
CLIargocd CLIflux CLI
알림Notification 설정Alert/Provider CRD
Health Check커스텀 Lua 스크립트Kustomization 헬스 체크
Drift 감지실시간 UI 표시주기적 Reconciliation
Progressive DeliveryArgo Rollouts 연동Flagger 연동
러닝 커브중간 (UI가 도움)높음 (CRD 기반 설정)

5. ApplicationSet

5.1 List Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: production
            url: https://prod-k8s.example.com
            namespace: production
            replicas: "5"
          - cluster: staging
            url: https://staging-k8s.example.com
            namespace: staging
            replicas: "2"
          - cluster: development
            url: https://dev-k8s.example.com
            namespace: development
            replicas: "1"
  template:
    metadata:
      name: "api-server-{{cluster}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "apps/api-server/overlays/{{cluster}}"
        kustomize:
          commonLabels:
            cluster: "{{cluster}}"
      destination:
        server: "{{url}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.2 Cluster Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-monitoring
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            monitoring: enabled
        values:
          prometheus_retention: "30d"
  template:
    metadata:
      name: "monitoring-{{name}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-charts.git
        targetRevision: main
        path: charts/monitoring
        helm:
          values: |
            cluster: "{{name}}"
            prometheus:
              retention: "{{values.prometheus_retention}}"
            grafana:
              ingress:
                host: "grafana-{{name}}.example.com"
      destination:
        server: "{{server}}"
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

5.3 Git Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/k8s-manifests.git
        revision: main
        directories:
          - path: "apps/*/overlays/production"
          - path: "apps/legacy-*"
            exclude: true
  template:
    metadata:
      name: "{{path[1]}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
# Git File Generator: config.json 파일 기반
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dynamic-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/app-config.git
        revision: main
        files:
          - path: "environments/*/config.json"
  template:
    metadata:
      name: "{{app.name}}-{{environment}}"
    spec:
      project: "{{app.project}}"
      source:
        repoURL: "{{app.repo}}"
        targetRevision: "{{app.revision}}"
        path: "{{app.path}}"
      destination:
        server: "{{cluster.url}}"
        namespace: "{{app.namespace}}"

5.4 Matrix Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cross-cluster-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: production
          - list:
              elements:
                - app: api-server
                  chart: api-server
                  namespace: backend
                - app: web-frontend
                  chart: web-frontend
                  namespace: frontend
                - app: worker
                  chart: worker
                  namespace: backend
  template:
    metadata:
      name: "{{app}}-{{name}}"
    spec:
      project: production
      source:
        repoURL: https://github.com/myorg/helm-charts.git
        targetRevision: main
        path: "charts/{{chart}}"
        helm:
          valueFiles:
            - "values-{{metadata.labels.region}}.yaml"
      destination:
        server: "{{server}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.5 Pull Request Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pr-preview
  namespace: argocd
spec:
  generators:
    - pullRequest:
        github:
          owner: myorg
          repo: api-server
          tokenRef:
            secretName: github-token
            key: token
          labels:
            - preview
        requeueAfterSeconds: 30
  template:
    metadata:
      name: "pr-{{number}}-api-server"
      labels:
        preview: "true"
    spec:
      project: previews
      source:
        repoURL: https://github.com/myorg/api-server.git
        targetRevision: "{{head_sha}}"
        path: deploy/preview
        kustomize:
          namePrefix: "pr-{{number}}-"
          commonLabels:
            pr: "{{number}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-{{number}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

6. Image Updater

6.1 ArgoCD Image Updater

# 설치
# kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

# Application에 이미지 업데이트 설정
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  annotations:
    # 이미지 목록과 업데이트 전략
    argocd-image-updater.argoproj.io/image-list: "app=myregistry.com/api-server"
    argocd-image-updater.argoproj.io/app.update-strategy: semver
    argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$"
    argocd-image-updater.argoproj.io/app.ignore-tags: "latest, nightly"
    # Git write-back 방식
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: "kustomization:../../base"
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production

6.2 Flux Image Automation

# ImageRepository: 이미지 레지스트리 감시
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: api-server
  namespace: flux-system
spec:
  image: myregistry.com/api-server
  interval: 5m
  secretRef:
    name: registry-credentials
---
# ImagePolicy: 어떤 태그를 선택할지 정의
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: api-server
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: api-server
  policy:
    semver:
      range: ">=1.0.0"
  filterTags:
    pattern: "^v(?P<version>[0-9]+\\.[0-9]+\\.[0-9]+)$"
    extract: "$version"
---
# ImageUpdateAutomation: Git 자동 커밋
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: auto-update
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux-bot
        email: flux@myorg.com
      messageTemplate: |
        chore(image): update images

        Automated image update:
        {{range .Changed.Changes}}
        - {{.OldValue}} -> {{.NewValue}}
        {{end}}
    push:
      branch: main
  update:
    path: ./deploy
    strategy: Setters

7. Helm + GitOps

7.1 ArgoCD에서 Helm 사용

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nginx-ingress
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://kubernetes.github.io/ingress-nginx
    chart: ingress-nginx
    targetRevision: 4.9.1
    helm:
      releaseName: nginx-ingress
      values: |
        controller:
          replicaCount: 3
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          metrics:
            enabled: true
            serviceMonitor:
              enabled: true
          admissionWebhooks:
            enabled: true
      valueFiles:
        - values-production.yaml
      parameters:
        - name: controller.service.type
          value: LoadBalancer
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-system
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true

7.2 Umbrella Chart 패턴

# Chart.yaml (umbrella chart)
apiVersion: v2
name: platform-stack
version: 1.0.0
dependencies:
  - name: prometheus
    version: 25.x.x
    repository: https://prometheus-community.github.io/helm-charts
    condition: prometheus.enabled
  - name: grafana
    version: 7.x.x
    repository: https://grafana.github.io/helm-charts
    condition: grafana.enabled
  - name: loki
    version: 5.x.x
    repository: https://grafana.github.io/helm-charts
    condition: loki.enabled

8. Kustomize + GitOps

8.1 환경별 Overlays

apps/api-server/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    development/
      kustomization.yaml
      patches/
        replicas.yaml
    staging/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
    production/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
        hpa.yaml
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: api-server
---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namePrefix: prod-
patches:
  - path: patches/replicas.yaml
  - path: patches/resources.yaml
  - path: patches/hpa.yaml
configMapGenerator:
  - name: api-config
    literals:
      - LOG_LEVEL=warn
      - ENVIRONMENT=production
---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 5

9. 시크릿 관리

9.1 Sealed Secrets

# 설치
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# 시크릿 암호화
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=super-secret \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  --format yaml > sealed-db-credentials.yaml
# sealed-db-credentials.yaml (Git에 커밋 가능)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgB1k2j3l4m5n6o7p8...
    password: AgC9d0e1f2g3h4i5j6...
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

9.2 External Secrets Operator

# SecretStore: Vault 연결
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: production-app
          serviceAccountRef:
            name: vault-auth
---
# ExternalSecret: Vault에서 시크릿 동기화
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.production:5432/myapp"
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
---
# ClusterSecretStore: 클러스터 전체에서 사용
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

9.3 SOPS (Secrets OPerationS)

# .sops.yaml 설정 파일
creation_rules:
  - path_regex: ".*\\.enc\\.yaml$"
    encrypted_regex: "^(data|stringData)$"
    age: age1ql3z7hjy54pw3hyww5ay...
  - path_regex: "environments/production/.*"
    kms: "arn:aws:kms:ap-northeast-2:123456789012:key/abc-123"
  - path_regex: "environments/staging/.*"
    pgp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
# SOPS로 암호화
sops --encrypt --in-place secrets.enc.yaml

# Flux에서 SOPS 사용을 위한 Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age-key

10. Progressive Delivery

10.1 Argo Rollouts - 카나리 배포

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: myregistry.com/api-server:v2.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi
  strategy:
    canary:
      canaryService: api-server-canary
      stableService: api-server-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: api-server-vsvc
              routes:
                - primary
      steps:
        - setWeight: 5
        - pause:
            duration: 5m
        - setWeight: 20
        - pause:
            duration: 10m
        - setWeight: 50
        - pause:
            duration: 10m
        - setWeight: 80
        - pause:
            duration: 5m
      analysis:
        templates:
          - templateName: success-rate
          - templateName: latency
        startingStep: 1
        args:
          - name: service-name
            value: api-server-canary
---
# AnalysisTemplate: 성공률 검증
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 60s
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
---
# AnalysisTemplate: 지연 시간 검증
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency
spec:
  args:
    - name: service-name
  metrics:
    - name: p99-latency
      interval: 60s
      successCondition: result[0] < 500
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m]))
              by (le)
            ) * 1000

10.2 Argo Rollouts - 블루그린 배포

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-frontend
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      containers:
        - name: web
          image: myregistry.com/web-frontend:v3.0.0
          ports:
            - containerPort: 3000
  strategy:
    blueGreen:
      activeService: web-frontend-active
      previewService: web-frontend-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
        args:
          - name: preview-url
            value: "http://web-frontend-preview.production.svc:3000"
      postPromotionAnalysis:
        templates:
          - templateName: success-rate
        args:
          - name: service-name
            value: web-frontend-active

11. 멀티 클러스터 GitOps

11.1 Hub-Spoke 패턴

management-cluster/          (Hub)
  argocd/
    applicationsets/
      monitoring.yaml         -> 모든 클러스터에 모니터링 배포
      logging.yaml            -> 모든 클러스터에 로깅 배포
      apps.yaml               -> 환경별 앱 배포
    projects/
      platform.yaml
      production.yaml
      staging.yaml
    clusters/
      prod-us-east.yaml
      prod-ap-northeast.yaml
      staging-us-east.yaml

11.2 저장소 구조: Monorepo vs Polyrepo

Monorepo 구조:

k8s-manifests/
  apps/
    api-server/
      base/
      overlays/
        development/
        staging/
        production/
    web-frontend/
      base/
      overlays/
        development/
        staging/
        production/
  platform/
    monitoring/
    logging/
    ingress/
  clusters/
    production/
    staging/

Polyrepo 구조:

# repo: api-server-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: web-frontend-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: platform-config
monitoring/
logging/
ingress/
항목MonorepoPolyrepo
가시성전체 시스템 한눈에서비스별 독립적
접근 제어디렉토리 기반 (CODEOWNERS)리포지토리별
CI/CD변경 감지 복잡단순 트리거
의존성 관리쉬움버전 관리 필요
규모중소 규모 적합대규모 조직 적합

11.3 App-of-Apps 패턴

# 루트 Application: 다른 Application들을 관리
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: clusters/production
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

12. Drift 감지와 Reconciliation

12.1 ArgoCD Drift 감지

ArgoCD는 실시간으로 Git의 원하는 상태와 클러스터의 실제 상태를 비교한다.

# 특정 필드를 Drift 감지에서 제외
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
spec:
  ignoreDifferences:
    # HPA가 관리하는 replicas 무시
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    # 자동 생성되는 annotation 무시
    - group: ""
      kind: Service
      jqPathExpressions:
        - .metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-*"
    # MutatingWebhook이 주입하는 sidecar 무시
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - istio-sidecar-injector

12.2 Flux Reconciliation

# 강제 Reconciliation
# flux reconcile kustomization api-server --with-source

# Reconciliation 상태 확인
# flux get kustomizations
# flux get helmreleases -A

# 특정 리소스 일시 중단
# flux suspend kustomization api-server
# flux resume kustomization api-server

13. 실전 퀴즈

Q1: GitOps의 Pull 기반 모델이 Push 기반(전통적 CI/CD)보다 보안적으로 우수한 이유는?

정답: Pull 기반 모델에서는 클러스터 내부의 에이전트(ArgoCD/Flux)가 Git을 폴링하여 변경 사항을 적용한다. CI 파이프라인에 클러스터 접근 자격 증명을 노출할 필요가 없다.

  • Push 모델: CI 서버가 kubectl/helm으로 직접 배포 -> 클러스터 자격 증명이 CI에 필요
  • Pull 모델: 클러스터 내부 에이전트가 Git을 감시 -> 클러스터 자격 증명이 외부에 노출되지 않음
  • 공격 표면(Attack Surface) 감소: CI 시스템이 침해되어도 클러스터에 직접 접근 불가
Q2: ApplicationSet의 Matrix Generator는 어떤 상황에서 유용한가?

정답: Matrix Generator는 두 개의 생성기를 조합하여 카르테시안 곱(모든 조합)을 만든다. 여러 클러스터에 여러 애플리케이션을 배포할 때 유용하다.

  • 예: 3개 클러스터(prod-us, prod-eu, prod-ap) x 5개 앱 = 15개 Application 자동 생성
  • Cluster Generator + List Generator 조합이 가장 일반적
  • 중복 설정 없이 모든 조합을 자동으로 관리
Q3: ArgoCD Image Updater의 write-back-method가 git인 경우와 argocd인 경우의 차이는?

정답: git 방식은 이미지 태그 변경을 Git 리포지토리에 커밋한다. argocd 방식은 ArgoCD Application의 파라미터를 직접 수정한다.

  • git: Git이 단일 진실의 원천으로 유지됨. 감사 추적 가능. 속도는 느림
  • argocd: Git을 수정하지 않음. 빠르지만 Git과 실제 상태가 불일치할 수 있음
  • 프로덕션에서는 git 방식을 권장
Q4: Sealed Secrets vs External Secrets Operator의 주요 차이점은?

정답: Sealed Secrets는 시크릿을 암호화하여 Git에 저장하고, External Secrets Operator는 외부 시크릿 저장소(Vault, AWS SM 등)에서 런타임에 동기화한다.

  • Sealed Secrets: Git에 암호화된 시크릿 저장. 오프라인에서도 작동. 키 로테이션이 복잡
  • External Secrets Operator: 외부 저장소에서 실시간 동기화. 중앙 집중 관리. 외부 서비스 의존성
  • 소규모 팀: Sealed Secrets가 더 간단
  • 대규모 조직: External Secrets + Vault 조합 권장
Q5: Argo Rollouts의 카나리 배포에서 AnalysisTemplate이 실패하면 어떤 일이 발생하는가?

정답: AnalysisTemplate의 메트릭이 successCondition을 만족하지 못하고 failureLimit에 도달하면, Rollout이 자동으로 롤백된다.

  • 카나리 트래픽이 0%로 되돌아감
  • stable 버전이 모든 트래픽을 받음
  • Rollout 상태가 Degraded로 변경
  • 수동 재시도: kubectl argo rollouts retry rollout api-server
  • 분석 결과 확인: kubectl argo rollouts get rollout api-server

14. 참고 자료

  1. ArgoCD 공식 문서
  2. Flux 공식 문서
  3. ApplicationSet 문서
  4. ArgoCD Image Updater
  5. Argo Rollouts
  6. Sealed Secrets
  7. External Secrets Operator
  8. SOPS
  9. Flagger
  10. GitOps Principles - OpenGitOps
  11. Kustomize
  12. Helm
  13. Weave GitOps

GitOps Complete Guide 2025: ArgoCD vs Flux, ApplicationSet, Image Updater, Multi-Cluster

Table of Contents

1. Introduction: What is GitOps

GitOps is an operational methodology that uses Git repositories as the single source of truth for infrastructure and applications. Declarative configuration files are stored in Git, and automated processes synchronize the actual cluster state with the desired state in Git.

The 4 Principles of GitOps:

  1. Declarative - The desired state of the system is described declaratively
  2. Versioned - The desired state is stored in Git with full history tracking
  3. Pulled - Agents automatically detect and apply desired state changes (Pull-based)
  4. Continuously Reconciled - Differences between actual and desired state are continuously detected and corrected

2. ArgoCD Deep Dive

2.1 Architecture

ArgoCD consists of these core components:

  • API Server: Interface for UI, CLI, and CI/CD systems
  • Repository Server: Internal service that generates manifests from Git repositories
  • Application Controller: Monitors running application state and synchronizes
# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# CLI login
argocd login argocd.example.com --grpc-web

# Register cluster
argocd cluster add production-context --name production

2.2 Application CRD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
    kustomize:
      namePrefix: prod-
      commonLabels:
        env: production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jqPathExpressions:
        - .spec.metrics[].resource.target

2.3 Sync Hooks

# Pre-sync: DB migration
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp/db-migrate:latest
          command: ["./migrate", "up"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url
      restartPolicy: Never
  backoffLimit: 3
---
# Post-sync: Slack notification
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-deploy
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"API Server deployed successfully to production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never
---
# SyncFail: Failure notification
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-fail
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"ALERT: API Server deployment FAILED in production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never

2.4 Custom Health Checks

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.argoproj.io_Rollout: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.currentStepIndex ~= nil then
        hs.status = "Progressing"
        hs.message = "Rollout in progress"
      end
      if obj.status.phase == "Healthy" then
        hs.status = "Healthy"
        hs.message = "Rollout is healthy"
      end
      if obj.status.phase == "Degraded" then
        hs.status = "Degraded"
        hs.message = "Rollout is degraded"
      end
    end
    return hs
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

2.5 RBAC Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Project admins
    p, role:project-admin, applications, *, production/*, allow
    p, role:project-admin, applications, sync, production/*, allow
    p, role:project-admin, logs, get, production/*, allow

    # Developers: read + sync only
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, staging/*, allow
    p, role:developer, logs, get, */*, allow

    # CI/CD systems
    p, role:ci-cd, applications, sync, */*, allow
    p, role:ci-cd, applications, get, */*, allow

    # Group mappings
    g, platform-team, role:admin
    g, backend-team, role:project-admin
    g, frontend-team, role:developer
    g, github-actions, role:ci-cd
  scopes: '[groups, email]'

3. Flux Deep Dive

3.1 Core Resources

# GitRepository: Define Git source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-repo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/platform-config
  ref:
    branch: main
  secretRef:
    name: git-credentials
  ignore: |
    # exclude all
    /*
    # include deploy dir
    !/deploy
---
# Kustomization: Define what to deploy where
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  retryInterval: 2m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server/production
  prune: true
  wait: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api-server
      namespace: production
  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: api-server
        spec:
          replicas: 5
      target:
        kind: Deployment
        name: api-server
---
# HelmRelease: Deploy Helm chart
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: production
spec:
  interval: 10m
  chart:
    spec:
      chart: redis
      version: "18.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    architecture: replication
    auth:
      enabled: true
      existingSecret: redis-auth
    replica:
      replicaCount: 3
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
  valuesFrom:
    - kind: ConfigMap
      name: redis-values
      valuesKey: values.yaml
  upgrade:
    remediation:
      retries: 3
  rollback:
    cleanupOnFail: true

3.2 Flux Notification Setup

# Provider: Slack notifications
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-webhook
---
# Alert: Notifications for specific events
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: deployment-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"
  exclusionList:
    - ".*upgrade.*retries exhausted.*"
  summary: "Flux deployment issue detected"

4. ArgoCD vs Flux Comparison

AspectArgoCDFlux
UIRich built-in web UISeparate UI needed (Weave GitOps, etc.)
ArchitectureCentralizedDistributed, namespace-scoped
CRD CountFew (Application, AppProject)Many (GitRepo, Kustomization, HelmRelease, etc.)
Helm SupportTemplate rendering then applyNative HelmRelease CRD
KustomizeBuilt-in supportBuilt-in support
Multi-tenancyAppProject isolationNamespace-based isolation
Image Auto-updateArgoCD Image UpdaterFlux Image Automation
SSOOIDC/SAML/LDAP supportDepends on UI tool
RBACFine-grained role-based accessKubernetes RBAC
CLIargocd CLIflux CLI
NotificationsNotification settingsAlert/Provider CRDs
Health CheckCustom Lua scriptsKustomization health checks
Drift DetectionReal-time UI displayPeriodic reconciliation
Progressive DeliveryArgo Rollouts integrationFlagger integration
Learning CurveMedium (UI helps)Higher (CRD-based config)

5. ApplicationSet

5.1 List Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: production
            url: https://prod-k8s.example.com
            namespace: production
            replicas: "5"
          - cluster: staging
            url: https://staging-k8s.example.com
            namespace: staging
            replicas: "2"
          - cluster: development
            url: https://dev-k8s.example.com
            namespace: development
            replicas: "1"
  template:
    metadata:
      name: "api-server-{{cluster}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "apps/api-server/overlays/{{cluster}}"
        kustomize:
          commonLabels:
            cluster: "{{cluster}}"
      destination:
        server: "{{url}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.2 Cluster Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-monitoring
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            monitoring: enabled
        values:
          prometheus_retention: "30d"
  template:
    metadata:
      name: "monitoring-{{name}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-charts.git
        targetRevision: main
        path: charts/monitoring
        helm:
          values: |
            cluster: "{{name}}"
            prometheus:
              retention: "{{values.prometheus_retention}}"
            grafana:
              ingress:
                host: "grafana-{{name}}.example.com"
      destination:
        server: "{{server}}"
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

5.3 Git Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/k8s-manifests.git
        revision: main
        directories:
          - path: "apps/*/overlays/production"
          - path: "apps/legacy-*"
            exclude: true
  template:
    metadata:
      name: "{{path[1]}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
# Git File Generator: based on config.json files
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dynamic-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/app-config.git
        revision: main
        files:
          - path: "environments/*/config.json"
  template:
    metadata:
      name: "{{app.name}}-{{environment}}"
    spec:
      project: "{{app.project}}"
      source:
        repoURL: "{{app.repo}}"
        targetRevision: "{{app.revision}}"
        path: "{{app.path}}"
      destination:
        server: "{{cluster.url}}"
        namespace: "{{app.namespace}}"

5.4 Matrix Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cross-cluster-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: production
          - list:
              elements:
                - app: api-server
                  chart: api-server
                  namespace: backend
                - app: web-frontend
                  chart: web-frontend
                  namespace: frontend
                - app: worker
                  chart: worker
                  namespace: backend
  template:
    metadata:
      name: "{{app}}-{{name}}"
    spec:
      project: production
      source:
        repoURL: https://github.com/myorg/helm-charts.git
        targetRevision: main
        path: "charts/{{chart}}"
        helm:
          valueFiles:
            - "values-{{metadata.labels.region}}.yaml"
      destination:
        server: "{{server}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.5 Pull Request Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pr-preview
  namespace: argocd
spec:
  generators:
    - pullRequest:
        github:
          owner: myorg
          repo: api-server
          tokenRef:
            secretName: github-token
            key: token
          labels:
            - preview
        requeueAfterSeconds: 30
  template:
    metadata:
      name: "pr-{{number}}-api-server"
      labels:
        preview: "true"
    spec:
      project: previews
      source:
        repoURL: https://github.com/myorg/api-server.git
        targetRevision: "{{head_sha}}"
        path: deploy/preview
        kustomize:
          namePrefix: "pr-{{number}}-"
          commonLabels:
            pr: "{{number}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-{{number}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

6. Image Updater

6.1 ArgoCD Image Updater

# Installation
# kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

# Application with image update configuration
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  annotations:
    # Image list and update strategy
    argocd-image-updater.argoproj.io/image-list: "app=myregistry.com/api-server"
    argocd-image-updater.argoproj.io/app.update-strategy: semver
    argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$"
    argocd-image-updater.argoproj.io/app.ignore-tags: "latest, nightly"
    # Git write-back method
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: "kustomization:../../base"
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production

6.2 Flux Image Automation

# ImageRepository: Watch image registry
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: api-server
  namespace: flux-system
spec:
  image: myregistry.com/api-server
  interval: 5m
  secretRef:
    name: registry-credentials
---
# ImagePolicy: Define which tags to select
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: api-server
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: api-server
  policy:
    semver:
      range: ">=1.0.0"
  filterTags:
    pattern: "^v(?P<version>[0-9]+\\.[0-9]+\\.[0-9]+)$"
    extract: "$version"
---
# ImageUpdateAutomation: Auto-commit to Git
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: auto-update
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux-bot
        email: flux@myorg.com
      messageTemplate: |
        chore(image): update images

        Automated image update:
        {{range .Changed.Changes}}
        - {{.OldValue}} -> {{.NewValue}}
        {{end}}
    push:
      branch: main
  update:
    path: ./deploy
    strategy: Setters

7. Helm + GitOps

7.1 Using Helm with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nginx-ingress
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://kubernetes.github.io/ingress-nginx
    chart: ingress-nginx
    targetRevision: 4.9.1
    helm:
      releaseName: nginx-ingress
      values: |
        controller:
          replicaCount: 3
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          metrics:
            enabled: true
            serviceMonitor:
              enabled: true
          admissionWebhooks:
            enabled: true
      valueFiles:
        - values-production.yaml
      parameters:
        - name: controller.service.type
          value: LoadBalancer
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-system
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true

7.2 Umbrella Chart Pattern

# Chart.yaml (umbrella chart)
apiVersion: v2
name: platform-stack
version: 1.0.0
dependencies:
  - name: prometheus
    version: 25.x.x
    repository: https://prometheus-community.github.io/helm-charts
    condition: prometheus.enabled
  - name: grafana
    version: 7.x.x
    repository: https://grafana.github.io/helm-charts
    condition: grafana.enabled
  - name: loki
    version: 5.x.x
    repository: https://grafana.github.io/helm-charts
    condition: loki.enabled

8. Kustomize + GitOps

8.1 Per-Environment Overlays

apps/api-server/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    development/
      kustomization.yaml
      patches/
        replicas.yaml
    staging/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
    production/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
        hpa.yaml
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: api-server
---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namePrefix: prod-
patches:
  - path: patches/replicas.yaml
  - path: patches/resources.yaml
  - path: patches/hpa.yaml
configMapGenerator:
  - name: api-config
    literals:
      - LOG_LEVEL=warn
      - ENVIRONMENT=production
---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 5

9. Secret Management

9.1 Sealed Secrets

# Installation
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Encrypt a secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=super-secret \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  --format yaml > sealed-db-credentials.yaml
# sealed-db-credentials.yaml (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgB1k2j3l4m5n6o7p8...
    password: AgC9d0e1f2g3h4i5j6...
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

9.2 External Secrets Operator

# SecretStore: Vault connection
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: production-app
          serviceAccountRef:
            name: vault-auth
---
# ExternalSecret: Sync secrets from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.production:5432/myapp"
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
---
# ClusterSecretStore: Cluster-wide usage
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

9.3 SOPS (Secrets OPerationS)

# .sops.yaml configuration file
creation_rules:
  - path_regex: ".*\\.enc\\.yaml$"
    encrypted_regex: "^(data|stringData)$"
    age: age1ql3z7hjy54pw3hyww5ay...
  - path_regex: "environments/production/.*"
    kms: "arn:aws:kms:ap-northeast-2:123456789012:key/abc-123"
  - path_regex: "environments/staging/.*"
    pgp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
# Encrypt with SOPS
sops --encrypt --in-place secrets.enc.yaml

# Flux Kustomization using SOPS
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age-key

10. Progressive Delivery

10.1 Argo Rollouts - Canary Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: myregistry.com/api-server:v2.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi
  strategy:
    canary:
      canaryService: api-server-canary
      stableService: api-server-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: api-server-vsvc
              routes:
                - primary
      steps:
        - setWeight: 5
        - pause:
            duration: 5m
        - setWeight: 20
        - pause:
            duration: 10m
        - setWeight: 50
        - pause:
            duration: 10m
        - setWeight: 80
        - pause:
            duration: 5m
      analysis:
        templates:
          - templateName: success-rate
          - templateName: latency
        startingStep: 1
        args:
          - name: service-name
            value: api-server-canary
---
# AnalysisTemplate: Success rate validation
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 60s
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
---
# AnalysisTemplate: Latency validation
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency
spec:
  args:
    - name: service-name
  metrics:
    - name: p99-latency
      interval: 60s
      successCondition: result[0] < 500
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m]))
              by (le)
            ) * 1000

10.2 Argo Rollouts - Blue-Green Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-frontend
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      containers:
        - name: web
          image: myregistry.com/web-frontend:v3.0.0
          ports:
            - containerPort: 3000
  strategy:
    blueGreen:
      activeService: web-frontend-active
      previewService: web-frontend-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
        args:
          - name: preview-url
            value: "http://web-frontend-preview.production.svc:3000"
      postPromotionAnalysis:
        templates:
          - templateName: success-rate
        args:
          - name: service-name
            value: web-frontend-active

11. Multi-Cluster GitOps

11.1 Hub-Spoke Pattern

management-cluster/          (Hub)
  argocd/
    applicationsets/
      monitoring.yaml         -> Deploy monitoring to all clusters
      logging.yaml            -> Deploy logging to all clusters
      apps.yaml               -> Deploy apps per environment
    projects/
      platform.yaml
      production.yaml
      staging.yaml
    clusters/
      prod-us-east.yaml
      prod-ap-northeast.yaml
      staging-us-east.yaml

11.2 Repository Structure: Monorepo vs Polyrepo

Monorepo Structure:

k8s-manifests/
  apps/
    api-server/
      base/
      overlays/
        development/
        staging/
        production/
    web-frontend/
      base/
      overlays/
        development/
        staging/
        production/
  platform/
    monitoring/
    logging/
    ingress/
  clusters/
    production/
    staging/

Polyrepo Structure:

# repo: api-server-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: web-frontend-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: platform-config
monitoring/
logging/
ingress/
AspectMonorepoPolyrepo
VisibilityEntire system at a glanceIndependent per service
Access ControlDirectory-based (CODEOWNERS)Per-repository
CI/CDComplex change detectionSimple triggers
Dependency MgmtEasyVersion management needed
ScaleSuits small-mediumSuits large organizations

11.3 App-of-Apps Pattern

# Root Application: manages other Applications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: clusters/production
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

12. Drift Detection and Reconciliation

12.1 ArgoCD Drift Detection

ArgoCD continuously compares the desired state in Git with the actual cluster state in real-time.

# Exclude specific fields from drift detection
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
spec:
  ignoreDifferences:
    # Ignore replicas managed by HPA
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    # Ignore auto-generated annotations
    - group: ""
      kind: Service
      jqPathExpressions:
        - .metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-*"
    # Ignore sidecar injected by MutatingWebhook
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - istio-sidecar-injector

12.2 Flux Reconciliation

# Force reconciliation
# flux reconcile kustomization api-server --with-source

# Check reconciliation status
# flux get kustomizations
# flux get helmreleases -A

# Suspend specific resource
# flux suspend kustomization api-server
# flux resume kustomization api-server

13. Practice Quiz

Q1: Why is the Pull-based GitOps model more secure than Push-based (traditional CI/CD)?

Answer: In the Pull-based model, an agent inside the cluster (ArgoCD/Flux) polls Git and applies changes. There is no need to expose cluster access credentials to the CI pipeline.

  • Push model: CI server deploys directly via kubectl/helm -> cluster credentials needed in CI
  • Pull model: In-cluster agent watches Git -> cluster credentials never exposed externally
  • Reduced attack surface: Even if the CI system is compromised, direct cluster access is not possible
Q2: When is the ApplicationSet Matrix Generator useful?

Answer: The Matrix Generator combines two generators to create a Cartesian product (all combinations). It is useful when deploying multiple applications to multiple clusters.

  • Example: 3 clusters (prod-us, prod-eu, prod-ap) x 5 apps = 15 Applications auto-generated
  • Cluster Generator + List Generator combination is most common
  • Manages all combinations automatically without duplicate configuration
Q3: What is the difference between git and argocd write-back methods in ArgoCD Image Updater?

Answer: The git method commits image tag changes to the Git repository. The argocd method directly modifies the ArgoCD Application parameters.

  • git: Git remains the single source of truth. Audit trail available. Slower
  • argocd: Does not modify Git. Faster but Git and actual state may diverge
  • git method is recommended for production
Q4: What are the key differences between Sealed Secrets and External Secrets Operator?

Answer: Sealed Secrets encrypts secrets and stores them in Git, while External Secrets Operator synchronizes from external secret stores (Vault, AWS SM, etc.) at runtime.

  • Sealed Secrets: Encrypted secrets stored in Git. Works offline. Key rotation is complex
  • External Secrets Operator: Real-time sync from external stores. Centralized management. External service dependency
  • Small teams: Sealed Secrets is simpler
  • Large organizations: External Secrets + Vault combination recommended
Q5: What happens when an AnalysisTemplate fails during Argo Rollouts canary deployment?

Answer: When the metrics in the AnalysisTemplate fail to meet the successCondition and reach the failureLimit, the Rollout automatically rolls back.

  • Canary traffic reverts to 0%
  • The stable version receives all traffic
  • Rollout status changes to Degraded
  • Manual retry: kubectl argo rollouts retry rollout api-server
  • Check analysis results: kubectl argo rollouts get rollout api-server

14. References

  1. ArgoCD Official Documentation
  2. Flux Official Documentation
  3. ApplicationSet Documentation
  4. ArgoCD Image Updater
  5. Argo Rollouts
  6. Sealed Secrets
  7. External Secrets Operator
  8. SOPS
  9. Flagger
  10. GitOps Principles - OpenGitOps
  11. Kustomize
  12. Helm
  13. Weave GitOps