Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

1. Introduction: What is GitOps

GitOps is an operational methodology that uses Git repositories as the single source of truth for infrastructure and applications. Declarative configuration files are stored in Git, and automated processes synchronize the actual cluster state with the desired state in Git.

The 4 Principles of GitOps:

Declarative - The desired state of the system is described declaratively
Versioned - The desired state is stored in Git with full history tracking
Pulled - Agents automatically detect and apply desired state changes (Pull-based)
Continuously Reconciled - Differences between actual and desired state are continuously detected and corrected

2. ArgoCD Deep Dive

2.1 Architecture

ArgoCD consists of these core components:

API Server: Interface for UI, CLI, and CI/CD systems
Repository Server: Internal service that generates manifests from Git repositories
Application Controller: Monitors running application state and synchronizes

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# CLI login
argocd login argocd.example.com --grpc-web

# Register cluster
argocd cluster add production-context --name production

2.2 Application CRD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
    kustomize:
      namePrefix: prod-
      commonLabels:
        env: production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jqPathExpressions:
        - .spec.metrics[].resource.target

2.3 Sync Hooks

# Pre-sync: DB migration
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp/db-migrate:latest
          command: ["./migrate", "up"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url
      restartPolicy: Never
  backoffLimit: 3
---
# Post-sync: Slack notification
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-deploy
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"API Server deployed successfully to production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never
---
# SyncFail: Failure notification
apiVersion: batch/v1
kind: Job
metadata:
  name: notify-fail
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl
          command:
            - /bin/sh
            - -c
            - |
              curl -X POST -H 'Content-type: application/json' \
                --data '{"text":"ALERT: API Server deployment FAILED in production"}' \
                https://hooks.slack.com/services/T00/B00/xxx
      restartPolicy: Never

2.4 Custom Health Checks

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.argoproj.io_Rollout: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.currentStepIndex ~= nil then
        hs.status = "Progressing"
        hs.message = "Rollout in progress"
      end
      if obj.status.phase == "Healthy" then
        hs.status = "Healthy"
        hs.message = "Rollout is healthy"
      end
      if obj.status.phase == "Degraded" then
        hs.status = "Degraded"
        hs.message = "Rollout is degraded"
      end
    end
    return hs
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

2.5 RBAC Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Project admins
    p, role:project-admin, applications, *, production/*, allow
    p, role:project-admin, applications, sync, production/*, allow
    p, role:project-admin, logs, get, production/*, allow

    # Developers: read + sync only
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, staging/*, allow
    p, role:developer, logs, get, */*, allow

    # CI/CD systems
    p, role:ci-cd, applications, sync, */*, allow
    p, role:ci-cd, applications, get, */*, allow

    # Group mappings
    g, platform-team, role:admin
    g, backend-team, role:project-admin
    g, frontend-team, role:developer
    g, github-actions, role:ci-cd
  scopes: '[groups, email]'

3. Flux Deep Dive

3.1 Core Resources

# GitRepository: Define Git source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-repo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/platform-config
  ref:
    branch: main
  secretRef:
    name: git-credentials
  ignore: |
    # exclude all
    /*
    # include deploy dir
    !/deploy
---
# Kustomization: Define what to deploy where
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  retryInterval: 2m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server/production
  prune: true
  wait: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api-server
      namespace: production
  patches:
    - patch: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: api-server
        spec:
          replicas: 5
      target:
        kind: Deployment
        name: api-server
---
# HelmRelease: Deploy Helm chart
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: production
spec:
  interval: 10m
  chart:
    spec:
      chart: redis
      version: "18.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    architecture: replication
    auth:
      enabled: true
      existingSecret: redis-auth
    replica:
      replicaCount: 3
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
  valuesFrom:
    - kind: ConfigMap
      name: redis-values
      valuesKey: values.yaml
  upgrade:
    remediation:
      retries: 3
  rollback:
    cleanupOnFail: true

3.2 Flux Notification Setup

# Provider: Slack notifications
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-webhook
---
# Alert: Notifications for specific events
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: deployment-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"
  exclusionList:
    - ".*upgrade.*retries exhausted.*"
  summary: "Flux deployment issue detected"

4. ArgoCD vs Flux Comparison

Aspect	ArgoCD	Flux
UI	Rich built-in web UI	Separate UI needed (Weave GitOps, etc.)
Architecture	Centralized	Distributed, namespace-scoped
CRD Count	Few (Application, AppProject)	Many (GitRepo, Kustomization, HelmRelease, etc.)
Helm Support	Template rendering then apply	Native HelmRelease CRD
Kustomize	Built-in support	Built-in support
Multi-tenancy	AppProject isolation	Namespace-based isolation
Image Auto-update	ArgoCD Image Updater	Flux Image Automation
SSO	OIDC/SAML/LDAP support	Depends on UI tool
RBAC	Fine-grained role-based access	Kubernetes RBAC
CLI	argocd CLI	flux CLI
Notifications	Notification settings	Alert/Provider CRDs
Health Check	Custom Lua scripts	Kustomization health checks
Drift Detection	Real-time UI display	Periodic reconciliation
Progressive Delivery	Argo Rollouts integration	Flagger integration
Learning Curve	Medium (UI helps)	Higher (CRD-based config)

5. ApplicationSet

5.1 List Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: production
            url: https://prod-k8s.example.com
            namespace: production
            replicas: "5"
          - cluster: staging
            url: https://staging-k8s.example.com
            namespace: staging
            replicas: "2"
          - cluster: development
            url: https://dev-k8s.example.com
            namespace: development
            replicas: "1"
  template:
    metadata:
      name: "api-server-{{cluster}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "apps/api-server/overlays/{{cluster}}"
        kustomize:
          commonLabels:
            cluster: "{{cluster}}"
      destination:
        server: "{{url}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.2 Cluster Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-monitoring
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            monitoring: enabled
        values:
          prometheus_retention: "30d"
  template:
    metadata:
      name: "monitoring-{{name}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-charts.git
        targetRevision: main
        path: charts/monitoring
        helm:
          values: |
            cluster: "{{name}}"
            prometheus:
              retention: "{{values.prometheus_retention}}"
            grafana:
              ingress:
                host: "grafana-{{name}}.example.com"
      destination:
        server: "{{server}}"
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

5.3 Git Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/k8s-manifests.git
        revision: main
        directories:
          - path: "apps/*/overlays/production"
          - path: "apps/legacy-*"
            exclude: true
  template:
    metadata:
      name: "{{path[1]}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
# Git File Generator: based on config.json files
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dynamic-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/app-config.git
        revision: main
        files:
          - path: "environments/*/config.json"
  template:
    metadata:
      name: "{{app.name}}-{{environment}}"
    spec:
      project: "{{app.project}}"
      source:
        repoURL: "{{app.repo}}"
        targetRevision: "{{app.revision}}"
        path: "{{app.path}}"
      destination:
        server: "{{cluster.url}}"
        namespace: "{{app.namespace}}"

5.4 Matrix Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cross-cluster-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: production
          - list:
              elements:
                - app: api-server
                  chart: api-server
                  namespace: backend
                - app: web-frontend
                  chart: web-frontend
                  namespace: frontend
                - app: worker
                  chart: worker
                  namespace: backend
  template:
    metadata:
      name: "{{app}}-{{name}}"
    spec:
      project: production
      source:
        repoURL: https://github.com/myorg/helm-charts.git
        targetRevision: main
        path: "charts/{{chart}}"
        helm:
          valueFiles:
            - "values-{{metadata.labels.region}}.yaml"
      destination:
        server: "{{server}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

5.5 Pull Request Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pr-preview
  namespace: argocd
spec:
  generators:
    - pullRequest:
        github:
          owner: myorg
          repo: api-server
          tokenRef:
            secretName: github-token
            key: token
          labels:
            - preview
        requeueAfterSeconds: 30
  template:
    metadata:
      name: "pr-{{number}}-api-server"
      labels:
        preview: "true"
    spec:
      project: previews
      source:
        repoURL: https://github.com/myorg/api-server.git
        targetRevision: "{{head_sha}}"
        path: deploy/preview
        kustomize:
          namePrefix: "pr-{{number}}-"
          commonLabels:
            pr: "{{number}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-{{number}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

6. Image Updater

6.1 ArgoCD Image Updater

# Installation
# kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

# Application with image update configuration
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
  namespace: argocd
  annotations:
    # Image list and update strategy
    argocd-image-updater.argoproj.io/image-list: "app=myregistry.com/api-server"
    argocd-image-updater.argoproj.io/app.update-strategy: semver
    argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$"
    argocd-image-updater.argoproj.io/app.ignore-tags: "latest, nightly"
    # Git write-back method
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: "kustomization:../../base"
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api-server/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production

6.2 Flux Image Automation

# ImageRepository: Watch image registry
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: api-server
  namespace: flux-system
spec:
  image: myregistry.com/api-server
  interval: 5m
  secretRef:
    name: registry-credentials
---
# ImagePolicy: Define which tags to select
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: api-server
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: api-server
  policy:
    semver:
      range: ">=1.0.0"
  filterTags:
    pattern: "^v(?P<version>[0-9]+\\.[0-9]+\\.[0-9]+)$"
    extract: "$version"
---
# ImageUpdateAutomation: Auto-commit to Git
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: auto-update
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux-bot
        email: flux@myorg.com
      messageTemplate: |
        chore(image): update images

        Automated image update:
        {{range .Changed.Changes}}
        - {{.OldValue}} -> {{.NewValue}}
        {{end}}
    push:
      branch: main
  update:
    path: ./deploy
    strategy: Setters

7. Helm + GitOps

7.1 Using Helm with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nginx-ingress
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://kubernetes.github.io/ingress-nginx
    chart: ingress-nginx
    targetRevision: 4.9.1
    helm:
      releaseName: nginx-ingress
      values: |
        controller:
          replicaCount: 3
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          metrics:
            enabled: true
            serviceMonitor:
              enabled: true
          admissionWebhooks:
            enabled: true
      valueFiles:
        - values-production.yaml
      parameters:
        - name: controller.service.type
          value: LoadBalancer
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-system
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true

7.2 Umbrella Chart Pattern

# Chart.yaml (umbrella chart)
apiVersion: v2
name: platform-stack
version: 1.0.0
dependencies:
  - name: prometheus
    version: 25.x.x
    repository: https://prometheus-community.github.io/helm-charts
    condition: prometheus.enabled
  - name: grafana
    version: 7.x.x
    repository: https://grafana.github.io/helm-charts
    condition: grafana.enabled
  - name: loki
    version: 5.x.x
    repository: https://grafana.github.io/helm-charts
    condition: loki.enabled

8. Kustomize + GitOps

8.1 Per-Environment Overlays

apps/api-server/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    development/
      kustomization.yaml
      patches/
        replicas.yaml
    staging/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
    production/
      kustomization.yaml
      patches/
        replicas.yaml
        resources.yaml
        hpa.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: api-server
---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namePrefix: prod-
patches:
  - path: patches/replicas.yaml
  - path: patches/resources.yaml
  - path: patches/hpa.yaml
configMapGenerator:
  - name: api-config
    literals:
      - LOG_LEVEL=warn
      - ENVIRONMENT=production
---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 5

9. Secret Management

9.1 Sealed Secrets

# Installation
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Encrypt a secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=super-secret \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  --format yaml > sealed-db-credentials.yaml

# sealed-db-credentials.yaml (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgB1k2j3l4m5n6o7p8...
    password: AgC9d0e1f2g3h4i5j6...
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

9.2 External Secrets Operator

# SecretStore: Vault connection
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: production-app
          serviceAccountRef:
            name: vault-auth
---
# ExternalSecret: Sync secrets from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.production:5432/myapp"
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
---
# ClusterSecretStore: Cluster-wide usage
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

9.3 SOPS (Secrets OPerationS)

# .sops.yaml configuration file
creation_rules:
  - path_regex: ".*\\.enc\\.yaml$"
    encrypted_regex: "^(data|stringData)$"
    age: age1ql3z7hjy54pw3hyww5ay...
  - path_regex: "environments/production/.*"
    kms: "arn:aws:kms:ap-northeast-2:123456789012:key/abc-123"
  - path_regex: "environments/staging/.*"
    pgp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"

# Encrypt with SOPS
sops --encrypt --in-place secrets.enc.yaml

# Flux Kustomization using SOPS

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-server
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-repo
  path: ./deploy/apps/api-server
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age-key

10. Progressive Delivery

10.1 Argo Rollouts - Canary Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: myregistry.com/api-server:v2.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi
  strategy:
    canary:
      canaryService: api-server-canary
      stableService: api-server-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: api-server-vsvc
              routes:
                - primary
      steps:
        - setWeight: 5
        - pause:
            duration: 5m
        - setWeight: 20
        - pause:
            duration: 10m
        - setWeight: 50
        - pause:
            duration: 10m
        - setWeight: 80
        - pause:
            duration: 5m
      analysis:
        templates:
          - templateName: success-rate
          - templateName: latency
        startingStep: 1
        args:
          - name: service-name
            value: api-server-canary
---
# AnalysisTemplate: Success rate validation
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 60s
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
---
# AnalysisTemplate: Latency validation
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency
spec:
  args:
    - name: service-name
  metrics:
    - name: p99-latency
      interval: 60s
      successCondition: result[0] < 500
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m]))
              by (le)
            ) * 1000

10.2 Argo Rollouts - Blue-Green Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-frontend
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      containers:
        - name: web
          image: myregistry.com/web-frontend:v3.0.0
          ports:
            - containerPort: 3000
  strategy:
    blueGreen:
      activeService: web-frontend-active
      previewService: web-frontend-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
        args:
          - name: preview-url
            value: "http://web-frontend-preview.production.svc:3000"
      postPromotionAnalysis:
        templates:
          - templateName: success-rate
        args:
          - name: service-name
            value: web-frontend-active

11. Multi-Cluster GitOps

11.1 Hub-Spoke Pattern

management-cluster/          (Hub)
  argocd/
    applicationsets/
      monitoring.yaml         -> Deploy monitoring to all clusters
      logging.yaml            -> Deploy logging to all clusters
      apps.yaml               -> Deploy apps per environment
    projects/
      platform.yaml
      production.yaml
      staging.yaml
    clusters/
      prod-us-east.yaml
      prod-ap-northeast.yaml
      staging-us-east.yaml

11.2 Repository Structure: Monorepo vs Polyrepo

Monorepo Structure:

k8s-manifests/
  apps/
    api-server/
      base/
      overlays/
        development/
        staging/
        production/
    web-frontend/
      base/
      overlays/
        development/
        staging/
        production/
  platform/
    monitoring/
    logging/
    ingress/
  clusters/
    production/
    staging/

Polyrepo Structure:

# repo: api-server-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: web-frontend-deploy
deploy/
  base/
  overlays/
    development/
    staging/
    production/

# repo: platform-config
monitoring/
logging/
ingress/

Aspect	Monorepo	Polyrepo
Visibility	Entire system at a glance	Independent per service
Access Control	Directory-based (CODEOWNERS)	Per-repository
CI/CD	Complex change detection	Simple triggers
Dependency Mgmt	Easy	Version management needed
Scale	Suits small-medium	Suits large organizations

11.3 App-of-Apps Pattern

# Root Application: manages other Applications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: clusters/production
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

12. Drift Detection and Reconciliation

12.1 ArgoCD Drift Detection

ArgoCD continuously compares the desired state in Git with the actual cluster state in real-time.

# Exclude specific fields from drift detection
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-server
spec:
  ignoreDifferences:
    # Ignore replicas managed by HPA
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    # Ignore auto-generated annotations
    - group: ""
      kind: Service
      jqPathExpressions:
        - .metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-*"
    # Ignore sidecar injected by MutatingWebhook
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - istio-sidecar-injector

12.2 Flux Reconciliation

# Force reconciliation
# flux reconcile kustomization api-server --with-source

# Check reconciliation status
# flux get kustomizations
# flux get helmreleases -A

# Suspend specific resource
# flux suspend kustomization api-server
# flux resume kustomization api-server

13. Practice Quiz

Q1: Why is the Pull-based GitOps model more secure than Push-based (traditional CI/CD)?

Answer: In the Pull-based model, an agent inside the cluster (ArgoCD/Flux) polls Git and applies changes. There is no need to expose cluster access credentials to the CI pipeline.

Push model: CI server deploys directly via kubectl/helm -> cluster credentials needed in CI
Pull model: In-cluster agent watches Git -> cluster credentials never exposed externally
Reduced attack surface: Even if the CI system is compromised, direct cluster access is not possible

Q2: When is the ApplicationSet Matrix Generator useful?

Answer: The Matrix Generator combines two generators to create a Cartesian product (all combinations). It is useful when deploying multiple applications to multiple clusters.

Example: 3 clusters (prod-us, prod-eu, prod-ap) x 5 apps = 15 Applications auto-generated
Cluster Generator + List Generator combination is most common
Manages all combinations automatically without duplicate configuration

Q3: What is the difference between git and argocd write-back methods in ArgoCD Image Updater?

Answer: The git method commits image tag changes to the Git repository. The argocd method directly modifies the ArgoCD Application parameters.

git: Git remains the single source of truth. Audit trail available. Slower
argocd: Does not modify Git. Faster but Git and actual state may diverge
git method is recommended for production

Q4: What are the key differences between Sealed Secrets and External Secrets Operator?

Answer: Sealed Secrets encrypts secrets and stores them in Git, while External Secrets Operator synchronizes from external secret stores (Vault, AWS SM, etc.) at runtime.

Sealed Secrets: Encrypted secrets stored in Git. Works offline. Key rotation is complex
External Secrets Operator: Real-time sync from external stores. Centralized management. External service dependency
Small teams: Sealed Secrets is simpler
Large organizations: External Secrets + Vault combination recommended

Q5: What happens when an AnalysisTemplate fails during Argo Rollouts canary deployment?

Answer: When the metrics in the AnalysisTemplate fail to meet the successCondition and reach the failureLimit, the Rollout automatically rolls back.

Canary traffic reverts to 0%
The stable version receives all traffic
Rollout status changes to Degraded
Manual retry: kubectl argo rollouts retry rollout api-server
Check analysis results: kubectl argo rollouts get rollout api-server

Table of Contents