[ArgoCD] Sync Engine Analysis: Everything About Synchronization Mechanisms

Pending --> Running --> Succeeded
                |
                +--> Failed --> (Retry or Manual)

PreSync --> Sync --> PostSync
              |
              +--> SyncFail (only on Sync failure)

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp/migration:latest
          command: ['./migrate', 'up']
      restartPolicy: Never

1. Sort all Sync Phase resources by Wave order
2. For each Wave group:
   a. Apply in resource type order
   b. Perform kubectl apply equivalent for each resource
   c. Wait until all resources in the Wave become Healthy
3. Proceed to next Wave

apiVersion: batch/v1
kind: Job
metadata:
  name: notification
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify
          image: curlimages/curl:latest
          command:
            - curl
            - -X
            - POST
            - https://hooks.slack.com/services/XXX
            - -d
            - '{"text":"Deployment successful"}'
      restartPolicy: Never

apiVersion: batch/v1
kind: Job
metadata:
  name: failure-notification
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: notify-failure
          image: curlimages/curl:latest
          command:
            - curl
            - -X
            - POST
            - https://hooks.slack.com/services/XXX
            - -d
            - '{"text":"Deployment FAILED"}'
      restartPolicy: Never

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync|Sync|PostSync|SyncFail|Skip
    argocd.argoproj.io/hook-delete-policy: HookSucceeded|HookFailed|BeforeHookCreation

1. Application Controller starts Sync
2. Identify hook resources for the current Phase
3. Sort hook resources by Wave order
4. Apply each hook resource to cluster (Job, Pod, etc.)
5. Wait for hook completion (success/failure)
6. On success: proceed to next step; on failure: abort Sync or transition to SyncFail
7. Clean up hook resources per Delete Policy

# Wave -1: Infrastructure resources (created first)
apiVersion: v1
kind: Namespace
metadata:
  name: my-app
  annotations:
    argocd.argoproj.io/sync-wave: '-1'

---
# Wave 0: Configuration resources (default)
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  annotations:
    argocd.argoproj.io/sync-wave: '0'

---
# Wave 1: Application resources
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    argocd.argoproj.io/sync-wave: '1'

---
# Wave 2: External access resources
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    argocd.argoproj.io/sync-wave: '2'

1. Group all resources by Wave number
2. Execute from lowest Wave first
3. Apply default resource type ordering within each Wave
4. Wait until all resources in current Wave are Healthy
5. Proceed to next Wave
6. Abort entire Sync if any Wave fails

Phase 1: Namespaces and base configs
  1. Namespace
  2. NetworkPolicy
  3. ResourceQuota
  4. LimitRange
  5. PodSecurityPolicy
  6. ServiceAccount
  7. Secret
  8. SecretList
  9. ConfigMap

Phase 2: RBAC
  10. ClusterRole
  11. ClusterRoleBinding
  12. Role
  13. RoleBinding

Phase 3: CRD
  14. CustomResourceDefinition

Phase 4: Storage and volumes
  15. PersistentVolume
  16. PersistentVolumeClaim
  17. StorageClass

Phase 5: Services
  18. Service
  19. Endpoints

Phase 6: Workloads
  20. DaemonSet
  21. Deployment
  22. ReplicaSet
  23. StatefulSet
  24. Job
  25. CronJob

Phase 7: Routing
  26. Ingress
  27. IngressClass
  28. APIService

metadata:
  annotations:
    argocd.argoproj.io/tracking-id: 'my-app:apps/Deployment:default/nginx'

metadata:
  labels:
    app.kubernetes.io/instance: my-app

APP_NAME:GROUP/KIND:NAMESPACE/NAME

Examples:
  my-app:apps/Deployment:default/nginx
  my-app:/Service:default/nginx-svc
  my-app:networking.k8s.io/Ingress:default/nginx-ingress

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  application.resourceTrackingMethod: annotation # annotation | label | annotation+label

1. Desired State: Manifests generated from Git (target state)
2. Live State: Actual state running in the cluster
3. Last Applied: Last applied configuration (recorded in annotation)

// Diff logic (simplified)
func diff(desired, live *unstructured.Unstructured) (*DiffResult, error) {
    // 1. Normalize
    normalizedDesired := normalize(desired)
    normalizedLive := normalize(live)

    // 2. Remove ignored fields
    removeIgnoredFields(normalizedDesired)
    removeIgnoredFields(normalizedLive)

    // 3. Structural comparison
    result := structuredMergeDiff(normalizedDesired, normalizedLive)

    return result, nil
}

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas # Ignore replica count managed by HPA
    - group: ''
      kind: ConfigMap
      jqPathExpressions:
        - .data.generated-field # Ignore auto-generated fields

data:
  resource.customizations.ignoreDifferences.all: |
    managedFieldsManagers:
      - kube-controller-manager
      - kube-scheduler

Healthy: All replicas Ready and update complete
Progressing: Rollout in progress (creating new ReplicaSet)
Degraded: Replicas failed to reach Ready state

Healthy: All replicas Ready and currentRevision == updateRevision
Progressing: Update in progress
Degraded: Replicas not Ready

Healthy: Running state with all containers Ready
Progressing: Pending or ContainerCreating state
Degraded: CrashLoopBackOff, ImagePullBackOff, etc.

Healthy: Successfully completed (Completed)
Progressing: Running (Active)
Degraded: Failed

# argocd-cm ConfigMap
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for _, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" then
            if condition.status == "True" then
              hs.status = "Healthy"
              hs.message = "Certificate is ready"
            else
              hs.status = "Degraded"
              hs.message = condition.message
            end
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

1. Generate complete resource list from Git manifests
2. Query ArgoCD-managed resources in cluster (tracking label/annotation)
3. Identify resources in cluster but not in Git (= Prune targets)
4. Delete resources according to deletion policy

- Recursively deletes owned resources
- Deleting a Deployment also deletes its ReplicaSet and Pods
- Uses Kubernetes garbage collection mechanism

- Owned resources deleted first, then parent resource
- Used when ordering must be guaranteed
- Order control via finalizers

# Add Prune prevention annotation to resource
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

# Disable Prune at Application level
spec:
  syncPolicy:
    automated:
      prune: false

spec:
  syncPolicy:
    automated:
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Retry 1: After 5s
Retry 2: After 10s (5s * 2)
Retry 3: After 20s (10s * 2)
Retry 4: After 40s (20s * 2)
Retry 5: After 80s (40s * 2) -> capped at 3m max

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - Replace=false
      - ServerSideApply=true
      - ApplyOutOfSyncOnly=true
      - Validate=true
      - RespectIgnoreDifferences=true

Benefits:
  - Field Ownership tracking
  - Conflict prevention between multiple controllers
  - More accurate 3-way merge
  - Performance improvement for large resources

Configuration:
  syncOptions:
    - ServerSideApply=true

spec:
  syncWindows:
    # Allow Sync only during weekday business hours
    - kind: allow
      schedule: '0 9 * * 1-5'
      duration: 9h
      applications:
        - '*'
      namespaces:
        - 'production'
    # Deny Sync on weekends
    - kind: deny
      schedule: '0 0 * * 0,6'
      duration: 24h
      applications:
        - '*'
    # Allow only manual Sync during off-hours
    - kind: allow
      schedule: '0 18 * * 1-5'
      duration: 15h
      manualSync: true
      applications:
        - 'critical-*'

1. deny takes priority over allow
2. More specific rules take priority at the same level
3. manualSync=true allows only manual Sync

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-app
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ServerSideApply=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 3
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 5m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jqPathExpressions:
        - .status