- 前言
- ArgoCD 架构概览
- Application 资源的基本构成
- 用 AppProject 实现 RBAC
- ApplicationSet 生成器
- 用 Sync Waves 控制部署顺序
- Sync Hook 的运用
- Secret 管理
- 多集群部署
- 回滚策略
- 监控配置
- 运维注意事项
- 故障案例与恢复流程
- 结语
- 参考资料

前言
GitOps 是一种把 Git 当作 Single Source of Truth、以声明式方式管理基础设施与应用期望状态的运维范式。ArgoCD 是在 Kubernetes 环境中实现 GitOps 最广泛使用的工具,作为 CNCF Graduated 项目,已经完成了生产环境验证。
本文将结合实战代码,介绍 ArgoCD 的核心能力:通过 ApplicationSet 实现多集群与多环境部署,通过 Sync Waves 和 Hook 控制部署顺序,以及 RBAC 与 Secret 管理等生产环境所需的全部配置。
ArgoCD 架构概览
ArgoCD 由以下核心组件构成。
| 组件 | 职责 | 主要功能 |
|---|---|---|
| API Server | Web UI、CLI、CI/CD 集成 | 提供 gRPC/REST API,处理 RBAC |
| Repo Server | Git 仓库管理 | 生成清单(Helm、Kustomize、Plain YAML) |
| Application Controller | 核心调和循环 | 监视集群状态,执行同步 |
| ApplicationSet Controller | 批量生成 Application | 基于模板自动生成 Application |
| Redis | 缓存 | 清单缓存、集群状态缓存 |
| Dex | 认证 | SSO、OIDC、LDAP 等外部认证集成 |
ArgoCD 安装
# 创建 Namespace 并安装 ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# 或者用 Helm 安装(生产环境推荐)
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
--namespace argocd \
--create-namespace \
--set server.service.type=LoadBalancer \
--set configs.params."server\.insecure"=true \
--set controller.replicas=2 \
--set repoServer.replicas=2 \
--set redis-ha.enabled=true
# 确认初始 admin 密码
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d
Application 资源的基本构成
Application 规格详解
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-application
namespace: argocd
# Finalizer:删除 Application 时一并删除集群资源
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: production
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: main
path: apps/my-app/overlays/production
# Kustomize 选项
kustomize:
namePrefix: prod-
commonLabels:
env: production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true # 在集群中一并删除 Git 里已被删除的资源
selfHeal: true # 发生手动变更时自动恢复
allowEmpty: false # 防止部署空清单
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas # 由 HPA 管理,因此忽略
Helm 源配置
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prometheus-stack
namespace: argocd
spec:
project: monitoring
source:
repoURL: https://prometheus-community.github.io/helm-charts
chart: kube-prometheus-stack
targetRevision: 65.1.0
helm:
releaseName: prometheus
valuesObject:
grafana:
enabled: true
adminPassword: vault:secret/grafana#password
prometheus:
prometheusSpec:
retention: 30d
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 100Gi
destination:
server: https://kubernetes.default.svc
namespace: monitoring
用 AppProject 实现 RBAC
项目定义
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: production
namespace: argocd
spec:
description: 'Production environment project'
# 允许的源仓库
sourceRepos:
- 'https://github.com/myorg/k8s-manifests.git'
- 'https://github.com/myorg/helm-charts.git'
# 允许的部署目标
destinations:
- namespace: 'production'
server: 'https://kubernetes.default.svc'
- namespace: 'production-*'
server: 'https://kubernetes.default.svc'
# 允许的集群级资源
clusterResourceWhitelist:
- group: ''
kind: Namespace
- group: 'rbac.authorization.k8s.io'
kind: ClusterRole
- group: 'rbac.authorization.k8s.io'
kind: ClusterRoleBinding
# 禁止的命名空间级资源
namespaceResourceBlacklist:
- group: ''
kind: ResourceQuota
- group: ''
kind: LimitRange
# RBAC 角色定义
roles:
- name: deployer
description: 'Can sync and manage applications'
policies:
- p, proj:production:deployer, applications, get, production/*, allow
- p, proj:production:deployer, applications, sync, production/*, allow
- p, proj:production:deployer, applications, action/*, production/*, allow
groups:
- platform-team
- name: viewer
description: 'Read-only access'
policies:
- p, proj:production:viewer, applications, get, production/*, allow
groups:
- dev-team
ApplicationSet 生成器
Git Directory 生成器
基于 Git 仓库的目录结构自动生成 Application。
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: app-of-apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ['missingkey=error']
generators:
- git:
repoURL: https://github.com/myorg/k8s-manifests.git
revision: main
directories:
- path: 'apps/*/overlays/production'
- path: 'apps/deprecated-*'
exclude: true
template:
metadata:
name: '{{.path.basename}}'
namespace: argocd
labels:
app.kubernetes.io/managed-by: applicationset
spec:
project: production
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: main
path: '{{.path.path}}'
destination:
server: https://kubernetes.default.svc
namespace: '{{.path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
Cluster 生成器
基于已注册的集群,自动化多集群部署。
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: multi-cluster-apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ['missingkey=error']
generators:
- clusters:
selector:
matchLabels:
env: production
matchExpressions:
- key: region
operator: In
values:
- ap-northeast-2
- us-west-2
- eu-west-1
template:
metadata:
name: 'app-{{.name}}'
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: main
path: 'apps/my-app/overlays/{{.metadata.labels.env}}'
kustomize:
commonLabels:
cluster: '{{.name}}'
region: '{{.metadata.labels.region}}'
destination:
server: '{{.server}}'
namespace: production
Matrix 生成器(复合组合)
把两个生成器组合起来,生成所有组合。
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: matrix-deployment
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ['missingkey=error']
generators:
- matrix:
generators:
# 生成器 1:集群列表
- clusters:
selector:
matchLabels:
env: production
# 生成器 2:从 Git 目录取得应用列表
- git:
repoURL: https://github.com/myorg/k8s-manifests.git
revision: main
directories:
- path: 'apps/*'
template:
metadata:
name: '{{.path.basename}}-{{.name}}'
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: main
path: '{{.path.path}}/overlays/production'
destination:
server: '{{.server}}'
namespace: '{{.path.basename}}'
List 生成器
基于静态的值列表生成 Application。
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: environment-apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ['missingkey=error']
generators:
- list:
elements:
- env: dev
cluster: https://dev-k8s.example.com
revision: develop
replicas: '1'
- env: staging
cluster: https://staging-k8s.example.com
revision: release/v2.5
replicas: '2'
- env: production
cluster: https://kubernetes.default.svc
revision: main
replicas: '3'
template:
metadata:
name: 'my-app-{{.env}}'
namespace: argocd
spec:
project: '{{.env}}'
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: '{{.revision}}'
path: 'apps/my-app/overlays/{{.env}}'
destination:
server: '{{.cluster}}'
namespace: 'my-app-{{.env}}'
用 Sync Waves 控制部署顺序
Sync Wave 基本概念
Sync Wave 通过 argocd.argoproj.io/sync-wave 注解定义,使用整数值,从较小的值开始依次部署。默认值是 0。
# Wave -2:命名空间与 RBAC(前置条件)
apiVersion: v1
kind: Namespace
metadata:
name: production
annotations:
argocd.argoproj.io/sync-wave: '-2'
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-service-account
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '-2'
---
# Wave -1:ConfigMap 与 Secret(配置)
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '-1'
data:
APP_ENV: production
LOG_LEVEL: info
---
# Wave 0:核心应用(默认值)
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '0'
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
serviceAccountName: app-service-account
containers:
- name: app
image: myorg/my-app:v2.5.0
ports:
- containerPort: 8080
envFrom:
- configMapRef:
name: app-config
---
apiVersion: v1
kind: Service
metadata:
name: my-app
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '0'
spec:
selector:
app: my-app
ports:
- port: 8080
targetPort: 8080
---
# Wave 1:依赖资源(Ingress、HPA)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: my-app-hpa
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '1'
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
---
# Wave 2:监控
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-monitor
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '2'
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: http
interval: 15s
Sync Hook 的运用
PreSync Hook:部署前的数据库迁移
apiVersion: batch/v1
kind: Job
metadata:
name: db-migration
namespace: production
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
argocd.argoproj.io/sync-wave: '-1'
spec:
backoffLimit: 3
template:
spec:
containers:
- name: migrate
image: myorg/db-migrator:v2.5.0
command: ['./migrate', 'up']
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-credentials
key: url
restartPolicy: Never
PostSync Hook:部署后的验证
apiVersion: batch/v1
kind: Job
metadata:
name: smoke-test
namespace: production
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 1
template:
spec:
containers:
- name: smoke-test
image: myorg/smoke-tester:latest
command:
- /bin/sh
- -c
- |
echo "Running smoke tests..."
curl -sf http://my-app.production.svc:8080/healthz || exit 1
curl -sf http://my-app.production.svc:8080/api/v1/status || exit 1
echo "All smoke tests passed!"
restartPolicy: Never
SyncFail Hook:同步失败时的通知
apiVersion: batch/v1
kind: Job
metadata:
name: sync-fail-notification
namespace: production
annotations:
argocd.argoproj.io/hook: SyncFail
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 1
template:
spec:
containers:
- name: notify
image: curlimages/curl:latest
command:
- /bin/sh
- -c
- |
curl -X POST "$SLACK_WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d '{"text":"ArgoCD Sync FAILED for my-app in production!"}'
env:
- name: SLACK_WEBHOOK_URL
valueFrom:
secretKeyRef:
name: slack-webhook
key: url
restartPolicy: Never
Secret 管理
Sealed Secrets 集成
# 安装 Sealed Secrets 控制器
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
--namespace kube-system
# 用 kubeseal CLI 加密 Secret
kubectl create secret generic db-credentials \
--namespace production \
--from-literal=url='postgresql://user:pass@db:5432/mydb' \
--dry-run=client -o yaml | \
kubeseal --format yaml > sealed-db-credentials.yaml
# 可以提交到 Git 的 SealedSecret
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-credentials
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '-2'
spec:
encryptedData:
url: AgA2X5N0Q...encrypted...base64==
template:
metadata:
name: db-credentials
namespace: production
type: Opaque
External Secrets Operator 集成
# ExternalSecret:从 AWS Secrets Manager 同步 Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
annotations:
argocd.argoproj.io/sync-wave: '-2'
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secretsmanager
kind: ClusterSecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: url
remoteRef:
key: production/database
property: connection_url
- secretKey: password
remoteRef:
key: production/database
property: password
---
# ClusterSecretStore 定义
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: aws-secretsmanager
spec:
provider:
aws:
service: SecretsManager
region: ap-northeast-2
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
namespace: external-secrets
多集群部署
集群注册
# 注册目标集群(基于 kubeconfig 上下文)
argocd cluster add staging-cluster \
--name staging \
--label env=staging \
--label region=ap-northeast-2
argocd cluster add production-cluster \
--name production \
--label env=production \
--label region=ap-northeast-2
# 确认已注册的集群
argocd cluster list
按集群拆分配置
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-services
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ['missingkey=error']
generators:
- matrix:
generators:
- clusters:
selector:
matchLabels:
env: production
- list:
elements:
- app: cert-manager
namespace: cert-manager
chart: cert-manager
repoURL: https://charts.jetstack.io
targetRevision: v1.16.0
- app: external-secrets
namespace: external-secrets
chart: external-secrets
repoURL: https://charts.external-secrets.io
targetRevision: 0.12.0
- app: metrics-server
namespace: kube-system
chart: metrics-server
repoURL: https://kubernetes-sigs.github.io/metrics-server
targetRevision: 3.12.0
template:
metadata:
name: '{{.app}}-{{.name}}'
namespace: argocd
spec:
project: platform
source:
repoURL: '{{.repoURL}}'
chart: '{{.chart}}'
targetRevision: '{{.targetRevision}}'
helm:
releaseName: '{{.app}}'
destination:
server: '{{.server}}'
namespace: '{{.namespace}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
回滚策略
自动回滚配置
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/myorg/k8s-manifests.git
targetRevision: main
path: apps/my-app/overlays/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
retry:
limit: 3
backoff:
duration: 10s
factor: 2
maxDuration: 5m
通过 CLI 手动回滚
# 确认部署历史
argocd app history my-app
# 回滚到特定修订版本
argocd app rollback my-app 5
# 或者同步到特定的 Git 提交
argocd app sync my-app --revision abc123def
# 确认同步状态
argocd app get my-app
argocd app wait my-app --health
监控配置
ArgoCD Prometheus 指标
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: argocd-metrics
namespace: argocd
spec:
selector:
matchLabels:
app.kubernetes.io/part-of: argocd
endpoints:
- port: metrics
interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: argocd-alerts
namespace: argocd
spec:
groups:
- name: argocd
rules:
- alert: ArgoCDAppOutOfSync
expr: |
argocd_app_info{sync_status="OutOfSync"} == 1
for: 10m
labels:
severity: warning
annotations:
summary: 'Application {{ "{{" }} $labels.name {{ "}}" }} is OutOfSync for more than 10 minutes'
- alert: ArgoCDAppDegraded
expr: |
argocd_app_info{health_status="Degraded"} == 1
for: 5m
labels:
severity: critical
annotations:
summary: 'Application {{ "{{" }} $labels.name {{ "}}" }} is Degraded'
- alert: ArgoCDSyncFailed
expr: |
increase(argocd_app_sync_total{phase="Failed"}[10m]) > 0
labels:
severity: critical
annotations:
summary: 'ArgoCD sync failed for {{ "{{" }} $labels.name {{ "}}" }}'
Slack 通知配置
# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
service.slack: |
token: $slack-token
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Error', 'Failed']
send: [app-sync-failed]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [app-health-degraded]
trigger.on-sync-succeeded: |
- when: app.status.operationState.phase in ['Succeeded']
oncePer: app.status.sync.revision
send: [app-sync-succeeded]
template.app-sync-failed: |
slack:
attachments: |
[{
"color": "#E96D76",
"title": "Sync Failed: {{.app.metadata.name}}",
"text": "Application {{.app.metadata.name}} sync failed.\nRevision: {{.app.status.sync.revision}}\nMessage: {{.app.status.operationState.message}}"
}]
template.app-health-degraded: |
slack:
attachments: |
[{
"color": "#f4c030",
"title": "Health Degraded: {{.app.metadata.name}}",
"text": "Application {{.app.metadata.name}} health is degraded."
}]
template.app-sync-succeeded: |
slack:
attachments: |
[{
"color": "#18BE52",
"title": "Sync Succeeded: {{.app.metadata.name}}",
"text": "Application {{.app.metadata.name}} synced successfully.\nRevision: {{.app.status.sync.revision}}"
}]
运维注意事项
1. ApplicationSet 安全
如果 ApplicationSet 的 project 字段被模板化,开发者就可能把 Application 创建到权限过大的项目中。必须始终把配置限制为:只从管理员控制的来源引用 project 字段。
2. 自动同步的注意点
automated.prune: true 会把 Git 中已移除的资源从集群里删除。一旦误删清单,生产资源可能立刻被删除,因此建议给重要资源加上 argocd.argoproj.io/sync-options: Prune=false 注解。
3. Webhook 安全
如果 ArgoCD 可以被公网访问,务必配置 Webhook Secret,以防止 DDoS 攻击。
4. Repo Server 资源
处理大型仓库或 Helm chart 时,Repo Server 的内存使用量可能急剧上升。需要设置合适的资源限制与副本数。
5. 抽象层级限制
使用 Application of Applications 模式时,应避免超过 3 层的抽象。4-5 层的嵌套会让调试变得极其困难。
故障案例与恢复流程
案例 1:Sync 无限循环
# 症状:Application 反复停留在 Syncing 状态
# 原因:未配置 ignoreDifferences 导致检测到 drift
# 诊断
argocd app diff my-app
argocd app get my-app --show-operation
# 恢复:添加 ignoreDifferences
kubectl patch application my-app -n argocd --type merge -p '{
"spec": {
"ignoreDifferences": [
{
"group": "apps",
"kind": "Deployment",
"jsonPointers": ["/spec/replicas"]
}
]
}
}'
案例 2:Repo Server OOM
# 症状:Application 处于 Unknown 状态,清单生成失败
# 诊断
kubectl logs -n argocd deploy/argocd-repo-server --previous
kubectl top pods -n argocd
# 恢复:提高资源上限
kubectl patch deployment argocd-repo-server -n argocd --type json -p '[
{
"op": "replace",
"path": "/spec/template/spec/containers/0/resources/limits/memory",
"value": "4Gi"
}
]'
# 重启 Repo Server
kubectl rollout restart deployment/argocd-repo-server -n argocd
案例 3:Secret 同步失败(External Secrets)
# 症状:Application 是 Healthy,但 Pod 处于 CrashLoopBackOff
# 原因:ExternalSecret 尚未完成同步
# 诊断
kubectl get externalsecret -n production
kubectl describe externalsecret db-credentials -n production
# 恢复:强制同步 ExternalSecret
kubectl annotate externalsecret db-credentials \
-n production \
force-sync=$(date +%s) --overwrite
# 或者用 Sync Wave 保证顺序
# ExternalSecret: sync-wave=-2, Deployment: sync-wave=0
案例 4:ApplicationSet 意外删除
# 症状:修改 ApplicationSet 之后,原有的 Application 被删除
# 原因:生成器配置有误,导致匹配到的条目减少
# 预防:配置 preserveResourcesOnDeletion
kubectl patch applicationset my-appset -n argocd --type merge -p '{
"spec": {
"syncPolicy": {
"preserveResourcesOnDeletion": true
}
}
}'
# 恢复:从 Git 历史还原之前的 ApplicationSet 配置
git log --oneline -- applicationsets/my-appset.yaml
git checkout HEAD~1 -- applicationsets/my-appset.yaml
git commit -m "Revert ApplicationSet to restore applications"
git push
结语
作为 GitOps 的核心工具,ArgoCD 提供了生产环境所需的全部能力:通过 ApplicationSet 实现多集群自动化,通过 Sync Waves 与 Hook 精细控制部署顺序,以及通过 RBAC 和 Secret 管理强化安全。
关键在于始终把 Git 维持为 Single Source of Truth,同时设定恰当的自动化程度。与其对所有环境都启用自动同步,不如采取渐进式做法:开发与预发布环境用自动同步,生产环境用手动同步。也请务必同时配置好监控与告警,以便迅速发现并应对同步失败或健康检查异常。
参考资料
현재 단락 (1/749)
GitOps 是一种把 Git 当作 Single Source of Truth、以声明式方式管理基础设施与应用期望状态的运维范式。ArgoCD 是在 Kubernetes 环境中实现 GitOps...