LearnwithVishnu
LearnwithVishnu
Basics → Production → Architect
← Home
🐙ArgoCD
BeginnerEngineerProductionArchitectGitOps continuous delivery — sync, App of Apps, RBAC, multi-cluster
What is ArgoCDApplicationApp of AppsSync WavesRBACTroubleshootInterview Q&A

🐙 What is ArgoCD?

GitOps — The Core Concept

Traditional CD (Jenkins push)GitOps (ArgoCD pull)
How it worksJenkins runs kubectl apply when triggeredArgoCD polls Git, applies when changed
CredentialsJenkins has cluster credentialsOnly ArgoCD has cluster credentials
Audit trailJenkins build log (can be deleted)Git history (permanent, immutable)
RollbackRe-run old Jenkins buildgit revert — ArgoCD auto-applies
Drift detectionNoneAuto-detects and can auto-revert manual changes
Access controlAnyone with Jenkins access can deployDeployment = PR approval in Git
Install ArgoCD

📄 Application Object

The ArgoCD Application is the core object

One Application = one service + one environment. It defines: WHERE to get the config (Git repo, path, branch), WHERE to deploy (cluster + namespace), and HOW to sync (automatic or manual, with or without self-healing).

Complete Application YAML with all options

🏗️ App of Apps Pattern

Managing 15+ services

Instead of creating 15 Application objects manually, define them as YAML files in Git. A parent Application watches that directory. Add a service = add one YAML file. Bootstrap an environment = sync one parent app.

App of Apps + Git promotion workflow

⏳ Sync Waves & Hooks

Controlling deployment order

Wave 1 (database) must be ready before Wave 2 (app). PreSync hooks run database migrations. PostSync hooks run smoke tests. SyncFail hooks send alerts.

Sync waves and hooks

🔐 RBAC & Projects

Multi-team isolation

Projects in ArgoCD isolate teams. Team A cannot see or sync Team B's applications. Sync windows prevent production deployments outside business hours.

Projects + RBAC + sync windows

🔍 Troubleshooting

OutOfSync, sync fails, rollback

🔁 ApplicationSets — Deploy to Many Clusters at Once

The problem ApplicationSets solve

Without ApplicationSets: you create 10 ArgoCD Applications manually — one per environment or cluster. Each one is nearly identical. When you need to add a new environment, you create another Application manually. With ApplicationSets: define a template once, and a Generator automatically creates Applications for every environment, cluster, or Git directory.

Three key generators

GeneratorWhat it doesUse case
List GeneratorDefine a list of clusters/environments explicitlyFixed number of known environments
Git Generator (Directories)Create one Application per directory in a Git repoEach microservice in its own folder
Cluster GeneratorCreate one Application per registered clusterSame app deployed to all clusters
# ApplicationSet — deploy same app to dev, staging, prod
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-service-appset
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: dev
        url: https://dev-cluster.example.com
        namespace: payment-dev
        values:
          replicas: "1"
          imageTag: "latest"
      - cluster: staging
        url: https://staging-cluster.example.com
        namespace: payment-staging
        values:
          replicas: "2"
          imageTag: "stable"
      - cluster: production
        url: https://prod-cluster.example.com
        namespace: payment-prod
        values:
          replicas: "5"
          imageTag: "v2.1.0"
  template:
    metadata:
      name: "payment-service-{{cluster}}"
    spec:
      project: payment-team
      source:
        repoURL: https://github.com/company/payment-service
        targetRevision: HEAD
        path: helm/payment-service
        helm:
          values: |
            replicaCount: {{values.replicas}}
            image:
              tag: {{values.imageTag}}
      destination:
        server: "{{url}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

🔔 ArgoCD Notifications and Webhooks

Get alerted when deployments succeed, fail, or drift

ArgoCD Notifications sends alerts to Slack, Teams, email, PagerDuty, and more when Application state changes. Install via: kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/stable/manifests/install.yaml

Common notification triggers

TriggerWhen it fires
on-sync-succeededDeployment completed successfully
on-sync-failedSync attempt failed
on-health-degradedApp health became Degraded (pods crashing)
on-deployedNew version successfully deployed
on-sync-status-unknownArgoCD cannot determine sync status

Annotation to enable notifications on an Application

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments
    notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts
    notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts

Git webhook for instant sync (not polling)

By default ArgoCD polls Git every 3 minutes. For instant sync on push: configure a webhook in GitHub/GitLab pointing to https://argocd.example.com/api/webhook. ArgoCD receives the push event and syncs immediately — sub-second response to Git changes.

🌐 Multi-Cluster Management with ArgoCD

One ArgoCD managing many clusters

ArgoCD runs in one cluster (the management cluster) but can deploy to many target clusters. This is the standard enterprise pattern — one ArgoCD instance manages dev, staging, production, and regional clusters.

Register a cluster

# Login to ArgoCD
argocd login argocd.example.com

# Add a new cluster (must have kubeconfig context set up)
argocd cluster add production-cluster-context --name production

# List registered clusters
argocd cluster list

Hub-and-spoke architecture

Management Cluster
└── ArgoCD (the hub)
    ├── deploys to → Dev Cluster (spoke)
    ├── deploys to → Staging Cluster (spoke)
    ├── deploys to → Production EU Cluster (spoke)
    └── deploys to → Production US Cluster (spoke)

Security for multi-cluster

ArgoCD creates a ServiceAccount in each target cluster with the minimum permissions needed to deploy the resources defined in Applications. The credentials are stored as Kubernetes Secrets in the ArgoCD namespace. Use ArgoCD Projects to restrict which Applications can deploy to which clusters — Production project can only deploy to production cluster, accessed only by the release team.

🎯 Interview Questions

ARGOCD · ARCHITECT
What is GitOps and how does ArgoCD implement it?
GitOps is an operational model where Git is the single source of truth for infrastructure and application state. Every change to a production system happens through a Git commit and PR review — no direct kubectl apply, no console clicks, no shell scripts run manually. ArgoCD implements GitOps by continuously watching a Git repository. Every 3 minutes (default), ArgoCD fetches the desired state from Git and compares it with the actual state of the Kubernetes cluster. If they differ (sync needed), ArgoCD can automatically apply the Git state to the cluster. The critical shift: developers push to Git. ArgoCD pulls and applies. Nobody needs direct cluster credentials. Every deployment is a commit with author, timestamp, and review. Rollback is git revert — takes 2 minutes instead of 30. Drift detection: if someone manually kubectl applies something, ArgoCD marks the app as OutOfSync and can revert it automatically with selfHeal: true. At HPE: ArgoCD manages all telecom platform deployments. When a microservice update causes issues, rollback is git revert the image tag change. ArgoCD applies it within 3 minutes.
ARGOCD · ENGINEER
What is the App of Apps pattern in ArgoCD?
App of Apps is a pattern where a parent ArgoCD Application watches a directory in Git that contains other Application YAML files. The parent app manages the child apps. When you commit a new Application YAML file to that directory, ArgoCD automatically creates and manages that new application. Use App of Apps when: you have many services (10+) and want to manage them consistently, you want a single sync to bootstrap an entire environment, or you want ArgoCD to manage its own applications (self-managing). The directory structure: argocd/production/ contains: sro-app.yaml, com-app.yaml, monitoring.yaml. The parent app watches that directory. Adding a new service is just adding a new YAML file. Promoted from HPE design: we bootstrap staging with one parent app sync that creates 15 child applications in the right order using sync waves. Complete environment up in under 10 minutes.
ARGOCD · PRODUCTION
ArgoCD application is OutOfSync. Walk through diagnosis and fix.
OutOfSync means the actual cluster state differs from what is in Git. Step 1: identify what is different. argocd app diff app-name shows the exact diff — like git diff but for Kubernetes resources. Step 2: determine cause. Common causes: someone ran kubectl apply manually (config drift), Kubernetes mutated a resource (annotations, status fields), the app changed its own config at runtime (bad practice), or a new commit to Git changed something. Step 3: if the diff is intentional Git change — sync it. argocd app sync app-name. Step 4: if the diff is manual drift — the Git version is correct, revert the manual change. With selfHeal: true, ArgoCD does this automatically. Without it: argocd app sync --force. Step 5: if neither — investigate what changed. kubectl describe resource shows last-applied-configuration. Check audit logs. Step 6: prevent recurrence. Enable selfHeal: true in sync policy. Restrict direct kubectl access using Kubernetes RBAC or admission webhooks. Rule: in production, Git is always right. Manual changes should be exceptions with immediate follow-up to update Git.
ARGOCD · ARCHITECT
How do you manage multi-cluster deployments with ArgoCD?
ArgoCD can manage multiple Kubernetes clusters from one ArgoCD instance. Register each cluster: argocd cluster add context-name. Each Application's destination.server points to the cluster API endpoint. For environment promotion: three clusters (dev, staging, prod), separate Application objects pointing to each cluster, same Git repo but different branches or folder paths (environments/dev/, environments/staging/, environments/prod/). Promotion flow: merge PR to dev branch → ArgoCD syncs dev cluster. After QA approval: PR to update image tag in environments/staging/ → ArgoCD syncs staging cluster. After approval: PR to update environments/prod/ → ArgoCD syncs prod cluster. Each environment has its own ArgoCD Project with RBAC. Dev cluster: auto-sync allowed for all. Prod cluster: sync windows restrict deployments to business hours, requires manual sync approval. ApplicationSets (ArgoCD feature): generate Applications for multiple clusters from one template — useful when you have 10+ clusters following the same pattern (multi-region or multi-tenant).
Continue Learning
🔧 Jenkins⚡ GitHub Actions☸️ Kubernetes🏠 All Topics
ARGOCD · ENGINEER
What is ArgoCD drift detection and self-healing? How do you control it?
Drift detection: ArgoCD continuously compares the desired state in Git (the Helm chart or YAML manifests) with the actual state in the Kubernetes cluster. If someone runs kubectl edit deployment manually, or a HPA changes the replica count, ArgoCD detects the difference and shows the Application as OutOfSync. Self-healing (selfHeal: true): when enabled, ArgoCD automatically reverts any manual changes back to what Git says. This is the GitOps principle — Git is the only source of truth. If a developer runs kubectl scale deployment to 10, ArgoCD reverts it back to 3 (what values.yaml says) within seconds. Control: sync policies are configured per Application. For production, some teams prefer selfHeal: false — drift is detected and alerted, but not automatically reverted, requiring human approval to sync. For development environments, selfHeal: true gives full automation. Prune: controls whether resources deleted from Git are also deleted from the cluster. prune: true means removing a Kubernetes resource from your Git repo removes it from the cluster on next sync. prune: false means orphaned resources are left in the cluster — useful during migration periods.
ARGOCD · ENGINEER
Explain ArgoCD sync waves and hooks with a real deployment example.
Sync waves control the ORDER in which resources are applied during a sync. By default all resources sync simultaneously — this causes issues when a database Job must complete before application Pods start. Waves use the annotation: argocd.argoproj.io/sync-wave: "N". Lower number = earlier. Example: wave -1: Namespace and RBAC resources. Wave 0: Database migration Job (must complete first). Wave 1: Backend API Deployment and Service. Wave 2: Frontend Deployment. Wave 3: Ingress. ArgoCD waits for each wave to be healthy before proceeding to the next. Sync hooks are actions that run at specific points in the sync lifecycle: PreSync (runs before sync begins — take a DB backup), Sync (runs during sync — custom deploy logic), PostSync (runs after sync completes — run smoke tests), SyncFail (runs if sync fails — rollback or notify). Example PostSync hook: a Kubernetes Job that runs integration tests after deployment. If the Job fails, ArgoCD marks the sync as failed and notifies the team. This is how you implement automated smoke testing as part of your GitOps pipeline without any separate CI step for post-deploy validation.
ARGOCD · PRODUCTION
ArgoCD Application is stuck in Degraded health. How do you troubleshoot?
Degraded means the Application deployed (Synced) but the Kubernetes resources are not healthy. Step 1: ArgoCD UI → click the Application → Resource Tree view. Find the degraded resource (red icon). Usually a Deployment, StatefulSet, or Pod. Step 2: click on the Pod resource in ArgoCD — it shows the pod status directly. Or: kubectl get pods -n namespace and kubectl describe pod for the degraded pod. Common causes: CrashLoopBackOff — application is crashing on startup. kubectl logs pod-name --previous shows logs from the crashed container. Check application startup errors, missing environment variables, failed DB connection. OOMKilled — pod exceeds memory limit. kubectl describe pod shows OOMKilled in Last State. Increase memory limit in Helm values, commit to Git, ArgoCD syncs the new limit. ImagePullBackOff — image tag does not exist or registry credentials missing. Check image tag in ArgoCD Application details vs what actually exists in the registry. Readiness probe failing — app starts but fails the HTTP health check. Check if the /health endpoint is responding on the correct port. Wrong Secret or ConfigMap — app cannot start because required secret does not exist in the namespace. Check ArgoCD notification alerts to get immediate visibility. In production at HPE, 90% of Degraded states were either CrashLoopBackOff from a misconfigured env var in the values file, or ImagePullBackOff from a failed image push earlier in the pipeline.
ARGOCD · ARCHITECT
How do you use ArgoCD Projects for multi-team isolation?
ArgoCD Projects provide multi-tenant isolation — different teams can use the same ArgoCD instance without interfering with each other. A Project defines: which Git repositories the team can use as sources, which clusters and namespaces they can deploy to, which Kubernetes resource types they can manage, who has access (RBAC). Example: payment-team Project — source: github.com/company/payment-* repos only, destination: production cluster, payment-prod namespace only, resources: all standard K8s resources but NOT ClusterRole (cannot escalate privileges), RBAC: payment engineers can Sync, payment leads can promote to production. If a payment engineer tries to deploy to the logistics namespace, ArgoCD rejects it at the Project policy level. Default project allows everything — never use it in production. Create dedicated projects for each team or application domain. Combined with AppProjects and OIDC-based SSO (GitHub teams mapped to ArgoCD roles), you get enterprise-grade multi-tenant GitOps without running separate ArgoCD instances.
🤖
AI Assistant
Ask anything about this topic
👋 Hi! I have read this page and can answer your questions.

Try asking: "Explain this topic in simple terms" or "Give me an example" or ask any specific question.