ArgoCD

Everything that runs in this cluster is declared in one Git repository. ArgoCD is the agent that makes the cluster's actual state match that declaration — on a schedule, automatically, and with a visible audit trail. If I delete a deployment by accident, ArgoCD puts it back. If I want to change something, I change the YAML and push.

01 Overview

I run ArgoCD as the single source of deployment truth. The cluster is bootstrapped once with a root application that points at the repo; that root then discovers and manages every other application — the app-of-apps pattern. From that point on, the cluster is fully declarative.

The practical upshot: I can wipe the node, reinstall k3s, apply a single manifest, and walk away while the entire stack rebuilds itself in the right order.

Key facts

02 The app-of-apps tree

A single root Application is the only thing applied manually. It renders the k8s/apps/ directory of child Application manifests, each of which manages one component. Adding a new service to the cluster means adding one file to the repo.

The three self-managed entries — root, argocd, and longhorn — are installed imperatively during bootstrap and then adopted by their own Application manifests in k8s/apps/ with zero diff on first sync. Root watches the same directory it lives in, making the whole tree self-sustaining after a single kubectl apply.

03 A typical Application

Here's the manifest that manages Vault. Every child app follows this pattern — point at a path in the repo, pick a destination namespace, and let ArgoCD own it. Note selfHeal and prune: the cluster is not allowed to drift. Vault gets sync-wave: "-1" so it starts before ESO and all downstream apps.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"  # deploys before ESO and all apps
spec:
  project: default
  source:
    repoURL: https://github.com/AlyBadawy/hl-beta
    targetRevision: main
    path: k8s/components/vault
  destination:
    server: https://kubernetes.default.svc
    namespace: security
  syncPolicy:
    automated:
      prune: true      # delete resources removed from git
      selfHeal: true   # revert manual cluster changes
    syncOptions:
      - CreateNamespace=true

04 Bootstrapping & access

On a fresh node, the provisioning script handles everything, including restoring Longhorn volumes from backup, and installing ArgoCD imperatively. Then activate-gitops.sh applies the root Application and hands full ownership to ArgoCD. After that, the cluster is entirely GitOps-driven.

# steps 1–8: provision server, bootstrap ArgoCD, restore Longhorn volumes
$ ./provision/rebuild.sh
  prompts for: SERVER_IP, VAULT_UNSEAL_KEY, CLOUDFLARE_API_TOKEN
  then runs unattended through step 7; step 8 requires manual Longhorn UI interaction

# final step: apply root app-of-apps, GitOps takes over
$ ./provision/activate-gitops.sh
application.argoproj.io/root created
→ vault starts (wave -1) → auto-unseal CronJob unseals Vault
→ ESO syncs all secrets from Vault → apps start

# access ArgoCD UI before ingress is live
$ kubectl port-forward -n argocd svc/argocd-server 8080:80
$ kubectl -n argocd get secret argocd-initial-admin-secret \
    -o jsonpath="{.data.password}" | base64 -d

05 Sync behaviour

ArgoCD polls the repo every three minutes, but a GitHub webhook makes most syncs feel instant. The reconciliation loop is the same whether triggered by a push, a drift, or a manual click:

1. Compare — render the manifests in Git (via Kustomize + Helm inflation) and diff against live cluster state.
2. Sync — apply the difference, respecting sync waves and hooks.
3. Heal — if live state drifts from Git, revert it automatically.
4. Report — surface Synced / OutOfSync and Healthy / Degraded per app.