aly badawy/homelab
all systems operational
// guides · provisioning

How to rebuild

Complete walkthrough for reprovisioning from a blank Ubuntu 26.04 server. Steps 1–8 run unattended after three prompts. The Longhorn restore is the only manual gate.

~30 min hands-on Ubuntu 26.04 k3s v1.36.1+k3s1

01 Prerequisites

Before running anything, have these ready:

Item Details
Server Ubuntu 26.04, reachable by SSH, with NOPASSWD sudo for user homelab.
SSH key Your key in ~/.ssh/authorized_keys on the server. ssh homelab@<ip> must work without a password prompt.
Vault unseal key From your offline backup. Seeded as a Kubernetes Secret in Step 6 — Vault cannot unseal without it.
Cloudflare API token Zone:DNS:Edit permission for alybadawy.com. cert-manager uses it for DNS-01 wildcard TLS.
Local tools kubectl, kustomize, helm, git installed and on PATH on the machine running the scripts.
NAS reachable 172.20.20.2 reachable from the server for NFS mounts and backup restore.

02 Run rebuild.sh

One script orchestrates Steps 1–8. It collects all three inputs upfront, then runs fully unattended until the Longhorn restore step.

terminal bash 
# clone the repo if you haven't already
$ git clone https://github.com/AlyBadawy/hl-beta && cd hl-beta

# run the rebuild orchestrator — prompts for IP, Vault key, Cloudflare token
$ ./provision/rebuild.sh

The script prompts for three values then runs unattended:

  1. Server IP address — validated as a valid IPv4 address.
  2. Vault unseal key — entered silently (no echo). From your offline backup.
  3. Cloudflare API token — entered silently. Must have Zone:DNS:Edit on alybadawy.com.

03 What each step does

Step Script What happens
1 — SSH check check-ssh-connection Validates SSH key auth works and NOPASSWD sudo is configured. Blocks if either fails.
2 — Dependencies update-dependencies Runs apt upgrade, installs packages (nfs-common, open-iscsi, apparmor, socat, etc.), disables SWAP. Optional — can skip with N.
3 — NAS mounts mount-nas Creates /mnt/nas/{homelab,backups,immich,nextcloud}, adds NFS entries to /etc/fstab, mounts and verifies. Idempotent.
4 — k3s install install-k3s Installs k3s v1.36.1+k3s1 with Traefik disabled. Waits for node Ready. Copies kubeconfig locally. Skips if already installed.
5 — Cluster config configure-cluster Creates cluster-config namespace and ConfigMap with domain, NAS paths, admin email, server IP. Applies kernel tuning (inotify, vm.max_map_count).
6 — Vault key inline in rebuild.sh Creates the security namespace and seeds vault-unseal-key Secret. Required before GitOps runs — without it Vault stays sealed and ESO cannot sync.
7 — ArgoCD bootstrap-argocd Installs ArgoCD via Kustomize + Helm, waits for argocd-server ready, creates networking namespace and cloudflare-api-token Secret. Stops before deploying any apps.
8 — Longhorn restore-volumes Installs Longhorn, configures NAS backup target. On a rebuild: opens the UI on localhost:9000 and waits for manual volume restore confirmation.

04 Longhorn volume restore (Step 8)

This is the only manual step. The script opens the Longhorn UI via port-forward and waits for you to confirm before continuing.

Fresh cluster? If there are no backups to restore, answer y when the script asks "Is this a fresh cluster?" — it configures the backup target and exits immediately. Skip to Step 5 below.

For a rebuild (restoring from NAS backups):

  1. Open http://localhost:9000 (the script starts the port-forward automatically).
  2. In the left sidebar, click Backup. Click the sync icon if no backups appear.
  3. For each volume, select the latest backup → Restore. Use the PVC name from the mapping table below.
  4. Go to Volume and wait for each volume to show state Detached (restore complete).
  5. For each restored volume, click Create PV/PVC and set the correct namespace and PVC name.
  6. Return to the terminal and press Enter to confirm.
volume → namespace/PVC mapping text 
Longhorn volume name   →   namespace / PVC name
────────────────────────────────────────────────
vault-data          →   security / vault-data
postgres-data       →   db       / postgres-data
nextcloud-config      →   cloud    / nextcloud-config
authentik-media     →   security / authentik-media
authentik-templates →   security / authentik-templates

05 Activate GitOps

After rebuild.sh completes, verify everything is in place, then hand cluster ownership to ArgoCD:

verification + activation bash 
# verify secrets are seeded
$ kubectl get secret vault-unseal-key -n security
$ kubectl get secret cloudflare-api-token -n networking

# verify ArgoCD is healthy
$ kubectl get pods -n argocd

# verify Longhorn PVCs are bound (if rebuild mode)
$ kubectl get pvc -A | grep -E 'lh$'

# activate GitOps — applies root.yaml and hands ownership to ArgoCD
$ ./provision/activate-gitops.sh

After activation, ArgoCD discovers all applications in k8s/apps/ and syncs them. Vault starts first (sync-wave -1), unseals within ~6 minutes, then ESO syncs all secrets and the rest of the apps come up. See Bootstrapping GitOps for the full wave sequence.

Total rebuild time: ~30–45 minutes. About 5 minutes is hands-on (prompts + Longhorn restore). The rest is waiting for packages to install, k3s to start, ArgoCD to sync, and Vault to unseal.
last updated 2026-06-08 · view source on GitHub