aly badawy/homelab
all systems operational
// cluster · secrets

External Secrets (ESO)

ESO is the bridge between Vault and the apps that need secrets. Apps don't know about Vault — they consume native Kubernetes Secrets. ESO creates and keeps those secrets in sync from Vault KV v2.

Synced namespace: security Vault provider k8s auth

The External Secrets Operator runs in the security namespace alongside Vault. A single ClusterSecretStore named k8s-secrets connects to Vault using Kubernetes auth. Each app defines one or more ExternalSecret resources that declare which Vault paths to pull and which Kubernetes Secret to create from them.

01 ClusterSecretStore

The ClusterSecretStore is the cluster-wide connection to Vault. ESO authenticates to Vault using the eso-vault-auth ServiceAccount via the Kubernetes auth method — no static tokens or passwords stored anywhere.

k8s/components/external-secrets/cluster-secret-store.yaml yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-vault-auth
  namespace: security
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: k8s-secrets
spec:
  provider:
    vault:
      server: http://vault.security.svc.cluster.local:8200
      path: secret        # KV v2 mount name
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso-reader
          serviceAccountRef:
            name: eso-vault-auth
            namespace: security

02 ExternalSecret pattern

Each app has an ExternalSecret resource that maps Vault KV keys to a Kubernetes Secret. The remoteRef.key is the Vault path under secret/; remoteRef.property is the key name within that path.

example: k8s/components/db/external-secrets.yaml yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: postgres-secret
  namespace: db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: k8s-secrets
    kind: ClusterSecretStore
  target:
    name: postgres-secret  # the Kubernetes Secret to create
    creationPolicy: Owner
  data:
    - secretKey: POSTGRES_USER
      remoteRef:
        key: postgres-secret      # Vault path: secret/postgres-secret
        property: POSTGRES_USER   # key within that KV entry
    - secretKey: POSTGRES_PASSWORD
      remoteRef:
        key: postgres-secret
        property: POSTGRES_PASSWORD

03 Rotating a secret

Update the value in Vault, then force ESO to pick it up immediately:

secret rotation bash
# 1. update the value in Vault (patch preserves other keys)
$ vault kv patch secret/postgres-secret POSTGRES_PASSWORD="new-password"

# 2. force ESO to sync immediately (instead of waiting for refreshInterval)
$ kubectl annotate externalsecret postgres-secret -n db \
    force-sync=$(date +%s) --overwrite

# 3. verify the k8s Secret was updated
$ kubectl get secret postgres-secret -n db \
    -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d; echo

04 Recovery after reboot

When Vault restarts sealed, ESO enters exponential backoff — the ClusterSecretStore shows Degraded. Even after Vault unseals, ESO stays in backoff and doesn't reconnect on its own.

The eso-recovery CronJob (runs every minute in the security namespace) detects when Vault is ready but the store is still degraded and restarts the three ESO deployments to clear the backoff. During normal operation it exits immediately — cheap to run every minute.

Fully automatic. You don't need to restart ESO manually after a reboot. The recovery sequence — unseal Vault, detect store degraded, restart ESO — takes ~6 minutes from boot with no human intervention.
last updated 2026-06-08 · view source on GitHub