Skip to content

Wave 9 — Backups

Three different jobs. Do not mix them.

What Tool Where bytes go
Kubernetes API objects talosctl etcd snapshot Workstation → NAS/S3 file
Longhorn volume contents Longhorn backupstore MinIO bucket
Postgres Barman Cloud plugin + ObjectStore MinIO prefix per database

MinIO install: Unraid extras. Seal keys: secrets. Skip this wave until the endpoint exists. The Applications are harmless idle if you never create an ObjectStore or a backup target.

Validation

Do not point Longhorn or Barman at MinIO until a worker can reach the API:

# from the workstation, then from a debug pod on a worker:
curl -sI http://10.0.0.2:9000/minio/health/live
mc ls nas/longhorn-backups

Timeout or 403 with the app key (not root) means stop. The cluster masquerades as the node IP; Unraid must allow .21–.29.

Longhorn → MinIO

Set a backup target. Helm 1.12 uses defaultBackupStore (not only the old defaultSettings.backupTarget).

  1. Bucket longhorn-backups on MinIO. Dedicated access key.
  2. Seal Secret longhorn-backup-s3 in namespace longhorn:
# keys (stringData before you seal)
AWS_ACCESS_KEY_ID: CHANGEME
AWS_SECRET_ACCESS_KEY: CHANGEME
AWS_ENDPOINTS: http://10.0.0.2:9000
AWS_REGION: us-east-1
  1. In values/longhorn/values.yaml:
defaultBackupStore:
  backupTarget: s3://longhorn-backups@us-east-1/
  backupTargetCredentialSecret: longhorn-backup-s3
  pollInterval: "300"

The @us-east-1 is the S3 region token Longhorn expects. MinIO ignores region as long as AWS_ENDPOINTS points at the API. Trailing slash on the URL matters; copy the docs form.

  1. Sync. Longhorn UI → Backup: target should be Ready.

Validation

Do not trust this for real volumes until: target Ready in the UI, a throwaway PVC snapshot → Backup → Restore to a new volume, file is still there. Unavailable target = endpoint, Secret keys, or trailing slash on backupTarget.

Workers must reach 10.0.0.2:9000. This is not off-site unless MinIO is.

plugin-barman-cloud

Barman Cloud plugin (usage). The Application only installs the plugin in cnpg-system. Each database needs its own ObjectStore + Secret + ScheduledBackup in the app namespace. Example: day-2.

etcd-backup

Talos: do not mount kubeadm hostPath certs. Use talosctl etcd snapshot. The Application ships a suspended CronJob and a ConfigMap with the workstation command:

talosctl --nodes 10.0.0.11 etcd snapshot ./etcd-$(date -u +%Y%m%dT%H%M%SZ).snapshot
# then: copy to the NAS, or:
mc cp ./etcd-*.snapshot nas/etcd-snapshots/

Automate with a systemd timer on the workstation or a small always-on box that has talosctl + TALOSCONFIG. Do not unsuspend the in-cluster CronJob and expect it to speak the Talos API — that pod does not have your talosconfig.

Restore: Talos day-2. Snapshot ≠ Longhorn bytes.

kubeadm / kubespray: you would snapshot with etcdctl and host-mounted etcd certs. That path is cluster-specific and is not in this repo.

Skip: delete applications/etcd-backup.yaml.