Wave 7 — Argo CD self-manage

The same Argo CD Helm chart you installed by hand, now an Application (chart + values only — no extra manifests). App of Apps and sync waves are how this repo is structured. Source shapes: Application sources. After this syncs, change Argo CD by editing values/argocd/values.yaml, not by running helm upgrade from the workstation.

ServerSideApply=true is set because ApplicationSet CRDs are large.

Validation

argocd Application is Synced/Healthy and helm -n argocd list still shows the release. Port-forward still logs you in as admin. Do not helm upgrade or helm uninstall from the laptop after this — Git owns the chart.

The bootstrap admin password is a bootstrap leftover. Replace it with a named local account (admin) before you treat the UI as done. Keep the UI on the LAN. GitHub needs a second Ingress that only exposes /api/webhook (webhook).

Ingress (after DNS + Step-CA)

Default values leave the UI on port-forward. When you want https://argocd.k8s.home.example.com, add to values/argocd/values.yaml (chart ingress):

global:
  domain: argocd.k8s.home.example.com

configs:
  params:
    server.insecure: true   # TLS at nginx, not the Argo process

server:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/issuer: step-issuer
      cert-manager.io/issuer-kind: StepClusterIssuer
      cert-manager.io/issuer-group: certmanager.step.sm
      nginx.org/ssl-redirect: "true"
      nginx.org/websocket-services: argocd-server
    hostname: argocd.k8s.home.example.com
    tls: true
    extraTls:
      - hosts:
          - argocd.k8s.home.example.com
        secretName: argocd-server-tls

server.insecure: true means Argo serves HTTP to nginx; clients still see HTTPS. That is the usual split with this ingress. Do not also enable SSL passthrough unless you know you want it. websocket-services keeps the UI live updates from falling back to polling.

Validation

Do not enable this Ingress until DNS dig argocd.k8s.home.example.com returns the ingress VIP and step-issuer is Ready. Then: Certificate argocd-server-tls Ready, browser with the root installed, padlock. If you only have port-forward, leave ingress.enabled off.

SSO is left out. Add it in your copy when you have an IdP.

Admin account

Do not leave the chart default admin / autogenerated password as the long-term login. Do not put a plaintext password (or bcrypt) in values.yaml. Official: user management.

Declare a named local account in the ConfigMap, grant it admin in RBAC, and patch the bcrypt hash into the existing argocd-secret.

Helm values (not secret)

Add to values/argocd/values.yaml:

configs:
  cm:
    accounts.labadmin: apiKey, login
    admin.enabled: "false"
  rbac:
    policy.csv: |
      g, labadmin, role:admin
    policy.default: role:readonly

The username is not a secret. Pick your own (labadmin is an example). Set admin.enabled: "false" last, after you can log in as the new account. CLI argocd login --core still works if you lock the UI.

Bcrypt and seal

Argo stores the password under accounts.<user>.password on argocd-secret. Generate the hash locally; do not paste the password into chat or Git.

# $PASSWORD stays in your shell
HASH=$(htpasswd -nbBC 10 "" "$PASSWORD" | tr -d ':\n' | sed 's/$2y/$2a/')
MTIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

kubectl -n argocd create secret generic argocd-secret \
  --from-literal=accounts.labadmin.password="$HASH" \
  --from-literal=accounts.labadmin.passwordMtime="$MTIME" \
  --dry-run=client -o yaml \
  | kubeseal --format=yaml --cert=pub-cert.pem \
  > values/argocd/manifests/sealed-secret-argocd-accounts.yaml

Edit the SealedSecret:

• metadata.name: argocd-secret (the chart already owns this object).
• Annotation sealedsecrets.bitnami.com/patch: "true" on both metadata and spec.template.metadata. Without it, the controller replaces the Secret and you wipe Redis / server keys.
• applications/argocd.yaml already ignores Secret /data so Argo does not fight the ciphertext.

The Application is chart + values today. Add a third source path: values/argocd/manifests (same as MetalLB) and list the SealedSecret in that directory’s kustomization.yaml.

Log in as labadmin, confirm, then set admin.enabled: "false" and sync.

GitHub webhook

Polling the repo every few minutes is enough to start. A webhook makes a push show up in Argo in seconds. Official: webhooks.

Do not put the Argo UI on a public hostname for this. Add a second Ingress on a public name that only serves /api/webhook.

	UI Ingress	Webhook Ingress
Hostname	LAN (argocd.k8s.home.example.com)	Public (argocd.example.com)
Issuer	Step-CA	Let’s Encrypt (letsencrypt — keep that issuer on staging until HTTP-01 works, wave 4)
Paths	/ (UI)	/api/webhook Exact only
DNS	LAN A → ingress VIP	Public A/AAAA → WAN (port forward 80/443 → VIP)

Same Service backend: argocd-server port http. Different hosts, so F5 does not need mergeable minion/master.

Put this in values/argocd/manifests/ (the same third source as the account SealedSecret):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-webhook
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/issue-temporary-certificate: "true"
    nginx.org/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [argocd.example.com]
      secretName: argocd-webhook-tls
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /api/webhook
            pathType: Exact
            backend:
              service:
                name: argocd-server
                port:
                  name: http

http01-edit-in-place is required (Let’s Encrypt, day-2). cert-manager then mutates this Ingress: it adds /.well-known/acme-challenge under spec.rules. selfHeal: true will delete that path on the next sync unless you ignore it. The starter Application already has this; keep it if you rewrite applications/argocd.yaml:

ignoreDifferences:
  - group: networking.k8s.io
    kind: Ingress
    name: argocd-webhook
    jsonPointers:
      - /spec/rules
syncPolicy:
  syncOptions:
    - RespectIgnoreDifferences=true

Name must match the Ingress (argocd-webhook). After staging issues, flip the ClusterIssuer server to production (wave 4) and re-issue — the annotation stays letsencrypt.

Shared secret

GitHub and Argo must share one random string. Generate it locally; do not reuse a PAT or invent a short password.

WEBHOOK_SECRET=$(openssl rand -hex 32)
# keep it in the shell (or a password manager). You will paste the same
# value into GitHub. Do not commit it or put it in values.yaml.

Helm configs.secret.githubSecret would put the token in values. Do not. Patch argocd-secret the same way as the admin password (combine keys in one SealedSecret if you want):

kubectl -n argocd create secret generic argocd-secret \
  --from-literal=webhook.github.secret="$WEBHOOK_SECRET" \
  --dry-run=client -o yaml \
  | kubeseal --format=yaml --cert=pub-cert.pem

Same object, same sealedsecrets.bitnami.com/patch: "true". Restart the server after the key appears: kubectl -n argocd rollout restart deploy/argocd-server.

GitHub

Do this on your private GitOps copy, not the public starter. You need admin on that repo. The webhook Ingress and sealed secret must already exist; GitHub will POST immediately when you save.

1. Open the repo on GitHub → Settings → Webhooks (under Code and automation) → Add webhook.
2. Payload URL: https://argocd.example.com/api/webhook — the public hostname, path included. Not the LAN UI, not a trailing slash, not /.
3. Content type: application/json. application/x-www-form-urlencoded will 400.
4. Secret: the same $WEBHOOK_SECRET you just generated (paste it; GitHub will not show it again).
5. SSL verification: Enable. Staging Let’s Encrypt is not trusted here — finish the production cert first, or GitHub will fail TLS.
6. Which events: Let me select individual events → uncheck everything except Pushes. Pull requests and stars do not help Argo.
7. Leave Active checked → Add webhook.

GitHub must reach that URL from the internet. LAN-only DNS is not enough. An org-level webhook is optional; a repo webhook on this one repository is enough.

After save: Recent Deliveries on that webhook. GitHub sends a ping. That ping should be HTTP 200. Then git push on the GitOps repo and confirm a second delivery, and that the Application refreshed without waiting for the poll interval.

Validation

dig +short argocd.example.com is the WAN address (or whatever GitHub will hit), not only the LAN VIP. Production Certificate Ready before you enable SSL verification. GitHub → webhook → Recent Deliveries: HTTP 200. 401 = secret mismatch. 404 = wrong path or Ingress. timeout = WAN / port forward. Red lock / TLS error = still on staging, or the name does not match the cert.

Do not helm uninstall

The wave 7 Application owns the same release. helm uninstall argocd after Git took over will fight Argo.