Database auth errors after rotating passwords
Symptom
The API pod keeps restarting. kubectl logs -n platform deploy/platform-api shows:
⚠️ Redis error: WRONGPASS invalid username-password pair or user is disabled.
[ioredis] Unhandled error event: ReplyError: WRONGPASS ...
Or for Postgres:
error: password authentication failed for user "postgres"
What the platform does by itself
Rotate and recover keep durable copies before any database password change:
platformctl backup(also run automatically byrecover) copies/etc/platform/.envand the Kubernetes Secrets (platform-env,platform-credentials,platform-oidc-signing-key, database secrets) to/var/lib/platform/backups/<timestamp>/.- One-click rotate (portal) writes the previous/next passwords to the
platform/platform-credentialsSecret (POSTGRES_PASSWORD_PREV,POSTGRES_PASSWORD_NEXT, ...) and toplatform/platform-rotate-pendingbeforeALTER USER. If neither can be written, rotate refuses to change Postgres. The API no longer writes host files;platformctl updatecopies the new values to/etc/platformafterwards. platformctl recoverreads the Secrets first, writes the passwords to the root-only host copy/etc/platform/.envand/etc/platform/credentials/{admin,postgres,redis}(mode 0600) before anyALTER USER, tries stored passwords first and does not ALTER if one still works, then syncsplatform-credentials.- On API boot,
getDb()tries env, pending,platform-credentials(incl. previous/next),platform-envand the Bitnami secret, then syncs the working password toplatform-credentials.
Passwords are not only flashed in the terminal. Read them with:
sudo cat /etc/platform/credentials/admin
sudo cat /etc/platform/credentials/postgres
sudo grep -E '^(ADMIN|POSTGRES|REDIS)_PASSWORD=' /etc/platform/.env
If a pod is still crash-looping
On the k3s host:
curl -fsSL https://github.com/Mpratyush54/SERVER-automation/releases/latest/download/install.sh | bash
sudo platformctl backup
sudo platformctl recover
sudo platformctl update
recover prints the backup directory and /etc/platform/credentials/* paths. Admin login is in credentials/admin and ADMIN_PASSWORD in /etc/platform/.env.
Roll to an image that includes in-cluster credential recovery (platform-api after this change) with platformctl update. A new API pod heals on start as long as any stored password copy still matches the database role.
Manual fallback (only if every stored copy is wrong)
Do not recreate platform-env or platform-credentials with kubectl create secret … | kubectl apply. That wipes other keys.
Patch a single key (credentials live in platform-credentials):
# Encode a known-good password (the value Postgres actually accepts)
ENC=$(printf '%s' 'known-good-password' | base64 -w0)
kubectl -n platform patch secret platform-credentials --type merge -p "{\"data\":{\"POSTGRES_PASSWORD\":\"$ENC\"}}"
kubectl -n databases patch secret postgresql --type merge -p "{\"data\":{\"postgres-password\":\"$ENC\",\"password\":\"$ENC\"}}"
kubectl -n platform rollout restart deployment/platform-api
Keep /etc/platform/.env in sync so a later bootstrap does not re-apply a stale password:
sudo sed -i "s/^POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD='known-good-password'/" /etc/platform/.env