Secret Rotation

Secret rotation replaces an active secret value with a new one. GCP Secret Manager supports multiple versions per secret, which means the old value remains accessible (as a non-latest version) until explicitly disabled or destroyed.

Rotation Flow

1. Generate new value (provider dashboard or CLI)
2. Add new version to Secret Manager
3. Trigger deploy (or manually re-run sync-secrets-vm.sh)
4. Verify the app works with the new value
5. Disable the old version in Secret Manager

The key insight: adding a new version does not take effect until the VM re-reads secrets. The VM reads secrets only during sync-secrets-vm.sh, which runs at deploy time. To apply a rotation without deploying new code, SSH into the VM and re-run the script manually.

Quick Rotation (No Code Deploy)

For rotating a secret without waiting for a code change:

# 1. Set the new value in Secret Manager
echo -n "new-value" | gcloud secrets versions add trovella-{name} \
  --project=trovella-prod --data-file=-

# 2. SSH into the VM and re-run the sync script
gcloud compute ssh trovella-prod-vm \
  --zone=us-central1-a --project=trovella-prod \
  --tunnel-through-iap \
  --command="cd /opt/trovella && sudo ./sync-secrets-vm.sh && sudo docker compose -f docker-compose.prod.yml restart web"

Only restart the web container if the rotated secret is used by the Next.js app. If it is used by inngest or typesense, restart those containers instead (or use up -d to restart all).

Rotation Procedures by Category

API Keys (Anthropic, Google AI, Resend, Typesense)

Impact during rotation: Requests using the old key will fail after the provider revokes it. Rotate during low-traffic periods.

Step	Action
1	Generate a new API key in the provider's dashboard
2	Add the new key as the latest version in Secret Manager
3	Deploy or manually sync (see Quick Rotation above)
4	Verify the feature works (make a test API call)
5	Revoke the old key in the provider's dashboard
6	Disable the old version in Secret Manager

Secrets affected: trovella-anthropic-api-key, trovella-google-ai-api-key, trovella-resend-api-key, trovella-typesense-api-key

OAuth Credentials (Google)

Impact during rotation: Active user sessions continue to work (they use BETTER_AUTH_SECRET, not the OAuth credentials). New sign-ins will fail until the new credentials are deployed.

Step	Action
1	Create new OAuth credentials in Google Cloud Console
2	Update both trovella-google-oauth-client-id and trovella-google-oauth-client-secret
3	Deploy or manually sync
4	Test a new Google sign-in flow end-to-end
5	Delete the old credentials in Google Cloud Console

Both secrets must be updated together -- a mismatched client ID and secret will reject all sign-in attempts.

Better Auth Secret

Impact during rotation: All existing sessions are invalidated. Every signed-in user will be signed out and must sign in again.

Step	Action
1	Generate a new random string (at least 32 characters): openssl rand -base64 48
2	Add the new value as the latest version of trovella-better-auth-secret
3	Deploy or manually sync
4	Verify sign-in works with the new secret

There is no way to rotate BETTER_AUTH_SECRET without invalidating sessions. Plan for user impact. Consider rotating during off-peak hours.

Database URL

Impact during rotation: The application loses database connectivity during the transition. This is a high-impact rotation.

The DATABASE_URL typically changes only when:

• The Cloud SQL instance is replaced (rare)
• The database password is rotated
• The database user changes

Step	Action
1	Update the password in Cloud SQL (via gcloud sql users set-password or Console)
2	Update trovella-database-url with the new connection string
3	Deploy or manually sync
4	The sync script rewrites the host to cloud-sql-proxy:5432 automatically
5	Verify database connectivity via the health endpoint

If only the password changed, the connection string format stays the same -- only the password segment changes.

Redis Credentials (Upstash)

Impact during rotation: Cache reads will fail until the new credentials are deployed. The app degrades gracefully (cache misses fall through to the database).

Step	Action
1	Rotate credentials in the Upstash dashboard
2	Update both trovella-upstash-redis-url and trovella-upstash-redis-token
3	Deploy or manually sync
4	Verify cache connectivity via the health endpoint

Inngest Keys

Impact during rotation: Background job event delivery fails until both the app and Inngest server use the new keys. Since both read from the same .env file on the VM, a single sync + restart updates both.

Step	Action
1	Generate new keys (or rotate in Inngest dashboard if using Inngest Cloud)
2	Update both trovella-inngest-event-key and trovella-inngest-signing-key
3	Deploy or manually sync, then restart both web and inngest containers
4	Verify a background job executes successfully

Sentry DSN

Impact during rotation: Error reporting stops until the new DSN is deployed. No user-facing impact.

Step	Action
1	Create a new DSN in Sentry project settings
2	Update trovella-sentry-dsn
3	For server-side: deploy or manually sync
4	For client-side: a full Docker image rebuild is required (the DSN is baked in as NEXT_PUBLIC_SENTRY_DSN)

Disabling Old Versions

After confirming the new version works, disable the old one to prevent accidental rollback:

# List versions
gcloud secrets versions list trovella-{name} --project=trovella-prod

# Disable a specific version (keeps it recoverable)
gcloud secrets versions disable trovella-{name}/versions/{N} --project=trovella-prod

# Destroy a version permanently (irreversible)
gcloud secrets versions destroy trovella-{name}/versions/{N} --project=trovella-prod

Use disable first. Only destroy old versions after confirming the application has been stable with the new version for at least 24 hours.

Pre-Rotation Checklist

Before rotating any secret:

• Identify which services and packages use the secret (see Environment Variables)
• Determine the user impact (session invalidation, API downtime, cache miss spike)
• Choose a low-traffic window if the rotation causes downtime
• Have the new value ready before starting
• Know whether a code deploy or manual sync is sufficient
• For NEXT_PUBLIC_* variables, a Docker image rebuild is required -- a manual sync alone is not enough

Future: Automated Rotation

GCP Secret Manager supports automatic rotation via Pub/Sub notifications and Cloud Functions. This is not implemented for Trovella's current scale (solo founder, 14 secrets, infrequent rotation). The trigger for adding automated rotation would be:

• Multiple team members who might forget rotation procedures
• Compliance requirements mandating rotation frequency
• Secrets that expire on a fixed schedule (e.g., certificates)