VM Secret Sync

How the sync-secrets-vm.sh script pulls secrets from GCP Secret Manager to the production VM at deploy time.

The infra/sync-secrets-vm.sh script is the mechanism that delivers secrets from GCP Secret Manager to the production VM. It runs on every deploy, immediately before docker compose up.

When It Runs

The deploy-prod CI job executes this sequence on the VM via SSH:

sudo mv ~/docker-compose.prod.yml ~/Caddyfile ~/sync-secrets-vm.sh /opt/trovella/
cd /opt/trovella
sudo chmod +x sync-secrets-vm.sh
sudo ./sync-secrets-vm.sh
sudo docker compose -f docker-compose.prod.yml pull
sudo docker compose -f docker-compose.prod.yml up -d --remove-orphans

The script always runs, even if no secrets changed. This ensures the .env file stays in sync with Secret Manager on every deploy.

How It Works

1. Define the Secret-to-Env Mapping

The script declares an array mapping Secret Manager IDs to environment variable names:

SECRETS=(
  "trovella-anthropic-api-key:ANTHROPIC_API_KEY"
  "trovella-google-ai-api-key:GOOGLE_AI_API_KEY"
  "trovella-database-url:DATABASE_URL"
  "trovella-better-auth-secret:BETTER_AUTH_SECRET"
  "trovella-better-auth-url:BETTER_AUTH_URL"
  "trovella-google-oauth-client-id:GOOGLE_CLIENT_ID"
  "trovella-google-oauth-client-secret:GOOGLE_CLIENT_SECRET"
  "trovella-resend-api-key:RESEND_API_KEY"
  "trovella-upstash-redis-url:REDIS_URL"
  "trovella-upstash-redis-token:UPSTASH_REDIS_TOKEN"
  "trovella-sentry-dsn:SENTRY_DSN"
  "trovella-inngest-event-key:INNGEST_EVENT_KEY"
  "trovella-inngest-signing-key:INNGEST_SIGNING_KEY"
  "trovella-typesense-api-key:TYPESENSE_API_KEY"
)

The format is secret-manager-id:ENV_VAR_NAME. The left side is the full Secret Manager secret ID. The right side is the environment variable name that application code reads.

2. Write Static Configuration

Before pulling secrets, the script writes non-secret configuration values:

echo "NODE_ENV=production" >> "$TEMP_FILE"
echo "HOSTNAME=0.0.0.0" >> "$TEMP_FILE"
echo "PORT=3000" >> "$TEMP_FILE"
echo "NEXT_PUBLIC_BETTER_AUTH_URL=https://trovella.ai" >> "$TEMP_FILE"
echo "TYPESENSE_URL=http://typesense:8108" >> "$TEMP_FILE"
echo "INNGEST_BASE_URL=http://inngest:8288" >> "$TEMP_FILE"
echo "CLOUD_SQL_CONNECTION_NAME=trovella-prod:us-central1:trovella-prod" >> "$TEMP_FILE"

These use Docker service names (typesense, inngest) as hostnames because containers communicate on the Docker Compose network.

3. Pull Each Secret

The script iterates over the mapping array and calls gcloud secrets versions access for each:

for entry in "${SECRETS[@]}"; do
  SECRET_ID="${entry%%:*}"
  ENV_VAR="${entry##*:}"

  VALUE=$(gcloud secrets versions access latest \
    --secret="$SECRET_ID" \
    --project="$PROJECT" 2>/dev/null) || VALUE=""

  if [ -n "$VALUE" ]; then
    echo "${ENV_VAR}=\"${VALUE}\"" >> "$TEMP_FILE"
    echo "  ok $ENV_VAR"
  else
    echo "  SKIP $ENV_VAR (no version found)"
  fi
done

Key behaviors:

Always reads latest version. There is no version pinning. The most recent secret version is always used.
Graceful skip on missing secrets. If a secret has no versions (newly created shell, or disabled), the variable is omitted with a SKIP log line. The deploy continues.
Values are double-quoted. This handles values containing spaces or special characters in the .env file.

4. Rewrite DATABASE_URL

After all secrets are pulled, the script rewrites the DATABASE_URL host to route through the Cloud SQL Auth Proxy container:

sed -i -E 's|(DATABASE_URL="postgresql://[^@]+@)[^:/]+:[0-9]+/|\1cloud-sql-proxy:5432/|' "$TEMP_FILE"

The secret in Secret Manager stores the direct Cloud SQL connection URL (public IP). On the VM, the cloud-sql-proxy container handles TLS and IAM authentication to Cloud SQL. The application connects to cloud-sql-proxy:5432 on the Docker network, which forwards to the real database.

This rewrite is identical in purpose to the sslmode=disable rewrite in the CI migration job (see Data & Storage -- CI Deployment), but uses the Docker service name instead of 127.0.0.1.

5. Atomic File Replacement

The script writes to a temporary file and atomically moves it into place:

TEMP_FILE=$(mktemp)
# ... write all vars to $TEMP_FILE ...
mv "$TEMP_FILE" "$ENV_FILE"
chmod 600 "$ENV_FILE"

This prevents Docker Compose from reading a partially-written .env file if a container restarts during the sync. The chmod 600 restricts the file to root only.

Output File

The resulting /opt/trovella/.env file looks like this (structure only, no real values):

# Auto-generated from GCP Secret Manager (trovella-prod)
# Generated at: 2026-04-08T12:00:00Z

NODE_ENV=production
HOSTNAME=0.0.0.0
PORT=3000
NEXT_PUBLIC_BETTER_AUTH_URL=https://trovella.ai
TYPESENSE_URL=http://typesense:8108
INNGEST_BASE_URL=http://inngest:8288
CLOUD_SQL_CONNECTION_NAME=trovella-prod:us-central1:trovella-prod

ANTHROPIC_API_KEY="sk-ant-..."
GOOGLE_AI_API_KEY="..."
DATABASE_URL="postgresql://user:pass@cloud-sql-proxy:5432/trovella"
BETTER_AUTH_SECRET="..."
# ... remaining secrets ...

How Containers Read It

The docker-compose.prod.yml file uses env_file to inject variables:

services:
  web:
    env_file: /opt/trovella/.env
    # ...

  cloud-sql-proxy:
    env_file: /opt/trovella/.env
    # ...

The typesense and inngest containers read specific variables directly from the .env file via ${VAR} interpolation in the compose file:

typesense:
  environment:
    TYPESENSE_API_KEY: ${TYPESENSE_API_KEY}

inngest:
  command: >
    inngest start
    --event-key ${INNGEST_EVENT_KEY}
    --signing-key ${INNGEST_SIGNING_KEY}

Authentication

The script runs on the VM with sudo, using the VM service account's ambient credentials from the GCE metadata server. No key files, no gcloud auth, no tokens. The service account (trovella-vm-prod@trovella-prod.iam.gserviceaccount.com) has roles/secretmanager.secretAccessor on the trovella-prod project.

See Secret Manager -- IAM Access Control for the Terraform configuration.

Troubleshooting

Secret shows SKIP in deploy logs

The secret exists in Secret Manager but has no versions. Set a value:

echo -n "value" | gcloud secrets versions add trovella-{name} \
  --project=trovella-prod --data-file=-

App starts but a feature is broken

Check if the env var is present:

gcloud compute ssh trovella-prod-vm \
  --zone=us-central1-a --project=trovella-prod \
  --tunnel-through-iap \
  --command="sudo grep NEW_VAR /opt/trovella/.env"

If missing, verify the variable is in both the SECRETS array in sync-secrets-vm.sh and has a value in Secret Manager.

Permission denied on gcloud secrets

The VM service account does not have secretmanager.secretAccessor. Check the IAM binding:

gcloud projects get-iam-policy trovella-prod \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/secretmanager.secretAccessor" \
  --format="table(bindings.members)"