Application Deployment Guide

This guide provides step-by-step instructions for deploying the Serko Northsky application to the Kubernetes cluster. It covers secret configuration in Google Secret Manager, Kubernetes deployment, and Langfuse setup.

Prerequisites

Before deploying the application, ensure:

1. Infrastructure is deployed - Run pulumi up to create GKE, databases, and storage
2. Tools are installed - See Prerequisites for required tools
3. Cluster access - Verify with kubectl get nodes

# Verify cluster connection
source infra/env.sh dev
kubectl get nodes

Deployment Overview

Step 1: Configure Secrets in Secret Manager

Application Secrets

The application requires a JSON secret named {namespace}-app-config in Google Secret Manager.

Required Secret Keys

Key	Description	Example
database_url	PostgreSQL connection string	postgresql://user:pass@host:5432/db
openai_api_key	OpenAI API key for AI features	sk-...
firebase_service_account_content	Firebase SA JSON (base64)	{...}
environment	Environment name	dev, test, prod
debug	Debug mode	true or false
log_level	Logging level	INFO, DEBUG, WARNING
backend_cors_origins	Allowed CORS origins	https://app.dev.serko-northsky.com
api-url	Backend API URL	https://app.dev.serko-northsky.com/api
next-public-firebase-config	Firebase web config JSON	{...}

Create the Secret

# Set environment
ENV=dev  # or test, prod
PROJECT_ID=serko-northsky-dev  # adjust per environment
NAMESPACE=serko-northsky-$ENV

# Create the secret JSON file
cat > /tmp/app-config.json << 'EOF'
{
  "database_url": "postgresql://postgres:YOUR_PASSWORD@10.x.x.x:5432/postgres",
  "openai_api_key": "sk-your-openai-key",
  "firebase_service_account_content": "{...base64 encoded or raw JSON...}",
  "environment": "dev",
  "debug": "true",
  "log_level": "INFO",
  "backend_cors_origins": "https://app.dev.serko-northsky.com,http://localhost:3000",
  "api-url": "https://app.dev.serko-northsky.com/api",
  "next-public-firebase-config": "{\"apiKey\":\"...\",\"authDomain\":\"...\"}"
}
EOF

# Create the secret in Secret Manager
gcloud secrets create "${NAMESPACE}-app-config" \
    --project=$PROJECT_ID \
    --replication-policy="automatic"

# Add the secret version
gcloud secrets versions add "${NAMESPACE}-app-config" \
    --project=$PROJECT_ID \
    --data-file=/tmp/app-config.json

# Clean up local file
rm /tmp/app-config.json

Get Database URL from Pulumi

# Get the database URL from Pulumi outputs
cd infra/pulumi
source ../env.sh $ENV
pulumi stack output databaseUrl --show-secrets

Langfuse Secrets (Optional)

If deploying Langfuse for LLM observability, create the {namespace}-langfuse-credentials secret.

Required Langfuse Keys

Key	Description	Example
database-url	Langfuse PostgreSQL URL	postgresql://langfuse:pass@host:5432/langfuse
redis-url	Redis connection URL	redis://:auth@host:6379
redis-host	Redis hostname	10.x.x.x
redis-port	Redis port	6379
redis-auth	Redis auth string	your-redis-auth
s3-access-key-id	GCS HMAC access key	GOOG...
s3-secret-access-key	GCS HMAC secret	...
nextauth-secret	NextAuth session secret	Random 32-byte hex
salt	Encryption salt	Random 32-byte hex
encryption-key	Data encryption key	Random 32-byte hex

Generate Random Secrets

# Generate required random secrets
echo "nextauth-secret: $(openssl rand -hex 32)"
echo "salt: $(openssl rand -hex 32)"
echo "encryption-key: $(openssl rand -hex 32)"

Create GCS HMAC Keys

Langfuse uses S3-compatible storage. For GCS, create HMAC keys:

# Create HMAC key for the Langfuse service account
gcloud storage hmac create langfuse@${PROJECT_ID}.iam.gserviceaccount.com \
    --project=$PROJECT_ID

# Note the accessId and secret from the output

Create Langfuse Secret

# Create the Langfuse credentials JSON
cat > /tmp/langfuse-creds.json << 'EOF'
{
  "database-url": "postgresql://langfuse:PASSWORD@10.x.x.x:5432/langfuse?sslmode=require",
  "redis-url": "redis://:REDIS_AUTH@10.x.x.x:6379",
  "redis-host": "10.x.x.x",
  "redis-port": "6379",
  "redis-auth": "REDIS_AUTH_STRING",
  "s3-access-key-id": "GOOG_ACCESS_KEY_ID",
  "s3-secret-access-key": "GCS_SECRET_ACCESS_KEY",
  "nextauth-secret": "GENERATED_HEX_32",
  "salt": "GENERATED_HEX_32",
  "encryption-key": "GENERATED_HEX_32"
}
EOF

# Create the secret
gcloud secrets create "${NAMESPACE}-langfuse-credentials" \
    --project=$PROJECT_ID \
    --replication-policy="automatic"

# Add the version
gcloud secrets versions add "${NAMESPACE}-langfuse-credentials" \
    --project=$PROJECT_ID \
    --data-file=/tmp/langfuse-creds.json

# Clean up
rm /tmp/langfuse-creds.json

Verify Secrets

# List secrets in the project
gcloud secrets list --project=$PROJECT_ID

# Verify secret content (be careful with sensitive data)
gcloud secrets versions access latest \
    --secret="${NAMESPACE}-app-config" \
    --project=$PROJECT_ID | jq .

Step 2: Grant Secret Access

Ensure the GKE service account can access the secrets:

# Get the app service account email
APP_SA="serko-northsky-${ENV}-app@${PROJECT_ID}.iam.gserviceaccount.com"

# Grant secret accessor role for app-config
gcloud secrets add-iam-policy-binding "${NAMESPACE}-app-config" \
    --project=$PROJECT_ID \
    --member="serviceAccount:${APP_SA}" \
    --role="roles/secretmanager.secretAccessor"

# Grant for Langfuse credentials (if using Langfuse)
gcloud secrets add-iam-policy-binding "${NAMESPACE}-langfuse-credentials" \
    --project=$PROJECT_ID \
    --member="serviceAccount:${APP_SA}" \
    --role="roles/secretmanager.secretAccessor"

Step 3: Deploy to Kubernetes

Connect to the Cluster

# Source environment and connect
cd infra
source env.sh dev  # or test, prod

# Verify connection
kubectl get namespaces

Deploy with Helm

The deployment script handles External Secrets Operator installation and Helm deployment:

cd infra/k8s/helm

# Deploy to development
./deploy-helm.sh dev

# Deploy to production
./deploy-helm.sh prod

Deployment Options

# Preview deployment (dry-run)
./deploy-helm.sh dev --dry-run

# Deploy specific components only
./deploy-helm.sh dev -c backend,frontend

# Deploy with debug output
./deploy-helm.sh dev --debug

# Check deployment status
./deploy-helm.sh dev --status

# Rollback to previous release
./deploy-helm.sh dev --rollback

Manual Deployment

For more control, deploy manually:

cd infra/k8s/helm/serko-northsky

# Preview rendered templates
helm template serko-northsky . \
  -f values/components/global.yaml \
  -f values/components/backend.yaml \
  -f values/components/frontend.yaml \
  -f values/components/tools.yaml \
  -f values/components/docs.yaml \
  -f values/components/ingress.yaml \
  -f values/components/security.yaml \
  -f values/environments/dev/overrides.yaml

# Deploy
helm upgrade --install serko-northsky . \
  --namespace serko-northsky-dev \
  --create-namespace \
  --wait \
  --timeout 5m \
  -f values/components/global.yaml \
  -f values/components/backend.yaml \
  -f values/components/frontend.yaml \
  -f values/components/tools.yaml \
  -f values/components/docs.yaml \
  -f values/components/ingress.yaml \
  -f values/components/security.yaml \
  -f values/environments/dev/overrides.yaml

Step 4: Verify Deployment

Check Pod Status

# List all pods
kubectl get pods -n serko-northsky-dev

# Expected output:
# NAME                        READY   STATUS    RESTARTS   AGE
# backend-xxx                 1/1     Running   0          2m
# frontend-xxx                1/1     Running   0          2m
# tools-xxx                   1/1     Running   0          2m
# docs-xxx                    1/1     Running   0          2m

Check Services

# List services
kubectl get services -n serko-northsky-dev

# Check ingress
kubectl get ingress -n serko-northsky-dev

Check Secrets Sync

# Verify External Secrets are synced
kubectl get externalsecrets -n serko-northsky-dev

# Check the synced Kubernetes secrets
kubectl get secrets -n serko-northsky-dev

View Logs

# Backend logs
kubectl logs -f deployment/backend -n serko-northsky-dev

# Frontend logs
kubectl logs -f deployment/frontend -n serko-northsky-dev

# All pods with label
kubectl logs -l app=backend -n serko-northsky-dev --tail=100

Test Endpoints

# Health check (from within cluster or via port-forward)
kubectl port-forward svc/backend-service 8000:8000 -n serko-northsky-dev &
curl http://localhost:8000/api/health

# Or via ingress (once DNS is configured)
curl https://app.dev.serko-northsky.com/api/health

Step 5: Enable Langfuse (Optional)

To enable Langfuse for LLM observability:

1. Update Environment Values

Edit infra/k8s/helm/serko-northsky/values/environments/dev/langfuse-overrides.yaml:

langfuse:
  enabled: true
  config:
    nextPublicUrl: "https://langfuse.dev.serko-northsky.com"

clickhouse:
  enabled: true

2. Deploy with Langfuse

cd infra/k8s/helm

# Deploy including Langfuse component
./deploy-helm.sh dev -c langfuse

3. Verify Langfuse

# Check Langfuse pods
kubectl get pods -n serko-northsky-dev -l app=langfuse

# Check ClickHouse
kubectl get pods -n serko-northsky-dev -l app=clickhouse

# View Langfuse logs
kubectl logs -f deployment/langfuse-web -n serko-northsky-dev

4. Access Langfuse

Once deployed and DNS configured:

https://langfuse.dev.serko-northsky.com

Troubleshooting

Secret Sync Issues

# Check ExternalSecret status
kubectl describe externalsecret app-config-external-secret -n serko-northsky-dev

# Check SecretStore
kubectl describe secretstore app-secretstore -n serko-northsky-dev

# Verify service account annotation
kubectl get sa app-service-account -n serko-northsky-dev -o yaml

Pod CrashLoopBackOff

# Get pod events
kubectl describe pod <pod-name> -n serko-northsky-dev

# Check recent logs
kubectl logs <pod-name> -n serko-northsky-dev --previous

Image Pull Errors

# Verify Workload Identity
kubectl get sa app-service-account -n serko-northsky-dev -o yaml | grep gcp-service-account

# Check if images exist
gcloud artifacts docker images list \
    us-central1-docker.pkg.dev/${PROJECT_ID}/serko-northsky

Database Connection Issues

# Test database connectivity from a pod
kubectl exec -it deployment/backend -n serko-northsky-dev -- \
    python -c "import psycopg2; print(psycopg2.connect('$DATABASE_URL'))"

# Check if AlloyDB IP is accessible
kubectl exec -it deployment/backend -n serko-northsky-dev -- \
    nc -zv 10.x.x.x 5432

CI/CD Deployment

For automated deployments via GitHub Actions, see the workflows:

• Development: Triggered on push to develop branch
• Production: Manual trigger from main branch

GitHub Actions Secrets Required

Secret	Description
GCP_SA_KEY_DEV	Dev service account JSON key
GCP_SA_KEY_PROD	Prod service account JSON key
GCP_PROJECT_ID_DEV	Dev project ID
GCP_PROJECT_ID_PROD	Prod project ID

Quick Reference

Full Deployment Checklist

1. Verify infrastructure is deployed (pulumi stack output)
2. Create {namespace}-app-config secret in Secret Manager
3. Grant service account access to secrets
4. Connect to GKE cluster (source env.sh dev)
5. Deploy with Helm (./deploy-helm.sh dev)
6. Verify pods are running (kubectl get pods)
7. Verify secrets are synced (kubectl get externalsecrets)
8. Test health endpoint
9. (Optional) Enable and deploy Langfuse

Common Commands

# Environment setup
source infra/env.sh dev

# Deploy
./deploy-helm.sh dev

# Status check
./deploy-helm.sh dev --status

# View pods
kubectl get pods -n serko-northsky-dev

# View logs
kubectl logs -f deployment/backend -n serko-northsky-dev

# Restart pods
./rollout-pods.sh dev all

# Rollback
./deploy-helm.sh dev --rollback