feat: Implement multi-environment support for dev/staging/prod deployments

[doughayden]

Feature Request

Add support for multiple deployment environments (development, staging, production) with proper configuration isolation and promotion workflows.

Level of Effort: 🔥 Large (6-8 days)

• Terraform refactoring: 3-4 days for environment separation
• Configuration management: 1-2 days for environment-specific configs
• CI/CD pipeline updates: 1-2 days for multi-environment workflows
• Testing and validation: 1 day for environment verification

Current State

Single environment deployment:

• All infrastructure deployed to one environment
• Configuration mixed between development and production settings
• No environment isolation or promotion workflow
• Single Terraform state for all resources

Limitations:

• Cannot test changes safely before production
• No staging environment for integration testing
• Configuration changes affect production directly
• Limited ability to experiment with new features

Recommended Implementation

1. Environment Structure

terraform/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── terraform.tfvars
│   │   └── backend.tf
│   ├── staging/
│   │   ├── main.tf
│   │   ├── terraform.tfvars
│   │   └── backend.tf
│   └── prod/
│       ├── main.tf
│       ├── terraform.tfvars
│       └── backend.tf
├── modules/
│   ├── answer-app/
│   ├── loadbalancer/
│   └── monitoring/

2. Environment-Specific Configuration

Development Environment:

# terraform/environments/dev/terraform.tfvars
project_id = "answer-app-dev"
domain = "dev.answer-app.example.com"
min_instances = 0
max_instances = 2
cpu_limit = "1"
memory_limit = "1Gi"
enable_iap = false
log_level = "DEBUG"

Staging Environment:

# terraform/environments/staging/terraform.tfvars
project_id = "answer-app-staging"
domain = "staging.answer-app.example.com"
min_instances = 1
max_instances = 5
cpu_limit = "2"
memory_limit = "2Gi"
enable_iap = true
log_level = "INFO"

Production Environment:

# terraform/environments/prod/terraform.tfvars
project_id = "answer-app-prod"
domain = "answer-app.example.com"
min_instances = 2
max_instances = 10
cpu_limit = "4"
memory_limit = "4Gi"
enable_iap = true
log_level = "WARNING"

3. Application Configuration

Environment-specific config files:

# src/answer_app/config/dev.yaml
environment: development
log_level: DEBUG
bigquery:
  dataset_id: "answer_app_dev"
discovery_engine:
  location: "us-central1"
  preamble: "[DEV] You are a helpful assistant..."

# src/answer_app/config/prod.yaml
environment: production
log_level: WARNING
bigquery:
  dataset_id: "answer_app_prod"
discovery_engine:
  location: "us-central1"
  preamble: "You are a helpful assistant..."

4. CI/CD Pipeline Updates

Environment-specific workflows:

# .github/workflows/deploy-dev.yml
name: Deploy to Development
on:
  push:
    branches: [develop]

# .github/workflows/deploy-staging.yml
name: Deploy to Staging
on:
  push:
    branches: [main]

# .github/workflows/deploy-prod.yml
name: Deploy to Production
on:
  release:
    types: [published]

Implementation Areas

Terraform Changes:

• Separate Terraform states: One per environment
• Environment modules: Reusable infrastructure components
• Variable management: Environment-specific configurations
• State backend separation: Isolated state storage

Application Changes:

• Configuration loading: Environment-aware config selection
• Environment detection: Runtime environment identification
• Feature flags: Environment-specific feature enablement
• Logging configuration: Environment-appropriate log levels

Infrastructure Updates:

• Project separation: Separate GCP projects per environment
• Domain management: Environment-specific domains
• Secret management: Environment-isolated secrets
• Network isolation: Separate VPCs/subnets if needed

Environment Promotion Workflow

1. Development → Staging

# Promote code to staging
git checkout main
git merge develop
git push origin main

# Deploy to staging automatically via CI/CD

2. Staging → Production

# Create release after staging validation
gh release create v1.2.0 --title "Release v1.2.0"

# Production deployment triggered automatically

3. Configuration Promotion

• Environment-specific configurations maintained separately
• Promote configuration changes through same workflow
• Validate configuration compatibility between environments

Resource Management

Cost Optimization:

• Development: Minimal resources, scale-to-zero when possible
• Staging: Production-like but smaller scale
• Production: Full resources with appropriate scaling

Data Management:

• Development: Synthetic/anonymized data
• Staging: Subset of production data (if appropriate)
• Production: Full production dataset

Testing Strategy

Environment Validation:

• Smoke tests: Basic functionality verification per environment
• Integration tests: Run against staging before production
• Performance tests: Staging environment performance validation
• Security tests: Environment-specific security scanning

Acceptance Criteria

• Separate Terraform configurations for each environment
• Environment-specific application configurations
• Isolated GCP projects/resources per environment
• Automated deployment workflows for each environment
• Environment promotion process documented
• Cost optimization implemented for non-production environments
• Testing strategy for each environment defined
• Rollback procedures documented

Priority

Low - Valuable for scaling team and reducing production risk, but not critical for current single-environment operation.

When to Implement

This becomes more valuable when:

• Team size grows beyond 2-3 developers
• Production changes require more validation
• Feature experimentation needs safe environment
• Multiple stakeholders need access to different environments
• Compliance or security requirements mandate environment separation