Skip to content

Latest commit

 

History

History
1242 lines (898 loc) · 26.5 KB

File metadata and controls

1242 lines (898 loc) · 26.5 KB

Python Management CLI

Modern Python-based CLI for DevStack Core with service profiles, colored output, and enhanced user experience.

Version: 1.3.0 | Last Updated: 2025-01-18

Table of Contents


Overview

devstack is a modern Python-based management CLI for DevStack Core with comprehensive service profile support. It provides a beautiful terminal experience with colored output, tables, and progress indicators powered by the Rich library.

Key Features:

  • 🎯 Profile Management: minimal, standard, full, and reference profiles
  • 🔐 Vault Integration: Complete HashiCorp Vault lifecycle management
  • 🎨 Beautiful UI: Rich library with colored output, tables, progress bars
  • 🏥 Health Checks: Monitor all services with color-coded status
  • 📦 Backup/Restore: Complete data protection for all services
  • 🔄 Auto-Detection: Automatic venv activation and environment loading
  • 📚 Comprehensive Help: Built-in help for every command with examples

Getting Help

The script includes comprehensive help documentation accessible via standard help commands:

# Main help - shows all commands
./devstack --help
./devstack -h

# Help for specific command - shows all options and examples
./devstack start --help
./devstack logs --help
./devstack vault-show-password --help

# Version information
./devstack --version

Every command includes:

  • ARGUMENTS: Required and optional parameters
  • OPTIONS: All available flags with defaults
  • EXAMPLES: Real-world usage demonstrations
  • NOTES: Important warnings, tips, and gotchas

Examples:

# See all available --profile options
./devstack start --help

# Learn about log viewing options (--follow, --tail)
./devstack logs --help

# Understand which services have stored passwords
./devstack vault-show-password --help

Installation

Prerequisites

  • macOS: Apple Silicon (M1/M2/M3/M4) or Linux (x86_64/arm64)
  • Python: 3.8 or higher (Python 3.14.0 confirmed working)
  • Homebrew: Package manager (macOS)

Install Dependencies

Option 1: Using uv (Recommended)

# Install prerequisites
brew install colima docker docker-compose uv

# Clone repository
cd ~/devstack-core

# Setup Python virtual environment (one-time)
uv venv
uv pip install -r scripts/requirements.txt

# The wrapper script automatically activates the venv
./devstack --help

Option 2: Using pip with Manual venv

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip3 install -r scripts/requirements.txt

# Deactivate when done
deactivate

# Note: The wrapper script handles activation automatically

Option 3: Using Homebrew Python

# Install Homebrew Python
brew install python@3

# Install dependencies system-wide (not recommended)
/opt/homebrew/bin/pip3 install click rich PyYAML python-dotenv

Verify Installation

# Test dependencies are available
python3 -c "import click, rich, yaml; print('✓ All dependencies installed!')"

# Test the script
./devstack --version
./devstack --help

Python Dependencies

The script requires these packages (installed automatically):

  • click >= 8.1.0 - CLI framework
  • rich >= 13.0.0 - Beautiful terminal output
  • PyYAML >= 6.0 - Configuration file parsing
  • python-dotenv >= 1.0.0 - Environment variable loading

Quick Start

First-Time Setup

# 1. Install dependencies (see Installation section)
cd ~/devstack-core
uv venv && uv pip install -r scripts/requirements.txt

# 2. Configure environment
cp .env.example .env
# Edit .env if needed (optional - auto-generated by Vault)

# 3. Start with standard profile (recommended)
./devstack start --profile standard

# 4. Initialize Vault (one-time setup)
./devstack vault-init

# 5. Bootstrap Vault PKI and credentials
./devstack vault-bootstrap

# 6. Initialize Redis cluster (for standard/full profiles)
./devstack redis-cluster-init

# 7. Initialize Forgejo Git server (optional)
./devstack forgejo-init

# 8. Verify all services are healthy
./devstack health

Daily Operations

# Morning: Start development environment
./devstack start --profile standard

# Check status and health
./devstack status
./devstack health

# View service logs
./devstack logs postgres

# Get service credentials
./devstack vault-show-password postgres

# Evening: Stop environment
./devstack stop

Common Workflows

Start Minimal Stack (Git + Postgres + Redis):

./devstack start --profile minimal

Start Full Stack with Reference Apps:

./devstack start --profile standard --profile reference

Backup Before Maintenance:

./devstack backup

Restore from Backup:

./devstack restore
./devstack restore 20250118_143022

Complete Command Reference

Core Management

start - Start Environment

Starts Colima VM and Docker services with specified profile(s).

Usage:

./devstack start [--profile PROFILE] [--detach/--no-detach]

Options:

  • --profile, -p - Service profile to start (minimal/standard/full/reference)
  • --detach, -d - Run in background (default: True)
  • Can specify multiple --profile flags

Examples:

# Start with standard profile (recommended)
./devstack start --profile standard

# Start minimal profile (lightweight, 2GB RAM)
./devstack start --profile minimal

# Combine profiles (standard + reference apps)
./devstack start --profile standard --profile reference

# Start in foreground (see logs in real-time)
./devstack start --profile minimal --no-detach

What it does:

  1. Checks if Colima is already running
  2. Starts Colima VM with configured resources
  3. Loads profile-specific environment variables
  4. Starts Docker Compose services for specified profiles
  5. Shows service status after startup

stop - Stop Environment

Stops Docker services and optionally Colima VM.

Usage:

./devstack stop [--profile PROFILE]

Options:

  • --profile, -p - Stop specific profile services only (keeps VM running)

Examples:

# Stop everything (services + Colima VM)
./devstack stop

# Stop only reference profile services
./devstack stop --profile reference

# Stop only standard services (keeps minimal running)
./devstack stop --profile standard

What it does:

  1. Stops Docker Compose services
  2. Stops Colima VM (if no --profile specified)
  3. Releases system resources

restart - Restart Services

Restarts Docker services without restarting Colima VM.

Usage:

./devstack restart

What it does:

  1. Restarts all running Docker containers
  2. VM stays running (faster than stop + start)
  3. Useful for applying configuration changes

status - Show Status

Displays status of Colima VM and all running services.

Usage:

./devstack status

Output includes:

  • Colima VM status and IP address
  • Running services count
  • CPU and memory allocation
  • Disk usage
  • Table of all containers with status

health - Health Check

Performs health checks on all running services with color-coded output.

Usage:

./devstack health

Output:

  • Green: Service is healthy and running
  • ⚠️ Yellow: Running but no health check or starting
  • Red: Stopped, unhealthy, or exited

Checks:

  • Container running status
  • Docker health check results
  • Dependency availability
  • Response times (where applicable)

logs - View Logs

View logs for all services or a specific service.

Usage:

./devstack logs [SERVICE] [--follow] [--tail N]

Arguments:

  • SERVICE - Service name (optional, shows all if omitted)

Options:

  • --follow, -f - Follow log output in real-time
  • --tail, -n - Number of lines to show (default: 100)

Examples:

# View all service logs (last 100 lines)
./devstack logs

# View PostgreSQL logs
./devstack logs postgres

# Follow Vault logs in real-time
./devstack logs vault -f

# Show last 500 lines
./devstack logs --tail 500 postgres

# Follow multiple services (use docker compose directly)
docker compose logs -f postgres vault redis-1

shell - Container Shell

Open an interactive shell in a running container.

Usage:

./devstack shell <SERVICE> [--shell SHELL]

Arguments:

  • SERVICE - Service name (required)

Options:

  • --shell, -s - Shell to use (default: sh, can use bash if available)

Examples:

# Shell into PostgreSQL container
./devstack shell postgres

# Use bash instead of sh
./devstack shell vault --shell bash

# Shell into Redis node
./devstack shell redis-1

# Common commands after entering shell:
# PostgreSQL: psql -U $POSTGRES_USER -d $POSTGRES_DB
# Redis: redis-cli -a $REDIS_PASSWORD
# MongoDB: mongosh -u $MONGODB_USER -p $MONGODB_PASSWORD
# Vault: vault status

ip - Show IP Address

Display Colima VM IP address.

Usage:

./devstack ip

Output:

  • VM IP address (e.g., 192.168.106.2)
  • Network interface information

Use cases:

  • Accessing services from libvirt VMs
  • Configuring network clients
  • Debugging network connectivity

Vault Operations

vault-init - Initialize Vault

Initialize Vault and generate unseal keys (first-time setup).

Usage:

./devstack vault-init

What it does:

  1. Initializes Vault with Shamir's Secret Sharing
  2. Generates 5 unseal keys (3 required to unseal)
  3. Generates root token
  4. Saves keys to ~/.config/vault/keys.json
  5. Saves root token to ~/.config/vault/root-token
  6. Automatically unseals Vault

⚠️ Important: Back up ~/.config/vault/ immediately!

One-time command: Run only after first start.


vault-unseal - Unseal Vault

Manually unseal Vault using stored unseal keys.

Usage:

./devstack vault-unseal

When to use:

  • Vault is sealed after unexpected shutdown
  • Auto-unseal mechanism failed
  • Manual intervention required
  • After Colima VM restart

What it does:

  1. Reads unseal keys from ~/.config/vault/keys.json
  2. Unseals Vault using first 3 of 5 keys
  3. Displays seal status

vault-status - Vault Status

Display Vault seal status and root token information.

Usage:

./devstack vault-status

Output:

  • Sealed status (true/false)
  • Initialized status (true/false)
  • Vault version
  • Root token location
  • Instructions to set VAULT_TOKEN environment variable

vault-token - Display Root Token

Print Vault root token to stdout (for scripting).

Usage:

./devstack vault-token

# Use in scripts
export VAULT_TOKEN=$(./devstack vault-token)
vault status

Output:

  • Raw token (no formatting) suitable for automation

vault-bootstrap - Bootstrap Vault

Bootstrap Vault with PKI infrastructure and service credentials.

Usage:

./devstack vault-bootstrap

Prerequisites:

  • Vault must be initialized and unsealed
  • Run vault-init first

What it does:

  1. Enables PKI secrets engine at pki/
  2. Generates Root CA (10-year validity)
  3. Generates Intermediate CA (5-year validity)
  4. Configures certificate roles for all services
  5. Enables KV v2 secrets engine at secret/
  6. Generates random passwords for all services:
    • PostgreSQL (admin user)
    • MySQL (root user)
    • Redis (AUTH password for all 3 nodes)
    • RabbitMQ (admin user)
    • MongoDB (root user)
    • Forgejo (admin user)
  7. Exports CA certificate chain to ~/.config/vault/ca/
  8. Creates Forgejo database in PostgreSQL
  9. Configures AppRole authentication for all services

One-time setup command: Run after vault-init.


vault-ca-cert - Export CA Certificate

Export Vault CA certificate chain to stdout.

Usage:

./devstack vault-ca-cert > ca.pem

# Install on macOS keychain
./devstack vault-ca-cert | \
  sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /dev/stdin

# Install on Linux
./devstack vault-ca-cert | sudo tee /usr/local/share/ca-certificates/devstack-ca.crt
sudo update-ca-certificates

Output:

  • PEM-encoded CA certificate chain (stdout)
  • Certificate location information (stderr)

vault-show-password - Show Service Credentials

Retrieve and display service credentials from Vault.

Usage:

./devstack vault-show-password <SERVICE>

Arguments:

  • SERVICE - Service name (required)

Available services:

  • postgres - PostgreSQL admin password
  • mysql - MySQL root password
  • redis-1, redis-2, redis-3 - Redis AUTH passwords
  • rabbitmq - RabbitMQ admin password
  • mongodb - MongoDB root password
  • forgejo - Admin username, email, and password

Examples:

# Get PostgreSQL password
./devstack vault-show-password postgres

# Get Forgejo admin credentials
./devstack vault-show-password forgejo

# Get Redis password for scripting
REDIS_PASSWORD=$(./devstack vault-show-password redis-1 | grep password | awk '{print $2}')

⚠️ Security Warning: Displays passwords in plaintext. Use carefully.


Service-Specific

forgejo-init - Initialize Forgejo

Initialize Forgejo Git server via automated bootstrap.

Usage:

./devstack forgejo-init

Prerequisites:

  1. Colima and services running
  2. Vault initialized and bootstrapped
  3. PostgreSQL database created (vault-bootstrap does this)

What it does:

  1. Checks Forgejo container is running
  2. Checks PostgreSQL is accessible
  3. Runs automated installation script
  4. Configures Forgejo with Vault-generated credentials
  5. Creates admin user account
  6. Sets up PostgreSQL database connection

After completion:

  • Access Forgejo at http://localhost:3000
  • Get admin credentials: ./devstack vault-show-password forgejo
  • Clone repos via SSH: ssh://git@localhost:2222/username/repo.git

redis-cluster-init - Initialize Redis Cluster

Initialize 3-node Redis cluster (required for standard/full profiles).

Usage:

./devstack redis-cluster-init

Prerequisites:

  • Started with --profile standard or --profile full
  • All 3 Redis nodes running (redis-1, redis-2, redis-3)

What it does:

  1. Checks Redis containers are running
  2. Fetches Redis password from Vault
  3. Creates 3-node cluster with automatic slot distribution:
    • redis-1: slots 0-5460
    • redis-2: slots 5461-10922
    • redis-3: slots 10923-16383
  4. Verifies cluster formation
  5. Displays cluster nodes

⚠️ Not needed for minimal profile (uses standalone Redis)

Verify cluster:

# Connect to cluster
redis-cli -c -h localhost -p 6379

# Check cluster nodes
redis-cli -h localhost -p 6379 cluster nodes

# Check cluster info
redis-cli -h localhost -p 6379 cluster info

Data Management

backup - Backup Data

Backup all service data to timestamped directory.

Usage:

./devstack backup

What it backs up:

  • PostgreSQL: Complete pg_dumpall of all databases
  • MySQL: Complete mysqldump of all databases
  • MongoDB: Binary mongodump archive
  • Forgejo: Tarball of /data directory (repos + config)
  • .env: Configuration file

Backup location:

  • ./backups/YYYYMMDD_HHMMSS/
  • Example: ./backups/20250118_143022/

Output:

  • Progress indicators for each service
  • Final backup size
  • Backup directory path

⚠️ Important: Always backup before:

  • Running reset command
  • Major upgrades
  • Configuration changes
  • Database migrations

restore - Restore from Backup

Restore service data from a timestamped backup directory.

Usage:

# List available backups
./devstack restore

# Restore specific backup
./devstack restore 20250118_143022

Arguments:

  • BACKUP_NAME - Backup directory name (optional)

⚠️ DATA LOSS WARNING: This command OVERWRITES:

  • All PostgreSQL databases
  • All MySQL databases
  • All MongoDB databases
  • Forgejo data directory
  • .env configuration file

What it does:

  1. Lists available backups (if no backup specified)
  2. Validates backup directory exists
  3. Prompts for confirmation
  4. Restores PostgreSQL databases
  5. Restores MySQL databases (with Vault password)
  6. Restores MongoDB databases
  7. Restores Forgejo data
  8. Restores .env configuration
  9. Recommends restarting services

After restore:

# Restart services to apply changes
./devstack restart

# Verify restoration
./devstack health

reset - Reset Environment

Completely destroy and delete Colima VM - DESTRUCTIVE OPERATION.

Usage:

./devstack reset

⚠️ DATA LOSS WARNING ⚠️

This command DESTROYS:

  • All Docker containers and images
  • All Docker volumes (databases, Git repos, files)
  • Colima VM disk and configuration
  • ALL service data

This command PRESERVES:

  • Vault keys/tokens in ~/.config/vault/ (on host)
  • Backups in ./backups/ (on host)
  • .env configuration (on host)

Confirmation required: Yes (interactive prompt)

⚠️ ALWAYS RUN backup FIRST!

Use cases:

  • Complete environment corruption
  • Starting fresh after major changes
  • Cleaning up before deleting project

Profile Management

profiles - List Profiles

List all available service profiles with details.

Usage:

./devstack profiles

Output includes:

  • Profile name
  • Number of services
  • RAM estimate
  • Description and use case
  • Custom profiles (from profiles.yaml)

Service Profiles

Profile Comparison

Profile Services RAM Startup Time Use Case
minimal 5 2GB ~20s Git + essential dev
standard 10 4GB ~45s Full dev stack + Redis cluster
full 18 6GB ~60s Everything + observability
reference +5 +1GB +15s Educational API examples

Minimal Profile (5 services)

Services: vault, postgres, pgbouncer, forgejo, redis-1 (standalone)

Use Cases:

  • Personal Git repository hosting
  • Simple CRUD application development
  • Learning DevStack Core
  • CI/CD pipelines (lightweight testing)

Start:

./devstack start --profile minimal

Redis: Single standalone instance (no cluster)


Standard Profile (10 services)

Services: All minimal + mysql, mongodb, redis-2, redis-3, rabbitmq

Use Cases:

  • Multi-database application development
  • Redis cluster testing and development
  • Message queue integration
  • Full-featured local development

Start:

./devstack start --profile standard

# Initialize Redis cluster (first time)
./devstack redis-cluster-init

Redis: 3-node cluster with 16,384 slots distributed


Full Profile (18 services)

Services: All standard + prometheus, grafana, loki, vector, cadvisor, redis-exporters

Use Cases:

  • Performance testing and optimization
  • Production troubleshooting simulation
  • Observability pattern learning
  • Load testing with metrics collection

Start:

./devstack start --profile full

# Initialize Redis cluster (first time)
./devstack redis-cluster-init

Observability Stack:


Reference Profile (5 services)

Services: reference-api, api-first, golang-api, nodejs-api, rust-api (~40%)

Use Cases:

  • Learning API design patterns
  • Comparing language implementations
  • Testing integration patterns
  • Code-first vs API-first comparison

Start:

# Must combine with standard or full
./devstack start --profile standard --profile reference

Access:


Environment Variables

Colima Configuration

# Colima profile name (default: default)
export COLIMA_PROFILE=default

# CPU cores (default: 4)
export COLIMA_CPU=4

# Memory in GB (default: 8)
export COLIMA_MEMORY=8

# Disk size in GB (default: 60)
export COLIMA_DISK=60

Environment Loading Priority

The script loads environment variables in this order (highest to lowest priority):

  1. Shell environment (highest priority)
  2. configs/profiles/<profile>.env (profile-specific overrides)
  3. Root .env file (project defaults)
  4. docker-compose.yml defaults (lowest priority)

Example:

# Profile-specific variables take precedence
./devstack start --profile standard
# Loads configs/profiles/standard.env automatically

Migration from Bash Script

Command Comparison

Bash Script Python CLI Status Notes
start start ✅ Enhanced Added profile support
stop stop ✅ Enhanced Added profile filtering
restart restart ✅ Identical Same functionality
status status ✅ Enhanced Rich tables, better formatting
logs logs ✅ Identical Same functionality
shell shell ✅ Identical Same functionality
ip ip ✅ Identical Same functionality
health health ✅ Enhanced Color-coded output
backup backup ✅ Enhanced Progress bars
N/A restore ✅ NEW Restore from backups
vault-* vault-* ✅ Identical All Vault commands
forgejo-init forgejo-init ✅ Identical Same functionality
redis-cluster-init redis-cluster-init ✅ Identical Same functionality
help --help ✅ Enhanced Click framework help
N/A profiles ✅ NEW List all profiles
reset reset ✅ Enhanced Added confirmation

Python CLI Enhancements

  1. Better UI:

    • Rich library for beautiful tables
    • Colored output with semantic meaning
    • Progress bars for long operations
    • Spinners for waiting operations
  2. Profile Support:

    • Native --profile flag
    • Automatic environment loading
    • Multiple profiles can be combined
  3. Improved Safety:

    • Confirmation prompts for destructive ops
    • Better error messages
    • Validation before operations
    • Helpful suggestions
  4. Automation Friendly:

    • Proper exit codes for CI/CD
    • Raw output modes for scripting
    • Better error handling

Migration Steps

For existing bash script users:

  1. Install Python dependencies:

    cd ~/devstack-core
    uv venv
    uv pip install -r scripts/requirements.txt
  2. Test basic commands:

    ./devstack --help
    ./devstack status
    ./devstack health
  3. Update aliases/scripts:

    • Old: ./devstack.sh start
    • New: ./devstack start
  4. All data preserved: The Python CLI uses the same Docker Compose files and configurations.


Troubleshooting

Dependencies Not Found

Problem: ModuleNotFoundError: No module named 'click'

Solution:

# Install dependencies
uv venv
uv pip install -r scripts/requirements.txt

# Or use pip directly
pip3 install --user click rich PyYAML python-dotenv

Script Not Executable

Problem: Permission denied: ./devstack

Solution:

chmod +x devstack

Colima Not Found

Problem: Command not found: colima

Solution:

# Install Colima
brew install colima

Profile Not Loading

Problem: Environment variables from profile not taking effect

Diagnosis:

# Check if profile .env file exists
ls -la configs/profiles/standard.env

# Manually load to debug
set -a
source configs/profiles/standard.env
set +a
env | grep REDIS

Solution:

  • Verify profile .env file exists
  • Check for syntax errors in .env file
  • Ensure no conflicting environment variables

Redis Cluster Init Fails

Problem: Error initializing Redis cluster

Diagnosis:

# Check redis containers are running
docker ps | grep redis

# Check if already initialized
redis-cli -h localhost -p 6379 cluster nodes

Solution:

# Reset if needed (WARNING: destroys data)
docker compose down
docker volume rm devstack-core_redis_1_data devstack-core_redis_2_data devstack-core_redis_3_data
./devstack start --profile standard
./devstack redis-cluster-init

Vault Sealed After Restart

Problem: Services can't connect to Vault after Colima restart

Solution:

# Check Vault status
./devstack vault-status

# Unseal if needed
./devstack vault-unseal

# Restart services that depend on Vault
./devstack restart

Advanced Usage

Custom Profiles

Create a custom profile environment file:

# Create custom profile
cat > configs/profiles/my-custom.env << 'EOF'
# Custom Redis configuration
REDIS_MAX_MEMORY=1024mb
REDIS_MAXMEMORY_POLICY=allkeys-lfu

# Custom PostgreSQL configuration
POSTGRES_MAX_CONNECTIONS=200
POSTGRES_SHARED_BUFFERS=512MB

# Enable observability
ENABLE_METRICS=true
ENABLE_LOGS=true
EOF

# Load and start
set -a
source configs/profiles/my-custom.env
set +a
docker compose --profile standard up -d

Scripting and Automation

Use the Python CLI in CI/CD pipelines:

#!/bin/bash
# CI/CD integration example

set -e  # Exit on error

# Start minimal profile for testing
./devstack start --profile minimal

# Wait for services to be healthy
sleep 30
if ! ./devstack health; then
  echo "Services not healthy"
  exit 1
fi

# Run tests
pytest tests/

# Cleanup
./devstack stop

Monitoring

Continuous health monitoring:

# Watch status in real-time
watch -n 5 './devstack status'

# Continuous health monitoring
while true; do
  ./devstack health
  sleep 60
done

# Alert on unhealthy services
if ! ./devstack health | grep -q "All services healthy"; then
  echo "ALERT: Services unhealthy" | mail -s "DevStack Alert" admin@example.com
fi

See Also


Last Updated: 2025-01-18 Version: 1.3.0