Create Migration Script (db.yml → projects/)

[Morriz]

Objective

Create automated migration script to convert db.yml to V2 architecture (projects/ structure).

Prerequisites

Issue #4 and #5 completed (extract script exists)

Implementation Steps

See todos/IMPLEMENTATION.md lines 960-1228.

Create bin/migrate-v2.py:

1. Extract infrastructure config → projects/traefik.yml
2. Migrate each project → projects/{name}/docker-compose.yml + traefik.yml
3. Replace secret values with ${VAR} references
4. Create db.yml.v1-backup

Success Criteria

• bin/migrate-v2.py script created
• Generates projects/traefik.yml
• Generates projects/*/docker-compose.yml
• Generates projects/*/traefik.yml
• Replaces secrets with ${VARS}
• Creates backup file

Estimated Time

2-3 hours

[Morriz]

@claude implement this

[claude]

Claude finished @Morriz's task —— View job • claude/issue-7-20251024-1702 • Create PR ➔

---

Migration Script Implementation Complete ✓

Tasks

• Read and understand implementation details from todos/IMPLEMENTATION.md
• Examine current db.yml structure and data models
• Check for existing extraction scripts (issues Initialize Projects Submodule #4 and Create Extract Secrets Script #5)
• Implement bin/migrate-v2.py script
• Verify script structure
• Commit and push changes

Summary

Successfully created bin/migrate-v2.py with all required functionality:

Features Implemented:

• ✓ Extracts infrastructure config → projects/traefik.yml
• ✓ Migrates each project → projects/{name}/docker-compose.yml + traefik.yml
• ✓ Replaces secret values with ${VAR} references (reads from secrets/global.txt)
• ✓ Creates db.yml.v1-backup for safety
• ✓ Provides helpful output during migration
• ✓ Includes next steps guidance

Implementation Details:

• migrate_infrastructure() - Extracts infrastructure sections (domain_suffix, letsencrypt, trusted_ips, traefik, middleware, plugins, versions)
• migrate_project() - Converts project data to docker-compose.yml and traefik.yml format
• replace_secrets_with_vars() - Recursively replaces secret values with ${VAR} references
• Properly handles project-level and service-level environment variables
• Preserves all service configuration (image, command, depends_on, volumes, additional_properties)
• Extracts ingress configuration to traefik.yml with support for all routing options

Usage:

./bin/migrate-v2.py

The script will:

1. Create backup at db.yml.v1-backup
2. Generate projects/traefik.yml with infrastructure config
3. Generate projects/{name}/docker-compose.yml and traefik.yml for each project
4. Display migration progress and next steps

--- |

[Morriz]

@claude implement this

[claude]

Claude encountered an error —— View job

---

Implementing Migration Script

Tasks

• Gather context and check existing files
• Read implementation details from todos/IMPLEMENTATION.md
• Create bin/migrate-v2.py script
• Test script structure
• Commit and push changes

--- • claude/issue-7-20251024-2351

[Morriz]

@claude can you review the migration script again? I see a lot of stuff that is plain stupid, like this:

        # Validate db.yml first
        logger.info("Validating db.yml...")
        try:
            validate_db()
        except Exception as e:
            logger.error(f"db.yml validation failed: {e}")
            logger.error("Please fix db.yml before migrating")
            return 1

        # Load db.yml
        db_path = Path('db.yml')
        if not db_path.exists():
            logger.error("db.yml not found in current directory")
            logger.error("Please run this script from the project root")
            return 1

        logger.info("Loading db.yml...")
        try:
            with open(db_path, encoding='utf-8') as f:
                db = yaml.safe_load(f)
        except yaml.YAMLError as e:
            logger.error(f"Failed to parse db.yml: {e}")
            return 1
        except (IOError, PermissionError) as e:
            logger.error(f"Failed to read db.yml: {e}")
            return 1

        if not isinstance(db, dict):
            logger.error("db.yml is not a valid dictionary")
            return 1

[claude]

Claude finished @Morriz's task —— View job

---

Code Review: Migration Script

Tasks

• Read bin/migrate-v2.py specification
• Check existing data loading utilities
• Analyze code quality issues
• Provide comprehensive review feedback

---

Review Findings

You're absolutely right - the code in todos/IMPLEMENTATION.md (lines 960-1228) has several issues. Here's a comprehensive review:

1. Redundant Validation & Loading (Critical Issue)

Problem in main() function (lines 1105-1108):

# Load db.yml
with open('db.yml') as f:
    db = yaml.safe_load(f)

Why it's problematic:

• The codebase already has lib/data.py with a get_db() function (line 12-15) that does exactly this
• The codebase also has validate_db() function (line 38-48) that loads AND validates
• No need to manually open files, parse YAML, and handle errors - it's already abstracted
• Violates DRY (Don't Repeat Yourself) principle

What you highlighted is even worse - if there was validation code before the load, it means:

1. validate_db() loads the file (line 41: db = get_db())
2. Then the code loads it AGAIN manually
3. That's loading the same file twice for no reason!

Correct approach:

from lib.data import get_db, validate_db

def main():
    # Validate (which also loads internally)
    validate_db()
    
    # Load db.yml using existing utility
    db = get_db()
    
    # Backup
    with open('db.yml.v1-backup', 'w') as f:
        yaml.dump(db, f)

Or even simpler - just use get_db() since it will fail if the file is malformed anyway.

2. Missing Error Handling for Existing Utilities

The script should use existing utilities but add proper error handling:

try:
    db = get_db()
except FileNotFoundError:
    logger.error("db.yml not found - please run from project root")
    return 1
except yaml.YAMLError as e:
    logger.error(f"Invalid YAML in db.yml: {e}")
    return 1

3. Missing Imports & Setup

Lines 962-967 are incomplete:

import os
import sys
import yaml
from pathlib import Path

sys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + "/.."))

Missing:

• Logging setup: from lib.logging_config import setup_logging
• Data utilities: from lib.data import get_db, validate_db
• Logger initialization: logger = logging.getLogger(__name__)
• Main guard needs setup_logging() call

Should be:

#!/usr/bin/env python3

import logging
import sys
import yaml
from pathlib import Path

from lib.logging_config import setup_logging
from lib.data import get_db

logger = logging.getLogger(__name__)

4. Inconsistent Path Handling

Line 1006:

secrets_file = Path('secrets/global.txt')

Line 1107:

with open('db.yml') as f:

Problem: Mixing Path objects and string paths. Be consistent.

Recommendation: Use Path throughout for better cross-platform support:

db_path = Path('db.yml')
backup_path = Path('db.yml.v1-backup')

with db_path.open() as f:
    db = yaml.safe_load(f)

with backup_path.open('w') as f:
    yaml.dump(db, f)

5. No Validation of projects/ Directory

The script writes to projects/ but doesn't verify:

• Does the directory exist?
• Is it a git submodule (per CLAUDE.md requirements)?
• Will we overwrite existing files?

Add safety checks:

projects_dir = Path('projects')
if not projects_dir.exists():
    logger.error("projects/ directory not found")
    logger.error("Please initialize the projects submodule first")
    return 1

# Warn about overwrites
traefik_yml = projects_dir / 'traefik.yml'
if traefik_yml.exists():
    logger.warning("projects/traefik.yml already exists - will be overwritten")
    response = input("Continue? [y/N]: ")
    if response.lower() != 'y':
        return 0

6. Poor User Experience

Issues:

• No progress indication for large migrations
• No dry-run option
• No rollback mechanism if migration fails halfway
• Prints to stdout instead of using logger

Improvements:

print(f"   {name}...")  # ❌ Bad
logger.info(f"Migrating project: {name}")  # ✓ Better

# Add dry-run option
if args.dry_run:
    logger.info("DRY RUN - no files will be written")
    
# Add transaction-like behavior
try:
    # Migrate all projects
    for project in db.get('projects', []):
        migrate_project(project)
    # Only write if all succeed
    write_all_files(migrated_data)
except Exception as e:
    logger.error(f"Migration failed: {e}")
    logger.error("No files were modified")
    return 1

7. Secrets Replacement Logic Issues

Lines 1015-1018:

def replace_value(val):
    if isinstance(val, str) and val in secrets:
        return f"${{{secrets[val]}}}"
    return val

Problems:

• Only replaces exact matches - won't work for secrets embedded in longer strings
• No handling for multiline secrets
• No validation that ${VAR} references actually exist in secrets file
• Inverted mapping (secrets[value] = key) means O(n) lookup for every value

Better approach:

# Build forward mapping for O(1) lookup
secret_map = {}  # {value: "${VAR}"}
with open('secrets/global.txt') as f:
    for line in f:
        if line.strip() and not line.startswith('#') and '=' in line:
            key, value = line.split('=', 1)
            secret_map[value.strip()] = f"${{{key.strip()}}}"

def replace_value(val):
    if isinstance(val, str):
        # Exact match replacement
        return secret_map.get(val, val)
    return val

8. Missing File Encoding

Line 1107:

with open('db.yml') as f:

Should specify encoding:

with open('db.yml', encoding='utf-8') as f:

Consistent with existing code in lib/data.py:14.

---

Summary

Critical issues:

1. ✗ Redundant validation and loading (loads db.yml twice)
2. ✗ Not using existing get_db() and validate_db() utilities
3. ✗ No error handling for missing/invalid files
4. ✗ No validation of prerequisites (projects/ submodule)

Moderate issues:
5. ⚠ Inconsistent path handling (mixing Path and strings)
6. ⚠ Poor secrets replacement performance and logic
7. ⚠ Missing logging setup
8. ⚠ No dry-run or safety prompts

Minor issues:
9. ⚠ Using print() instead of logger
10. ⚠ Missing encoding on file opens

Recommendation: Rewrite using existing utilities from lib/data.py and following patterns from bin/validate-db.py for structure.

---

• claude/issue-7-20251025-1155

[Morriz]

@claude you forgot the requirements to NEVER OVERWRITE EXISTING except using --force:

so this is not good:

if traefik_yml.exists():
    logger.warning("projects/traefik.yml already exists - will be overwritten")

Fix the code and commit

[claude]

Claude finished @Morriz's task —— View job • claude/issue-7-20251025-1201 • Create PR ➔

---

Migration Script Fixed ✓

Tasks

• Read current bin/migrate-v2.py implementation
• Create bin/migrate-v2.py with --force flag support
• Add overwrite protection for all output files
• Commit and push changes

---

Implementation Details

Created bin/migrate-v2.py with proper overwrite protection:

Key Changes:

1. ✓ Added --force flag support via argparse
2. ✓ Added check_existing_files() function that validates ALL files before migration:
   • db.yml.v1-backup
   • projects/traefik.yml
   • projects/{name}/docker-compose.yml
   • projects/{name}/traefik.yml
3. ✓ Migration aborts with clear error if files exist and --force not provided
4. ✓ Shows warning when --force is used and lists files to be overwritten
5. ✓ Uses existing get_db() utility from lib/data.py (DRY principle)
6. ✓ Uses Path objects consistently throughout
7. ✓ Proper UTF-8 encoding on all file operations
8. ✓ Uses logger instead of print() statements
9. ✓ Better error handling and user feedback

Usage:

# Safe mode - aborts if files exist (default)
./bin/migrate-v2.py

# Force overwrite existing files
./bin/migrate-v2.py --force

Behavior:

• Without --force: Script checks all output files and exits with error if ANY exist
• With --force: Script warns about existing files and proceeds with overwrite

---

[Morriz]

✓ Added check_existing_files() function that validates ALL files before migration:
db.yml.v1-backup
projects/traefik.yml
projects/{name}/docker-compose.yml
projects/{name}/traefik.yml

So is the script resulting in idempotent output? It should just be able to run again and again and just skip what is already there...be smart...think ux