[RFC] Improve Ansible Installer Production Readiness

[dnviti]

Summary

The current Ansible installer () provides a solid foundation for deployments but lacks some features expected in enterprise-grade installation systems.

Current Strengths

• Idempotency: SHA256 hash-based diff detection
• Stateful: Profile, state, and status encrypted with AES
• Drift detection: Monitors compose/environment changes
• Capability-based: Modular service selection based on capabilities
• Healthcheck: Verifies container health, API endpoints, and migrations
• Multi-backend: Supports both Podman Compose and Kubernetes (Helm)

Missing Features

Feature	Priority	Description
Rolling update	HIGH	Sequential container restart instead of simultaneous recreation
Pre/post hooks	MEDIUM	Ability to inject custom logic before/after deployment phases
Circuit breaker	MEDIUM	Stop deployment if critical services fail
Canary/blue-green	LOW	Deploy to subset of nodes for validation
Multi-target deployment	LOW	Parallel deployment to multiple hosts

Technical Notes

• Rolling update blocked in install_apply.yml — restarts all containers at once
• No hook system exists in the codebase
• Healthcheck continues even if individual services fail

Proposal

Enhance the installer to include:

1. Rolling update support: Implement sequential service restart with health validation between steps
2. Hook system: Add pre_apply, post_apply, pre_migrate, post_migrate hooks
3. Circuit breaker: Abort deployment if critical services (postgres, control-plane-api) fail healthcheck

Files Involved

• deployment/ansible/playbooks/tasks/install_apply.yml — main logic
• deployment/ansible/scripts/install_model.py — diff and resolution
• deployment/ansible/roles/deploy/tasks/apply.yml — container application
• deployment/ansible/roles/healthcheck/tasks/main.yml — health verification

Test Plan

1. Run make install and verify no regression in dev mode
2. Implement rolling update for non-critical services
3. Add pre/post hooks and verify execution order
4. Test circuit breaker with intentional service failure