Drift Detection

Drift Detection

Configuration drift occurs when VPS runtime configuration differs from git-tracked configuration. strut can detect, report, and automatically fix drift.

Quick Reference

strut my-stack drift detect --env prod              # Detect drift
strut my-stack drift report --env prod              # Detailed report
strut my-stack drift report --env prod --json       # JSON report
strut my-stack drift fix --env prod                 # Fix drift
strut my-stack drift fix --dry-run --env prod       # Preview fix
strut my-stack drift history --env prod             # View history
strut my-stack drift auto-fix enable --env prod     # Enable auto-fix
strut my-stack drift auto-fix disable --env prod    # Disable auto-fix
strut my-stack drift images --env prod              # Stale image-digest check (since v0.31.0)

Commit Staleness (since v0.29.0)

drift detect now also checks whether the VPS checkout is behind origin. If the remote branch has commits that haven't been pulled, strut reports this as drift (commit staleness). This catches cases where a deploy was skipped or a strut sync hasn't run yet.

Image-Digest Drift (since v0.31.0)

Config drift catches files that diverged from git. drift images catches a different kind of drift entirely: a mutable tag that moved on the registry after you deployed it. If your compose file pins ghcr.io/org/app:latest (or any other floating tag), the container running right now can silently be a different image than what :latest currently resolves to — with no file-level drift to detect, since the compose file itself never changed.

strut my-stack drift images --env prod
strut my-stack drift images --env prod --json

For each running container, strut compares the digest it's actually running against what the tag currently resolves to on the registry (docker manifest inspect), skipping any image already pinned to a digest (image@sha256:...):

✗ ghcr.io/myorg/app:latest — stale (running: sha256:abc1..., registry: sha256:def5...)
✓ ghcr.io/myorg/db:5.0 — current

A "stale" result means: someone pushed a new image under that tag since your last deploy, and your running container is still on the old digest. That's not necessarily wrong — deliberate, staged rollouts often work exactly this way — but it's worth knowing about, especially for security-relevant base images that get patched upstream.

Common Causes of Drift

• Manual changes on VPS (emergency hotfixes, experiments)
• Failed deployments that partially applied
• Configuration experiments that weren't reverted
• Emergency hotfixes not committed back to git

Detection

Manual

strut my-stack drift detect --env prod

Automatic (Cron)

# Hourly (default)
0 * * * * /path/to/strut my-stack drift detect --env prod

# Every 30 minutes
*/30 * * * * /path/to/strut my-stack drift detect --env prod

Fixing Drift

Manual Fix

strut my-stack drift fix --env prod

The fix process:

1. Backs up current VPS configuration
2. Applies git-tracked configuration
3. Runs health checks
4. If health checks fail, restores backup
5. Logs the event and sends notification

Auto-Fix

When enabled, drift is automatically corrected on detection:

strut my-stack drift auto-fix enable --env prod

Auto-fix creates a backup before every fix, runs health checks after, and rolls back if health checks fail. Disable before intentional experiments:

strut my-stack drift auto-fix disable --env prod

.drift-ignore

Some files legitimately differ at runtime. Add them to .drift-ignore in the stack directory:

# Runtime-generated
*.log
*.pid
*.sock

# Local overrides
.env
.env.local
docker-compose.override.yml

# SSL (managed by certbot)
nginx/conf.d/ssl.conf

# Temporary
tmp/*
cache/*

Supports glob patterns: *, ?, **, [abc].

Drift History

Events stored in stacks/<stack>/drift-history/ as JSON with:

• Timestamp, stack, status
• Files that drifted (with git/VPS hashes and diffs)
• Resolution method (auto-fix, manual-fix, ignored)
• Health check result

Common Workflows

Emergency Hotfix (Intentional Drift)

1. Make emergency change on VPS
2. Drift will be detected on next check
3. Commit the change to git so drift resolves
4. Drift resolves on next detection cycle

Configuration Experiment

1. Disable auto-fix: strut my-stack drift auto-fix disable --env prod
2. Make experimental changes on VPS
3. Test changes
4. If successful, commit to git
5. Re-enable auto-fix

Alerts

Drift alerts integrate with the monitoring stack:

• Drift detected → warning
• Auto-fix applied → info
• Auto-fix failed → critical
• Health checks failed after fix → critical

Troubleshooting

False Positives

# Check line endings (CRLF vs LF)
file stacks/<stack>/docker-compose.yml
dos2unix stacks/<stack>/docker-compose.yml

Drift Not Detected

crontab -l | grep drift          # Check cron
strut my-stack drift detect --env prod   # Run manually
strut my-stack shell --env prod          # Check SSH

Best Practices

1. Use .drift-ignore for runtime-generated files
2. Enable auto-fix on production stacks
3. Review drift history weekly
4. Always commit emergency hotfixes back to git
5. Disable auto-fix before intentional experiments
6. Make changes in git, not on VPS — let deployments propagate