Bug: range destroy can orphan AWS resources and become unretryable

[Brad-Edwards]

Summary

Range destroy can leave AWS range resources running while CTF/CMS state says the range is gone or in progress, and the normal portal/CTF API path cannot retry the teardown without manual DB repair.

This was observed during cleanup of the AWS dev Polaris/CTF ranges on 2026-06-11 in aws-dev / us-east-2.

Impact

• Participant ranges can continue running and incurring cost after users/events believe they are destroyed.
• CTF organizer UI can no longer destroy the range because CTFParticipant.range_instance_id is cleared.
• Mission Control destroy-by-request can return Range not found because CMS has soft-deleted the RangeInstance.
• Engine may show failed while EC2 instances and Terraform state still exist.
• Operators are pushed toward manual DB repair or direct ECS/terraform intervention, which risks inconsistent state.

Observed Failure Modes

1. CTF destroy clears CTFParticipant.range_instance_id immediately after calling CMS.
2. CMS destroy sets RangeInstance.status=destroying and deleted_at immediately, before teardown success is confirmed.
3. If the provisioner fails, AWS resources remain running but the retry paths are broken:
   • CTF endpoint returns No range assigned to participant.
   • Mission Control/CMS destroy by request can return Range not found because it uses the default manager and misses soft-deleted RangeInstance rows.
4. Engine destroy retry has an idempotency trap: when a range is already destroying, it returns success without necessarily starting a new worker. If the previous worker died, this can strand the range.
5. CMS range event handling logged RangeInstance not found for destroyed callbacks even though soft-deleted rows existed, so CMS remained destroying after engine marked ranges destroyed.
6. Provisioner task failures also contributed:
   • Deployed image could not write Terraform files under /app as UID 1000.
   • Terraform repeatedly failed provider discovery against registry.terraform.io, requiring repeated retries and larger registry timeout/retry settings.

Expected Behavior

Destroy should be durable and retryable through the portal API:

• Do not hide or unlink the range from CTF/CMS until engine teardown reaches a terminal success, or preserve enough durable linkage for retry.
• Destroy retry APIs should find soft-deleted/in-progress rows when resources may still exist.
• A failed teardown should leave the system in a state where the same API can retry safely.
• Destroyed engine events should update CMS state reliably for soft-deleted rows.
• Provisioner should have a writable Terraform working directory and reliable provider availability.

Suggested Fixes

• Change CTF destroy behavior so participant.range_instance_id is not cleared until engine/CMS confirms destroyed, or store a durable teardown reference that the UI/API can use for retries.
• Change CMS destroy-by-request lookup to use RangeInstance.all_objects when handling teardown/retry paths.
• Adjust engine destroy idempotency: if status is destroying but no active worker exists or the latest worker failed, allow/restart teardown instead of returning success as a no-op.
• Fix CMS range event handling so destroyed status events can reliably update soft-deleted RangeInstance rows.
• Fix the provisioner runtime so Terraform runs from a writable workspace, not unwritable /app as UID 1000.
• Avoid live registry dependency during teardown where possible: cache/pre-bundle providers or configure reliable registry access/timeouts/retries.
• Add tests covering failed destroy followed by retry through the same CTF and Mission Control API paths.

Acceptance Criteria

• A failed destroy can be retried through the same portal/CTF API without manual DB edits.
• CTF UI still exposes enough state/control to retry or observe teardown after a worker failure.
• CMS and engine converge to destroyed after successful teardown.
• No EC2 range instances remain when CMS/engine report destroyed.
• Provisioner teardown does not fail due to local filesystem permissions.
• Terraform provider discovery is not a recurring teardown blocker.