Repository navigation
Use explicit CTF event egress policy for bulk participant ranges #2372
Description
Activity
🛠️ Picked up by /implement - driver codex, branch
2372-ctf-shared-egress, 2026-09-24T00:27:58.567Z.gc workflow phase recorded:
plan(issue #2372). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Scope and acceptance
Fix issue #2372 for generic CTF ranges. Path B (regression): participant/spare launch policy must derive from the immutable event workspace, not the owner's personal workspace, while CMS/Engine ownership and quota remain on the owner's authorized personal workspace. Path A: shared-VPC GCP egress must enroll non-
nonerange subnets into a serialized, explicitly scoped regional NAT instead of one Cloud Router per range; preserve vpc-per-range behavior and current zero-egress firewall+no-NAT invariant. Path A: an opt-in public-web image profile must continue to allow research only under an admitted non-noneposture. Path A: capacity and audit documentation must state the operational bounds and policy-change semantics. No private scenario identifiers or infrastructure enter this repository.Design
- Extend the internal CTF→CMS launch options with a narrow event-policy workspace fact and event owner. CTF participant, managed spare, and recovery callers derive both from the persisted event server-side. Public serializers and generic range calls cannot supply it. CMS rejects a policy override for non-CTF source, validates current event-owner workspace authority via
workspaces.services, and reauthorizes/locks both owner and policy workspaces in stable ID order in the reservation transaction. The Request, RangeInstance, Engine Range, quota and interactive access retain the participant/spare's workspace. Pre-admission and authoritative pinned egress use the event policy source. No fallback to personal policy on denial. Reuse the canonical enum and existing workspace policy audit; record bounded policy source/effective mode in launch audit, not provisioner payload. - Preserve the pinned mode across ownership transfer. Before assigning a ready spare, compare its persisted effective mode with the current event policy under event/assignment coordination; refuse or replace an incompatible spare, never silently adopt a different posture. Participant recovery and manual/scheduled launches use the same launch seam.
- For shared-VPC GCP, plan a deterministic provisioner-owned regional Router/NAT namespace distinct from Terraform's migration bridge. Explicitly list only non-
nonesubnets. Serialize read-modify-write on the shared Router through an existing provisioner PostgreSQL connection and advisory lock keyed by project/network/region; re-read provider state under lock, idempotently add/remove entries, wait for operations and verify actual state. Shard subnet lists at Google's 50-subnet/gateway ceiling, bounded by the 50-gateway/router and five-router/network/region limits; report classified capacity exhaustion before creating range resources. Destroy removes enrollment before subnet deletion and does not delete shared infrastructure used by others. Old per-range routers are recognized on replay/destroy for migration, without double-enrolling or leaking. VPC-per-range retains its independent Router/NAT. - Keep the current per-range firewall/tag isolation, private API lane and the existing
allow_public_web_egressprofile boolean. Do not add an event internet switch, external guest IP, all-subnet NAT, or change Terraform-owned NAT. Changes are confined to the CTF/CMS service boundary, GCE plan/apply/destroy clients, tests, ADR registry/design and operator capacity documentation.
TDD and verification
- Path B: first add DB-backed failing tests for two participants with distinct personal workspaces and a differently configured event workspace, plus a fresh managed spare, proving owner binding and pinned event mode independently; then implement. Add boundary denial, concurrent policy mutation, no fallback, policy-change-after-launch and spare-transfer tests.
- Path A: first add failing GCE plan/provider tests for 20 two-subnet ranges plus two spares, concurrent enrollment, 50-entry sharding, no-enrollment for
none, replay/partial failure/old-router cleanup, destroy ordering and capacity exhaustion. Implement the smallest shared NAT coordinator that makes them pass. Add public-web positive and zero-egress negative rules/reachability checks using synthetic profiles. - Run affected CTF/CMS/workspaces and GCE provisioner suites, ruff check/format, import-linter, full ADR guard, and relevant Terraform lint if Terraform changes (not currently planned). Verify docs explain gateway/subnet and NAT port/IP capacity, quota headroom, shared failure domain, and audited event-policy changes. Use hosted CI and Sonar in the normal delivery workflow.
Whole-repository considerations
Security: CTF organizer/participant authorization, workspace role and locked admission, CMS service caller shape, closed egress enum, GCE profile parser/firewall validator and opaque error envelopes remain enforcing boundaries. No secrets or workspace identities in argv, guest metadata, provider labels or cloud jobs. Maintainability: reuse
workspaces.services, CMS reservation/dispatch, existing GCE plan/resources/ops andprovisioner_db.get_db_connection; avoid a parallel policy schema. Extensibility: a typed policy-workspace selection keeps future CTF workspace policy independent of participant ownership, and deterministic gateway sharding supports more subnets without multiplying routers. Whole repo: inspect CTF participant/spare/recovery, CMS launch/reassign/warm claim, Engine range pinning, RAES and legacy GCE plan paths, provider clients/destroy, Terraform migration-bridge ownership, ADR-017/026/046 and deployment docs. No new structural gate or requirement is introduced.- Extend the internal CTF→CMS launch options with a narrow event-policy workspace fact and event owner. CTF participant, managed spare, and recovery callers derive both from the persisted event server-side. Public serializers and generic range calls cannot supply it. CMS rejects a policy override for non-CTF source, validates current event-owner workspace authority via
gc_codex_review — sanitized deferred publication for issue #2372, cycle 1 of 1
Reviewed revision:307fe2ad34db0408bf8f402afd84969ee937842f155b387a7a10cca41369a3e5Architectural read
Event policy and participant ownership remain separate, but shared NAT migration replay and spare retry must be tightened before delivery.
Verdict:
don't-shipNotes
- Also bound shared NAT gateway names within the provider resource-name limit.
Findings
core-F1 — Align shared NAT containment rules
- Classification:
one-off - Decision:
fix - Location:
docs/adr/index.yaml - Rationale: Update the binding GCP containment rule to permit serialized, subnet-explicit provisioner-owned shared NAT while preserving zero-egress isolation.
core-F2 — Preserve migration-bridge replay
- Classification:
one-off - Decision:
fix - Location:
shifter/engine/provisioner/gcp_range_cell_shared_nat.py - Rationale: Add a safe replay path for existing bridge-managed subnets without double NAT or unsafe handoff.
security-F1 — Recheck reserved spare egress on retry
- Classification:
one-off - Decision:
fix - Location:
shifter/shifter_platform/ctf/services/range/recovery_steps.py - Rationale: Revalidate the current event egress policy before old-range teardown on every recovery retry.
Review decision record — codex cycle 1 (issue #2372)
Reviewer: codex
Cycle: 1
Verdict:don't-shipArchitectural read:
Event policy and participant ownership remain separate, but shared NAT migration replay and spare retry must be tightened before delivery.
Blocking findings: 3
Finding 1 —
one-off- ID:
core-F1 - Title: Align shared NAT containment rules
- Location:
docs/adr/index.yaml - Decision: fix
- Rationale: Update the binding GCP containment rule to permit serialized, subnet-explicit provisioner-owned shared NAT while preserving zero-egress isolation.
Finding 2 —
one-off- ID:
core-F2 - Title: Preserve migration-bridge replay
- Location:
shifter/engine/provisioner/gcp_range_cell_shared_nat.py - Decision: fix
- Rationale: Add a safe replay path for existing bridge-managed subnets without double NAT or unsafe handoff.
Finding 3 —
one-off- ID:
security-F1 - Title: Recheck reserved spare egress on retry
- Location:
shifter/shifter_platform/ctf/services/range/recovery_steps.py - Decision: fix
- Rationale: Revalidate the current event egress policy before old-range teardown on every recovery retry.
Notes (non-blocking, no decisions):
- Also bound shared NAT gateway names within the provider resource-name limit.
- ID:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd406c68a2d98f0a5ff4056541412c16971e08893 - Outcome:
merged_clean - Published feature head:
4d4cbc323eff8dbc5e14dcaa024449414bc9ad40 - Synchronized tree:
a9ed32b425719e79b4a414f874480ea1378697eb
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd406c68a2d98f0a5ff4056541412c16971e08893 - Outcome:
already_current - Published feature head:
ca0c22ef935f86fd5a6beed2d58df9f7a7f1b3f0 - Synchronized tree:
b658d0dccc5fb3af88aea23409d53f079929beaa
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd406c68a2d98f0a5ff4056541412c16971e08893 - Outcome:
already_current - Published feature head:
bc87fd6151f10f70ebe14150df131292a987d837 - Synchronized tree:
18357d83b288694d66a1b55ee5fec8f982554791
- Source:
Pre-PR base synchronization
- Source:
refs/remotes/origin/devatd406c68a2d98f0a5ff4056541412c16971e08893 - Outcome:
already_current - Published feature head:
a83bbffd0326eba1d28b8717b7290337307f3110 - Synchronized tree:
4ebab2bee02f01ff5df83a4c187e60a80e072616
- Source:
Ready for review — issue #2372
PR: #2381
Plan: #2372 (comment)Outcome
CTF participants, managed spares, and recovery launches now use the organizer event's authorized egress policy while retaining personal range ownership. GCP shared-VPC ranges use serialized, explicitly scoped regional NAT so multi-seat events do not consume one router per range.
PR #2381 is green and mergeable at a83bbff; live Nazgul scale qualification is in progress.
Files changed
Added:
docs/architecture/ctf-event-egress-preflight-2372.mdshifter/engine/provisioner/gcp_range_cell_shared_nat.pyshifter/engine/provisioner/tests/test_gcp_shared_nat.pyshifter/shifter_platform/cms/services/_raes_range_dispatch.pyshifter/shifter_platform/tests/ctf/test_services/test_range_recovery_validation.py
Modified:
docs/adr/index.yamldocs/dev/gcp-range-cell-deploy.mddocs/features/ctf-organizer-guide.mddocs/requirements/PLAT-238/requirement.mddocs/technical/platform_infrastructure/gcp-infrastructure.mdshifter/engine/provisioner/gcp_range_cell_clients.pyshifter/engine/provisioner/gcp_range_cell_destroy.pyshifter/engine/provisioner/gcp_range_cell_naming.pyshifter/engine/provisioner/gcp_range_cell_plan.pyshifter/engine/provisioner/gcp_range_cell_types.pyshifter/engine/provisioner/gcp_range_cells.pyshifter/engine/provisioner/raes_gcp_apply.pyshifter/engine/provisioner/raes_gcp_destroy.pyshifter/engine/provisioner/raes_gcp_plan.pyshifter/engine/provisioner/tests/test_gcp_range_cells.pyshifter/engine/provisioner/tests/test_raes_gcp_apply.pyshifter/engine/provisioner/tests/test_raes_gcp_plan.pyshifter/shifter_platform/cms/services/__init__.pyshifter/shifter_platform/cms/services/_non_user_range_launch.pyshifter/shifter_platform/cms/services/_raes_range_create.pyshifter/shifter_platform/cms/services/_range_launch_common.pyshifter/shifter_platform/cms/services/_range_reassign.pyshifter/shifter_platform/cms/services/_range_workspace.pyshifter/shifter_platform/cms/services/_warm_pool_claim.pyshifter/shifter_platform/cms/services/_warm_pool_reconcile.pyshifter/shifter_platform/ctf/bridges.pyshifter/shifter_platform/ctf/services/range/provision.pyshifter/shifter_platform/ctf/services/range/recovery_steps.pyshifter/shifter_platform/ctf/services/range/spares.pyshifter/shifter_platform/engine/services/__init__.pyshifter/shifter_platform/engine/services/_range_by_request.pyshifter/shifter_platform/tests/cms/test_warm_pool_claim.pyshifter/shifter_platform/tests/ctf/test_services/test_range_recovery.pyshifter/shifter_platform/tests/ctf/test_services/test_range_spares.py
Reviews
- Codex: One pre-push review cycle on the implementation resolved all three findings and one note; later Sonar-driven refactors passed focused tests and the complete hosted checks.
Traceability reconciliation
Proposed in this PR — verified against the merged tree in Phase E.
- IMPLEMENTS / TESTS / DOCUMENTS added: 0
- Links updated: 0
- Stale links removed: 0
Status
- CI: ✅ green
- SonarCloud: ✅ passed
- PR ready for user review and merge. Phase E validates the merged requirement state, posts the final report, and closes the issue automatically once this PR merges.
Phase E could not finalize PR #2381
The merge was detected and the delivery-readiness record was read, but finalization did
not complete. This issue is intentionally left open.- Reason:
delivery_readiness_head_mismatch - Detail: delivery-readiness for PR fix(ctf): scale event egress without per-range routers #2381 was recorded against a different head than the merged one
- Repair:
re_record_phase_d_readiness_for_the_merged_head_and_rerun_phase_e - Readiness record: comment 5806855753
Repair the cause, then re-run the Ground Control Phase E workflow for this pull request.
- Reason:
CTF participant provisioning passes the participant user into the CMS range creation path. The resulting range uses the participant's personal workspace egress policy, even when the event has a separate organizer-owned workspace. Managed spare users likewise receive a fresh personal workspace with the default policy before their range is created. In GCP range-cell mode, each status-quo range creates its own Cloud Router/NAT, exhausting the per-network router limit after only a few participants or spares. Changing the event workspace policy does not prevent this.
Make the launch workspace/egress policy explicit for CTF participant and spare ranges, with an authorization and ownership model that preserves participant isolation and organizer control. Replace the one-Cloud-Router-per-range design with a scalable egress path for shared-VPC events, and allow opt-in participant web research using the existing profile-level public-web egress contract. Provide tests for bulk participants with distinct personal workspaces, managed spare users, and an event workspace configured differently from those personal workspaces. Document expected network capacity and audit policy changes.