feat(range): shared participant OpenVPN server pool on GCP - #2491
Open
Brad-Edwards wants to merge 10 commits into
Open
Brad-Edwards wants to merge 10 commits into
Brad-Edwards wants to merge 10 commits into
Conversation
…eways Participant OpenVPN moves from one gateway VM per range to a shared, horizontally scaled server pool (#2480). Portal: - VpnSession records live sessions in PostgreSQL: one active session per range, and a newer connection supersedes the older one. - authorize/renew/end re-check access on every call (READY range, current generation and owner, access deadline), so destroy, reassignment, the end of the event, or a newer session ends a live tunnel within one heartbeat. - /api/v1/cms/vpn-control/ admits only the pool service account through its Google identity token (exact audience, email and numeric subject). - Profiles must pin the pool server certificate name. Provisioner: - Profiles are signed by one tenant CA and point at the pool address; the realization contract and the download path are unchanged. - Each OpenVPN range gets one ingress rule that admits the pool's networks to the target node on its declared participant channels only. - The per-range gateway VM, its firewall envelope, health probe, server and issuer secrets, the gateway service-account slot and the legacy GCE and Terraform-path VPN plumbing (never reachable) are removed. Refs #2480
One OpenVPN 2.6 process per pool VM with a controller on its management interface. Every connection is authorized by the portal, which returns the single target address and ports; a per-client nftables allowance admits only that tuple and only that route is pushed. Sessions heartbeat to the portal, revoked sessions are killed, and every session is dropped after 90 seconds without portal confirmation. The component has its own lint, SAST, and test lanes, Sonar coverage, and Dependabot entries. Refs #2480
A regional, autoscaled managed instance group of Container-Optimized OS VMs in its own subnet of the range VPC, behind one external passthrough UDP/1194 load balancer on a reserved address. The pool is sized from the shared-service capacity profile (25 participants per e2-standard-2 server, one spare, 30% CPU target) and checked by the capacity drift inspector. The deploy builds, attests, scans, and provenance-checks the pool image with the other release images, creates the tenant CA and renews the pool server certificate directly in Secret Manager (keys never enter Terraform state), records the verified digest, and replaces the servers by surging new ones in first. The deploy identity gets a custom role for the pool's templates, instance group, and autoscaler whose VM, disk, and template permissions are conditioned to the pool's names, plus actAs on the pool identity only. The per-range gateway identity pool, its 1195 health port, and its secret probe canaries are removed. Refs #2480
ADR-039-R10 now describes termination on the shared pool, portal-authorized sessions, revocation within one heartbeat, and fail-closed behavior; ADR-008-R7 and ADR-063-R5 drop the per-range gateway identity pool. The stale gateway service-account exception is removed. Deploy, capacity, CTF, and containment docs and code comments describe the pool, and the OpenVPN preflights note the superseded termination. Refs #2480
# Conflicts: # platform/charts/shifter/tests/test_chart_contract.py
Owner
Author
|
HOLD: do not merge. Live validation on a GCP tenant is still in progress (isolation, one session per range, revocation, failover, fail-closed). I'll remove this hold when it passes. |
…lope The pool server's metadata, Secret Manager, and portal calls now go through one opener that handles only http and https, follows no redirects, ignores proxy variables, and refuses other schemes (an OpenerDirector without UnknownHandler silently returned None for file: URLs). This resolves bandit B310 at the root. A credential-free Terraform contract test pins the pool envelope: one UDP 1194 listener, probe-range-only health ingress, 22/3389 range egress, deny-all below every allow, no public addresses, Shielded VM, surge replacement, and a pool identity that never reads the CA. The module is registered in the validation inventory with that contract. The gcloud subprocess in the PKI step follows the scripts/gcp argv-only annotation convention, and detect-private-key excludes the two modules that hold only the OpenVPN static-key armor labels. Refs #2480
…ings The pool image now defaults to an unprivileged user and cannot start without the deploy's hardened invocation: the VM starts it as root only so setpriv can switch to uid 10001 keeping NET_ADMIN as its sole capability (verified in the two-network lab: OpenVPN and the controller run as 10001 with CapEff NET_ADMIN only, isolation and heartbeat revocation unchanged). OpenVPN no longer switches to nobody, and its management socket admits only that user. The plain-HTTP health server is replaced by a TCP listener that opens once the server is serving, with a TCP load-balancer health check. The HTTPS handler uses the default verifying context. The tunnel network is a module variable passed to the server instead of a hardcoded default. Remaining smells: named refusal codes and management markers, a split target resolver, docstrings, suppression comments Sonar can parse, the capacity profile invariants in their own module, and a Dockerfile that follows the repo convention (digest-only FROM, maintainer label, sorted packages). Refs #2480
Owner
Author
|
Live validation on a GCP tenant passed (deployed from this branch's commits):
Hold lifted once CI is green on the dev-merge commit just pushed. |
|
Owner
Author
|
HOLD LIFTED. CI and SonarCloud are green on dd2433b (current with dev) and the tenant validation above passed. Ready for your review and merge. |
The controller is the container's PID 1 and ignored SIGTERM, so docker stop waited out its timeout and SIGKILLed OpenVPN; the configured explicit-exit-notify never fired and clients sat out the full keepalive timeout on every planned removal (deploy, scale-in, VM replacement). The controller now passes SIGTERM/SIGINT to OpenVPN, which tells every client to reconnect at once, and exits cleanly when it was asked to stop. The GCP bootstrap guide now has an OpenVPN step: apply global/cicd-oidc for the conditioned pool roles, set RANGE_OPENVPN_ENABLED, and enable through the CI deploy, because the local gdc-bootstrap path never deploys the pool. Refs #2480
Owner
Author
|
HOLD: do not merge. New commit 24c0309 fixes a shutdown bug: the exit notification never reached clients, so planned server removals cost participants the full keepalive timeout. It also adds the OpenVPN step to the GCP bootstrap guide. A keepalive change may follow. Both still need CI and a live failover recheck on the tenant. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2480
Replaces per-range OpenVPN gateway VMs with one shared, horizontally scaled OpenVPN server pool per deployment (GCP). AWS follows in #2481.
What changes
modules/vpn-pool): regional autoscaled MIG of Container-Optimized OS VMs in its own subnet of the range VPC, no public IPs, behind one external passthrough UDP/1194 load balancer on a reserved address (client-IP affinity, HTTP health check). NAT scoped to the pool subnet for the portal call only. Firewall envelope: allows at 900, denies at 950.shifter/engine/openvpn): OpenVPN 2.6 plus a controller on the management interface. Each connection is authorized by the portal, which returns one target address and its ports; a per-client nftables allowance (forward policy drop) admits only that tuple, and only that route is pushed. Container: own netns,--cap-drop ALL+ NET_ADMIN/SETUID/SETGID, read-only root, no-new-privileges; OpenVPN drops tonobody.VpnSession, one active session per range (newer wins), 15 s heartbeat / 60 s lease. Destroy, pause, owner change, deadline, or a newer session disconnects within one heartbeat; a server that cannot confirm sessions drops all clients after 90 s. Controller authenticates with an exact-audience Google ID token pinned to the pool SA's email and numeric ID.notAfter= deadline andverify-x509-namepinned. Created/renewed byscripts/gcp/ensure_vpn_pki.pyat deploy; private keys never enter Terraform state._gcp-dev.yml): build, attest, exact-digest Trivy scan, and provenance-verify the pool image with the other release images; ensure PKI; write the release record (digest) only after the gates; surge-replace servers (max-unavailable 0), wait stable, require all backends healthy.<prefix>-vpn-*names (notcompute.instanceAdmin.v1, which the release-security guard forbids); actAs only on the pool SA. Requires applyingglobal/cicd-oidcper identity profile before enabling the pool.VpnPoolCapacityin the shared-service profiles (25 participants per e2-standard-2, +1 spare, 30% CPU target, max = 2x min) and drift checking viacheck_event_capacity_drift.py --vpn-pool.sh-vpn-pool-*), the 1195 health port, per-range server secrets, the gateway secret-probe canaries, the stale gateway SA exception.Verification
terraform validateon the environment and CI-identity roots.