From 0d8dcd7a620650f73fc91acb2d48fbfb75d43fd1 Mon Sep 17 00:00:00 2001 From: Orga Shih Date: Tue, 19 May 2026 01:51:24 +0800 Subject: [PATCH 1/3] factory: generates gcp/vm stubs (Compute Engine provider, full vm suite + new virtual_device_hardening step) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a complete GCP Compute Engine provider under `isvctl/configs/providers/gcp/` covering the full vm test suite — `launch_instance`, `list_instances`, `verify_tags`, `serial_console`, `console_rbac`, `virtual_device_hardening` (new step from PR #413), `stop_instance`, `start_instance`, `reboot_instance`, `describe_instance`, `deploy_nim` (shared) — plus teardown. AWS oracle contract shape preserved across all steps; GCE-specific divergences encoded per step where the API surface differs from EC2 (zone capacity walk, async-operation cleanup tracker semantics, verified-reuse `instance_created` ownership flag, IAM token-creator propagation budget, instance metadata fields for serial-console RBAC probes). Also adds operator setup documentation at `docs/references/gcp.md` (linked from `docs/getting-started.md`) covering authentication (ADC / service-account key), project-ID resolution order, required IAM roles, L4 GPU quota and zone list, NIM-step `NGC_API_KEY` handling, GPU-image default + how to override for Docker-based tests, and org-policy considerations operators commonly need to check before a clean run. GCP gets its own reference page rather than mixing into `references/aws.md` since AWS is treated as the canonical oracle example; future target NCPs follow the same per-NCP pattern. The default `--image-project` / `--image-family` are the public GCP Deep Learning VM Image (`deeplearning-platform-release/common-cu129-ubuntu-2204-nvidia-580`) so a clean checkout can run the VM suite without any private-image entitlement. The DLVM ships with the NVIDIA driver + CUDA toolkit; operators who need Docker (for the `deploy_nim` step) override the default by exporting `GCP_VM_IMAGE` / `GCP_VM_IMAGE_PROJECT` in their shell / `.env` (the provider config reads both via Jinja, identical pattern to the existing `GCP_VM_SKIP_TEARDOWN` precedent), or skip NIM by leaving `NGC_API_KEY` unset (the shared script short-circuits cleanly). `--set image=... --set image_project=...` is also accepted for one-off ad-hoc invocations. Mirrors two adjacent upstream changes that landed on main while this branch was in-flight: - PR #413 (commit 2cdf4d76, "feat: add virtual device hardening validation" by @mresvanis): the vm suite contract gained `virtual_device_hardening` as a new test step. The GCP mirror is a 302-line stub at `gcp/scripts/vm/virtual_device_hardening.py` — provider-evidence preamble naming Compute Engine (no customer-facing USB / clipboard surface on GCE tenant VMs), guest-side SSH probes using the same PROBE_SENTINEL framing and pattern lists as the AWS oracle, emitting the three REQUIRED_TESTS subtests (`usb_devices_disabled`, `clipboard_disabled`, `unnecessary_virtual_devices_absent`). Adds a sister `ssh_run` helper to `gcp/scripts/common/ssh_utils.py` returning `(exit_code, stdout, stderr)` with sentinel exit codes for TimeoutExpired (124) and OSError (255). - PR #424 (commit 6e1458a7, "fix: handle AWS VM validation regressions" by @abegnoche): adds `IdentityAgent=none` alongside the existing `IdentitiesOnly=yes` so explicit `-i ` is the only SSH credential considered on operator hosts running a multi-identity ssh-agent. Mirrored to the GCP `_SSH_OPTS` tuple so every GCP SSH helper benefits uniformly. Verification: - **Static checks**: PASS (46/46). - **Independent post-generation review on branch HEAD**: `PASS p1=0 p2=0`. - **GCP orphans after teardown**: 0 on every run. - **Live execution end-to-end** — verified under both supported operator configurations: - canonical / custom-image (`NGC_API_KEY` set + `GCP_VM_IMAGE` + `GCP_VM_IMAGE_PROJECT` set to a Docker-equipped image): 3 consecutive full-domain PASS — 9 test steps + NIM deploy / teardown + cleanup, ~1650s per run. Post-upstream-refresh re-verification (with the new `virtual_device_hardening` step + IdentityAgent flag + `ssh_run` helper) also PASS — domain-gate live run `test-f65be511` 330.8s, all 11 vm steps + NIM deploy/teardown + cleanup, 0 orphans. - public-default (all three env vars unset → suite falls through to the DLVM): 1 expected FAIL on `host_os.ContainerRuntimeCheck` (Docker not available — DLVM ships NVIDIA driver + CUDA but not Docker; this is documented in `docs/references/gcp.md` §5 as a known portability gap of the public default). Everything else PASS: all 9 lifecycle steps, NIM steps cleanly `runtime_skip`, both `deploy_nim` and `teardown_nim` policy-skip honored end-to-end. 1113.5s. Known gaps: - `ContainerRuntimeCheck` fails on the public DLVM default image (documented above). Operators who need a fully green run override the image via `GCP_VM_IMAGE` / `GCP_VM_IMAGE_PROJECT` to a Docker-equipped image. The public default is published as "boot + lifecycle works without any private entitlement", not as "every validator passes" — see `docs/references/gcp.md` §5. Branch history before squash-for-PR: 32 commits — initial factory-generated foundation + per-step worker + domain-gate + final-review-fix commits, plus 7 operator hand-fixes addressing post-generation review findings (cleanup-bool propagation, socket-timeout retry, zone-walk contract, probe-network arg threading, factory-vocabulary scrub, warm-reuse stability gate, symmetric NIM teardown policy-skip, env-var override architecture), plus 2 docs commits (operator setup reference + ContainerRuntimeCheck clarification), plus a single squashed factory-update-mode regen against the new upstream main tip 9b79e340 bringing the two adjacent upstream mirrors above. Squashed here for a single-commit upstream PR. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Orga Shih --- .factory_version | 11 + docs/getting-started.md | 1 + docs/references/gcp.md | 141 +++ isvctl/configs/providers/gcp/config/vm.yaml | 359 ++++++ .../providers/gcp/scripts/common/__init__.py | 2 + .../providers/gcp/scripts/common/compute.py | 915 +++++++++++++++ .../providers/gcp/scripts/common/errors.py | 135 +++ .../providers/gcp/scripts/common/ssh_utils.py | 283 +++++ .../providers/gcp/scripts/vm/console_rbac.py | 1003 +++++++++++++++++ .../gcp/scripts/vm/describe_instance.py | 91 ++ .../providers/gcp/scripts/vm/describe_tags.py | 75 ++ .../gcp/scripts/vm/launch_instance.py | 880 +++++++++++++++ .../gcp/scripts/vm/list_instances.py | 119 ++ .../gcp/scripts/vm/reboot_instance.py | 242 ++++ .../gcp/scripts/vm/serial_console.py | 114 ++ .../gcp/scripts/vm/start_instance.py | 176 +++ .../providers/gcp/scripts/vm/stop_instance.py | 153 +++ .../providers/gcp/scripts/vm/teardown.py | 372 ++++++ .../scripts/vm/virtual_device_hardening.py | 302 +++++ .../configs/providers/shared/teardown_nim.py | 42 + isvctl/pyproject.toml | 1 + uv.lock | 139 +++ 22 files changed, 5556 insertions(+) create mode 100644 .factory_version create mode 100644 docs/references/gcp.md create mode 100644 isvctl/configs/providers/gcp/config/vm.yaml create mode 100644 isvctl/configs/providers/gcp/scripts/common/__init__.py create mode 100644 isvctl/configs/providers/gcp/scripts/common/compute.py create mode 100644 isvctl/configs/providers/gcp/scripts/common/errors.py create mode 100644 isvctl/configs/providers/gcp/scripts/common/ssh_utils.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/console_rbac.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/describe_instance.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/describe_tags.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/launch_instance.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/list_instances.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/reboot_instance.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/serial_console.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/start_instance.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/stop_instance.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/teardown.py create mode 100644 isvctl/configs/providers/gcp/scripts/vm/virtual_device_hardening.py diff --git a/.factory_version b/.factory_version new file mode 100644 index 00000000..39dae5db --- /dev/null +++ b/.factory_version @@ -0,0 +1,11 @@ +{ + "ncps": { + "gcp": { + "vm": { + "factory_commit": "21869ef830d5a088250206c37c757c4ffc28eeeb", + "isv_commit": "9b79e34037435b8567e3cd79e18dae01ae1ea0e2" + } + } + }, + "source_repo": "git@github.com:NVIDIA/ISV-NCP-Validation-Suite.git" +} diff --git a/docs/getting-started.md b/docs/getting-started.md index 93c848b1..3b0fd30a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -136,6 +136,7 @@ See [Remote Deployment Guide](guides/remote-deployment.md) for details. - [my-isv Scaffold](../isvctl/configs/providers/my-isv/scripts/README.md) - Adding your own platform? Start here - [Validation Test Suites](../isvctl/configs/suites/README.md) - The platform-agnostic validation contract - [AWS Reference Implementation](references/aws.md) - Working AWS examples to study +- [GCP Target Implementation](references/gcp.md) - Operator setup for running tests on GCP - [Configuration Guide](guides/configuration.md) - Config file format and options - [External Validation Guide](guides/external-validation-guide.md) - Custom validations without modifying the repo - [Local Development](guides/local-development.md) - Running tests locally diff --git a/docs/references/gcp.md b/docs/references/gcp.md new file mode 100644 index 00000000..1538b4a2 --- /dev/null +++ b/docs/references/gcp.md @@ -0,0 +1,141 @@ +# GCP Target Implementation + +The GCP implementation is a target-platform port of the ISV validation framework. The [AWS reference](aws.md) is the canonical example for how the suite expects each provider to behave; the GCP scripts under `providers/gcp/` translate that contract onto Google Cloud's API surface (Compute Engine, IAM, etc.). + +This page covers the operator setup needed to run `isvctl` tests against GCP. + +## Available Modules + +| Domain | Config | Scripts | Test Suite | +|--------|--------|---------|------------| +| **VM** | [`providers/gcp/config/vm.yaml`](../../isvctl/configs/providers/gcp/config/vm.yaml) | [`providers/gcp/scripts/vm/`](../../isvctl/configs/providers/gcp/scripts/vm/) | [`suites/vm.yaml`](../../isvctl/configs/suites/vm.yaml) | + +Shared GCP utilities (compute helpers, SSH wrappers, retry envelopes, error classifiers) are in [`providers/gcp/scripts/common/`](../../isvctl/configs/providers/gcp/scripts/common/). + +Other domains (IAM, Network, Bare Metal, EKS, Control Plane, Image Registry, Security) are not yet implemented for GCP. + +## Prerequisites + +### 1. Operator GCP environment + +- A GCP project with **billing enabled** and the **Compute Engine API** enabled. +- **L4 GPU quota** (`NVIDIA_L4_GPUS`) of at least 1 in at least one zone listed under [Supported zones](#supported-zones-for-l4-gpu-vm-tests) below. The VM domain's `launch_instance` step provisions a `g2-standard-8` instance for the lifecycle, console-RBAC, and NIM subtests. +- The principal running the tests (user or service account) needs roughly these roles on the project (consolidate via custom role if preferred): + - `roles/compute.admin` — create / delete / start / stop / reboot instances and firewall rules. + - `roles/iam.serviceAccountAdmin` + `roles/iam.serviceAccountUser` — `console_rbac` self-provisions short-lived probe service accounts and mints access tokens against them. + - `roles/iam.serviceAccountTokenCreator` on the probe SAs (granted dynamically by the test itself, but the project-level binding is needed to allow the test to grant it). + +### 2. Authentication + +`isvctl` calls into the GCP Python SDKs (`google-cloud-compute`, `google-auth`, etc.) which use Application Default Credentials. Either: + +```bash +# Option A — user credentials (recommended for local dev / interactive runs) +gcloud auth application-default login + +# Option B — service account key file +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json +``` + +### 3. Project ID resolution + +The GCP scripts resolve the active project ID in this order: + +1. Explicit `--project` argument (rarely passed by the suite; reserved for ad-hoc invocations). +2. `GOOGLE_CLOUD_PROJECT` environment variable. +3. `GCLOUD_PROJECT` environment variable. +4. The project bundled with Application Default Credentials (set by `gcloud auth application-default login`). + +If none resolve, every step fails fast with a structured `credentials_missing` error. + +```bash +# Most operators just set this once: +export GOOGLE_CLOUD_PROJECT=your-project-id +``` + +### 4. NIM API key (for `deploy_nim` step) + +The shared NIM deployment script reads `NGC_API_KEY` from the environment. If unset, the step short-circuits with `success=True, skipped=True` and the rest of the suite continues — useful for environments where NIM isn't part of the operator's validation scope. + +```bash +export NGC_API_KEY=nvapi-... +``` + +### 5. GPU image (and Docker requirement for `deploy_nim`) + +`launch_instance` defaults to the public GCP Deep Learning VM Image: + +| Field | Default | +|---|---| +| `--image-project` | `deeplearning-platform-release` | +| `--image-family` | `common-cu129-ubuntu-2204-nvidia-580` | + +This image is published by Google, ships with the NVIDIA driver + CUDA toolkit, and works out-of-the-box for lifecycle / serial-console / RBAC / describe coverage. + +**It does NOT ship Docker.** The `deploy_nim` step pulls and runs a NIM container, so it requires a Docker engine on the launched VM. Operators who need NIM coverage have two options: + +1. **Bring a custom image** that has Docker + NVIDIA Container Toolkit preinstalled. The recommended path is to set the override once in your shell / `.env` so every run reuses the same pin: + + ```bash + # Add to your .env or shell environment. + export GCP_VM_IMAGE= + export GCP_VM_IMAGE_PROJECT= + ``` + + The provider config reads both env vars via Jinja and falls back to the public DLVM when either is unset. For one-off runs you can also override per-invocation: + + ```bash + uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml \ + --set image_project= \ + --set image= + ``` + + Either path wires `image_project` and `image` through to `launch_instance`'s `--image-project` / `--ami-id` arguments. The image short-name resolves in the operator project first; if not found there, the resolver falls back to the default DLVM project. + +2. **Skip NIM** by leaving `NGC_API_KEY` unset (see §4 above). The `deploy_nim` and `teardown_nim` steps short-circuit cleanly and every instance-lifecycle step proceeds. The run still reports `[FAIL] TEST` because of `ContainerRuntimeCheck` (see note below) — accept that as a documented limitation of the default image. + +**Note on `ContainerRuntimeCheck`**: the `host_os` validator group includes `ContainerRuntimeCheck`, which asserts Docker is installed and runnable on the launched VM. It runs on every `vm` invocation regardless of `NGC_API_KEY`. On the default DLVM image this validator fails with `Docker not available`, and the run reports `[FAIL] TEST` even though every instance-lifecycle and NIM-policy-skip step passes. Only option 1 (a custom image with Docker preinstalled) produces a fully clean PASS. + +Operators without an NGC entitlement should pick option 2; operators with one and no custom image can install Docker on the default image inside their own cloud-init / startup-script, but the simplest path is a custom image where `docker run --gpus all` works at boot. + +## Running GCP Validations + +```bash +# Prerequisites: ADC + GOOGLE_CLOUD_PROJECT set per "Authentication" above. +# Optional: NGC_API_KEY for NIM coverage. + +uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml +``` + +The VM suite exercises 11 subtests end-to-end: launch (with GPU + cloud-init + SSH stability gate), tag verification, serial console output, console-RBAC probe (creates two short-lived probe service accounts + a second probe VM), idempotent stop / start / reboot lifecycle, describe (host OS / driver / CPU / container runtime checks), NIM deploy + inference, teardown of all created resources. Wall-clock is roughly 30–45 minutes on a clean operator environment; capacity stockout in one zone triggers a documented walk to the next zone in the preferred list. + +## Supported zones for L4 GPU VM tests + +The VM domain's zone-walk prefers GCP zones with observed L4 capacity. The reviewed list (in priority order) is: + +``` +us-central1-a / -b / -c us-east4-a / -b / -c us-east1-c / -d +us-west1-a / -b us-west4-a / -b +europe-west4-a / -b europe-west1-b / -c +asia-southeast1-a / -b / -c asia-northeast1-a / -c asia-east1-a / -b / -c +``` + +The list reflects multi-week capacity observation from spring 2026; capacity drifts, so an operator can probe a zone before a long run with: + +```bash +gcloud compute accelerator-types list --filter='name=nvidia-l4 AND zone:' +``` + +## Org-policy considerations + +GCP organizations sometimes apply policies that block specific operations the VM suite needs. Common ones to check or exempt: + +- `compute.disableSerialPortAccess` — must allow. The `console_rbac` test validates that serial-console access is properly RBAC-restricted; the test itself reads serial output via IAM-mediated short-lived tokens. +- `iam.disableServiceAccountKeyCreation` — does NOT need to be disabled. The suite uses `IAMCredentials.generateAccessToken` (no key material), not service-account JSON keys. +- `compute.requireOsLogin` — must allow per-instance SSH-key metadata, which the suite uses to establish SSH for the post-launch stability gate and NIM health checks. If OS Login is enforced project-wide, the SSH gate fails; either exempt the test instances or grant the operator the OS Login roles. + +## Resources + +- GCP IAM permissions for Compute Engine: +- L4 GPU zones (current as of GCP docs): +- Application Default Credentials: diff --git a/isvctl/configs/providers/gcp/config/vm.yaml b/isvctl/configs/providers/gcp/config/vm.yaml new file mode 100644 index 00000000..ed36f6d5 --- /dev/null +++ b/isvctl/configs/providers/gcp/config/vm.yaml @@ -0,0 +1,359 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +# GCP VM (Compute Engine) Validation Configuration. +# +# Imports the provider-agnostic VM contract from suites/vm.yaml and +# wires every step to a Compute Engine stub under +# providers/gcp/scripts/vm/. +# +# Steps mirror the AWS oracle order. Verified-reuse propagation: +# launch_instance emits zone, instance_created, firewall_created, +# key_created, key_file, firewall_name. Compute Engine has no managed +# key-pair store so launch_instance emits result.key_name = null and +# teardown does not accept or forward --key-name. Every downstream +# zonal step reads {{steps.launch_instance.zone}}; teardown reads +# instance_created, firewall_created, and key_created so destruction +# gates on each ownership bit (verified-reuse cleanup contract). +# instance_created is False on the +# GCP_VM_INSTANCE_ID / GCP_VM_KEY_FILE adoption path so the long-lived +# operator VM is preserved across the run. +# +# Validator override under cloud_init translates the CloudInitCheck's +# default AWS metadata URL to the Compute Engine metadata server + +# Metadata-Flavor header. +# +# Usage: +# uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml + +import: + - ../../../suites/vm.yaml + +version: "1.0" + +commands: + vm: + phases: ["setup", "test", "teardown"] + steps: + # 1. Launch a GPU VM (compute_v1.insert). Cap parity with the AWS + # oracle's launch step; per-zone op waits and the multi-zone walk + # are bounded inside the stub via deadline budgets, not summed into + # the orchestrator cap. + - name: launch_instance + phase: setup + command: "python3 ../scripts/vm/launch_instance.py" + args: + - "--name" + - "isv-test-gpu" + - "--instance-type" + - "{{instance_type}}" + - "--region" + - "{{region}}" + - "--vpc-id" + - "{{network}}" + - "--subnet-id" + - "{{subnet}}" + - "--ami-id" + - "{{image}}" + - "--image-project" + - "{{image_project}}" + - "--firewall-name" + - "{{firewall_name}}" + - "--key-name" + - "{{key_name}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 900 + + # 2. List instances filtered by network (compute_v1.aggregated_list). + - name: list_instances + phase: test + command: "python3 ../scripts/vm/list_instances.py" + args: + - "--vpc-id" + - "{{steps.launch_instance.vpc_id}}" + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + timeout: 120 + + # 3. Verify labels round-trip to canonical tags (compute_v1.get). + - name: verify_tags + phase: test + command: "python3 ../scripts/vm/describe_tags.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + timeout: 60 + + # 4. Serial console probe (compute_v1.get_serial_port_output). + - name: serial_console + phase: test + command: "python3 ../scripts/vm/serial_console.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + timeout: 60 + + # 4b. Console RBAC probe — three subtests. Default path + # self-provisions denied/allowed probe service accounts and a + # second VM, then mints short-lived tokens via IAMCredentials REST + # to call getSerialPortOutput as each principal. + - name: console_rbac + phase: test + command: "python3 ../scripts/vm/console_rbac.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--network" + - "{{steps.launch_instance.vpc_id | default(network, true)}}" + timeout: 600 + + # 4c. Virtual-device hardening evidence — three subtests + # (usb_devices_disabled, clipboard_disabled, + # unnecessary_virtual_devices_absent). Provider preamble records + # that Compute Engine has no customer-facing USB / clipboard + # surface; guest-side SSH probes (single session, sentinel-framed) + # flip a subtest to failed only on real Linux-level signals. + - name: virtual_device_hardening + phase: test + command: "python3 ../scripts/vm/virtual_device_hardening.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--public-ip" + - "{{steps.launch_instance.public_ip}}" + - "--key-file" + - "{{steps.launch_instance.key_file}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 120 + + # 5. Stop (compute_v1.stop) — pre-gates on cloud-init. Cap parity + # with the AWS oracle stop step; cloud-init waits and op-poll + # retries are bounded by deadlines inside the stub. + - name: stop_instance + phase: test + command: "python3 ../scripts/vm/stop_instance.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--key-file" + - "{{steps.launch_instance.key_file}}" + - "--public-ip" + - "{{steps.launch_instance.public_ip}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 600 + + # 6. Start (compute_v1.start) — re-reads fresh public IP. Cap + # parity with the AWS oracle's start step (AWS uses 300s, this is + # 2x to absorb GCP's slower compute_v1.start operation polling and + # cloud-init re-arming on g2-standard-8). SSH stability and + # cloud-init waits are bounded inside the stub via deadlines so + # this cap is not a sum of internal waits. + - name: start_instance + phase: test + command: "python3 ../scripts/vm/start_instance.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--key-file" + - "{{steps.launch_instance.key_file}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 600 + + # 7. Reset (compute_v1.reset) — SSH-drop wait + stability gate. + # Cap parity with the AWS oracle reboot step; ssh-drop, ssh-stable, + # and cloud-init waits are bounded by deadlines inside the stub. + - name: reboot_instance + phase: test + command: "python3 ../scripts/vm/reboot_instance.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--key-file" + - "{{steps.launch_instance.key_file}}" + - "--public-ip" + - "{{steps.start_instance.public_ip}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 900 + + # 8. Describe (compute_v1.get) — post-reboot host validation anchor. + - name: describe_instance + phase: test + command: "python3 ../scripts/vm/describe_instance.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--key-file" + - "{{steps.launch_instance.key_file}}" + - "--ssh-user" + - "{{ssh_user}}" + timeout: 120 + + # 9. Deploy NIM (shared paramiko script). On canonical GCP runs the + # operator's environment provides NGC_API_KEY (forwarded by the + # harness server); the shared script gracefully skips with + # success=True / skipped=True / rc=0 when the credential is missing, + # so leaving the step active here keeps the NIM validation honest on + # equipped operators without producing false failures otherwise. + - name: deploy_nim + phase: test + command: "python3 ../../shared/deploy_nim.py" + args: + - "--host" + - "{{steps.reboot_instance.public_ip}}" + - "--key-file" + - "{{steps.reboot_instance.key_file}}" + - "--user" + - "{{steps.reboot_instance.ssh_user | default(ssh_user, true)}}" + timeout: 1800 + + # 10. Teardown NIM (shared paramiko script). Non-empty sentinel + # defaults so an upstream skip doesn't collapse the args (empty + # defaults are DROPPED by the orchestrator's _render_args). + - name: teardown_nim + phase: teardown + command: "python3 ../../shared/teardown_nim.py" + args: + - "--host" + - "{{steps.reboot_instance.public_ip | default(steps.start_instance.public_ip, true) | default(steps.launch_instance.public_ip, true) | default('none', true)}}" + - "--key-file" + - "{{steps.reboot_instance.key_file | default(steps.start_instance.key_file, true) | default(steps.launch_instance.key_file, true) | default('none', true)}}" + - "--user" + - "{{steps.reboot_instance.ssh_user | default(ssh_user, true)}}" + timeout: 180 + + # 11. Teardown — gates instance + firewall + key cleanup on + # verified-reuse flags. instance_created gates BOTH the primary + # and the leaked-zone deletes so a run that adopted a + # long-lived operator VM (GCP_VM_INSTANCE_ID / GCP_VM_KEY_FILE) + # cannot destroy resources it did not create. + - name: teardown + phase: teardown + command: "python3 ../scripts/vm/teardown.py" + args: + - "--instance-id" + - "{{steps.launch_instance.instance_id | default('none', true)}}" + - "--region" + - "{{region}}" + - "--zone" + - "{{steps.launch_instance.zone | default(zone)}}" + - "--delete-key-pair" + - "--delete-security-group" + - "--instance-created" + - "{{steps.launch_instance.instance_created | default('false')}}" + - "--firewall-name" + - "{{steps.launch_instance.firewall_name | default('none')}}" + - "--firewall-created" + - "{{steps.launch_instance.firewall_created | default('false')}}" + - "--key-file" + # The producer initializes key_file to a sentinel and only sets + # the real path after key generation succeeds. Boolean-mode + # defaulting keeps empty or missing values from collapsing the + # argv pair and shifting following flags into the value slot. + - "{{steps.launch_instance.key_file | default('none', true)}}" + - "--key-created" + - "{{steps.launch_instance.key_created | default('false')}}" + - "--leaked-zones" + # Happy path emits leaked_zones=[]; join(',') is an empty string. + # Boolean-mode defaulting converts both undefined and empty + # values to the sentinel so the teardown argv stays aligned. + - "{{steps.launch_instance.leaked_zones | join(',') | default('none', true)}}" + - "{{teardown_flag}}" + # Cap parity with the AWS oracle teardown; per-resource op waits + # and leaked-zone cleanup are bounded inside the stub via + # deadlines. + timeout: 600 + +tests: + cluster_name: "gcp-vm-validation" + description: "GCP VM (Compute Engine) validation tests" + + settings: + region: "us-central1" + zone: "us-central1-a" + # g2-standard-8 + nvidia-l4 is the validated GPU pin. The pin + # trades a $0.17/hr premium for the V1 NIM engine path (~2-3 min + # cold-start) and enough vCPU/RAM headroom for ContainerRuntimeCheck + # side-containers. + instance_type: "g2-standard-8" + network: "default" + # The "none" sentinel survives the orchestrator's empty-arg dropping + # so the --subnet-id / --ami-id / --image-project flag/value argv + # pairs stay aligned. launch_instance.py treats this literal as + # absent and falls back to the default subnet for the resolved + # zone / the canonical GPU image. + # + # `image` / `image_project` Jinja-read from the operator's + # environment first (`GCP_VM_IMAGE` / `GCP_VM_IMAGE_PROJECT`, + # typically set in `.env` or shell env) and fall back to the + # "none" sentinel when those env vars are unset. The stub then + # falls back to its public DLVM default. Operators with a + # Docker-equipped custom image set `GCP_VM_IMAGE=` + # and `GCP_VM_IMAGE_PROJECT=` once; subsequent runs + # reuse the same pin. Single-run override via + # `--set image=...` / `--set image_project=...` is also supported + # for ad-hoc invocations. + subnet: "none" + # default('none', true) — second arg makes the filter fire on + # *falsy* values too (empty string, not just undefined), so + # GCP_VM_IMAGE= (set but empty in .env) collapses to the "none" + # sentinel the stub treats as "use the public DLVM default." + image: "{{env.GCP_VM_IMAGE | default('none', true)}}" + image_project: "{{env.GCP_VM_IMAGE_PROJECT | default('none', true)}}" + firewall_name: "isv-test-vm-ssh" + key_name: "isv-test-key" + ssh_user: "ubuntu" + teardown_flag: "{{(env.GCP_VM_SKIP_TEARDOWN == 'true') | ternary('--skip-destroy', '')}}" + + # GCP-specific validator overrides. + # GROUP-NESTED shape — the FLAT shape is silently dropped by + # isvtest/src/isvtest/main.py:244 and the validator falls back to the + # AWS default when the upstream validator exposes a mode/option flag. + validations: + cloud_init: + step: launch_instance + checks: + CloudInitCheck: + metadata_url: "http://metadata.google.internal/computeMetadata/v1/" + metadata_headers: + Metadata-Flavor: "Google" diff --git a/isvctl/configs/providers/gcp/scripts/common/__init__.py b/isvctl/configs/providers/gcp/scripts/common/__init__.py new file mode 100644 index 00000000..d6f9ca70 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/common/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary diff --git a/isvctl/configs/providers/gcp/scripts/common/compute.py b/isvctl/configs/providers/gcp/scripts/common/compute.py new file mode 100644 index 00000000..f6f16a53 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/common/compute.py @@ -0,0 +1,915 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Shared Compute Engine helpers for GCP VM stubs. + +This module is the canonical home for Compute Engine divergences from +the AWS oracle: + + * ``resolve_project`` — env / ADC project resolution because the + harness does NOT forward GOOGLE_CLOUD_PROJECT to spawned stubs. + * ``narrow_region_to_zone`` / ``zone_to_region`` — Compute Engine + instance APIs are zone-scoped; provider configs may supply either. + * ``canonical_state`` — translate Compute Engine raw status to the + canonical lifecycle vocabulary the suite expects. + * ``wait_for_zonal_op`` / ``wait_for_global_op`` — block on a + Compute Operation's terminal DONE. + * ``poll_instance_state`` — poll instances.get for canonical state. + * ``wait_for_public_ip`` — ephemeral external IPs are released on stop; + post-start / post-reset code MUST re-read rather than reuse a cached + arg. + * ``generate_ssh_keypair`` — local PEM/.pub pair; returns + ``(path, created)`` for the verified-reuse cleanup contract. + * ``ensure_ssh_firewall`` — verified-reuse SSH firewall rule on the + target network; returns ``(name, created)``. + * Label projection helpers — canonical mixed-case Name/CreatedBy tags + project to api-valid lowercase labels on create and back on read. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +import time +import uuid +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import google.auth +from google.api_core import exceptions as gax +from google.cloud import compute_v1 + +# --------------------------------------------------------------------- # +# Auth / project resolution # +# --------------------------------------------------------------------- # + +# The harness does NOT forward GOOGLE_CLOUD_PROJECT/GCLOUD_PROJECT to +# spawned stubs. Fall back to +# Application Default Credentials' bundled project_id when neither --project +# nor an env var is set so operators with `gcloud auth application-default +# login` don't have to thread the project through every call. +_PROJECT_ENV_VARS: tuple[str, ...] = ("GOOGLE_CLOUD_PROJECT", "GCLOUD_PROJECT") + + +def resolve_project(arg_value: str | None = None) -> str: + """Resolve the active GCP project ID. + + Order: explicit ``--project`` arg, then ``GOOGLE_CLOUD_PROJECT`` / + ``GCLOUD_PROJECT`` env var, then ``google.auth.default()`` (ADC). + Raises ``RuntimeError`` if nothing resolves so the failure surfaces as + a structured ``credentials_missing``-class error rather than a hidden + AttributeError downstream. + """ + if arg_value: + return arg_value + for var in _PROJECT_ENV_VARS: + val = os.environ.get(var, "").strip() + if val: + return val + try: + _, project = google.auth.default() + except Exception as e: + raise RuntimeError( + f"Could not resolve GCP project ID via ADC: {e}. " + "Run `gcloud auth application-default login` or pass --project." + ) from e + if not project: + raise RuntimeError( + "GCP project ID not found. Set GOOGLE_CLOUD_PROJECT, pass --project, " + "or run `gcloud auth application-default login` with a project quota." + ) + return project + + +# --------------------------------------------------------------------- # +# Run-id suffixing # +# --------------------------------------------------------------------- # + + +def unique_suffix(base: str, *, length: int = 8) -> str: + """Append the suite's ``RUN_ID`` (or a fresh UUID8) to ``base``. + + Compute Engine resource names ARE the API IDs (name-collision + risk). Every user-supplied name in this provider — instance, + firewall, local + key file — MUST flow through this helper so that: + + * Concurrent test runs (different ``RUN_ID``s) don't collide + on ``AlreadyExists`` during create. + * Operators can group artifacts by run id (``gcloud compute + instances list --filter "name~$RUN_ID"``). + * Same-session teardown deletes only its own resources. + + Falls back to a random UUID8 only when ``RUN_ID`` is unset (e.g. + manual stub invocation without the harness setting the env var). + The helper MUST NOT raise on missing env var — that would block + ad-hoc reproduction. + """ + sid = os.environ.get("RUN_ID") or os.environ.get("LS_RUN_ID") or "" + return f"{base}-{sid[:length] if sid else uuid.uuid4().hex[:length]}" + + +# --------------------------------------------------------------------- # +# Zone / region helpers # +# --------------------------------------------------------------------- # + + +# Preferred GPU zones for L4 capacity walk. The list starts from the +# reviewed regional priority order and includes capacity-advised sibling +# zones observed in live Compute Engine stockout responses, so the +# multi-zone walker is not limited to stale static availability notes. +# Refresh this in lock-step with the knowledge file when operators update +# priorities. +PREFERRED_ZONES: tuple[str, ...] = ( + "us-central1-a", + "us-central1-b", + "us-central1-c", + "us-east4-a", + "us-east4-b", + "us-east4-c", + # us-east1 has L4 capacity in c + d only; us-east1-b deliberately + # omitted (no L4). + "us-east1-c", + "us-east1-d", + "us-west1-a", + "us-west1-b", + "us-west4-a", + "us-west4-b", + "europe-west4-a", + "europe-west4-b", + "europe-west1-b", + "europe-west1-c", + "asia-southeast1-a", + "asia-southeast1-b", + "asia-southeast1-c", + "asia-northeast1-a", + "asia-northeast1-c", + "asia-east1-a", + "asia-east1-b", + "asia-east1-c", +) + + +_ZONE_RE = re.compile(r"^[a-z]+-[a-z]+[0-9]+-[a-z]+$") + + +# Compute Engine GPU machine-type families that REJECT the default +# `Scheduling.on_host_maintenance=MIGRATE` and require TERMINATE + +# automatic_restart. This Scheduling override is GPU-only — non-GPU +# types must keep the API +# default to preserve live-migrate behavior. The prefix list covers the +# documented GPU families (g2 = L4, a2 = A100, a3 = H100/H200, n1 with +# attached guest accelerators); broader matching belongs in the +# knowledge file, not in the source. +_GPU_MACHINE_PREFIXES: tuple[str, ...] = ("g2-", "a2-", "a3-") + + +def is_gpu_machine_type(machine_type: str) -> bool: + """Return True iff the machine type is a known GPU-bearing family.""" + return any(machine_type.startswith(p) for p in _GPU_MACHINE_PREFIXES) + + +def _is_full_zone(value: str) -> bool: + """Return True iff ``value`` is a full zone (region prefix + letter).""" + return bool(_ZONE_RE.match(value)) + + +_REGION_LOOKUP_RETRY_TRANSIENT: tuple[type[BaseException], ...] = ( + gax.ServiceUnavailable, + gax.DeadlineExceeded, + gax.Aborted, + gax.ResourceExhausted, + gax.InternalServerError, +) + + +def _list_region_zones( + project: str, + region: str, + *, + attempts: int = 3, + backoff: float = 1.0, +) -> list[str]: + """Return the live zones the GCP API reports for ``region``. + + The static ``PREFERRED_ZONES`` list omits regions the suite has + not pinned for L4 capacity; the helper consults the API so a + valid region without + a preferred-list entry still walks its OWN zones first instead of + silently leaving the region for the preferred-list head. + + For ``zone_capacity_handling`` the helper MUST distinguish: + + * Transient lookup errors (HTTP 5xx / 429 / quota) — retry with + backoff, then return ``[]`` so the caller can fall back to the + offline prefix-match path against ``PREFERRED_ZONES``. + * Terminal errors (``NotFound`` / ``PermissionDenied`` / + ``InvalidArgument`` / ``Unauthenticated``) — raise structured so + the operator's region request is NEVER silently substituted by a + different region's zones. + """ + last_transient: BaseException | None = None + for attempt in range(1, attempts + 1): + try: + client = compute_v1.RegionsClient() + region_obj = client.get(project=project, region=region) + return [url.rsplit("/", 1)[-1] for url in region_obj.zones or ()] + except _REGION_LOOKUP_RETRY_TRANSIENT as e: + last_transient = e + print( + f" region zones lookup transient ({type(e).__name__}); " + f"attempt {attempt}/{attempts}", + file=sys.stderr, + ) + time.sleep(backoff * attempt) + continue + except ( + gax.NotFound, + gax.PermissionDenied, + gax.InvalidArgument, + gax.Unauthenticated, + ) as e: + msg = f"region '{region}' is invalid or unauthorized: {e}" + raise RuntimeError(msg) from e + except gax.GoogleAPICallError as e: + # Other GCP API call errors are treated as terminal — never + # silently substitute a different region's zones. + msg = f"region '{region}' lookup failed: {e}" + raise RuntimeError(msg) from e + if last_transient is not None: + print( + f" region zones lookup exhausted after {attempts} attempts: " + f"{last_transient}", + file=sys.stderr, + ) + return [] + + +def select_zones( + region_or_zone: str | None, + *, + project: str | None = None, + zone_walk: bool = True, +) -> list[str]: + """Return the ordered list of candidate zones for an ``instances.insert`` walk. + + Zone-capacity handling: + + * A FULL zone string (e.g., ``us-central1-f``) is a single-zone + pin: honor it as the ONLY candidate, no walk fallback. + * A region prefix (e.g., ``us-central1``) is expanded via the GCP + RegionsClient. Intersect with ``PREFERRED_ZONES``: if the + intersection is non-empty, walk those zones in preferred-list + order, then fall back cross-region to the rest of + ``PREFERRED_ZONES``. ONLY when the region has no preferred + zone live (e.g., region missing from the curated capacity list) + do we fall back to the region's other live zones — preserves + operator's reviewed capacity contract while keeping an + offline-region escape hatch. + * An empty value falls back to the full ``PREFERRED_ZONES``. + + Setting ``zone_walk=False`` returns at most one candidate so + callers can disable the walk locally (mirrors the AWS oracle's + no-fallback behavior for unit tests). + """ + if not region_or_zone: + return list(PREFERRED_ZONES) if zone_walk else list(PREFERRED_ZONES[:1]) + + if _is_full_zone(region_or_zone): + return [region_or_zone] + + # Region prefix: consult the API so a region missing from the + # preferred list still walks its OWN zones before cross-region. + region_zones: list[str] = [] + if project: + region_zones = _list_region_zones(project, region_or_zone) + preferred_in_region = [z for z in PREFERRED_ZONES if z in region_zones] + other_in_region = [z for z in region_zones if z not in preferred_in_region] + # Fall back to prefix match against the preferred list when the API + # lookup is unavailable so single-zone pins outside the list still + # work in offline environments. + if not region_zones: + preferred_in_region = [z for z in PREFERRED_ZONES if z.startswith(f"{region_or_zone}-")] + cross_region = [ + z + for z in PREFERRED_ZONES + if z not in preferred_in_region and z not in other_in_region + ] + # Zone-capacity contract: intersect with PREFERRED_ZONES when + # non-empty; only fall back to nonpreferred live zones when the + # intersection is empty (region missing from the curated capacity + # list). + if preferred_in_region: + candidates = preferred_in_region + cross_region + else: + candidates = other_in_region + cross_region + if not zone_walk: + candidates = candidates[:1] + return candidates or list(PREFERRED_ZONES) + + +# Compute Engine wire wordings for zone-unavailable errors. The +# classifier covers all four observed shapes (sync stockout, async DONE +# with errors, machine-type-not-in-zone, polling-fallback RuntimeError). +_ZONE_TOKENS_CASE_SENSITIVE = ("ZONE_RESOURCE",) +_ZONE_TOKENS_CASE_INSENSITIVE = ( + "stockout", + "does not have enough resources", +) + + +def is_zone_unavailable(err: Exception, op: Any = None) -> bool: + """Return True iff ``err`` (or the optional ``op``) signals zone-unavailable. + + Four shapes: + + 1. ``ResourceExhausted`` / 503 sync. + 2. Async op DONE with ZONE_RESOURCE / STOCKOUT in + ``op.error.errors[].code``. + 3. HTTP 400 ``machineType ... does not exist in zone``. + 4. ``RuntimeError`` from the wait helper joining + ``code:message`` — matches ZONE_RESOURCE, STOCKOUT, or the + human-readable "does not have enough resources" sentence. + """ + if isinstance(err, gax.ResourceExhausted): + return True + msg = str(err) + if "does not exist in zone" in msg and "machineType" in msg: + return True + if isinstance(err, RuntimeError): + if any(tok in msg for tok in _ZONE_TOKENS_CASE_SENSITIVE): + return True + msg_upper = msg.upper() + if any(tok.upper() in msg_upper for tok in _ZONE_TOKENS_CASE_INSENSITIVE): + return True + if op is not None and getattr(op, "error", None): + for e in op.error.errors: + code = (getattr(e, "code", "") or "").upper() + if "ZONE_RESOURCE" in code or "STOCKOUT" in code: + return True + return False + + +def delete_failed_zonal_instance(project: str, zone: str, name: str) -> bool: + """Best-effort delete of a partial async-insert in a failed zone. + + Compute Engine's async DONE-with-errors shape leaves a phantom + instance record in the failed zone. The multi-zone walker MUST + call this between zones (``zone_capacity_handling`` shape 2). + Returns True iff the cleanup + completed or the record was already absent. + """ + try: + op = compute_v1.InstancesClient().delete(project=project, zone=zone, instance=name) + except gax.NotFound: + return True + except gax.GoogleAPICallError as e: + print(f" warn: failed-zone cleanup raised: {e}", file=sys.stderr) + return False + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + try: + wait_for_zonal_op(project, zone, op_name, timeout=120) + except Exception as e: + print(f" warn: failed-zone cleanup wait raised: {e}", file=sys.stderr) + return False + return True + + +# --------------------------------------------------------------------- # +# Zone / region helpers (region<->zone narrowing) # +# --------------------------------------------------------------------- # + + +def zone_to_region(zone: str) -> str: + """Strip the trailing zone letter from a zone (us-central1-a -> us-central1).""" + parts = zone.rsplit("-", 1) + if len(parts) == 2 and len(parts[1]) == 1 and parts[1].isalpha(): + return parts[0] + return zone + + +def narrow_region_to_zone(region_or_zone: str, default_zone_letter: str = "a") -> str: + """Return a zone from either a region or a zone input. + + Compute Engine instance APIs are zone-scoped, but the suite contract + passes a single ``--region`` argument. Treat already-zone inputs + (trailing single-letter suffix) as pinned; otherwise append the default + suffix. The provider config wires ``--zone`` separately so explicit + pins always win over this fallback. + """ + parts = region_or_zone.rsplit("-", 1) + if len(parts) == 2 and len(parts[1]) == 1 and parts[1].isalpha(): + return region_or_zone + return f"{region_or_zone}-{default_zone_letter}" + + +# --------------------------------------------------------------------- # +# State translation (enumerated emitted set) # +# --------------------------------------------------------------------- # + + +# Mapping from Compute Engine ``Instance.status`` enum values to the +# canonical lifecycle vocabulary used by the suite validators. The emitted +# set is enumerated here so downstream code only branches on documented +# values. +# +# Emitted set: "running", "stopped", "starting", "stopping", "unknown". +# +# TERMINATED in Compute Engine means "stopped" (not deleted); REPAIRING +# is mapped to "unknown" because the guest is unreachable during host- +# failure recovery and treating it as running would weaken oracle parity. +# DEPROVISIONING (the post-delete transient) maps to "stopping" so the +# canonical "if state in ('stopping',)" branches downstream see it. +_RAW_TO_CANONICAL: dict[str, str] = { + "PROVISIONING": "starting", + "STAGING": "starting", + "RUNNING": "running", + "STOPPING": "stopping", + "STOPPED": "stopped", + "SUSPENDING": "stopping", + "SUSPENDED": "stopped", + "TERMINATED": "stopped", + "REPAIRING": "unknown", + "DEPROVISIONING": "stopping", +} + + +def canonical_state(raw: str | None) -> str: + """Translate a Compute Engine raw status to the canonical lifecycle vocabulary. + + Emitted set: ``running``, ``stopped``, ``starting``, ``stopping``, + ``unknown``. Downstream code MUST only branch on these. + """ + if not raw: + return "unknown" + return _RAW_TO_CANONICAL.get(raw.upper(), "unknown") + + +# --------------------------------------------------------------------- # +# Operation waiters # +# --------------------------------------------------------------------- # + + +def wait_for_zonal_op(project: str, zone: str, operation_name: str, *, timeout: int = 600) -> compute_v1.Operation: + """Block until a zonal Compute Operation reaches DONE. + + Raises ``RuntimeError`` if the operation's error list is non-empty. + The joined message includes ``op.error.errors[].code`` so the + multi-zone walk classifier can match canonical STOCKOUT tokens + (zone_capacity_handling shape 4). + """ + client = compute_v1.ZoneOperationsClient() + deadline = time.monotonic() + timeout + while True: + op = client.get(project=project, zone=zone, operation=operation_name) + if op.status == compute_v1.Operation.Status.DONE: + if op.error and op.error.errors: + msg = "; ".join(f"{getattr(e, 'code', '')}:{getattr(e, 'message', str(e))}" for e in op.error.errors) + raise RuntimeError(f"Zonal op {operation_name} failed: {msg}") + return op + if time.monotonic() >= deadline: + raise TimeoutError(f"Zonal operation {operation_name} did not complete in {timeout}s") + time.sleep(3) + + +def retry_zonal_lifecycle_op( + op_fn: Callable[[], Any], + project: str, + zone: str, + *, + resource_desc: str, + on_sync_success: Callable[[], None] | None = None, + attempts: int = 3, + backoffs: tuple[int, ...] = (60, 120), + op_timeout: int = 600, +) -> compute_v1.Operation | None: + """Run a zonal lifecycle op with stockout retry-in-place. + + Lifecycle ops (``instances.start`` / ``stop`` / ``reset``) are + zone-bound — they + cannot walk to a different zone on STOCKOUT. The only recovery is + retry-with-backoff in the same zone, 3 attempts max with 60s / 120s + backoffs. + + ``op_fn`` must perform the synchronous API call and return the + ``Operation``. ``on_sync_success`` (if supplied) fires AFTER each + synchronous return but BEFORE the async wait — callers stamp their + ``_initiated`` tracker there so the idempotent-lifecycle + invariant holds even across retries. + + Stockout shapes covered: + * Synchronous ``ResourceExhausted`` raise (shape 1). + * Async DONE-with-errors observed by ``wait_for_zonal_op`` and + re-raised as ``RuntimeError`` carrying STOCKOUT / ZONE_RESOURCE + tokens (shapes 2 / 4). + + Non-stockout exceptions re-raise immediately so transient API errors + do not waste two backoffs. + """ + last_err: Exception | None = None + for attempt_idx in range(attempts): + op: Any = None + try: + op = op_fn() + if on_sync_success is not None: + on_sync_success() + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + return wait_for_zonal_op(project, zone, op_name, timeout=op_timeout) + return None + except Exception as e: + last_err = e + if not is_zone_unavailable(e, op=op): + raise + if attempt_idx >= attempts - 1: + break + wait = backoffs[min(attempt_idx, len(backoffs) - 1)] + print( + f" stockout on {resource_desc} attempt {attempt_idx + 1}/{attempts}; sleeping {wait}s before retry", + file=sys.stderr, + ) + time.sleep(wait) + raise RuntimeError( + f"Stockout retry exhausted for {resource_desc} after {attempts} attempts: {last_err}" + ) from last_err + + +def wait_for_global_op(project: str, operation_name: str, *, timeout: int = 600) -> compute_v1.Operation: + """Block until a global Compute Operation reaches DONE.""" + client = compute_v1.GlobalOperationsClient() + deadline = time.monotonic() + timeout + while True: + op = client.get(project=project, operation=operation_name) + if op.status == compute_v1.Operation.Status.DONE: + if op.error and op.error.errors: + msg = "; ".join(f"{getattr(e, 'code', '')}:{getattr(e, 'message', str(e))}" for e in op.error.errors) + raise RuntimeError(f"Global op {operation_name} failed: {msg}") + return op + if time.monotonic() >= deadline: + raise TimeoutError(f"Global operation {operation_name} did not complete in {timeout}s") + time.sleep(3) + + +# --------------------------------------------------------------------- # +# Instance polling # +# --------------------------------------------------------------------- # + + +def get_instance(project: str, zone: str, name: str) -> compute_v1.Instance: + """Wrapper around InstancesClient().get for convenience.""" + return compute_v1.InstancesClient().get(project=project, zone=zone, instance=name) + + +def poll_instance_state( + project: str, + zone: str, + name: str, + *, + target_canonical: str, + timeout: int = 300, + interval: int = 5, +) -> str: + """Poll instances.get until canonical_state(status) == target_canonical. + + Returns the final canonical state; raises ``TimeoutError`` on budget + exhaustion so the caller can record a structured timeout instead of + silently treating a never-reached state as success. + """ + deadline = time.monotonic() + timeout + while True: + inst = get_instance(project, zone, name) + cstate = canonical_state(inst.status) + if cstate == target_canonical: + return cstate + if time.monotonic() >= deadline: + raise TimeoutError(f"Instance {name} did not reach {target_canonical!r} (last={cstate!r}) in {timeout}s") + time.sleep(interval) + + +def first_external_ip(instance: compute_v1.Instance) -> str | None: + """Return ``networkInterfaces[0].accessConfigs[0].natIP`` if present.""" + for nic in instance.network_interfaces: + for cfg in nic.access_configs: + ip = getattr(cfg, "nat_i_p", None) or getattr(cfg, "nat_ip", None) + if ip: + return ip + return None + + +def first_internal_ip(instance: compute_v1.Instance) -> str | None: + if not instance.network_interfaces: + return None + return instance.network_interfaces[0].network_i_p or None + + +def wait_for_public_ip( + project: str, + zone: str, + name: str, + *, + timeout: int = 120, + interval: int = 5, +) -> str | None: + """Poll instances.get until an external IP is observable. + + Ephemeral external IPs are released on stop in Compute Engine, so + post-start / post-reset code MUST re-read rather than rely on a + cached arg. Terminal classes (NotFound, Unauthenticated, + PermissionDenied) are re-raised so a wrong instance / wrong zone / + bad credentials surfaces as a structured error rather than a silent + timeout. + """ + deadline = time.monotonic() + timeout + while True: + try: + inst = get_instance(project, zone, name) + ip = first_external_ip(inst) + if ip: + return ip + except (gax.ServiceUnavailable, gax.InternalServerError, gax.GatewayTimeout, gax.DeadlineExceeded) as e: + print(f" warn: wait_for_public_ip transient: {e}", file=sys.stderr) + if time.monotonic() >= deadline: + return None + time.sleep(interval) + + +def short_name(self_link: str | None) -> str: + """Return the trailing path segment from a Compute Engine self-link. + + Use this when verifying scope-binding fields by exact match. + Substring / startswith comparisons accept supersets and adopt + resources from the wrong scope. + """ + if not self_link: + return "" + return self_link.rsplit("/", 1)[-1] + + +def resolve_image( + image_project: str, + image_arg: str, +) -> compute_v1.Image: + """Resolve an operator-supplied image short-name or family to a concrete Image. + + The operator's ``args.project`` (forwarded as ``image_project`` + here) MUST be the FIRST lookup scope — + vendor-default constants are only an explicit fallback. The resolver + tries: + 1. ``images.get(project=image_project, image=image_arg)`` (exact name) + 2. ``images.get_from_family(project=image_project, family=image_arg)`` + (family alias) + Raising the original ``NotFound`` keeps the failure inspectable when + nothing matches. + """ + client = compute_v1.ImagesClient() + try: + return client.get(project=image_project, image=image_arg) + except gax.NotFound: + return client.get_from_family(project=image_project, family=image_arg) + + +# --------------------------------------------------------------------- # +# Tag/label projection # +# --------------------------------------------------------------------- # + + +# Compute Engine label keys must match [a-z]([-a-z0-9_]*). The suite +# expects canonical mixed-case Name / CreatedBy keys; we project to +# api-valid forms on create and back on read so InstanceTagCheck stays +# unchanged. +_TAG_TO_LABEL: dict[str, str] = { + "Name": "isv_name", + "CreatedBy": "createdby", +} +_LABEL_TO_TAG: dict[str, str] = {v: k for k, v in _TAG_TO_LABEL.items()} + + +def canonical_tags_to_labels(tags: dict[str, str]) -> dict[str, str]: + """Project canonical Name/CreatedBy tag keys to api-valid Compute Engine labels.""" + out: dict[str, str] = {} + for k, v in tags.items(): + label_key = _TAG_TO_LABEL.get(k, k.lower()) + label_val = re.sub(r"[^a-z0-9_-]", "-", (v or "").lower())[:63] + out[label_key] = label_val + return out + + +def labels_to_canonical_tags(labels: dict[str, str] | None) -> dict[str, str]: + """Project Compute Engine labels back to canonical suite tag names.""" + if not labels: + return {} + out: dict[str, str] = {} + for k, v in labels.items(): + out[_LABEL_TO_TAG.get(k, k)] = v + return out + + +# --------------------------------------------------------------------- # +# Local SSH key pair (verified-reuse, returns created bool) # +# --------------------------------------------------------------------- # + + +_KEY_NAME_RE = re.compile(r"[A-Za-z0-9_.-]{1,255}") + + +def sanitize_key_name(key_name: str) -> str: + """Reject key names that could escape /tmp when composed into a path.""" + if not key_name or not _KEY_NAME_RE.fullmatch(key_name): + raise ValueError(f"invalid key name {key_name!r}: must match [A-Za-z0-9_.-] (1-255 chars).") + return key_name + + +def read_ssh_pubkey(priv_path: str) -> str: + """Read the OpenSSH public-key line that pairs with ``priv_path``. + + ``ssh-keygen`` derives ``.pub`` from the FULL private path, so a + private key at ``isv-test-key.pem`` produces ``isv-test-key.pem.pub``. + The pairing lives in one place — local artifacts count as resources too. + """ + return Path(f"{priv_path}.pub").read_text().strip() + + +def generate_ssh_keypair( + key_name: str, + key_dir: str | Path | None = None, +) -> tuple[str, bool]: + """Generate or verified-reuse a local OpenSSH key pair. + + Compute Engine has no managed key-pair store; the SSH public key is + attached via instance metadata. The local PEM + ``.pub`` pair is the + artifact that survives the run, so it must follow the verified-reuse + cleanup contract: + + * Returns ``(private_key_path, created)``. + * ``created`` is True only when this call generated a fresh pair. + * An adopted pair (both files non-empty) returns ``created=False`` so + teardown skips local deletion. + """ + sanitize_key_name(key_name) + base = Path(key_dir) if key_dir else Path("/tmp") + base.mkdir(parents=True, exist_ok=True) + priv = base / f"{key_name}.pem" + pub = base / f"{key_name}.pem.pub" + + if priv.exists() and pub.exists() and priv.stat().st_size > 0 and pub.stat().st_size > 0: + print(f" reusing existing local SSH key pair: {priv}", file=sys.stderr) + return str(priv), False + + # Wipe partial state and regenerate. + for p in (priv, pub): + if p.exists(): + try: + p.chmod(0o600) + except OSError: + pass + p.unlink(missing_ok=True) + + subprocess.run( + ["ssh-keygen", "-t", "rsa", "-b", "2048", "-N", "", "-q", "-f", str(priv)], + check=True, + ) + priv.chmod(0o400) + print(f" generated SSH key pair: {priv}", file=sys.stderr) + return str(priv), True + + +def delete_local_keypair(priv_path: str) -> bool: + """Remove the local PEM + ``.pub`` pair. Returns True iff both files are gone.""" + ok = True + for p in (Path(priv_path), Path(priv_path + ".pub")): + try: + if p.exists(): + p.chmod(0o600) + p.unlink() + except OSError as e: + print(f" warn: could not remove {p}: {e}", file=sys.stderr) + ok = False + return ok + + +# --------------------------------------------------------------------- # +# Firewall (verified-reuse, returns created bool) # +# --------------------------------------------------------------------- # + + +# Compute Engine firewall rules do NOT accept a labels field on the proto. +# The closest analog to the AWS oracle's CreatedBy=isvtest tag-based +# check is to embed +# the ownership marker in the rule's description and require an exact +# match on adopt. +_ISV_OWNERSHIP_MARKER = "createdby=isvtest" +_ISV_FIREWALL_DESCRIPTION = f"ISV validation SSH firewall rule ({_ISV_OWNERSHIP_MARKER})" +ISV_NETWORK_TAG = "isv-test-vm" + + +def _firewall_matches_ssh_shape(rule: compute_v1.Firewall, network_short: str) -> bool: + """Return True iff the existing firewall rule matches the SSH-allow shape. + + Rule #7 invariant: every caller-depended property must be verified on + reuse-adoption. A rule with the ownership marker + description + port + shape but ``disabled=True`` would be silently adopted, then SSH would + time out at cloud-init wait and teardown would skip the rule (because + ``firewall_created=False``). Reject disabled rules up front. + """ + if rule.disabled: + return False + if short_name(rule.network) != network_short: + return False + if rule.direction != "INGRESS": + return False + # Rule #7 requires every caller-depended invariant be verified on + # reuse-adoption. Use set-equality (not membership) so a rule with + # extra CIDRs (`["0.0.0.0/0", "10.0.0.0/8"]`) or extra target tags + # is REJECTED rather than silently adopted with `firewall_created= + # False` — adopting a superset rule weakens the test contract AND + # persists post-run because teardown gates on the `firewall_created` + # flag. + if set(rule.source_ranges) != {"0.0.0.0/0"}: + return False + if set(rule.target_tags) != {ISV_NETWORK_TAG}: + return False + # Require exactly one allowed entry: tcp/22. Multiple entries or + # additional ports broaden the ingress beyond the caller-declared + # scope and must be rejected. + if len(rule.allowed) != 1: + return False + allowed = rule.allowed[0] + if allowed.I_p_protocol.lower() != "tcp": + return False + if list(allowed.ports) != ["22"]: + return False + return True + + +def _firewall_has_isv_ownership(rule: compute_v1.Firewall) -> bool: + return _ISV_OWNERSHIP_MARKER in (rule.description or "").lower() + + +def insert_ssh_firewall( + project: str, + name: str, + network_short: str, +) -> tuple[str, Any]: + """Submit a verified-reuse SSH firewall insert and return ``(name, op)``. + + Stamp-before-wait split of the previous ``ensure_ssh_firewall`` + (cleanup-tracker sub-rule): the caller stamps + ``firewall_created=True`` IMMEDIATELY after this + function returns ``op != None``, BEFORE running ``wait_for_global_op``. + A post-insert wait failure then leaves the caller with the truthful + ``firewall_created=True`` so cleanup-on-failure deletes the + accepted-but-uncomfirmed rule. + + Returns ``(name, op)``. ``op`` is ``None`` iff the call adopted a + verified-reuse existing rule (no wait required); otherwise the + caller MUST block on ``wait_for_global_op(project, op.name, ...)``. + + Adoption is gated on the same three checks as before: ownership + marker present, description matches what we'd produce, shape + matches (network/direction/source/target tag/tcp22). + """ + fw_client = compute_v1.FirewallsClient() + network_url = f"projects/{project}/global/networks/{network_short}" + + rule = compute_v1.Firewall() + rule.name = name + rule.network = network_url + rule.direction = "INGRESS" + rule.priority = 1000 + rule.source_ranges = ["0.0.0.0/0"] + rule.target_tags = [ISV_NETWORK_TAG] + rule.description = _ISV_FIREWALL_DESCRIPTION + + allowed = compute_v1.Allowed() + allowed.I_p_protocol = "tcp" + allowed.ports = ["22"] + rule.allowed = [allowed] + + try: + op = fw_client.insert(project=project, firewall_resource=rule) + except gax.Conflict: + existing = fw_client.get(project=project, firewall=name) + if not _firewall_has_isv_ownership(existing): + raise RuntimeError( + f"firewall {name!r} exists in {project} without ownership marker " + f"{_ISV_OWNERSHIP_MARKER!r}; refusing to adopt" + ) from None + if (existing.description or "") != _ISV_FIREWALL_DESCRIPTION: + raise RuntimeError( + f"firewall {name!r} description differs: expected " + f"{_ISV_FIREWALL_DESCRIPTION!r}, got {existing.description!r}" + ) from None + if not _firewall_matches_ssh_shape(existing, network_short): + raise RuntimeError( + f"firewall {name!r} exists but shape (network/direction/source/tags/tcp22) " + "does not match; refusing to adopt" + ) from None + print(f" reusing verified firewall rule: {name}", file=sys.stderr) + return name, None + + print(f" inserted firewall rule {name} (op pending)", file=sys.stderr) + return name, op diff --git a/isvctl/configs/providers/gcp/scripts/common/errors.py b/isvctl/configs/providers/gcp/scripts/common/errors.py new file mode 100644 index 00000000..b0d9454f --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/common/errors.py @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""GCP error classification + retry helpers for VM stubs. + +Mirrors providers/aws/scripts/common/errors.py so the structured output +shape stays identical across providers: + {"success": false, "error_type": "", "error": ""} + +Categories match the AWS oracle: + credentials_missing, credentials_invalid, access_denied, + api_error, unknown_error. +""" + +from __future__ import annotations + +import functools +import json +import logging +import time +from collections.abc import Callable +from typing import Any + +from google.api_core import exceptions as gax +from google.auth import exceptions as auth_exceptions + +logger = logging.getLogger(__name__) + +ALREADY_GONE_EXCEPTIONS: tuple[type[Exception], ...] = (gax.NotFound,) + +TRANSIENT_EXCEPTIONS: tuple[type[Exception], ...] = ( + gax.ServiceUnavailable, + gax.InternalServerError, + gax.GatewayTimeout, + gax.DeadlineExceeded, + gax.TooManyRequests, + gax.Aborted, + gax.RetryError, +) + + +def classify_gcp_error(e: Exception) -> tuple[str, str]: + """Translate a GCP exception into (error_type, message). + + Categories mirror providers/aws/scripts/common/errors.classify_aws_error. + Order: credentials_missing (no ADC at all) before credentials_invalid + (ADC present but rejected) so a missing-setup operator gets the + setup-pointing message rather than the "invalid credentials" one. + """ + if isinstance(e, auth_exceptions.DefaultCredentialsError): + return "credentials_missing", f"GCP credentials missing or not configured: {e}" + if isinstance(e, gax.Unauthenticated): + return "credentials_invalid", f"GCP credentials invalid or missing: {e}" + if isinstance(e, gax.PermissionDenied): + return "access_denied", f"Access denied: {e}" + if isinstance(e, gax.NotFound): + return "api_error", str(e) + if isinstance(e, gax.GoogleAPICallError): + return "api_error", str(e) + if isinstance(e, gax.RetryError): + return "api_error", str(e) + return "unknown_error", str(e) + + +def delete_with_retry( + fn: Callable[..., Any], + *args: Any, + resource_desc: str = "resource", + attempts: int = 3, + backoff_seconds: float = 2.0, + **kwargs: Any, +) -> bool: + """Call ``fn`` with bounded retry on transient GCP errors. + + Never raises. Returns True iff the call succeeded or the resource was + already gone (NotFound counts as success — the desired terminal state + is reached). Mirrors providers/aws/scripts/common/errors.delete_with_retry + so callers can write provider-portable cleanup blocks. + + The bool return MUST be consumed by the caller and AND-ed into the + teardown result — helpers that return ``bool`` for batch-cleanup + safety MUST surface the bool into ``result['success']``. + """ + last_error: Exception | None = None + for attempt in range(1, attempts + 1): + try: + fn(*args, **kwargs) + return True + except ALREADY_GONE_EXCEPTIONS: + return True + except TRANSIENT_EXCEPTIONS as e: + if attempt < attempts: + last_error = e + delay = backoff_seconds * attempt + logger.warning( + "Transient error deleting %s (attempt %d/%d): %s; retrying in %.1fs", + resource_desc, attempt, attempts, e, delay, + ) + time.sleep(delay) + continue + logger.exception("Failed to delete %s after %d attempts", resource_desc, attempts) + return False + except gax.GoogleAPICallError: + logger.exception("Non-transient API error deleting %s", resource_desc) + return False + except Exception: + logger.exception("Unexpected error deleting %s", resource_desc) + return False + + if last_error is not None: + logger.error("Exhausted retries deleting %s: %s", resource_desc, last_error) + return False + + +def handle_gcp_errors[**P](func: Callable[P, int]) -> Callable[P, int]: + """Decorator that catches uncaught GCP errors and emits structured JSON. + + Mirrors providers/aws/scripts/common/errors.handle_aws_errors. Scripts + still print their own JSON and return 0/1; this decorator only handles + exceptions that escape main (e.g. client construction). + """ + + @functools.wraps(func) + def wrapper(*args: P.args, **kwargs: P.kwargs) -> int: + try: + return func(*args, **kwargs) + except Exception as e: + error_type, error_msg = classify_gcp_error(e) + print(json.dumps( + {"success": False, "error_type": error_type, "error": error_msg}, + indent=2, + )) + return 1 + + return wrapper diff --git a/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py b/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py new file mode 100644 index 00000000..e93c21ce --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py @@ -0,0 +1,283 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""SSH helpers for GCP VM stubs. + +Defensive option set per the sister-stub consistency rule: every SSH +call site in the GCP target must +use the same flag set, including ``IdentitiesOnly=yes`` + +``PasswordAuthentication=no`` so operator environments with multiple +loaded agent keys don't exhaust ``MaxAuthTries`` before the ``-i`` key +gets tried. +""" + +from __future__ import annotations + +import shlex +import subprocess +import sys +import time + +# Canonical SSH options. Keep the set identical across every stub that +# runs SSH from the orchestrator host (sister-stub consistency rule). +# `IdentitiesOnly=yes` + `IdentityAgent=none` together ensure the explicit +# `-i ` argument is the ONLY credential SSH considers, even when +# the operator has an ssh-agent running with multiple identities. Without +# the agent disable, agent-offered keys are tried first and exhaust +# MaxAuthTries before the `-i` key is reached, producing spurious auth +# failures. Mirrors the AWS oracle's ssh_utils canonical options. +_SSH_OPTS: tuple[str, ...] = ( + "-o", + "StrictHostKeyChecking=no", + "-o", + "UserKnownHostsFile=/dev/null", + "-o", + "IdentitiesOnly=yes", + "-o", + "IdentityAgent=none", + "-o", + "PasswordAuthentication=no", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=5", +) + + +def _ssh_argv(host: str, user: str, key_file: str, remote_cmd: str) -> list[str]: + return ["ssh", *_SSH_OPTS, "-i", key_file, f"{user}@{host}", remote_cmd] + + +def _try_ssh(host: str, user: str, key_file: str, remote_cmd: str = "exit 0") -> bool: + """Single SSH probe. Returns True on rc=0; False on rc!=0 / OSError / timeout.""" + try: + result = subprocess.run( + _ssh_argv(host, user, key_file, remote_cmd), + capture_output=True, + timeout=15, + ) + return result.returncode == 0 + except (subprocess.TimeoutExpired, OSError): + return False + + +def ssh_run( + host: str, + user: str, + key_file: str, + command: str, + *, + timeout: int = 30, + connect_timeout: int = 10, +) -> tuple[int, str, str]: + """Run a single command over SSH. Returns ``(exit_code, stdout, stderr)``. + + Sister to the AWS oracle's ``common.ssh_utils.ssh_run`` — same return + contract so guest probes can be reused verbatim. Errors map to fixed + sentinel exit codes (124 for timeout, 255 for OSError) instead of + raising, mirroring the AWS helper. + """ + opts = list(_SSH_OPTS) + # Override the canonical ConnectTimeout with the caller-supplied value. + try: + idx = opts.index("ConnectTimeout=5") + opts[idx] = f"ConnectTimeout={connect_timeout}" + except ValueError: + opts.extend(["-o", f"ConnectTimeout={connect_timeout}"]) + try: + proc = subprocess.run( + ["ssh", *opts, "-i", key_file, f"{user}@{host}", "--", command], + capture_output=True, + timeout=timeout, + text=True, + check=False, + ) + except subprocess.TimeoutExpired as err: + return 124, "", f"TimeoutExpired: {err}" + except OSError as err: + return 255, "", f"OSError: {err}" + return proc.returncode, proc.stdout, proc.stderr + + +def wait_for_ssh( + host: str, + user: str, + key_file: str, + max_attempts: int = 30, + interval: int = 10, +) -> bool: + """Poll until SSH accepts a connection. Mirrors AWS oracle's wait_for_ssh. + + Single-success gate — NOT sufficient as a post-lifecycle stability + gate (see ``wait_for_ssh_stable``). + """ + for attempt in range(1, max_attempts + 1): + if _try_ssh(host, user, key_file): + print(f" SSH ready after attempt {attempt}", file=sys.stderr) + return True + print(f" waiting for SSH... (attempt {attempt}/{max_attempts})", file=sys.stderr) + time.sleep(interval) + return False + + +def wait_for_ssh_drop( + host: str, + user: str, + key_file: str, + max_attempts: int = 18, + interval: int = 5, +) -> bool: + """Poll for SSH connection FAILURE. + + Used after ``instances.reset`` to confirm the pre-reboot sshd has + dropped before the post-reboot stability gate. On async soft-reboot + APIs, wait for SSH to DROP before waiting for it to stabilize. + Default budget ~90s. + """ + for attempt in range(1, max_attempts + 1): + if not _try_ssh(host, user, key_file): + print(f" SSH dropped after attempt {attempt}", file=sys.stderr) + return True + print(f" waiting for SSH to drop... (attempt {attempt}/{max_attempts})", file=sys.stderr) + time.sleep(interval) + return False + + +def wait_for_ssh_stable( + host: str, + user: str, + key_file: str, + consecutive: int = 3, + interval: int = 10, + max_attempts: int = 36, +) -> bool: + """Block until SSH responds ``consecutive`` times in a row. + + Compute Engine acks lifecycle calls (start, reset) before the guest + agent finishes; sshd may transiently restart and rewrite + authorized_keys mid-replay. Downstream validators racing the replay + flake without this gate. + """ + if consecutive < 1: + consecutive = 1 + + streak = 0 + for attempt in range(1, max_attempts + 1): + if _try_ssh(host, user, key_file): + streak += 1 + print(f" SSH probe {streak}/{consecutive} ok (attempt {attempt})", file=sys.stderr) + if streak >= consecutive: + return True + time.sleep(interval) + continue + if streak > 0: + print(f" SSH probe {streak + 1} failed; resetting streak", file=sys.stderr) + streak = 0 + time.sleep(interval) + return False + + +def wait_for_cloud_init( + host: str, + user: str, + key_file: str, + timeout_seconds: int = 600, + transport_backoff: int = 10, +) -> bool: + """Run ``cloud-init status --wait`` over SSH. + + Wait-command exit codes: 0 and warnings both mean done. rc 0 (clean) + AND rc 2 (recoverable warnings) BOTH count as terminal completion. + Only rc 1 (fatal) is failure. + + The shell wrapper captures the upstream rc into the stdout marker + ``CLOUDINIT_RC=N`` so we never consult ``r.returncode`` from the + ``|| echo ...`` chain (which would always be 0 because the echo + succeeded — shell-based wait helpers that capture non-zero exit + codes via ``|| echo`` must parse the echoed RC before consulting + ``r.returncode``). + + Distinguish transport-level failure (sshd refused / dropped + connection — no ``CLOUDINIT_RC`` marker ever lands) from semantic + failure (cloud-init returned rc=1). Transport failures retry within + the deadline (with a short backoff between probes); semantic + failures terminate immediately so the operator sees the cause. + """ + remote_script = "sudo cloud-init status --wait && echo CLOUDINIT_RC=0 || echo CLOUDINIT_RC=$?" + deadline = time.monotonic() + timeout_seconds + last_transport_rc: int | None = None + while True: + remaining = int(deadline - time.monotonic()) + if remaining <= 0: + print( + f" cloud-init wait deadline exhausted (last transport rc={last_transport_rc})", + file=sys.stderr, + ) + return False + try: + result = subprocess.run( + ["ssh", *_SSH_OPTS, "-i", key_file, f"{user}@{host}", remote_script], + capture_output=True, + text=True, + timeout=remaining, + ) + except subprocess.TimeoutExpired: + print(f" cloud-init wait timed out after {timeout_seconds}s", file=sys.stderr) + return False + except OSError as e: + print(f" cloud-init wait failed (OSError): {e}", file=sys.stderr) + return False + + rc_marker: int | None = None + for line in result.stdout.splitlines(): + line = line.strip() + if line.startswith("CLOUDINIT_RC="): + try: + rc_marker = int(line.split("=", 1)[1]) + except ValueError: + rc_marker = None + + if rc_marker is None: + # SSH never reached the echo — transport-level failure + # (connect drop, auth refusal, command-not-found). Retry + # within the deadline rather than terminate. + last_transport_rc = result.returncode + print( + f" cloud-init wait: no marker (ssh rc={result.returncode}); transport retry in {transport_backoff}s", + file=sys.stderr, + ) + time.sleep(transport_backoff) + continue + + if rc_marker in (0, 2): + print(f" cloud-init wait completed (rc={rc_marker})", file=sys.stderr) + return True + # Semantic failure (rc=1 or other non-zero) — terminate; the + # guest reached the echo, so it has a definite answer. + print(f" cloud-init wait reported fatal rc={rc_marker}", file=sys.stderr) + return False + + +def get_uptime_via_ssh(host: str, user: str, key_file: str) -> float | None: + """Return ``/proc/uptime`` seconds via SSH, or None on failure. + + Used for reboot affirmation (post-reset uptime should be substantially + less than pre-reset uptime — see ``reboot_instance.py``). + """ + try: + result = subprocess.run( + _ssh_argv(host, user, key_file, "cat /proc/uptime | cut -d' ' -f1"), + capture_output=True, + text=True, + timeout=30, + ) + if result.returncode == 0: + return float(result.stdout.strip()) + except (subprocess.TimeoutExpired, ValueError, OSError): + pass + return None + + +def quote(s: str) -> str: + """Shell-quote a string for inline composition in SSH commands.""" + return shlex.quote(s) diff --git a/isvctl/configs/providers/gcp/scripts/vm/console_rbac.py b/isvctl/configs/providers/gcp/scripts/vm/console_rbac.py new file mode 100644 index 00000000..9242c6f4 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/console_rbac.py @@ -0,0 +1,1003 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Console RBAC probe for Compute Engine serial console access. + +The suite contract requires three subtests: + 1. denied_principal_cannot_access_console + 2. allowed_principal_can_access_console + 3. allowed_principal_is_resource_scoped + +GCP has no AWS-style ``simulate_principal_policy`` equivalent for Compute +Engine serial console access (``instances.testIamPermissions`` and +``instances.getSerialPortOutput`` both evaluate the caller). Per the +reviewed GCP knowledge file, RBAC evidence must come from REAL probe +principals: + + * a denied service account WITHOUT + ``compute.instances.getSerialPortOutput`` on the target VM, + * an allowed service account WITH that permission scoped to the target + VM only, + * a real second VM where the allowed SA must still be denied. + +The default path is SELF-PROVISIONED: this stub creates two temporary +probe service accounts and a second probe VM, grants the caller +``roles/iam.serviceAccountTokenCreator`` on the probe SAs, grants the +allowed SA a minimal serial-output role scoped to the target VM only, +mints short-lived access tokens via the IAMCredentials REST +``generateAccessToken`` endpoint, and probes +``instances.getSerialPortOutput`` as each principal. Cleanup deletes the +temporary SAs and probe VM and removes the IAM bindings with +read-modify-write retry on etag conflicts. + +The pre-provisioned env-var path (``GCP_DENIED_PRINCIPAL_SA``, +``GCP_ALLOWED_PRINCIPAL_SA``, ``GCP_OTHER_INSTANCE_ID``) is a FALLBACK +for projects where the operator cannot allow IAM mutation. The fallback +is opt-in via the env vars themselves; otherwise the self-provisioned +path runs. + +Per the reviewed knowledge file (target_sdk_quirks #4 / #5): + * Direct IAMCredentials REST token minting is the stable path. Avoid + ``google.auth.impersonated_credentials.Credentials`` with local + authorized-user ADC — its refresh code can call a private + ``_refresh_token`` member that is a string on authorized-user + credentials, raising ``TypeError: 'str' object is not callable``. + * Resolve the ADC caller from tokeninfo when local user ADC has no + ``service_account_email`` and an empty ``account``. + +HTTP 404 on the second VM probe is treated as a FAILURE, not as proof of +RBAC scoping (the resource being missing means IAM enforcement could not +be observed). +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + narrow_region_to_zone, + resolve_project, + unique_suffix, + wait_for_zonal_op, +) +from common.errors import handle_gcp_errors + +# Serial console permission. The validator's ``restricted_actions`` field +# is populated with this exact permission name so downstream audits can +# correlate the IAM action with the API call. +_CONSOLE_PERMISSION = "compute.instances.getSerialPortOutput" +_RESTRICTED_ACTIONS = (_CONSOLE_PERMISSION,) + +# Compute Engine REST endpoints. +_COMPUTE_BASE = "https://compute.googleapis.com/compute/v1" +_IAM_BASE = "https://iam.googleapis.com/v1" +_IAM_CREDENTIALS_BASE = "https://iamcredentials.googleapis.com/v1" +_TOKENINFO_URL = "https://oauth2.googleapis.com/tokeninfo" + +# Default per-call timeout for raw HTTP calls (seconds). The enclosing +# step timeout is the real deadline; this is just a safeguard against +# the urllib request hanging on a single API call. +_HTTP_TIMEOUT_S = 30 + +# Minimal predefined role granting serial-port-output read. ``roles/ +# compute.viewer`` includes the permission and is broader than necessary; +# the knowledge file lists it as the acceptable predefined role for this +# probe ("``roles/compute.viewer`` or an equivalent minimal serial-output +# custom role"). A custom role would be tighter but requires extra +# provisioning the probe does not need. +_ALLOWED_TARGET_ROLE = "roles/compute.viewer" +_TOKEN_CREATOR_ROLE = "roles/iam.serviceAccountTokenCreator" + +# Self-provisioning is the DEFAULT path. The pre-provisioned fallback is +# opt-in by exporting all three env vars. +_DENIED_SA_ENV = "GCP_DENIED_PRINCIPAL_SA" +_ALLOWED_SA_ENV = "GCP_ALLOWED_PRINCIPAL_SA" +_OTHER_INSTANCE_ENV = "GCP_OTHER_INSTANCE_ID" +_OTHER_INSTANCE_ZONE_ENV = "GCP_OTHER_INSTANCE_ZONE" +# Operators that cannot grant IAM mutations to the caller can set this +# to ``"0"`` to force-skip the self-provisioned path even when no fallback +# env vars are supplied; otherwise the stub will try self-provisioning +# and surface an honest failure if any step is denied. +_SELF_PROVISION_ENABLED_ENV = "GCP_SELF_PROVISION_RBAC" + + +def _http_request( + method: str, + url: str, + token: str, + *, + body: dict[str, Any] | None = None, + extra_headers: dict[str, str] | None = None, +) -> tuple[int, dict[str, Any]]: + """Issue an HTTPS call with a bearer token; return ``(status, body)``. + + Body is the parsed JSON when the response is JSON, or ``{}`` when + the response is empty / non-JSON. Errors raise ``urllib.error.HTTPError`` + which the caller catches to read the status code. + """ + headers = { + "Authorization": f"Bearer {token}", + "Content-Type": "application/json", + "Accept": "application/json", + } + if extra_headers: + headers.update(extra_headers) + data = json.dumps(body).encode("utf-8") if body is not None else None + request = urllib.request.Request(url, method=method, data=data, headers=headers) + try: + with urllib.request.urlopen(request, timeout=_HTTP_TIMEOUT_S) as response: + status = response.getcode() or 0 + raw = response.read() + except urllib.error.HTTPError: + raise + parsed: dict[str, Any] = {} + if raw: + try: + parsed = json.loads(raw.decode("utf-8")) + except (ValueError, UnicodeDecodeError): + parsed = {"raw": raw.decode("utf-8", errors="replace")} + return status, parsed + + +def _http_error_body(error: urllib.error.HTTPError) -> dict[str, Any]: + """Best-effort extract JSON body from an HTTPError.""" + try: + raw = error.read() + except Exception: + return {} + if not raw: + return {} + try: + return json.loads(raw.decode("utf-8")) + except (ValueError, UnicodeDecodeError): + return {"raw": raw.decode("utf-8", errors="replace")} + + +def _adc_access_token() -> str: + """Refresh ADC and return the access token string.""" + import google.auth + import google.auth.transport.requests + from google.auth.credentials import Credentials + + raw_creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) + creds: Credentials = raw_creds # type: ignore[assignment] + creds.refresh(google.auth.transport.requests.Request()) + token = getattr(creds, "token", None) + if not isinstance(token, str) or not token: + msg = "ADC refresh produced no access token" + raise RuntimeError(msg) + return token + + +def _resolve_caller_member(access_token: str) -> str: + """Resolve the calling principal to a ``user:`` / ``serviceAccount:`` member. + + Local user ADC (``gcloud auth application-default login``) commonly + has no ``service_account_email`` and an empty ``account`` attribute; + in that case the only reliable identifier is the tokeninfo endpoint, + which returns the authenticated email for the refreshed access token. + """ + import google.auth + + creds, _ = google.auth.default() + sa_email = getattr(creds, "service_account_email", None) + if isinstance(sa_email, str) and sa_email: + return f"serviceAccount:{sa_email}" + account = getattr(creds, "account", "") + if isinstance(account, str) and account: + # gcloud user ADC populates ``account`` with the user email. + return f"user:{account}" + + # Last resort: probe the tokeninfo endpoint. + url = f"{_TOKENINFO_URL}?access_token={urllib.parse.quote(access_token)}" + with urllib.request.urlopen(url, timeout=_HTTP_TIMEOUT_S) as response: + info = json.loads(response.read().decode("utf-8")) + email = info.get("email") or info.get("audience") or "" + if not isinstance(email, str) or not email: + msg = "could not resolve caller principal from ADC or tokeninfo" + raise RuntimeError(msg) + member_type = "serviceAccount" if email.endswith(".gserviceaccount.com") else "user" + return f"{member_type}:{email}" + + +def _create_service_account( + *, + project: str, + token: str, + sa_id: str, + display_name: str, +) -> str: + """Create a service account and return its email.""" + url = f"{_IAM_BASE}/projects/{project}/serviceAccounts" + body = { + "accountId": sa_id, + "serviceAccount": {"displayName": display_name}, + } + _, response = _http_request("POST", url, token, body=body) + email = response.get("email") + if not isinstance(email, str) or not email: + msg = f"create_service_account: response missing email: {response}" + raise RuntimeError(msg) + return email + + +def _delete_service_account(*, project: str, token: str, email: str) -> bool: + """Delete a service account; NotFound is success.""" + url = f"{_IAM_BASE}/projects/{project}/serviceAccounts/{email}" + try: + _http_request("DELETE", url, token) + except urllib.error.HTTPError as e: + if e.code == 404: + return True + print(f" delete_service_account({email}) HTTP {e.code}: {_http_error_body(e)}", file=sys.stderr) + return False + return True + + +def _service_account_resource(project: str, sa_email: str) -> str: + """REST resource path for an SA (used by SA-resource IAM policy calls).""" + return f"projects/{project}/serviceAccounts/{sa_email}" + + +def _modify_iam_policy( + *, + get_url: str, + set_url: str, + token: str, + operation: str, + role: str, + member: str, + get_method: str = "POST", + attempts: int = 5, + backoff: float = 1.0, +) -> bool: + """Read-modify-write an IAM policy with etag retry. + + ``operation`` is ``"add"`` or ``"remove"``. Returns True on success. + + ``get_method`` selects the HTTP verb for ``getIamPolicy``. The IAM API + (service-account resources at ``iam.googleapis.com``) uses + ``POST {resource}:getIamPolicy`` with a JSON body, while the Compute + Engine API (instance / disk / zonal resources at + ``compute.googleapis.com``) uses + ``GET {resource}/getIamPolicy?optionsRequestedPolicyVersion=3`` and + rejects POST with HTTP 400. ``setIamPolicy`` is POST on both APIs. + """ + last_error: str | None = None + for attempt in range(1, attempts + 1): + try: + if get_method == "GET": + _, policy = _http_request( + "GET", + f"{get_url}?optionsRequestedPolicyVersion=3", + token, + ) + else: + _, policy = _http_request( + "POST", + get_url, + token, + body={"options": {"requestedPolicyVersion": 3}}, + ) + except urllib.error.HTTPError as e: + last_error = f"getIamPolicy HTTP {e.code}: {_http_error_body(e)}" + if e.code in (404, 403): + # No policy / no permission — caller decides whether this is fatal. + return False + if e.code == 429 or 500 <= e.code < 600: + time.sleep(backoff * attempt) + continue + # Other 4xx (malformed request, etc.) — retry will not help. + return False + except (urllib.error.URLError, TimeoutError, OSError) as e: + # Transient socket-level failure (connect/read timeout, connection + # reset, DNS). On Python >= 3.10 a mid-read `socket.timeout` is + # `TimeoutError` and may NOT be wrapped in `URLError`, so catch + # both directly to avoid escaping into the outer try. + last_error = f"getIamPolicy network error ({type(e).__name__}): {e}" + time.sleep(backoff * attempt) + continue + + bindings = list(policy.get("bindings", []) or []) + # Locate or create the role binding. + target_idx = next((i for i, b in enumerate(bindings) if b.get("role") == role), None) + if operation == "add": + if target_idx is None: + bindings.append({"role": role, "members": [member]}) + else: + members = list(bindings[target_idx].get("members", [])) + if member not in members: + members.append(member) + bindings[target_idx]["members"] = members + elif operation == "remove": + if target_idx is None: + return True # already absent + members = [m for m in bindings[target_idx].get("members", []) if m != member] + if members: + bindings[target_idx]["members"] = members + else: + bindings.pop(target_idx) + else: + msg = f"invalid operation: {operation!r}" + raise ValueError(msg) + + new_policy = { + "bindings": bindings, + "etag": policy.get("etag", ""), + "version": policy.get("version", 1), + } + try: + _http_request("POST", set_url, token, body={"policy": new_policy}) + return True + except urllib.error.HTTPError as e: + last_error = f"setIamPolicy HTTP {e.code}: {_http_error_body(e)}" + # 409 stale etag (refresh GET on next iter), 429 rate-limit, and 5xx + # transient server errors all warrant the read-modify-write retry. + if e.code in (409, 429) or 500 <= e.code < 600: + time.sleep(backoff * attempt) + continue + return False + except (urllib.error.URLError, TimeoutError, OSError) as e: + # Same socket-level coverage as the getIamPolicy arm — mid-read + # timeouts can escape URLError on Python >= 3.10. + last_error = f"setIamPolicy network error ({type(e).__name__}): {e}" + time.sleep(backoff * attempt) + continue + + if last_error: + print(f" iam_policy_retry exhausted: {last_error}", file=sys.stderr) + return False + + +def _grant_token_creator(*, project: str, token: str, sa_email: str, member: str) -> bool: + """Grant the caller TokenCreator on a probe SA.""" + base = f"{_IAM_BASE}/projects/{project}/serviceAccounts/{sa_email}" + return _modify_iam_policy( + get_url=f"{base}:getIamPolicy", + set_url=f"{base}:setIamPolicy", + token=token, + operation="add", + role=_TOKEN_CREATOR_ROLE, + member=member, + ) + + +def _revoke_token_creator(*, project: str, token: str, sa_email: str, member: str) -> bool: + base = f"{_IAM_BASE}/projects/{project}/serviceAccounts/{sa_email}" + return _modify_iam_policy( + get_url=f"{base}:getIamPolicy", + set_url=f"{base}:setIamPolicy", + token=token, + operation="remove", + role=_TOKEN_CREATOR_ROLE, + member=member, + ) + + +def _grant_target_role( + *, + project: str, + zone: str, + instance: str, + token: str, + sa_email: str, +) -> bool: + """Grant the allowed SA the target-VM serial-output role.""" + base = f"{_COMPUTE_BASE}/projects/{project}/zones/{zone}/instances/{instance}" + return _modify_iam_policy( + get_url=f"{base}/getIamPolicy", + set_url=f"{base}/setIamPolicy", + token=token, + operation="add", + role=_ALLOWED_TARGET_ROLE, + member=f"serviceAccount:{sa_email}", + get_method="GET", + ) + + +def _revoke_target_role( + *, + project: str, + zone: str, + instance: str, + token: str, + sa_email: str, +) -> bool: + base = f"{_COMPUTE_BASE}/projects/{project}/zones/{zone}/instances/{instance}" + return _modify_iam_policy( + get_url=f"{base}/getIamPolicy", + set_url=f"{base}/setIamPolicy", + token=token, + operation="remove", + role=_ALLOWED_TARGET_ROLE, + member=f"serviceAccount:{sa_email}", + get_method="GET", + ) + + +def _mint_access_token( + *, + token: str, + sa_email: str, + attempts: int = 12, + delay: float = 15.0, +) -> str: + """Mint a short-lived access token for ``sa_email`` via IAMCredentials REST. + + TokenCreator IAM bindings on Compute Engine probe service accounts + are eventually-consistent; observed convergence in this suite has + required up to ~3 minutes after the binding is granted. Retry on + HTTP 403 with + a 12 x 15s budget so the probe doesn't nondeterministically fail + with ``iam.serviceAccounts.getAccessToken denied`` against bindings + that converge a few seconds after the call. + """ + url = f"{_IAM_CREDENTIALS_BASE}/projects/-/serviceAccounts/{sa_email}:generateAccessToken" + body = { + "scope": ["https://www.googleapis.com/auth/cloud-platform"], + "lifetime": "300s", + } + last_error: str | None = None + for attempt in range(1, attempts + 1): + try: + _, response = _http_request("POST", url, token, body=body) + except urllib.error.HTTPError as e: + last_error = f"HTTP {e.code}: {_http_error_body(e)}" + if e.code == 403 and attempt < attempts: + print( + f" generateAccessToken attempt {attempt}/{attempts} HTTP 403 " + f"(propagation); retrying in {delay:.0f}s", + file=sys.stderr, + ) + time.sleep(delay) + continue + msg = f"generateAccessToken failed: {last_error}" + raise RuntimeError(msg) from e + minted = response.get("accessToken") + if isinstance(minted, str) and minted: + return minted + last_error = f"response missing accessToken: {response}" + if attempt < attempts: + time.sleep(delay) + continue + msg = f"generateAccessToken failed: {last_error}" + raise RuntimeError(msg) + msg = f"generateAccessToken failed after {attempts} attempts: {last_error}" + raise RuntimeError(msg) + + +def _probe_serial_console( + *, + project: str, + zone: str, + instance: str, + access_token: str, +) -> tuple[int, str]: + """Probe ``getSerialPortOutput`` with ``access_token``. + + Returns ``(http_status, evidence_text)``. Honest signal: the HTTP + status comes from a real probe (200 = allowed, 403 = denied, 404 = + diagnostic gap). The evidence text records the response body / error + detail for audit. + """ + url = f"{_COMPUTE_BASE}/projects/{project}/zones/{zone}/instances/{instance}/serialPort?port=1" + try: + status, response = _http_request("GET", url, access_token) + except urllib.error.HTTPError as e: + body = _http_error_body(e) + message = body.get("error", {}).get("message", "") + return e.code, f"HTTP {e.code}: {message or body}" + contents = response.get("contents") or "" + return status, f"HTTP {status}: contents_length={len(contents)}" + + +def _submit_probe_vm_insert( + *, + project: str, + zone: str, + network: str, + token: str, + name: str, +) -> tuple[bool, str, str]: + """Submit the probe VM insert; return ``(ack_ok, op_name, evidence)``. + + The probe VM is e2-micro with the Debian image family — no GPU, no + NIM, no persistent disk reuse. It exists solely so the allowed-SA + probe can produce a real HTTP 403 against an instance the SA was + NOT granted access to. + + Stamp-before-wait split: this function only submits the insert and + returns the op_name. The caller stamps its probe-VM cleanup tracker + BEFORE blocking on ``wait_for_zonal_op`` so a wait-side failure + still has the partial-create name on disk for teardown. + """ + url = f"{_COMPUTE_BASE}/projects/{project}/zones/{zone}/instances" + body = { + "name": name, + "machineType": f"zones/{zone}/machineTypes/e2-micro", + "disks": [ + { + "boot": True, + "autoDelete": True, + "initializeParams": { + "sourceImage": "projects/debian-cloud/global/images/family/debian-12", + "diskType": f"zones/{zone}/diskTypes/pd-balanced", + "diskSizeGb": "10", + }, + } + ], + "networkInterfaces": [ + { + "network": f"projects/{project}/global/networks/{network}", + } + ], + "labels": { + "createdby": "isvtest", + "isv_role": "console-rbac-probe", + }, + } + try: + _, op = _http_request("POST", url, token, body=body) + except urllib.error.HTTPError as e: + return False, "", f"insert HTTP {e.code}: {_http_error_body(e)}" + op_name = op.get("name", "") + if not op_name: + return False, "", f"insert response missing operation name: {op}" + return True, op_name, f"probe VM {name} insert accepted in {zone}" + + +def _wait_probe_vm_insert( + *, + project: str, + zone: str, + op_name: str, +) -> tuple[bool, str]: + """Block on the probe VM insert op; return ``(ok, evidence)``.""" + try: + wait_for_zonal_op(project, zone, op_name, timeout=300) + except Exception as e: + return False, f"insert wait failed: {e}" + return True, "probe VM insert wait done" + + +def _delete_probe_vm(*, project: str, zone: str, token: str, name: str) -> bool: + """Delete the probe VM; NotFound counts as success.""" + url = f"{_COMPUTE_BASE}/projects/{project}/zones/{zone}/instances/{name}" + try: + _, op = _http_request("DELETE", url, token) + except urllib.error.HTTPError as e: + if e.code == 404: + return True + print(f" delete_probe_vm({name}) HTTP {e.code}: {_http_error_body(e)}", file=sys.stderr) + return False + op_name = op.get("name", "") + if op_name: + try: + wait_for_zonal_op(project, zone, op_name, timeout=180) + except Exception as e: + print(f" delete_probe_vm wait failed: {e}", file=sys.stderr) + return False + return True + + +def _self_provisioned_probe( + *, + project: str, + zone: str, + instance: str, + network: str, + result: dict[str, Any], +) -> int: + """Run the self-provisioned RBAC probe. + + Returns 0 ONLY when every subtest passed AND cleanup succeeded on + every probe resource. Cleanup runs in the ``finally`` block so the + SAs / VM / IAM bindings created by this run are removed even on + partial failure; the ``cleanup_errors`` list is then AND-ed into + ``result['success']`` and the return code, mirroring the AWS oracle + (providers/aws/scripts/vm/console_rbac.py — cleanup failures flip + success to False). + """ + caller_token = _adc_access_token() + caller_member = _resolve_caller_member(caller_token) + result["caller"] = caller_member + + suffix = unique_suffix("rbac", length=8).split("-", 1)[-1] + denied_sa_id = f"isv-rbac-denied-{suffix}"[:30] + allowed_sa_id = f"isv-rbac-allowed-{suffix}"[:30] + probe_vm_name = f"isv-rbac-probe-{suffix}"[:62] + + created: dict[str, Any] = { + "denied_sa": "", + "allowed_sa": "", + "probe_vm": "", + "token_creator_denied": False, + "token_creator_allowed": False, + "target_role_allowed": False, + } + cleanup_errors: list[str] = [] + subtests_passed = False + early_failure: str | None = None + + try: + # 1. Create probe service accounts. + print(f"Creating denied probe SA {denied_sa_id}...", file=sys.stderr) + denied_email = _create_service_account( + project=project, + token=caller_token, + sa_id=denied_sa_id, + display_name="ISV RBAC denied probe", + ) + created["denied_sa"] = denied_email + + print(f"Creating allowed probe SA {allowed_sa_id}...", file=sys.stderr) + allowed_email = _create_service_account( + project=project, + token=caller_token, + sa_id=allowed_sa_id, + display_name="ISV RBAC allowed probe", + ) + created["allowed_sa"] = allowed_email + + # 2. Grant the caller TokenCreator on both probe SAs so we can + # mint access tokens for them. + created["token_creator_denied"] = _grant_token_creator( + project=project, + token=caller_token, + sa_email=denied_email, + member=caller_member, + ) + created["token_creator_allowed"] = _grant_token_creator( + project=project, + token=caller_token, + sa_email=allowed_email, + member=caller_member, + ) + if not (created["token_creator_denied"] and created["token_creator_allowed"]): + early_failure = "could not grant TokenCreator on probe SAs" + result["error"] = early_failure + return 1 + + # 3. Grant the allowed SA the serial-output role on the TARGET VM + # only. The denied SA gets nothing; the allowed SA's binding is + # scoped to this one instance so the resource-scope subtest + # against the second VM is a genuine deny. + created["target_role_allowed"] = _grant_target_role( + project=project, + zone=zone, + instance=instance, + token=caller_token, + sa_email=allowed_email, + ) + if not created["target_role_allowed"]: + early_failure = "could not grant target-VM role to allowed probe SA" + result["error"] = early_failure + return 1 + + # 4. Submit the second probe VM. The allowed SA was NOT granted + # any role on this VM, so the resource-scope subtest is a real + # HTTP 403 (or honest failure if 404 — the instance is missing). + # Stamp-before-wait: record the probe VM name in the cleanup + # tracker IMMEDIATELY after the insert ack so a wait-side + # failure still has the partial-create name on disk for the + # finally-block teardown. + ack_ok, probe_op, ack_evidence = _submit_probe_vm_insert( + project=project, + zone=zone, + network=network, + token=caller_token, + name=probe_vm_name, + ) + if not ack_ok: + early_failure = f"could not create probe VM: {ack_evidence}" + result["error"] = early_failure + return 1 + created["probe_vm"] = probe_vm_name + wait_ok, wait_evidence = _wait_probe_vm_insert( + project=project, + zone=zone, + op_name=probe_op, + ) + if not wait_ok: + early_failure = f"probe VM insert wait failed: {wait_evidence}" + result["error"] = early_failure + return 1 + + # 5. Mint short-lived access tokens (with TokenCreator-propagation + # retry budget — see _mint_access_token) and run the three + # subtests. + denied_token = _mint_access_token(token=caller_token, sa_email=denied_email) + allowed_token = _mint_access_token(token=caller_token, sa_email=allowed_email) + + denied_status, denied_evidence = _probe_serial_console( + project=project, + zone=zone, + instance=instance, + access_token=denied_token, + ) + result["tests"]["denied_principal_cannot_access_console"] = { + "passed": denied_status == 403, + "principal": f"serviceAccount:{denied_email}", + "evidence": denied_evidence, + } + + allowed_status, allowed_evidence = _probe_serial_console( + project=project, + zone=zone, + instance=instance, + access_token=allowed_token, + ) + result["tests"]["allowed_principal_can_access_console"] = { + "passed": allowed_status == 200, + "principal": f"serviceAccount:{allowed_email}", + "evidence": allowed_evidence, + } + + scope_status, scope_evidence = _probe_serial_console( + project=project, + zone=zone, + instance=probe_vm_name, + access_token=allowed_token, + ) + # HTTP 404 is NOT proof of scoping — the resource is missing, so + # IAM enforcement could not be observed. Only 403 counts. + result["tests"]["allowed_principal_is_resource_scoped"] = { + "passed": scope_status == 403, + "principal": f"serviceAccount:{allowed_email}", + "evidence": f"probe_vm={probe_vm_name}; {scope_evidence}", + } + + result["access_restricted"] = ( + result["tests"]["denied_principal_cannot_access_console"]["passed"] + and result["tests"]["allowed_principal_is_resource_scoped"]["passed"] + ) + subtests_passed = all(t["passed"] for t in result["tests"].values()) + if not subtests_passed: + result["error"] = "one or more console RBAC subtests failed; see tests.* evidence" + # Defer the final success/rc computation to the finally block so + # cleanup failures can flip success to False (matches AWS oracle). + return 0 # placeholder — finally overrides with the cleanup-AND-ed rc + + except Exception as e: + # Capture probe-setup errors (SA create/grant HTTP errors, token + # mint failures, etc.) so the operator sees a structured root + # cause rather than a generic three-False-subtest failure. The + # finally block still runs to clean up partial probe resources. + early_failure = f"{type(e).__name__}: {e}" + result["error"] = early_failure + return 1 + finally: + # Cleanup runs unconditionally so this stub never leaks probe + # resources / IAM bindings even on partial failure. Each + # cleanup helper returns bool and is AND-ed into success. + if created["target_role_allowed"]: + ok = _revoke_target_role( + project=project, + zone=zone, + instance=instance, + token=caller_token, + sa_email=created["allowed_sa"], + ) + if not ok: + cleanup_errors.append(f"revoke target role on {instance}") + if created["probe_vm"]: + ok = _delete_probe_vm( + project=project, + zone=zone, + token=caller_token, + name=created["probe_vm"], + ) + if not ok: + cleanup_errors.append(f"delete probe VM {created['probe_vm']}") + for sa_email, created_flag in ( + (created["denied_sa"], created["token_creator_denied"]), + (created["allowed_sa"], created["token_creator_allowed"]), + ): + if not sa_email: + continue + if created_flag: + if not _revoke_token_creator( + project=project, + token=caller_token, + sa_email=sa_email, + member=caller_member, + ): + cleanup_errors.append(f"revoke tokenCreator on {sa_email}") + if not _delete_service_account(project=project, token=caller_token, email=sa_email): + cleanup_errors.append(f"delete service account {sa_email}") + if cleanup_errors: + result["cleanup_errors"] = cleanup_errors + # Final success/rc: subtests AND cleanup AND no early failure. + final_success = subtests_passed and not cleanup_errors and early_failure is None + result["success"] = final_success + if cleanup_errors and not result.get("error"): + result["error"] = "console RBAC cleanup failed: " + "; ".join(cleanup_errors) + elif cleanup_errors and result.get("error"): + result["error"] = f"{result['error']}; cleanup failed: {'; '.join(cleanup_errors)}" + return 0 if final_success else 1 + + +def _preprovisioned_probe( + *, + project: str, + zone: str, + instance: str, + denied_sa: str, + allowed_sa: str, + other_instance: str, + other_zone: str, + result: dict[str, Any], +) -> int: + """Run the pre-provisioned RBAC probe with operator-supplied principals. + + The fallback path for projects where IAM mutation is not allowed. + Operators must pre-create denied / allowed SAs, grant the caller + TokenCreator on both, scope the allowed SA's serial-output role to + the target VM, and create a real ``GCP_OTHER_INSTANCE_ID`` that the + allowed SA has NOT been granted access to. + + Workflow exceptions (ADC failure, token-mint propagation timeout, + HTTP errors from getSerialPortOutput) are caught here so the + contract-shaped result populated by ``main()`` survives — mirrors + the AWS oracle's try/except around its console RBAC workflow. + Escaping to ``handle_gcp_errors`` would drop ``platform``, + ``test_name``, ``rbac_model``, ``access_restricted``, and + ``tests.*`` from the printed JSON. + """ + try: + caller_token = _adc_access_token() + result["caller"] = _resolve_caller_member(caller_token) + denied_token = _mint_access_token(token=caller_token, sa_email=denied_sa) + allowed_token = _mint_access_token(token=caller_token, sa_email=allowed_sa) + + denied_status, denied_evidence = _probe_serial_console( + project=project, + zone=zone, + instance=instance, + access_token=denied_token, + ) + result["tests"]["denied_principal_cannot_access_console"] = { + "passed": denied_status == 403, + "principal": f"serviceAccount:{denied_sa}", + "evidence": denied_evidence, + } + + allowed_status, allowed_evidence = _probe_serial_console( + project=project, + zone=zone, + instance=instance, + access_token=allowed_token, + ) + result["tests"]["allowed_principal_can_access_console"] = { + "passed": allowed_status == 200, + "principal": f"serviceAccount:{allowed_sa}", + "evidence": allowed_evidence, + } + + scope_status, scope_evidence = _probe_serial_console( + project=project, + zone=other_zone, + instance=other_instance, + access_token=allowed_token, + ) + result["tests"]["allowed_principal_is_resource_scoped"] = { + "passed": scope_status == 403, + "principal": f"serviceAccount:{allowed_sa}", + "evidence": f"other_instance={other_instance}; {scope_evidence}", + } + + result["access_restricted"] = ( + result["tests"]["denied_principal_cannot_access_console"]["passed"] + and result["tests"]["allowed_principal_is_resource_scoped"]["passed"] + ) + all_passed = all(t["passed"] for t in result["tests"].values()) + result["success"] = all_passed + if not all_passed: + result["error"] = "one or more console RBAC subtests failed; see tests.* evidence" + return 0 if all_passed else 1 + except Exception as e: + result["success"] = False + result["access_restricted"] = False + result["error"] = str(e) + return 1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Console RBAC probe (Compute Engine)") + parser.add_argument("--instance-id", required=True, help="Target instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument( + "--network", + default="default", + help="Network for the probe VM (self-provisioned path)", + ) + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "test_name": "console_rbac", + "instance_id": args.instance_id, + "project": project, + "zone": zone, + "rbac_model": "gcp-iam", + "access_restricted": False, + "restricted_actions": list(_RESTRICTED_ACTIONS), + "tests": { + "denied_principal_cannot_access_console": {"passed": False, "principal": "", "evidence": ""}, + "allowed_principal_can_access_console": {"passed": False, "principal": "", "evidence": ""}, + "allowed_principal_is_resource_scoped": {"passed": False, "principal": "", "evidence": ""}, + }, + } + + denied_sa = os.environ.get(_DENIED_SA_ENV, "").strip() + allowed_sa = os.environ.get(_ALLOWED_SA_ENV, "").strip() + other_instance = os.environ.get(_OTHER_INSTANCE_ENV, "").strip() + other_zone = os.environ.get(_OTHER_INSTANCE_ZONE_ENV, "").strip() or zone + + # The pre-provisioned fallback runs only when the operator supplies + # all three env vars. Otherwise the default self-provisioned path + # runs (unless explicitly disabled via _SELF_PROVISION_ENABLED_ENV=0). + if denied_sa and allowed_sa and other_instance: + result["mode"] = "preprovisioned" + rc = _preprovisioned_probe( + project=project, + zone=zone, + instance=args.instance_id, + denied_sa=denied_sa, + allowed_sa=allowed_sa, + other_instance=other_instance, + other_zone=other_zone, + result=result, + ) + print(json.dumps(result, indent=2, default=str)) + return rc + + if os.environ.get(_SELF_PROVISION_ENABLED_ENV, "1").strip() in {"0", "false", "no"}: + # Intentional opt-out via env var. Treat as a clean policy-skip + # (rc=0, success=True, skipped=True) — same shape as deploy_nim's + # missing-NGC_API_KEY skip — so the orchestrator's + # StepSuccessCheck reads this as "step short-circuited cleanly," + # not as a failed RBAC probe. + result["mode"] = "skipped" + result["skipped"] = True + result["success"] = True + result["skip_reason"] = ( + f"{_SELF_PROVISION_ENABLED_ENV} disables the self-provisioned probe and no " + f"{_DENIED_SA_ENV} / {_ALLOWED_SA_ENV} / {_OTHER_INSTANCE_ENV} fallback was supplied" + ) + print(json.dumps(result, indent=2, default=str)) + return 0 + + result["mode"] = "self_provisioned" + rc = _self_provisioned_probe( + project=project, + zone=zone, + instance=args.instance_id, + network=args.network, + result=result, + ) + print(json.dumps(result, indent=2, default=str)) + return rc + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/describe_instance.py b/isvctl/configs/providers/gcp/scripts/vm/describe_instance.py new file mode 100644 index 00000000..d463e422 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/describe_instance.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Describe a Compute Engine VM as the anchor for host-level validators. + +Per the suite contract this step runs AFTER reboot so host-level checks +(ConnectivityCheck, GpuCheck, etc.) bind to the post-reboot guest. The +stub itself just reads the current state via ``instances.get`` and +forwards SSH ingredients. + +Divergences: + * Compute Engine reports the zone as a self-link; emit the short name + under ``availability_zone`` for parity with the AWS oracle. + * Use ``canonical_state(...)`` so downstream code branches on the + documented vocabulary, not the raw GCE enum. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + canonical_state, + first_external_ip, + first_internal_ip, + get_instance, + narrow_region_to_zone, + resolve_project, + short_name, +) +from common.errors import handle_gcp_errors +from google.api_core import exceptions as gax + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Describe a Compute Engine VM") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument("--key-file", required=True, help="SSH private key path") + parser.add_argument("--ssh-user", default="ubuntu", help="SSH username") + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "region": args.region, + "zone": zone, + "project": project, + "key_file": args.key_file, + "ssh_user": args.ssh_user, + } + + try: + inst = get_instance(project, zone, args.instance_id) + result["state"] = canonical_state(inst.status) + result["instance_type"] = short_name(inst.machine_type) + result["public_ip"] = first_external_ip(inst) + result["private_ip"] = first_internal_ip(inst) + if inst.network_interfaces: + result["vpc_id"] = short_name(inst.network_interfaces[0].network) + if inst.network_interfaces[0].subnetwork: + result["subnet_id"] = short_name(inst.network_interfaces[0].subnetwork) + result["availability_zone"] = short_name(inst.zone) + result["launch_time"] = getattr(inst, "creation_timestamp", None) + result["success"] = True + + except gax.NotFound as e: + result["error"] = f"Instance {args.instance_id} not found: {e}" + except Exception as e: + result["error"] = str(e) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/describe_tags.py b/isvctl/configs/providers/gcp/scripts/vm/describe_tags.py new file mode 100644 index 00000000..3d3d00f8 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/describe_tags.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Read Compute Engine labels and emit them as canonical-cased tags. + +Compute Engine label keys must match ``[a-z]([-a-z0-9_]*)`` so the +launch step writes lowercase labels. The suite contract expects mixed- +case ``Name`` / ``CreatedBy`` keys — we project labels back to canonical +casing here so ``InstanceTagCheck.required_keys`` stays unchanged across +providers. + +Validator-consumed fields must derive from a real signal: the values +come from the actual labels on the live instance, never hardcoded. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + get_instance, + labels_to_canonical_tags, + narrow_region_to_zone, + resolve_project, +) +from common.errors import handle_gcp_errors + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Describe Compute Engine instance labels") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "tags": {}, + "tag_count": 0, + "region": args.region, + "zone": zone, + "project": project, + } + + try: + inst = get_instance(project, zone, args.instance_id) + labels = dict(getattr(inst, "labels", {}) or {}) + tags = labels_to_canonical_tags(labels) + result["tags"] = tags + result["tag_count"] = len(tags) + result["success"] = True + + except Exception as e: + result["error"] = str(e) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/launch_instance.py b/isvctl/configs/providers/gcp/scripts/vm/launch_instance.py new file mode 100644 index 00000000..c0a7ebbe --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/launch_instance.py @@ -0,0 +1,880 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Launch a GPU VM on Compute Engine for VM-domain validation. + +Translates the AWS oracle's ``launch_instance`` workflow to Compute +Engine. Documented divergences: + + * No managed key-pair store — generate a local PEM/.pub pair and + attach the public key via instance metadata. + * Firewall rules are project-global and bound by network tag, not + attached per-instance — create / verified-reuse a TCP/22 INGRESS + rule on the launch network and assign the matching network tag. + * GPU-bearing machine types reject ``onHostMaintenance=MIGRATE`` + (HTTP 400); force ``TERMINATE`` + ``automatic_restart=true``. + * ``instances.insert`` returns DONE before the guest is reachable — + poll RUNNING, then run a best-effort SSH-or-cloud-init readiness gate. + * Public IP is assigned only when an ``accessConfigs`` entry of type + ``ONE_TO_ONE_NAT`` is requested on the NIC. + * Compute Engine label keys must be lowercase. Project canonical + mixed-case ``Name`` / ``CreatedBy`` keys to api-valid labels on + create and back on read so ``InstanceTagCheck.required_keys`` does + not change per provider. + * Emit the effective ``zone``, ``firewall_created``, ``key_created`` + so every downstream zonal step + teardown can read them via + ``{{steps.launch_instance.X}}`` (verified-reuse cleanup contract). + +Operator-supplied image identifiers are resolved against +``args.image_project`` FIRST — short-name identifiers MUST be resolved +against the operator's chosen project/account/region scope first. +A vendor-default fallback is allowed only as an explicit second attempt. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + ISV_NETWORK_TAG, + canonical_state, + canonical_tags_to_labels, + delete_failed_zonal_instance, + delete_local_keypair, + first_external_ip, + first_internal_ip, + generate_ssh_keypair, + get_instance, + insert_ssh_firewall, + is_gpu_machine_type, + is_zone_unavailable, + narrow_region_to_zone, + poll_instance_state, + read_ssh_pubkey, + resolve_image, + resolve_project, + retry_zonal_lifecycle_op, + select_zones, + short_name, + unique_suffix, + wait_for_global_op, + wait_for_public_ip, + wait_for_zonal_op, +) +from common.errors import delete_with_retry, handle_gcp_errors +from common.ssh_utils import wait_for_cloud_init, wait_for_ssh, wait_for_ssh_stable +from google.api_core import exceptions as gax +from google.cloud import compute_v1 + +# GCP Deep Learning VM Image — the closest public equivalent to AWS's +# Deep Learning AMIs. Ships with the NVIDIA driver + CUDA toolkit +# preinstalled. Does NOT ship Docker; for tests that require a container +# runtime (e.g. the NIM deploy step) operators must either supply a +# custom image via --image-project / --image-family / --ami-id, or +# install Docker out-of-band before invoking the suite. The image lives +# in a public GCP-published project so no operator-specific entitlement +# is needed. +DEFAULT_IMAGE_FAMILY = "common-cu129-ubuntu-2204-nvidia-580" +DEFAULT_IMAGE_PROJECT = "deeplearning-platform-release" +DEFAULT_NETWORK = "default" +DEFAULT_FIREWALL_NAME = "isv-test-vm-ssh" +DEFAULT_KEY_NAME = "isv-test-key" +DEFAULT_SSH_USER = "ubuntu" + +# Bound the per-attempt wait so the 3-attempt delete_with_retry +# does not multiply 600s zonal-op + 120s global-op budgets into the +# enclosing step timeout. Cleanup-on-failure runs from inside the +# launch_instance step, whose budget already covers happy-path waits; +# delete waits beyond 180s instance / 120s firewall are diminishing +# returns under transient control-plane errors. +_CLEANUP_INSTANCE_WAIT_S = 180 +_CLEANUP_FIREWALL_WAIT_S = 120 + + +def _build_instance_resource( + *, + project: str, + zone: str, + name: str, + machine_type: str, + source_image: str, + network_name: str, + subnet_name: str | None, + ssh_user: str, + ssh_pubkey: str, + labels: dict[str, str], +) -> compute_v1.Instance: + """Build a Compute Engine ``Instance`` resource for ``instances.insert``. + + Every property here serializes via proto-plus (so it survives the + REST encode); ad-hoc ``obj._properties[...] = ...`` mutations would + be silently dropped. + + Subnetwork (when supplied) MUST be the regional URL; ``machine_type`` + MUST be the zonal URL — bare tokens are rejected by the proto wire + layer. + """ + instance = compute_v1.Instance() + instance.name = name + instance.machine_type = f"zones/{zone}/machineTypes/{machine_type}" + + boot = compute_v1.AttachedDisk() + boot.boot = True + boot.auto_delete = True + init = compute_v1.AttachedDiskInitializeParams() + init.source_image = source_image + init.disk_size_gb = 100 + boot.initialize_params = init + instance.disks = [boot] + + nic = compute_v1.NetworkInterface() + nic.network = f"projects/{project}/global/networks/{network_name}" + if subnet_name: + region = zone.rsplit("-", 1)[0] + nic.subnetwork = f"projects/{project}/regions/{region}/subnetworks/{subnet_name}" + nat = compute_v1.AccessConfig() + nat.type_ = "ONE_TO_ONE_NAT" + nat.name = "External NAT" + nic.access_configs = [nat] + instance.network_interfaces = [nic] + + instance.tags = compute_v1.Tags(items=[ISV_NETWORK_TAG]) + + # GPU machine types REJECT the default `MIGRATE` and require + # `TERMINATE` + `automatic_restart`. This override is GPU-only — + # non-GPU types must keep the API default (MIGRATE) to preserve + # live-migrate behavior. + if is_gpu_machine_type(machine_type): + sched = compute_v1.Scheduling() + sched.on_host_maintenance = "TERMINATE" + sched.automatic_restart = True + instance.scheduling = sched + + instance.labels = labels + + ssh_item = compute_v1.Items() + ssh_item.key = "ssh-keys" + ssh_item.value = f"{ssh_user}:{ssh_pubkey}" + instance.metadata = compute_v1.Metadata(items=[ssh_item]) + + return instance + + +def _delete_instance_op(project: str, zone: str, name: str) -> None: + """Submit ``instances.delete`` and wait on the zonal op (NotFound is idempotent).""" + try: + op = compute_v1.InstancesClient().delete(project=project, zone=zone, instance=name) + except gax.NotFound: + return + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + wait_for_zonal_op(project, zone, op_name, timeout=_CLEANUP_INSTANCE_WAIT_S) + + +def _delete_firewall_op(project: str, name: str) -> None: + """Submit ``firewalls.delete`` and wait on the global op (NotFound is idempotent).""" + try: + op = compute_v1.FirewallsClient().delete(project=project, firewall=name) + except gax.NotFound: + return + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + wait_for_global_op(project, op_name, timeout=_CLEANUP_FIREWALL_WAIT_S) + + +def _find_ssh_firewall_for_instance( + project: str, + inst: compute_v1.Instance, +) -> str | None: + """Best-effort: derive the SSH firewall name covering ``inst``. + + Used on the reuse-existing-instance path to surface the security + handle from live state rather than fabricate one. Returns None when + no rule matches; emitting None is more honest than a stand-in + (the validator + teardown gating both see "no firewall to manage"). + """ + if not inst.network_interfaces: + return None + network_short = short_name(inst.network_interfaces[0].network) + inst_tags = set(getattr(inst.tags, "items", []) or []) + if not inst_tags: + return None + try: + rules = compute_v1.FirewallsClient().list(project=project) + except gax.GoogleAPICallError: + return None + for rule in rules: + if short_name(rule.network) != network_short: + continue + if rule.direction != "INGRESS": + continue + if not (set(rule.target_tags) & inst_tags): + continue + for allowed in rule.allowed: + if allowed.I_p_protocol.lower() == "tcp" and "22" in list(allowed.ports): + return rule.name + return None + + +def _reuse_existing_instance( + *, + project: str, + zone: str, + instance_id: str, + key_file: str, + ssh_user: str, +) -> int: + """Mirror the AWS oracle's ``AWS_VM_INSTANCE_ID``/``AWS_VM_KEY_FILE`` reuse path. + + GCP equivalents are ``GCP_VM_INSTANCE_ID`` / ``GCP_VM_KEY_FILE``. When + both are set, the stub describes the existing instance (and starts it + if it's canonically stopped) instead of provisioning a new one — the + dev workflow for iterating against a long-lived VM. + + Verified-reuse semantics: ``firewall_created`` / ``key_created`` stay + False so teardown's gates skip destruction of pre-existing resources. + """ + print(f"Reusing existing instance {instance_id}", file=sys.stderr) + + # Reuse-branch must not fabricate keys it doesn't have evidence for. + # Initialize fields as None and only fill them when live state + # provides a value. + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": instance_id, + "project": project, + "zone": zone, + "availability_zone": zone, + "key_file": key_file, + "key_created": False, + "firewall_created": False, + # Verified-reuse ownership: adoption path NEVER created the + # instance, so teardown must skip both the primary and any + # leaked-zone delete. False stays False — there is no in-stub + # branch that could promote it on the reuse path. + "instance_created": False, + "firewall_name": None, + "security_group_id": None, + "key_name": None, + "ssh_user": ssh_user, + "reused": True, + "tags": {}, + } + + started_in_reuse = False + try: + inst = get_instance(project, zone, instance_id) + cstate = canonical_state(inst.status) + + if cstate == "stopped": + print(f" {instance_id} is stopped — starting it", file=sys.stderr) + # Sister-stub consistency (rule #4): the dedicated + # `start_instance.py` wraps the start sync+wait pair in the + # in-zone retry-with-backoff envelope (3 attempts, 60s/120s + # backoff). The reuse-from-stopped path runs the SAME + # lifecycle op against the SAME zone-bound instance and MUST + # honor the same recovery contract — operators stockout-flake + # here exactly as they would on the canonical start step. + client = compute_v1.InstancesClient() + retry_zonal_lifecycle_op( + lambda: client.start(project=project, zone=zone, instance=instance_id), + project, + zone, + resource_desc=f"reuse-start {instance_id}", + ) + poll_instance_state(project, zone, instance_id, target_canonical="running", timeout=300) + inst = get_instance(project, zone, instance_id) + cstate = canonical_state(inst.status) + started_in_reuse = True + + result["state"] = cstate + result["instance_type"] = short_name(inst.machine_type) + result["public_ip"] = first_external_ip(inst) or wait_for_public_ip(project, zone, instance_id, timeout=120) + result["private_ip"] = first_internal_ip(inst) + if inst.network_interfaces: + result["vpc_id"] = short_name(inst.network_interfaces[0].network) + if inst.network_interfaces[0].subnetwork: + result["subnet_id"] = short_name(inst.network_interfaces[0].subnetwork) + # Only emit canonical tag keys when their backing labels are + # actually present on the live instance. Fabricating defaults + # here would diverge from the AWS oracle reuse path, which + # emits exactly what the API returned. + actual_labels = dict(getattr(inst, "labels", {}) or {}) + derived_tags: dict[str, str] = {} + if "isv_name" in actual_labels: + derived_tags["Name"] = actual_labels["isv_name"] + if "createdby" in actual_labels: + derived_tags["CreatedBy"] = actual_labels["createdby"] + result["tags"] = derived_tags + + # Derive the firewall handle from live state if possible — never + # fabricate (rule #1 oracle-derivation parity). The gating flags + # stay False either way, so this is purely informational. + derived_fw = _find_ssh_firewall_for_instance(project, inst) + if derived_fw: + result["firewall_name"] = derived_fw + result["security_group_id"] = derived_fw + + # Compute Engine has no managed key-pair store and no live + # `KeyName` field on the instance record. There is no portable + # counterpart to the AWS oracle signal, and the reviewed GCP + # knowledge file requires emitting `result["key_name"] = None` + # rather than synthesizing a basename / `@reuse` token. + # Local PEM identity flows through `key_file` alone. + result["key_name"] = None + + if cstate != "running" or not result["public_ip"]: + result["error"] = f"Instance {instance_id} is {cstate!r} or has no external IP" + print(json.dumps(result, indent=2, default=str)) + return 1 + + # The reuse branch must enforce the same readiness gate as the + # create branch — consecutive-success stability, not first-SSH. + # A reused VM whose sshd transiently flakes during the probe + # would otherwise pass on a single success and surface as an + # unstable readiness state to downstream validators. + ssh_ok = wait_for_ssh_stable( + host=result["public_ip"], + user=ssh_user, + key_file=key_file, + consecutive=3, + interval=10, + max_attempts=36, + ) + cloud_init_ok = False + if ssh_ok: + cloud_init_ok = wait_for_cloud_init( + host=result["public_ip"], + user=ssh_user, + key_file=key_file, + timeout_seconds=600, + ) + result["ssh_ready"] = ssh_ok + result["cloud_init_ok"] = cloud_init_ok + # When the reuse branch just started the VM, both SSH stability + # AND cloud-init completion are required to match the dedicated + # start_instance step's success contract. Downstream validators + # would otherwise race a guest whose cloud-init replay is still + # rewriting authorized_keys / fstab. For a guest that was already + # running when adopted, SSH stability alone is enough — cloud-init + # only completes once per boot. + if started_in_reuse: + ready = ssh_ok and cloud_init_ok + else: + ready = ssh_ok or cloud_init_ok + if ready: + result["success"] = True + else: + result["error"] = ( + f"Instance {instance_id} is RUNNING but the reuse-path " + "readiness gate (ssh stability + cloud-init completion) " + "did not pass" + ) + + except Exception as e: + result["error"] = str(e) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Launch a GPU VM on Compute Engine") + parser.add_argument("--name", default="isv-test-gpu", help="Instance name") + parser.add_argument( + "--instance-type", + required=True, + help="Compute Engine machineType (e.g., g2-standard-8)", + ) + parser.add_argument( + "--region", + required=True, + help="GCP region or zone; if a region is given it's narrowed to -a", + ) + parser.add_argument("--zone", default=None, help="GCP zone (overrides region narrowing)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument("--vpc-id", default=DEFAULT_NETWORK, help="Network short name") + parser.add_argument("--subnet-id", default=None, help="Subnetwork short name") + parser.add_argument( + "--image-family", + default=DEFAULT_IMAGE_FAMILY, + help="GCP image family (resolved to a concrete image at runtime)", + ) + parser.add_argument( + "--image-project", + default=None, + help=( + "GCP project hosting the image. When omitted: short-name " + "--ami-id resolves in the operator project first (parameter-" + "surface parity with the AWS oracle, where AMI IDs are " + "account-scoped); --image-family resolves in the default " + "project (where the canonical GPU image lives). Pass " + "explicitly to override either fallback." + ), + ) + parser.add_argument( + "--ami-id", + default=None, + help=( + "Parameter-surface parity with the AWS oracle's --ami-id; " + "if set, overrides --image-family lookup with a literal image" + ), + ) + parser.add_argument("--key-name", default=DEFAULT_KEY_NAME, help="Local SSH key label") + parser.add_argument( + "--firewall-name", + default=DEFAULT_FIREWALL_NAME, + help="SSH firewall rule name", + ) + parser.add_argument("--ssh-user", default=DEFAULT_SSH_USER, help="SSH username") + args = parser.parse_args() + + # The provider config wires --subnet-id / --ami-id from settings that + # default to the literal "none" sentinel so the orchestrator does not + # collapse the flag/value pair. Treat the sentinel as "operator did + # not supply" — the default + # subnet for the resolved zone and the canonical image family take + # over. + if args.subnet_id == "none": + args.subnet_id = None + if args.ami_id == "none": + args.ami_id = None + if args.image_project == "none": + args.image_project = None + + project = resolve_project(args.project) + initial_zone = args.zone or narrow_region_to_zone(args.region) + + # Reuse-existing-instance branch (AWS oracle parity). + reuse_instance = os.environ.get("GCP_VM_INSTANCE_ID") + reuse_key = os.environ.get("GCP_VM_KEY_FILE") + if reuse_instance and reuse_key: + return _reuse_existing_instance( + project=project, + zone=initial_zone, + instance_id=reuse_instance, + key_file=reuse_key, + ssh_user=args.ssh_user, + ) + + # Apply the RUN_ID suffix. Compute Engine names ARE the API IDs — + # without the suffix, parallel runs collide on AlreadyExists during + # create and /tmp/.pem clobbers across sessions (name-collision + # risk). The suffix lives at runtime, NOT in provider config (only + # the team-letter belongs in config). + instance_name = unique_suffix(args.name) + firewall_name_suffixed = unique_suffix(args.firewall_name) + key_name_suffixed = unique_suffix(args.key_name) + + # Multi-zone walk candidates. select_zones honors a single-zone pin + # (full ``us-central1-a`` form) and otherwise queries the operator- + # supplied region's live zones via the GCP API before iterating + # preferred-in-region → other-in-region → cross-region + # (zone_capacity_handling). Passing the resolved project lets the + # helper query the regions API so a + # valid region missing from PREFERRED_ZONES still walks its OWN + # zones first. + candidate_zones = select_zones(args.zone or args.region, project=project) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + # Stays None until instances.insert ack; emitted only when an + # API-accepted name exists (AWS oracle parity). + "instance_id": None, + "instance_type": args.instance_type, + "region": args.region, + "zone": initial_zone, + "availability_zone": initial_zone, + "project": project, + "vpc_id": args.vpc_id, + "subnet_id": args.subnet_id, + # Compute Engine has no managed key-pair store; local PEM + # identity flows through `key_file` end-to-end. The CLI accepts + # `--key-name` for AWS-oracle invocation parity but the emitted + # result is unconditionally None per the reviewed GCP knowledge. + "key_name": None, + # Producer-side sentinel defense: initialize to the canonical + # sentinel so an exception BEFORE generate_ssh_keypair (e.g., + # resolve_image raising) still emits a non-empty value. The + # consumer template in vm.yaml uses `default('none', true)` + # (boolean mode) to collapse both undefined AND empty/sentinel + # values to the same downstream arg; matching the producer side + # keeps the contract tight under partial-failure JSON. + "key_file": "none", + "key_created": False, + # Verified-reuse ownership for the instance itself. Stays False + # until the instances.insert ack returns; teardown gates primary + # and leaked-zone deletes on this so a pre-RUNNING failure + # (e.g., image resolve) cannot make teardown destroy an + # operator-supplied VM that this run never touched. + "instance_created": False, + "firewall_name": firewall_name_suffixed, + "firewall_created": False, + "security_group_id": firewall_name_suffixed, + "ssh_user": args.ssh_user, + "state": "", + "public_ip": None, + "private_ip": None, + # Filled after resolve_image() — never echo the requested family. + "ami_id": "", + "tags": {}, + # leaked_zones flows into teardown's --leaked-zones arg so any + # partial-create in a failed zone gets a second-chance delete + # (zone_capacity_handling). + "leaked_zones": [], + } + + # Per-resource trackers for the cleanup-on-failure block. + instance_created = False + zone = initial_zone # tracked separately so the walk can update it + key_priv: str | None = None + key_created = False + firewall_created = False + fw_name = firewall_name_suffixed + + try: + # 0. Resolve image. Three operator-supplied shapes are honored + # (operator scope wins): + # * Full self-link (``https://...`` or ``projects/

/global/ + # images/``) — pass through verbatim; Compute Engine + # accepts it as ``sourceImage``. + # * Short name OR family alias under ``--image-project`` — + # route through ``resolve_image`` which tries + # ``images.get`` then ``images.get_from_family``. This is + # the AWS-oracle parameter-surface parity case: the + # operator passes ``--ami-id `` (mirroring AWS) and + # the stub resolves it inside the operator's chosen + # project, NOT a hardcoded vendor default. + # ``--ami-id`` is treated as a literal-or-short hint; the family + # alias branch uses ``--image-family`` so operators can supply + # either without ambiguity. + if args.ami_id: + literal_image = args.ami_id + is_full_path = literal_image.startswith(("projects/", "https://")) + if is_full_path: + resolved_source_image = literal_image + result["ami_id"] = short_name(literal_image) + result["ami_self_link"] = literal_image + result["ami_name"] = short_name(literal_image) + else: + # Short name — operator-scope parameter-surface parity: + # AMI IDs are account-scoped on AWS, so the AWS-oracle + # invocation pattern (`--ami-id `) MUST resolve in + # the operator project on GCP. Try operator project + # first; on NotFound (e.g., operator following a tutorial + # that names a vendor-default image) fall back to the + # vendor default. An explicit `--image-project` wins + # over both. + explicit_project = args.image_project + operator_scope = explicit_project or project + try: + image = resolve_image(operator_scope, literal_image) + except gax.NotFound: + if explicit_project: + # Operator explicitly named the scope — do not + # silently substitute. Surface the error. + raise RuntimeError( + f"Image {literal_image!r} not found in project {explicit_project!r}" + ) from None + if operator_scope == DEFAULT_IMAGE_PROJECT: + # Already searched the default project; nothing + # more to try. + raise RuntimeError(f"Image {literal_image!r} not found in project {operator_scope!r}") from None + print( + f" image {literal_image!r} not in operator project " + f"{operator_scope!r}; falling back to default project " + f"{DEFAULT_IMAGE_PROJECT!r}", + file=sys.stderr, + ) + try: + image = resolve_image(DEFAULT_IMAGE_PROJECT, literal_image) + except gax.NotFound as e: + raise RuntimeError( + f"Image {literal_image!r} not found in operator project " + f"{operator_scope!r} or default project " + f"{DEFAULT_IMAGE_PROJECT!r}: {e}" + ) from e + resolved_source_image = image.self_link + result["ami_id"] = short_name(image.self_link) + result["ami_name"] = image.name + result["ami_self_link"] = image.self_link + else: + # Image-family lookup — the canonical GPU image lives in + # the default project, so the family-default route reads from + # there unless the operator overrides --image-project. + family_scope = args.image_project or DEFAULT_IMAGE_PROJECT + try: + image = resolve_image(family_scope, args.image_family) + resolved_source_image = image.self_link + result["ami_id"] = short_name(image.self_link) + result["ami_name"] = image.name + result["ami_self_link"] = image.self_link + except gax.NotFound as e: + raise RuntimeError(f"Image {args.image_family!r} in project {family_scope!r} not found: {e}") from e + + # 1. Local SSH key pair (verified-reuse). Use the run-id-suffixed + # name so /tmp/-.pem can't collide between sessions. + # The tuple-unpack shape matches the drift-guard contract. + key_priv, key_created = generate_ssh_keypair(key_name_suffixed) + ssh_pubkey = read_ssh_pubkey(key_priv) + result["key_file"] = key_priv + result["key_created"] = key_created + + # 2. SSH firewall on the target network (verified-reuse). + # Stamp-before-wait pattern — insert returns ``(name, op)``; the + # caller stamps ``firewall_created`` BEFORE the wait so a + # wait-side failure leaves the truthful flag for cleanup + # (cleanup-tracker pattern). ``op is None`` when the helper + # adopted a verified-reuse + # existing rule, in which case ``firewall_created`` stays False. + fw_name, fw_op = insert_ssh_firewall( + project=project, + name=firewall_name_suffixed, + network_short=args.vpc_id, + ) + if fw_op is not None: + firewall_created = True + result["firewall_created"] = True + wait_for_global_op(project, fw_op.name, timeout=120) + result["firewall_name"] = fw_name + result["security_group_id"] = fw_name + + # 3. Build / insert with multi-zone walk on STOCKOUT. + # Canonical tag projection happens at the boundary; the emitted + # ``tags`` dict comes from a live readback further below. The + # ``Name`` tag carries the same suffixed instance_name so + # ``gcloud compute instances list --filter "labels.name~$RUN_ID"`` + # works for cross-resource grouping. + canonical_tags = {"Name": instance_name, "CreatedBy": "isvtest"} + labels = canonical_tags_to_labels(canonical_tags) + + instances_client = compute_v1.InstancesClient() + last_error: Exception | None = None + op = None + op_name = "" + for candidate_idx, candidate_zone in enumerate(candidate_zones, start=1): + print( + f"Inserting instance {instance_name} in " + f"{project}/{candidate_zone} [{candidate_idx}/{len(candidate_zones)}]...", + file=sys.stderr, + ) + instance_resource = _build_instance_resource( + project=project, + zone=candidate_zone, + name=instance_name, + machine_type=args.instance_type, + source_image=resolved_source_image, + network_name=args.vpc_id, + subnet_name=args.subnet_id, + ssh_user=args.ssh_user, + ssh_pubkey=ssh_pubkey, + labels=labels, + ) + try: + op = instances_client.insert( + project=project, + zone=candidate_zone, + instance_resource=instance_resource, + ) + # Stamp-before-wait: set the cleanup tracker AND + # result['instance_id'] / result['zone'] IMMEDIATELY + # after the insert ack, BEFORE the wait. A wait-side + # failure then leaves the truthful identifier on disk + # for teardown. result['instance_created'] is stamped + # on the same tick so the teardown ownership flag + # forwarded via vm.yaml stays in sync with the + # in-process tracker driving cleanup-on-failure. + instance_created = True + zone = candidate_zone + result["instance_id"] = instance_name + result["instance_created"] = True + result["zone"] = candidate_zone + result["availability_zone"] = candidate_zone + + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + wait_for_zonal_op(project, candidate_zone, op_name, timeout=600) + # Insert + DONE successful — break out of the walk. + break + except Exception as exc: + # Async DONE-with-errors raises from wait_for_zonal_op. + # Sync stockout raises from insert. is_zone_unavailable + # covers all four shapes; treat non-zone errors as fatal. + if not is_zone_unavailable(exc, op=op): + raise + last_error = exc + # Shape 2: clean up the partial async-insert before + # moving to the next zone, so the failed zone doesn't + # leak a phantom instance record. + if instance_created: + print( + f" zone {candidate_zone} unavailable; cleaning partial create", + file=sys.stderr, + ) + cleaned = delete_failed_zonal_instance(project, candidate_zone, instance_name) + if not cleaned: + result["leaked_zones"].append(candidate_zone) + # Reset the per-zone tracker so the next iteration's + # insert ack stamps it fresh. Do NOT null + # result['instance_id'] / result['zone'] — the + # instance_name is deterministic across walker + # attempts (suffix is rolled once before the loop), + # so the stamped value remains a valid teardown + # target whether the walker succeeds later or + # exhausts every candidate. Clearing it broke the + # cleanup-provenance chain on full-walk exhaustion + # (rule #6) — teardown then lost the deterministic + # name and the leaked instance in zone A survived. + instance_created = False + else: + # Sync stockout — no partial state to clean. + result["leaked_zones"].append(candidate_zone) + print(f" walking past {candidate_zone} (stockout-class)", file=sys.stderr) + op = None + op_name = "" + continue + else: + # Exhausted every candidate — raise the most recent + # zone-unavailable error so the operator sees the actual + # cause rather than a generic "no zones tried" message. + raise RuntimeError( + f"Zone-walk exhausted ({len(candidate_zones)} candidates); last error: {last_error}" + ) from last_error + + # 5. Poll canonical 'running'. + print("Waiting for RUNNING status...", file=sys.stderr) + result["state"] = poll_instance_state( + project, + zone, + instance_name, + target_canonical="running", + timeout=300, + ) + + # 6. Re-read instance for IPs + label round-trip. + inst = get_instance(project, zone, instance_name) + result["public_ip"] = first_external_ip(inst) or wait_for_public_ip(project, zone, instance_name, timeout=120) + if not result["public_ip"]: + raise RuntimeError("Instance has no external IP after RUNNING (timed out polling)") + result["private_ip"] = first_internal_ip(inst) + result["vpc_id"] = short_name(inst.network_interfaces[0].network) + if inst.network_interfaces[0].subnetwork: + result["subnet_id"] = short_name(inst.network_interfaces[0].subnetwork) + # Only emit canonical tag keys when the backing label is actually + # present on the live instance. Falling back to the REQUESTED + # values would fabricate a vacuous readback round-trip and mask + # any regression in `canonical_tags_to_labels` projection (rule + # #1 oracle-derivation parity; mirrors the reuse-branch shape). + actual_labels = dict(getattr(inst, "labels", {}) or {}) + derived_tags: dict[str, str] = {} + if "isv_name" in actual_labels: + derived_tags["Name"] = actual_labels["isv_name"] + if "createdby" in actual_labels: + derived_tags["CreatedBy"] = actual_labels["createdby"] + result["tags"] = derived_tags + + # 7. Best-effort readiness gate — SSH OR cloud-init counts as + # success. Failing BOTH is the only honest reason to call launch + # failed at this point. + ssh_ok = wait_for_ssh( + host=result["public_ip"], + user=args.ssh_user, + key_file=key_priv, + max_attempts=20, + interval=10, + ) + cloud_init_ok = False + if ssh_ok: + cloud_init_ok = wait_for_cloud_init( + host=result["public_ip"], + user=args.ssh_user, + key_file=key_priv, + timeout_seconds=600, + ) + # Compute Engine's guest-agent restarts sshd shortly after + # cloud-init completes (refreshes authorized_keys / host + # keys). Downstream validators (e.g. CloudInitCheck) connect + # via paramiko immediately after this step returns and race + # that restart, surfacing as "Error reading SSH protocol + # banner: Connection reset by peer". Require 3 consecutive + # SSH successes here so the post-cloud-init bounce is washed + # out before we hand control back. Mirrors the reuse-branch + # readiness gate. + if cloud_init_ok: + ssh_stable_ok = wait_for_ssh_stable( + host=result["public_ip"], + user=args.ssh_user, + key_file=key_priv, + consecutive=3, + interval=10, + max_attempts=24, + ) + if not ssh_stable_ok: + print( + " SSH did not stabilize after cloud-init; continuing on best-effort", + file=sys.stderr, + ) + result["ssh_stable"] = ssh_stable_ok + result["ssh_ready"] = ssh_ok + result["cloud_init_ok"] = cloud_init_ok + + if not (ssh_ok or cloud_init_ok): + raise RuntimeError( + "Launch reached RUNNING but neither SSH nor cloud-init became observable within the step timeout" + ) + + result["success"] = True + print("Launch succeeded", file=sys.stderr) + + except Exception as e: + result.setdefault("error", str(e)) + result["success"] = False + # Cleanup-on-failure — gate on per-resource trackers so a failed + # verified-reuse adoption doesn't take a pre-existing shared + # resource with it. + try: + if instance_created: + print( + f"Cleanup-on-failure: deleting instance {instance_name}", + file=sys.stderr, + ) + delete_with_retry( + _delete_instance_op, + project, + zone, + instance_name, + resource_desc=f"instance {instance_name}", + ) + if firewall_created: + print( + f"Cleanup-on-failure: deleting firewall {fw_name}", + file=sys.stderr, + ) + delete_with_retry( + _delete_firewall_op, + project, + fw_name, + resource_desc=f"firewall {fw_name}", + ) + if key_created and key_priv: + delete_local_keypair(key_priv) + except Exception as cleanup_exc: + print(f"Cleanup-on-failure error: {cleanup_exc}", file=sys.stderr) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/list_instances.py b/isvctl/configs/providers/gcp/scripts/vm/list_instances.py new file mode 100644 index 00000000..13e6a2b5 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/list_instances.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""List Compute Engine instances filtered by network. + +Translates the AWS oracle's region-scoped ``describe_instances(VPC=...)`` +to Compute Engine's zone-scoped reality: + + * ``instances.list`` is zone-scoped; cross-zone listing requires + ``aggregatedList``. + * Filter aggregatedList output to zones inside the operator-supplied + region (oracle parity — AWS describe_instances doesn't leak in + instances from other regions). + * Network/VPC match is exact-equality on the trailing path segment of + the network self-link (per the scope-binding-comparison rule: + substring / startswith accept supersets). + +The validator (``InstanceListCheck``) only reads ``instances`` and +``found_target``; the canonical state translation comes from +``common.compute.canonical_state``. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + canonical_state, + first_external_ip, + first_internal_ip, + narrow_region_to_zone, + resolve_project, + short_name, + zone_to_region, +) +from common.errors import handle_gcp_errors +from google.cloud import compute_v1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="List Compute Engine instances by network") + parser.add_argument("--vpc-id", required=True, help="Network short name") + parser.add_argument("--instance-id", help="Specific instance to look up") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + args = parser.parse_args() + + project = resolve_project(args.project) + effective_zone = args.zone or narrow_region_to_zone(args.region) + target_region = zone_to_region(effective_zone) + # Compute Engine aggregatedList keys zones as ``zones/-``. + # Use the region prefix so the cross-region instances are filtered out + # — region-scoped oracle vs zone-scoped target: never silently fall back. + zone_prefix = f"zones/{target_region}-" + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instances": [], + "count": 0, + "total_count": 0, + "region": args.region, + "zone": effective_zone, + "project": project, + } + + try: + client = compute_v1.InstancesClient() + request = compute_v1.AggregatedListInstancesRequest( + project=project, + max_results=500, + ) + for zone_key, scoped in client.aggregated_list(request=request): + if not zone_key.startswith(zone_prefix): + continue + for inst in scoped.instances or []: + inst_network = "" + if inst.network_interfaces: + inst_network = short_name(inst.network_interfaces[0].network) + if inst_network != args.vpc_id: + continue + result["instances"].append( + { + "instance_id": inst.name, + "instance_type": short_name(inst.machine_type), + "state": canonical_state(inst.status), + "public_ip": first_external_ip(inst), + "private_ip": first_internal_ip(inst), + "vpc_id": inst_network, + } + ) + + result["count"] = len(result["instances"]) + result["total_count"] = result["count"] + + if args.instance_id: + result["target_instance"] = args.instance_id + result["found_target"] = any(i["instance_id"] == args.instance_id for i in result["instances"]) + + result["success"] = True + + except Exception as e: + result["error"] = str(e) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/reboot_instance.py b/isvctl/configs/providers/gcp/scripts/vm/reboot_instance.py new file mode 100644 index 00000000..54271d78 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/reboot_instance.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Reboot a Compute Engine VM via ``instances.reset`` and affirm recovery. + +Compute Engine has no soft-reboot equivalent of the AWS oracle's +``reboot_instances``; ``instances.reset`` is a HARD reset that returns +before the guest restarts. On async soft-reboot APIs, wait for SSH to +DROP before waiting for it to stabilize: + + 1. Sample pre-reset uptime over SSH (best-effort). + 2. Pre-gate on ``cloud-init status --wait`` so the reset doesn't land + mid-init (pre-lifecycle defensive gate). + 3. Issue ``instances.reset``; record the request timestamp. + 4. Wait for SSH to STOP responding (90s budget) — confirms the pre- + reset sshd has dropped, so subsequent uptime/boot reads cannot + falsely confirm reboot. + 5. Poll canonical 'running'; re-read the public IP from live state. + 6. Stability gate against the post-reset sshd. + 7. Sample post-reset uptime. Confirm via + ``boot_started_at >= reboot_requested_at`` OR + ``post_uptime < pre_uptime`` — emit ``reboot_confirmed`` as a real + bool, never literal True. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + canonical_state, + first_external_ip, + first_internal_ip, + get_instance, + narrow_region_to_zone, + poll_instance_state, + resolve_project, + retry_zonal_lifecycle_op, + wait_for_public_ip, +) +from common.errors import handle_gcp_errors +from common.ssh_utils import ( + get_uptime_via_ssh, + wait_for_cloud_init, + wait_for_ssh_drop, + wait_for_ssh_stable, +) +from google.cloud import compute_v1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Reboot a Compute Engine VM via reset") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument("--key-file", required=True, help="SSH private key path") + parser.add_argument("--public-ip", required=True, help="Pre-reset public IP (re-read after)") + parser.add_argument("--ssh-user", default="ubuntu", help="SSH username") + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "region": args.region, + "zone": zone, + "project": project, + "key_file": args.key_file, + "ssh_user": args.ssh_user, + "reboot_initiated": False, + "ssh_ready": False, + "reboot_confirmed": False, + "ssh_drop_observed": False, + } + + try: + # 1. Pre-check + pre-uptime sample (best-effort: a missing sample + # is recoverable as long as the boot-timestamp check succeeds). + print("Verifying instance is running before reboot...", file=sys.stderr) + inst = get_instance(project, zone, args.instance_id) + cstate = canonical_state(inst.status) + if cstate != "running": + result["state"] = cstate + result["error"] = f"Instance is {cstate!r}, expected running" + print(json.dumps(result, indent=2, default=str)) + return 1 + + pre_uptime = get_uptime_via_ssh(args.public_ip, args.ssh_user, args.key_file) + if pre_uptime is not None: + result["pre_reboot_uptime"] = round(pre_uptime, 1) + print(f" pre-reboot uptime: {pre_uptime:.0f}s", file=sys.stderr) + + # 2. Pre-gate on cloud-init wait — refuse to reset mid-init. + cloud_init_pre = wait_for_cloud_init( + host=args.public_ip, + user=args.ssh_user, + key_file=args.key_file, + timeout_seconds=600, + ) + result["cloud_init_pre_reboot"] = cloud_init_pre + if not cloud_init_pre: + result["error"] = ( + "cloud-init did not complete cleanly (rc != 0/2) before reset; " + "refusing to reset a guest in an unsettled state" + ) + print(json.dumps(result, indent=2, default=str)) + return 1 + + # 3. Reset. Stamp the request timestamp BEFORE the API call so + # the post-reset boot_started_at >= reboot_requested_at check + # has a stable comparison anchor. Lifecycle ops are zone-bound + # (cannot walk on STOCKOUT); wrap sync+wait in the in-zone + # retry-with-backoff envelope (3 attempts, 60s/120s backoff). + print(f"Resetting instance {args.instance_id}...", file=sys.stderr) + reboot_requested_at = time.time() + + def _stamp_reboot_initiated() -> None: + result["reboot_initiated"] = True + + client = compute_v1.InstancesClient() + retry_zonal_lifecycle_op( + lambda: client.reset(project=project, zone=zone, instance=args.instance_id), + project, + zone, + resource_desc=f"reset {args.instance_id}", + on_sync_success=_stamp_reboot_initiated, + op_timeout=300, + ) + + # 4. SSH-drop wait — best-effort observability signal. If the + # guest reboots and sshd recovers faster than our probe interval, + # we may miss the drop window entirely (the budget extends past + # the canonical reset+boot timeline of a g2-standard-8 in + # practice). The boot_started_at >= reboot_requested_at gate + # below is the LOAD-BEARING reboot signal — it is robust whether + # or not we observe the drop, because a lingering pre-reset + # uptime would compute to a boot_started_at far earlier than + # reboot_requested_at and the gate would correctly reject the + # reboot. Matches the AWS oracle's no-SSH-drop pattern while + # preserving the observation for diagnostics. + print("Waiting for pre-reset SSH to drop (best-effort)...", file=sys.stderr) + drop_observed = wait_for_ssh_drop( + host=args.public_ip, + user=args.ssh_user, + key_file=args.key_file, + max_attempts=18, + interval=5, + ) + result["ssh_drop_observed"] = drop_observed + if not drop_observed: + print( + " WARNING: SSH-drop not observed within budget; relying on " + "boot_started_at >= reboot_requested_at to adjudicate reboot", + file=sys.stderr, + ) + + # 5. Poll canonical 'running' + re-read public IP from live state. + print("Waiting for canonical 'running' state...", file=sys.stderr) + result["state"] = poll_instance_state( + project, + zone, + args.instance_id, + target_canonical="running", + timeout=300, + ) + inst = get_instance(project, zone, args.instance_id) + result["private_ip"] = first_internal_ip(inst) + public_ip = first_external_ip(inst) or wait_for_public_ip(project, zone, args.instance_id, timeout=120) + if not public_ip: + result["error"] = "Instance has no external IP after reset (timed out polling)" + print(json.dumps(result, indent=2, default=str)) + return 1 + result["public_ip"] = public_ip + + # 6. Stability gate against the fresh sshd. + print("Waiting for post-reset SSH to stabilize...", file=sys.stderr) + ssh_ok = wait_for_ssh_stable( + host=public_ip, + user=args.ssh_user, + key_file=args.key_file, + consecutive=3, + interval=10, + max_attempts=36, + ) + result["ssh_ready"] = ssh_ok + if not ssh_ok: + result["error"] = "SSH did not stabilize after reset" + print(json.dumps(result, indent=2, default=str)) + return 1 + + # 7. Post-reset uptime. The boot-started-at comparison is the + # primary signal; uptime-decreased is a fallback when the + # pre-reset sample succeeded. + post_uptime = get_uptime_via_ssh(public_ip, args.ssh_user, args.key_file) + if post_uptime is None: + result["error"] = "Could not sample post-reset uptime via SSH" + print(json.dumps(result, indent=2, default=str)) + return 1 + result["uptime_seconds"] = round(post_uptime, 1) + print(f" post-reboot uptime: {post_uptime:.0f}s", file=sys.stderr) + + boot_started_at = time.time() - post_uptime + if boot_started_at >= reboot_requested_at: + result["reboot_confirmed"] = True + print(" reboot confirmed (boot time follows reset request)", file=sys.stderr) + elif pre_uptime is not None and post_uptime < pre_uptime: + result["reboot_confirmed"] = True + print(" reboot confirmed (uptime decreased)", file=sys.stderr) + else: + result["reboot_confirmed"] = False + result["error"] = ( + "Reboot not affirmed: post-reset boot time precedes reset request " + "and pre-reset uptime sample missing or did not decrease" + ) + + result["success"] = result["reboot_confirmed"] + if result["success"]: + print("Reboot completed", file=sys.stderr) + + except Exception as e: + result["error"] = str(e) + print(f"ERROR: {e}", file=sys.stderr) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/serial_console.py b/isvctl/configs/providers/gcp/scripts/vm/serial_console.py new file mode 100644 index 00000000..548ade70 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/serial_console.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Probe Compute Engine serial console output. + +Compute Engine has NO account-level toggle equivalent to AWS's +``get_serial_console_access_status`` — access is gated by IAM. Both +``console_available`` and ``serial_access_enabled`` are derived from a +single ``getSerialPortOutput`` permission probe: + + * ``serial_access_enabled``: True iff the call succeeded under the + active credentials (False on PermissionDenied / Unauthenticated). + * ``console_available``: True iff returned ``contents`` are non-empty. + +Validator-consumed fields must derive from a real signal: no field is +hardcoded; both flip to False on real permission/auth failures so the +validator sees the actual platform state on this run. The AWS oracle requires step success only when at +least one probe yielded a usable result, so this stub matches that +contract: ``success = console_available or serial_access_enabled``. +Permission/auth denials with no readable output exit rc=1 with a +structured error so the orchestrator sees an honest failure rather than +a green step on a denied probe. + +This stub uses the explicit ``GetSerialPortOutputInstanceRequest`` +Request object rather than the flattened kwarg form, because the SDK +does NOT accept ``port=`` as a flattened kwarg. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + narrow_region_to_zone, + resolve_project, +) +from common.errors import handle_gcp_errors +from google.api_core import exceptions as gax +from google.cloud import compute_v1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Probe Compute Engine serial console") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument( + "--port", + type=int, + default=1, + help="Serial port (1..4); 1 is the default boot console", + ) + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "console_available": False, + "serial_access_enabled": False, + "output_length": 0, + "region": args.region, + "zone": zone, + "project": project, + } + + try: + client = compute_v1.InstancesClient() + request = compute_v1.GetSerialPortOutputInstanceRequest( + project=project, + zone=zone, + instance=args.instance_id, + port=args.port, + ) + try: + response = client.get_serial_port_output(request=request) + result["serial_access_enabled"] = True + contents = response.contents or "" + result["output_length"] = len(contents) + if contents: + result["console_available"] = True + result["output_snippet"] = contents[-500:] if len(contents) > 500 else contents + except (gax.PermissionDenied, gax.Unauthenticated) as e: + result["serial_access_enabled"] = False + result["error"] = f"Serial console access denied: {e}" + except gax.NotFound as e: + result["error"] = f"Instance not found: {e}" + + # Per the AWS oracle: step succeeds only when at least one probe + # returned a usable result. A denied probe with no readable output + # is an honest failure for the validator to see, not a green step. + result["success"] = bool(result["console_available"] or result["serial_access_enabled"]) + + except Exception as e: + result["error"] = str(e) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/start_instance.py b/isvctl/configs/providers/gcp/scripts/vm/start_instance.py new file mode 100644 index 00000000..be81f0e3 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/start_instance.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Start a stopped Compute Engine VM and gate success on a stable guest. + +Divergences from the AWS oracle: + * Compute Engine reports raw ``TERMINATED`` for the canonical stopped + state; use ``canonical_state(...)``. + * Ephemeral external IPs are RELEASED on stop. ``--public-ip`` may be + forwarded for diagnostics, but every post-start emission MUST come + from a fresh ``instances.get`` / ``wait_for_public_ip`` read — + public IP is NOT preserved across stop/start on Compute Engine. + * First-SSH-success is not enough: the guest agent may rewrite + authorized_keys mid-cloud-init replay. Gate on (1) cloud-init + completion AND (2) N consecutive successful SSH probes — + post-lifecycle steps gate on stability, not first SSH success. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + canonical_state, + first_external_ip, + first_internal_ip, + get_instance, + narrow_region_to_zone, + poll_instance_state, + resolve_project, + retry_zonal_lifecycle_op, + wait_for_public_ip, +) +from common.errors import handle_gcp_errors +from common.ssh_utils import wait_for_cloud_init, wait_for_ssh_stable +from google.cloud import compute_v1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Start a stopped Compute Engine VM") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument("--key-file", required=True, help="SSH private key path") + parser.add_argument( + "--public-ip", + default=None, + help="Pre-stop public IP (informational; re-read after start)", + ) + parser.add_argument("--ssh-user", default="ubuntu", help="SSH username") + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "region": args.region, + "zone": zone, + "project": project, + "key_file": args.key_file, + "ssh_user": args.ssh_user, + "start_initiated": False, + "ssh_ready": False, + } + + try: + # 1. Pre-check current state. The AWS oracle and GCP knowledge + # both require the canonical stopped state before issuing start; + # a previously-running VM or a skipped/no-op stop step must not + # produce a green start, otherwise the lifecycle test no longer + # proves stop→start behavior (oracle parity rule #1). + print("Verifying instance is stopped before start...", file=sys.stderr) + inst = get_instance(project, zone, args.instance_id) + cstate = canonical_state(inst.status) + + if cstate != "stopped": + result["state"] = cstate + result["error"] = f"Instance is {cstate!r}, expected stopped" + print(json.dumps(result, indent=2, default=str)) + return 1 + + # 2. Start; wait on zonal op then poll for canonical 'running'. + # Lifecycle ops are zone-bound (cannot walk on STOCKOUT) — wrap + # the sync+wait pair in the in-zone retry-with-backoff envelope + # (zone_capacity_handling: 3 attempts, 60s/120s backoff). The + # post-API stamp keeps + # start_initiated tied to a real API acknowledgement rather than + # firing speculatively. + print(f"Starting instance {args.instance_id}...", file=sys.stderr) + + def _stamp_start_initiated() -> None: + result["start_initiated"] = True + + client = compute_v1.InstancesClient() + retry_zonal_lifecycle_op( + lambda: client.start(project=project, zone=zone, instance=args.instance_id), + project, + zone, + resource_desc=f"start {args.instance_id}", + on_sync_success=_stamp_start_initiated, + ) + + print("Waiting for canonical 'running' state...", file=sys.stderr) + result["state"] = poll_instance_state( + project, + zone, + args.instance_id, + target_canonical="running", + timeout=300, + ) + + # 3. Re-read details from live state — public IP is the critical + # one because Compute Engine releases the ephemeral on stop and + # assigns a fresh one on start. + inst = get_instance(project, zone, args.instance_id) + result["private_ip"] = first_internal_ip(inst) + fresh_ip = first_external_ip(inst) or wait_for_public_ip(project, zone, args.instance_id, timeout=120) + if not fresh_ip: + result["error"] = "Instance has no external IP after start (timed out polling)" + print(json.dumps(result, indent=2, default=str)) + return 1 + result["public_ip"] = fresh_ip + + # 4. Stability gate. Consecutive SSH successes + cloud-init wait. + print("Waiting for SSH to stabilize after start...", file=sys.stderr) + ssh_ok = wait_for_ssh_stable( + host=fresh_ip, + user=args.ssh_user, + key_file=args.key_file, + consecutive=3, + interval=10, + max_attempts=36, + ) + result["ssh_ready"] = ssh_ok + if not ssh_ok: + result["error"] = "SSH did not stabilize after start" + print(json.dumps(result, indent=2, default=str)) + return 1 + + cloud_init_ok = wait_for_cloud_init( + host=fresh_ip, + user=args.ssh_user, + key_file=args.key_file, + timeout_seconds=600, + ) + result["cloud_init_ok"] = cloud_init_ok + if not cloud_init_ok: + result["error"] = "cloud-init did not complete after start (rc != 0/2)" + print(json.dumps(result, indent=2, default=str)) + return 1 + + result["success"] = True + print("Start completed", file=sys.stderr) + + except Exception as e: + result["error"] = str(e) + print(f"ERROR: {e}", file=sys.stderr) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/stop_instance.py b/isvctl/configs/providers/gcp/scripts/vm/stop_instance.py new file mode 100644 index 00000000..e34326bc --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/stop_instance.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Stop a Compute Engine VM and verify the canonical 'stopped' state. + +Divergences from the AWS oracle: + * Compute Engine accepts ``instances.stop`` mid-cloud-init, leaving + the guest dirty on next boot. Pre-gate on ``cloud-init status + --wait`` over SSH (exit codes 0 and 2 are terminal). + * ``instances.stop`` returns a zonal Operation — wait on the op, + then poll ``instances.get`` until ``canonical_state == 'stopped'`` + (Compute Engine reports raw ``TERMINATED``). +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + canonical_state, + get_instance, + narrow_region_to_zone, + poll_instance_state, + resolve_project, + retry_zonal_lifecycle_op, +) +from common.errors import handle_gcp_errors +from common.ssh_utils import wait_for_cloud_init +from google.cloud import compute_v1 + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Stop a Compute Engine VM") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument( + "--key-file", + default=None, + help="SSH private key for the pre-stop cloud-init wait", + ) + parser.add_argument("--public-ip", default=None, help="Pre-stop public IP") + parser.add_argument("--ssh-user", default="ubuntu", help="SSH username") + args = parser.parse_args() + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "instance_id": args.instance_id, + "region": args.region, + "zone": zone, + "project": project, + "stop_initiated": False, + "cloud_init_pre_stop": None, + } + + try: + # 1. Pre-check current state. Idempotent no-op when already stopped. + print("Checking instance state before stop...", file=sys.stderr) + inst = get_instance(project, zone, args.instance_id) + cstate = canonical_state(inst.status) + + if cstate == "stopped": + # Idempotent no-op: instance is already stopped; we did not + # issue a stop request, so honestly report stop_initiated=False + # (rule #9: every reported boolean reflects a real action, + # not an aspirational success flag). + result["state"] = cstate + result["stop_initiated"] = False + result["success"] = True + print(f" {args.instance_id} already stopped (no-op)", file=sys.stderr) + print(json.dumps(result, indent=2, default=str)) + return 0 + + if cstate != "running": + result["state"] = cstate + result["error"] = f"Instance is {cstate!r}, expected running" + print(json.dumps(result, indent=2, default=str)) + return 1 + + # 2. Pre-gate on cloud-init wait. Compute Engine accepts stop mid- + # cloud-init, but the next boot will be dirty. If SSH ingredients + # are supplied, the bool MUST be surfaced — helpers that return + # ``bool`` for batch-cleanup safety MUST surface the bool into + # ``result['success']``. + if args.public_ip and args.key_file: + print("Pre-gating stop on cloud-init wait...", file=sys.stderr) + cloud_init_pre = wait_for_cloud_init( + host=args.public_ip, + user=args.ssh_user, + key_file=args.key_file, + timeout_seconds=600, + ) + result["cloud_init_pre_stop"] = cloud_init_pre + if not cloud_init_pre: + result["error"] = ( + "cloud-init did not complete cleanly (rc != 0/2) before stop; refusing to stop a dirty guest" + ) + print(json.dumps(result, indent=2, default=str)) + return 1 + + # 3. Stop. The op is zonal — wait on completion, then poll state. + # Lifecycle ops are zone-bound (cannot walk on STOCKOUT); the + # in-zone retry envelope wraps sync+wait (3 attempts, 60s/120s + # backoff) and stamps stop_initiated on first sync success. + print(f"Stopping instance {args.instance_id}...", file=sys.stderr) + + def _stamp_stop_initiated() -> None: + result["stop_initiated"] = True + + client = compute_v1.InstancesClient() + retry_zonal_lifecycle_op( + lambda: client.stop(project=project, zone=zone, instance=args.instance_id), + project, + zone, + resource_desc=f"stop {args.instance_id}", + on_sync_success=_stamp_stop_initiated, + ) + + # 4. Poll canonical 'stopped' (Compute Engine raw 'TERMINATED'). + print("Waiting for canonical 'stopped' state...", file=sys.stderr) + result["state"] = poll_instance_state( + project, + zone, + args.instance_id, + target_canonical="stopped", + timeout=600, + ) + result["success"] = True + print("Stop completed", file=sys.stderr) + + except Exception as e: + result["error"] = str(e) + print(f"ERROR: {e}", file=sys.stderr) + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/teardown.py b/isvctl/configs/providers/gcp/scripts/vm/teardown.py new file mode 100644 index 00000000..b8afb9be --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/teardown.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Teardown a Compute Engine VM + verified-reuse companions. + +Mirrors the AWS oracle's teardown.py shape (instance + SG + key pair), +translated to Compute Engine: + + * ``instances.delete`` is zonal; ``firewalls.delete`` is project-global. + * Instance, firewall, and local key pair are ALL verified-reuse — + destruction MUST gate on the ``_created: bool`` flags forwarded + from launch_instance via ``{{steps.launch_instance.X}}`` (cleanup + contract). For the instance this + means a run started with ``GCP_VM_INSTANCE_ID`` / ``GCP_VM_KEY_FILE`` + against an operator-supplied long-lived VM emits + ``instance_created=False`` and teardown skips both the primary and + every leaked-zone delete so the adopted VM survives. + * ``--skip-destroy`` short-circuits to success BEFORE resolving the + project, so an expired-ADC environment can still no-op cleanly — + preservation-mode flags MUST be evaluated before any auth-resolving + helper. + * NotFound on the cloud-side preflight is idempotent SUCCESS for the + instance read, but must NOT short-circuit local PEM/.pub cleanup — + NotFound-on-cloud-read idempotency must not short-circuit + local-artifact cleanup. + * Each cleanup helper returns ``bool``; the final ``success`` is the + AND of every per-resource bool — helpers that return ``bool`` for + batch-cleanup safety MUST surface the bool into + ``result['success']``. + +Sentinel handling: the provider config wires bool / path args with the +non-empty defaults ``'none'`` / ``'false'`` — forwarded inter-step +Jinja args MUST use ``| default()``. The stub +treats ``none`` / ``null`` / ``""`` / ``false`` as "no artifact +tracked". +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.compute import ( + delete_local_keypair, + get_instance, + narrow_region_to_zone, + resolve_project, + wait_for_global_op, + wait_for_zonal_op, +) +from common.errors import delete_with_retry, handle_gcp_errors +from google.api_core import exceptions as gax +from google.cloud import compute_v1 + +_FALSY_SENTINELS = {"", "none", "null", "false"} + +# Bound the per-attempt wait so the 3-attempt delete_with_retry +# does not multiply zonal/global operation budgets into the +# enclosing teardown step timeout. The leaked-zones walk iterates over +# multiple zones with the same retry helper; without this bound, a +# transient throttle on a single zone could exhaust the step budget +# before later zones are even attempted. Firewall deletes have exceeded +# 120s in live GCP runs, so keep the global-op wait comfortably above +# that observed path while still inside the enclosing teardown budget. +_TEARDOWN_INSTANCE_WAIT_S = 180 +_TEARDOWN_FIREWALL_WAIT_S = 300 + + +def _truthy(arg: str | None) -> bool: + """Per-arg sentinel check. Treats both "" / 'none' / 'null' / 'false' as falsy.""" + if arg is None: + return False + return arg.strip().lower() not in _FALSY_SENTINELS + + +def _delete_instance_op(project: str, zone: str, name: str) -> None: + """Delete an instance and wait on the zonal op (NotFound is idempotent).""" + try: + op = compute_v1.InstancesClient().delete(project=project, zone=zone, instance=name) + except gax.NotFound: + return + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + wait_for_zonal_op(project, zone, op_name, timeout=_TEARDOWN_INSTANCE_WAIT_S) + + +def _delete_firewall_op(project: str, name: str) -> None: + """Delete a firewall rule and wait on the global op (NotFound is idempotent).""" + try: + op = compute_v1.FirewallsClient().delete(project=project, firewall=name) + except gax.NotFound: + return + op_name = getattr(op, "name", None) or getattr(op, "operation", "") + if op_name: + wait_for_global_op(project, op_name, timeout=_TEARDOWN_FIREWALL_WAIT_S) + + +@handle_gcp_errors +def main() -> int: + parser = argparse.ArgumentParser(description="Teardown a Compute Engine VM + companions") + parser.add_argument("--instance-id", required=True, help="Instance name") + parser.add_argument("--region", required=True, help="GCP region or zone") + parser.add_argument("--zone", default=None, help="GCP zone (overrides region)") + parser.add_argument("--project", default=None, help="GCP project ID (ADC fallback)") + parser.add_argument( + "--delete-key-pair", + action="store_true", + help="Delete the local SSH key pair if --key-created is truthy", + ) + parser.add_argument( + "--delete-security-group", + action="store_true", + help="Delete the SSH firewall rule if --firewall-created is truthy", + ) + parser.add_argument( + "--skip-destroy", + action="store_true", + help="Short-circuit to success (preserve cloud state) BEFORE resolving auth", + ) + parser.add_argument("--firewall-name", default="none", help="Firewall rule name") + parser.add_argument( + "--firewall-created", + default="false", + help="Bool sentinel forwarded from launch_instance.firewall_created", + ) + parser.add_argument( + "--instance-created", + default="false", + help=( + "Bool sentinel forwarded from launch_instance.instance_created. " + "False skips both the primary and every leaked-zone instance " + "delete so a verified-reuse adoption of an operator-supplied " + "long-lived VM is never destroyed by this teardown." + ), + ) + parser.add_argument( + "--key-file", + default="none", + help="Local SSH PEM path forwarded from launch_instance.key_file", + ) + parser.add_argument( + "--key-created", + default="false", + help="Bool sentinel forwarded from launch_instance.key_created", + ) + parser.add_argument( + "--leaked-zones", + default="", + help=( + "Comma-separated zones the multi-zone walker accumulated " + "partial-create leaks in. Teardown best-effort-deletes the " + "instance in each before completing." + ), + ) + args = parser.parse_args() + + result: dict[str, Any] = { + "success": False, + "platform": "vm", + "resources_destroyed": False, + "deleted": { + "instances": [], + "firewall_rules": [], + "key_files": [], + }, + "resources_deleted": [], # flat list shape matching AWS oracle / my-isv + "message": "", + } + + # Preservation-mode flag short-circuits BEFORE any cloud / auth call + # so an expired-credentials environment still no-ops cleanly. + if args.skip_destroy: + result["success"] = True + result["instance_id"] = args.instance_id + result["message"] = f"Instance {args.instance_id} preserved (--skip-destroy); delete manually when done." + print(json.dumps(result, indent=2, default=str)) + return 0 + + project = resolve_project(args.project) + zone = args.zone or narrow_region_to_zone(args.region) + + firewall_created = _truthy(args.firewall_created) + key_created = _truthy(args.key_created) + instance_created = _truthy(args.instance_created) + instance_id = args.instance_id if _truthy(args.instance_id) else None + fw_name = args.firewall_name if _truthy(args.firewall_name) else None + key_file = args.key_file if _truthy(args.key_file) else None + + # Per-resource booleans surfaced into the final success. + instance_ok = True + firewall_ok = True + key_ok = True + + # 1. Instance delete. Each preflight read is scoped narrow so a + # transient probe error doesn't poison sibling cleanup blocks — + # teardown preflight reads MUST NOT share an exception handler with + # the cleanup blocks. + # + # The verified-reuse ownership gate (instance_created) bypasses + # the preflight entirely — there is no observable difference + # between "adopted, still present" and "we created and it's still + # present", so we MUST trust the forwarded ownership bit rather + # than the live state. + instance_present = False + if instance_id and not instance_created: + print( + f"Skipping instance delete for {instance_id} (instance_created=false; " + "verified-reuse adoption — never destroy resources this run did not create)", + file=sys.stderr, + ) + result.setdefault("warnings", []).append( + f"instance {instance_id} preserved (verified-reuse adoption: instance_created=false)" + ) + elif instance_id: + print(f"Deleting instance {instance_id} in {zone}...", file=sys.stderr) + try: + get_instance(project, zone, instance_id) + instance_present = True + except gax.NotFound: + print(f" instance {instance_id} already absent (NotFound)", file=sys.stderr) + result.setdefault("warnings", []).append(f"instance {instance_id} not found at teardown — already deleted") + except Exception as e: + # Transient/API error during preflight: treat as present and let + # delete_with_retry handle NotFound idempotency. + print(f" warn: instance preflight failed: {e}", file=sys.stderr) + result.setdefault("warnings", []).append(f"instance preflight read failed: {e}") + instance_present = True + else: + print("Skipping instance delete (no instance id was produced)", file=sys.stderr) + + if instance_present: + assert instance_id is not None + instance_ok = delete_with_retry( + _delete_instance_op, + project, + zone, + instance_id, + resource_desc=f"instance {instance_id}", + ) + if instance_ok: + result["deleted"]["instances"].append(instance_id) + result["resources_deleted"].append(f"instance:{instance_id}") + + # 1b. The multi-zone walker may have accumulated zones where a partial + # async insert leaked; best-effort delete in each so phantom records do + # not survive the run. Drop falsy sentinels so the per-zone delete loop + # only walks real zone strings. + leaked = [ + z.strip() + for z in (args.leaked_zones or "").split(",") + if z.strip() and z.strip().lower() not in _FALSY_SENTINELS + ] + if instance_id and not instance_created and leaked: + # Verified-reuse adoption never invoked the multi-zone walker + # (the walker runs only on the create path), so a leaked_zones + # list arriving here is impossible under normal flow. If + # something upstream wires it anyway, refuse to touch a + # not-ours name in any zone. + result.setdefault("warnings", []).append( + f"leaked-zone cleanup skipped: instance_created=false (preserving adopted {instance_id})" + ) + elif instance_id and instance_created: + for leak_zone in leaked: + if leak_zone == zone: + continue # already handled above + print(f"Leaked-zone cleanup: instance {instance_id} in {leak_zone}", file=sys.stderr) + leak_ok = delete_with_retry( + _delete_instance_op, + project, + leak_zone, + instance_id, + resource_desc=f"instance {instance_id}@{leak_zone}", + ) + # Leaked-zone failure surfaces into the aggregate success so the + # operator sees an honest partial-cleanup verdict; the per-zone + # delete is best-effort but its outcome is NOT swallowed. + if not leak_ok: + instance_ok = False + result.setdefault("warnings", []).append(f"leaked-zone delete failed: {instance_id}@{leak_zone}") + else: + result["deleted"]["instances"].append(f"{instance_id}@{leak_zone}") + result["resources_deleted"].append(f"instance:{instance_id}@{leak_zone}") + elif leaked: + result.setdefault("warnings", []).append( + f"leaked zones ignored because no instance id was produced: {', '.join(leaked)}" + ) + + # 2. Firewall — gated on the verified-reuse flag forwarded by + # launch_instance. NotFound is idempotent success; transient is + # local-only (does not bypass key cleanup below). + if args.delete_security_group: + if firewall_created and fw_name: + firewall_present = False + try: + compute_v1.FirewallsClient().get(project=project, firewall=fw_name) + firewall_present = True + except gax.NotFound: + print(f" firewall {fw_name} already absent (NotFound)", file=sys.stderr) + except Exception as e: + print(f" warn: firewall preflight failed: {e}", file=sys.stderr) + result.setdefault("warnings", []).append(f"firewall preflight read failed: {e}") + firewall_present = True + + if firewall_present: + print(f"Deleting firewall rule {fw_name}...", file=sys.stderr) + firewall_ok = delete_with_retry( + _delete_firewall_op, + project, + fw_name, + resource_desc=f"firewall {fw_name}", + ) + if firewall_ok: + result["deleted"]["firewall_rules"].append(fw_name) + result["resources_deleted"].append(f"firewall_rule:{fw_name}") + else: + print( + " skipping firewall delete (firewall_created=false or no name)", + file=sys.stderr, + ) + + # 3. Local SSH key pair — gated on key_created. Runs regardless of + # the instance preflight outcome (cloud-side NotFound must NOT + # short-circuit local cleanup). ``delete_local_keypair`` handles + # both halves of the pair so the .pub is removed even when the PEM + # was already gone from a prior run. + if args.delete_key_pair: + if key_created and key_file: + pub_path = key_file + ".pub" + priv_present = os.path.exists(key_file) + pub_present = os.path.exists(pub_path) + if priv_present or pub_present: + print( + f"Deleting local SSH key pair: {key_file} (priv={priv_present}, pub={pub_present})", + file=sys.stderr, + ) + key_ok = delete_local_keypair(key_file) + if key_ok: + if priv_present: + result["deleted"]["key_files"].append(key_file) + result["resources_deleted"].append(f"key_file:{key_file}") + if pub_present: + result["deleted"]["key_files"].append(pub_path) + result["resources_deleted"].append(f"key_file:{pub_path}") + else: + print(f" local SSH key pair already absent: {key_file} + .pub", file=sys.stderr) + else: + print( + " skipping local key cleanup (key_created=false or no path)", + file=sys.stderr, + ) + + # 4. Surface every per-resource bool into final success. + result["success"] = bool(instance_ok and firewall_ok and key_ok) + result["resources_destroyed"] = result["success"] + if result["success"]: + result["message"] = "Instance and verified-reuse companions deleted" + else: + result["message"] = f"Cleanup partial: instance_ok={instance_ok}, firewall_ok={firewall_ok}, key_ok={key_ok}" + + print(json.dumps(result, indent=2, default=str)) + return 0 if result["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/gcp/scripts/vm/virtual_device_hardening.py b/isvctl/configs/providers/gcp/scripts/vm/virtual_device_hardening.py new file mode 100644 index 00000000..a6af2d58 --- /dev/null +++ b/isvctl/configs/providers/gcp/scripts/vm/virtual_device_hardening.py @@ -0,0 +1,302 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: LicenseRef-NvidiaProprietary + +"""Validate Compute Engine VM virtual-device hardening evidence. + +Compute Engine does not expose customer-facing USB redirection or +shared-clipboard controls for tenant VMs (equivalent posture to EC2 — +attached devices are persistent disks, NICs, and local SSD when +configured). This script records that provider evidence and, when SSH +details are available, adds conservative guest-side probes for USB +controllers/devices, clipboard agents, and desktop-style virtual +peripherals. + +The PROBE_SENTINEL framing + pattern tuples + REQUIRED_TESTS list are +reused verbatim from the AWS oracle because the probes are Linux-level +guest commands with no cloud-vendor surface. Only the provider-evidence +preamble names Compute Engine (not EC2). + +Usage: + python3 virtual_device_hardening.py --instance-id \ + --region --public-ip --key-file +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # providers/gcp/scripts/ + +from common.errors import handle_gcp_errors +from common.ssh_utils import ssh_run + +CLIPBOARD_PATTERNS = ( + "spice-vdagent", + "vdagent", + "xrdp-chansrv", + "vncconfig", + "vmtoolsd", +) +UNNECESSARY_DEVICE_PATTERNS = ( + "floppy", + "cd-rom", + "cdrom", + "qxl", + "spice", + "open-vm-tools", + "vgauth", + "vmware", + "virtualbox", + "vbox", + "tablet", + "audio", +) +USB_DEVICE_PATTERNS = ("usb controller", "usb host") +REQUIRED_TESTS = ( + "usb_devices_disabled", + "clipboard_disabled", + "unnecessary_virtual_devices_absent", +) + +PROBE_SENTINEL = "---ISVCTL-PROBE---" +PROBES: tuple[tuple[str, str], ...] = ( + ( + "usb_count", + "if [ -d /sys/bus/usb/devices ]; then " + "find /sys/bus/usb/devices -mindepth 1 -maxdepth 1 -type l -print 2>/dev/null | wc -l; " + "else echo 0; fi", + ), + ("pci_devices", "command -v lspci >/dev/null 2>&1 && lspci || true"), + ("processes", "ps -eo comm= 2>/dev/null || true"), + ( + "services", + "command -v systemctl >/dev/null 2>&1 && " + "systemctl list-units --type=service --state=running --no-pager --output=json 2>/dev/null || true", + ), + ( + "device_paths", + "find /dev -maxdepth 1 \\( -name fd0 -o -name sr0 -o -name cdrom -o -name dvd \\) -print 2>/dev/null || true", + ), +) + +SIGNAL_BINDINGS: tuple[tuple[str, str, str], ...] = ( + ("usb_signals", "usb_devices_disabled", "USB device/controller signals detected"), + ("clipboard_signals", "clipboard_disabled", "Clipboard-sharing agent signals detected"), + ( + "unnecessary_device_signals", + "unnecessary_virtual_devices_absent", + "Unnecessary virtual device signals detected", + ), +) + + +def _compact(text: str, max_length: int = 240) -> str: + """Collapse whitespace and cap length for one-line diagnostics.""" + compact = " ".join(text.split()) + if len(compact) <= max_length: + return compact + return f"{compact[: max_length - 3]}..." + + +def _matching_lines(text: str, patterns: tuple[str, ...]) -> list[str]: + """Return lines containing any case-insensitive pattern.""" + matches: list[str] = [] + for line in text.splitlines(): + stripped = line.strip() + if not stripped: + continue + lowered = stripped.lower() + if any(pattern in lowered for pattern in patterns): + matches.append(stripped) + return matches + + +def _running_service_unit_names(systemctl_output: str) -> list[str]: + """Return running service unit names from ``systemctl --output=json``.""" + text = systemctl_output.strip() + if not text: + return [] + + try: + units = json.loads(text) + except json.JSONDecodeError as e: + msg = "systemctl did not return valid JSON" + raise ValueError(msg) from e + + if not isinstance(units, list): + msg = "systemctl JSON output must be a list" + raise ValueError(msg) + + services: list[str] = [] + for unit in units: + if not isinstance(unit, dict): + continue + normalized = {str(key).lower(): value for key, value in unit.items()} + unit_name = str(normalized.get("unit") or normalized.get("name") or "") + if not unit_name.endswith(".service"): + continue + active = str(normalized.get("active") or "").lower() + sub = str(normalized.get("sub") or "").lower() + if active != "active" or sub != "running": + continue + services.append(unit_name) + return services + + +def _combined_probe_script() -> str: + """Build one shell script that emits every probe's output between sentinel markers.""" + parts: list[str] = [] + for name, command in PROBES: + parts.append(f"echo '{PROBE_SENTINEL} {name}'") + parts.append(f"({command})") + return "\n".join(parts) + + +def _split_probe_outputs(combined: str) -> dict[str, str]: + """Split combined probe output back into a per-probe dict.""" + outputs: dict[str, list[str]] = {name: [] for name, _ in PROBES} + current: str | None = None + for line in combined.splitlines(): + if line.startswith(PROBE_SENTINEL): + current = line[len(PROBE_SENTINEL) :].strip() + continue + if current in outputs: + outputs[current].append(line) + return {name: "\n".join(lines) for name, lines in outputs.items()} + + +def _run_combined_probe(host: str, user: str, key_file: str, timeout: int) -> tuple[dict[str, str] | None, str | None]: + """Run every guest probe in one SSH session. Returns (outputs, error).""" + exit_code, stdout, stderr = ssh_run( + host, + user, + key_file, + _combined_probe_script(), + timeout=timeout, + connect_timeout=min(timeout, 10), + ) + if exit_code != 0: + return None, _compact(stderr or stdout or f"SSH command exited {exit_code}") + return _split_probe_outputs(stdout), None + + +def _collect_guest_probe(host: str, user: str, key_file: str, timeout: int) -> dict[str, Any]: + """Collect optional guest-side virtual-device hardening evidence.""" + if not host or not key_file: + return {"status": "skipped", "reason": "missing SSH details"} + + outputs, error = _run_combined_probe(host, user, key_file, timeout) + if error is not None: + return {"status": "unavailable", "error": error} + if outputs is None: + return {"status": "unavailable", "error": "guest probe returned no output"} + + usb_count_raw = outputs.get("usb_count", "0").strip() + if not usb_count_raw.isdigit(): + msg = f"Expected integer output, got {usb_count_raw!r}" + raise ValueError(msg) + usb_count = int(usb_count_raw) + pci_devices = outputs.get("pci_devices", "") + processes = outputs.get("processes", "") + services = "\n".join(_running_service_unit_names(outputs.get("services", ""))) + device_paths = outputs.get("device_paths", "") + + usb_signals = [f"usb device entries present: {usb_count}"] if usb_count else [] + usb_signals.extend(_matching_lines(pci_devices, USB_DEVICE_PATTERNS)) + + clipboard_signals = _matching_lines(f"{processes}\n{services}", CLIPBOARD_PATTERNS) + unnecessary_signals = _matching_lines(f"{pci_devices}\n{processes}\n{services}", UNNECESSARY_DEVICE_PATTERNS) + unnecessary_signals.extend(line.strip() for line in device_paths.splitlines() if line.strip()) + + return { + "status": "completed", + "usb_device_count": usb_count, + "usb_signals": usb_signals, + "clipboard_signals": clipboard_signals, + "unnecessary_device_signals": unnecessary_signals, + } + + +def _base_tests() -> dict[str, dict[str, Any]]: + """Return passing provider-control evidence before optional guest probes.""" + return { + "usb_devices_disabled": { + "passed": True, + "probes": ["gce_no_customer_usb_redirection_api"], + "message": "Compute Engine exposes no tenant-facing USB redirection or attach surface", + }, + "clipboard_disabled": { + "passed": True, + "probes": ["gce_no_shared_clipboard_api"], + "message": "Compute Engine exposes no tenant-facing shared clipboard surface", + }, + "unnecessary_virtual_devices_absent": { + "passed": True, + "probes": ["gce_no_desktop_virtualization_peripheral_api"], + "message": "Compute Engine exposes no customer-controlled desktop peripheral redirection surface", + }, + } + + +def _apply_guest_probe(tests: dict[str, dict[str, Any]], guest_probe: dict[str, Any]) -> None: + """Mark failing tests for any guest-side signal in a completed probe.""" + if guest_probe.get("status") != "completed": + return + + for signal_key, test_name, error_msg in SIGNAL_BINDINGS: + signals = list(guest_probe.get(signal_key, [])) + if signals: + tests[test_name].update({"passed": False, "error": f"{error_msg}: {_compact('; '.join(signals))}"}) + + +@handle_gcp_errors +def main() -> int: + """Validate Compute Engine virtual-device hardening and emit structured JSON.""" + parser = argparse.ArgumentParser(description="Validate Compute Engine VM virtual-device hardening") + parser.add_argument("--instance-id", required=True, help="Compute Engine instance name") + parser.add_argument("--region", default="", help="(unused) forwarded by orchestrator") + parser.add_argument("--zone", default="", help="(unused) forwarded by orchestrator") + parser.add_argument("--public-ip", default="", help="Optional SSH host for guest probes") + parser.add_argument("--key-file", default="", help="Optional SSH private key path for guest probes") + parser.add_argument("--ssh-user", default="ubuntu", help="SSH username") + parser.add_argument( + "--ssh-timeout", + type=int, + default=60, + help="Total seconds for the combined guest probe SSH command", + ) + args = parser.parse_args() + + tests = _base_tests() + guest_probe_error: str | None = None + try: + guest_probe = _collect_guest_probe(args.public_ip, args.ssh_user, args.key_file, args.ssh_timeout) + except ValueError as e: + guest_probe_error = _compact(str(e)) + guest_probe = {"status": "unavailable"} + if guest_probe_error is None and guest_probe.get("status") == "unavailable": + guest_probe_error = _compact(str(guest_probe.get("error") or "guest probe unavailable")) + + _apply_guest_probe(tests, guest_probe) + + success = guest_probe_error is None and all(tests[name].get("passed") is True for name in REQUIRED_TESTS) + result: dict[str, Any] = { + "success": success, + "platform": "vm", + "test_name": "virtual_device_hardening", + "instance_id": args.instance_id, + "tests": tests, + } + if guest_probe_error is not None: + result["error"] = guest_probe_error + print(json.dumps(result, indent=2)) + return 0 if success else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isvctl/configs/providers/shared/teardown_nim.py b/isvctl/configs/providers/shared/teardown_nim.py index 06d4ea1e..6fb5e30c 100644 --- a/isvctl/configs/providers/shared/teardown_nim.py +++ b/isvctl/configs/providers/shared/teardown_nim.py @@ -31,6 +31,7 @@ import argparse import json +import os import re import sys from typing import Any @@ -69,6 +70,16 @@ def main() -> int: parser.add_argument("--user", default="ubuntu", help="SSH username") parser.add_argument("--container-name", default="isv-nim", help="Docker container name") parser.add_argument("--remove-image", action="store_true", help="Also remove the container image") + parser.add_argument( + "--ngc-api-key", + default=os.environ.get("NGC_API_KEY", "") or os.environ.get("NGC_NIM_API_KEY", ""), + help=( + "NGC API key (defaults to NGC_API_KEY / NGC_NIM_API_KEY env var). " + "When absent the step short-circuits with success=True, skipped=True " + "to mirror deploy_nim's policy-skip — deploy was a no-op, so there " + "is no container to tear down." + ), + ) args = parser.parse_args() if not _CONTAINER_NAME_RE.match(args.container_name): @@ -80,11 +91,42 @@ def main() -> int: result: dict[str, Any] = { "success": False, "platform": "vm", + "skipped": False, "container_removed": False, "image_removed": False, "container_name": args.container_name, } + # Policy-skip on missing NGC_API_KEY (mirrors deploy_nim's policy-skip + # shape — when deploy_nim was skipped because the operator's env has + # no NGC entitlement, there is no container to tear down). Equivalent + # symmetric handling so the documented "leave NGC_API_KEY unset to + # opt out of NIM coverage" path stays green end-to-end. + if not args.ngc_api_key: + result["success"] = True + result["skipped"] = True + result["skip_reason"] = "NGC_API_KEY not set (deploy_nim was skipped, nothing to tear down)" + print(json.dumps(result, indent=2)) + return 0 + + # Sentinel-skip path: when the producing step (launch / start / + # reboot) was skipped or failed, the provider config forwards + # "none" / "null" / "" sentinels rather than dropping the argv pair. + # Treat any sentinel host or key as "no instance was ever ready for + # SSH" and emit the canonical policy-skip JSON (rc=0, success=True, + # skipped=True) so the orchestrator's StepSuccessCheck does not + # turn a failed-setup run red on teardown. + _SENTINELS = {"none", "null", ""} + if args.host.strip().lower() in _SENTINELS or args.key_file.strip().lower() in _SENTINELS: + result["success"] = True + result["skipped"] = True + result["skip_reason"] = ( + f"sentinel host/key forwarded (host={args.host!r}, key_file={args.key_file!r}); " + "upstream lifecycle step did not produce a reachable instance, nothing to tear down" + ) + print(json.dumps(result, indent=2)) + return 0 + ssh = None try: ssh = ssh_connect(args.host, args.user, args.key_file) diff --git a/isvctl/pyproject.toml b/isvctl/pyproject.toml index f76f86a6..e3848afe 100644 --- a/isvctl/pyproject.toml +++ b/isvctl/pyproject.toml @@ -13,6 +13,7 @@ dependencies = [ "pydantic>=2.10.0,<3.0.0", "pyyaml>=6.0.2,<7.0.0", "typer>=0.21.1,<1.0.0", + "google-cloud-compute>=1.30.0,<2.0.0", ] [project.scripts] diff --git a/uv.lock b/uv.lock index 55809a52..80b25f0a 100644 --- a/uv.lock +++ b/uv.lock @@ -1,6 +1,11 @@ version = 1 revision = 3 requires-python = ">=3.12" +resolution-markers = [ + "python_full_version >= '3.14'", + "python_full_version == '3.13.*'", + "python_full_version < '3.13'", +] [manifest] members = [ @@ -501,6 +506,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/76/91/7216b27286936c16f5b4d0c530087e4a54eead683e6b0b73dd0c64844af6/filelock-3.20.0-py3-none-any.whl", hash = "sha256:339b4732ffda5cd79b13f4e2711a31b0365ce445d95d243bb996273d072546a2", size = 16054, upload-time = "2025-10-08T18:03:48.35Z" }, ] +[[package]] +name = "google-api-core" +version = "2.30.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "google-auth" }, + { name = "googleapis-common-protos" }, + { name = "proto-plus" }, + { name = "protobuf" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/16/ce/502a57fb0ec752026d24df1280b162294b22a0afb98a326084f9a979138b/google_api_core-2.30.3.tar.gz", hash = "sha256:e601a37f148585319b26db36e219df68c5d07b6382cff2d580e83404e44d641b", size = 177001, upload-time = "2026-04-10T00:41:28.035Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/15/e56f351cf6ef1cfea58e6ac226a7318ed1deb2218c4b3cc9bd9e4b786c5a/google_api_core-2.30.3-py3-none-any.whl", hash = "sha256:a85761ba72c444dad5d611c2220633480b2b6be2521eca69cca2dbb3ffd6bfe8", size = 173274, upload-time = "2026-04-09T22:57:16.198Z" }, +] + +[package.optional-dependencies] +grpc = [ + { name = "grpcio" }, + { name = "grpcio-status" }, +] + [[package]] name = "google-auth" version = "2.42.1" @@ -515,6 +542,89 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/92/05/adeb6c495aec4f9d93f9e2fc29eeef6e14d452bba11d15bdb874ce1d5b10/google_auth-2.42.1-py2.py3-none-any.whl", hash = "sha256:eb73d71c91fc95dbd221a2eb87477c278a355e7367a35c0d84e6b0e5f9b4ad11", size = 222550, upload-time = "2025-10-30T16:42:17.878Z" }, ] +[[package]] +name = "google-cloud-compute" +version = "1.47.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "google-api-core", extra = ["grpc"] }, + { name = "google-auth" }, + { name = "grpcio" }, + { name = "proto-plus" }, + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9f/b9/e9c462aaef5ff2d7abc306821d11ec1f0c8cdfa4840cd26d8fa87f41f5ee/google_cloud_compute-1.47.0.tar.gz", hash = "sha256:f2c7909299f230428b0b12e52e031efe76c39be5d28cae9998fe1130a223fc3a", size = 5084010, upload-time = "2026-03-30T22:51:26.576Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/37/1b7941d5741096cbdd04e040b16858c9af3ec6a324915994b83ce584ea0f/google_cloud_compute-1.47.0-py3-none-any.whl", hash = "sha256:7e0329d1b226ec948cd6064aa88ba6f16d4556585a13b1ec2494f751783749d3", size = 3846242, upload-time = "2026-03-30T22:49:04.879Z" }, +] + +[[package]] +name = "googleapis-common-protos" +version = "1.75.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b5/c8/f439cffde755cffa462bfbb156278fa6f9d09119719af9814b858fd4f81f/googleapis_common_protos-1.75.0.tar.gz", hash = "sha256:53a062ff3c32552fbd62c11fe23768b78e4ddf0494d5e5fd97d3f4689c75fbbd", size = 151035, upload-time = "2026-05-07T08:04:49.423Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/c8/e2645aa8ed02fd4c7a2f59d68783b65b1f3cbdfe39a6308e156509d1fee8/googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed", size = 300631, upload-time = "2026-05-07T08:03:30.345Z" }, +] + +[[package]] +name = "grpcio" +version = "1.80.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b7/48/af6173dbca4454f4637a4678b67f52ca7e0c1ed7d5894d89d434fecede05/grpcio-1.80.0.tar.gz", hash = "sha256:29aca15edd0688c22ba01d7cc01cb000d72b2033f4a3c72a81a19b56fd143257", size = 12978905, upload-time = "2026-03-30T08:49:10.502Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/e8/a2b749265eb3415abc94f2e619bbd9e9707bebdda787e61c593004ec927a/grpcio-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:c624cc9f1008361014378c9d776de7182b11fe8b2e5a81bc69f23a295f2a1ad0", size = 6015616, upload-time = "2026-03-30T08:47:13.428Z" }, + { url = "https://files.pythonhosted.org/packages/3e/97/b1282161a15d699d1e90c360df18d19165a045ce1c343c7f313f5e8a0b77/grpcio-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:f49eddcac43c3bf350c0385366a58f36bed8cc2c0ec35ef7b74b49e56552c0c2", size = 12014204, upload-time = "2026-03-30T08:47:15.873Z" }, + { url = "https://files.pythonhosted.org/packages/6e/5e/d319c6e997b50c155ac5a8cb12f5173d5b42677510e886d250d50264949d/grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d334591df610ab94714048e0d5b4f3dd5ad1bee74dfec11eee344220077a79de", size = 6563866, upload-time = "2026-03-30T08:47:18.588Z" }, + { url = "https://files.pythonhosted.org/packages/ae/f6/fdd975a2cb4d78eb67769a7b3b3830970bfa2e919f1decf724ae4445f42c/grpcio-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0cb517eb1d0d0aaf1d87af7cc5b801d686557c1d88b2619f5e31fab3c2315921", size = 7273060, upload-time = "2026-03-30T08:47:21.113Z" }, + { url = "https://files.pythonhosted.org/packages/db/f0/a3deb5feba60d9538a962913e37bd2e69a195f1c3376a3dd44fe0427e996/grpcio-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4e78c4ac0d97dc2e569b2f4bcbbb447491167cb358d1a389fc4af71ab6f70411", size = 6782121, upload-time = "2026-03-30T08:47:23.827Z" }, + { url = "https://files.pythonhosted.org/packages/ca/84/36c6dcfddc093e108141f757c407902a05085e0c328007cb090d56646cdf/grpcio-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2ed770b4c06984f3b47eb0517b1c69ad0b84ef3f40128f51448433be904634cd", size = 7383811, upload-time = "2026-03-30T08:47:26.517Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ef/f3a77e3dc5b471a0ec86c564c98d6adfa3510d38f8ee99010410858d591e/grpcio-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:256507e2f524092f1473071a05e65a5b10d84b82e3ff24c5b571513cfaa61e2f", size = 8393860, upload-time = "2026-03-30T08:47:29.439Z" }, + { url = "https://files.pythonhosted.org/packages/9b/8d/9d4d27ed7f33d109c50d6b5ce578a9914aa68edab75d65869a17e630a8d1/grpcio-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9a6284a5d907c37db53350645567c522be314bac859a64a7a5ca63b77bb7958f", size = 7830132, upload-time = "2026-03-30T08:47:33.254Z" }, + { url = "https://files.pythonhosted.org/packages/14/e4/9990b41c6d7a44e1e9dee8ac11d7a9802ba1378b40d77468a7761d1ad288/grpcio-1.80.0-cp312-cp312-win32.whl", hash = "sha256:c71309cfce2f22be26aa4a847357c502db6c621f1a49825ae98aa0907595b193", size = 4140904, upload-time = "2026-03-30T08:47:35.319Z" }, + { url = "https://files.pythonhosted.org/packages/2f/2c/296f6138caca1f4b92a31ace4ae1b87dab692fc16a7a3417af3bb3c805bf/grpcio-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:9fe648599c0e37594c4809d81a9e77bd138cc82eb8baa71b6a86af65426723ff", size = 4880944, upload-time = "2026-03-30T08:47:37.831Z" }, + { url = "https://files.pythonhosted.org/packages/2f/3a/7c3c25789e3f069e581dc342e03613c5b1cb012c4e8c7d9d5cf960a75856/grpcio-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e9e408fc016dffd20661f0126c53d8a31c2821b5c13c5d67a0f5ed5de93319ad", size = 6017243, upload-time = "2026-03-30T08:47:40.075Z" }, + { url = "https://files.pythonhosted.org/packages/04/19/21a9806eb8240e174fd1ab0cd5b9aa948bb0e05c2f2f55f9d5d7405e6d08/grpcio-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:92d787312e613754d4d8b9ca6d3297e69994a7912a32fa38c4c4e01c272974b0", size = 12010840, upload-time = "2026-03-30T08:47:43.11Z" }, + { url = "https://files.pythonhosted.org/packages/18/3a/23347d35f76f639e807fb7a36fad3068aed100996849a33809591f26eca6/grpcio-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac393b58aa16991a2f1144ec578084d544038c12242da3a215966b512904d0f", size = 6567644, upload-time = "2026-03-30T08:47:46.806Z" }, + { url = "https://files.pythonhosted.org/packages/ff/40/96e07ecb604a6a67ae6ab151e3e35b132875d98bc68ec65f3e5ab3e781d7/grpcio-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:68e5851ac4b9afe07e7f84483803ad167852570d65326b34d54ca560bfa53fb6", size = 7277830, upload-time = "2026-03-30T08:47:49.643Z" }, + { url = "https://files.pythonhosted.org/packages/9b/e2/da1506ecea1f34a5e365964644b35edef53803052b763ca214ba3870c856/grpcio-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:873ff5d17d68992ef6605330127425d2fc4e77e612fa3c3e0ed4e668685e3140", size = 6783216, upload-time = "2026-03-30T08:47:52.817Z" }, + { url = "https://files.pythonhosted.org/packages/44/83/3b20ff58d0c3b7f6caaa3af9a4174d4023701df40a3f39f7f1c8e7c48f9d/grpcio-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2bea16af2750fd0a899bf1abd9022244418b55d1f37da2202249ba4ba673838d", size = 7385866, upload-time = "2026-03-30T08:47:55.687Z" }, + { url = "https://files.pythonhosted.org/packages/47/45/55c507599c5520416de5eefecc927d6a0d7af55e91cfffb2e410607e5744/grpcio-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba0db34f7e1d803a878284cd70e4c63cb6ae2510ba51937bf8f45ba997cefcf7", size = 8391602, upload-time = "2026-03-30T08:47:58.303Z" }, + { url = "https://files.pythonhosted.org/packages/10/bb/dd06f4c24c01db9cf11341b547d0a016b2c90ed7dbbb086a5710df7dd1d7/grpcio-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8eb613f02d34721f1acf3626dfdb3545bd3c8505b0e52bf8b5710a28d02e8aa7", size = 7826752, upload-time = "2026-03-30T08:48:01.311Z" }, + { url = "https://files.pythonhosted.org/packages/f9/1e/9d67992ba23371fd63d4527096eb8c6b76d74d52b500df992a3343fd7251/grpcio-1.80.0-cp313-cp313-win32.whl", hash = "sha256:93b6f823810720912fd131f561f91f5fed0fda372b6b7028a2681b8194d5d294", size = 4142310, upload-time = "2026-03-30T08:48:04.594Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e6/283326a27da9e2c3038bc93eeea36fb118ce0b2d03922a9cda6688f53c5b/grpcio-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:e172cf795a3ba5246d3529e4d34c53db70e888fa582a8ffebd2e6e48bc0cba50", size = 4882833, upload-time = "2026-03-30T08:48:07.363Z" }, + { url = "https://files.pythonhosted.org/packages/c5/6d/e65307ce20f5a09244ba9e9d8476e99fb039de7154f37fb85f26978b59c3/grpcio-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:3d4147a97c8344d065d01bbf8b6acec2cf86fb0400d40696c8bdad34a64ffc0e", size = 6017376, upload-time = "2026-03-30T08:48:10.005Z" }, + { url = "https://files.pythonhosted.org/packages/69/10/9cef5d9650c72625a699c549940f0abb3c4bfdb5ed45a5ce431f92f31806/grpcio-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d8e11f167935b3eb089ac9038e1a063e6d7dbe995c0bb4a661e614583352e76f", size = 12018133, upload-time = "2026-03-30T08:48:12.927Z" }, + { url = "https://files.pythonhosted.org/packages/04/82/983aabaad82ba26113caceeb9091706a0696b25da004fe3defb5b346e15b/grpcio-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f14b618fc30de822681ee986cfdcc2d9327229dc4c98aed16896761cacd468b9", size = 6574748, upload-time = "2026-03-30T08:48:16.386Z" }, + { url = "https://files.pythonhosted.org/packages/07/d7/031666ef155aa0bf399ed7e19439656c38bbd143779ae0861b038ce82abd/grpcio-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4ed39fbdcf9b87370f6e8df4e39ca7b38b3e5e9d1b0013c7b6be9639d6578d14", size = 7277711, upload-time = "2026-03-30T08:48:19.627Z" }, + { url = "https://files.pythonhosted.org/packages/e8/43/f437a78f7f4f1d311804189e8f11fb311a01049b2e08557c1068d470cb2e/grpcio-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2dcc70e9f0ba987526e8e8603a610fb4f460e42899e74e7a518bf3c68fe1bf05", size = 6785372, upload-time = "2026-03-30T08:48:22.373Z" }, + { url = "https://files.pythonhosted.org/packages/93/3d/f6558e9c6296cb4227faa5c43c54a34c68d32654b829f53288313d16a86e/grpcio-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:448c884b668b868562b1bda833c5fce6272d26e1926ec46747cda05741d302c1", size = 7395268, upload-time = "2026-03-30T08:48:25.638Z" }, + { url = "https://files.pythonhosted.org/packages/06/21/0fdd77e84720b08843c371a2efa6f2e19dbebf56adc72df73d891f5506f0/grpcio-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a1dc80fe55685b4a543555e6eef975303b36c8db1023b1599b094b92aa77965f", size = 8392000, upload-time = "2026-03-30T08:48:28.974Z" }, + { url = "https://files.pythonhosted.org/packages/f5/68/67f4947ed55d2e69f2cc199ab9fd85e0a0034d813bbeef84df6d2ba4d4b7/grpcio-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:31b9ac4ad1aa28ffee5503821fafd09e4da0a261ce1c1281c6c8da0423c83b6e", size = 7828477, upload-time = "2026-03-30T08:48:32.054Z" }, + { url = "https://files.pythonhosted.org/packages/44/b6/8d4096691b2e385e8271911a0de4f35f0a6c7d05aff7098e296c3de86939/grpcio-1.80.0-cp314-cp314-win32.whl", hash = "sha256:367ce30ba67d05e0592470428f0ec1c31714cab9ef19b8f2e37be1f4c7d32fae", size = 4218563, upload-time = "2026-03-30T08:48:34.538Z" }, + { url = "https://files.pythonhosted.org/packages/e5/8c/bbe6baf2557262834f2070cf668515fa308b2d38a4bbf771f8f7872a7036/grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f", size = 5019457, upload-time = "2026-03-30T08:48:37.308Z" }, +] + +[[package]] +name = "grpcio-status" +version = "1.80.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "googleapis-common-protos" }, + { name = "grpcio" }, + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/ed/105f619bdd00cb47a49aa2feea6232ea2bbb04199d52a22cc6a7d603b5cb/grpcio_status-1.80.0.tar.gz", hash = "sha256:df73802a4c89a3ea88aa2aff971e886fccce162bc2e6511408b3d67a144381cd", size = 13901, upload-time = "2026-03-30T08:54:34.784Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/80/58cd2dfc19a07d022abe44bde7c365627f6c7cb6f692ada6c65ca437d09a/grpcio_status-1.80.0-py3-none-any.whl", hash = "sha256:4b56990363af50dbf2c2ebb80f1967185c07d87aa25aa2bea45ddb75fc181dbe", size = 14638, upload-time = "2026-03-30T08:54:01.569Z" }, +] + [[package]] name = "identify" version = "2.6.15" @@ -603,6 +713,7 @@ name = "isvctl" version = "0.7.0" source = { editable = "isvctl" } dependencies = [ + { name = "google-cloud-compute" }, { name = "isvreporter" }, { name = "isvtest" }, { name = "jinja2" }, @@ -613,6 +724,7 @@ dependencies = [ [package.metadata] requires-dist = [ + { name = "google-cloud-compute", specifier = ">=1.30.0,<2.0.0" }, { name = "isvreporter", editable = "isvreporter" }, { name = "isvtest", editable = "isvtest" }, { name = "jinja2", specifier = ">=3.1.6,<4.0.0" }, @@ -968,6 +1080,33 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5b/a5/987a405322d78a73b66e39e4a90e4ef156fd7141bf71df987e50717c321b/pre_commit-4.3.0-py2.py3-none-any.whl", hash = "sha256:2b0747ad7e6e967169136edffee14c16e148a778a54e4f967921aa1ebf2308d8", size = 220965, upload-time = "2025-08-09T18:56:13.192Z" }, ] +[[package]] +name = "proto-plus" +version = "1.28.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c9/56/e647b0c675392d2da368da7b6f158f7368b18542fd6f7d7400a2f39de000/proto_plus-1.28.0.tar.gz", hash = "sha256:38e5696342835b08fc116f30a25665b29531cda9d5d5643e9b81fc312385abd9", size = 57221, upload-time = "2026-05-07T08:04:50.811Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7c/20/b122d4626976acb81132036d2ad1bb35a1a8775fceb837ec30964622516a/proto_plus-1.28.0-py3-none-any.whl", hash = "sha256:a630604310899e73c59ec302e5765c058d412b2f090b9c79c8822589f14955b8", size = 50410, upload-time = "2026-05-07T08:03:31.962Z" }, +] + +[[package]] +name = "protobuf" +version = "6.33.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/66/70/e908e9c5e52ef7c3a6c7902c9dfbb34c7e29c25d2f81ade3856445fd5c94/protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135", size = 444531, upload-time = "2026-03-18T19:05:00.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fc/9f/2f509339e89cfa6f6a4c4ff50438db9ca488dec341f7e454adad60150b00/protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3", size = 425739, upload-time = "2026-03-18T19:04:48.373Z" }, + { url = "https://files.pythonhosted.org/packages/76/5d/683efcd4798e0030c1bab27374fd13a89f7c2515fb1f3123efdfaa5eab57/protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326", size = 437089, upload-time = "2026-03-18T19:04:50.381Z" }, + { url = "https://files.pythonhosted.org/packages/5c/01/a3c3ed5cd186f39e7880f8303cc51385a198a81469d53d0fdecf1f64d929/protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a", size = 427737, upload-time = "2026-03-18T19:04:51.866Z" }, + { url = "https://files.pythonhosted.org/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2", size = 324610, upload-time = "2026-03-18T19:04:53.096Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ca/25afc144934014700c52e05103c2421997482d561f3101ff352e1292fb81/protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3", size = 339381, upload-time = "2026-03-18T19:04:54.616Z" }, + { url = "https://files.pythonhosted.org/packages/16/92/d1e32e3e0d894fe00b15ce28ad4944ab692713f2e7f0a99787405e43533a/protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593", size = 323436, upload-time = "2026-03-18T19:04:55.768Z" }, + { url = "https://files.pythonhosted.org/packages/c4/72/02445137af02769918a93807b2b7890047c32bfb9f90371cbc12688819eb/protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901", size = 170656, upload-time = "2026-03-18T19:04:59.826Z" }, +] + [[package]] name = "pyasn1" version = "0.6.3" From 8ceb61531e21bd9571fd81d64ee72fddfd9b9a34 Mon Sep 17 00:00:00 2001 From: Orga Shih Date: Tue, 19 May 2026 02:08:32 +0800 Subject: [PATCH 2/3] fix(gcp/common): add PEP 257 docstrings to missing helpers (CodeRabbit review feedback) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three helper functions surfaced by CodeRabbit's docstring-coverage quality gate (78.72% < 80% threshold on PR #427) lacked PEP 257 docstrings: - common/compute.py: first_internal_ip (returns the nic0 internal IPv4 or None when the instance has no NICs / hasn't allocated yet). - common/compute.py: _firewall_has_isv_ownership (verified-reuse ownership marker check — distinguishes firewalls this suite created from operator-owned firewalls that happen to match the test name). - common/ssh_utils.py: _ssh_argv (canonical ssh argv builder — shared _SSH_OPTS + -i + @ + — so every subprocess SSH call in this module uses identical option semantics, per the canonical-options consistency rule). Each function gets a one-paragraph docstring (one-line summary + a second line of context where it adds value). Coverage on the gcp/scripts tree post-fix: 83.15% (74/89), above the 80% threshold. Kept as a separate commit (not amended into the squash) so the PR's "Commits" tab shows a transparent response to CodeRabbit's review feedback — easier for the upstream maintainer to see exactly what changed in response to a specific review-cycle pass. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Orga Shih --- .../configs/providers/gcp/scripts/common/compute.py | 12 ++++++++++++ .../providers/gcp/scripts/common/ssh_utils.py | 6 ++++++ 2 files changed, 18 insertions(+) diff --git a/isvctl/configs/providers/gcp/scripts/common/compute.py b/isvctl/configs/providers/gcp/scripts/common/compute.py index f6f16a53..a48159dc 100644 --- a/isvctl/configs/providers/gcp/scripts/common/compute.py +++ b/isvctl/configs/providers/gcp/scripts/common/compute.py @@ -601,6 +601,12 @@ def first_external_ip(instance: compute_v1.Instance) -> str | None: def first_internal_ip(instance: compute_v1.Instance) -> str | None: + """Return the primary internal IPv4 of ``instance`` (``network_interfaces[0].network_i_p``), or ``None``. + + Returns ``None`` when the instance has no NICs or when nic0's + ``network_i_p`` is empty (instance still provisioning, or unusual + no-NIC shape). + """ if not instance.network_interfaces: return None return instance.network_interfaces[0].network_i_p or None @@ -846,6 +852,12 @@ def _firewall_matches_ssh_shape(rule: compute_v1.Firewall, network_short: str) - def _firewall_has_isv_ownership(rule: compute_v1.Firewall) -> bool: + """Return ``True`` iff ``rule.description`` carries the ISV ownership marker. + + Used by verified-reuse to distinguish firewalls that this suite + created (and is therefore safe to mutate / delete) from operator- + owned firewalls that happen to match the test's name pattern. + """ return _ISV_OWNERSHIP_MARKER in (rule.description or "").lower() diff --git a/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py b/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py index e93c21ce..54fc96ba 100644 --- a/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py +++ b/isvctl/configs/providers/gcp/scripts/common/ssh_utils.py @@ -45,6 +45,12 @@ def _ssh_argv(host: str, user: str, key_file: str, remote_cmd: str) -> list[str]: + """Build a canonical ``ssh`` argv: shared ``_SSH_OPTS`` + ``-i `` + ``@`` + ````. + + Used by ``_try_ssh`` / ``get_uptime_via_ssh`` / cloud-init waiters so + every subprocess SSH call in this module uses identical option + semantics (canonical-options consistency rule). + """ return ["ssh", *_SSH_OPTS, "-i", key_file, f"{user}@{host}", remote_cmd] From 2d2b89f6bc36435baf0cb113016d1a02bad6caa4 Mon Sep 17 00:00:00 2001 From: Orga Shih Date: Tue, 19 May 2026 02:16:26 +0800 Subject: [PATCH 3/3] docs(references/gcp): specify text language for L4-zones code fence (MD040) CodeRabbit's review of PR #427 flagged the bare ``` fence around the L4-supported-zones region list as a markdownlint MD040 violation. Changing to ```text marks the block explicitly as plain text so linters don't trip while preserving the rendering (text is the de-facto fallback for un-tagged fences). One-character class change; no behavior or rendering difference. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Orga Shih --- docs/references/gcp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/references/gcp.md b/docs/references/gcp.md index 1538b4a2..90ac7012 100644 --- a/docs/references/gcp.md +++ b/docs/references/gcp.md @@ -113,7 +113,7 @@ The VM suite exercises 11 subtests end-to-end: launch (with GPU + cloud-init + S The VM domain's zone-walk prefers GCP zones with observed L4 capacity. The reviewed list (in priority order) is: -``` +```text us-central1-a / -b / -c us-east4-a / -b / -c us-east1-c / -d us-west1-a / -b us-west4-a / -b europe-west4-a / -b europe-west1-b / -c