-
Notifications
You must be signed in to change notification settings - Fork 17
feat: generates gcp/vm stubs #427
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sinorga
wants to merge
3
commits into
NVIDIA:main
Choose a base branch
from
sinorga:agent/gcp-workers-286265d1-e5eb-4ffe-bf5f-c8e7a46ce147
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "ncps": { | ||
| "gcp": { | ||
| "vm": { | ||
| "factory_commit": "21869ef830d5a088250206c37c757c4ffc28eeeb", | ||
| "isv_commit": "9b79e34037435b8567e3cd79e18dae01ae1ea0e2" | ||
| } | ||
| } | ||
| }, | ||
| "source_repo": "git@github.com:NVIDIA/ISV-NCP-Validation-Suite.git" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| # GCP Target Implementation | ||
|
|
||
| The GCP implementation is a target-platform port of the ISV validation framework. The [AWS reference](aws.md) is the canonical example for how the suite expects each provider to behave; the GCP scripts under `providers/gcp/` translate that contract onto Google Cloud's API surface (Compute Engine, IAM, etc.). | ||
|
|
||
| This page covers the operator setup needed to run `isvctl` tests against GCP. | ||
|
|
||
| ## Available Modules | ||
|
|
||
| | Domain | Config | Scripts | Test Suite | | ||
| |--------|--------|---------|------------| | ||
| | **VM** | [`providers/gcp/config/vm.yaml`](../../isvctl/configs/providers/gcp/config/vm.yaml) | [`providers/gcp/scripts/vm/`](../../isvctl/configs/providers/gcp/scripts/vm/) | [`suites/vm.yaml`](../../isvctl/configs/suites/vm.yaml) | | ||
|
|
||
| Shared GCP utilities (compute helpers, SSH wrappers, retry envelopes, error classifiers) are in [`providers/gcp/scripts/common/`](../../isvctl/configs/providers/gcp/scripts/common/). | ||
|
|
||
| Other domains (IAM, Network, Bare Metal, EKS, Control Plane, Image Registry, Security) are not yet implemented for GCP. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ### 1. Operator GCP environment | ||
|
|
||
| - A GCP project with **billing enabled** and the **Compute Engine API** enabled. | ||
| - **L4 GPU quota** (`NVIDIA_L4_GPUS`) of at least 1 in at least one zone listed under [Supported zones](#supported-zones-for-l4-gpu-vm-tests) below. The VM domain's `launch_instance` step provisions a `g2-standard-8` instance for the lifecycle, console-RBAC, and NIM subtests. | ||
| - The principal running the tests (user or service account) needs roughly these roles on the project (consolidate via custom role if preferred): | ||
| - `roles/compute.admin` — create / delete / start / stop / reboot instances and firewall rules. | ||
| - `roles/iam.serviceAccountAdmin` + `roles/iam.serviceAccountUser` — `console_rbac` self-provisions short-lived probe service accounts and mints access tokens against them. | ||
| - `roles/iam.serviceAccountTokenCreator` on the probe SAs (granted dynamically by the test itself, but the project-level binding is needed to allow the test to grant it). | ||
|
|
||
| ### 2. Authentication | ||
|
|
||
| `isvctl` calls into the GCP Python SDKs (`google-cloud-compute`, `google-auth`, etc.) which use Application Default Credentials. Either: | ||
|
|
||
| ```bash | ||
| # Option A — user credentials (recommended for local dev / interactive runs) | ||
| gcloud auth application-default login | ||
|
|
||
| # Option B — service account key file | ||
| export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json | ||
| ``` | ||
|
|
||
| ### 3. Project ID resolution | ||
|
|
||
| The GCP scripts resolve the active project ID in this order: | ||
|
|
||
| 1. Explicit `--project` argument (rarely passed by the suite; reserved for ad-hoc invocations). | ||
| 2. `GOOGLE_CLOUD_PROJECT` environment variable. | ||
| 3. `GCLOUD_PROJECT` environment variable. | ||
| 4. The project bundled with Application Default Credentials (set by `gcloud auth application-default login`). | ||
|
|
||
| If none resolve, every step fails fast with a structured `credentials_missing` error. | ||
|
|
||
| ```bash | ||
| # Most operators just set this once: | ||
| export GOOGLE_CLOUD_PROJECT=your-project-id | ||
| ``` | ||
|
|
||
| ### 4. NIM API key (for `deploy_nim` step) | ||
|
|
||
| The shared NIM deployment script reads `NGC_API_KEY` from the environment. If unset, the step short-circuits with `success=True, skipped=True` and the rest of the suite continues — useful for environments where NIM isn't part of the operator's validation scope. | ||
|
|
||
| ```bash | ||
| export NGC_API_KEY=nvapi-... | ||
| ``` | ||
|
|
||
| ### 5. GPU image (and Docker requirement for `deploy_nim`) | ||
|
|
||
| `launch_instance` defaults to the public GCP Deep Learning VM Image: | ||
|
|
||
| | Field | Default | | ||
| |---|---| | ||
| | `--image-project` | `deeplearning-platform-release` | | ||
| | `--image-family` | `common-cu129-ubuntu-2204-nvidia-580` | | ||
|
|
||
| This image is published by Google, ships with the NVIDIA driver + CUDA toolkit, and works out-of-the-box for lifecycle / serial-console / RBAC / describe coverage. | ||
|
|
||
| **It does NOT ship Docker.** The `deploy_nim` step pulls and runs a NIM container, so it requires a Docker engine on the launched VM. Operators who need NIM coverage have two options: | ||
|
|
||
| 1. **Bring a custom image** that has Docker + NVIDIA Container Toolkit preinstalled. The recommended path is to set the override once in your shell / `.env` so every run reuses the same pin: | ||
|
|
||
| ```bash | ||
| # Add to your .env or shell environment. | ||
| export GCP_VM_IMAGE=<your-image-family-or-name> | ||
| export GCP_VM_IMAGE_PROJECT=<your-gcp-project> | ||
| ``` | ||
|
|
||
| The provider config reads both env vars via Jinja and falls back to the public DLVM when either is unset. For one-off runs you can also override per-invocation: | ||
|
|
||
| ```bash | ||
| uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml \ | ||
| --set image_project=<your-gcp-project> \ | ||
| --set image=<your-image-family-or-name> | ||
| ``` | ||
|
|
||
| Either path wires `image_project` and `image` through to `launch_instance`'s `--image-project` / `--ami-id` arguments. The image short-name resolves in the operator project first; if not found there, the resolver falls back to the default DLVM project. | ||
|
|
||
| 2. **Skip NIM** by leaving `NGC_API_KEY` unset (see §4 above). The `deploy_nim` and `teardown_nim` steps short-circuit cleanly and every instance-lifecycle step proceeds. The run still reports `[FAIL] TEST` because of `ContainerRuntimeCheck` (see note below) — accept that as a documented limitation of the default image. | ||
|
|
||
| **Note on `ContainerRuntimeCheck`**: the `host_os` validator group includes `ContainerRuntimeCheck`, which asserts Docker is installed and runnable on the launched VM. It runs on every `vm` invocation regardless of `NGC_API_KEY`. On the default DLVM image this validator fails with `Docker not available`, and the run reports `[FAIL] TEST` even though every instance-lifecycle and NIM-policy-skip step passes. Only option 1 (a custom image with Docker preinstalled) produces a fully clean PASS. | ||
|
|
||
| Operators without an NGC entitlement should pick option 2; operators with one and no custom image can install Docker on the default image inside their own cloud-init / startup-script, but the simplest path is a custom image where `docker run --gpus all` works at boot. | ||
|
|
||
| ## Running GCP Validations | ||
|
|
||
| ```bash | ||
| # Prerequisites: ADC + GOOGLE_CLOUD_PROJECT set per "Authentication" above. | ||
| # Optional: NGC_API_KEY for NIM coverage. | ||
|
|
||
| uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml | ||
| ``` | ||
|
|
||
| The VM suite exercises 11 subtests end-to-end: launch (with GPU + cloud-init + SSH stability gate), tag verification, serial console output, console-RBAC probe (creates two short-lived probe service accounts + a second probe VM), idempotent stop / start / reboot lifecycle, describe (host OS / driver / CPU / container runtime checks), NIM deploy + inference, teardown of all created resources. Wall-clock is roughly 30–45 minutes on a clean operator environment; capacity stockout in one zone triggers a documented walk to the next zone in the preferred list. | ||
|
|
||
| ## Supported zones for L4 GPU VM tests | ||
|
|
||
| The VM domain's zone-walk prefers GCP zones with observed L4 capacity. The reviewed list (in priority order) is: | ||
|
|
||
| ```text | ||
| us-central1-a / -b / -c us-east4-a / -b / -c us-east1-c / -d | ||
| us-west1-a / -b us-west4-a / -b | ||
| europe-west4-a / -b europe-west1-b / -c | ||
| asia-southeast1-a / -b / -c asia-northeast1-a / -c asia-east1-a / -b / -c | ||
| ``` | ||
|
|
||
| The list reflects multi-week capacity observation from spring 2026; capacity drifts, so an operator can probe a zone before a long run with: | ||
|
|
||
| ```bash | ||
| gcloud compute accelerator-types list --filter='name=nvidia-l4 AND zone:<zone>' | ||
| ``` | ||
|
|
||
| ## Org-policy considerations | ||
|
|
||
| GCP organizations sometimes apply policies that block specific operations the VM suite needs. Common ones to check or exempt: | ||
|
|
||
| - `compute.disableSerialPortAccess` — must allow. The `console_rbac` test validates that serial-console access is properly RBAC-restricted; the test itself reads serial output via IAM-mediated short-lived tokens. | ||
| - `iam.disableServiceAccountKeyCreation` — does NOT need to be disabled. The suite uses `IAMCredentials.generateAccessToken` (no key material), not service-account JSON keys. | ||
| - `compute.requireOsLogin` — must allow per-instance SSH-key metadata, which the suite uses to establish SSH for the post-launch stability gate and NIM health checks. If OS Login is enforced project-wide, the SSH gate fails; either exempt the test instances or grant the operator the OS Login roles. | ||
|
|
||
| ## Resources | ||
|
|
||
| - GCP IAM permissions for Compute Engine: <https://cloud.google.com/compute/docs/access/iam> | ||
| - L4 GPU zones (current as of GCP docs): <https://cloud.google.com/compute/docs/gpus/gpu-regions-zones> | ||
| - Application Default Credentials: <https://cloud.google.com/docs/authentication/application-default-credentials> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is this file for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's for updating flow when isv repo changed and the factory need to update the generated content. it's easier to find the commit to diff instead of searching commit history. if you think it's better to not injecting any factory related metadata in to isv repo, I can think about other solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be best to keep separate if you can thank