Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .factory_version

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this file for?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's for updating flow when isv repo changed and the factory need to update the generated content. it's easier to find the commit to diff instead of searching commit history. if you think it's better to not injecting any factory related metadata in to isv repo, I can think about other solution.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be best to keep separate if you can thank

Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"ncps": {
"gcp": {
"vm": {
"factory_commit": "21869ef830d5a088250206c37c757c4ffc28eeeb",
"isv_commit": "9b79e34037435b8567e3cd79e18dae01ae1ea0e2"
}
}
},
"source_repo": "git@github.com:NVIDIA/ISV-NCP-Validation-Suite.git"
}
1 change: 1 addition & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ See [Remote Deployment Guide](guides/remote-deployment.md) for details.
- [my-isv Scaffold](../isvctl/configs/providers/my-isv/scripts/README.md) - Adding your own platform? Start here
- [Validation Test Suites](../isvctl/configs/suites/README.md) - The platform-agnostic validation contract
- [AWS Reference Implementation](references/aws.md) - Working AWS examples to study
- [GCP Target Implementation](references/gcp.md) - Operator setup for running tests on GCP
- [Configuration Guide](guides/configuration.md) - Config file format and options
- [External Validation Guide](guides/external-validation-guide.md) - Custom validations without modifying the repo
- [Local Development](guides/local-development.md) - Running tests locally
Expand Down
141 changes: 141 additions & 0 deletions docs/references/gcp.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# GCP Target Implementation

The GCP implementation is a target-platform port of the ISV validation framework. The [AWS reference](aws.md) is the canonical example for how the suite expects each provider to behave; the GCP scripts under `providers/gcp/` translate that contract onto Google Cloud's API surface (Compute Engine, IAM, etc.).

This page covers the operator setup needed to run `isvctl` tests against GCP.

## Available Modules

| Domain | Config | Scripts | Test Suite |
|--------|--------|---------|------------|
| **VM** | [`providers/gcp/config/vm.yaml`](../../isvctl/configs/providers/gcp/config/vm.yaml) | [`providers/gcp/scripts/vm/`](../../isvctl/configs/providers/gcp/scripts/vm/) | [`suites/vm.yaml`](../../isvctl/configs/suites/vm.yaml) |

Shared GCP utilities (compute helpers, SSH wrappers, retry envelopes, error classifiers) are in [`providers/gcp/scripts/common/`](../../isvctl/configs/providers/gcp/scripts/common/).

Other domains (IAM, Network, Bare Metal, EKS, Control Plane, Image Registry, Security) are not yet implemented for GCP.

## Prerequisites

### 1. Operator GCP environment

- A GCP project with **billing enabled** and the **Compute Engine API** enabled.
- **L4 GPU quota** (`NVIDIA_L4_GPUS`) of at least 1 in at least one zone listed under [Supported zones](#supported-zones-for-l4-gpu-vm-tests) below. The VM domain's `launch_instance` step provisions a `g2-standard-8` instance for the lifecycle, console-RBAC, and NIM subtests.
- The principal running the tests (user or service account) needs roughly these roles on the project (consolidate via custom role if preferred):
- `roles/compute.admin` — create / delete / start / stop / reboot instances and firewall rules.
- `roles/iam.serviceAccountAdmin` + `roles/iam.serviceAccountUser` — `console_rbac` self-provisions short-lived probe service accounts and mints access tokens against them.
- `roles/iam.serviceAccountTokenCreator` on the probe SAs (granted dynamically by the test itself, but the project-level binding is needed to allow the test to grant it).

### 2. Authentication

`isvctl` calls into the GCP Python SDKs (`google-cloud-compute`, `google-auth`, etc.) which use Application Default Credentials. Either:

```bash
# Option A — user credentials (recommended for local dev / interactive runs)
gcloud auth application-default login

# Option B — service account key file
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
```

### 3. Project ID resolution

The GCP scripts resolve the active project ID in this order:

1. Explicit `--project` argument (rarely passed by the suite; reserved for ad-hoc invocations).
2. `GOOGLE_CLOUD_PROJECT` environment variable.
3. `GCLOUD_PROJECT` environment variable.
4. The project bundled with Application Default Credentials (set by `gcloud auth application-default login`).

If none resolve, every step fails fast with a structured `credentials_missing` error.

```bash
# Most operators just set this once:
export GOOGLE_CLOUD_PROJECT=your-project-id
```

### 4. NIM API key (for `deploy_nim` step)

The shared NIM deployment script reads `NGC_API_KEY` from the environment. If unset, the step short-circuits with `success=True, skipped=True` and the rest of the suite continues — useful for environments where NIM isn't part of the operator's validation scope.

```bash
export NGC_API_KEY=nvapi-...
```

### 5. GPU image (and Docker requirement for `deploy_nim`)

`launch_instance` defaults to the public GCP Deep Learning VM Image:

| Field | Default |
|---|---|
| `--image-project` | `deeplearning-platform-release` |
| `--image-family` | `common-cu129-ubuntu-2204-nvidia-580` |

This image is published by Google, ships with the NVIDIA driver + CUDA toolkit, and works out-of-the-box for lifecycle / serial-console / RBAC / describe coverage.

**It does NOT ship Docker.** The `deploy_nim` step pulls and runs a NIM container, so it requires a Docker engine on the launched VM. Operators who need NIM coverage have two options:

1. **Bring a custom image** that has Docker + NVIDIA Container Toolkit preinstalled. The recommended path is to set the override once in your shell / `.env` so every run reuses the same pin:

```bash
# Add to your .env or shell environment.
export GCP_VM_IMAGE=<your-image-family-or-name>
export GCP_VM_IMAGE_PROJECT=<your-gcp-project>
```

The provider config reads both env vars via Jinja and falls back to the public DLVM when either is unset. For one-off runs you can also override per-invocation:

```bash
uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml \
--set image_project=<your-gcp-project> \
--set image=<your-image-family-or-name>
```

Either path wires `image_project` and `image` through to `launch_instance`'s `--image-project` / `--ami-id` arguments. The image short-name resolves in the operator project first; if not found there, the resolver falls back to the default DLVM project.

2. **Skip NIM** by leaving `NGC_API_KEY` unset (see §4 above). The `deploy_nim` and `teardown_nim` steps short-circuit cleanly and every instance-lifecycle step proceeds. The run still reports `[FAIL] TEST` because of `ContainerRuntimeCheck` (see note below) — accept that as a documented limitation of the default image.

**Note on `ContainerRuntimeCheck`**: the `host_os` validator group includes `ContainerRuntimeCheck`, which asserts Docker is installed and runnable on the launched VM. It runs on every `vm` invocation regardless of `NGC_API_KEY`. On the default DLVM image this validator fails with `Docker not available`, and the run reports `[FAIL] TEST` even though every instance-lifecycle and NIM-policy-skip step passes. Only option 1 (a custom image with Docker preinstalled) produces a fully clean PASS.

Operators without an NGC entitlement should pick option 2; operators with one and no custom image can install Docker on the default image inside their own cloud-init / startup-script, but the simplest path is a custom image where `docker run --gpus all` works at boot.

## Running GCP Validations

```bash
# Prerequisites: ADC + GOOGLE_CLOUD_PROJECT set per "Authentication" above.
# Optional: NGC_API_KEY for NIM coverage.

uv run isvctl test run -f isvctl/configs/providers/gcp/config/vm.yaml
```

The VM suite exercises 11 subtests end-to-end: launch (with GPU + cloud-init + SSH stability gate), tag verification, serial console output, console-RBAC probe (creates two short-lived probe service accounts + a second probe VM), idempotent stop / start / reboot lifecycle, describe (host OS / driver / CPU / container runtime checks), NIM deploy + inference, teardown of all created resources. Wall-clock is roughly 30–45 minutes on a clean operator environment; capacity stockout in one zone triggers a documented walk to the next zone in the preferred list.

## Supported zones for L4 GPU VM tests

The VM domain's zone-walk prefers GCP zones with observed L4 capacity. The reviewed list (in priority order) is:

```text
us-central1-a / -b / -c us-east4-a / -b / -c us-east1-c / -d
us-west1-a / -b us-west4-a / -b
europe-west4-a / -b europe-west1-b / -c
asia-southeast1-a / -b / -c asia-northeast1-a / -c asia-east1-a / -b / -c
```

The list reflects multi-week capacity observation from spring 2026; capacity drifts, so an operator can probe a zone before a long run with:

```bash
gcloud compute accelerator-types list --filter='name=nvidia-l4 AND zone:<zone>'
```

## Org-policy considerations

GCP organizations sometimes apply policies that block specific operations the VM suite needs. Common ones to check or exempt:

- `compute.disableSerialPortAccess` — must allow. The `console_rbac` test validates that serial-console access is properly RBAC-restricted; the test itself reads serial output via IAM-mediated short-lived tokens.
- `iam.disableServiceAccountKeyCreation` — does NOT need to be disabled. The suite uses `IAMCredentials.generateAccessToken` (no key material), not service-account JSON keys.
- `compute.requireOsLogin` — must allow per-instance SSH-key metadata, which the suite uses to establish SSH for the post-launch stability gate and NIM health checks. If OS Login is enforced project-wide, the SSH gate fails; either exempt the test instances or grant the operator the OS Login roles.

## Resources

- GCP IAM permissions for Compute Engine: <https://cloud.google.com/compute/docs/access/iam>
- L4 GPU zones (current as of GCP docs): <https://cloud.google.com/compute/docs/gpus/gpu-regions-zones>
- Application Default Credentials: <https://cloud.google.com/docs/authentication/application-default-credentials>
Loading