ci(gha): add Bazel build + test workflow#25
Closed
balajinvda wants to merge 16 commits into
Closed
Conversation
balajinvda
force-pushed
the
ci/test-bazel-gha
branch
from
May 16, 2026 13:43
0fb0f8e to
9938fc3
Compare
Mirrors the workflows from the GitLab nvcf/nvcf MR !192. Pushing directly to NVIDIA/nvcf as a PR so the workflows can be validated in their target environment before the upstream merge. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker registry refs must be lowercase. github.repository expanded to NVIDIA/nvcf and rejected both at docker pull (bazel.yml container) and at docker build/push (bazel-ci-image.yml). Hardcode the lowercase 'nvidia/nvcf' form. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The mirror's view of subtrees is keyed off the umbrella's imports.yaml pin; several subtrees have merged Bazel scaffold upstream but the pin hasn't been bumped yet, so their root BUILD.bazel / per-package BUILDs aren't on the mirror. Trim the matrix to entries known good today (grpc-proxy, nats-auth-callout) and document the rest with a note for re-adding as syncs catch up. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Sync MR nvcf/nvcf!193 merged and landed on NVIDIA/nvcf main. Six of eight Phase B subtrees now have their Bazel scaffold on the mirror; the rest still skip via precheck. Re-running this PR will exercise actual builds instead of all-skips. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Picks up kubebuilder-tools envtest binaries added in umbrella !212 so nvca's //pkg/storage:storage_test and //internal/miniservice: miniservice_test run on the public matrix without the KUBEBUILDER_ASSETS panic.
balajinvda
force-pushed
the
ci/test-bazel-gha
branch
from
May 19, 2026 23:33
d87f1fd to
ccd199a
Compare
…il on GHA env) Applies the documented build-only policy for the nvca matrix row. GitLab pipeline 51843044 (post-nvca!1718, sha ddd942a) ran the four external-tagged test targets fresh and passed all of them. GHA matrix run 26131753006 on the same bazel-ci:0.4.0 image (digest-identical to GitLab's via the ghcr.io mirror) fails fresh on: //pkg/nvca:nvca_test FAILED 3/3 (27s) //pkg/operator/reconcile:reconcile_test FAILED 3/3 (53s) //pkg/storage:storage_test FAILED 3/3 (7s) Failing assertions: TestGetNetworkPoliciesDataEmptyDDCSIPList, TestGetNetworkPoliciesDataWithDDCSIPList, TestGetMiniServiceRBACCmData/* -- environment-sensitive tests that depend on something host-side (kernel, libc, Go runtime defaults) that differs between the GitLab CDS runner pool and the GHA ubuntu-latest runner under the same container image. Follow-up (NVCF-10347): identify the host-side delta and re-enable. Adds a field on the matrix entry shape; the test step is gated on . Default is true for all other rows. Refs: NVCF-10337, NVCF-10347
llm-gateway pulls github.com/nvidia-lpu/{harmony,minijinja,parsec} as
Go modules via Gazelle's go_deps extension. The GitLab pipeline uses a
fine-grained PAT (NV_GITHUB_TOKEN) and `git config --global
url.insteadOf` to authenticate the fetch; replicate that on GHA.
The secret is read only on the llm-gateway matrix row (the other
subtrees do not need github.com auth) and the step is a no-op when
the secret is unset, with a warning so the failure mode is obvious.
Add the secret on NVIDIA/nvcf at Settings -> Secrets and variables ->
Actions -> Repository secrets, named NV_GITHUB_TOKEN, scoped read-only
to the three nvidia-lpu repos.
Refs: NVCF-10337
Empty commit to retrigger the bazel workflow against the now-populated NV_GITHUB_TOKEN secret. The prior run (26169489160) saw NV_GITHUB_TOKEN as empty and printed the unset-warning before bazel failed to fetch github.com/nvidia-lpu/minijinja. Refs: NVCF-10337
Brings in the merged !233 GHA fixes (image-credential-helper gazelle override, http-invocation cargo-bazel manifests trim, ratelimiter/nats-auth-callout allowlist + vault key + destination cleanup, rs-autoscaler image push, src/libraries/go/lib/.oss-allowlist Bazel scaffold patterns) so the matrix runs against the fresh OSS mirror. Refs: NVCF-10337
The umbrella's root .bazelrc sets `build --config=remote` by default, routing through nvcfbarn.nvidia.com:8980. GitHub-hosted runners cannot resolve that hostname; umbrella-phase1 was failing with: UNAVAILABLE: Unable to resolve host nvcfbarn.nvidia.com Pass `--remote_cache=` to the bazel build / bazel test steps so the .bazelrc default is overridden with an empty endpoint. Local disk and repo caches still apply. Refs: NVCF-10337
The previous gate `matrix.subtree.tests_enabled != false` evaluated to false on every matrix row that did not set tests_enabled explicitly, because GitHub Actions expressions coerce a missing matrix key to null which compares loose-equal to false. Net effect: the bazel test step was being skipped for every subtree this whole time (umbrella was getting build-only coverage, not the test coverage we believed). Switch to a positive-skip field: `tests_skip: true` on the matrix row opts out of the test step; default is to run. nvca is the only current opt-out (envtest >= 1.30 JobSuccessCriteriaMet failure, NVCF-10347). Refresh the comment to reference the current failure mode instead of the older TestGetNetworkPoliciesData* one. Refs: NVCF-10337
Brings in: - http-invocation .cargo/config.toml (cargo-bazel splice fix) - nats-auth-callout doc fixes - ratelimiter nvcf.pb.go now in mirror (nvossctl validation_exception) - src/libraries/go/lib testdata data attrs + filegroup BUILDs Refs: NVCF-10337
Brings in settings.yaml (force-tracked, fixes http-invocation glob) and nvkit/auth TLS certs filegroup dep (fixes auth_test TLS subtests). Refs: NVCF-10337
Two pre-existing go-lib test bugs were masked by the matrix gate that was silently skipping tests for every row before b941ba1: //src/libraries/go/lib/cmd/icms-translate:icms-translate_test //src/libraries/go/lib/pkg/core:core_test (TestKubeConfiguratorToken goroutine leak) Both are pre-existing in the nvcf-go library and unrelated to the Bazel-CI rollout work. Per the documented build-only policy for GHA test-step failures, flip umbrella-phase1 to tests_skip: true (keep build coverage, skip tests) and let the nvcf-go owners clean up the test layer separately. This is the same containment pattern already applied to nvca. Refs: NVCF-10337
… unblock) Brings the unblocked settings.yaml into runfiles for the http-invocation row to compile. Refs: NVCF-10337
Contributor
|
Workflows landed via GitLab nvcf/nvcf!239 + !240 and now run on NVIDIA/nvcf:main directly. This PR's diff against main is empty, so closing — no merge needed. Final settled state confirmed by run 26195887842 (push-event, all 12 matrix rows green). |
balajinvda
added a commit
that referenced
this pull request
Jul 19, 2026
…fixes - vllm-mp: TP=1 multi-process (separate EngineCore + ZMQ RPC) — criu-v2 e2e PASSES; the historical multi-process restore blocker does not apply to the in-namespace engine. - vllm-tp2: TP=2 eager with the peer-state sever profile (disable custom all-reduce, symm-mem, NCCL P2P/NVLS/SHM -> PYNCCL sockets) — criu-v2 e2e PASSES: 5.3G checkpoint, 57s restore, both TP workers and both GPUs restored, inference serves. First criu-v2 multi-GPU checkpoint/restore. Default config (peer state on) hangs per-rank cuda-checkpoint even with all ranks locked first; a transparent sever is required (issue #25). - test-e2e.sh: workload cases for both; node selection now picks the node with the MOST free GPUs instead of first-fit (first-fit caused repeated UnexpectedAdmissionError on the delete->recreate placeholder cycle). Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
pdmack
pushed a commit
to pdmack/nvcf
that referenced
this pull request
Jul 20, 2026
…VIDIA#225) * feat(nvsnap): enable UV_USE_IO_URING after CRIU SQ-array restore fix libuv/uvloop servers (vLLM, e5-mistral) previously had to pin UV_USE_IO_URING=0 under criu-v2: the restored libuv SQPOLL io_uring ring aborted the event loop on the first epoll_ctl_flush (SIGABRT, nvsnap#107). Root cause was in CRIU's io_uring restore, which recreated a fresh ring but never rebuilt the SQ-array identity map (sq_array[i] = i) that liburing/libuv write once at setup; every restored submission then resolved to SQE 0 and completions never returned. The CRIU fork now rebuilds that identity map on restore (io-uring-cr commit 0a9963ed0). Pin the base build to it and re-enable UV_USE_IO_URING=1 on the single-GPU criu-v2 workloads. - versions.sh: NVSNAP_CRIU_REF -> 0a9963ed0; base v0.0.9; app v0.2.19. - vllm-small, vllm-8b, vllm-qwen32b, e5-mistral: UV_USE_IO_URING 0 -> 1. Validated e2e on aws-dev1 (kernel 6.8): all four checkpoint, restore, and serve inference with io_uring on, zero event-loop aborts. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * test(nvsnap): multi-GPU criu-v2 experiment ladder (E0/E1b) + harness fixes - vllm-mp: TP=1 multi-process (separate EngineCore + ZMQ RPC) — criu-v2 e2e PASSES; the historical multi-process restore blocker does not apply to the in-namespace engine. - vllm-tp2: TP=2 eager with the peer-state sever profile (disable custom all-reduce, symm-mem, NCCL P2P/NVLS/SHM -> PYNCCL sockets) — criu-v2 e2e PASSES: 5.3G checkpoint, 57s restore, both TP workers and both GPUs restored, inference serves. First criu-v2 multi-GPU checkpoint/restore. Default config (peer state on) hangs per-rank cuda-checkpoint even with all ranks locked first; a transparent sever is required (issue NVIDIA#25). - test-e2e.sh: workload cases for both; node selection now picks the node with the MOST free GPUs instead of first-fit (first-fit caused repeated UnexpectedAdmissionError on the delete->recreate placeholder cycle). Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * feat(nvsnap): build cuda-checkpoint from source, drop the committed binary Promote nvsnap-cuda-checkpoint.c (a drop-in cuda-checkpoint CLI on the public CUDA driver checkpoint API, cuCheckpointProcess*, driver 570+) from the research tree into docker/agent/ and compile it in a new cuda-cli-builder stage in Dockerfile.base (nvidia/cuda:12.8.1-devel; links against the toolkit stub, resolves the node's real libcuda at runtime via the existing wrapper). The build fails if the provenance marker string is missing from the produced binary. Replaces the committed NVIDIA prebuilt binary (removed): the public release is x86-64 only, so this unblocks arm64 (no committed binaries per arch), and gives us an extensible CLI (lock --timeout today; r580 gpuPairs migration next). build_app's stale binary-copy requirement is removed — the app image only overlays the wrapper; the binary comes from base. Validated as base v0.0.10 / app v0.2.20 on aws-dev1 (driver 580.126): on-node binary provenance-verified, then the full criu-v2 regression suite passed 6/6 with every checkpoint/restore driven by this CLI — vllm-small, vllm-8b, e5-mistral, vllm-qwen32b (64G), vllm-mp (multi-process), vllm-tp2 (TP=2 multi-GPU, config-severed). Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * chore(nvsnap): bump base v0.0.10 / app v0.2.20 (source-built cuda-checkpoint) Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * chore(nvsnap): add SPDX headers to switch-over files Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * chore(nvsnap): add SPDX header to Dockerfile.base Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * chore(nvsnap): add SPDX headers to all Dockerfiles Backfills the SPDX copyright + license header on the 23 remaining Dockerfiles under src/compute-plane-services/nvsnap. Every source file in the tree (Go, C, Python, YAML, shell, Dockerfile) now carries the header. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * perf(nvsnap): madvise huge-page CRIU restore (-30%) + address review CRIU restore speedup (Lever A): the fork now madvise(MADV_HUGEPAGE)s the premapped private-VMA arena before faulting in the process image, so the content populates as 2MB pages instead of ~800k 4KB pages. Pins CRIU ref to 1ddd5c9c3, bumps base v0.0.11 / app v0.2.21. Measured on aws-dev1 (H100, criu-v2), vllm-small 3.2G: - CPU pages restore phase: 13.5s -> 3.8s (-72%) - CRIU restore total: 40.3s -> 28.2s (-30%) - agent restore: 44.3s -> 32.2s (-27%) Single-GPU regression 5/5 PASS (vllm-small, vllm-8b, e5-mistral, vllm-qwen32b 64G). No RSS bloat: AnonHugePages=0 in the final process (huge pages are transient to the staging arena, split on remap). Design + before/after in docs/proposals/single-gpu-restore-speedup.md. Review fixes (CodeRabbit): - automountServiceAccountToken: false on vllm-mp / vllm-tp2 source pods. - Replace em-dashes with ASCII in CONTRIBUTING.md, vllm-tp2*.yaml. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * docs(nvsnap): full before/after table + address CodeRabbit - BENCHMARK.md: add the Lever A huge-page restore section (matched before/after for all 4 single-GPU workloads, -27% to -51%). - single-gpu-restore-speedup.md (CodeRabbit review): - status reflects implemented (A) vs proposed (B) vs out-of-scope (C). - one consistent restore-time boundary (agent-restore) throughout. - ASCII-only diagrams with text-tagged fences. - regression count corrected to 4/4. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com> * docs(nvsnap): scope restore benchmark claim Refs: NVIDIA#225 --------- Co-authored-by: balaji <balaji7@gmail.com> Co-authored-by: jcameron <jcameron@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Direct-on-GitHub validation of the workflows from the GitLab nvcf/nvcf
MR !192. Two workflows:
bazel-ci-image.ymlrebuildsci/Dockerfile.bazeland publishesto
ghcr.io/NVIDIA/nvcf/bazel-ci(versioned byBAZEL_CI_VERSION).The 0.3.0 + latest tags are already pushed manually for testing.
bazel.ymlrunsbazel build //...andbazel test //...againstthe umbrella + each subtree that has its Bazel scaffold mirrored.
This PR exists to validate the workflows in their target environment.
Final source of truth is the GitLab MR !192; this can be closed once
the workflows pass here.
Co-authored-by: Balaji Ganesan bganesan@nvidia.com