Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .buildkite/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ The four test steps depend on the pre-commit gate. Each suite runs its files
sequentially inside one step because these queues boot a fresh EC2 instance
per job — a per-file matrix would be mostly boot + pip-install time.
Most always-on CPU steps use the standard `python:3.11` image and install their
lightweight dependencies at runtime. `upstream-sync-cpu` uses
`vllm/vime:latest` because the synchronized GLM and checkpoint tests import the
image-pinned Megatron stack even though they do not allocate a GPU.
lightweight dependencies at runtime. `upstream-sync-cpu` uses `VIME_CI_IMAGE`
(defaulting to `vllm/vime:latest`) because the synchronized GLM and checkpoint
tests import the image-pinned Megatron stack even though they do not allocate a GPU.

## Creating the pipeline (one-time, Buildkite UI)

Expand Down Expand Up @@ -70,8 +70,8 @@ startup, so a warm HF cache is all they need. `WANDB_API_KEY` is not wired up
yet; runs report without wandb until it's added (e.g. as a k8s secret in the
pod spec).

GPU jobs use `vllm/vime:latest`. Rebuild and publish that image before validating
Dockerfile or vLLM patch changes.
Set `VIME_CI_IMAGE` to an immutable candidate digest for image-backed jobs;
otherwise they use `vllm/vime:latest`. Do not update `latest` before merge.

## Keeping it in sync

Expand Down
2 changes: 1 addition & 1 deletion .buildkite/gpu_suites.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import subprocess

GPU_QUEUE = "mithril-h100-pool"
CI_IMAGE = "vllm/vime:latest"
CI_IMAGE = os.environ.get("VIME_CI_IMAGE", "vllm/vime:latest")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In Python, os.environ.get("VIME_CI_IMAGE", "vllm/vime:latest") only falls back to the default if the environment variable is completely unset. If VIME_CI_IMAGE is set to an empty string (e.g., VIME_CI_IMAGE=""), it will return "", which is an invalid image name.

Using os.environ.get("VIME_CI_IMAGE") or "vllm/vime:latest" ensures that both unset and empty values correctly fall back to the default image, matching the bash fallback behavior "$${VIME_CI_IMAGE:-vllm/vime:latest}" used in pipeline.yml.

Suggested change
CI_IMAGE = os.environ.get("VIME_CI_IMAGE", "vllm/vime:latest")
CI_IMAGE = os.environ.get("VIME_CI_IMAGE") or "vllm/vime:latest"

HF_CACHE_HOST_PATH = "/mnt/hf-cache"
HF_HOME = "/root/.cache/huggingface"
NODE_INSTANCE_TYPE = "gpu-h100-sxm"
Expand Down
2 changes: 1 addition & 1 deletion .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ steps:
-e GIT_CONFIG_PARAMETERS="'safe.directory=/workspace'" \
-e GLOO_SOCKET_IFNAME=lo -e TP_SOCKET_IFNAME=lo \
-v "$$PWD:/workspace" -w /workspace \
vllm/vime:latest bash -lc '
"$${VIME_CI_IMAGE:-vllm/vime:latest}" bash -lc '
set -euo pipefail
pip install -q -e . --no-deps --break-system-packages
for test_file in \
Expand Down
5 changes: 3 additions & 2 deletions docs/en/developer_guide/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,9 @@ After the CPU steps pass, the Buildkite build exposes a block step named
- `ckpt`

`.buildkite/gpu_suites.py` expands each selected suite into one Buildkite job
per test. GPU tests use `vllm/vime:latest`; rebuild and publish that image
before validating a Dockerfile or vLLM patch change.
per test. Set `VIME_CI_IMAGE` to an immutable candidate digest when validating
Dockerfile or vLLM patch changes. Jobs otherwise use `vllm/vime:latest`, which
must not be updated before the change merges.

## Registering tests

Expand Down
4 changes: 2 additions & 2 deletions docs/zh/developer_guide/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ block step。可以选择一个或多个套件:
- `ckpt`

`.buildkite/gpu_suites.py` 会把所选套件展开为每个测试一个 Buildkite
job。GPU 测试使用 `vllm/vime:latest`;验证 Dockerfile 或 vLLM patch 修改前,
需要先重建并发布该镜像
job。验证 Dockerfile 或 vLLM patch 修改时,通过 `VIME_CI_IMAGE` 指定不可变的
候选镜像 digest;未设置时使用 `vllm/vime:latest`,且 PR 合入前不得更新该标签

## 注册测试

Expand Down