[feature]support vllm-ascend delta disk weight

.buildkite/README-NPU-CI.md

@@ -0,0 +1,111 @@
+# vime NPU CI on Buildkite
+
+The NPU pipeline lives in [`pipeline-npu.yaml`](./pipeline-npu.yaml). It validates
+vime on Ascend NPU hardware.
+
+The NPU suites are behind a **block step** (`:rocket: Run NPU test suites?`):
+click it in the Buildkite UI like GPU CI test suites.
+
+## Pipeline steps
+
+Three steps run in order:
+
+**`pre-commit-npu`** runs the pre-commit gate on all files, always.
+
+**`npu-gate`** is a block step that pauses the pipeline for manual interaction.
+It appears only on PR triggers (`build.source != "schedule"`).  You select which
+NPU suites to run via a multi-select field:
+
+- **`image-build`** — triggers a fresh image build in the `image-build-npu` step;
+  when selected, `smk` is automatically included and all test steps use the newly
+  built image instead of the pre-built default.
+- **`smk`** — runs the smoke test suite.
+- **`nightly`** — runs the nightly test suite.
+
+**`image-build-npu`** the `image-build-npu` step will build and
+push a fresh NPU test image tagged with the current commit. When `image-build` 
+is selected, all test steps generated by `upload-npu-suites` use the newly built
+image instead of the pre-built default image.
+
+This allows testing code changes that require an updated NPU environment
+(e.g., modifications to `docker/Dockerfile.npu`) before they are merged.
+
+**`upload-npu-suites`** reads the `NPU_SUITES` environment variable (or
+`buildkite-agent meta-data get npu-suites` for PR triggers) and generates
+individual test jobs via [`npu_suites.py`](./npu_suites.py).
+
+## Triggers
+
+The pipeline supports two trigger modes:
+
+**PR trigger.** The `npu-gate` block step appears for manual suite selection.
+Suites not selected in the block step are skipped. The `image-build-npu` step
+builds a new image only when `image-build` is chosen in the block.
+
+**Schedule trigger.** There is no block step — the `npu-gate` step is omitted
+entirely. A new image is **always** built, and the suites are determined by the
+`NPU_SUITES` environment variable (set in the scheduled build's pipeline
+configuration), which lists suite names from the `SUITES` dict.
+
+## Adding a test
+
+Suites and test mappings are defined in [`npu_suites.py`](./npu_suites.py). Two
+suites are predefined — `smk` (runs with the `run-ci-npu-smk` label) and `nightly`
+(runs on schedule or with the `run-ci-npu-nightly` label).
+
+Each entry is a 4-tuple:
+
+```python
+("test-qwen3-4B-npu.py", "npu-8", "", {})
+# (test_name, resource_class, extra_args, env_overrides)
+```
+
+- **`test_name`** — the test script under `tests/`, e.g. `test-qwen3-4B-npu.py`.
+- **`resource_class`** — NPU count for the pod (`npu-2`, `npu-4`, `npu-8`,
+  `npu-16`). 
+- **`extra_args`** — extra CLI arguments passed to the test script.
+- **`env_overrides`** — extra environment variables for the test step, e.g.
+  `{"USE_DEEPEP": "1"}`.
+
+A test script should:
+
+1. **Download models and datasets to `HF_HOME`** before training so subsequent
+   steps reuse the cache. The CI sets `HF_HOME=/root/.cache/huggingface`.
+
+2. **Run training** via `train.py` or `train_async.py` with the appropriate
+   NPU-specific arguments.
+
+To register a new suite, add a key to the `SUITES` dict and update
+`selected_suites()` if it should run conditionally. Also add a corresponding
+option in the block step's `fields[].options` list in
+[`pipeline-npu.yaml`](./pipeline-npu.yaml) so it can be selected in the
+Buildkite UI.
+
+To add a test to an existing suite, simply append a new entry tuple to its
+list in `SUITES`.
+
+## Adding or removing a patch
+
+Patches are applied explicitly by [`docker/Dockerfile.npu`](../docker/Dockerfile.npu)
+and reconciled at runtime in the order declared by
+[`docker/npu_patch/series.conf`](../docker/npu_patch/series.conf).
+Each entry declares `target_worktree|image_patch|source_patch`: the image keeps
+the flat `image_patch` under `/opt/npu_patch`, while runtime CI reads
+`source_patch` relative to the current VIME checkout.
+
+When adding a patch file, add the matching Dockerfile COPY/apply operations and
+the `series.conf` entry in the same order. When deleting a patch, remove the
+Dockerfile operation and `series.conf` entry before deleting the patch file;
+the previous image retains the OLD bytes required for runtime revert. Rename
+and reorder changes must likewise update both declarations. Ordinary patch
+changes must not add patch-specific branches to the CI script.
+
+Patch-only changes reuse the default image and are reconciled when the test
+container starts. Changes that add, remove, or upgrade external repositories,
+modify installed dependencies, or otherwise change the Docker image must be
+validated with `image-build`.
+
+The initial rollout of this mechanism requires publishing a new default image
+that contains `/opt/npu_patch/series.conf`, then updating
+`DEFAULT_CI_IMAGE` in `npu_suites.py`. The legacy default image contains patch
+bytes under `/tmp/npu_patch` but not the OLD series required by this reconciler.

.buildkite/npu_suites.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""Emit Buildkite steps for the NPU suites.
+
+Piped into `buildkite-agent pipeline upload` by the npu step in pipeline.yml.
+The suites and their configurations are defined here.
+
+NPU jobs run on the ascend-a3 queue with resource_class determining NPU count
+(e.g., "npu-8" means 8 NPUs).
+
+The selection is read from the NPU_SUITES environment variable. For local
+testing, set NPU_SUITES=smk,nightly instead of having a buildkite-agent on PATH.
+
+stdlib only — runs with the agent host's python3.
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+NPU_QUEUE = "ascend-a3"
+DEFAULT_CI_IMAGE = "quay.io/ascend/vime:vime-latest"
+IMAGE_REGISTRY = "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
+IMAGE_NAME = "vime-ci-npu"
+VIME_IMAGE_TAG = os.environ.get("BUILDKITE_COMMIT", "latest")
+BUILDKITE_SOURCE = os.environ.get("BUILDKITE_SOURCE", "")
+
+# (test_name, resource_class, extra_args, env_overrides)
+SUITES = {
+    "smk": [
+        ("test_qwen3_4B_npu.py", "npu-8", "", {}),
+        ("test_qwen3_30B_A3B_npu.py", "npu-16", "", {}),
+        ("test_qwen3_vl_8B_npu.py", "npu-8", "", {}),
+        ("test_glm4.7_30B_A3B_npu.py", "npu-16", "", {}),
+    ],
+    "nightly": [],
+}
+
+
+def _read_suite_values() -> list[str]:
+    raw = os.environ.get("NPU_SUITES")
+    if raw is None:
+        try:
+            raw = subprocess.run(
+                ["buildkite-agent", "meta-data", "get", "npu-suites"],
+                check=True,
+                capture_output=True,
+                text=True,
+            ).stdout
+        except subprocess.CalledProcessError:
+            raw = ""
+    return [v.strip() for v in raw.replace(",", "\n").splitlines()]
+
+
+def _ci_image() -> str:
+    values = _read_suite_values()
+    if ("image-build" in values) or (BUILDKITE_SOURCE == "schedule"):
+        return f"{IMAGE_REGISTRY}/{IMAGE_NAME}:{VIME_IMAGE_TAG}"
+    return DEFAULT_CI_IMAGE
+
+
+def selected_suites() -> list:
+    values = _read_suite_values()
+    unknown = [v for v in values if v and v not in SUITES and v != "image-build"]
+    if unknown:
+        raise SystemExit(f"unknown suite(s) {unknown}; expected {sorted(SUITES)}")
+    if "image-build" in values:
+        # image-build auto-includes smk tests
+        values.append("smk")
+    return [s for s in SUITES if s in values]
+
+
+def npu_step(suite: str, test_name: str, resource_class: str, extra_args: str, env: dict) -> dict:
+    step_env = {
+        "VIME_TEST_ENABLE_INFINITE_RUN": "false",
+        "BUILDKITE_PULL_REQUEST": os.environ.get("BUILDKITE_PULL_REQUEST", "false"),
+        "BUILDKITE_COMMIT": os.environ.get("BUILDKITE_COMMIT", ""),
+        "HF_TOKEN": "${HF_TOKEN}",
+        "HF_ENDPOINT": "https://hf-mirror.com",
+        "ASCEND_RT_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15",
+        "IMAGE_REGISTRY": IMAGE_REGISTRY,
+        "IMAGE_NAME": IMAGE_NAME,
+        "VIME_IMAGE_TAG": VIME_IMAGE_TAG,
+        "HF_HOME": "/root/.cache/huggingface",
+        **env,
+    }
+
+    commands = "\n".join(
+        [
+            'echo "INFO: update NPU environment"',
+            'if [ -n "${BUILDKITE_COMMIT}" ]; then',
+            "  source /workspace/build/buildkite/.buildkite/scripts/update-npu-environment.sh",
+            "fi",
+            "export HF_HUB_OFFLINE=0",
+            f"python tests/{test_name}{' ' + extra_args if extra_args else ''}",
+        ]
+    )
+    command = f"bash -c '{commands}'"
+
+    label = f":fire: {suite}: {test_name}{' ' + extra_args if extra_args else ''}"
+    step = {
+        "label": label,
+        "depends_on": "image-build-npu",
+        "command": command,
+        "agents": {
+            "queue": NPU_QUEUE,
+            "resource_class": resource_class,
+        },
+        "timeout_in_minutes": 180,
+        "image": _ci_image(),
+        "plugins": [
+            {
+                "kubernetes": {
+                    "podSpecPatch": {
+                        "imagePullSecrets": [{"name": "swr-secret"}],
+                    },
+                }
+            }
+        ],
+        "env": step_env,
+    }
+    return step
+
+
+def main() -> None:
+    steps = [npu_step(suite, *entry) for suite in selected_suites() for entry in SUITES[suite]]
+    json_str = json.dumps({"steps": steps}, indent=2)
+
+    print("--- Generated Pipeline JSON:", file=sys.stderr)
+    print(json_str, file=sys.stderr)
+
+    print(json_str)
+
+
+if __name__ == "__main__":
+    main()

.buildkite/pipeline-npu-image.yaml

@@ -0,0 +1,88 @@
+steps:
+  - label: ":docker: Build and Push NPU Test Image - A3"
+    key: image-build-npu
+    depends_on: 
+      - pre-commit-npu
+      - npu-gate
+    timeout_in_minutes: 240
+    skip: __SKIP_IMAGE_BUILD__
+    agents:
+      queue: "ascend-a3"
+      resource_class: "npu-2"
+    plugins:
+      - kubernetes:
+          metadata:
+            annotations:
+              vault.hashicorp.com/agent-init-first: "true"
+              vault.hashicorp.com/agent-inject: "true"
+              vault.hashicorp.com/agent-inject-perms-ca.pem: "0400"
+              vault.hashicorp.com/agent-inject-perms-cert.pem: "0400"
+              vault.hashicorp.com/agent-inject-perms-config.json: "0400"
+              vault.hashicorp.com/agent-inject-perms-key.pem: "0400"
+              vault.hashicorp.com/agent-inject-secret-ca.pem: internal/data/ascend/buildkitd
+              vault.hashicorp.com/agent-inject-secret-cert.pem: internal/data/ascend/buildkitd
+              vault.hashicorp.com/agent-inject-secret-config.json: internal/data/ascend/buildkitd
+              vault.hashicorp.com/agent-inject-secret-key.pem: internal/data/ascend/buildkitd
+              vault.hashicorp.com/agent-inject-template-ca.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.RootCA }}\n{{- end }}"
+              vault.hashicorp.com/agent-inject-template-cert.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaCert }}\n{{- end }}"
+              vault.hashicorp.com/agent-inject-template-config.json: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.dockerConfig }}\n{{- end }}"
+              vault.hashicorp.com/agent-inject-template-key.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaKey }}\n{{- end }}"
+              vault.hashicorp.com/agent-pre-populate-only: "true"
+              vault.hashicorp.com/agent-run-as-group: "1000"
+              vault.hashicorp.com/agent-run-as-user: "1000"
+              vault.hashicorp.com/agent-service-account-token-volume-name: token-vol
+              vault.hashicorp.com/role: ascend-gha-runners
+              vault.hashicorp.com/secret-volume-path: /home/user/.docker/
+              vault.hashicorp.com/tls-skip-verify: "true"
+          podSpecPatch:
+            volumes:
+            - name: token-vol
+              projected:
+                defaultMode: 420
+                sources:
+                - serviceAccountToken:
+                    audience: api
+                    expirationSeconds: 600
+                    path: token
+    env:
+      VIME_IMAGE_TAG: "${BUILDKITE_COMMIT}"
+      IMAGE_NAME: "vime-ci-npu"
+      IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
+      BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
+    command: |
+      set -ex
+      echo "--- Building and pushing NPU Test Image"
+      echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"
+      echo "buildkitd address: $${BUILDKITD_ADDR}"
+
+      if ! command -v buildctl &> /dev/null; then
+        echo "Installing buildctl..."
+        mkdir -p /tmp/buildkit
+        BUILDKIT_VERSION="v0.29.0"
+        wget -q "https://gh-proxy.test.osinfra.cn/https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
+        tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
+        cp /tmp/buildkit/bin/buildctl /usr/local/bin/
+      fi
+
+      sed -i '/^RUN git config --global http.sslVerify false/i RUN git config --global url."https://gh-proxy.test.osinfra.cn/https://github.com/".insteadOf "https://github.com/"' docker/Dockerfile.npu
+      sed -i '/^# syntax=docker\/dockerfile:1\.7$$/d' docker/Dockerfile.npu
+
+      export DOCKER_CONFIG=/home/user/.docker
+      buildctl \
+        --addr="$${BUILDKITD_ADDR}" \
+        --tlscacert=/home/user/.docker/ca.pem \
+        --tlscert=/home/user/.docker/cert.pem \
+        --tlskey=/home/user/.docker/key.pem \
+        build \
+        --frontend dockerfile.v0 \
+        --local context=. \
+        --local dockerfile=./docker \
+        --opt filename=Dockerfile.npu \
+        --opt build-arg:APTMIRROR=http://cache-service.nginx-pypi-cache.svc.cluster.local:8081 \
+        --opt build-arg:PIP_INDEX_URL=http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
+        --secret id=dockerconfig,src=/home/user/.docker/config.json \
+        --output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG},push=true \
+        --progress=plain
+
+      echo "--- Image pushed successfully"
+      echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"