diff --git a/examples/vime_qwen3_8b_rocm_ablation/README.md b/examples/vime_qwen3_8b_rocm_ablation/README.md index 31674e3e..0a37eaaa 100644 --- a/examples/vime_qwen3_8b_rocm_ablation/README.md +++ b/examples/vime_qwen3_8b_rocm_ablation/README.md @@ -1,5 +1,10 @@ # Vime Qwen3-8B ROCm Attention ablation +The canonical launch command is maintained in +[`../vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md`](../vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md#rocm-entry-points). +This file documents the ROCm-specific acceptance contract; keep host paths and +commands in the shared runbook. + This is the ROCm end-to-end counterpart of PR230's production/RL-Kernel operator matrix. It launches the real Vime orchestration once per Attention cell and requires runtime evidence from both sides: @@ -50,28 +55,11 @@ workers. `rocm_python_entrypoint.sh` is provided for launchers that replace their Python executable. Point `RL_KERNEL_REAL_PYTHON` at the real interpreter and configure Vime to invoke this wrapper. -## Review the launch contract - -Without `--run`, the runner writes only a review summary and does not require a -ROCm host: - -```bash -python examples/vime_qwen3_8b_rocm_ablation/run.py \ - --output-dir /tmp/rocm-attention-ablation \ - -- bash /path/to/vime/scripts/run-qwen3-8B-rocm.sh -``` - -## Execute the full matrix - -```bash -python examples/vime_qwen3_8b_rocm_ablation/run.py \ - --run \ - --output-dir /tmp/rocm-attention-ablation \ - -- bash /path/to/vime/scripts/run-qwen3-8B-rocm.sh -``` +## Run contract -Use `--case R/R` (repeatable) to run a subset while debugging. The final -acceptance run should execute all four cells. +Use the shared runbook for the dry-run and full-matrix commands. `--case R/R` +is repeatable for debugging a subset; the final acceptance run should execute +all four cells. ## Evidence and pass boundary diff --git a/examples/vime_qwen3_8b_tp2_cp2/README.md b/examples/vime_qwen3_8b_tp2_cp2/README.md deleted file mode 100644 index 1bfe3735..00000000 --- a/examples/vime_qwen3_8b_tp2_cp2/README.md +++ /dev/null @@ -1,83 +0,0 @@ -# Vime Qwen3-8B TP=2 CP=2 validation - -This example is the recommended reproducible entry point for the Vime-side -linear_logp integration. It keeps framework glue in Vime and keeps the -numerical provider, contract, provenance, and report in RL-Kernel. - -The example is deliberately strict: - -- Megatron training uses `TP=2`, `CP=2`, `PP=1`, and four actor ranks. -- vLLM rollout uses processed logprobs and `top_p=1.0`. -- Vime must load `rl_engine.integrations.vime.linear_logp_provider.provider` in `strict` mode. -- A native fallback or a missing provider marker is not reported as a pass. -- Attention and FFN are not declared consistent from configuration alone. They - require executed Megatron and vLLM readbacks, so the report marks them - `unclaimed` until those artifacts are supplied. The readback must use - `rlkernel.operator_runtime_evidence.v1` and report exact-zero comparison - metrics for both sides. - -The Vime companion must be installed or checked out separately. This example -does not modify `vllm-project/vime`. - -The executable entry point is the Vime script -`scripts/run-qwen3-8B-rlkernel-tp2-cp2.sh`. Its default topology is an -8-GPU H100 node with four Megatron actor GPUs and four vLLM rollout GPUs. The -script refuses to start on a different GPU count or GPU class. Set -`COLOCATE=1` only when intentionally testing the colocated path; that mode is -not the default 8-GPU train/infer split. - -## Dry run - -```bash -python examples/vime_qwen3_8b_tp2_cp2/run.py \ - --vime-root /path/to/RL-Align/vime \ - --rl-kernel-root /path/to/RL-Kernel \ - --output reports/qwen3_8b_tp2_cp2.validation.json -``` - -## Execute - -The Vime script expects model/checkpoint/data paths through environment -variables. Override them before adding `--run`: - -```bash -export MODEL_ROOT=/models/Qwen3-8B -export TORCH_DIST_ROOT=/models/Qwen3-8B_torch_dist -export PROMPT_DATA=/data/dapo-math-17k.jsonl -export RL_KERNEL_ROOT=/path/to/RL-Kernel -export MEGATRON_ROOT=/path/to/Megatron-LM - -python examples/vime_qwen3_8b_tp2_cp2/run.py \ - --vime-root /path/to/RL-Align/vime \ - --rl-kernel-root "$RL_KERNEL_ROOT" \ - --output reports/qwen3_8b_tp2_cp2.validation.json \ - --run -``` - -For a real 8xH100 run, the model, Megatron torch-dist checkpoint, prompt data, -and Megatron checkout must already exist on the host. The first run can omit -`VIME_CKPT`; the script will initialize from `TORCH_DIST_ROOT` and save the -Vime checkpoint there. Use `NUM_ROLLOUT=1` for the integration smoke test and -increase it only after the provider marker is observed. - -When the Megatron/vLLM launch also emits the operator readback artifact, pass -it explicitly: - -```bash -python examples/vime_qwen3_8b_tp2_cp2/run.py \ - --vime-root /path/to/RL-Align/vime \ - --rl-kernel-root "$RL_KERNEL_ROOT" \ - --runtime-evidence reports/qwen3_8b_tp2_cp2.runtime-evidence.json \ - --output reports/qwen3_8b_tp2_cp2.validation.json \ - --run -``` - -The evidence file is intentionally post-execution. It must include training -and rollout identities for `attention` and `ffn`, plus `passed: true` and -exact-zero `out`, backward, and (for attention) `LSE` comparison metrics. A -configured backend without this readback remains `unclaimed`. - -The runner writes a JSON report and a sibling combined log. The report records -the exact command, both repository revisions, provider backend identity, strict -fallback status, and the claim boundary. It does not fabricate numerical drift -when the GPU run was not executed. diff --git a/examples/vime_qwen3_8b_tp2_cp2/aligned_python_entrypoint.sh b/examples/vime_qwen3_8b_tp2_cp2/aligned_python_entrypoint.sh deleted file mode 100755 index eb1cf61b..00000000 --- a/examples/vime_qwen3_8b_tp2_cp2/aligned_python_entrypoint.sh +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -REAL_PYTHON="${RL_KERNEL_REAL_PYTHON:?RL_KERNEL_REAL_PYTHON must name the real Python executable}" -RL_KERNEL_ROOT="${RL_KERNEL_ROOT:-$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)}" -export RL_KERNEL_ROOT -export PYTHONPATH="${RL_KERNEL_ROOT}:${PYTHONPATH:-}" - -if [[ "${1:-}" == "train.py" || "${1:-}" == */train.py ]]; then - # Ray job submission does not always forward the shell exports used by the - # outer launcher. A strict linear-logp job must still install the matching - # vLLM hooks; otherwise it silently falls back to native attention/FFN and - # loses the R/R performance path. Preserve explicit ablation selections. - strict_linear_logp=0 - rollout_batch_size="" - n_samples_per_prompt="1" - explicit_vllm_execution_config=0 - previous_arg="" - for current_arg in "$@"; do - if [[ "${previous_arg}" == "--linear-logp-provider-mode" && "${current_arg}" == "strict" ]]; then - strict_linear_logp=1 - elif [[ "${previous_arg}" == "--rollout-batch-size" ]]; then - rollout_batch_size="${current_arg}" - elif [[ "${previous_arg}" == "--n-samples-per-prompt" ]]; then - n_samples_per_prompt="${current_arg}" - fi - - case "${current_arg}" in - --linear-logp-provider-mode=strict) - strict_linear_logp=1 - ;; - --rollout-batch-size=*) - rollout_batch_size="${current_arg#*=}" - ;; - --n-samples-per-prompt=*) - n_samples_per_prompt="${current_arg#*=}" - ;; - --vllm-enforce-eager|--vllm-optimization-level|--vllm-optimization-level=*|--vllm-compilation-config|--vllm-compilation-config=*) - explicit_vllm_execution_config=1 - ;; - esac - previous_arg="${current_arg}" - done - - strict_cudagraph_args=() - if [[ "${strict_linear_logp}" == "1" ]]; then - export RL_KERNEL_VLLM_INTEGRATION="${RL_KERNEL_VLLM_INTEGRATION:-1}" - export RL_KERNEL_CUDA_ONLY="${RL_KERNEL_CUDA_ONLY:-1}" - export VIME_RL_KERNEL_STRICT="${VIME_RL_KERNEL_STRICT:-1}" - export RL_KERNEL_ATTENTION_CASE="${RL_KERNEL_ATTENTION_CASE:-R/R}" - export RL_KERNEL_FFN_CASE="${RL_KERNEL_FFN_CASE:-R/R}" - export RL_KERNEL_LOGP_CASE="${RL_KERNEL_LOGP_CASE:-R/R}" - - # Strict rollout kernels preserve their arithmetic order under CUDA Graph. - # Capturing the complete decode graph removes the per-layer host-launch - # gaps that otherwise dominate small decode batches. Capture every exact - # batch size: padding a strict custom kernel to a larger sparse graph can - # access invalid slots and, more importantly, changes the tested contract. - # Explicit vLLM execution flags always win so callers can opt out. - if [[ "${explicit_vllm_execution_config}" == "0" ]]; then - if [[ "${rollout_batch_size}" =~ ^[1-9][0-9]*$ && "${n_samples_per_prompt}" =~ ^[1-9][0-9]*$ ]]; then - max_capture_size=$((rollout_batch_size * n_samples_per_prompt)) - if [[ -n "${RL_KERNEL_VLLM_CUDAGRAPH_MAX_CAPTURE_SIZE:-}" ]]; then - max_capture_size="${RL_KERNEL_VLLM_CUDAGRAPH_MAX_CAPTURE_SIZE}" - fi - if ! [[ "${max_capture_size}" =~ ^[1-9][0-9]*$ ]]; then - echo "RL_KERNEL_VLLM_CUDAGRAPH_MAX_CAPTURE_SIZE must be a positive integer" >&2 - exit 2 - fi - - capture_sizes="[" - for ((batch_size = 1; batch_size <= max_capture_size; batch_size++)); do - if ((batch_size > 1)); then - capture_sizes+="," - fi - capture_sizes+="${batch_size}" - done - capture_sizes+="]" - compilation_config="{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":${capture_sizes},\"max_cudagraph_capture_size\":${max_capture_size}}" - strict_cudagraph_args=( - --vllm-optimization-level 0 - --vllm-compilation-config "${compilation_config}" - ) - echo "[RL-Kernel] strict vLLM full-decode CUDA Graph capture sizes: ${capture_sizes}" >&2 - else - echo "[RL-Kernel] strict CUDA Graph disabled: rollout batch size is unavailable" >&2 - fi - fi - fi - exec "${REAL_PYTHON}" "$@" \ - --seed 1234 \ - --rollout-seed 42 \ - --vllm-enable-deterministic-inference \ - --vllm-attention-backend flash_attn \ - --vllm-disable-custom-all-reduce \ - --deterministic-mode \ - --accumulate-allreduce-grads-in-fp32 \ - "${strict_cudagraph_args[@]}" -fi - -exec "${REAL_PYTHON}" "$@" diff --git a/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json b/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json deleted file mode 100644 index 2351e554..00000000 --- a/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "schema_version": "rlkernel.vime_qwen3_8b_tp2_cp2.v1", - "model": "Qwen/Qwen3-8B", - "training": { - "framework": "megatron", - "tensor_model_parallel_size": 2, - "context_parallel_size": 2, - "pipeline_model_parallel_size": 1, - "world_size": 4, - "dtype": "bf16" - }, - "rollout": { - "framework": "vllm", - "top_p": 1.0, - "logprobs_mode": "processed_logprobs" - }, - "linear_logp_provider": { - "path": "rl_engine.integrations.vime.linear_logp_provider.provider", - "mode": "strict", - "backend_id": "pytorch-vocab-parallel-logp-ws2", - "real_vocab_size": 151936, - "padded_vocab_size": 152064, - "num_vocab_tiles": 64 - }, - "operator_evidence": { - "logp": { - "training": "rl-kernel-provider", - "rollout": "vllm-native-processed-logprobs", - "required_runtime_marker": "linear_logp provider active" - }, - "attention": { - "training": "runtime-readback-required", - "rollout": "runtime-readback-required", - "status": "not_claimed_without_Megatron_and_vLLM_readback" - }, - "ffn": { - "training": "runtime-readback-required", - "rollout": "runtime-readback-required", - "status": "not_claimed_without_Megatron_and_vLLM_readback" - } - }, - "vime_script": "scripts/run-qwen3-8B-rlkernel-tp2-cp2.sh" -} diff --git a/examples/vime_qwen3_8b_tp2_cp2/run.py b/examples/vime_qwen3_8b_tp2_cp2/run.py deleted file mode 100644 index bba0c939..00000000 --- a/examples/vime_qwen3_8b_tp2_cp2/run.py +++ /dev/null @@ -1,271 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2026 RL-Kernel Contributors - -"""Run and archive the Vime Qwen3-8B TP=2/CP=2 validation entry point. - -This is an integration example, not a synthetic pass generator. A dry run -only records the exact launch contract. ``--run`` executes Vime and records -whether the strict RL-Kernel provider was actually observed in the log. The -report deliberately leaves attention/FFN unclaimed until both framework -readbacks are supplied. -""" - -from __future__ import annotations - -import argparse -import json -import os -import subprocess -from datetime import datetime, timezone -from pathlib import Path -from typing import Any, Mapping - -DEFAULT_CONFIG = Path(__file__).with_name("qwen3_8b_tp2_cp2.json") -PROVIDER_MARKER = "linear_logp provider active" -FALLBACK_MARKERS = ("using native path", "fallback=True", "fallback=true") -RUNTIME_EVIDENCE_SCHEMA = "rlkernel.operator_runtime_evidence.v1" -_OPERATOR_METRICS = { - "attention": ("out_max_abs", "lse_max_abs", "dq_max_abs", "dk_max_abs", "dv_max_abs"), - "ffn": ("out_max_abs", "dx_max_abs", "dw_max_abs"), -} - - -def load_config(path: Path) -> dict[str, Any]: - with path.open(encoding="utf-8") as handle: - value = json.load(handle) - if not isinstance(value, dict): - raise ValueError("example config must contain a JSON object") - return value - - -def validate_config(config: Mapping[str, Any]) -> None: - training = config.get("training") - rollout = config.get("rollout") - provider = config.get("linear_logp_provider") - if ( - not isinstance(training, Mapping) - or not isinstance(rollout, Mapping) - or not isinstance(provider, Mapping) - ): - raise ValueError("training, rollout, and linear_logp_provider sections are required") - expected = { - "tensor_model_parallel_size": 2, - "context_parallel_size": 2, - "pipeline_model_parallel_size": 1, - "world_size": 4, - } - for name, value in expected.items(): - if training.get(name) != value: - raise ValueError(f"training.{name} must be {value!r}") - if rollout.get("top_p") != 1.0: - raise ValueError("rollout.top_p must remain 1.0 for the strict provider contract") - if provider.get("mode") != "strict": - raise ValueError("linear_logp_provider.mode must be strict") - if provider.get("path") != "rl_engine.integrations.vime.linear_logp_provider.provider": - raise ValueError("example must use the RL-Kernel Vime provider") - if provider.get("backend_id") != "pytorch-vocab-parallel-logp-ws2": - raise ValueError("example must pin the WS2 vocab-parallel backend") - - -def load_runtime_evidence(path: Path | None) -> dict[str, Any] | None: - """Load post-execution readback without treating configuration as evidence.""" - - if path is None: - return None - with path.open(encoding="utf-8") as handle: - value = json.load(handle) - if not isinstance(value, dict) or value.get("schema_version") != RUNTIME_EVIDENCE_SCHEMA: - raise ValueError(f"runtime evidence must use schema {RUNTIME_EVIDENCE_SCHEMA!r}") - return value - - -def _operator_evidence_status(evidence: Mapping[str, Any] | None, operator: str) -> str: - if evidence is None: - return "unclaimed" - operators = evidence.get("operators") - item = operators.get(operator) if isinstance(operators, Mapping) else None - if not isinstance(item, Mapping): - return "unclaimed" - training = item.get("training") - rollout = item.get("rollout") - comparison = item.get("comparison") - if not isinstance(training, Mapping) or not isinstance(rollout, Mapping): - return "unclaimed" - if not isinstance(comparison, Mapping) or comparison.get("passed") is not True: - return "failed" - required_identity = ("implementation_id", "backend_id", "contract_id") - if any(not training.get(name) or not rollout.get(name) for name in required_identity): - return "failed" - if training["implementation_id"] != rollout["implementation_id"]: - return "failed" - for metric in _OPERATOR_METRICS[operator]: - value = comparison.get(metric) - if not isinstance(value, (int, float)) or isinstance(value, bool) or value != 0.0: - return "failed" - return "passed" - - -def validate_runtime_evidence(evidence: Mapping[str, Any] | None) -> None: - """Reject malformed evidence before it can affect a report.""" - - if evidence is None: - return - for operator in _OPERATOR_METRICS: - status = _operator_evidence_status(evidence, operator) - if status == "failed": - raise ValueError(f"runtime evidence for {operator} is incomplete or non-zero") - - -def _revision(path: Path) -> str | None: - try: - result = subprocess.run( - ["git", "-C", str(path), "rev-parse", "HEAD"], - check=True, - capture_output=True, - text=True, - ) - except (OSError, subprocess.CalledProcessError): - return None - return result.stdout.strip() or None - - -def build_environment(vime_root: Path, rl_kernel_root: Path) -> dict[str, str]: - env = dict(os.environ) - existing = [str(vime_root), str(rl_kernel_root), "/root/Megatron-LM"] - if env.get("PYTHONPATH"): - existing.append(env["PYTHONPATH"]) - env["PYTHONPATH"] = os.pathsep.join(existing) - env["RL_KERNEL_ROOT"] = str(rl_kernel_root) - env["TP_SIZE"] = "2" - env["CP_SIZE"] = "2" - env["ROLLOUT_TOP_P"] = "1.0" - return env - - -def build_command(config: Mapping[str, Any], vime_root: Path) -> list[str]: - script = vime_root / str(config.get("vime_script", "")) - if not script.is_file(): - raise FileNotFoundError(f"Vime entry script does not exist: {script}") - return ["bash", str(script)] - - -def build_report( - config: Mapping[str, Any], - *, - vime_root: Path, - rl_kernel_root: Path, - command: list[str], - status: str, - returncode: int | None, - log_text: str, - log_path: Path | None, - runtime_evidence: Mapping[str, Any] | None = None, - runtime_evidence_path: Path | None = None, -) -> dict[str, Any]: - provider_active = PROVIDER_MARKER in log_text - fallback_observed = any(marker in log_text for marker in FALLBACK_MARKERS) - strict_provider_passed = status == "passed" and provider_active and not fallback_observed - effective_status = ( - "passed" if strict_provider_passed else ("failed" if status == "passed" else status) - ) - attention_status = _operator_evidence_status(runtime_evidence, "attention") - ffn_status = _operator_evidence_status(runtime_evidence, "ffn") - return { - "schema_version": "rlkernel.vime_validation_report.v1", - "created_at": datetime.now(timezone.utc).isoformat(), - "status": effective_status, - "claim_boundary": { - "qwen3_8b_tp2_cp2_vime_training": strict_provider_passed, - "attention_train_infer_consistency": attention_status, - "ffn_train_infer_consistency": ffn_status, - "reason": ( - "attention and FFN require executed Megatron/vLLM runtime readbacks; " - "the evidence contract accepts only exact-zero comparison metrics" - ), - }, - "config": dict(config), - "topology": config["training"], - "provider": { - "configured_path": config["linear_logp_provider"]["path"], - "configured_mode": config["linear_logp_provider"]["mode"], - "backend_id": config["linear_logp_provider"]["backend_id"], - "active_observed": provider_active, - "fallback_observed": fallback_observed, - }, - "command": command, - "returncode": returncode, - "artifacts": { - "log": None if log_path is None else str(log_path), - "runtime_evidence": ( - None if runtime_evidence_path is None else str(runtime_evidence_path) - ), - }, - "runtime_evidence": None if runtime_evidence is None else dict(runtime_evidence), - "revisions": { - "vime": _revision(vime_root), - "rl_kernel": _revision(rl_kernel_root), - }, - } - - -def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG) - parser.add_argument("--vime-root", type=Path, default=Path(os.environ.get("VIME_ROOT", "."))) - parser.add_argument( - "--rl-kernel-root", type=Path, default=Path(os.environ.get("RL_KERNEL_ROOT", ".")) - ) - parser.add_argument("--output", type=Path, default=Path("qwen3_8b_tp2_cp2.validation.json")) - parser.add_argument( - "--runtime-evidence", - type=Path, - default=None, - help="post-execution Megatron/vLLM operator readback JSON (strict exact-zero contract)", - ) - parser.add_argument("--run", action="store_true", help="execute the Vime script") - args = parser.parse_args(argv) - - config = load_config(args.config) - validate_config(config) - runtime_evidence = load_runtime_evidence(args.runtime_evidence) - validate_runtime_evidence(runtime_evidence) - vime_root = args.vime_root.resolve() - rl_kernel_root = args.rl_kernel_root.resolve() - command = build_command(config, vime_root) - - status = "not_run" - returncode: int | None = None - log_text = "" - log_path: Path | None = None - if args.run: - args.output.parent.mkdir(parents=True, exist_ok=True) - log_path = args.output.with_suffix(".log") - env = build_environment(vime_root, rl_kernel_root) - with log_path.open("w", encoding="utf-8") as log_handle: - process = subprocess.run( - command, cwd=vime_root, env=env, stdout=log_handle, stderr=subprocess.STDOUT - ) - returncode = process.returncode - log_text = log_path.read_text(encoding="utf-8", errors="replace") - status = "passed" if returncode == 0 else "failed" - - report = build_report( - config, - vime_root=vime_root, - rl_kernel_root=rl_kernel_root, - command=command, - status=status, - returncode=returncode, - log_text=log_text, - log_path=log_path, - runtime_evidence=runtime_evidence, - runtime_evidence_path=args.runtime_evidence, - ) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") - print(json.dumps(report, indent=2, sort_keys=True)) - return 0 if report["status"] in {"passed", "not_run"} else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/vime_qwen3_8b_tp2_cp2/validate_artifacts.py b/examples/vime_qwen3_8b_tp2_cp2/validate_artifacts.py deleted file mode 100644 index bc7a1705..00000000 --- a/examples/vime_qwen3_8b_tp2_cp2/validate_artifacts.py +++ /dev/null @@ -1,212 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2026 RL-Kernel Contributors - -"""Validate CUDA-only framework readbacks and Vime train/rollout Logp dumps.""" - -from __future__ import annotations - -import argparse -import json -import math -from pathlib import Path -from typing import Any, Mapping - -import torch - -from rl_engine.integrations.runtime import _contains_triton, _runtime_platform - -_FRAMEWORKS = (("megatron", "training"), ("vllm", "rollout")) -_MODULES = ("attention", "ffn", "logp") -_STRICT_LOGP_BACKEND = "rlkernel.linear_logp.bitwise.v1" -_BACKEND_PREFIXES = ("rlkernel.", "pytorch-vocab-parallel-logp") - - -def load_readbacks(directory: Path) -> list[dict[str, Any]]: - values: list[dict[str, Any]] = [] - for path in sorted(directory.glob("*.json")): - value = json.loads(path.read_text(encoding="utf-8")) - if not isinstance(value, dict): - raise ValueError(f"readback must contain an object: {path}") - value["_path"] = str(path) - values.append(value) - if not values: - raise ValueError(f"no framework readbacks found in {directory}") - return values - - -def validate_readbacks(readbacks: list[dict[str, Any]]) -> dict[str, Any]: - errors: list[str] = [] - frameworks: dict[str, Any] = {} - for framework, target in _FRAMEWORKS: - matching = [ - value - for value in readbacks - if value.get("framework") == framework and value.get("target") == target - ] - label = f"{framework}/{target}" - if not matching: - errors.append(f"missing {label} readback") - continue - module_summary: dict[str, Any] = {} - for value in matching: - if value.get("fallbacks"): - errors.append(f"{label} recorded fallback: {value['fallbacks']}") - for module in _MODULES: - hook_count = sum(module in value.get("installed_hooks", {}) for value in matching) - records = [ - value["operators"][module] - for value in matching - if isinstance(value.get("operators"), Mapping) and module in value["operators"] - ] - call_count = sum(int(record.get("call_count", 0)) for record in records) - if hook_count == 0: - errors.append(f"{label} {module} hook was not installed") - if call_count == 0: - errors.append(f"{label} {module} had zero calls") - backends = sorted({str(record.get("backend_id", "")) for record in records}) - for record in records: - backend = str(record.get("backend_id", "")) - if module == "logp" and backend != _STRICT_LOGP_BACKEND: - errors.append( - f"{label} logp used {backend!r}, expected {_STRICT_LOGP_BACKEND!r}" - ) - elif not backend.startswith(_BACKEND_PREFIXES): - errors.append(f"{label} {module} used unexpected backend {backend!r}") - if _contains_triton(record): - errors.append(f"{label} {module} used Triton") - if _runtime_platform(record.get("provenance")) != "cuda": - errors.append(f"{label} {module} did not report CUDA execution") - module_summary[module] = { - "installed_processes": hook_count, - "call_count": call_count, - "backend_ids": backends, - } - frameworks[label] = { - "readback_count": len(matching), - "modules": module_summary, - } - return {"passed": not errors, "errors": errors, "frameworks": frameworks} - - -def _load_train_dump(path: Path) -> Mapping[str, Any]: - value = torch.load(path, map_location="cpu", weights_only=False) - if not isinstance(value, Mapping): - raise ValueError(f"train dump must contain a mapping: {path}") - return value - - -def compare_train_rollout_logps(paths: list[Path]) -> dict[str, Any]: - sample_count = 0 - element_count = 0 - mismatch_count = 0 - max_abs_diff = 0.0 - errors: list[str] = [] - for path in paths: - payload = _load_train_dump(path) - samples = payload.get("samples") - if not isinstance(samples, list): - rollout_data = payload.get("rollout_data") - if not isinstance(rollout_data, Mapping): - errors.append(f"{path} has neither samples nor rollout_data") - continue - training_values = rollout_data.get("log_probs") - rollout_values = rollout_data.get("rollout_log_probs") - if not isinstance(training_values, (list, tuple)) or not isinstance( - rollout_values, (list, tuple) - ): - errors.append(f"{path} rollout_data lacks list log_probs/rollout_log_probs") - continue - if len(training_values) != len(rollout_values): - errors.append( - f"{path} logprob list length mismatch: " - f"{len(training_values)} != {len(rollout_values)}" - ) - samples = [ - {"log_probs": training, "rollout_log_probs": rollout} - for training, rollout in zip(training_values, rollout_values, strict=False) - ] - for sample_index, sample in enumerate(samples): - if not isinstance(sample, Mapping): - errors.append(f"{path} sample {sample_index} is not a mapping") - continue - training = sample.get("log_probs") - rollout = sample.get("rollout_log_probs") - if not isinstance(training, torch.Tensor) or not isinstance(rollout, torch.Tensor): - errors.append( - f"{path} sample {sample_index} lacks tensor log_probs/rollout_log_probs" - ) - continue - sample_count += 1 - if training.shape != rollout.shape: - errors.append( - f"{path} sample {sample_index} shape mismatch: " - f"{tuple(training.shape)} != {tuple(rollout.shape)}" - ) - continue - if training.dtype != rollout.dtype: - errors.append( - f"{path} sample {sample_index} dtype mismatch: " - f"{training.dtype} != {rollout.dtype}" - ) - element_count += training.numel() - mismatch_count += int(torch.ne(training, rollout).sum().item()) - if training.numel(): - diff = (training.float() - rollout.float()).abs() - if not bool(torch.isfinite(diff).all().item()): - errors.append(f"{path} sample {sample_index} has non-finite drift") - else: - max_abs_diff = max(max_abs_diff, float(diff.max().item())) - if not paths: - errors.append("no Vime train dump was found") - if sample_count == 0: - errors.append("no comparable train/rollout samples were found") - torch_equal = not errors and mismatch_count == 0 - return { - "passed": torch_equal and max_abs_diff == 0.0, - "torch_equal": torch_equal, - "mismatch_count": mismatch_count, - "max_abs_diff": max_abs_diff if math.isfinite(max_abs_diff) else None, - "sample_count": sample_count, - "element_count": element_count, - "errors": errors, - "artifacts": [str(path) for path in paths], - } - - -def validate_artifacts(readback_dir: Path, train_data_dir: Path) -> dict[str, Any]: - readbacks = validate_readbacks(load_readbacks(readback_dir)) - train_paths = sorted(train_data_dir.glob("*.pt")) - bitwise = compare_train_rollout_logps(train_paths) - return { - "schema_version": "rlkernel.vime_cuda_bitwise_validation.v1", - "passed": bool(readbacks["passed"] and bitwise["passed"]), - "runtime_policy": {"platform": "cuda", "triton_allowed": False}, - "readbacks": readbacks, - "train_rollout_logp": bitwise, - } - - -def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--readback-dir", type=Path, required=True) - parser.add_argument("--train-data-dir", type=Path, required=True) - parser.add_argument("--output", type=Path, required=True) - args = parser.parse_args(argv) - - try: - report = validate_artifacts(args.readback_dir, args.train_data_dir) - except Exception as exc: - report = { - "schema_version": "rlkernel.vime_cuda_bitwise_validation.v1", - "passed": False, - "runtime_policy": {"platform": "cuda", "triton_allowed": False}, - "error": str(exc), - } - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") - print(json.dumps(report, indent=2, sort_keys=True)) - return 0 if report["passed"] else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/vime_qwen3_8b_tp4_cp2_200/README.md b/examples/vime_qwen3_8b_tp4_cp2_200/README.md index e51aceb6..6140963e 100644 --- a/examples/vime_qwen3_8b_tp4_cp2_200/README.md +++ b/examples/vime_qwen3_8b_tp4_cp2_200/README.md @@ -16,6 +16,10 @@ succeeds, every expected operator route has runtime execution evidence, no fallback or Triton route is observed for an R/R arm, the requested number of steps is present, and vLLM CUDA Graph evidence matches the manifest. +All launch commands are maintained in [`REPRODUCTION.md`](REPRODUCTION.md). +This README describes the experiment and its acceptance boundary; it does not +duplicate host-specific launch commands. + ## Ablation matrix | Group | VIME `--use-rollout-logprobs` | Attention / FFN / logp | Purpose | @@ -32,6 +36,16 @@ directly. `R/R` selects RL-Kernel on both sides and installs the strict RL-Kernel linear-logp provider. All four groups use the same prompts, initial checkpoint, sampling settings, seeds, TP4/CP2 topology, and batch sizes. +The CUDA module ablation is the `M000`-`M111` matrix defined by the same +`run_arm.py` and `experiment_matrix.json`; it is intentionally kept under this +TP4/CP2 example rather than duplicated in a CUDA-only directory. The canonical +command is in [`REPRODUCTION.md`](REPRODUCTION.md#cuda-module-ablation). + +The ROCm P/R and Attention-only runners are separate because they select HIP, +AITER/CK, and RCCL-specific routes that cannot pass the CUDA validation gates. +The Attention-only runner is a historical attribution diagnostic, not a second +definition of the module matrix. + Do not interpret G10/G11 as evidence that train and rollout recomputation is bitwise equal: framework reuse changes which stored logp enters the RL loss. The direct numerical claim comes from G01/G11 and the runtime comparison @@ -107,60 +121,12 @@ The converter requires `pyarrow`. The small `qwen3_8b_multiround_math.jsonl` file is a developer fixture and must not be used for experiment or reward claims. -## Run one arm - -The complete host setup, data and checkpoint preparation, exact historical -revision table, formal 200-step launch commands, Ray log capture, validation, -and performance-analysis commands are recorded in -[`REPRODUCTION.md`](REPRODUCTION.md). The short example below is schematic; -every expanded path and command is recorded in `manifest.json`. - -```bash -python examples/vime_qwen3_8b_tp4_cp2_200/run_arm.py \ - --group G01 \ - --num-rollout 8 \ - --seed 1234 \ - --rollout-seed 1234 \ - --output-root /data/vime-200/runs/short \ - --rl-kernel-root /path/to/RL-Kernel \ - --vime-root /path/to/vime \ - --megatron-root /path/to/Megatron-LM \ - --model-root /models/Qwen3-8B \ - --ref-load /models/Qwen3-8B_torch_dist \ - --prompt-data /data/dapo-math-17k.vime.jsonl \ - --python /path/to/python \ - --ray-bin /path/to/ray \ - --wait -``` - -`run_arm.py` refuses to reuse an existing run ID. It records repository -revisions, command line, environment, data hash, GPU inventory, topology, -seeds, batch parameters, and CUDA Graph contract before submission. +## Reproduction -After the Ray job finishes, save its combined log as `run.log` in the run -directory and validate it: - -```bash -python examples/vime_qwen3_8b_tp4_cp2_200/validate_run.py \ - --run-dir /data/vime-200/runs/short/ \ - --seal -``` - -## Aggregate and plot - -Only sealed runs are collected. `collect_results.py` writes one row per run, -one row per training step, and group-level summaries. - -```bash -python examples/vime_qwen3_8b_tp4_cp2_200/collect_results.py \ - --runs-root /data/vime-200/runs \ - --output-dir /data/vime-200/results - -python examples/vime_qwen3_8b_tp4_cp2_200/plot_results.py \ - --rounds-csv /data/vime-200/results/rounds.csv \ - --phase convergence \ - --output-dir /data/vime-200/results/figures -``` +Use [`REPRODUCTION.md`](REPRODUCTION.md) for the complete host setup, data and +checkpoint preparation, CUDA 200-step launch, CUDA module matrix, ROCm +entrypoints, Ray log capture, validation, and performance analysis commands. +The runbook is the single source of truth for commands and paths. The plotting step requires Matplotlib. It produces: diff --git a/examples/vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md b/examples/vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md index 0697fe4d..3388261d 100644 --- a/examples/vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md +++ b/examples/vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md @@ -40,7 +40,7 @@ when using another machine. ```bash set -euo pipefail -export EXPERIMENT_ROOT=/home/ellm/ljj/vime_qwen3_8b_tp2_cp2_200_experiment +export EXPERIMENT_ROOT=/home/ellm/ljj/vime_qwen3_8b_tp4_cp2_200_experiment export DATA_ROOT=/data/ellm/vime_qwen3_8b_tp4_cp2_200_experiment export RLK_ROOT=$EXPERIMENT_ROOT/RL-Kernel export VIME_ROOT=$EXPERIMENT_ROOT/vime @@ -259,6 +259,76 @@ test -f "$RUN_DIR/COMPLETE" --output-dir "$DATA_ROOT/results/figures" ``` +## CUDA module ablation + +The CUDA module matrix is part of this TP4/CP2 example. It is not a separate +CUDA directory: `run_supplement_suite.py` drives the eight `M000`-`M111` arms +through the same validated `run_arm.py` and `validate_run.py` path. + +Run it only with a fresh suite ID and an empty output directory: + +```bash +export MODULE_SUITE_ID=cuda-module-$(date -u +%Y%m%dT%H%M%SZ) +export MODULE_ROOT="$DATA_ROOT/runs/module/$MODULE_SUITE_ID" +mkdir -p "$MODULE_ROOT" + +env -u PYTHONPATH \ + CUDNN_FRONTEND_CUDART_LIB_NAME="$CUDA_RUNTIME_ROOT/lib/libcudart.so.12" \ + "$PYTHON" "$EXAMPLE_ROOT/run_supplement_suite.py" \ + --phase module \ + --suite-id "$MODULE_SUITE_ID" \ + --output-root "$MODULE_ROOT" \ + --rl-kernel-root "$RLK_ROOT" \ + --vime-root "$VIME_ROOT" \ + --megatron-root "$MEGATRON_ROOT" \ + --model-root "$HF_MODEL_ROOT" \ + --ref-load "$TORCH_DIST_ROOT" \ + --prompt-data "$PROMPT_DATA" \ + --python "$PYTHON" \ + --ray-bin "$RAY" \ + --extra-pythonpath "$RUNTIME_SITE" \ + --extra-pythonpath "$CUDA_PYTHON_SITE" \ + --extra-pythonpath "$TE218_ROOT" \ + --ld-library-path "$CUDA_RUNTIME_ROOT/lib:$TE218_ROOT/transformer_engine/wheel_lib" +``` + +The resulting summary is `$MODULE_ROOT/$MODULE_SUITE_ID.summary.json`. The +matrix uses eight rollouts, one prompt with eight samples, seed 1234, and the +same TP4/CP2 validation gates as the long run. Do not copy the old TP2/CP2 +commands; that example has been retired. + +## ROCm entry points + +ROCm is a separate backend and must not use the CUDA `run_arm.py` command: the +CUDA runbook requires `nvidia-smi`, CUDA Graph evidence, and Transformer Engine +libraries. The maintained ROCm launchers remain available, but their commands +are documented here so the repository has one command index: + +```bash +# Full ROCm VIME operator matrix (P/P, P/R, R/P, R/R) +python examples/vime_qwen3_8b_rocm_ablation/run.py \ + --run \ + --output-dir /tmp/rocm-vime-ablation \ + -- bash /path/to/vime/scripts/run-qwen3-8B-rocm.sh + +# Historical Attention-only cross-configuration diagnostic +python examples/vime_rocm_attention_ablation/run.py \ + --vime-root /work/vime \ + --rl-kernel-root /work/RL-Kernel \ + --megatron-root /work/Megatron-LM-vime \ + --model-root /app/model/Qwen3-8B \ + --reference-checkpoint /app/model/Qwen3-8B_torch_dist \ + --prompt-data /app/model/dapo-math-17k/dapo-math-17k.jsonl \ + --run-dir /work/RL-Kernel/runs/vime-rocm-attention-$(date -u +%Y%m%dT%H%M%SZ) \ + --run +``` + +The Attention-only runner was introduced earlier for ROCm route attribution +(PR #385) and fixes FFN/Logp while varying only the Attention P/R pairing. It +is not the CUDA/ROCm `M000`-`M111` module matrix and its results must not be +presented as the primary TP4/CP2 ablation. Keep it only when reproducing that +historical diagnostic; use the full-path matrix for current evidence. + ## Performance analysis commands The performance analyser reads the emitted per-step timers instead of rounded diff --git a/examples/vime_rocm_attention_ablation/README.md b/examples/vime_rocm_attention_ablation/README.md index c9ce8913..eb49ee2c 100644 --- a/examples/vime_rocm_attention_ablation/README.md +++ b/examples/vime_rocm_attention_ablation/README.md @@ -1,5 +1,10 @@ # Vime ROCm Attention operator ablation +The canonical command is maintained in +[`../vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md`](../vime_qwen3_8b_tp4_cp2_200/REPRODUCTION.md#rocm-entry-points). +This file records the historical Attention-only route-attribution contract; +it is not the primary CUDA/ROCm `M000`-`M111` module matrix. + This example runs one real Vime rollout/training step for each Attention implementation pairing: @@ -46,7 +51,7 @@ Use a ROCm environment with Vime, Megatron-LM, vLLM, AITER, and this RL-Kernel checkout installed (normally `pip install -e /work/RL-Kernel`). A source-only `PYTHONPATH` entry is insufficient because vLLM discovers RL-Kernel through the installed `vllm.general_plugins` entry point. The launcher is based on Vime's -Qwen3-8B AMD launcher and the existing `vime_qwen3_8b_tp2_cp2` example, but is +Qwen3-8B AMD launcher and the TP4/CP2 experiment, but is parameterized and avoids their CUDA-only flags. The default formal topology reuses PR #377's colocated eight-GPU schedule: @@ -75,58 +80,21 @@ run directory, it also requires the reference checkpoint's `latest_checkpointed_iteration.txt` marker and verifies that the installed RL-Kernel distribution exposes the expected vLLM plugin entry point. -## Inspect the plan - -No configuration JSON is checked into the repository. Supply paths on the CLI -or through the matching environment variables: - -```bash -python examples/vime_rocm_attention_ablation/run.py \ - --vime-root /work/vime \ - --rl-kernel-root /work/RL-Kernel \ - --megatron-root /work/Megatron-LM-vime \ - --model-root /app/model/Qwen3-8B \ - --reference-checkpoint /app/model/Qwen3-8B_torch_dist \ - --prompt-data /app/model/dapo-math-17k/dapo-math-17k.jsonl -``` +## Inspect and execute the plan -Without `--run`, this prints the exact four-arm plan and does not start Ray or -write results. +Use the shared runbook for the canonical command and current host paths. The +runner still supports a dry-run (omit `--run`) and a full four-arm execution. ## Execute all four arms -```bash -python examples/vime_rocm_attention_ablation/run.py \ - --vime-root /work/vime \ - --rl-kernel-root /work/RL-Kernel \ - --megatron-root /work/Megatron-LM-vime \ - --model-root /app/model/Qwen3-8B \ - --reference-checkpoint /app/model/Qwen3-8B_torch_dist \ - --prompt-data /app/model/dapo-math-17k/dapo-math-17k.jsonl \ - --run-dir /work/RL-Kernel/runs/vime-rocm-attention-$(date -u +%Y%m%dT%H%M%SZ) \ - --run -``` - -The runner content-hashes the prompt dataset, launcher, and small checkpoint +Use the shared runbook for the canonical command and current host paths. The +runner content-hashes the prompt dataset, launcher, and small checkpoint index/config manifests. For the large model/checkpoint trees it seals every relative file name, size, nanosecond mtime, and symlink target without rereading all 8B weight shards. It also records each source revision, tracked dirty state, and tracked-diff digest. Dirty checkouts are allowed, but the complete seal must remain identical before and after the four arms. -The default resource arguments are equivalent to: - -```bash -python examples/vime_rocm_attention_ablation/run.py \ - ... \ - --visible-gpus 0,1,2,3,4,5,6,7 \ - --num-gpus 8 \ - --tp-size 4 \ - --cp-size 2 \ - --rollout-tp-size 4 \ - --run -``` - CP remains an Attention matrix dimension here; sequence parallelism remains off even when CP is greater than one. @@ -188,15 +156,3 @@ python examples/vime_rocm_attention_ablation/validate_artifacts.py \ Do not add `matrix-plan.json`, validation JSON, rollout dumps, mismatch sidecars, checkpoints, or MI300X result files to the PR. Publish them as CI/job artifacts when needed. - -## Full-native PR377 workload - -The standalone P/P runner selects production attention, FFN, and logp on both -the Megatron and vLLM sides. It uses actor TP4/CP2, two TP4 rollout engines, -round-robin routing, eight samples, a 7168-token response limit, and Vime's -rollout-logprob framework consistency mode. Three rounds are the default: - -```bash -python -m examples.vime_rocm_attention_ablation.run_full_pp_pr377_workload \ - --run-dir /app/model/vime-runs/pr394-full-native-pp-vime-tis -``` diff --git a/tests/test_vime_qwen3_example.py b/tests/test_vime_qwen3_example.py deleted file mode 100644 index b3b10928..00000000 --- a/tests/test_vime_qwen3_example.py +++ /dev/null @@ -1,140 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 - -from __future__ import annotations - -from pathlib import Path - -import pytest - -from examples.vime_qwen3_8b_tp2_cp2.run import ( - build_report, - load_config, - validate_config, - validate_runtime_evidence, -) - -ROOT = Path(__file__).parents[1] -CONFIG = ROOT / "examples" / "vime_qwen3_8b_tp2_cp2" / "qwen3_8b_tp2_cp2.json" - - -def test_qwen3_example_config_is_strict_and_explicit(): - config = load_config(CONFIG) - validate_config(config) - assert config["training"]["tensor_model_parallel_size"] == 2 - assert config["training"]["context_parallel_size"] == 2 - assert config["linear_logp_provider"]["mode"] == "strict" - - -def test_qwen3_example_report_does_not_claim_unread_back_attention_or_ffn(tmp_path): - config = load_config(CONFIG) - report = build_report( - config, - vime_root=tmp_path / "vime", - rl_kernel_root=tmp_path / "rl-kernel", - command=["bash", "run.sh"], - status="passed", - returncode=0, - log_text="linear_logp provider active: backend_id=pytorch-vocab-parallel-logp-ws2", - log_path=tmp_path / "run.log", - ) - assert report["status"] == "passed" - assert report["claim_boundary"]["qwen3_8b_tp2_cp2_vime_training"] is True - assert report["claim_boundary"]["attention_train_infer_consistency"] == "unclaimed" - assert report["claim_boundary"]["ffn_train_infer_consistency"] == "unclaimed" - assert report["provider"]["fallback_observed"] is False - - -def test_qwen3_example_fails_closed_when_provider_marker_is_missing(tmp_path): - config = load_config(CONFIG) - report = build_report( - config, - vime_root=tmp_path / "vime", - rl_kernel_root=tmp_path / "rl-kernel", - command=["bash", "run.sh"], - status="passed", - returncode=0, - log_text="training completed without provider provenance", - log_path=None, - ) - assert report["status"] == "failed" - assert report["claim_boundary"]["qwen3_8b_tp2_cp2_vime_training"] is False - - -def _runtime_evidence(): - return { - "schema_version": "rlkernel.operator_runtime_evidence.v1", - "operators": { - "attention": { - "training": { - "implementation_id": "rlk.attn", - "backend_id": "rlk", - "contract_id": "a", - }, - "rollout": { - "implementation_id": "rlk.attn", - "backend_id": "rlk", - "contract_id": "a", - }, - "comparison": { - "passed": True, - "out_max_abs": 0.0, - "lse_max_abs": 0.0, - "dq_max_abs": 0.0, - "dk_max_abs": 0.0, - "dv_max_abs": 0.0, - }, - }, - "ffn": { - "training": { - "implementation_id": "rlk.ffn", - "backend_id": "rlk", - "contract_id": "f", - }, - "rollout": { - "implementation_id": "rlk.ffn", - "backend_id": "rlk", - "contract_id": "f", - }, - "comparison": { - "passed": True, - "out_max_abs": 0.0, - "dx_max_abs": 0.0, - "dw_max_abs": 0.0, - }, - }, - }, - } - - -def test_qwen3_example_accepts_only_exact_zero_runtime_evidence(tmp_path): - evidence = _runtime_evidence() - validate_runtime_evidence(evidence) - config = load_config(CONFIG) - report = build_report( - config, - vime_root=tmp_path / "vime", - rl_kernel_root=tmp_path / "rl-kernel", - command=["bash", "run.sh"], - status="passed", - returncode=0, - log_text="linear_logp provider active: backend_id=pytorch-vocab-parallel-logp-ws2", - log_path=None, - runtime_evidence=evidence, - ) - assert report["claim_boundary"]["attention_train_infer_consistency"] == "passed" - assert report["claim_boundary"]["ffn_train_infer_consistency"] == "passed" - - -def test_qwen3_example_rejects_nonzero_runtime_evidence(): - evidence = _runtime_evidence() - evidence["operators"]["attention"]["comparison"]["out_max_abs"] = 1e-6 - with pytest.raises(ValueError, match="attention"): - validate_runtime_evidence(evidence) - - -@pytest.mark.parametrize("bad_path", ["", "other.provider"]) -def test_qwen3_example_rejects_non_rlkernel_provider(bad_path): - config = load_config(CONFIG) - config["linear_logp_provider"]["path"] = bad_path - with pytest.raises(ValueError, match="RL-Kernel Vime provider"): - validate_config(config) diff --git a/tests/test_vime_validation_artifacts.py b/tests/test_vime_validation_artifacts.py index 64eba21b..26beb36a 100644 --- a/tests/test_vime_validation_artifacts.py +++ b/tests/test_vime_validation_artifacts.py @@ -6,7 +6,7 @@ import torch -from examples.vime_qwen3_8b_tp2_cp2.validate_artifacts import validate_artifacts +from examples.vime_qwen3_8b_tp4_cp2_200.validate_artifacts import validate_artifacts def _write_readback(directory, framework, target, *, triton_used=False):