Conversation
…ect#257) * fix(data): reuse stored multimodal_inputs in length filter filter_long_prompt re-extracted vision info from sample.prompt via process_vision_info in the multimodal branch. When apply_chat_template is set, sample.prompt is the rendered *string* (not a conversation list), so process_vision_info -> qwen_vl_utils crashed with "TypeError: string indices must be integers, not 'str'". This made prompt-length filtering unusable for any VLM dataset: setting --rollout-max-context-len (which derives rollout_max_prompt_len) or --rollout-max-prompt-len / --eval-max-prompt-len activates the filter and hits the crash. Reuse the multimodal inputs already computed during dataset construction (sample.multimodal_inputs) instead of recomputing them from the string prompt. Add CPU unit tests covering the multimodal branch and a mixed text-only + multimodal dataset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * Delete tests/test_filter_long_prompt_multimodal.py Signed-off-by: aoshen02 <aoshen@inferact.ai> * Update data.py Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: aoshen02 <aoshen@inferact.ai>
…llm-project#283) Replace all references to `inferactinc/public:vime-*` with `vllm/vime:*` across CI, docs, and the release justfile. The image is now published under the official vllm DockerHub namespace (`vllm/vime:latest`, `vllm/vime:test-latest`) as a multi-arch manifest (amd64 + arm64). Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…llm-project#280) tau-bench RunConfig defines agent_strategy, not agent. The old key was silently ignored by Pydantic, and resolve_tau_config accessed tau_config.agent which raises AttributeError at rollout time. Signed-off-by: kaiyuan <kyxiezju@163.com>
… 20B support (vllm-project#260) * restore: bring back deleted examples, scripts, and agent doc Restore files that were either deleted by vllm-project#126 ("trim examples to qwen3 only") or never synced from slime: **Reverted from pre-vllm-project#126 (translated):** - scripts/low_precision/run-qwen3-4b-fp8.sh - scripts/low_precision/run-qwen3-30b-a3b-fp8.sh - scripts/run-glm4-9B.sh - scripts/run-moonlight-16B-A3B.sh - scripts/run-qwen3-4B-base-sft.sh - scripts/run-qwen3-32B.sh - scripts/run-qwen3.5-35B-A3B-sft.sh **New from slime@44d29ee (translated):** - docs/en/get_started/agent.md - examples/fully_async/run-qwen2.5-0.5B-fully_async.sh All sglang engine flags translated to vllm equivalents (§2.4). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: unify pkill pattern to '[v]llm serve|VLL[M]::' Standardize all scripts to use the bracket-escaped pkill pattern that avoids matching pkill itself and also catches vLLM's renamed subprocesses (VLLM::EngineCore, VLLM::Worker_TP*). Matches the canonical pattern in command_utils.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * scripts: complete slime-exact translation of all 29 run scripts Translate all slime scripts to vime following SGLANG_TO_VLLM_TRANSLATION.md: - sglang→vllm prefix swap for CLI flags and variables - _slime→_vime for checkpoint paths - EP: --sglang-ep-size N → --vllm-enable-expert-parallel (boolean) - Speculative: multi-param → --vllm-speculative-config JSON (§5.2) - Delete genuinely sglang-coupled params (DP-attention, DeepEP, NSA, etc.) - flashinfer → FLASHINFER case fix (§2.4) 23 new scripts + 6 existing updated to match slime@cutoff. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(scripts): correct model config source path in FP8 low_precision scripts The FP8 scripts used `${SCRIPT_DIR}/../scripts/models/` which resolves to `scripts/scripts/models/` (non-existent). Changed to `../models/` to match the INT4 scripts. Same fix as slime PR #2094. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(gpt-oss): fused BF16 format, bridge API patch, bshd qkv format Three fixes needed to run GPT-OSS 20B RLHF on vLLM backend: 1. hf_weight_iterator_bridge: match Megatron-Bridge 0.5.0 API _patch_bridge_expert_cache_to_cpu monkey-patches GPTOSSBridge. maybe_modify_converted_hf_weight gained a 4th `hf_state_dict` parameter; the patched wrapper only accepted 3, causing TypeError during weight sync. 2. run-gpt-oss-20B: point --hf-checkpoint at fused BF16 format vLLM's _load_weights_other expects gate_up_proj [E, hidden, 2*ffn] (fused). The old per-expert split format (experts.{e}.gate_proj.weight) causes KeyError on bias loading. Use tools/convert_gpt_oss_to_fused.py to convert an existing per-expert checkpoint, or re-run preprocess_gpt_oss.py to produce fused format directly. 3. run-gpt-oss-20B: add --qkv-format bshd + fix seq-length GPT-OSS uses learnable softmax (sink attention). TransformerEngine disables all attention backends when softmax_type=learnable and qkv_format=thd (packed sequences). --qkv-format bshd avoids this. --use-dynamic-batch-size is incompatible with bshd; replaced with fixed --seq-length 10240 (covers 8192 max response + prompt headroom). tools/convert_gpt_oss_to_fused.py: new tool to convert per-expert BF16 checkpoint (output of old preprocess_gpt_oss.py) to the fused HF format expected by vLLM without re-running the slow MXFP4 dequantization. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(scripts): replace pkill -9 vllm with precise -f pattern (21 files) pkill -9 vllm matches any process named "vllm" and can inadvertently kill unrelated vllm processes (e.g. background services). Use the same pattern as PR vllm-project#220 which targets only vllm serve and Ray VLL[M]:: actors: pkill -9 -f '[v]llm serve|VLL[M]::' Also updates the inline form used in multi-node SSH worker restart commands (run-qwen3-235B-A22B*.sh, run-qwen3.5-27B.sh, etc.). Skipped: scripts/run-gpt-oss-20B.sh (uses pkill -9 -f "vllm serve" already), scripts/run-minimax-m2.sh and run-glm4.7-*.sh (already used -f "vllm serve"). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * chore(scripts): remove run-qwen3-4B-amd.sh from this PR AMD-specific script is out of scope for the gb300-complete-port PR. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * sync(docs+scripts): port docs/examples from slime-44d29ee, fix script translations - Add missing EN/ZH docs: low-precision, on-policy-distillation, get_started/agent, pd-disaggregation (heterogeneous server groups fix), examples zh docs - Add missing examples: on_policy_distillation, eval_multi_task, delta_weight_sync, geo3k images - Fix vLLM flag translations across all example docs: - --vllm-mem-fraction-static → --vllm-gpu-memory-utilization - Remove non-existent dp-attention flags (--vllm-enable-dp-attention, --vllm-dp-size, --vllm-moe-dense-tp-size, --vllm-enable-dp-lm-head, --vllm-ep-size) - --vllm-ep-num-redundant-experts → --vllm-eplb-config - --vllm-cuda-graph-bs → --vllm-max-cudagraph-capture-size - sglang speculative flags → --vllm-speculative-config JSON - GLM-4.7 MTP: method=eagle → method=mtp, num_speculative_tokens=4 → 3 - sgl-router → vllm-router; THUDM/vime → vllm-project/vime - Fix scripts: restore run-kimi-k2-Instruct/Thinking/qwen3-4B/qwen3-235B-A22B to slime-44d29ee-as-vime + pkill precision fix only; restore int4 python3 path Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * revert(scripts): pkill -9 -f pattern back to pkill -9 vllm, align with slime Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * revert(pkill): align all remaining kill patterns with slime (pkill -9 vllm) Covers examples/, docs/, tests/, and vime/utils -- previously missed in the scripts/ revert. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * chore: remove gpt-oss-20B script and convert tool (moved to separate PR) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-project#295) The rollout_buffer README (English and Chinese) linked to docs/{en,zh}/models/qwen3-4B.md, but there is no models/ directory — the referenced doc lives under examples/. Point both links to docs/{en,zh}/examples/qwen3-4B.md so the setup instructions resolve. Signed-off-by: ajinkya-metica <ajinkya@metica.com> Co-authored-by: ajinkya-metica <ajinkya@metica.com>
…eview] (vllm-project#286) * sync(slime #2014..#2125): diff3 3-way merge, conflicts preserved Mechanical commit 1 of 2 (per knowledge/rl/slime-to-vime-sync-sop.md §2). diff3 translated 3-way merge on upstream/main (incl vllm-project#260/vllm-project#280/vllm-project#283/vllm-project#257): ours = vime@main, base = translate(slime@#2013), theirs = translate(slime@#2125) Translation fixes vs prior attempt: - casing: SGLang->vLLM (prose) / SGLang<X>->VLLM<X> (identifiers); killed VLlm artifact (was 35 files) - dotted module refs slime.X->vime.X now translated (was leaking 'from slime.backends') These resolved 9 spurious conflicts (46->37 files). Results: 61 clean / 37 conflict (diff3 markers preserved) / 50 new-to-vime / 5 del. Conflict markers use readable -L labels (ours/base/theirs). Resolve in commit 2. Non-conflict provenance fix: vimerl/vime -> vllm/vime in 2 example docs. Engine patch handling (docker/patch/) deferred to commit 2 per SOP §4.5. Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync(slime #2014..#2125): resolve all conflicts (commit 2) Resolved all 37 conflict files / 84 diff3 blocks per agent_run RESOLUTION_POLICY. Principle: keep vime vLLM impl (ours) + incorporate slime's new features (theirs). Highlights: - vLLM API form kept everywhere: /inference/v1/generate, choices parsing, AsyncEngineArgs, vLLM flag names (--vllm-gpu-memory-utilization etc). - Dropped all vllm.srt.* imports (non-existent in real vLLM). - Accepted new slime features: delta-weight-sync CLI args, append_response_tokens (Sample), get_server_info/start_external_rollout_servers/get_rollout_num_engines, old-router(<=0.2.1) compat, TrajectoryManager adapter design (vime already adopted it). - Kept vime-only: --rollout-external, add_router_arguments, _get_metrics_router_addr, reinit_wandb_primary_with_open_metrics, update_tracking_open_metrics, modal sandbox, VIME_AGENT_* env names, local-vLLM tau-bench user sim. - Engine patches (docker/patch/): kept ours vllm.patch (22-line MoE fix), dropped theirs sglang 2674-line content; deleted sglang-only vllm-top_p.patch (per SOP 4.5). - Dockerfile kept ours (vllm/vllm-openai base); version.txt accepted theirs nightly. - run-deepseek-r1.sh: dropped /sgl-workspace dead-path env. Deviations from policy (documented): vllm_rollout.py abort path kept ours pause/drain (abort_servers_until_idle would break partial-rollout drain + leave paused_workers unbound). README ecosystem section left empty (ours) pending de-translation of provenance. All changed .py py_compile clean; zero conflict markers; no sglang/slime leakage. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): pre-commit lint/format on resolved files - re-add dropped `from vllm_router.launch_router import RouterArgs` import in vime/utils/arguments.py (add_router_arguments uses it; F821 from conflict resolution) - drop unused base_top_p_token_ids/offsets in vllm_streaming_rollout.py (F841; came from theirs but ours's choices-parsing path doesn't use them) - black/isort autoformat (anthropic.py, test_agent/*, arguments.py) - pipeline.yml: agent tests moved to tests/test_agent/; wire new CPU tests pre-commit: all hooks pass (ruff/autoflake/isort/black/yaml). Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): make CPU CI green (engine import, docker build, agent/utils tests) Local CPU CI (pre-commit + plugin + agent + utils) all green on 8xH200 host in python:3.11 containers. Fixes found while running it: - vllm_engine.py: make 'import vllm_router'/'packaging.parse' lazy (inside _register_to_router); top-level import broke CPU import (vllm_router absent in CPU CI). The old-router(<=0.2.1) compat branch is theirs-accepted. - docker/Dockerfile: TMS_CUDA_MAJOR=12 for torch_memory_saver pin (its build backend now requires it for CUDA wheels; base is cu129). Unblocks image build. - agent/adapters/common.py: _run_turn called parse_model_output() without the required tokenizer= kwarg -> 500s in adapter tests. Pass tokenizer=tok. - tests/test_agent/_fakes.py: FakeVLLMServer served sglang /generate + meta_info; retarget to vime /inference/v1/generate + choices shape + x-session-id header. - tests/test_agent/test_adapters.py: parse_model_output(tokenizer=...) + assert vime body keys (token_ids/max_tokens). - tests/utils/test_vllm_config.py: vLLMConfig->VllmConfig (4 sites); fake router returns 3-tuple (ip,port,prom) matching _start_router; drop spurious resolve(). - tests/test_megatron_argument_validation.py: add num_gpus_per_node=8 to the vime_validate_args fixture (vime colocate override needs it). - .buildkite/pipeline.yml: agent tests -> tests/test_agent/*; +cispo_loss, +logprob_response_spans (CPU-safe); test_rollout_metrics stays GPU-only (imports vllm). Engine patch verdict (PATCH_ASSESSMENT.md): P1-P7 vLLM doesn't need (NIXL/Mooncake native); kept ours vllm.patch, dropped sglang content + top_p.patch. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): unblock GPU run (scipy pin, router arg dup, top-p-replay gate) Found running GPU CI on 8xH200 with vllm/vime:latest: - docker/Dockerfile: pin scipy<1.14 next to numpy<2 (scipy drifted to 1.18 which needs numpy>=2 and uses removed np.long -> 'import vllm' crash). - vllm_utils/arguments.py: drop sync-added RouterArgs.add_cli_args + its import in add_vllm_router_arguments. main exposes the full router surface only in utils.add_router_arguments; the duplicate re-registered --router-request-timeout-secs -> argparse conflict at train startup. - megatron_utils/loss.py: get_rollout_top_p_logprob_kwargs falls back to full-vocab logprob when top-p nucleus token ids are absent instead of raising. slime's top-p-replay needs engine-returned top-p tokens; vime's vLLM /inference/v1/generate does not expose them (sglang-only). Matches vime pre-sync behavior; flagged in OVERNIGHT_REPORT for review. Image import smoke + Megatron ckpt load + 4x VLLMEngine bringup + NCCL weight transfer all confirmed working in-image before this. Signed-off-by: aoshen02 <aoshen@inferact.ai> * ci(gpu): drop deleted test_qwen2.5_0.5B_ppo_critic_only_short from short suite slime deleted tests/test_qwen2.5_0.5B_ppo_critic_only_short.py this window (#2014..#2125); gpu_suites.py still listed it -> 'no such file' exit 2. Critic-only path is still covered by test_qwen3_4B_ppo_train_critic_only (megatron suite). Other 3 short tests (gsm8k_async, gsm8k, fully_async) pass on 8xH200. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): restore base_log_probs init in streaming rollout (None+list crash) GPU test_qwen3_4B_streaming_partial_rollout hit 'TypeError: unsupported operand +: NoneType and list' at vllm_streaming_rollout.py:234. The conflict resolution changed base_log_probs from main's `list(sample.rollout_log_probs or [])` to a None-able form; a fresh sample (rollout_log_probs=None) then did None + call_log_probs. Restored main's form. Real sync-resolution regression caught by GPU CI. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): streaming rollout uses append_response_tokens (was renamed update_from_meta_info) slime #2110 renamed Sample.update_from_meta_info -> append_response_tokens; vllm_rollout was updated but vllm_streaming_rollout still called the old name (AttributeError at generate_streaming). Streaming already accumulates tokens incrementally for partial-rollout, so call append_response_tokens(meta_info=meta) with tokens omitted -> metadata-only finalize (no double-append). Caught by GPU test_qwen3_4B_streaming_partial_rollout. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): restore vime unconditional colocate rollout_num_gpus re-derive 3-way compare (slime / vime-main / PR) showed the merge made a broken hybrid: it KEPT vime's num_gpus_per_node colocate override (which assumes rollout_num_gpus is forced to actor_num_gpus_per_node*actor_num_nodes) but REPLACED vime's unconditional re-derive (`!= -> re-derive`) with slime's `is None`-only form. When a colocate test's rollout_num_gpus is non-None but mismatches, it was left mis-sized -> engine/GPU misplacement -> mixed_offload IPC-UUID mismatch + ckpt 'Free memory < util'. slime passes (no override, self-consistent is-None); vime main passes (override + unconditional re-derive, coupled). Restore vime's re-derive; keep slime's new rollout_num_gpus==0 branch (checked first). Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): slime-consistency review pass (docs, rollout routed_experts, comments) Docs (EN+zh parity): - fault-tolerance: revert junk 'server'->'engine' mistranslation (keep correct /health endpoint, verified vs vllm_engine.py) - vllm-config: 'ServerArgs'->'EngineArgs' (sglang class -> vLLM AsyncEngineArgs u FrontendArgs); fix duplicated 'vllm-router (vllm-router)' alias - customization: restore over-deleted 'custom_generate -> list[Sample]' section + signature (dropped only the vime-absent search-r1 example link) Rollout: - routed_experts now flows through the slime-identical Sample._apply_meta_info (single assignment site, torch.int32 tensor matching downstream) instead of an inline numpy assign; vLLM .npy-on-choice decode stays (engine wire-format delta). Both vllm_rollout and vllm_streaming_rollout. Comments for future syncers: - --opd-teacher-model + on_policy_distillation: engine-driven divergence (vLLM model field; sglang /generate has none) - overrides / _vllm_server_field_names: AsyncEngineArgs u FrontendArgs == slime's sglang ServerArgs Examples/docker/etc: drop vime-absent npu/retool/search-r1/tau-bench files; restore eval_multi_task; docker alignment with slime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(ci): pre-commit green — define base in streaming MM render (F821) + isort/black on test_agent Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(ci): correct two mistranslated CPU tests (colocate rollout-gpu re-derive; parse_model_output tokenizer) - test_..._preserves_larger_rollout_gpus_under_colocate asserted slime behavior (==12); vime re-derives to actor*nodes=8 under colocate (commit 9701304). Renamed + assert ==8 + divergence note. vime-main never had the test; slime does. - test_parse_model_output_plain_text_no_parsers called parse_model_output without the required tokenizer kwarg (vllm-project#198 made it required for vLLM parsers). Pass tokenizer=None (unused on the no-parser path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): streaming rollout posts to /inference/v1/generate, not sglang /generate The slime diff3 merge took slime's sglang endpoint (/generate) for the streaming rollout POST instead of keeping vime's vLLM endpoint (/inference/v1/generate). main (7198547) had the correct URL; the sync regressed it (and dropped the base var). Result: 404 Not Found at vllm_streaming_rollout.py:182 -> test_qwen3_4B_streaming_partial_rollout fails. Caught on a clean h200 node. The file's own docstrings already say /inference/v1/generate throughout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): thread rollout port cursor globally across multi-model engines The diff3 merge adopted slime's deferred rollout-engine init (start_rollout_servers now returns pending_init_handles awaited by the caller instead of ray.get-ing each model's engines before the next). That broke an implicit invariant the per-model port_cursors reset relied on: in main, model 0's engines were fully bound before model 1 allocated ports, so the free-port bind-test in _allocate_rollout_engine_addr_and_ports_normal skipped model 0's ports. With deferred init, model 1 allocates while model 0 is unbound, the bind-test sees the base ports free, and a second model (e.g. mixed_offload's frozen "ref") lands on the same 15000-15003 as the actor. The actor's POST /update_weights to :15002 then hits the never-started ref engine -> vLLM 500 "start_weight_update must be called before update_weights" (test_vllm_config_mixed_offload[_ft]). Fix: initialize port_cursors once before the model loop so the per-node next-free cursor is monotonic across all models, keeping every engine's ports disjoint regardless of bind timing. Single-model behaviour is unchanged; the per-model reset only existed to scope cursors that are already node-keyed. Caught on h200 GPU CI (new nightly-dev-20260618a image, which added the start_weight_update-before-update_weights enforcement that exposed the collision). Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): restore robust pkill pattern so ckpt cleanup kills vLLM ray actors PR vllm-project#260 ("complete slime-exact port") changed execute_train's pre-launch cleanup from pkill -9 -f '[v]llm serve|VLL[M]::' to pkill -9 vllm as an over-literal sglang->vllm translation. But the vLLM rollout engine runs as Ray actor processes whose process *name* is python/ray, with "VLLM::" only in the command line — so `pkill -9 vllm` (name match, no -f) does not kill them. Leftover engine processes from the ckpt test's save phase survive into the load phase, holding ~115 GiB, so the load-phase engine starts with ~24/139 GiB free and dies with "Free memory ... less than desired GPU memory utilization (0.8, 111.84 GiB)" (test_qwen3_4B_ckpt.py, both --async-save and not). Restore the cmdline-match pattern `-f '[v]llm serve|VLL[M]::'`. Bisected on h200: ckpt PASSES at 289ee6d / e62d44f (old pattern, 4/4 runs) and FAILS at 7198547/main + PR (new pattern, 0/3), same old image -> code regression in vllm-project#260. Verified: pkill-fixed PR code + new pr286 image -> ckpt PASS (579s). Signed-off-by: aoshen02 <aoshen@inferact.ai> * chore(sync): replace all `pkill -9 vllm` with cmdline-match pattern Same root cause as 764e1e1 (command_utils.py): `pkill -9 vllm` matches by process *name*, but vLLM rollout engines run as Ray actor processes (python/ray named, "VLLM::" only in the command line), so the name match never kills them. Apply the robust cmdline pattern `pkill -9 -f '[v]llm serve|VLL[M]::'` everywhere the bare `pkill -9 vllm` cleanup was used across run/example scripts, so leftover engines don't squat GPUs across runs. No logic change beyond the kill pattern. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(docker): restore vLLM core.py partial-wake sleep-guard (#44483) Commit d41f0aa removed the core.py sleep-guard hunk from docker/patch/latest/vllm.patch on the assumption it was "already in v0.23.0". It is not: stock v0.23.0 `vllm/v1/engine/core.py` calls `resume_scheduler()` and `execute_dummy_batch()` even during a partial (weights-only) wake. So a colocate DP+EP pd_mooncake rollout, right after `POST /wake_up?tags=weights` (KV cache still released under level-2 sleep), has its DP busy-loop fire a decode-shaped dummy batch that touches freed KV -> the scheduler_metadata write in flashattn_mla.py:234 (MLA, glm4.7) and flash_attn.py:547 (FA3, qwen3.6) raises `CUDA error: invalid argument`. Restore the guard (`if not self.model_executor.is_sleeping` around resume_scheduler; `if not self.is_sleeping()` around execute_dummy_batch), keeping the all2all_utils weight-reload fix. This is the vllm-project#173 sleep-guard patch re-expressed against v0.23.0 line numbers. Verified: git-apply --check clean against stock v0.23.0; both guards land; glm4.7/qwen3.6 pd_mooncake reproduced the crash without it (the 8-day-old image that still carried the guard passes both). Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(docker): drop scipy<1.14 pin, mirror slime numpy<2 only The scipy<1.14 pin (added in 0cda11c while chasing the pd_mooncake crash) was a red herring: the real cause was the dropped core.py partial-wake sleep-guard, now restored. slime-2125-as-vime pins only `numpy<2`; this restores that exact line. numpy 1.26.4 + scipy resolved naturally matches the working baseline. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(docker): keep FlashQLA install gated behind INSTALL_FLASHQLA=0 slime installs FlashQLA unconditionally, but it is sm90/Hopper-only. Restore vime's original gated form (default off; --qwen-gdn-backend fla elsewhere). CI build passes --build-arg INSTALL_FLASHQLA=1 to include it. Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs(vllm-config): fix inference-only FAQ — vime launches engines in-process The mechanical mirror of slime's answer steered users to vLLM's standalone `vllm serve` (slime's `launch_server` analog) for inference-only. That is misleading for vime: like slime, vime launches the vLLM engines in-process from `--vllm-config` (same in-process path as training), so a rollout-only run serves directly with no separate server process. Point standalone users to `--rollout-external-engine-addrs` instead. EN + ZH. Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs(debug): restore INT4 / Compressed-Tensors checkpoint section vime's debug.md was missing slime's "INT4 / Compressed-Tensors Quantization Checkpoint Issues" section (slime #1642) — dropped in an earlier sync, not present on main. Restore it (EN + ZH), translated sglang→vLLM / Megatron→vLLM. Covers the quantization_config.ignore list, all-zero MoE router weights (mlp.gate.weight) when mis-quantized, missing safetensors shards, and diagnosis via --check-weight-update-equal / --debug-rollout-only. Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs(vllm-config): use _run_vllm_server for inference-only FAQ Keep slime's wording; the only engine-coupled fix is the standalone launcher name. slime's `launch_server` is its in-process engine entry; vime's analog is `_run_vllm_server` (vllm_engine.py, launched via multiprocessing.Process), not the standalone `vllm serve` CLI. EN + ZH. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(docker): restore scipy pin (scipy<1.18) — vime base needs it Reverts the scipy-pin removal in b359b24, which was wrong: vime's vllm/vllm-openai base ships no scipy, so unpinned the build pulls scipy>=1.18, which hard-requires numpy>=2 and uses np.long (removed numpy>=1.24) -> crashes against the numpy<2 reinstall (Megatron needs numpy 1.x). slime's sglang base resolves scipy 1.17.1 natively (numpy-1.x compatible), so slime needs no pin; this is a base-image divergence, not a red herring. Pin boundary is 1.18 (slime runs 1.17.1), not the earlier 1.14 over-estimate. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(scripts): properly translate sglang args in glm5.2-744B + glm4.7-355B-delta These two were prefix-swapped (--sglang-X -> --vllm-X) without semantic mapping, leaving ~20 args that aren't vLLM AsyncEngineArgs (argparse would reject). Apply the knowledge/rl/sglang-to-vllm-translation.md §5.5 mappings: - dp-size->data-parallel-size, ep-size->enable-expert-parallel, max-running-requests-> max-num-seqs, cuda-graph-max-bs->max-cudagraph-capture-size - 5x/4x --speculative-* -> one --vllm-speculative-config JSON (§5.2) - DeepEP: per-group deepep_mode auto/low_latency -> all2all_backend deepep_high_throughput/ low_latency in the --vllm-config overrides (vLLM has no 'auto'; PD encodes it per-role) - watchdog-timeout -> env VLLM_ENGINE_ITERATION_TIMEOUT_S - drop sglang-only: dp-attention / dp-lm-head / moe-dense-tp / disable-overlap-schedule / NSA backends (vLLM selects DeepSeek sparse attn per model) / engine delta-receiver knobs - flag PD mooncake transport (-> --vllm-kv-transfer-config) as fabric-specific TODO These are 744B/355B scripts not runnable in CI — translations are SOP-mapped but hardware-unvalidated (flagged inline). Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(args): hard-guard unverified delta weight-sync mode --update-weight-mode=delta (PR vllm-project#278 lineage) is not yet validated on vime+vLLM (vLLM exposes dense/sparse_flat only, not slime's gap-delta/ zstd encoding). Raise NotImplementedError at arg-validation so it fails fast at startup instead of crashing mid weight-sync. Downstream delta code is kept untouched; remove this raise once a real delta-load run passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs,test: correct rollout engine endpoint to /inference/v1/generate The agent rollout engine is reached at vime's vLLM ``/inference/v1/generate`` (see vllm_rollout.get_model_url default + common.call_vllm_generate), not the bare ``/generate`` of sglang. Fix the imprecise path in adapter/test docstrings and comments, and rewrite the vllm-config.md custom-rollout examples (en+zh): they were still sglang-shaped (``/generate`` path + ``{"text":..., "return_logprob": True}`` body). Use vime's real request schema instead -- ``{"model","token_ids", "sampling_params"}`` with ``max_tokens``/``logprobs``, ``prompt_logprobs`` for fixed-sequence scoring, and the ``choices[0]`` response shape. No code/logic change: comments, docstrings, and doc examples only. The Megatron training server's own ``/generate`` endpoint and the sglang citation in arguments.py are correct and left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * style(args): collapse delta-guard message to one line (black) The delta-guard NotImplementedError message was split across two adjacent string literals; black on the CI (line-length 119) collapses/normalizes it. Make it a single clean literal so pre-commit is green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * test(args): assert delta weight-sync is guarded off (not per-condition) The hard delta guard (7bb19e6) raises NotImplementedError at the top of the delta branch, making the downstream colocate / unknown-transport rejections unreachable. Replace test_update_weight_delta_rejects_colocate and test_update_weight_delta_rejects_unknown_transport (whose ValueError paths no longer fire) with a single test_update_weight_delta_disabled that asserts the guard raises for any delta config. Breadcrumb left to restore the per-condition tests when delta is verified and the guard is lifted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(sync): restore dropped weight-sync metrics chain + delta dispatch (slime parity) A mirror cross-check of the delta-weight-sync surface (slime #1806/#1991) found the #2014..#2125 sync had silently dropped several slime-faithful pieces: * extra_metrics logging chain — slime threads weight-update metrics from the actor through log_perf_data -> log_perf_data_raw. vime dropped the param at all three layers, so weight-update metrics were never logged. Restored: train_metric_utils.log_perf_data_raw(extra_metrics=...), data.log_perf_data passthrough, and actor passing self.weight_updater.pop_metrics(). * pop_metrics on UpdateWeightFromDistributed — the default (non-colocate, nccl) weight_updater. slime gives all three updaters a pop_metrics() stub so the actor can call it uniformly; vime kept it on tensor/disk but dropped it on distributed, which would AttributeError once the actor calls it. Restored the ~5-line stub (delta-specific plumbing stays dropped — vime+vLLM has no DeltaSpec). * actor delta-mode dispatch branch — restores the elif selecting UpdateWeightFromDistributedDelta. Dead code behind the validation guard that rejects --update-weight-mode=delta, so vime mirrors slime with the guard as the single divergence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * chore(sync): align comments to the mechanical mirror Comment-only pass; no behavior change. Aligns vime comments to what a faithful slime->vime translation would carry: * Strip vime-divergence rationale markers (delta guard, opd-teacher-model, top-p fallback, colocate/delta tests). The rationale belongs in the divergence manifest, not inline; the guarded code + self-explanatory NotImplementedError messages stand on their own. * De-verbose vLLM-specific comments to mirror scale: the router-args block, the AsyncEngineArgs u FrontendArgs docstring, and the MoE-replay / streaming-rollout blocks that slime does not carry at that length. * Restore slime-original comments the sync had dropped or naively translated, with judgment translation of sglang-specific terms: session_id routing ("vLLM router", not the mechanical "Model Gateway"), "Prepare payload for vLLM server", the unique-session_id loop, and the pending-tasks wait. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * docs(delta): note delta weight-sync not yet verified on vime+vLLM (PR vllm-project#286 review) Per review on PR vllm-project#286: delta weight sync is documented here but the arg guard disables --update-weight-mode=delta. Add a top-of-page note (en + zh) so users see it before hitting NotImplementedError at argparse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ect#291) * Add examples/mem_agent for long-context MemAgent RL Co-authored-by: Cursor Signed-off-by: kaiyuan <kyxiezju@163.com> * Address review: harden mem_agent convert and eval scripts Co-authored-by: Cursor Signed-off-by: kaiyuan <kyxiezju@163.com> --------- Signed-off-by: kaiyuan <kyxiezju@163.com>
…m-project#296) Under --partial-rollout, abort() (pause -> drain -> resume) deadlocks: /pause?mode=abort puts the scheduler in PAUSED_NEW, and a /generate that races in after the pause parks in the waiting queue and never returns until /resume, which runs after the drain. Reordering to pause -> resume -> drain avoids the hang, but resume reopens the whole queue so the long tail runs to COMPLETION -- breaking partial rollout's "truncate the tail, resume it next step" semantics. Switch to a delete-type abort instead: - vLLM: add POST /abort_requests to the RLHF api_router -> EngineClient.abort() (removes queued requests from the waiting queue and finish-aborts running ones, whose partial output returns on the original /generate stream). It does not pause the scheduler, so there is no /resume and no deadlock. Shipped as a build-time patch in docker/patch/latest/vllm.patch. - vime: server_control.abort_inflight_requests() replaces the unused, slime-mirrored abort_servers_until_idle / _v1_loads helper (vLLM has neither /abort_request nor /v1/loads). abort() re-issues the sweep across drain waves and converges on state.pendings, with a timeout bounding how long a late multi-turn straggler can run before being truncated to partial. - vllm_engine: drop the legacy version gate in _register_to_router. vime ships its own vllm-router, so only the /workers payload path is needed. Adds delete-type abort unit tests. AI-assisted change; reviewed by a human before submission. Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Josephasafg <ajgard7@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…-project#303) rollout/prefix_cache_hit_rate (and avg_cached_tokens_per_sample) were structurally 0 due to three coupled gaps: 1. vLLM's non-streaming /inference/v1/generate builds a `usage` block but silently drops it: GenerateResponse (v0.23.0 serve/disagg/protocol.py) never declared a `usage` field and has no extra="allow", so pydantic discards it. Patched in docker/patch/latest/vllm.patch, mirroring GenerateStreamResponse. (Upstream fix filed against vllm-project/vllm.) 2. The rollout parser read usage.prompt_tokens/completion_tokens but never usage.prompt_tokens_details.cached_tokens -> PrefixCacheInfo numerator pinned to 0. Now read in both vllm_rollout and vllm_streaming_rollout. 3. Streaming additionally needs stream_options.include_usage=True for vLLM to emit the terminal usage SSE chunk. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-project#317) * fix(docker): abort-all in the /abort_requests vLLM patch must abort by internal ids The bundled /abort_requests endpoint (merged in vllm-project#296) populated request_ids from output_processor.request_states (internal ids) but called engine.abort() with the default internal=False, so they were treated as external, matched nothing, and POST /abort_requests {} silently aborted no requests under default request-id randomization. Abort the all-in-flight list as internal. Mirrors vllm-project/vllm#47173. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * style: reject malformed JSON in /abort_requests patch with 400 Match the sibling dev endpoints and the Rust frontend (400 on malformed JSON) instead of silently treating it as empty. Mirrors vllm-project/vllm#47173. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(docker): abort-all patch must also abort parallel-sampling parents The /abort_requests patch enumerated request_states (child internal ids only), so with n>1 the ParentRequest entry leaked. Include parent_requests keys in the abort-all set. Mirrors vllm PR #47173. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> * feat(docker): cu13 image variant + TMS cu13 preload Port the cu13 build support from vllm-project#307: ENABLE_CUDA_13 branches the apt dev headers, cublas header, TransformerEngine (source-built for cu13), TMS_CUDA_MAJOR auto-detect, and the cudnn pin. justfile gains a build-cu13 target and a VARIANT-prefixed manifest. actor_group preloads the cu13 TMS .so. Also switch the vLLM patch apply to --allow-empty so the build survives once the patch is emptied upstream. Excludes vllm-project#307's NCCL_CUMEM_ENABLE default flip (0->1) and the glm5.2 scripts by request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AGENTS.mdforvllm-agent-infraCLAUDE.mdWhy
The repo can now load the shared
vllm-agent-infraagent contract frommainusing the project-local checkout.Validation
bash vllm-agent-infra/setup.sh /home/aoshen/vimevllm-agent-infra/knowledge/frontmatter/index metadata.Note
Direct pushes to
mainare disabled, so this branch is opened as a draft PR instead.