diff --git a/amd-docs/conf.py b/amd-docs/conf.py index bf5ad549fae..13146c4eb07 100644 --- a/amd-docs/conf.py +++ b/amd-docs/conf.py @@ -12,7 +12,7 @@ flavor options: rocm, rocm-docs-home, rocm-blogs, rocm-ds, instinct, ai-developer-hub, local, generic ''' html_theme = "rocm_docs_theme" -html_theme_options = {"flavor": "rocm-llmext", "repository_url": "https://github.com/ROCm/verl/"} +html_theme_options = {"flavor": "rocm-llmext", "repository_url": "https://github.com/AMD-Ecosystem/verl/"} ''' docs_header_version is used to manually configure the version in the header. If @@ -20,7 +20,7 @@ the documentation page will contain the given version string. ''' html_context = { - "docs_header_version": "26.02" + "docs_header_version": "26.06" } diff --git a/amd-docs/data/dapo_7b_math_fsdp2_4_4_complete.png b/amd-docs/data/dapo_7b_math_fsdp2_4_4_complete.png new file mode 100644 index 00000000000..68ea09d7a24 Binary files /dev/null and b/amd-docs/data/dapo_7b_math_fsdp2_4_4_complete.png differ diff --git a/amd-docs/data/geo3k_qwen25vl_7b_megatron_4_4_complete.png b/amd-docs/data/geo3k_qwen25vl_7b_megatron_4_4_complete.png new file mode 100644 index 00000000000..197a69050cf Binary files /dev/null and b/amd-docs/data/geo3k_qwen25vl_7b_megatron_4_4_complete.png differ diff --git a/amd-docs/examples/verl-async-examples.rst b/amd-docs/examples/verl-async-examples.rst new file mode 100644 index 00000000000..6592dd38b1a --- /dev/null +++ b/amd-docs/examples/verl-async-examples.rst @@ -0,0 +1,102 @@ +.. meta:: + :description: verl fully asynchronous examples + :keywords: verl, programming, ROCm, example, DAPO, GRPO, Megatron, FSDP2 + +.. _run-a-fully-async-verl-example: + +******************************************************************** +Run fully asynchronous verl examples +******************************************************************** + +This guide shows how to run fully asynchronous verl examples on AMD GPUs with ROCm. +It covers preparing data and models, launching fully asynchronous GRPO training on a +vision-language model with Megatron, and running fully asynchronous DAPO math +reasoning training with FSDP2. + +Megatron example +-------------------------------------------------------------------- + +The `geo3k_qwen25vl_7b_megatron_4_4.sh `_ example launches fully asynchronous GRPO training for ``Qwen2.5-VL-7B-Instruct`` on the Geometry3k vision-math dataset using verl's fully asynchronous policy with the Megatron trainer configuration. + +1. Download `prepare_geo3k_qwen25vl_7b_megatron_4_4.sh `_ and run: + + .. code-block:: bash + + export HF_TOKEN=your_token + # or: hf auth login + cd /workspace/verl + bash verl/experimental/fully_async_policy/shell/data_model_preparation/prepare_geo3k_qwen25vl_7b_megatron_4_4.sh + +2. Run the example: + + .. code-block:: bash + + export HF_MODEL_PATH=${HOME}/models/Qwen2.5-VL-7B-Instruct + cd /workspace/verl + bash verl/experimental/fully_async_policy/shell/geo3k_qwen25vl_7b_megatron_4_4.sh + + This example will take several hours to run. Once the example completes, the output should resemble the following: + + .. image:: ../data/geo3k_qwen25vl_7b_megatron_4_4_complete.png + :alt: Expected terminal output after geo3k_qwen25vl_7b_megatron_4_4.sh completes + +DAPO example +-------------------------------------------------------------------- + +The `dapo_7b_math_fsdp2_4_4.sh `__ +example launches fully asynchronous DAPO reinforcement learning training using ``Qwen2.5-Math-7B`` on math reasoning tasks. + +1. Download `prepare_dapo_7b_math_fsdp2_4_4.sh `_ and run: + + .. code-block:: bash + + export HF_TOKEN=your_token + # or: hf auth login + cd /workspace/verl + bash verl/experimental/fully_async_policy/shell/data_model_preparation/prepare_dapo_7b_math_fsdp2_4_4.sh + +2. Run the example: + + .. code-block:: bash + + cd /workspace/verl + bash verl/experimental/fully_async_policy/shell/dapo_7b_math_fsdp2_4_4.sh + + This example will take several hours to run. Once the example completes, the output should resemble the following: + + .. image:: ../data/dapo_7b_math_fsdp2_4_4_complete.png + :alt: Expected terminal output after dapo_7b_math_fsdp2_4_4.sh completes + +3. If you encounter out of memory issues, reduce ``max_position_embeddings`` in the ``config.json`` to 4096: + + .. code-block:: bash + + python -c " + import json, pathlib + config_path = pathlib.Path.home() / 'verl/models/Qwen2.5-Math-7B/config.json' + with open(config_path) as f: + config = json.load(f) + config['max_position_embeddings'] = 4096 + with open(config_path, 'w') as f: + json.dump(config, f, indent=2) + " + +4. Apply these changes as overrides in ``dapo_7b_math_fsdp2_4_4.sh``: + + .. code-block:: bash + + # line 55: default uses * 2 + actor_ppo_max_token_len=$((max_prompt_length + max_response_length)) + + # line 56: default uses * 3 + infer_ppo_max_token_len=$(((max_prompt_length + max_response_length) * 2)) + + # line 61: default is fsdp_size=2 + fsdp_size=4 + + # line 74: default is total_rollout_steps=$(((512*100))) + total_rollout_steps=$((100)) + + # line 105: default is max_position_embeddings=32768 + +actor_rollout_ref.model.override_config.max_position_embeddings=4096 \ + diff --git a/amd-docs/examples/verl-examples.rst b/amd-docs/examples/verl-examples.rst index fe683eb16af..952f87f9b63 100644 --- a/amd-docs/examples/verl-examples.rst +++ b/amd-docs/examples/verl-examples.rst @@ -13,7 +13,7 @@ It covers data preparation, model loading checks, configuration, environment variables, running PPO and GRPO, and launching multi-node training with Slurm. -1. Prepare the GSM8K dataset using the provided preprocessing script from the examples directory at `https://github.com/ROCm/verl/tree/main/examples/data_preprocess `__. +1. Prepare the GSM8K dataset using the provided preprocessing script from the examples directory at `https://github.com/AMD-Ecosystem/verl/tree/main/examples/data_preprocess `__. .. code-block:: bash diff --git a/amd-docs/index.rst b/amd-docs/index.rst index 8a213c498ed..35fcaef0118 100644 --- a/amd-docs/index.rst +++ b/amd-docs/index.rst @@ -22,10 +22,10 @@ verl achieves high training and generation throughput by building on existing LL Its 3D-HybridEngine reduces memory use and communication overhead when switching between training and inference, improving overall performance. -verl is part of the `ROCm-LLMExt toolkit +verl is part of the `AMD LLM Extension toolkit `__. -The verl public repository is located at `https://github.com/ROCm/verl `__. +The verl public repository is located at `https://github.com/AMD-Ecosystem/verl `__. .. grid:: 2 :gutter: 3 @@ -37,6 +37,7 @@ The verl public repository is located at `https://github.com/ROCm/verl ` + * :doc:`Run fully asynchronous verl examples ` .. grid-item-card:: Reference @@ -44,6 +45,6 @@ The verl public repository is located at `https://github.com/ROCm/verl `__ To contribute to the documentation, refer to -`Contributing to verl `__. +`Contributing to verl `__. You can find licensing information on the :doc:`Licensing ` page. diff --git a/amd-docs/install/verl-install.rst b/amd-docs/install/verl-install.rst index 0a4332ed2a4..3e93da6a04f 100644 --- a/amd-docs/install/verl-install.rst +++ b/amd-docs/install/verl-install.rst @@ -15,7 +15,7 @@ To use verl `0.7.1 `__, - **ROCm version:** `7.0.2 `__ - **Operating system:** Ubuntu 22.04 -- **GPU platform:** AMD Instinct™ MI300X, MI308, MI325X, MI350X, or MI355X +- **GPU platform:** AMD Instinct™ MI300X, MI325X, MI355X - **PyTorch:** `2.9.1 `__ - **Python:** `3.12 `__ - **vLLM:** `0.20.2 `__ @@ -42,7 +42,7 @@ Prebuilt Docker images with verl configured for ROCm are available on `Docker Hu .. code-block:: bash - docker pull rocm/verl:verl-release-v0.7.1amd0 + docker pull rocm/verl:verl-0.7.1.amd0_rocm7.0.2_ubuntu22.04_py3.12_vllm0.20.2 2. Launch and connect to the Docker container @@ -61,7 +61,7 @@ Prebuilt Docker images with verl configured for ROCm are available on `Docker Hu -v "$HOME:$HOME" \ --shm-size 128G \ -w "$PWD" \ - rocm/verl:verl-release-v0.7.1amd0 \ + rocm/verl:verl-0.7.1.amd0_rocm7.0.2_ubuntu22.04_py3.12_vllm0.18 \ /bin/bash .. _build-your-verl-rocm-docker-image: @@ -69,11 +69,11 @@ Prebuilt Docker images with verl configured for ROCm are available on `Docker Hu Build your own Docker image -------------------------------------------------------------------------------- -1. Clone the `https://github.com/ROCm/verl `_ repository +1. Clone the `https://github.com/AMD-Ecosystem/verl `_ repository .. code-block:: bash - git clone https://github.com/ROCm/verl.git + git clone https://github.com/AMD-Ecosystem/verl.git 2. Build the Docker container using the Dockerfile in the ``verl/docker`` directory @@ -82,7 +82,7 @@ Build your own Docker image cd verl docker build --build-arg VLLM_REPO=https://github.com/vllm-project/vllm.git \ --build-arg VLLM_BRANCH=4ca5cd5740c0cd7788cdfa8b7ec6a27335607a48 \ - --build-arg VERL_REPO=https://github.com/ROCm/verl.git \ + --build-arg VERL_REPO=https://github.com/AMD-Ecosystem/verl.git \ --build-arg VERL_BRANCH=0eb50ec4a33cda97e05ed8caab9c7f17a30c05a9 \ -f docker/Dockerfile.rocm7 -t my-rocm-verl . diff --git a/amd-docs/sphinx/_toc.yml.in b/amd-docs/sphinx/_toc.yml.in index 32fedaacb20..07a7f939841 100644 --- a/amd-docs/sphinx/_toc.yml.in +++ b/amd-docs/sphinx/_toc.yml.in @@ -16,6 +16,8 @@ subtrees: entries: - file: examples/verl-examples.rst title: Run a verl example + - file: examples/verl-async-examples.rst + title: Run fully asynchronous verl examples - caption: Reference entries: @@ -28,7 +30,7 @@ subtrees: entries: - url: https://rocm.blogs.amd.com/artificial-intelligence/verl-large-scale-rocm7/README.html title: ROCm verl blog - - url: https://github.com/ROCm/verl/blob/main/CONTRIBUTING.md + - url: https://github.com/AMD-Ecosystem/verl/blob/main/CONTRIBUTING.md title: Contribute to verl - file: about/license.rst title: License diff --git a/amd-docs/sphinx/requirements.in b/amd-docs/sphinx/requirements.in index c831419df4d..834f75631ea 100644 --- a/amd-docs/sphinx/requirements.in +++ b/amd-docs/sphinx/requirements.in @@ -1 +1 @@ -rocm-docs-core==1.32.0 +rocm-docs-core==1.37.0 diff --git a/amd-docs/sphinx/requirements.txt b/amd-docs/sphinx/requirements.txt index cd4e2f3aab0..d8cf7dcf575 100644 --- a/amd-docs/sphinx/requirements.txt +++ b/amd-docs/sphinx/requirements.txt @@ -8,30 +8,30 @@ accessible-pygments==0.0.5 # via pydata-sphinx-theme alabaster==1.0.0 # via sphinx -asttokens==3.0.0 +asttokens==3.0.2 # via stack-data -attrs==25.3.0 +attrs==26.1.0 # via # jsonschema # jupyter-cache # referencing -babel==2.17.0 +babel==2.18.0 # via # pydata-sphinx-theme # sphinx -beautifulsoup4==4.13.4 +beautifulsoup4==4.15.0 # via pydata-sphinx-theme breathe==4.36.0 # via rocm-docs-core -certifi==2025.4.26 +certifi==2026.6.17 # via requests -cffi==1.17.1 +cffi==2.1.0 # via # cryptography # pynacl -charset-normalizer==3.4.2 +charset-normalizer==3.4.9 # via requests -click==8.2.1 +click==8.4.2 # via # jupyter-cache # sphinx-external-toc @@ -40,88 +40,88 @@ colorama==0.4.6 # click # ipython # sphinx -comm==0.2.2 +comm==0.2.3 # via ipykernel -cryptography==45.0.3 +cryptography==49.0.0 # via pyjwt -debugpy==1.8.14 +debugpy==1.8.21 # via ipykernel -decorator==5.2.1 +decorator==5.3.1 # via ipython -deprecated==1.2.18 - # via pygithub -docutils==0.21.2 +docutils==0.22.4 # via # myst-parser # pydata-sphinx-theme # sphinx -executing==2.2.0 +executing==2.2.1 # via stack-data -fastjsonschema==2.21.1 +fastjsonschema==2.21.2 # via # nbformat # rocm-docs-core gitdb==4.0.12 # via gitpython -gitpython==3.1.44 +gitpython==3.1.51 # via rocm-docs-core -greenlet==3.2.2 +greenlet==3.5.3 # via sqlalchemy -idna==3.10 +idna==3.18 # via requests -imagesize==1.4.1 +imagesize==2.0.0 # via sphinx -importlib-metadata==8.7.0 +importlib-metadata==9.0.0 # via # jupyter-cache # myst-nb -ipykernel==6.29.5 +ipykernel==7.3.0 # via myst-nb -ipython==8.36.0 +ipython==9.15.0 # via # ipykernel # myst-nb -jedi==0.19.2 +ipython-pygments-lexers==1.1.1 + # via ipython +jedi==0.20.0 # via ipython jinja2==3.1.6 # via # myst-parser # sphinx -jsonschema==4.24.0 +jsonschema==4.26.0 # via nbformat -jsonschema-specifications==2025.4.1 +jsonschema-specifications==2025.9.1 # via jsonschema jupyter-cache==1.0.1 # via myst-nb -jupyter-client==8.6.3 +jupyter-client==8.9.1 # via # ipykernel # nbclient -jupyter-core==5.8.1 +jupyter-core==5.9.1 # via # ipykernel # jupyter-client # nbclient # nbformat -markdown-it-py==3.0.0 +markdown-it-py==4.2.0 # via # mdit-py-plugins # myst-parser -markupsafe==3.0.2 +markupsafe==3.0.3 # via jinja2 -matplotlib-inline==0.1.7 +matplotlib-inline==0.2.2 # via # ipykernel # ipython -mdit-py-plugins==0.4.2 +mdit-py-plugins==0.6.1 # via myst-parser mdurl==0.1.2 # via markdown-it-py -myst-nb==1.2.0 +myst-nb==1.4.0 # via rocm-docs-core -myst-parser==4.0.1 +myst-parser==5.1.0 # via myst-nb -nbclient==0.10.2 +nbclient==0.11.0 # via # jupyter-cache # myst-nb @@ -130,77 +130,81 @@ nbformat==5.10.4 # jupyter-cache # myst-nb # nbclient -nest-asyncio==1.6.0 +nest-asyncio2==1.7.2 # via ipykernel -packaging==25.0 +packaging==26.2 # via # ipykernel - # pydata-sphinx-theme # sphinx -parso==0.8.4 +parso==0.8.7 # via jedi -platformdirs==4.3.8 +platformdirs==4.10.0 # via jupyter-core -prompt-toolkit==3.0.51 +prompt-toolkit==3.0.52 # via ipython -psutil==7.0.0 - # via ipykernel +psutil==7.2.2 + # via + # ipykernel + # ipython pure-eval==0.2.3 # via stack-data -pycparser==2.22 +pycparser==3.0 # via cffi -pydata-sphinx-theme==0.15.4 +pydata-sphinx-theme==0.16.1 # via # rocm-docs-core # sphinx-book-theme -pygithub==2.6.1 +pygithub==2.9.1 # via rocm-docs-core -pygments==2.19.1 +pygments==2.20.0 # via # accessible-pygments # ipython + # ipython-pygments-lexers # pydata-sphinx-theme # sphinx -pyjwt[crypto]==2.10.1 +pyjwt[crypto]==2.13.0 # via pygithub -pynacl==1.5.0 +pynacl==1.6.2 # via pygithub python-dateutil==2.9.0.post0 # via jupyter-client -pyyaml==6.0.2 +pyyaml==6.0.3 # via # jupyter-cache # myst-nb # myst-parser # rocm-docs-core # sphinx-external-toc -pyzmq==26.4.0 +pyzmq==27.1.0 # via # ipykernel # jupyter-client -referencing==0.36.2 +referencing==0.37.0 # via # jsonschema # jsonschema-specifications -requests==2.32.3 +requests==2.34.2 # via # pygithub # sphinx -rocm-docs-core==1.32.0 +rocm-docs-core==1.37.0 # via -r requirements.in -rpds-py==0.25.1 +roman-numerals==4.1.0 + # via sphinx +rpds-py==2026.6.3 # via # jsonschema # referencing six==1.17.0 # via python-dateutil -smmap==5.0.2 +smmap==5.0.3 # via gitdb -snowballstemmer==3.0.1 +snowballstemmer==3.1.1 # via sphinx -soupsieve==2.7 +soupsieve==2.8.4 # via beautifulsoup4 -sphinx==8.1.3 +sphinx==9.1.0 # via # breathe # myst-nb @@ -211,15 +215,18 @@ sphinx==8.1.3 # sphinx-copybutton # sphinx-design # sphinx-external-toc + # sphinx-multitoc-numbering # sphinx-notfound-page -sphinx-book-theme==1.1.4 +sphinx-book-theme==1.2.0 # via rocm-docs-core sphinx-copybutton==0.5.2 # via rocm-docs-core -sphinx-design==0.6.1 +sphinx-design==0.7.0 # via rocm-docs-core -sphinx-external-toc==1.0.1 +sphinx-external-toc==1.1.0 # via rocm-docs-core +sphinx-multitoc-numbering==0.1.3 + # via sphinx-external-toc sphinx-notfound-page==1.1.0 # via rocm-docs-core sphinxcontrib-applehelp==2.0.0 @@ -234,19 +241,18 @@ sphinxcontrib-qthelp==2.0.0 # via sphinx sphinxcontrib-serializinghtml==2.0.0 # via sphinx -sqlalchemy==2.0.41 +sqlalchemy==2.0.51 # via jupyter-cache stack-data==0.6.3 # via ipython -tabulate==0.9.0 +tabulate==0.10.0 # via jupyter-cache -tornado==6.5.1 +tornado==6.5.7 # via # ipykernel # jupyter-client -traitlets==5.14.3 +traitlets==5.15.1 # via - # comm # ipykernel # ipython # jupyter-client @@ -254,21 +260,20 @@ traitlets==5.14.3 # matplotlib-inline # nbclient # nbformat -typing-extensions==4.13.2 +typing-extensions==4.16.0 # via # beautifulsoup4 + # jupyter-client # myst-nb # pydata-sphinx-theme # pygithub # referencing # sqlalchemy -urllib3==2.4.0 +urllib3==2.7.0 # via # pygithub # requests -wcwidth==0.2.13 +wcwidth==0.8.2 # via prompt-toolkit -wrapt==1.17.2 - # via deprecated -zipp==3.22.0 +zipp==4.1.0 # via importlib-metadata diff --git a/amd-docs/what-is-verl.rst b/amd-docs/what-is-verl.rst index a0424e48e26..92b7605e5ed 100644 --- a/amd-docs/what-is-verl.rst +++ b/amd-docs/what-is-verl.rst @@ -1,67 +1,95 @@ -.. meta:: - :description: What is verl? - :keywords: verl, documentation, vLLM, reinforcement learning, deep learning, framework, GPU, AMD, ROCm, overview, introduction - -.. _what-is-verl: - -******************************************************************** -What is verl? -******************************************************************** - -Volcano Engine Reinforcement Learning for LLMs (`verl `_) is a reinforcement learning (RL) training library designed for the post-training of large language models. It is the open-source version of `HybridFlow `_, -which models post-training workflows as a dataflow graph. - -It provides a flexible and scalable system for implementing Reinforcement Learning from Human Feedback (RLHF) and other RL-based optimization workflows. The library integrates with modern language model training and inference stacks while prioritizing performance and modularity. - -verl is part of the `ROCm-LLMExt toolkit `_. - -Why verl? -==================================================================== - -verl is well suited for RL because: - -- Its **hybrid programming model** reduces complexity in RL dataflow construction while maintaining flexibility for a variety of algorithms. - -- The **modular APIs** support reuse and extension of existing infrastructure and model ecosystems, reducing engineering overhead. - -- **Performance and scalability** are core design goals, supporting efficient resource use across GPU clusters and multi-node, multi-framework training scenarios. - -- Active community engagement and open-source development make it suitable for both research and production workflows. - -verl features and use cases -==================================================================== - -verl provides the following features: - -- **Flexible RL Algorithms:** Supports extension and implementation of diverse RL algorithms using a hybrid programming model that unifies single-controller and multi-controller paradigms for efficient dataflow execution with minimal code. - -- **Modular Integration:** Integrates with existing LLM infrastructure such as PyTorch FSDP, FSDP2, Megatron-LM, vLLM, SGLang, and Hugging Face models, decoupling computation and data dependencies. - -- **Scalable Parallelism:** Flexible device mapping and parallelism support efficient use of multi-GPU and distributed cluster environments. - -- **High Performance:** Achieves training and rollout throughput through tight integration with optimized engines, and uses techniques such as efficient actor model resharding through the 3D-HybridEngine to reduce memory and communication overhead. - -- **Models and configuration:** Supports common model families such as Qwen, Llama, Gemma, and DeepSeek via Hugging Face, with YAML-based configuration and example scripts for datasets such as GSM8K. - -verl is commonly used in the following scenarios: - -- **RLHF Training for LLMs:** Train language models with RL algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) for RLHF and other alignment recipes. - -- **Agent Training:** Build RL-based agent training pipelines that interact with environments or tools. - -- **Research and Experimentation:** Rapidly prototype and evaluate different RL strategies and configurations on large-scale models. - -- **Production Deployments:** Integrate production-ready RL workflows using diverse backends and distributed computing resources. - -ROCm deployment and runtime -==================================================================== - -On ROCm, verl uses the same HybridFlow controller and worker layout as upstream, with AMD-specific container images and runtime. - -- ``rocm/verl`` ships verl, PyTorch, ROCm, and vLLM for the default AMD rollout stack - -- Ray worker placement includes HIP device visibility handling for multi-GPU scheduling on ROCm clusters - -- Layouts span single-node jobs and multi-node clusters, including Slurm-managed Ray deployments - - +.. meta:: + :description: What is verl? + :keywords: verl, documentation, vLLM, reinforcement learning, deep learning, framework, GPU, AMD, ROCm, overview, introduction + +.. _what-is-verl: + +******************************************************************** +What is verl? +******************************************************************** + +Volcano Engine Reinforcement Learning for LLMs (`verl `__) +is a reinforcement learning (RL) training library designed for the +post-training of large language models. It is the open-source version +of `HybridFlow `__, which models +post-training workflows as a dataflow graph. + +It provides a flexible and scalable system for implementing Reinforcement +Learning from Human Feedback (RLHF) and other RL-based optimization +workflows. The library integrates with modern language model training +and inference stacks while prioritizing performance and modularity. + +Why verl? +==================================================================== + +verl is well suited for RL because: + +- Its **hybrid programming model** reduces complexity in RL dataflow + construction while maintaining flexibility for a variety of algorithms. + +- The **modular APIs** support reuse and extension of existing + infrastructure and model ecosystems, reducing engineering overhead. + +- **Performance and scalability** are core design goals, supporting + efficient resource use across GPU clusters and multi-node, + multi-framework training scenarios. + +- Active community engagement and open-source development make it suitable + for both research and production workflows. + +verl features and use cases +==================================================================== + +verl provides the following features: + +- **Flexible RL Algorithms:** Supports extension and implementation + of diverse RL algorithms using a hybrid programming model that unifies + single-controller and multi-controller paradigms for efficient dataflow + execution with minimal code. + +- **Modular Integration:** Integrates with existing LLM infrastructure + such as PyTorch FSDP, FSDP2, Megatron-LM, vLLM, SGLang, and Hugging + Face models, decoupling computation and data dependencies. + +- **Scalable Parallelism:** Flexible device mapping and parallelism + support efficient use of multi-GPU and distributed cluster environments. + +- **High Performance:** Achieves training and rollout throughput through + tight integration with optimized engines, and uses techniques such + as efficient actor model resharding through the 3D-HybridEngine to + reduce memory and communication overhead. + +- **Models and configuration:** Supports common model families such as + Qwen, Llama, Gemma, and DeepSeek via Hugging Face, with YAML-based + configuration and example scripts for datasets such as GSM8K. + +verl is commonly used in the following scenarios: + +- **RLHF Training for LLMs:** Train language models with RL algorithms + such as Proximal Policy Optimization (PPO) and Group Relative Policy + Optimization (GRPO) for RLHF and other alignment recipes. + +- **Agent Training:** Build RL-based agent training pipelines that + interact with environments or tools. + +- **Research and Experimentation:** Rapidly prototype and evaluate different + RL strategies and configurations on large-scale models. + +- **Production Deployments:** Integrate production-ready RL workflows + using diverse backends and distributed computing resources. + +ROCm deployment and runtime +==================================================================== + +On ROCm, verl uses the same HybridFlow controller and worker layout as +upstream, with AMD-specific container images and runtime. + +- ``rocm/verl`` ships verl, PyTorch, ROCm, and vLLM for the default + AMD rollout stack. + +- Ray worker placement includes HIP device visibility handling for + multi-GPU scheduling on ROCm clusters. + +- Layouts span single-node jobs and multi-node clusters, including + Slurm-managed Ray deployments. +