Building cross-hardware and multi-model RL post-training infrastructure for kernel-level train–inference consistency.
Architecture · Current scope · Results · Hardware support · Quick start · Documentation
RL-Kernel is high-performance infrastructure for RL post-training. It provides deterministic operators for consistent numerical computation across rollout and training engines, together with hardware-specific kernels for faster execution and lower memory use in GRPO, PPO, and related workloads.
Today, the end-to-end path covers Qwen3-8B Dense with vime, vLLM, and Megatron-LM. Work on DeepSeek-V4 Flash MoE, Miles, and AReaL is ongoing.
Rollout and training engines can produce different log probabilities for the same tokens and model weights because their kernels, batching, and reduction orders differ. Those differences enter the policy ratios and KL terms used by RL algorithms.
- Exact train–inference consistency: deterministic operators keep rollout and training computations aligned. The published experiment records exact runtime LogP agreement across all 200 training steps.
- RL operators: deterministic attention, dense FFN, LogP, GRPO and PPO objectives, and collectives cover the numerical boundaries in RL post-training.
- Performance: fused computation and hardware-specific kernels reduce rollout time, memory use, and synchronization costs.
- vime integration: vime orchestrates vLLM rollout and Megatron-LM training, with RL-Kernel supplying the operators used by both engines.
- Hardware: NVIDIA SM90 and AMD gfx942 are supported. Ascend dav_c220 has partial operator coverage. Support for other hardware is in progress.
RL-Kernel sits between execution engines and accelerator backends. Its runtime adapters select the operator implementation for each backend while keeping the same numerical contract across rollout and training.
The architecture below shows how orchestration frameworks, execution engines, RL-Kernel operators, and hardware backends fit together.
The current end-to-end path uses Qwen3-8B Dense with vime.
| Area | Current | Next |
|---|---|---|
| Model | Qwen3-8B Dense | DeepSeek-V4-Flash-0731 MoE |
| Orchestration | vime | Miles and AReaL |
| Engines | vLLM rollout and Megatron-LM training | More rollout and training engines |
RL-Kernel currently supports the following hardware targets.
| Hardware | Architecture | Software | Status |
|---|---|---|---|
| NVIDIA H100, H200, GH200 | SM90 | CUDA | Supported |
| AMD Instinct MI300A, MI300X, MI325X | gfx942 | ROCm | Supported |
| Huawei Ascend dav_c220 | dav-2201 | CANN 9.1.0 and Ascend C | Partial |
| Moore Threads | In development | MUSA | In progress |
The published end-to-end benchmark was run on H100. The ROCm extension and backend checks were verified on MI300X. Ascend support is limited to dav_c220. Support for other hardware models is in progress.
Install Python 3.10 or newer, a PyTorch build matching your accelerator runtime, and the corresponding CUDA or ROCm compiler toolchain. Then clone RL-Kernel:
git clone https://github.com/RL-Align/RL-Kernel.git
cd RL-KernelFor the Qwen3-8B train–rollout commands and setup for vime with RL-Kernel on CUDA and ROCm, see the reproduction scripts.
Build against a visible NVIDIA GPU. Set TORCH_CUDA_ARCH_LIST when you want to pin the target architecture instead of relying on device detection.
# NVIDIA SM90: H100, H200, GH200
MAX_JOBS=8 \
RL_KERNEL_REQUIRE_EXT=1 \
TORCH_CUDA_ARCH_LIST="9.0+PTX" \
python3 -m pip install --no-build-isolation --no-deps -e .The CUDA build targets SM90 and has been tested on an NVIDIA H100 80GB HBM3. H100, H200, and GH200 use SM90. Support for other CUDA architectures is in progress.
Verify the loaded extension, GPU, SM capability, and required native symbol:
python3 -c "import torch, rl_engine._C as C; print('GPU:', torch.cuda.get_device_name(0)); print('Capability:', torch.cuda.get_device_capability(0)); print('Extension:', C.__file__); print('fused_logp:', hasattr(C, 'fused_logp')); assert hasattr(C, 'fused_logp'); print('H100 build: PASS')"The gfx942 build targets AMD Instinct MI300A, MI300X, and MI325X:
PYTORCH_ROCM_ARCH=gfx942 python3 setup.py developVerify the ROCm environment and required native symbol:
python3 scripts/check_rocm_env.py
python3 -c "import torch, rl_engine._C as C; print('GPU:', torch.cuda.get_device_name(0)); print('HIP:', torch.version.hip); print('Extension:', C.__file__); print('fused_logp:', hasattr(C, 'fused_logp')); assert hasattr(C, 'fused_logp'); print('MI300X build: PASS')"The extension and environment checks have been tested on AMD Instinct MI300X. Support for other ROCm architectures is in progress.
For CPU-only or pure-Python development, use an editable pip installation. Ascend has partial operator support on dav_c220 with the dav-2201 target. Moore Threads support is in progress. See the installation guide for backend dependencies and troubleshooting.
Join us on Slack or WeChat, and open an issue for bugs and feature requests. Contributions to kernels, framework integrations, hardware adaptation, and benchmarks are welcome. See the contributing guide.
RL-Kernel builds on the work of the open-source AI infrastructure community, including vime, vLLM, Megatron-LM, and FlashInfer. We thank their contributors and everyone helping bring RL-Kernel to new accelerators.
Licensed under the Apache License 2.0.