RL post-training that spans prototype to scale.
Warply is building one Python API for the RL post-training loop — generate, score, update, sync — that runs the same user code from a single GPU to asynchronous, disaggregated rollout/train clusters. Simple like a toy framework, scalable like an industrial one, locked to no platform.
Status: Pre-alpha, and mid-refocus (#11). Warply began as a serving control plane; it is being rebuilt around RL post-training, where the same disaggregation machinery solves a problem we hit ourselves. The API below is a design target, not yet runnable. The serving-era README is preserved in git history.
RL post-training tooling forces a bad choice:
- Simple frameworks don't scale. Prototype-friendly APIs hit a wall when you need real rollout throughput or multi-node training.
- Scalable frameworks are hostile. Industrial stacks bury the loop under distributed-runtime config, YAML sprawl, and opaque failures.
- Between them is a rewrite. Teams prototype in one framework, then rebuild everything in another to scale. The rewrite is the tax everyone pays and nobody defends.
The hard part of the loop — disaggregated rollout/train orchestration, weight sync, staleness — is infrastructure, and it should be invisible.
Design spec — implementation tracked in #13 and #14.
import warply as wp
def reward(prompt: str, completion: str) -> float:
... # any callable: programmatic check, verifier model, test suite
run = wp.RLRun(
policy="Qwen/Qwen2.5-7B-Instruct",
reward=reward,
algo="grpo",
)
run.train() # one GPU, today’s laptop-scale prototypeThe same code, disaggregated and asynchronous — a config change, not a rewrite:
run = wp.RLRun(
policy="Qwen/Qwen2.5-7B-Instruct",
reward=reward,
algo="grpo",
rollout=wp.Pool("4xH100", replicas=2), # inference workers (vLLM/SGLang)
train=wp.Pool("8xH100"), # FSDP2 trainer
sync="async", # staleness-tolerant weight sync
)
run.train()Inspection is first-class, because opaque training loops are the #1 complaint about existing stacks:
run.stats() # per-stage throughput: rollout tokens/s, samples/s, staleness, utilization
run.events() # lifecycle history: batches produced, weights synced, updates applied
wp.doctor(run) # offline preflight: memory fit, engine availability, reward sanity- Same code, one GPU → multi-node async. Scale is configuration, never a rewrite.
- No Ray in your face. Whatever runs underneath, users never write placement groups.
- Pure-Python, typed config. No YAML sprawl.
- Engines and trainers are not rebuilt. vLLM/SGLang generate; FSDP2 updates. Warply owns the seam: orchestration, sync, observability.
- The loop is never opaque. What's generating, what's stale, and why it's hung are always one call away.
| Status | |
|---|---|
Typed spec/config discipline, Pool, deterministic plan compilation |
Exists (serving-era, being repurposed) |
events(), stats(), doctor() observability primitives |
Exist (serving-era, being repurposed for the loop) |
| Single-node GRPO loop (vLLM rollouts, FSDP2, weight sync) | In progress — #13 |
| Async disaggregated rollout/train behind the same API | Planned — #14 |
| Benchmark vs. existing frameworks (incl. lines-of-config) | Planned — #16 |
| Gap-analysis design doc | Planned — #12 |
Nothing is claimed as working that isn't. When the MVP runs, this table will say so.
- LLMs first: GRPO with verifiable rewards, one model family, end-to-end.
- Async disaggregation: the prototype→scale span, demonstrated as a config change.
- VLMs next: same loop, multimodal rollouts.
- Further stages are earned by demand, not designed in advance.
git clone https://github.com/warply-ai/warply.git
cd warply
pip install -e ".[dev]"
ruff check warply tests
pytest -qCI runs on Python 3.10, 3.11, and 3.12.
- Direction and discussion: #11
- Issues: bugs, design, and build order
- Contributing guide: CONTRIBUTING.md
- Security policy: SECURITY.md
- Code of conduct: CODE_OF_CONDUCT.md
Apache 2.0. See LICENSE.