[NPU][Spike] Fully-Async training: Qwen3-4B GRPO on NPU

[floatlibai]

Fully_Async

Mode	Entry	Rollout path
Sync	train.py	default sync rollout
Async	train_async.py	vime.rollout.fully_async_rollout.generate_rollout_fully_async

[floatlibai]

Setup 1

Item	Value
Hardware	A3
Model	Qwen3-4B
Dataset (train)	dapo-math-17k
Method	GRPO
Steps	200
TP	2
Actor GPUs	2
Rollout GPUs	2 (disaggregated)
rollout-batch-size / n-samples / gbs	8 / 4 / 32
rollout-max-response-len	1k
use-kl-loss	True

Test result

---

Overall Performance Summary

┌────────────────────┬──────────────────┬─────────────────────────┬─────────────────┐
  │       Metric       │ Sync (117 steps) │ Fully Async (200 steps) │   Improvement   │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ avg step_time      │ 31.5s            │ 20.8s                   │ 1.51× speedup   │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ step_time stddev   │ 6.6s             │ 10.8s                   │ —               │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ avg train_wait     │ 18.7s (59.3%)    │ 9.8s (34.7%)            │ 1.91× reduction │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ avg train_time     │ 12.8s            │ 11.0s                   │ —               │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ avg actor_train    │ 9.9s             │ 8.9s                    │ —               │
  ├────────────────────┼──────────────────┼─────────────────────────┼─────────────────┤
  │ avg update_weights │ 1.7s             │ 2.9s                    │ —               │
  └────────────────────┴──────────────────┴─────────────────────────┴─────────────────┘

Oscillation of step_time and rollout_time in fully_async mode
TL;DR
The oscillation is a depth-1 pipeline bubble. When rollout_time > train_time, the warm queue cannot replenish fast enough during one training step, creating a 2-step cycle where every other step must wait for fresh generation. When rollout_time ≤ train_time (Setup 2), the queue stays ahead and the oscillation disappears.

Mechanism
In fully_async mode, generate_rollout_fully_async() reads from a warm queue that is continuously filled by a background rollout worker. The perf/rollout_time timer measures how long the call blocks waiting for enough samples — it does NOT measure the full generation time (which happens asynchronously).

                    ┌─ train_wait ─┬── actor_train (~9s) ──┐
Step N (even, fast):│  queue full  │   training             │
                    │  (1.2s)      │                         │
                    │              │  ┌─ bg rollout (~9s) ──┤
                    │              │  │ generates ~31% of   │
                    │              │  │ next batch           │
                    └──────────────┴──┴─────────────────────┘
                                        → queue underfilled for step N+1

                    ┌─── train_wait (~25s) ────┬── actor_train (~9s) ──┐
Step N+1 (odd,slow):│  waiting for generation  │   training             │
                    │  rollout_time = 29s      │                        │
                    │                          │  ┌─ bg rollout ───────┤
                    │  ┌─ bg rollout ────────┐ │  │ generates full     │
                    │  │ generates full batch │ │  │ batch (34s total)  │
                    │  └──────────────────────┘ │  │                    │
                    └──────────────────────────┴──┴────────────────────┘
                                                   → queue full for step N+2

The root cause: rollout_production_rate < train_consumption_rate

Factor	0720 (oscillates)	0723 (stable)
Actual rollout generation time	~29s	~67s
actor_train time	~9s	~99s
rollout/train ratio	3.2× (rollout slower)	0.7× (rollout faster)
Queue produced during 1 train step	~31% of batch	~148% of batch
Result	Queue underfills → bubble	Queue stays ahead → no wait

The warm queue is filled at the rate of the rollout worker and drained at the rate of the training loop. The pipeline has a fixed depth of 1 (prefetch of one step ahead). When the drain rate exceeds the fill rate, the queue periodically depletes, causing the alternating bubble pattern.

[floatlibai]

Setup 2

Item	Value
Hardware	A3
Model	Qwen3-4B
Dataset (train)	dapo-math-17k
Method	GRPO
Steps	200
TP	2
Actor GPUs	2
Rollout GPUs	2 (disaggregated)
rollout-batch-size / n-samples / gbs	32 / 8 / 256
rollout-max-response-len	2k
use-kl-loss	False

Test result

---

Core Performance Metrics (per-step, excluding cold-start step 0)

┌─────────────────────┬──────────────────┬──────────────────────────────────┬────────────────────┐
  │       Metric        │ Sync (200 steps) │ Fully Async (98 effective steps) │    Improvement     │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg step_time       │ 178.3s           │ 104.0s                           │ 1.72× speedup      │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg train_wait      │ 97.2s (54.6%)    │ 4.6s (4.4%)                      │ 21.1× reduction    │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg train_time      │ 81.4s            │ 99.6s                            │ —                  │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg wait_time_ratio │ 54.5%            │ 4.4%                             │ GPU idle: 55% → 4% │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg update_weights  │ 1.6s             │ ~2–3s                            │ —                  │
  ├─────────────────────┼──────────────────┼──────────────────────────────────┼────────────────────┤
  │ avg actor_train     │ 81.1s            │ ~99s                             │ —                  │
  └─────────────────────┴──────────────────┴──────────────────────────────────┴────────────────────┘

GPU Utilization

┌─────────────┬───────────────────┬────────────────────────────┐
  │    Mode     │ Effective Compute │ Idle (waiting for rollout) │
  ├─────────────┼───────────────────┼────────────────────────────┤
  │ Sync        │ ~45%              │ ~55%                       │
  ├─────────────┼───────────────────┼────────────────────────────┤
  │ Fully Async │ ~94%              │ ~4%                        │
  └─────────────┴───────────────────┴────────────────────────────┘

Fully async mode reduces GPU idle time from 54.5% to 4.4%, virtually eliminating the train-wait bubble.

[floatlibai]

Setup 2 with model runner v2 enabled

Model Runner v2's performance is pretty poor, which might be due to issues with the Gumbel operator.

When rollout_time >> train_time, the warm queue can never build a surplus. Every step degrades to waiting for a full synchronous generation — effectively equivalent to sync. Async only delivers gains when rollout_time ≤ train_time.