CPU training drops to ~1 step/sec after replay buffer warms up

[CatEllite312]

CPU-only training: severe step slowdown after ~2 episodes

Environment

• CPU only (no CUDA), single agent (--num_agents 1)
• Observed on: 2 x Intel Xeon E5-2667 v4, Linux

Problem

When training with a single agent on CPU, the first 1–2 episodes run at normal speed (~19 steps/sec), then throughput drops to ~1 step/sec and stays there for the rest of the session.

Root cause

The training loop calls agent.optimize_model() on every single step with no skip condition. On a GPU this is fine — a forward+backward pass on the 1.4M-parameter HybridDuelingDQN takes ~5–10ms. On CPU the same operation takes 500–900ms, which completely blocks the game loop.

The first 1–2 episodes are fast because optimize_model() skips internally when the replay buffer hasn't reached min_buffer_size yet. Once the buffer is warm, every step stalls waiting for the optimizer.

The project's default target_update_freq = 10000 has a comment "Increased for stability with 10 agents", which confirms the hyperparameters were tuned for multi-agent GPU training and not for single-agent CPU use.

Fix

1. Add train_freq to skip optimization on most steps (trainer.py, around line 2077):

# Before:
metrics = agent.optimize_model()

# After:
if total_steps % 4 == 0:
    metrics = agent.optimize_model()
else:
    metrics = None

This reduces CPU load by ~4× while keeping learning quality intact — all 4 skipped transitions are still stored in the replay buffer and will be sampled during the next optimize_model() call. train_freq=4 was the default in the original DeepMind DQN paper for exactly this reason.

2. Adjust target_update_freq proportionally (config.py):

# Before:
target_update_freq: int = 10000

# After (for single agent, train_freq=4):
target_update_freq: int = 2500

With train_freq=4, the target network would otherwise update only every 40 000 steps instead of 10 000, slowing convergence. Dividing by 4 restores the original effective update frequency.

Result

After the fix: consistent ~15–19 steps/sec throughout training on CPU, same as the first two episodes before the buffer warms up. Learning quality is unaffected — the replay buffer accumulates experience on every step regardless of whether optimize_model() is called.

Suggestion

It may be worth adding an explicit train_freq parameter to config.py (defaulting to 1 for GPU, recommended 4 for CPU) and documenting the CPU single-agent use case in the README.