[Task][Megatron] 4.1: Pipeline parallel 1F1B / interleaved schedules

[lvyufeng]

Parent Issue

Part of #206 — Megatron Roadmap, Phase 4 (Advanced Features / Stretch): Task 4.1

Description

Implement pipeline parallel schedules used by Megatron-LM.

Requirements

• 1F1B (one forward, one backward) schedule
• Interleaved 1F1B schedule
• Micro-batch splitting and pipeline stage assignment
• Correct gradient accumulation across micro-batches
• P2P communication for activation/gradient transfer between stages

Blocked By

• [Task][Megatron] 1.3: Validate async P2P semantics for pipeline communication #254 (Async P2P semantics)
• Phase 2 and Phase 3 tasks

Blocks

• Multi-dimensional parallelism (TP × PP × DP)