Phase 6: Add profiling-guided fast-path expansion telemetry

[inaniloquentee]

Background

Split from #6 Phase 6: Fast Path Expansion From Profiling Bottlenecks.

This issue owns the profiling and target-selection layer after the consistency/audit and operator-contract stack is available. The goal is to make fast-path work follow measured vime bottlenecks rather than convenient kernel boundaries, while keeping strict-consistency claims separated from opportunistic speed paths.

Scope

• Add clean warmup and stable-window profiling entry points for representative vime workloads.
• Add or formalize NVTX ranges for rollout wait, rollout execution, training forward, logprob, backward, optimizer, weight sync, and communication.
• Record communication metrics: AllReduce/AllGather count and time, rank skew, overlap status, and topology notes.
• Record compute/communication segment telemetry: compute time, exposed communication time, overlapped communication time, idle/wait time, stream/event plan, bucket assignment, and overlap fallback reason.
• Add a profiling report path that separates operator-level, actor-window, and full-step measurements.
• Use Phase 3-5 attribution outputs to choose fast-path targets, especially TP communication and high-count small elementwise/reduce/copy kernels.
• Add backend descriptors or telemetry fields that state whether communication is in-op, post-op NCCL, custom all-reduce, or overlapped bucket, and whether that policy is strict-eligible.
• Keep linear_logp operator timing/memory claims separate from full-step claims.

Non-Goals

• Do not implement hardware-specific Blackwell/B-card CUDA kernel replacements here; [Experiment] Build Blackwell train-inference mismatch baseline for VIME post-training #24 can consume this issue's profiling output to choose candidate kernels.
• Do not duplicate Phase 3 audit/replay/result-cube ownership from Phase 3: Build consistency audit harness and replay diagnostics #29.
• Do not duplicate Phase 4 single-card operator comparison from Phase 4: Add single-card operator comparisons and drift attribution #30.
• Do not duplicate Phase 5 distributed operator contracts or A0-A5 profiles from Phase 5: Add distributed operator contracts and A0-A5 profiles #31.
• Do not claim full-step speedup from isolated operator timing.
• Do not label communication overlap or bucket fusion as strict unless reduction order and bucket-boundary contracts are explicitly covered.

Acceptance Criteria

• Profiling output names warmup, measurement window, model/workload shape, world sizes, dtype, backend choices, and runtime/build fingerprints.
• NVTX/telemetry separates rollout, train compute, logprob, backward, optimizer, weight sync, and communication time.
• Communication telemetry records count/time/skew/overlap and enough topology metadata to compare runs.
• Fast-path candidates are selected from measured bottlenecks and linked back to Phase 3-5 attribution or profiling evidence.
• Reports clearly separate operator-level, actor-window, and full-step metrics.
• Strict-consistency eligibility is never inferred from performance telemetry alone.

Hardware / Execution Notes

• Unit tests for report builders and telemetry normalization should run on CPU or single-card developer machines.
• NVTX emission and single-GPU operator/actor-window profiling need one CUDA GPU.
• Real communication and rank-skew validation need distributed GPUs, typically at least 2 GPUs for TP/collectives and preferably 4-8 GPUs for representative TP/SP/CP sweeps.
• Blackwell/B-card hardware is only required for Blackwell-specific profiling or claims, not for the generic vime profiling framework.

Dependencies

• Roadmap: RL-Kernel integration roadmap for vime #6.
• Phase 3 audit harness: Phase 3: Build consistency audit harness and replay diagnostics #29.
• Phase 4 single-card operator attribution: Phase 4: Add single-card operator comparisons and drift attribution #30.
• Phase 5 distributed contracts: Phase 5: Add distributed operator contracts and A0-A5 profiles #31.
• Blackwell experiment consumer: [Experiment] Build Blackwell train-inference mismatch baseline for VIME post-training #24.