[roadmap] RL-Insight Q1 Roadmap

[tardis-key]

Core

• ci
• prepare simplified data from mstx/nvtx/torch_profile in /data. [data, torch_profile] test: add st of torch and mstx #23
• add ST in /tests/special_e2e. [data, torch_profile] test: add st of torch and mstx #23
• add UT [ci, mstx] fix: fix ut for merge_short_events and delete the unused data_type. #2
• basic pr check. [ci] fix: enable module check and pre-commit #4
• build the initial version of docs
• readme. [doc] chore: init readme and doc #7
• provice docs related data/pipeline/parser/visualizer, base on the existing features. [doc] chore: add guideline for offlinepipeline #40
• compile doc, setup rl-insight.readthedocs.io/deepwiki. [doc] feat: add readthedocs and deepwiki #15
• commuity
• Good first issue / Help wanted

Data

• InputData Specification
• define BaseClass fo Data, including basic behavior, format checking [data] feat: add DataChecker #28
• derive MultiJsonData, a universal data format shared by nvtx/mstx/torch_profile. It consists of json/json/gz from different directories, where the timeline data contains role/rank/start_time/end_time...
• derive VerlLogData, it will also serve as an import input for performance insights. @duesdues [data] feat: add VerlLogData validation rules #35
• OutputData Specification
• dervie SummaryEventData from BaseClass, clarify the struture and fields contained in results from different Parsers. @tifning [data] feat: add data check rule func ParserOutputValidatorRule #36

Offline pipeline

• define the current offline pipeline, (InputData->Parser->OutputData->Visualizer) e95ca74
  [mstx, torch_profile] feat: Refactor main pipeline to pipeline_runner #16
• parser & examples
• mstx e95ca74
• nvtx
• torch_profile [torch_profile, ci] feat: scheduling analysis based on profiling data for torch profiler with ut #1
• visualizer
• html timeline e95ca74
• png timeline

Online pipeline

• pre-search for online monitor
• define an online pipeline
• plugin: wandb/tensorboad/...

CollectorController

• cross-process data collection capability. Due to limitations in the config scope of DistProfiler, it is necessary to use complex parameter passing to obtain configurations for different scenarios, which typically results in long control-chains and usage restrictions. A global instance for cross-process data sharing can be used to achieve more flexible data collection.

Metric

• quantify some key metrics to evaluate training efficiency.

[duesdues]

I’ll take this task: [Data] "derive VerlLogData, which will also serve as an important input for performance insights."

[tifning]

i'll take this task:[DATA] "dervie SummaryEventData from BaseClass, clarify the struture and fields contained in results from different Parsers."

[tardis-key]

welcome! you can wait utill BaseClass for data is ready @duesdues @tifning

[Debonex]

I'll take this task: [DATA] "define BaseClass fo Data, including basic behavior, format checking"

[Gary-cjy]

I’ll take this task: [Data] "The visualizer outputs the timeline in PNG format"

[zhihaofang1017]

I have completed the work of [doc]“compile doc, setup rl-insight.readthedocs.io/deepwiki”, and the PR is: #15

[tardis-key]

Suggestions from @Kirrito-k423 in verl-project/verl#5439

This profiling RFC addresses an important gap for async mode debugging. A few suggestions:

1. Unified Trace Format
Consider defining a verl-specific trace format (JSON-based) that abstracts away hardware differences:

{
  "rank": 0,
  "timestamp_us": 1234567890,
  "event_type": "rollout_start",
  "duration_us": 50000,
  "metadata": {"batch_size": 16}
}

Converters for each hardware backend (nsys, mstx, torch.profiler) would emit this format.

2. Key Metrics to Track
For async mode specifically:

• Time between rollout completion and training start (staleness)
• Rollout/training overlap ratio
• Per-rank memory utilization over time
• NCCL/HCCL communication time breakdown

3. Visualization
Consider integrating with existing tools:

• Chrome Trace Viewer (catapult) - already used by torch.profiler
• Perfetto - more modern, better for large traces
• TensorBoard profiler plugin

4. Integration with wandb/tensorboard
Export key metrics to wandb/tensorboard for real-time monitoring during training.

Looking forward to the implementation!