[RFC] RL-Insight Online Monitoring System

[mengchengTang]

1. Summary

This RFC proposes an Online Monitoring subsystem in RL-Insight to improve live operational visibility during complex RL training (e.g., long context, agentic loops, distributed train/infer). The subsystem will provide:

• Metrics: counter increment, gauge set, histogram / distribution observe
• Traces: spans with attributes (latency and cross-component timelines)
• Logs (optional / phased): structured logging hook with a stable API surface
• Multi-process / multi-node ingestion and aggregation toward Grafana-friendly backends
• Grafana as the primary UI (metrics + traces + state-style timelines), including dashboard JSON export/import
• Optional offline export of time-sliced markers / trace-like JSON for post-hoc inspection

---

2. Motivation

Modern RL systems are increasingly distributed and agentic. Scalar experiment trackers are often insufficient for runtime questions, while continuous deep profiling is too expensive for day-to-day operation.

Pain Point	Description
Coarse signals	Tools like TensorBoard / WandB are strong for curves but often miss runtime structure (rank skew, stalls, phase boundaries, tool-loop latency).
Profiling cost	Profiling is rich but heavy for long runs; it should complement—not replace—lightweight online telemetry.
Cross-component correlation	Train/infer split and multi-turn tools require correlated metrics + spans across processes, not isolated logs.

RL-Insight focuses on performance insight with a well-defined data protocol; online monitoring extends that with always-on, low-friction telemetry aligned with the Grafana ecosystem.

---

3. Design Goals

• G1: Provide a small, framework-agnostic producer API (metrics + spans + structured log hook).
• G2: Support multi-node / multi-process ingestion with documented behavior under load (batching, backpressure, cardinality guidance).
• G3: Integrate with Grafana: Prometheus-style metrics, OTLP-compatible traces, and state timeline style panels where appropriate; ship versioned dashboard JSON for reproducible setup.
• G4: Define a versioned event schema so collectors, backends, and dashboards can evolve without breaking producers.
• G5: Provide an optional offline path: export time-sliced or trace-like JSON for external viewers (e.g., Chrome trace–compatible workflows where feasible).

---

4. Scope & Non-Goals

• In scope: online operational telemetry, reference local stack documentation (e.g., compose), and redaction hooks for sensitive fields.
• Out of scope (initially): replacing full GPU/kernel profilers as the primary analysis tool; hosted SaaS; guaranteed automatic root-cause attribution for every slowdown.

[mengchengTang]

@tardis-key

[tardis-key]

Here are two issues related to monitoring capabilities for reference
#45
#44

[tardis-key]

It is my preliminary design, waiting for further discussions with you.
This is the source file. Feel free to modify it directly based on this version.

Design

Task List

A. InsightCollector

1. Provide cross-process and cross-node data reporting and aggregation capabilities based on Ray.
2. Offer basic data reporting APIs, as well as function and variable monitoring capabilities.
3. Define the data format / event schema.
4. Provide offline data export capabilities (Chrome trace JSON, etc.) (Blocked by A.3).
5. Implement data reporting through data aggregation and the PlatformAdapter interface. Focus on performance overhead caused by data aggregation and reporting, as well as performance under high concurrency. (Blocked by C.1 and C.2)

B. RL Framework

1. Understand the training workflow of AgentLoop and PPO Trainer, and clarify the functions, variables, and other events that need to be monitored.
2. Complete code adaptation based on InsightCollector APIs by implementing decorator registration, property modification, and interface invocation. (Blocked by A.2)

C. PlatformAdapter

1. Provide data reporting and visualization capabilities for Grafana.
2. Provide data reporting and visualization capabilities for MLflow.
3. Provide data redirection capabilities for Wandb and MLflow (if necessary and easy to implement).
4. Different platforms support different data types; clarify the supported data type scope (metric/log/trace). If necessary, conduct secondary development for corresponding platforms to support custom data types.

D. Parser

1. Provide data analysis capabilities; the specific data to be analyzed is not determined yet.

Requirement Mapping

#46

• G1 → A.2
• G2 → A.1 / A.5
• G3 → C.1
• G4 → A.3
• G5 → A.4
  Support more fine-grained state monitoring on Rollout phase #45
• Implemented via manual framework collection in B.1
• Additionally, maximize reuse of MLflow capabilities through C.2
  support time-consuming monitoring on cpkt engine and tq. #44
• Implemented via B.1

[tardis-key]

#53

[tardis-key]

Thanks for your contribution! Online monitoring is now merged as an experimental feature, and follow-up tasks will be tracked in the roadmap #49 . This issue is closed.