Motivation
As the RL-Kernel and its upper-level RL-Engine scale to support large-scale distributed PPO/GRPO training, observability becomes critical. However, integrating Heavy-weight monitoring (like Prometheus) directly into the C++/CUDA operator level is an anti-pattern that introduces unacceptable overhead and breaks asynchronous CUDA streams.
We need a unified, low-overhead observability architecture that provides both micro-level hardware tracing (for kernel developers) and macro-level cluster monitoring (for SREs and researchers) via Grafana.
Architecture Design
We propose a Two-Tiered Observability System:
-
Micro-Level (C++ / CUDA Kernels): NVTX Tracing
- Inject NVIDIA Tools Extension (NVTX) markers in csrc/ops.cpp.
- Zero Overhead: These markers have negligible overhead and are only activated when profiled via Nsight Systems (nsys).
- Goal: Visualize exact kernel latency, GPU idle time, and stream synchronization events in flame graphs.
-
Macro-Level (Python Engine Layer): Prometheus Metrics
- Integrate prometheus_client at the Python Executor/Scheduler level (rl_engine).
- Run a lightweight background HTTP server to expose a /metrics endpoint.
- Goal: Track macro metrics for Grafana dashboards, such as:
- Output Token Throughput (Tokens/s)
- Hardware fallback rate (e.g., when inputs don't match specific kernel shapes)
- Rollout QPS and KV Cache fragmentation rate.
Proposed Implementation
Phase 1: NVTX Injection (Kernel Layer)
- Include <nvtx3/nvToolsExt.h> in csrc/ops.cpp.
- Wrap kernel launches with nvtxRangePushA and nvtxRangePop.
Phase 2: Prometheus Setup (Engine Layer)
- Add prometheus-client to requirements.txt.
- Implement a MetricsRegistry singleton in Python to handle Counter, Gauge, and Histogram.
- Expose port 8000 (or configurable) for Prometheus scraping.
Acceptance Criteria (DoD)
- C++ operators (e.g., prefix_shared_attention) are visible as distinct colored blocks in nsys timelines.
- The Python engine successfully exposes a /metrics endpoint.
- A sample grafana-dashboard.json is provided in the examples/ directory.
Motivation
As the RL-Kernel and its upper-level RL-Engine scale to support large-scale distributed PPO/GRPO training, observability becomes critical. However, integrating Heavy-weight monitoring (like Prometheus) directly into the C++/CUDA operator level is an anti-pattern that introduces unacceptable overhead and breaks asynchronous CUDA streams.
We need a unified, low-overhead observability architecture that provides both micro-level hardware tracing (for kernel developers) and macro-level cluster monitoring (for SREs and researchers) via Grafana.
Architecture Design
We propose a Two-Tiered Observability System:
Micro-Level (C++ / CUDA Kernels): NVTX Tracing
Macro-Level (Python Engine Layer): Prometheus Metrics
Proposed Implementation
Phase 1: NVTX Injection (Kernel Layer)
Phase 2: Prometheus Setup (Engine Layer)
Acceptance Criteria (DoD)