Repository navigation
[GPU] Improve Perfetto support for General-Purpose GPU (GPGPU) compute workloads #5097
Description
Overview
Modern GPUs are used far beyond traditional graphics rendering — powering scientific simulation, machine learning, image processing, physical modeling, and other massively parallel workloads through General-Purpose Computing on Graphics Processing Units (GPGPU). Perfetto's GPU tracing support was designed primarily around graphics pipelines (Vulkan, OpenGL, GLES), and while it can partially represent compute workloads today, several gaps in the data model, trace config, and UI make it less than ideal to analyze GPGPU workloads effectively.
This issue tracks a collection of related improvements spanning API coverage, counter collection controls, UI capabilities, and the render stage data model — with the goal of making Perfetto a first-class tracing backend for GPU compute workloads across CUDA, HIP, SYCL, and other GPGPU runtimes.
Motivation
GPGPU profiling tools like CUPTI (CUDA), ROCProfiler/ROCTracer (HIP/ROCm), and Intel's GPU profiling stack already expose rich performance data. Producers such as PyTorch Profiler (via libkineto) and vendor profiling SDKs typically have some Perfetto support today but typically limited to using track events and debug annotations to record GPU events. This limits the UI plugins and tools that can be built on top of Perfetto traces for analyzing GPU workloads. The improvements outlined below describe how to address some of these gaps.
Known Limitations & Proposed Improvements
1. Missing API Enums for CUDA / HIP
The current GpuRenderStageEvent and related protos lack enum values for CUDA and HIP APIs. Producers targeting these runtimes are forced to use placeholder or vendor-specific values, making cross-tool interoperability fragile.
Proposed: Add first-class enum entries for CUDA and HIP (and potentially SYCL, Metal Compute, OpenCL) in the relevant GPU proto definitions.
2. Multi-GPU Support in the UI
Perfetto's UI does not clearly differentiate or aggregate events across multiple GPU devices. For multi-GPU compute jobs running across 4, 8, or more devices, the current UI falls apart.
Proposed (v1):
- Surface
gpu_idas a first-class grouping dimension in the UI track hierarchy - Allow users to view per-GPU timelines side-by-side or in a collapsible grouped view
- Ensure SQL queries and metrics surface
gpu_idconsistently
Proposed (v2):
- Deprecate
gpu_idand introduce agpu_iidand a GPU description proto message (InternedGpu) - Providing more detailed data about GPUs in the trace. E.g. Vendor, arch, UUID, etc.
3. Counter Hierarchy Based on Hardware Blocks
GPU performance counters today appear as a flat list, making it hard to navigate the large number of counters exposed by hardware vendors (e.g., SIMT core, L2 cache, tensor cores).
Proposed: Introduce a hierarchical grouping mechanism for GPU counters in the UI, allowing producers to use the existing GpuCounterBlock to organize the counters.
4. Render Stages Split into Groups by API
Currently, render stages are presented as a single flat set of hw queues with events regardless of their originating API (Vulkan, OpenGL, CUDA, HIP, Metal, etc.). In workloads that mix graphics and compute — or that use multiple compute runtimes simultaneously — this makes it hard to attribute time to the correct API layer. The term "Render Stages" in the UI is also a bit confusing for GPGPU workloads that are limited to Compute.
Proposed: Allow render stage tracks to be optionally grouped by API (e.g., a Vulkan group vs. a CUDA group) in the UI.
5. Kernel Name as Event Name for Compute Stage Events
Today, compute stage events use a fixed set of render stage event names defined in the spec. For GPGPU workloads, the most meaningful label for a GPU event is the actual kernel name (e.g., volta_sgemm_128x64_nt, reduce_kernel<float>), not a generic stage name.
Proposed: Allow producers to supply a free-form name as the event name for compute-stage GPU events, as an alternative or override to the fixed render stage name. This would dramatically improve navigability of compute-heavy traces and bring Perfetto in line with how other GPGPU profilers surface kernel activity.
6. Fine-Grained Control Over Counter Collection Scope
GPGPU hardware typically supports instrumented collection of a large set of counters. Often represented as a higher level metric that is identified by a string and might be derived from a number of hardware counters. These type of performance counters/metrics are expensive to collect — enabling them for every kernel invocation introduces significant overhead and can perturb the workload being profiled. CUPTI and ROCProfiler already expose mechanisms like replay ranges and kernel filtering for this reason, but Perfetto has no equivalent concept in its trace config.
Proposed: Introduce a counter collection scoping mechanism in the trace config that allows consumers to be more specific about the instrumented counters they want and producers to restrict which kernels a given set of counters is collected for. Specifically:
- Kernel name filter: Collect counters only for kernels matching a name pattern (exact string or regex)
- Kernel skip / count: Skip the first N invocations of a matched kernel and collect for the subsequent M invocations — useful for steady-state profiling and warmup avoidance
- Explicit collection ranges: Define named start/end boundaries (analogous to CUPTI's NVTX ranges) within which counter collection is active, allowing producers to scope profiling to a specific region of interest within a longer-running workload
This maps naturally onto how profiling tools like Nsight Systems, CUPTI, and ROCProfiler already work, and would make Perfetto a viable backend for production-quality GPGPU performance analysis.
7. Counter Selection by Name Instead of ID
The current GpuCounterConfig in the trace config requires counters to be specified by numeric ID, as defined in GpuCounterDescriptor. This creates a brittle coupling between the trace config and the producer's internal counter numbering — IDs can vary across driver versions, vendors, and GPU generations, making configs non-portable across hardware.
Proposed: Allow counters to be specified by name (string) in GpuCounterConfig, in addition to the existing ID-based selection. The producer would be responsible for resolving names to IDs at trace start using the descriptors it has already emitted. Benefits include:
- Portable trace configs that work across GPU vendors and driver versions (e.g.
"l2_read_hit_rate"instead ofcounter_id: 47) - Easier human authoring and review of trace configs
- Better alignment with how CUPTI, ROCProfiler, and vendor tools already expose instrumented counters (metrics) by name
A recommended approach would be to add an optional counter_names repeated string field alongside the existing counter_ids in GpuCounterConfig, with the union of ids and names being collected.
8. Flow Event Connections Between Host Track Events and GPU Render Stage Events
Currently there is no mechanism to associate host-side track events with the GPU render stage events they produce. This makes it impossible to follow the causal chain from an API call on the CPU — such as a kernel enqueue, command buffer submission, or synchronization primitive — through to the corresponding GPU execution event in the trace UI.
This connection is particularly valuable for:
- Kernel dispatch: Linking a host-side
cuLaunchKernel/hipLaunchKernel/vkQueueSubmitcall to the render stage event representing that kernel's execution on the GPU - Synchronization: Connecting host-side calls that block on GPU completion (e.g.
cudaStreamSynchronize,vkWaitForFences) to the GPU events they are waiting on, making synchronization overhead and idle time easier to root cause
Proposed: Extend the GPU render stage event model to support Perfetto flow events linking host TrackEvent slices to GpuRenderStageEvent entries. Producers would emit flow IDs on both the host-side event and the corresponding GPU event, allowing the Perfetto UI to identify proceeding and following flows and render flow event arrows accordingly.
Suggested Approach
These improvements can be tackled independently. A reasonable order might be:
- Proto changes for CUDA/HIP enums and kernel name support (unblocks producers immediately)
- Counter selection by name and counter collection scoping (high value for GPGPU profiling tool integrations)
- Counter hierarchy in the data model
- UI changes for multi-GPU grouping and render stage API grouping
Activity
LalitMaganti commentedon Mar 12, 2026
In general I'm in favour of much/most of the changes proposed here. All seem sensible and well considered. 5/6/7 could be more controversial depending on their implementation and warrant further discussion when we get there
The only thing I will say is the Perfetto team itself has zero context or expertise with GPUs and so we need someone else to champion this externally.
I should also cc @emrekultursay and @olehkuznetsov who work on GPU related things as an fyi to this proposal and how it will interact with their downstream tools/data collection.
dreveman commentedon Mar 12, 2026
@LalitMaganti thanks for the quick feedback. I'm interested in implementing and maintaining these changes in upstream.
All this can be done using alternative protos, SQL modules and UI plugins (some of that already exists) but I would prefer to keep as much as possible in upstream and compatible with the GPU support we have there.
primiano commentedon Mar 12, 2026
A bunch of comments on different levels
Generally, one thing I always missed in our project is somebody to own all the ideas and vision for the GPU world. What we have today GPU-wise is really the union of a number of initiatives that happened over the years
I'd love if there was somebody (or a steering group of few trusted people) willing to lead the overall "here's how GPU tracing should work". GPU in perfetto is the area where very few people have a set of very strong opinions (and for once I am NOT one of them)
My main request, before we go down in the specifics, is that we get some alignment across these few people who contributed the most historically and that I feel proved to have good opinions on the situtation.
I would have to do a bit of digging, but I feel the right set of people is a union of pmuetschard@google and jbates@google, sidaths (ex-google, now arm) and somebody from QCom .
For us is very important to have both ARM and QCom in the discussion, as most of the GPU stuff we have in Android is really implemented by them in the GPU drivers, and we need them onboard to make some of these changes.
Now more specific
I like the idea overall, but the reality is that I have no idea how GPU counters even show up today. Any screenshot / link to trace?
Hmm here I'd like to understand more. My mental model for render stages, as it has been "sold" to me back then was:
so my understanding is that render stages were trying to mimic what the "cpu scheduling" tracks do, pivoting on the functional units (the gpu queues)
In my mind we need to have the same thing we have for CPU thread slices, where thread slices show BOTH on the cpu tracks (when they are running) and also in the process track groups (so you can see the evolution of a specific thread, regardless of which CPU it was scheduled on).
isn't something similar for the APIs vs hw queues?
I have no opinion / I don't know enough. sounds reasonable.
So this is probably where we should loop in people who worked on the driver side.
I remebmer hw counters being a major drama because the to big GPU vendors we have in Android operate in widely different ways. I think what we have today is the result of having tried to normalize for the lowest common denominator.
The problem here is that you need to convince people in ARM and QCom to change their data source, because that's what parses the config
LalitMaganti commentedon Mar 12, 2026
I guess maybe where @dreveman is coming from (please correct me) is that that are implementing these data sources and/or writing these protos synthetically for some custom GPU driver code etc they have and they're looking for ways of representing information they have.
I am guessing most of what they're proposing is "optional/opt-in things a GPU vendor could do" rather than "something everyone needs to switch to".
primiano commentedon Mar 12, 2026
yeah but what I want to avoid is a world where there are some options in the traceconfig but some of them work only on GPU verndor A, some others only on B and so on.
If anything ,we need to come up with a set of capabilities that GPU data sources advertise, so the UI and the tooling can tell you "you cannot refer to gpu counters by name"
But really most of the difficulty here is not implementing a GPU data source or editing the config. is really figuring out something that can work consistently (at least from a schema/proto viewpoint) across the vendors
emrekultursay commentedon Mar 12, 2026
Android Games & Graphics team and the Android Studio Profiler team have been driving cross-IHV GPU profiling/tracing conversations over the past 2 years. We'd be happy to take inputs from more sources.
We've been working very closely with S.LSI, which resulted in Sokatoa, and we've held many meetings with the profiling teams of ARM, Qualcomm, and IMG. One of our goals with these has always been to get more data pushed out via Perfetto, using unified solutions and data formats. We have sicooke@google from Games DevRel organizing these cross-vendor efforts. We are also working on a newer version of Android GPU Inspector (based on Perfetto), and its primary premise is to work with all Android GPUs.
We've also restarted updating Android requirements and are actively raising the bar. For instance, Android 17 CDD will require counter groups to be reported, and many CTS tests were added to enforce devices report GPU counters correctly. This is another area we are heavily working on.
dreveman commentedon Mar 12, 2026
The config changes are probably the least critical as custom vendor specific extensions to those protos are relatively easy to maintain.
I think the higher level question that needs to be answered is; do we want Perfetto to have first-class support for GPGPU workloads like Inference and Training for GenAI? It's already used for some of that today but it's typically done by dumping everything into track events, which makes the traces huge and drastically limits the usefulness as it can't be visualized and analyzed properly.
We can always create a separate set of protos, SQL tables, and UI plugins to serve the GPGPU purpose but the existing GPU support is very close to what is needed, which is why I'm proposing that it's extended to serve this additional use-case better. Most of what I'm interested in adding (excluding the trace config stuff) should fall into the category; if the producer like to provide more details about GPGPU workloads then that data is available for processing and the UI can visualize that better. But if not, nothing changes from now.
I might have gone into too many specific details in the description of this issue. I think a lot of that can be discussed separately. The main thing we need to agree on is if the current GPU support should be extended to have first-class support for GPGPU workloads or be limited to graphics on Android devices (which seems to be the case today?)
LalitMaganti commentedon Mar 12, 2026
This is already not the case: Mesa (on Linux/ChromeOS) uses these protos already. The design has mainly been driven by Android certainly but we already have many people using these these even outside Android that we support today.
As long as we're not breaking existing workflows and generally keeping things consistent I'm supportive But I would also want all users who are interested to be kept in the loop so that are keeping everyone aligned.
dreveman commentedon Mar 12, 2026
Ok, yeah that makes sense. There's a focus on mobile/desktop graphics, with some basic compute support, which makes sense.
We're on the same page here. I created this issue for that purpose but don't know who needs to be aligned so appreciate you adding those people.
primiano commentedon Mar 13, 2026
So after sleeping more on this, I think the best way to proceed is to split this proposals into:
Because they involve different sets people, and help maintain the discussion scoped and limited, which allows faster progress.
I would really love your inputs and, to your question, yes in these days it would be great having better support for GPGPU. It's just going to be trick if we try to do everything at once, because i need to mediate between people across several companies that care about slightly different angles of GPU tracing.
From my side I can help with:
WDYT?
dreveman commentedon Mar 13, 2026
Sounds good. 1 and 2 is already how I imagined code review would be split.
The UI and trace processor improvements shouldn't be very controversial. Especially not if they are guarded by some setting flags and the default behavior remains the same as before. I think most of the UI features that I've identified so far can be done without any changes to the data sources except for one that I didn't mention in the initial description of the issue but added just now.
This would need an extension to the render stage event.
ui: Improve GPU track naming
ui: Improve GPU track naming (#5203)
5 remaining items
dreveman commentedon Apr 13, 2026
Update
Quick progress report on the GPGPU work. Thanks everyone for the early feedback and code reviews (@LalitMaganti @olehkuznetsov @primiano @stevegolton) — I've been landing this in pieces that can be tested already and I think we can start looking at adopting it in more advanced producers (e.g. PyTorch/Kineto) once in-review changes have landed.
Landed
CUDA,HIPtoGpuRenderStageEventand related protosgpu_idis now surfaced as a first-class grouping dimension in the UI track hierarchy, SQL tables, and metricsugpuin SQL tablesGpuRenderStageEventnow acceptsnameoverriding the fixed stage enumInstrumentedSamplingConfigtoGpuCounterConfigevent_idfromTrackEvent→GpuRenderStageEventforcuLaunchKernel/hipLaunchKernel/vkQueueSubmitand sync primitivesdreveman commentedon Apr 24, 2026
The UI plugin needs more work and there are some other follow up changes planned but I we're now in a good place when it comes to general support for GPGPU workloads. Closing this.