The flydsl2flydsl performance harness
(tasks/flydsl2flydsl/*/test_kernel_harness.py) times each iteration like this:
start.record()
launch_fn(x, output, M)
end.record()
torch.cuda.synchronize()
times.append(start.elapsed_time(end))
launch_fn does a lot of host-side dispatch work between the two events. At that point the GPU stream has nothing else to run, so the GPU sits idle waiting for the CPU to enqueue the kernel, and that idle gap falls right between start and end — so elapsed_time(start, end) also counts the host launch overhead. For small ops / small shapes this host overhead dominates the measured time (the actual GPU compute is often just a few microseconds).
Why this is a problem
Does not reflect real deployment: real inference runs these kernels under CUDA/HIP Graph (graph-mode serving), which eliminates the per-launch host overhead via graph replay. So for small shapes the current metric does not represent how the op actually behaves in production.
Dilutes real GPU optimization: with a fixed host-launch floor, an agent's genuine GPU-compute improvements look negligible against that launch overhead.
Incentivizes the wrong thing: an agent can lower the measured time by optimizing the Python/launch path (e.g. caching the compiled dispatch state, reusing packed argument pointers) instead of the kernel itself — a "speedup" that disappears entirely once CUDA Graph is enabled.
Suggested fix
Measure only the actual GPU compute time, so the agent optimizes the kernel rather than the launch path.
The
flydsl2flydslperformance harness(
tasks/flydsl2flydsl/*/test_kernel_harness.py) times each iteration like this:launch_fn does a lot of host-side dispatch work between the two events. At that point the GPU stream has nothing else to run, so the GPU sits idle waiting for the CPU to enqueue the kernel, and that idle gap falls right between start and end — so elapsed_time(start, end) also counts the host launch overhead. For small ops / small shapes this host overhead dominates the measured time (the actual GPU compute is often just a few microseconds).
Why this is a problem
Does not reflect real deployment: real inference runs these kernels under CUDA/HIP Graph (graph-mode serving), which eliminates the per-launch host overhead via graph replay. So for small shapes the current metric does not represent how the op actually behaves in production.
Dilutes real GPU optimization: with a fixed host-launch floor, an agent's genuine GPU-compute improvements look negligible against that launch overhead.
Incentivizes the wrong thing: an agent can lower the measured time by optimizing the Python/launch path (e.g. caching the compiled dispatch state, reusing packed argument pointers) instead of the kernel itself — a "speedup" that disappears entirely once CUDA Graph is enabled.
Suggested fix
Measure only the actual GPU compute time, so the agent optimizes the kernel rather than the launch path.