Problem
Our Parquet kernels as currently constituted duplicate tightly coupled level-derived work: validity construction, output-position/rank calculation, nesting progress, andlist-offset state. decode_page_data_generic has a customized block-wide approach for this, while all the other kernels (GENERAL/BSS/delta binary) rely on the gpuDecodeLevels device function. This device function is designed to operate on a single warp, effectively imposing a warp-specialized pattern on every kernel using it. The gpuDecodeLevels function dates back to the earliest days of cuIO over 7 years ago, i.e. the age of Volta. The warp-specialized kernels were likely a reasonable choice at the time when GPU bandwidth was much lower and we had fewer tools available to hide the latency.
Modern GPUs, however, have much higher HBM bandwidth and larger caches, and newer PTX instructions supporting asynchronous staging of global reads into shared memory allow overlapping data and compute operations in more flexible ways. Conversely, the existing warp-specialized kernels have numerous drawbacks that cannot fundamentally be overcome: long register live ranges lead to high register pressure and reduce occupancy, while cross-warp coordination introduces expensive synchronization barriers. Based on profiling today, the costs of this approach now outweigh the benefits. Moreover, the proliferation in encodings supported by Parquet means that we have a much larger set of kernels that all reproduce this work and are all subject to the warp-specialization constraints. That needs to change.
Proposal
The common work that all decode kernels currently do should be moved to a separate preprocessing kernel that handles the validity and output position calculations. That new kernel will populate this information into global memory for subsequent kernels to consume. To fully take advantage of this change, every decode kernel will then need to be restructured to support proper utilization of the full thread block. This change is a substantial undertaking, and intermediate stages have a significant likelihood of regressing performance, so this issue outlines a safe, staged migration pathway to a page-global level-prepass contract. The goal is to handle all dispatching via runtime environment variables so that we can change kernels in stages without impacting existing code (except to temporarily increase build times).
Only after correctness and end-to-end performance evidence is established should the legacy implementation be removed.
Phase 1 — Feature-flagged dual-path rollout
Introduce internal host-only LIBCUDF_PARQUET_LEVEL_PREPASS=<uint32 bitmask>, resolved once per reader_impl read. There should also be a way to control this setting programmatically for structured testing purposes. Then, introduce the new preprocessing pass, which will filter which page types it runs on based on this bitmask. Each PR will add a new supported decode kernel. During this phase, the existing decode kernels will remain as-is, but new versions will be introduced and runtime switching will determine which one to use to avoid impacting performance by introducing device-side selection logic. We will leverage the programmatic setting to support testing both versions of the kernel in all existing parquet tests.
Proposed selector bits (this bitmask was AI-generated, I'll validate these further as I proceed)
| Bit |
Family |
0x001 |
Generic flat |
0x002 |
GENERAL/BSS flat |
0x004 |
Delta flat |
0x008 |
Generic nested |
0x010 |
GENERAL/BSS nested |
0x020 |
Delta nested |
0x040 |
Generic lists |
0x080 |
GENERAL/BSS lists |
0x100 |
Delta lists |
0x1ff |
All families |
An important constraint during this phase: the core structure of the existing decode kernels will not be modified. That means that existing warp-specializations will be maintained, and thread/warp assignments within the kernel will remain the same. For the warp-specialized kernels, warp 0 will simply become a (set of) global memory read(s) to populate the data. Where adaptor code is necessary to plug in the data from the global reads into the existing code (if the form is slightly different), such code will be introduced to minimize diffs, even if it means accepting some performance reductions.
Phase 2 — Ensure performance parity, switch defaults for some period of time, then retire legacy kernels
At this stage, we will compare the legacy selector 0 against the all-family selector 0x1ff. Our goal at this stage is to achieve performance parity across all our microbenchmarks and TPC benchmarks. We must ensure that we validate a wide range of parametrizations accounting for nullability and different degrees of nested data. Given the constraints imposed in phase one, we will not be able to take full advantage of the opportunities afforded by the refactor.
Phase 3 — Optimize and rearchitect prepass consumers
After we have achieved performance parity and switched over to the new kernels, we can begin optimizing each decode kernel under the new sets of constraints. This will involve redesigning these kernels from the ground up and will be parallelizable work. We should start with the highest priority kernels first, which should be decode_page_data_generic and decode_delta_binary, but all of the currently warp-specialized kernels are also candidates for rewriting at this stage. Given that all of them will have an effectively idle warp zero, we should be able to pick up significant performance gains.
Risks and rollback
The selection bitmask should allow us to avoid any meaningful performance regressions on main during the course of the rollout, so the main risk is discovering that at the end of phase 2 the prepass is substantially slower for some use cases and we need to demonstrate that we can speed up individual kernels enough to justify it. That risk can be mitigated by using agents to quickly prototype the full replacement prior to moving forward with this approach. Once we have that prototype in hand, the remaining risk is minimal.
Problem
Our Parquet kernels as currently constituted duplicate tightly coupled level-derived work: validity construction, output-position/rank calculation, nesting progress, andlist-offset state.
decode_page_data_generichas a customized block-wide approach for this, while all the other kernels (GENERAL/BSS/delta binary) rely on thegpuDecodeLevelsdevice function. This device function is designed to operate on a single warp, effectively imposing a warp-specialized pattern on every kernel using it. ThegpuDecodeLevelsfunction dates back to the earliest days of cuIO over 7 years ago, i.e. the age of Volta. The warp-specialized kernels were likely a reasonable choice at the time when GPU bandwidth was much lower and we had fewer tools available to hide the latency.Modern GPUs, however, have much higher HBM bandwidth and larger caches, and newer PTX instructions supporting asynchronous staging of global reads into shared memory allow overlapping data and compute operations in more flexible ways. Conversely, the existing warp-specialized kernels have numerous drawbacks that cannot fundamentally be overcome: long register live ranges lead to high register pressure and reduce occupancy, while cross-warp coordination introduces expensive synchronization barriers. Based on profiling today, the costs of this approach now outweigh the benefits. Moreover, the proliferation in encodings supported by Parquet means that we have a much larger set of kernels that all reproduce this work and are all subject to the warp-specialization constraints. That needs to change.
Proposal
The common work that all decode kernels currently do should be moved to a separate preprocessing kernel that handles the validity and output position calculations. That new kernel will populate this information into global memory for subsequent kernels to consume. To fully take advantage of this change, every decode kernel will then need to be restructured to support proper utilization of the full thread block. This change is a substantial undertaking, and intermediate stages have a significant likelihood of regressing performance, so this issue outlines a safe, staged migration pathway to a page-global level-prepass contract. The goal is to handle all dispatching via runtime environment variables so that we can change kernels in stages without impacting existing code (except to temporarily increase build times).
Only after correctness and end-to-end performance evidence is established should the legacy implementation be removed.
Phase 1 — Feature-flagged dual-path rollout
Introduce internal host-only
LIBCUDF_PARQUET_LEVEL_PREPASS=<uint32 bitmask>, resolved once perreader_implread. There should also be a way to control this setting programmatically for structured testing purposes. Then, introduce the new preprocessing pass, which will filter which page types it runs on based on this bitmask. Each PR will add a new supported decode kernel. During this phase, the existing decode kernels will remain as-is, but new versions will be introduced and runtime switching will determine which one to use to avoid impacting performance by introducing device-side selection logic. We will leverage the programmatic setting to support testing both versions of the kernel in all existing parquet tests.Proposed selector bits (this bitmask was AI-generated, I'll validate these further as I proceed)
0x0010x0020x0040x0080x0100x0200x0400x0800x1000x1ffAn important constraint during this phase: the core structure of the existing decode kernels will not be modified. That means that existing warp-specializations will be maintained, and thread/warp assignments within the kernel will remain the same. For the warp-specialized kernels, warp 0 will simply become a (set of) global memory read(s) to populate the data. Where adaptor code is necessary to plug in the data from the global reads into the existing code (if the form is slightly different), such code will be introduced to minimize diffs, even if it means accepting some performance reductions.
Phase 2 — Ensure performance parity, switch defaults for some period of time, then retire legacy kernels
At this stage, we will compare the legacy selector
0against the all-family selector0x1ff. Our goal at this stage is to achieve performance parity across all our microbenchmarks and TPC benchmarks. We must ensure that we validate a wide range of parametrizations accounting for nullability and different degrees of nested data. Given the constraints imposed in phase one, we will not be able to take full advantage of the opportunities afforded by the refactor.Phase 3 — Optimize and rearchitect prepass consumers
After we have achieved performance parity and switched over to the new kernels, we can begin optimizing each decode kernel under the new sets of constraints. This will involve redesigning these kernels from the ground up and will be parallelizable work. We should start with the highest priority kernels first, which should be decode_page_data_generic and decode_delta_binary, but all of the currently warp-specialized kernels are also candidates for rewriting at this stage. Given that all of them will have an effectively idle warp zero, we should be able to pick up significant performance gains.
Risks and rollback
The selection bitmask should allow us to avoid any meaningful performance regressions on main during the course of the rollout, so the main risk is discovering that at the end of phase 2 the prepass is substantially slower for some use cases and we need to demonstrate that we can speed up individual kernels enough to justify it. That risk can be mitigated by using agents to quickly prototype the full replacement prior to moving forward with this approach. Once we have that prototype in hand, the remaining risk is minimal.