Populate data_size for Iceberg V2 scans - #2118
Draft
parthosa wants to merge 2 commits into
Draft
Conversation
Iceberg DataSource V2 BatchScan reports its input volume via the SQL-plan metric "total data file size (bytes)", but data_source_information.csv reported data_size=0 for Iceberg scans because the I/O-metric helper only recognized the file-source label "size of files read". Recognize the Iceberg data-size metric in IoMetrics and route it to dataSize. The label is emitted by the Iceberg source independent of the compute engine, so it is handled at the base IoMetricsTrait level and thus inherited by the OSS, Photon, and Auron helpers. File-source scans (a distinct metric name on distinct nodes) are unchanged. Adds IoMetricsSuite covering recognition, routing, the file-source regression, engine helpers, and unsupported-name handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
parthosa
marked this pull request as draft
August 7, 2026 16:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2115
Problem
For Iceberg DataSource V2
BatchScan,data_source_information.csvreportsdata_size = 0even when the scan reads a large volume of data. Any downstream consumer that readsdata_size(e.g. QualX scan features) is blind to the true scan size for Iceberg tables.Root cause
Iceberg
BatchScanreports its input volume through the SQL-plan metrictotal data file size (bytes)(v2Custom_...TotalDataFileSize). Thedata_sizecolumn is filled by an I/O-metric helper (IoMetrics) that recognizes data-size metrics by exact name, and the only OSS label it knows is"size of files read". Iceberg's metric name isn't in that allowlist, soDataSourceViewfilters it out anddata_sizestays 0. The metric is still present insql_plan_metrics_for_application.csv— only the data-source view drops it.Fix
Recognize
"total data file size (bytes)"inIoMetricsand route it todataSize. Because the Iceberg source emits this metric independent of the compute engine, the label is added at the baseIoMetricsTraitlevel, so the OSS, Photon, and Auron helpers all inherit it. File-source scans are unaffected — they emit"size of files read"on distinct nodes, so the two never collide.Verification
IoMetricsSuite, 5 cases): Iceberg metric recognized and routed todataSize; file-source"size of files read"still routes correctly (regression); Photon/Auron helpers inherit the behavior; unrelated metric names still rejected.data_size(matching the rawtotal data file size (bytes)values) instead of 0, with no final-plan Iceberg row left at 0; a non-Iceberg log still populatesdata_sizeandscan_timefor Parquet file-source scans.Out of scope (separate follow-ups)
scan_time/scan_bwfor Iceberg: IcebergBatchScanemits no scan-time metric on either engine ("scan time"is CPU-FileSourceScan-only; GPU splits it intobuffer time+GPU decode time). It is genuinely absent rather than an allowlist miss, so it stays 0 andscan_bwdegrades safely to 0 in the featurizer — consistent with every GPU scan today.input_bytesRead: Spark does not populate V2 scan bytes in task input metrics, soinput_bytesRead_sumis 0 for Iceberg. That is a distinct path, tracked separately.