Iceberg V2 scan bytes missing from data source output

[parthosa]

Problem

Iceberg DataSource V2 BatchScan reports input bytes as total data file size (bytes). The profiler retains this metric in sql_plan_metrics_for_application.csv, but does not include it in data_source_information.csv. Consequently, data_size can be reported as zero for Iceberg scans.

Root cause

DataSourceView filters SQL plan metrics through the selected I/O-metric helper. The OSS helper recognizes only size of files read as a data-size metric, so it excludes Iceberg's V2 metric before generating the data source output.

This also affects consumers that derive scan-related features from data_size, scan_time, or scan_bw.

Scope

The metric is not lost during event-log parsing: AppSQLPlanAnalyzer records all SQL-plan metrics. The gap is limited to the I/O-metric filtering and data source output path in IoMetrics and DataSourceView.

QualX consumes data_size, scan_time, and derived scan_bw, so those features are also affected when the data source output omits the metric.

Relevant code:

• core/src/main/scala/com/nvidia/spark/rapids/tool/analysis/AppSQLPlanAnalyzer.scala
• core/src/main/scala/com/nvidia/spark/rapids/tool/views/IoMetrics.scala
• core/src/main/scala/com/nvidia/spark/rapids/tool/views/DataSourceView.scala
• user_tools/src/spark_rapids_tools/tools/qualx/featurizers/default.py

Acceptance criteria

• An Iceberg V2 scan populates data_size from total data file size (bytes) in data_source_information.csv.
• Existing file-source scans using size of files read are unchanged.

Related

• Revisit Scan ops for Iceberg #1949 covers Iceberg scan recognition and support decisions.

[eordentlich]

Are the inputBytesRead.* metrics also impacted?

[rishic3]

Are the inputBytesRead.* metrics also impacted?

inputBytesRead metrics are undercounted in Iceberg readers. This is tracked in apache/iceberg#17208.