Skip to content

Populate data_size for Iceberg V2 scans - #2118

Draft
parthosa wants to merge 2 commits into
NVIDIA:devfrom
parthosa:cudf-tools-2115
Draft

Populate data_size for Iceberg V2 scans #2118
parthosa wants to merge 2 commits into
NVIDIA:devfrom
parthosa:cudf-tools-2115

Conversation

@parthosa

@parthosa parthosa commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #2115

Problem

For Iceberg DataSource V2 BatchScan, data_source_information.csv reports data_size = 0 even when the scan reads a large volume of data. Any downstream consumer that reads data_size (e.g. QualX scan features) is blind to the true scan size for Iceberg tables.

Root cause

Iceberg BatchScan reports its input volume through the SQL-plan metric total data file size (bytes) (v2Custom_...TotalDataFileSize). The data_size column is filled by an I/O-metric helper (IoMetrics) that recognizes data-size metrics by exact name, and the only OSS label it knows is "size of files read". Iceberg's metric name isn't in that allowlist, so DataSourceView filters it out and data_size stays 0. The metric is still present in sql_plan_metrics_for_application.csv — only the data-source view drops it.

Fix

Recognize "total data file size (bytes)" in IoMetrics and route it to dataSize. Because the Iceberg source emits this metric independent of the compute engine, the label is added at the base IoMetricsTrait level, so the OSS, Photon, and Auron helpers all inherit it. File-source scans are unaffected — they emit "size of files read" on distinct nodes, so the two never collide.

Verification

  • Unit tests (IoMetricsSuite, 5 cases): Iceberg metric recognized and routed to dataSize; file-source "size of files read" still routes correctly (regression); Photon/Auron helpers inherit the behavior; unrelated metric names still rejected.
  • End-to-end on an Iceberg V2 event log: after the fix, final-plan Iceberg scans report their true multi-GB data_size (matching the raw total data file size (bytes) values) instead of 0, with no final-plan Iceberg row left at 0; a non-Iceberg log still populates data_size and scan_time for Parquet file-source scans.

Out of scope (separate follow-ups)

  • scan_time / scan_bw for Iceberg: Iceberg BatchScan emits no scan-time metric on either engine ("scan time" is CPU-FileSourceScan-only; GPU splits it into buffer time + GPU decode time). It is genuinely absent rather than an allowlist miss, so it stays 0 and scan_bw degrades safely to 0 in the featurizer — consistent with every GPU scan today.
  • Task-level input_bytesRead: Spark does not populate V2 scan bytes in task input metrics, so input_bytesRead_sum is 0 for Iceberg. That is a distinct path, tracked separately.

Iceberg DataSource V2 BatchScan reports its input volume via the SQL-plan
metric "total data file size (bytes)", but data_source_information.csv
reported data_size=0 for Iceberg scans because the I/O-metric helper only
recognized the file-source label "size of files read".

Recognize the Iceberg data-size metric in IoMetrics and route it to
dataSize. The label is emitted by the Iceberg source independent of the
compute engine, so it is handled at the base IoMetricsTrait level and thus
inherited by the OSS, Photon, and Auron helpers. File-source scans (a
distinct metric name on distinct nodes) are unchanged.

Adds IoMetricsSuite covering recognition, routing, the file-source
regression, engine helpers, and unsupported-name handling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
@github-actions github-actions Bot added the core_tools Scope the core module (scala) label Aug 7, 2026
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
@parthosa parthosa changed the title Populate data_size for Iceberg V2 scans (#2115) Populate data_size for Iceberg V2 scans Aug 7, 2026
@parthosa parthosa self-assigned this Aug 7, 2026
@parthosa
parthosa marked this pull request as draft August 7, 2026 16:38
@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with the narrowly scoped metric mapping supported by focused regression tests.

Summary

  • Recognizes total data file size (bytes) as an I/O metric across OSS, Photon, and Auron helpers.
  • Routes the Iceberg accumulator total into the existing dataSize field.
  • Adds focused tests covering recognition, routing, engine inheritance, existing file-source behavior, and unsupported labels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core_tools Scope the core module (scala)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg V2 scan bytes missing from data source output

2 participants