Skip to content

feat(scan): backport completed planning reports for dd-source - #36

Draft
DerGut wants to merge 5 commits into
alex.bianchi/upstream-main-arrow59from
jannik.steinmann/scan-reports-staging
Draft

DerGut wants to merge 5 commits into
alex.bianchi/upstream-main-arrow59from
jannik.steinmann/scan-reports-staging

Conversation

@DerGut

@DerGut DerGut commented Nov 20, 2026

Copy link
Copy Markdown

Backport Apache iceberg-rust #3708, #3709, #3732, and #3736 onto 9354b42a, the fork revision currently pinned by ddoghq/dd-source. Target the consumed compatibility branch rather than the fork's stale default main.

Adds opt-in, operation-scoped scan counters and asynchronous reporter callbacks after clean planning-stream exhaustion. Partial, failed, and snapshotless planning does not report. Duration includes consumer backpressure; these are planning metrics, not physical read metrics. Preserve Arrow 59, reqwest 0.12, and the fork's reader configuration API; adapt delete-file byte accounting to the fork's u64 accessor. No dependency or serialized task-format changes.

Validation: cargo +nightly-2026-04-16 test -p iceberg --lib --no-default-features --locked: 2,239 passed, 2 existing ignored. Formatting and Clippy (--all-targets --no-default-features --locked -- -D warnings) pass.

For dd-source consumption, pin all three direct Iceberg patches plus both Apache git-source patches to this branch's exact commit, then regenerate Cargo/Bazel crate metadata. A companion dd-source PR will attach a query-scoped reporter for staging validation.

fix: skip manifests with no live entries

Avoid opening manifests whose list counts prove there are no added or existing files. Cover normal and copy-on-write planning with a removed dead-only manifest.

(cherry picked from commit 14e0919)
Expose the reporter contract and read-only scan report types independently
from scan metric collection and delivery.

(cherry picked from commit 1a93bf0)
Instrument normal and copy-on-write planning with private, invocation-scoped
manifest, data-file, delete-file and deletion-vector counters. Count repeated
delete attachments separately from indexed deletes and use DV content sizes
for attached bytes. Cover pruning, empty/error paths and independent calls.

Production collects and discards these counters until a reporting-integration
change consumes them using the result types introduced by PR apache#3709.
PlanningCounts is a test-only assertion snapshot; production integration can
construct the public result types directly. Duration and stable finalization
after early termination remain follow-up work.

(cherry picked from commit 36b6610)
Submit operation-scoped reports asynchronously after clean stream exhaustion, with table-level configuration and scan overrides. Preserve stream items and errors, and omit reports for failed or abandoned plans.

(cherry picked from commit dcf9dfb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant