[EPIC] Minimize version-specific production classes

[gerashegalov]

Goal

Minimize production code that is compiled or packaged as Spark-version-specific when it can safely use a conventional root-jar layout.

This issue replaces the former umbrella PR #14834 as the durable tracker for the gradual unshimming work. It also advances #11926 by reducing the surface that would need generated version-specific package names.

Motivation

The parallel-world classloader remains necessary for code that depends on incompatible Spark APIs or ABI shapes, but treating common code as version-specific by default has costs:

• root-loaded integrations can fail when helpers are only visible through the shim classloader;
• the distribution contains unnecessary parallel-world copies;
• debugging and class ownership are harder to reason about; and
• adding a Spark shim rebuilds and republishes code that is not actually version-dependent.

Approach

• Move stable helper surfaces into narrowly-scoped Maven modules whose classes are safe in the jar root.
• Use binary identity and dependency analysis to verify root safety across supported shims.
• Replace temporary per-class root-promotion entries with module-level ownership or conditional root-safe source selection.
• Keep code version-specific when a supported Spark or integration release lacks the required API, or when commonizing it would require reflection or a broader compatibility design.
• Land the migration as reasonably-sized, independently validated PRs rather than recreating the monolithic change from Minimize the amount of version-specific classes [databricks] #14834.

Completed foundations

• Add common unshim packaging tooling [databricks] #15025 — common unshim packaging tooling
• Add SQL plugin helper module wiring [databricks] #15519 — SQL plugin helper-module wiring
• Optimize dist packaging hot paths #15732 — dist packaging hot-path improvements
• Move FlatBuffers format classes to helper module [databricks] #15739 — root-safe FlatBuffers format helpers
• Move Hadoop file I/O helpers to fileio module [databricks] #15760 — root-safe Hadoop file I/O helpers
• Move columnar base helpers to columnar module [fast-ut] #15761 — root-safe columnar base helpers
• Move Iceberg common helpers to root-safe module [fast-ut][reduced-it] #15762 — root-safe Iceberg common helpers
• Move Iceberg helpers to root-safe module [fast-ut][reduced-it] #15822 — root-safe Iceberg access, file I/O, and base helpers
• Extract Iceberg schema accessors to root-safe module [reduced-it] #15824 — root-safe Iceberg schema accessors
• Move Iceberg write access bridge to root-safe module [fast-ut][reduced-it] #15829 — root-safe Iceberg write-access bridge and pre-merge deployment smoke coverage
• Make Iceberg package-private access root-safe #15905 — root-safe Iceberg package-private access with fail-closed package auditing

Completed milestone: all current Iceberg package-private callers are root-safe

#15905 moved every current cuDF Plugin class that directly accesses an Iceberg package-private class or member to the assembled JAR root. Its package-phase audit now preserves that invariant across every selected Spark/Iceberg runtime world and rejects future callers placed in spark-shared or a versioned shim directory.

This removes the known classloader/package-identity exposure and turns the requirement into a build invariant. Iceberg code that genuinely depends on version-specific public APIs may still remain in its corresponding shim world.

Current priority milestone: eliminate Iceberg classes from spark-shared

• Move independent Iceberg helpers to root-safe module [reduced-it] #15841 — move the next independent Iceberg helper set into the root-safe module
• Finalize Iceberg root and shim class placement [reduced-it] #15944 — finish Iceberg root and shim placement using conditional root-safe sources

#15944 is a draft stacked on #15841. When the stack merges, the assembled distribution for the supported OSS shim matrix should contain no Iceberg classes under spark-shared: portable classes will appear once at the conventional JAR root, while classes with real Spark/Iceberg compatibility differences will remain in the appropriate version-specific directories.

The final step also removes the remaining Iceberg per-class promotion exceptions and handles helpers such as GpuSparkPlanningUtil and GpuBaseReader whose root safety applies only to a subset of supported Spark/Iceberg combinations.

Remaining work

• Merge Move independent Iceberg helpers to root-safe module [reduced-it] #15841 and Finalize Iceberg root and shim class placement [reduced-it] #15944, up-merge the stack as needed, and revalidate the final layout against current main.
• Continue extracting stable SQL plugin and other non-Iceberg helper clusters where module dependencies remain acyclic and narrow.
• Reduce legacy per-class root-promotion and keep-list entries as module ownership replaces them.
• Inventory the remaining production src/main/spark* sources and document or remove unjustified version-specific copies.
• Update shim architecture documentation with the final module ownership and compatibility rules.

Completion criteria

• Stable production helpers no longer live under version-specific source roots or spark-shared solely because of build layout.
• Root-safe module classes appear exactly once at the jar root and do not retain parallel-world copies.
• Remaining version-specific production classes have a concrete Spark/integration API or ABI reason to stay shimmed.
• Temporary per-class promotion lists are minimized and contain documented exceptions only.
• Packaging and compatibility checks pass across supported Spark, Scala, Databricks, Delta Lake, and Iceberg combinations affected by each migration.

Out of scope

• Eliminating the shim layer entirely.
• Commonizing code through reflection solely to force bytecode identity.
• Migrating version-specific test sources unless needed to validate a production-code move.