Skip to content

[Build] Test Delta against cached Spark source refs - #36

Open
murali-db wants to merge 16 commits into
masterfrom
stack/spark-source-cache-consumer
Open

murali-db wants to merge 16 commits into
masterfrom
stack/spark-source-cache-consumer

Conversation

@murali-db

@murali-db murali-db commented May 29, 2026 •

Copy link
Copy Markdown
Owner

Stacked PR

PR #35 has been merged into this fork's master. Use this link to review the remaining consumer changes in the fork.

Stack:

Review SHAs:

  • Fork base: 84ec2fc0e40341afab4d77d89852d0ee6ee62c38
  • Consumer head: 2899dea4712056d5258179abe18aeb36f1ce18a4

Summary

  • Reuses the existing non-blocking Delta Spark matrix lane for Spark 4.2 instead of adding a separate source-ref workflow.
  • Restores cached Apache Spark artifacts for the checked-in Spark 4.2 source ref before the Spark 4.2 shards run.
  • Makes Maven local a fallback repository for SBT repository-override builds, then verifies restored Spark Maven POMs before invoking tests.
  • Keeps the sparkCommit artifact-version overlay separate from the normal Spark compatibility matrix.

Test plan

murali-db added 2 commits May 29, 2026 09:48
Add a manual workflow and helper script that publish a selected Apache Spark source ref to the runner's local Maven cache under a deterministic commit-qualified version.
Keep the cache producer helper focused on the generic source-ref implementation so the workflow does not retain the old commit-only path.
murali-db added 2 commits June 2, 2026 07:45
Avoid deriving the Spark source artifact cache key from runner metadata so producer and consumer workflows share an explicit Ubuntu cache namespace.
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 7cdd305 to 5acaac5 Compare June 3, 2026 15:39
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 5acaac5 to 2b10780 Compare June 4, 2026 19:26
murali-db added 10 commits June 9, 2026 10:29
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/spark-source-cache-consumer branch from 2b10780 to 279bfa8 Compare June 9, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant