Repository navigation
Conversation
Add a manual workflow and helper script that publish a selected Apache Spark source ref to the runner's local Maven cache under a deterministic commit-qualified version.
Keep the cache producer helper focused on the generic source-ref implementation so the workflow does not retain the old commit-only path.
murali-db
force-pushed
the
stack/spark-source-cache-consumer
branch
from
May 29, 2026 19:37
0a32bfc to
5e38dc3
Compare
Avoid deriving the Spark source artifact cache key from runner metadata so producer and consumer workflows share an explicit Ubuntu cache namespace.
Keep the source cache workflow generic by resolving default Spark refs from CrossSparkVersions while still allowing manual source-ref overrides. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
murali-db
force-pushed
the
stack/spark-source-cache-consumer
branch
from
June 3, 2026 15:39
7cdd305 to
5acaac5
Compare
murali-db
force-pushed
the
stack/spark-source-cache-consumer
branch
from
June 4, 2026 19:26
5acaac5 to
2b10780
Compare
Add an opt-in Spark source ref overlay for dependency resolution and a non-blocking PR signal that restores prebuilt Spark artifacts from the cache instead of rebuilding them.
Source-built Spark refs replace sparkVersion.value with a commit-qualified Maven coordinate, so select shim directories from the configured compatibility spec instead of looking up that artifact version in ALL_SPECS.
When a Spark source ref is selected, include Maven local in the Spark-aware resolvers so restored commit-qualified Spark artifacts are visible to SBT.
Allow the source-ref workflow to opt in to Maven local while SBT repository override is enabled, and verify the restored Spark Maven artifacts before invoking SBT.
Only expose Maven local to source-built Spark runs, and make a missing source Spark cache fail once while the remaining Spark 4.2 shards skip until the cache is warmed.
Remove the standalone source Spark workflow so the non-blocking Spark 4.2 lane remains the only cache consumer. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Preserve the source-built Spark helper references and shared property/env lookup after restacking the consumer branch. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
murali-db
force-pushed
the
stack/spark-source-cache-consumer
branch
from
June 9, 2026 17:33
2b10780 to
279bfa8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR
PR #35 has been merged into this fork's
master. Use this link to review the remaining consumer changes in the fork.Stack:
Review SHAs:
84ec2fc0e40341afab4d77d89852d0ee6ee62c382899dea4712056d5258179abe18aeb36f1ce18a4Summary
Delta Sparkmatrix lane for Spark 4.2 instead of adding a separate source-ref workflow.sparkCommitartifact-version overlay separate from the normal Spark compatibility matrix.Test plan