Skip to content

Stack/irc spark int - #5

Open
yili-db wants to merge 6 commits into
mainfrom
stack/irc_spark_int
Open

yili-db wants to merge 6 commits into
mainfrom
stack/irc_spark_int

Conversation

@yili-db

@yili-db yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner

PR Checklist

  • A description of the changes is added to the description of this PR.
  • If there is a related issue, make sure it is linked to this PR.
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added or modified a feature, documentation in docs is updated

Description of changes

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

1 similar comment
@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

2 similar comments
@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

3 similar comments
@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

- Add `IcebergTableReadWriteTest` (+ managed/external subclasses) driving Iceberg tables through Iceberg's own REST `SparkCatalog` against UC's Iceberg REST catalog, on a local `file://` warehouse; reuses `BaseTableReadWriteTest`'s create/read/write matrix.
- Add `DeltaIcebergCrossFormatJoinTest`: one session wired with both the UC connector (Delta) and the Iceberg REST catalog, joining a Delta table against an Iceberg table.
- Refactor `BaseSparkIntegrationTest` so all suites share `createSparkSessionWithCatalogs`:
  - per-catalog `isIcebergCatalog` switch selects UC-connector vs Iceberg wiring; session extensions are derived from the catalog mix
  - extract `ucSingleCatalogConfig` / `icebergCatalogConfig` / `catalogConfKey`
- Make the shared `BaseTableReadWriteTest` tests Iceberg-aware via `testingIceberg()`, so `testTableOperations`, `testListTablesPagination`, and `testHyphenInTableName` run for Iceberg instead of being reimplemented; the non-Iceberg path is unchanged.
- `build.sbt`: add `iceberg-spark-runtime` (Spark 4.0/4.1 only) as a test dep and open `java.base/java.nio` in `Test / javaOptions` for Arrow reads.

Signed-off-by: Yi Li <yi.li@databricks.com>
Signed-off-by: Yi Li <yi.li@databricks.com>
Signed-off-by: Yi Li <yi.li@databricks.com>
Signed-off-by: Yi Li <yi.li@databricks.com>
Signed-off-by: Yi Li <yi.li@databricks.com>
@yili-db
yili-db force-pushed the stack/irc_spark_int branch from 7d475e4 to 2d08b12 Compare October 2, 2026 18:39
@yili-db

yili-db commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db
yili-db force-pushed the stack/irc_spark_int branch 2 times, most recently from bff15b9 to 7aea9e0 Compare October 2, 2026 21:30
Signed-off-by: Yi Li <yi.li@databricks.com>
@yili-db

yili-db commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

/review

1 similar comment
@yili-db

yili-db commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

/review

@yili-db

yili-db commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

/trigger

@yili-db
yili-db force-pushed the main branch 3 times, most recently from de39204 to 622669e Compare October 15, 2026 23:53
@yili-db

yili-db commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant