Skip to content

Add contract-first Iceberg REST read-only support (Polaris-aligned codegen)#457

Open
cbb330 wants to merge 3 commits intolinkedin:mainfrom
cbb330:chbush/iceberg-rest-contract-first-phase1
Open

Add contract-first Iceberg REST read-only support (Polaris-aligned codegen)#457
cbb330 wants to merge 3 commits intolinkedin:mainfrom
cbb330:chbush/iceberg-rest-contract-first-phase1

Conversation

@cbb330
Copy link
Collaborator

@cbb330 cbb330 commented Feb 17, 2026

Summary

Adds Phase 1 Iceberg REST Catalog support using contract-first codegen aligned with the Apache Polaris pattern. The server implements generated OpenAPI interfaces for read-only endpoints (config, list tables, load table, table exists) with comprehensive round-trip integration tests using a real Iceberg RESTCatalog client.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Client-facing API Changes

Adds read-only Iceberg REST Catalog endpoints via IcebergRestCatalogController:

  • GET /v1/config — returns prefix override (iceberg) for route isolation
  • GET /v1/iceberg/namespaces/{namespace}/tables — list tables
  • GET /v1/iceberg/namespaces/{namespace}/tables/{table} — load table metadata
  • HEAD /v1/iceberg/namespaces/{namespace}/tables/{table} — check table existence

All unimplemented endpoints (write, namespace, views, etc.) return 501 via generated defaults.

Internal API Changes

  • Upstream Iceberg REST OpenAPI spec at spec/iceberg-rest-catalog-open-api.yaml
  • importMappings/typeMappings map spec schemas to real Iceberg types following Polaris pattern (no model generation)
  • Post-processing for Iceberg 1.10-only types mapped to Object (compatibility with our 1.5.2 fork)
  • Controller implements generated CatalogApiApi + ConfigurationApiApi interfaces
  • IcebergRestHttpMessageConverter — custom AbstractHttpMessageConverter<RESTResponse> using Iceberg's RESTSerializers (kebab-case JSON) without interfering with existing Jackson converters
  • IcebergRestSerdeConfig — registers the converter via WebMvcConfigurer.extendMessageConverters
  • IcebergRestExceptionHandler — scoped @RestControllerAdvice returning Iceberg ErrorResponse format

New Features

Contract-first Iceberg REST read-only surface, compatible with stock Iceberg RESTCatalog, PyIceberg, and DuckDB clients.

Tests

  • IcebergRestCatalogRoundTripTest (21 tests) — round-trip integration tests using a real Iceberg RESTCatalog client: list tables (4), load table schema/partition/metadata (9), table exists HEAD (3), error handling (2), consistency (1), cross-namespace (2)
  • IcebergRestCatalogControllerTest (6 tests) — MockMvc unit tests for config, list, load, load not found, head exists, head not found

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

All 27 tests pass locally:

IcebergRestCatalogRoundTripTest > 21 tests PASSED
IcebergRestCatalogControllerTest > 6 tests PASSED
BUILD SUCCESSFUL

Manual validation with PyIceberg:

>>> from pyiceberg.catalog import load_catalog
>>> catalog = load_catalog("openhouse", type="rest", uri="http://localhost:8000", token="...")
>>> catalog.list_tables("testdb")
[('testdb', 'test_table')]

Manual validation with DuckDB:

>>> table = catalog.load_table('testdb.test_table')
>>> arrow_table = table.scan().to_arrow()
>>> print(conn.execute('SELECT * FROM arrow_table').fetchdf())
Empty DataFrame
Columns: [id, name, timestampCol]

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

Note: When Iceberg is upgraded to 1.7+, the round-trip tests should extend CatalogTests<RESTCatalog> with write tests disabled, matching the Polaris pattern. CatalogTests is not available in our current 1.5.2 fork.

teamurko
teamurko previously approved these changes Mar 10, 2026
import org.springframework.web.bind.annotation.RestController;

/**
* Read-only Iceberg REST Catalog surface.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does surface mean in this context?

- Replace hand-written OpenAPI spec with upstream Iceberg REST spec
  (iceberg-rest-catalog-open-api.yaml) following the Apache Polaris
  codegen pattern (importMappings/typeMappings, models=false)
- Controller now implements generated CatalogApiApi and
  ConfigurationApiApi interfaces; unimplemented endpoints return 501
- Add prefix routing via /v1/config so REST client uses
  /v1/iceberg/namespaces/... (separate from /v1/databases/...)
- Add IcebergRestHttpMessageConverter for Iceberg REST type
  serialization without interfering with existing Jackson converters
- Add comprehensive round-trip integration tests (21 tests) using
  a real RESTCatalog client: schema fidelity, partitions, metadata,
  list tables, table exists, error handling, consistency
- Update mock controller tests to use prefixed routes
@cbb330 cbb330 changed the title Add contract-first Iceberg REST read-only support with CI compatibility gates Add contract-first Iceberg REST read-only support (Polaris-aligned codegen) Mar 11, 2026
@cbb330
Copy link
Collaborator Author

cbb330 commented Mar 11, 2026

Superseded by #497 — rebased on latest main with contract-first codegen, upstream spec, and comprehensive round-trip tests.

@cbb330 cbb330 closed this Mar 11, 2026
@cbb330 cbb330 reopened this Mar 11, 2026
@cbb330 cbb330 force-pushed the chbush/iceberg-rest-contract-first-phase1 branch from 59a9742 to 2daee5b Compare March 11, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants