Skip to content

feat(writer): add Table::location_generator that follows table properties - #3730

Open
smaheshwar-pltr wants to merge 6 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator
Open

smaheshwar-pltr wants to merge 6 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator

Conversation

@smaheshwar-pltr

@smaheshwar-pltr smaheshwar-pltr commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

See the issue description for details. This PR proposes adding Table::location_generator() to select the default or object-storage layout from table properties, and use it in the copy-on-write writer and writer examples.

It's on Table rather than a public TableLocationGenerator::new(&TableMetadata) so that custom location providers can later be supplied through the catalog, like storage factories, without a second entry point. For now, Java's write.location-provider.impl is ignored.

We'd then use that also in Datafusion to respect the object storage property. apache/datafusion-iceberg#121 is how that might look (edit: needs updating with new API).

Are these changes tested?

Yes, with test_table_location_follows_properties and test_default_location_ignores_object_storage_property unit tests in crates/iceberg/src/writer/file_writer/location_generator.rs.

AI Disclosure

This PR was written by myself, Codex and Claude Code.

Writers that build a location generator from table metadata had to
choose between `DefaultLocationGenerator` and
`ObjectStorageLocationGenerator` themselves, or ignore
`write.object-storage.enabled`. Add `TableLocationGenerator`, which
makes that choice from the table properties, and use it in the
copy-on-write rewrite writer.

The layout is chosen internally, so more layouts can be added later
without breaking callers.
Generated-by: Codex
/// [`write.object-storage.enabled`](crate::spec::TableProperties::PROPERTY_WRITE_OBJECT_STORAGE_ENABLED)
/// is `true`, and [`DefaultLocationGenerator`] otherwise.
#[derive(Clone, Debug)]
pub struct TableLocationGenerator {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote up some context in #3729 on why I've gone with this approach but open to change / discussion if folks have thoughts. A new generator here lets us keep compatibility with current client behaviour, but the downside is that DefaultLocationGenerator reads naturally yet may not do what's desired.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IAlso, Java and Python support custom location providers so that is worth considering too w.r.t. the API design we introduce here. I think the proposed design should be able to support that by e.g. adding Custom(Arc<dyn <ErasedGenerator>) to the Layout enum)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should consider custom location provider here. enum seems like an acceptable workaround to me, but I don't want to block the custom location provider support for datafusion. Would appreciate more clarity here!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(You're right, I'm having a think here on the best shape)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you - on custom providers, I think they'd come in through the catalog builder like storage factories with the catalog handing those to each Table, not TableMetadata. So I've moved the entry point to Table::location_generator().

To check, I POC-ed what custom providers might look like here on top of this API choice i.e. Table::location_generator() doesn't change meaning e.g. DataFusion picks it up with no changes. (The wiring is LocationGeneratorFactory passed in through CatalogBuilder::with_location_generator_factory)

Separately, I do wonder why location generators aren't trait objects like Storage - i.e. if we were building this from scratch, we could pass Arc<dyn LocationGenerator> around instead with this PR's Table::location_generator() returning that too. #3738 is what that might look like (this is with custom providers on top).

That's breaking so I've kept this PR additive for now, but I am curious for thoughts on that alternative design.

@smaheshwar-pltr smaheshwar-pltr changed the title feat(writer): add a location generator that follows table properties feat(writer): add Table::location_generator that follows table properties Nov 20, 2026
Replace the public TableLocationGenerator::new with
Table::location_generator, so custom location providers can later be
supplied through the catalog without a second entry point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a location generator that follows the table layout properties

2 participants