Repository navigation
feat(writer): add Table::location_generator that follows table properties - #3730
smaheshwar-pltr wants to merge 6 commits into
Conversation
Writers that build a location generator from table metadata had to choose between `DefaultLocationGenerator` and `ObjectStorageLocationGenerator` themselves, or ignore `write.object-storage.enabled`. Add `TableLocationGenerator`, which makes that choice from the table properties, and use it in the copy-on-write rewrite writer. The layout is chosen internally, so more layouts can be added later without breaking callers.
Generated-by: Codex
Generated-by: Codex
Generated-by: Codex
| /// [`write.object-storage.enabled`](crate::spec::TableProperties::PROPERTY_WRITE_OBJECT_STORAGE_ENABLED) | ||
| /// is `true`, and [`DefaultLocationGenerator`] otherwise. | ||
| #[derive(Clone, Debug)] | ||
| pub struct TableLocationGenerator { |
There was a problem hiding this comment.
I wrote up some context in #3729 on why I've gone with this approach but open to change / discussion if folks have thoughts. A new generator here lets us keep compatibility with current client behaviour, but the downside is that DefaultLocationGenerator reads naturally yet may not do what's desired.
There was a problem hiding this comment.
IAlso, Java and Python support custom location providers so that is worth considering too w.r.t. the API design we introduce here. I think the proposed design should be able to support that by e.g. adding Custom(Arc<dyn <ErasedGenerator>) to the Layout enum)
There was a problem hiding this comment.
Yes, I think we should consider custom location provider here. enum seems like an acceptable workaround to me, but I don't want to block the custom location provider support for datafusion. Would appreciate more clarity here!
There was a problem hiding this comment.
(You're right, I'm having a think here on the best shape)
There was a problem hiding this comment.
Thank you - on custom providers, I think they'd come in through the catalog builder like storage factories with the catalog handing those to each Table, not TableMetadata. So I've moved the entry point to Table::location_generator().
To check, I POC-ed what custom providers might look like here on top of this API choice i.e. Table::location_generator() doesn't change meaning e.g. DataFusion picks it up with no changes. (The wiring is LocationGeneratorFactory passed in through CatalogBuilder::with_location_generator_factory)
Separately, I do wonder why location generators aren't trait objects like Storage - i.e. if we were building this from scratch, we could pass Arc<dyn LocationGenerator> around instead with this PR's Table::location_generator() returning that too. #3738 is what that might look like (this is with custom providers on top).
That's breaking so I've kept this PR additive for now, but I am curious for thoughts on that alternative design.
Table::location_generator that follows table properties
Replace the public TableLocationGenerator::new with Table::location_generator, so custom location providers can later be supplied through the catalog without a second entry point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a931e28 to
5dd75a7
Compare
Which issue does this PR close?
What changes are included in this PR?
See the issue description for details. This PR proposes adding
Table::location_generator()to select the default or object-storage layout from table properties, and use it in the copy-on-write writer and writer examples.It's on
Tablerather than a publicTableLocationGenerator::new(&TableMetadata)so that custom location providers can later be supplied through the catalog, like storage factories, without a second entry point. For now, Java'swrite.location-provider.implis ignored.We'd then use that also in Datafusion to respect the object storage property. apache/datafusion-iceberg#121 is how that might look (edit: needs updating with new API).
Are these changes tested?
Yes, with
test_table_location_follows_propertiesandtest_default_location_ignores_object_storage_propertyunit tests incrates/iceberg/src/writer/file_writer/location_generator.rs.AI Disclosure
This PR was written by myself, Codex and Claude Code.