Skip to content

fix: load each table from the catalog on lookup - #116

Open
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/fresh-table-lookup
Open

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:fix/fresh-table-lookup

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

IcebergSchemaProvider builds each table's provider once, when the catalog is registered, and every query reuses it with the schema the table had then. After another client changes the schema, INSERTs fail (#56) and reads return wrong results or panic (#57).

This PR resolves a table when a query references it, as DataFusion's remote catalog example does (suggested in #24): each query fetches the table from the catalog and uses it only for that query.

What changes are included in this PR?

  • IcebergSchemaProvider keeps only table names. Each table() lookup loads the table and builds a new provider, including for $ metadata tables.
  • Registering the catalog no longer loads every table.
  • Removes the now-unused IcebergCatalogTableProvider::metadata_table.

Table names are still listed once, so tables other clients create or drop stay invisible (rest of #24).

Are there any user-facing changes?

Queries see schema changes made by other clients. Each lookup costs one load_table, on top of the one scan/insert_into already does; a follow-up can have the provider reuse the loaded table. No API changes.

Are these changes tested?

Yes. tests/table_lookup.rs has a session use a table, another client change its schema, and the session use it again. Both tests fail on main with the errors from the issues.

@NoahKusaba
NoahKusaba marked this pull request as ready for review November 18, 2026 14:53
@NoahKusaba
NoahKusaba marked this pull request as draft November 19, 2026 18:51
@NoahKusaba
NoahKusaba force-pushed the fix/fresh-table-lookup branch from 60c87e8 to fc85b58 Compare November 19, 2026 19:10
IcebergSchemaProvider built every table's provider when the catalog was
registered and returned it to every query, so a session kept planning
against a table's old schema after another client changed it. Keep only
the table names and load the table on each lookup.

Closes apache#56, apache#57.
@NoahKusaba
NoahKusaba force-pushed the fix/fresh-table-lookup branch from fc85b58 to c21f917 Compare November 21, 2026 00:16
@NoahKusaba
NoahKusaba marked this pull request as ready for review November 21, 2026 04:22
@NoahKusaba

NoahKusaba commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

I feel good about this change, wanted to re-review after I noticed #24 design write-up. This is a pretty simple fix, and shouldn't be a burden to review.
cc @alexanderbianchi

@alexanderbianchi

alexanderbianchi commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Just took a quick look and 100% it addresses some of the concerns from #24. Will review more in depth in the next day or two. Note my stamp doesn't unblock.

@NoahKusaba

Copy link
Copy Markdown
Contributor Author

Just took a quick look and 100% it addresses some of the concerns from #24. Will review more in depth in the next day or two. Note my stamp doesn't unblock.

Yeah I'll make follow ups to fully address. I think this part can land independently first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants