Remove the index files of a dropped Lance table

[robinskil]

Problem

DROP TABLE leaves the index state of a Lance table in place. A new table with the same name reuses the same URI.

The new table then holds an index reference to a file that does not exist. COMPACT TABLE and UPDATE fail:

LanceError(IO): Object at location lance/beacon/mt_lifecycle.lance/_indices//page_data.lance not found

Evidence

• Run the three tests alone: 3 passed.
• Run the three tests after test_compact_keeps_indexes: 3 failed.

test_compact_keeps_indexes creates a btree index. The fixture then drops the table.

Failed tests

• test_lance_tables.py::test_compact_honours_target_rows_per_fragment
• test_lance_tables.py::test_compact_never_keeps_every_version
• test_lance_tables.py::test_compact_after_update_preserves_values

Task

1. Delete the _indices folder when the server drops a Lance table.
2. Add a Rust test. The test drops a table with an index, creates the table again, and runs a compaction.

Environment

Branch features/reimplement-atlas. Image beacon-integration:local.