Move the crawler to the streaming listing

[robinskil]

Problem

The crawler calls ListingFactory::list_datasets. That function collects every object into a vector, then classifies the vector. A store with 2853217 objects holds about 1 GB during one scan.

list_datasets is also the last function on the listing factory that knows about file formats. The factory resolves paths and reads stores. It does not need to know what a Zarr marker is.

Change

Change the call in beacon-db/beacon-core/src/crawler/engine.rs:

1. Resolve the target prefix with ListingFactory::listing.
2. Read the result with stream().
3. Classify each object with the same rule the listing module uses.

Then delete ListingFactory::list_datasets and enrich_with_object_metadata.

Result

The crawler holds one object at a time. The listing factory stops depending on FileFormatFactoryExt.

Notes

Move classify to a shared place first. It sits in beacon-functions/src/listing/provider.rs. The crawler sits in beacon-core.

The crawler still collects the datasets. group_into_tables needs the full set to detect Hive partitions. The saving is the object vector and the separate enrichment pass.

Follows #416.