Report the progress of an active query

[robinskil]

What is missing

Beacon holds no record of an active query. A slow query stays invisible until it ends.

Two questions have no answer today. What runs now? How far along is it?

The query store receives one row at the end of a query. A cancelled query records nothing. See #484.

The data already exists

This issue needs no new counter.

ExecutionPlanMetricsSet updates while the scan runs. The values are live.

AtlasScanMetrics already counts the work of an Atlas scan:

• atlas_datasets_scanned counts the datasets a partition read.
• atlas_datasets_pruned counts the datasets the statistics ruled out.
• atlas_index_rows counts the datasets a pruning index covers.

The plan holds the total dataset count. create_physical_plan makes one entry per dataset.

So a reader can compute progress today. Example: 42 of 190 datasets, and 12 ruled out.

What is absent

A place to read the values from.

No caller reaches the physical plan after run_query returns. The plan lives inside the stream alone.

Steps

1. Add a registry of active queries. Key it on the query id.
2. Hold the start time, the user and the physical plan in each entry.
3. Insert an entry when a query starts.
4. Remove the entry when the stream drops.
5. Add a read path for the registry. A SQL table function matches the existing style.
6. Add a task that logs the slow queries. Log one line per query above an age limit.

Why a timer, and not the read path

A collection holds up to millions of datasets. An event per dataset floods the log.

Sample the registry on a timer instead. Set the age limit in config. 30 seconds is a reasonable default.

A second use

The registry gives cancellation a home.

Beacon holds no cancellation token today. A client cancels a query only by a disconnect. No caller can cancel a query by its id.

Hang a token on each entry. Issues #483 and #484 both need one.

Design overlap

Issue #484 must decide how the query store marks a partial row.

This issue must decide what the registry reports.

Sequence the two together. One decision covers both.

Files

beacon-db/beacon-core/src/runtime.rs

beacon-db/beacon-core/src/metrics.rs

beacon-db/beacon-file-formats/beacon-arrow-atlas/src/datafusion/metrics.rs

Related

#481, #483, #484

Issue #485 adds the query id to the logs. That id keys this registry too.