Rewrite the detector core around DetectorLogic with bounded and unbounded data logics

CHANGELOG.md

@@ -19,14 +19,22 @@
 - `MovableLogic.stop()` is now the abstract `ensure_stopped()`, which must wait until the motion has stopped, and `StandardMovable.stop()` returns once it has. A subclass that still defines `stop()` gets a `DeprecationWarning`, and its `stop()` is used as `ensure_stopped()`. `Motor` now waits for `.DMOV` after writing `.STOP`, allowing its acceleration time to decelerate, and `InstantMotorMock` finishes a move on `.STOP`.
 - `ophyd_async` now needs `typing-extensions>=4.14`, for `KICKED_OFF`. Type-checking a `fly` annotated with `KICKED_OFF` needs pyright 1.1.410 or later.
 - Renamed `MotorFlyCtx` to `MotorPrepareCtx`, `PmacFlyCtx` and `PmacPrepareContext` to `PmacPrepareCtx`, and `SimMotorMoveLogic` to `SimMotorLogic`.
-- A flyer must now be staged before it is prepared: `prepare()` is only allowed after `stage()`, `kickoff()` after `prepare()` and `complete()` after `kickoff()`, and a verb in the wrong state raises `TransitionError` (a `RuntimeError`). Staging an already-staged flyer, even one with a verb running, used to silently reset it and now raises (see below). A fly that fails leaves the flyer staged. `complete()` returns the same status each time it is called after a `kickoff()`, until the next `prepare()` or `stage()`. After `complete()` a flyer can be prepared again without restaging. Stock bluesky plans that stage their devices (`bp.count`, `bp.scan`) keep working; `bp.fly` neither stages nor prepares, so stage the flyer and prepare it before `bp.fly`, for example with `bpp.stage_wrapper`.
+- A flyer must now be staged before it is prepared: `prepare()` is only allowed after `stage()`, `kickoff()` after `prepare()` and `complete()` after `kickoff()`, and a verb in the wrong state raises `TransitionError` (a `RuntimeError`). Staging an already-staged flyer, even one with a verb running, used to silently reset it and now raises (see below). A fly that fails leaves the flyer staged. `complete()` returns the same status each time it is called after a `kickoff()`, until the next `prepare()` or `stage()`. After `complete()` a flyer can be prepared again without restaging, but a prepared flyer cannot be prepared again until its fly has completed (a detector also after a `trigger()`). Stock bluesky plans that stage their devices (`bp.count`, `bp.scan`) keep working; `bp.fly` neither stages nor prepares, so stage the flyer and prepare it before `bp.fly`, for example with `bpp.stage_wrapper`.
 - Staging any already-staged Standard device (`StandardReadable`, `StandardMovable`, `StandardFlyable`, ...) now raises `TransitionError`, as ophyd raises `RedundantStaging`. For example, nesting `bpp.stage_wrapper` around a plan that also stages the same device now fails.
 - `unstage()` now cancels the `prepare()` or fly that is running. `unstage()` waits for it before calling `ensure_stopped()`, then raises any error it raised while being cancelled.
 - `MovableLogic` and `FlyableLogic` now share the base `Logic`, an `ABC`, so a subclass missing an abstract hook (`ensure_stopped`, `prepare_to_fly` or `fly`) raises `TypeError` when it is constructed. See ADR 0022 for the reasoning behind these hooks.
+- `DetectorAcquireLogic.start_acquiring()` and `wait_for_idle()` are replaced by one `acquire()`, an async generator that starts acquiring, yields once armed, then waits until idle. `StandardDetector` enters it in `prepare()` for external triggering and in `kickoff()` or `trigger()` for internal triggering, runs it to the end once the data is written, and closes it at its `yield` if the scan is cancelled, so it must own what it starts (`async with` a status, or cancel a task in `finally`). An `acquire()` that finishes without yielding, or yields more than once, raises `RuntimeError` naming the logic class. `ADAcquireLogic` and `ADContAcqAcquireLogic` no longer have an `acquire_status` attribute, nor `BlobAcquireLogic` a `task`.
+- `unstage()` now cancels a running `StandardDetector.trigger()`, and `kickoff()` raises `TransitionError` while one is running. After a failed `trigger()`, as after any error, `unstage()` and `stage()` the detector before using it again.
+- A `StandardDetector` with no data logic can now be kicked off: its fly has nothing to collect and completes once the acquisition has. `kickoff()` after `trigger()` raises the state machine's `TransitionError` (`cannot go to FLYING from TRIGGERED`).
+- A `BoundedDataLogic` that served the first point of a step scan and returns `None` at a later one raises `RuntimeError`. A data provider carried over to the next `prepare()` publishes from that prepare's baseline, so its first event page or stream datum no longer includes collections written before it.
+- `StreamableDataProvider.make_stream_docs(collections_written, collections_per_event)` is now `make_stream_docs(index)`, the event to publish up to, counted from the origin; the detector keeps the count of what it has published. `StreamResourceDataProvider.last_emitted` is removed, and an override of `make_stream_docs` must take the new arguments and keep no count of its own.
+- `StreamableDataProvider.make_datakeys(collections_per_event)` is now `make_datakeys()`: a provider is given `collections_per_event` when it is made.
+- `DataProvider`, the new base of `StreamableDataProvider`, has an abstract `start_new_resources(origin)`, which makes the provider count its events from collection `origin` with nothing yet published. An out-of-tree `StreamableDataProvider` subclass that does not implement it raises `TypeError` when it is made.
 - `SimMotor` is now a `StandardFlyable`, so like any flyer it must be staged before `prepare()`.
 - `SimMotor.stage()` now calls `SimMotorLogic.ensure_ready()`, which calls `ensure_stopped()` and so stops the simulated motion, waits for it to finish, and sets the setpoint to the current readback. `unstage()` cancels a fly in progress, then calls `ensure_stopped()`. The simulated motion runs in the task of the verb that moves, so `stop()` during a `prepare()` or fly, or `stage()` during a `set()`, cancels that verb, which raises `CancelledError`.
 - The progress of `StandardFlyable.prepare()` now comes from the `WatcherUpdate`s the logic's `prepare_to_fly` yields, replacing the heuristic that watched setpoint writes (#1448).
 - `WatchableAsyncStatus` closes the async generator it drives as soon as it is cancelled or fails, rather than when the generator is collected, and a watcher that raises is logged and ignored rather than failing the status.
+- A `StandardReadable` that a detector suppresses with its data keeps the hints of the signals it drops from `read()` and `describe()`, as the data has the same name. A data logic that hints such a name now makes `hints` raise.
 - `StandardReadable` now keeps one format per registered child, keyed by the child, so registering a child again replaces its format instead of adding a second contribution, and the order of `hints` is the order children were first registered. The private `_describe_config_funcs`, `_read_config_funcs` and `_has_hints` tuples are gone: a subclass that added to them now adds a hook to `_describe_configuration_funcs`, `_read_configuration_funcs` or `_hints_funcs` (see ADR 0023).
 - The deprecated `ConfigSignal` and `HintedSignal` aliases are removed from `ophyd_async.core`; use `StandardReadableFormat.CONFIG_SIGNAL`, `HINTED_SIGNAL` or `HINTED_UNCACHED_SIGNAL`. `StandardReadable.add_readables()` is deprecated in favour of `set_readable_format()` on each device.
 - `store_settings()` writes the readable format changes of the Device tree under the reserved key `<READABLE_FORMATS>`, and `apply_settings()` resets the tree to its class formats before applying them. Settings files stored before this have no such key and leave formats alone. `retrieve_settings()` warns about and drops any other `<...>` key.
@@ -35,9 +43,19 @@
 - The `prepare_single` data logic tier is removed, with `ReadableDataProvider`, `SignalDataProvider` and `adcore.PluginSignalDataLogic`. Register a plugin scalar such as `NDStatsIO.total` with `set_readable_format(signal, StandardReadableFormat.HINTED_SIGNAL)` instead.
 - The areaDetector driver and plugin classes (`ADBaseIO`, `NDROIIO`, `NDStatsIO` and `NDROIStatNIO`) are now `StandardReadable`s that declare their own configuration and read signals, so registering a plugin as `CHILD` of an `AreaDetector` puts them in `read()` and `read_configuration()`.
 - `ADAcquireLogic` sets `WaitForPlugins` on `stage()`, from its new `wait_for_plugins` argument (default on), rather than `AreaDetector` hiding it, so a step scan reads plugin scalars for the frame it has just taken.
+- `StandardDetector` is now a `StandardFlyable` whose `logic` is a `DetectorLogic` holding the trigger, acquire and data logics (see ADR 0026). A detector builds it in a `logic` `@cached_property` rather than passing the logics to `StandardDetector.__init__()`, which takes no arguments, and `add_detector_logics()` is removed. Use `DetectorLogic.with_device()` for an ad-hoc detector. Like any flyer it follows the `stage()`, `prepare()`, `kickoff()`, `complete()` ordering and raises `TransitionError` out of order.
+- Data logics are now either a `BoundedDataLogic`, told the number of collections in the scan, or an `UnboundedDataLogic`, and implement one `start_provider()` that sets up and arms the writer and returns a `StreamableDataProvider`, a `PageableDataProvider` or `None` to sit the stage out (see ADR 0025). `prepare_unbounded()` is replaced by it. Which logics serve is decided at the first `prepare()` after `stage()` and holds until the next `stage()`, and no two serving providers may make the same datakey.
+- `StandardDetector` exposes `collect_asset_docs()` or `collect_pages()`, whichever the serving providers produce, as an instance attribute bound by `prepare()`, so the bluesky run bundler asks for the right one.
+- A registered signal with the same name as a datakey a serving provider makes is left out of `read()`, `describe()` and `hints`, so the provider's data replaces it; `describe()` raises `ValueError` if the two differ in dtype or shape beyond one leading dimension.
+- `TriggerInfo.number_of_events` must now be positive.
+- `ADHDFDataLogic` sizes its HDF chunks from `TriggerInfo.flush_period`: when it and the collection period are both positive, `NumFramesChunks` is set to the frames collected in a flush period, `round(flush_period / collection_period)`, at least 1 (#1309). Otherwise the value on the IOC is used, 0 taken as 1. `NumFramesChunks` is now written back at every `prepare()` that starts the HDF writer, where it used to be written only to change 0 to 1.
+- `adcore.ADWriterFactory` is renamed `ADDataFactory`; the old name still works with a `DeprecationWarning`. Its factories take `always_enable`, default `False`: the plugin's `EnableCallbacks` PV is followed and the plugin sits out the scans of a stage if it is off at the first `prepare()`, so another data logic can serve them. The HDF writer used to be switched on whatever the PV said, so pass `always_enable=True` to keep that.
+- `StreamResourceDataProvider` takes `collections_written` (main's keyword was `collections_written_signal`), the signal counting the collections written, and `collections_per_event`.
+- `StreamableDataProvider.collections_written_signal` is now `DataProvider.collections_written`, an attribute every provider has, the signal counting the collections written.
 
 ### Added
 
+- `DetectorSuppressLogic`, set as `DetectorLogic.suppress`, returns from `hidden_datakeys()` the names of the readables the detector's data hides. `DetectorLogic.prepare_to_fly` asks it at the first `prepare()` after `stage()`, keeps the answer in the context and raises at a later `prepare()` if it differs.
 - `KICKED_OFF`, the sentinel `FlyableLogic.fly` yields once the scan is up to speed.
 - `watch_readback()` reports the progress of a move by watching a `MovableLogic` readback, for use from `FlyableLogic` hooks.
 - `SimMotor.prepare()` and `SimMotor.complete()` report progress to watchers, as `Motor` does.
@@ -49,6 +67,9 @@
 - `StandardReadable.set_readable_format()`, `reset_readable_formats()`, `get_readable_formats()` and `get_readable_format_changes()` (which yield each child with its format), to change how a child contributes to `read()`, `read_configuration()` and `hints` at runtime, the equivalent of `Kind` in ophyd v1 (see "How to change what a Device reports at runtime").
 - `ReadableEntry`, `ReadableFormats`, `walk_readable_formats()`, `apply_readable_formats()` and `READABLE_FORMATS_KEY` in `ophyd_async.core`, and a `readable_formats` field on `Settings`, so formats are stored and retrieved with the signal values.
 - pyright now runs with `reportMatchNotExhaustive`.
+- `BoundedDataLogic`, `UnboundedDataLogic`, `DetectorLogic`, `DataProvider`, `PageableDataProvider` and `EventPageDataProvider` in `ophyd_async.core`, and `ADDataFactory` in `ophyd_async.epics.adcore`.
+- `EventPageDataProvider(refresh=)`, an awaitable callable awaited before each page's arrays are read.
+- `TriggerInfo.flush_period`, how many seconds between flushes of the data being written, default 0 for whatever the hardware is set to. Each data logic's `start_provider()` is given it.
 
 ### Fixed
 
@@ -58,6 +79,7 @@
 - `unstage()` now ends a running `complete()`, such as for the PandA.
 - A `prepare()` that fails or is cancelled now stops the move to the run-up start of a `Motor` or `SimMotor`.
 - A failed `prepare()` no longer leaves the flyer prepared with the failed context, so there is nothing to kick off.
+- `unstage()` closes an acquisition that an externally triggered `StandardDetector.prepare()` armed and nothing ran, rather than leaving it to be closed when garbage collected.
 - `set_and_wait_for_other_value()` cancelled before the first value arrives, such as by `stop()` during a move, no longer leaves its value-watching task running.
 - A signal `get_value()`, `set()` or `trigger()` cancelled just as it finishes no longer swallows the cancellation on Python 3.11.
-- Leaving an `async with AsyncStatus(...)` block, `set_and_wait_for_other_value()` and the sim `BlobAcquireLogic.ensure_stopped()` no longer swallow a cancellation of the caller that arrives while they wait for the task they cancelled: they wait for that task to end, then raise the cancellation.
+- Leaving an `async with AsyncStatus(...)` block, `set_and_wait_for_other_value()` and closing the sim `BlobAcquireLogic.acquire()` no longer swallow a cancellation of the caller that arrives while they wait for the task they cancelled: they wait for that task to end, then raise the cancellation.

docs/conf.py

@@ -157,6 +157,7 @@ def setup(app: application.Sphinx):
     "ophyd_async.core._standard_base._StandardBase",
     "ophyd_async.core._flyable.PrepareT",
     "ophyd_async.core._flyable.CtxT",
+    "ophyd_async.core._detector.ProviderT",
     "ophyd_async.core._utils.P",
     "ophyd_async.core._utils.P.args",
     "ophyd_async.core._utils.P.kwargs",
@@ -193,6 +194,14 @@ def setup(app: application.Sphinx):
     "0.001",
     "1.0",
     "bluesky.protocols.T_co",
+    # TypeVars documented on StandardFlyable, added in #1359
+    "PrepareT",
+    "CtxT",
+    # event_model publishes no intersphinx entry for this TypedDict
+    "event_model.documents.PartialEventPage",
+    # The context DetectorLogic threads from prepare to unstage, private
+    # because nothing outside StandardDetector constructs or inspects one
+    "ophyd_async.core._detector.DetectorCtx",
 ]
 nitpick_ignore = []
 for var in obj_ignore:

docs/explanations/decisions/0012-detector-rewrite.md

@@ -9,6 +9,9 @@ Partly superseded by [ADR 0024](0024-standard-detector-is-a-standard-readable.md
 made `StandardDetector` a `StandardReadable` and removed the `prepare_single` tier along
 with `ReadableDataProvider`.
 
+The data logics and providers were reworked again in
+[ADR 0025](0025-bounded-data-logic-tier.md).
+
 ## Context
 
 The original `StandardDetector` implementation had several architectural issues:

docs/explanations/decisions/0021-single-logic-property.md

@@ -8,6 +8,9 @@ Accepted
 
 Supersedes part of ADR 0017, which introduced `movable_logic`.
 
+`StandardDetector` has since become a `StandardFlyable` whose `logic` is a `DetectorLogic`
+(see [ADR 0026](0026-detector-is-a-flyable.md)).
+
 ## Context
 
 A Device that both moves and flies had to expose the same logic object twice. `Motor`