diff --git a/CHANGELOG.md b/CHANGELOG.md index 63d284885b..5071fa5038 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,14 +19,22 @@ - `MovableLogic.stop()` is now the abstract `ensure_stopped()`, which must wait until the motion has stopped, and `StandardMovable.stop()` returns once it has. A subclass that still defines `stop()` gets a `DeprecationWarning`, and its `stop()` is used as `ensure_stopped()`. `Motor` now waits for `.DMOV` after writing `.STOP`, allowing its acceleration time to decelerate, and `InstantMotorMock` finishes a move on `.STOP`. - `ophyd_async` now needs `typing-extensions>=4.14`, for `KICKED_OFF`. Type-checking a `fly` annotated with `KICKED_OFF` needs pyright 1.1.410 or later. - Renamed `MotorFlyCtx` to `MotorPrepareCtx`, `PmacFlyCtx` and `PmacPrepareContext` to `PmacPrepareCtx`, and `SimMotorMoveLogic` to `SimMotorLogic`. -- A flyer must now be staged before it is prepared: `prepare()` is only allowed after `stage()`, `kickoff()` after `prepare()` and `complete()` after `kickoff()`, and a verb in the wrong state raises `TransitionError` (a `RuntimeError`). Staging an already-staged flyer, even one with a verb running, used to silently reset it and now raises (see below). A fly that fails leaves the flyer staged. `complete()` returns the same status each time it is called after a `kickoff()`, until the next `prepare()` or `stage()`. After `complete()` a flyer can be prepared again without restaging. Stock bluesky plans that stage their devices (`bp.count`, `bp.scan`) keep working; `bp.fly` neither stages nor prepares, so stage the flyer and prepare it before `bp.fly`, for example with `bpp.stage_wrapper`. +- A flyer must now be staged before it is prepared: `prepare()` is only allowed after `stage()`, `kickoff()` after `prepare()` and `complete()` after `kickoff()`, and a verb in the wrong state raises `TransitionError` (a `RuntimeError`). Staging an already-staged flyer, even one with a verb running, used to silently reset it and now raises (see below). A fly that fails leaves the flyer staged. `complete()` returns the same status each time it is called after a `kickoff()`, until the next `prepare()` or `stage()`. After `complete()` a flyer can be prepared again without restaging, but a prepared flyer cannot be prepared again until its fly has completed (a detector also after a `trigger()`). Stock bluesky plans that stage their devices (`bp.count`, `bp.scan`) keep working; `bp.fly` neither stages nor prepares, so stage the flyer and prepare it before `bp.fly`, for example with `bpp.stage_wrapper`. - Staging any already-staged Standard device (`StandardReadable`, `StandardMovable`, `StandardFlyable`, ...) now raises `TransitionError`, as ophyd raises `RedundantStaging`. For example, nesting `bpp.stage_wrapper` around a plan that also stages the same device now fails. - `unstage()` now cancels the `prepare()` or fly that is running. `unstage()` waits for it before calling `ensure_stopped()`, then raises any error it raised while being cancelled. - `MovableLogic` and `FlyableLogic` now share the base `Logic`, an `ABC`, so a subclass missing an abstract hook (`ensure_stopped`, `prepare_to_fly` or `fly`) raises `TypeError` when it is constructed. See ADR 0022 for the reasoning behind these hooks. +- `DetectorAcquireLogic.start_acquiring()` and `wait_for_idle()` are replaced by one `acquire()`, an async generator that starts acquiring, yields once armed, then waits until idle. `StandardDetector` enters it in `prepare()` for external triggering and in `kickoff()` or `trigger()` for internal triggering, runs it to the end once the data is written, and closes it at its `yield` if the scan is cancelled, so it must own what it starts (`async with` a status, or cancel a task in `finally`). An `acquire()` that finishes without yielding, or yields more than once, raises `RuntimeError` naming the logic class. `ADAcquireLogic` and `ADContAcqAcquireLogic` no longer have an `acquire_status` attribute, nor `BlobAcquireLogic` a `task`. +- `unstage()` now cancels a running `StandardDetector.trigger()`, and `kickoff()` raises `TransitionError` while one is running. After a failed `trigger()`, as after any error, `unstage()` and `stage()` the detector before using it again. +- A `StandardDetector` with no data logic can now be kicked off: its fly has nothing to collect and completes once the acquisition has. `kickoff()` after `trigger()` raises the state machine's `TransitionError` (`cannot go to FLYING from TRIGGERED`). +- A `BoundedDataLogic` that served the first point of a step scan and returns `None` at a later one raises `RuntimeError`. A data provider carried over to the next `prepare()` publishes from that prepare's baseline, so its first event page or stream datum no longer includes collections written before it. +- `StreamableDataProvider.make_stream_docs(collections_written, collections_per_event)` is now `make_stream_docs(index)`, the event to publish up to, counted from the origin; the detector keeps the count of what it has published. `StreamResourceDataProvider.last_emitted` is removed, and an override of `make_stream_docs` must take the new arguments and keep no count of its own. +- `StreamableDataProvider.make_datakeys(collections_per_event)` is now `make_datakeys()`: a provider is given `collections_per_event` when it is made. +- `DataProvider`, the new base of `StreamableDataProvider`, has an abstract `start_new_resources(origin)`, which makes the provider count its events from collection `origin` with nothing yet published. An out-of-tree `StreamableDataProvider` subclass that does not implement it raises `TypeError` when it is made. - `SimMotor` is now a `StandardFlyable`, so like any flyer it must be staged before `prepare()`. - `SimMotor.stage()` now calls `SimMotorLogic.ensure_ready()`, which calls `ensure_stopped()` and so stops the simulated motion, waits for it to finish, and sets the setpoint to the current readback. `unstage()` cancels a fly in progress, then calls `ensure_stopped()`. The simulated motion runs in the task of the verb that moves, so `stop()` during a `prepare()` or fly, or `stage()` during a `set()`, cancels that verb, which raises `CancelledError`. - The progress of `StandardFlyable.prepare()` now comes from the `WatcherUpdate`s the logic's `prepare_to_fly` yields, replacing the heuristic that watched setpoint writes (#1448). - `WatchableAsyncStatus` closes the async generator it drives as soon as it is cancelled or fails, rather than when the generator is collected, and a watcher that raises is logged and ignored rather than failing the status. +- A `StandardReadable` that a detector suppresses with its data keeps the hints of the signals it drops from `read()` and `describe()`, as the data has the same name. A data logic that hints such a name now makes `hints` raise. - `StandardReadable` now keeps one format per registered child, keyed by the child, so registering a child again replaces its format instead of adding a second contribution, and the order of `hints` is the order children were first registered. The private `_describe_config_funcs`, `_read_config_funcs` and `_has_hints` tuples are gone: a subclass that added to them now adds a hook to `_describe_configuration_funcs`, `_read_configuration_funcs` or `_hints_funcs` (see ADR 0023). - The deprecated `ConfigSignal` and `HintedSignal` aliases are removed from `ophyd_async.core`; use `StandardReadableFormat.CONFIG_SIGNAL`, `HINTED_SIGNAL` or `HINTED_UNCACHED_SIGNAL`. `StandardReadable.add_readables()` is deprecated in favour of `set_readable_format()` on each device. - `store_settings()` writes the readable format changes of the Device tree under the reserved key ``, and `apply_settings()` resets the tree to its class formats before applying them. Settings files stored before this have no such key and leave formats alone. `retrieve_settings()` warns about and drops any other `<...>` key. @@ -35,9 +43,19 @@ - The `prepare_single` data logic tier is removed, with `ReadableDataProvider`, `SignalDataProvider` and `adcore.PluginSignalDataLogic`. Register a plugin scalar such as `NDStatsIO.total` with `set_readable_format(signal, StandardReadableFormat.HINTED_SIGNAL)` instead. - The areaDetector driver and plugin classes (`ADBaseIO`, `NDROIIO`, `NDStatsIO` and `NDROIStatNIO`) are now `StandardReadable`s that declare their own configuration and read signals, so registering a plugin as `CHILD` of an `AreaDetector` puts them in `read()` and `read_configuration()`. - `ADAcquireLogic` sets `WaitForPlugins` on `stage()`, from its new `wait_for_plugins` argument (default on), rather than `AreaDetector` hiding it, so a step scan reads plugin scalars for the frame it has just taken. +- `StandardDetector` is now a `StandardFlyable` whose `logic` is a `DetectorLogic` holding the trigger, acquire and data logics (see ADR 0026). A detector builds it in a `logic` `@cached_property` rather than passing the logics to `StandardDetector.__init__()`, which takes no arguments, and `add_detector_logics()` is removed. Use `DetectorLogic.with_device()` for an ad-hoc detector. Like any flyer it follows the `stage()`, `prepare()`, `kickoff()`, `complete()` ordering and raises `TransitionError` out of order. +- Data logics are now either a `BoundedDataLogic`, told the number of collections in the scan, or an `UnboundedDataLogic`, and implement one `start_provider()` that sets up and arms the writer and returns a `StreamableDataProvider`, a `PageableDataProvider` or `None` to sit the stage out (see ADR 0025). `prepare_unbounded()` is replaced by it. Which logics serve is decided at the first `prepare()` after `stage()` and holds until the next `stage()`, and no two serving providers may make the same datakey. +- `StandardDetector` exposes `collect_asset_docs()` or `collect_pages()`, whichever the serving providers produce, as an instance attribute bound by `prepare()`, so the bluesky run bundler asks for the right one. +- A registered signal with the same name as a datakey a serving provider makes is left out of `read()`, `describe()` and `hints`, so the provider's data replaces it; `describe()` raises `ValueError` if the two differ in dtype or shape beyond one leading dimension. +- `TriggerInfo.number_of_events` must now be positive. +- `ADHDFDataLogic` sizes its HDF chunks from `TriggerInfo.flush_period`: when it and the collection period are both positive, `NumFramesChunks` is set to the frames collected in a flush period, `round(flush_period / collection_period)`, at least 1 (#1309). Otherwise the value on the IOC is used, 0 taken as 1. `NumFramesChunks` is now written back at every `prepare()` that starts the HDF writer, where it used to be written only to change 0 to 1. +- `adcore.ADWriterFactory` is renamed `ADDataFactory`; the old name still works with a `DeprecationWarning`. Its factories take `always_enable`, default `False`: the plugin's `EnableCallbacks` PV is followed and the plugin sits out the scans of a stage if it is off at the first `prepare()`, so another data logic can serve them. The HDF writer used to be switched on whatever the PV said, so pass `always_enable=True` to keep that. +- `StreamResourceDataProvider` takes `collections_written` (main's keyword was `collections_written_signal`), the signal counting the collections written, and `collections_per_event`. +- `StreamableDataProvider.collections_written_signal` is now `DataProvider.collections_written`, an attribute every provider has, the signal counting the collections written. ### Added +- `DetectorSuppressLogic`, set as `DetectorLogic.suppress`, returns from `hidden_datakeys()` the names of the readables the detector's data hides. `DetectorLogic.prepare_to_fly` asks it at the first `prepare()` after `stage()`, keeps the answer in the context and raises at a later `prepare()` if it differs. - `KICKED_OFF`, the sentinel `FlyableLogic.fly` yields once the scan is up to speed. - `watch_readback()` reports the progress of a move by watching a `MovableLogic` readback, for use from `FlyableLogic` hooks. - `SimMotor.prepare()` and `SimMotor.complete()` report progress to watchers, as `Motor` does. @@ -49,6 +67,9 @@ - `StandardReadable.set_readable_format()`, `reset_readable_formats()`, `get_readable_formats()` and `get_readable_format_changes()` (which yield each child with its format), to change how a child contributes to `read()`, `read_configuration()` and `hints` at runtime, the equivalent of `Kind` in ophyd v1 (see "How to change what a Device reports at runtime"). - `ReadableEntry`, `ReadableFormats`, `walk_readable_formats()`, `apply_readable_formats()` and `READABLE_FORMATS_KEY` in `ophyd_async.core`, and a `readable_formats` field on `Settings`, so formats are stored and retrieved with the signal values. - pyright now runs with `reportMatchNotExhaustive`. +- `BoundedDataLogic`, `UnboundedDataLogic`, `DetectorLogic`, `DataProvider`, `PageableDataProvider` and `EventPageDataProvider` in `ophyd_async.core`, and `ADDataFactory` in `ophyd_async.epics.adcore`. +- `EventPageDataProvider(refresh=)`, an awaitable callable awaited before each page's arrays are read. +- `TriggerInfo.flush_period`, how many seconds between flushes of the data being written, default 0 for whatever the hardware is set to. Each data logic's `start_provider()` is given it. ### Fixed @@ -58,6 +79,7 @@ - `unstage()` now ends a running `complete()`, such as for the PandA. - A `prepare()` that fails or is cancelled now stops the move to the run-up start of a `Motor` or `SimMotor`. - A failed `prepare()` no longer leaves the flyer prepared with the failed context, so there is nothing to kick off. +- `unstage()` closes an acquisition that an externally triggered `StandardDetector.prepare()` armed and nothing ran, rather than leaving it to be closed when garbage collected. - `set_and_wait_for_other_value()` cancelled before the first value arrives, such as by `stop()` during a move, no longer leaves its value-watching task running. - A signal `get_value()`, `set()` or `trigger()` cancelled just as it finishes no longer swallows the cancellation on Python 3.11. -- Leaving an `async with AsyncStatus(...)` block, `set_and_wait_for_other_value()` and the sim `BlobAcquireLogic.ensure_stopped()` no longer swallow a cancellation of the caller that arrives while they wait for the task they cancelled: they wait for that task to end, then raise the cancellation. +- Leaving an `async with AsyncStatus(...)` block, `set_and_wait_for_other_value()` and closing the sim `BlobAcquireLogic.acquire()` no longer swallow a cancellation of the caller that arrives while they wait for the task they cancelled: they wait for that task to end, then raise the cancellation. diff --git a/docs/conf.py b/docs/conf.py index 695b7f8ea4..3d5370cd7b 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -157,6 +157,7 @@ def setup(app: application.Sphinx): "ophyd_async.core._standard_base._StandardBase", "ophyd_async.core._flyable.PrepareT", "ophyd_async.core._flyable.CtxT", + "ophyd_async.core._detector.ProviderT", "ophyd_async.core._utils.P", "ophyd_async.core._utils.P.args", "ophyd_async.core._utils.P.kwargs", @@ -193,6 +194,14 @@ def setup(app: application.Sphinx): "0.001", "1.0", "bluesky.protocols.T_co", + # TypeVars documented on StandardFlyable, added in #1359 + "PrepareT", + "CtxT", + # event_model publishes no intersphinx entry for this TypedDict + "event_model.documents.PartialEventPage", + # The context DetectorLogic threads from prepare to unstage, private + # because nothing outside StandardDetector constructs or inspects one + "ophyd_async.core._detector.DetectorCtx", ] nitpick_ignore = [] for var in obj_ignore: diff --git a/docs/explanations/decisions/0012-detector-rewrite.md b/docs/explanations/decisions/0012-detector-rewrite.md index f1cc30a79a..8b0881a0f9 100644 --- a/docs/explanations/decisions/0012-detector-rewrite.md +++ b/docs/explanations/decisions/0012-detector-rewrite.md @@ -9,6 +9,9 @@ Partly superseded by [ADR 0024](0024-standard-detector-is-a-standard-readable.md made `StandardDetector` a `StandardReadable` and removed the `prepare_single` tier along with `ReadableDataProvider`. +The data logics and providers were reworked again in +[ADR 0025](0025-bounded-data-logic-tier.md). + ## Context The original `StandardDetector` implementation had several architectural issues: diff --git a/docs/explanations/decisions/0021-single-logic-property.md b/docs/explanations/decisions/0021-single-logic-property.md index 028ab0449b..8cbe0e5cd3 100644 --- a/docs/explanations/decisions/0021-single-logic-property.md +++ b/docs/explanations/decisions/0021-single-logic-property.md @@ -8,6 +8,9 @@ Accepted Supersedes part of ADR 0017, which introduced `movable_logic`. +`StandardDetector` has since become a `StandardFlyable` whose `logic` is a `DetectorLogic` +(see [ADR 0026](0026-detector-is-a-flyable.md)). + ## Context A Device that both moves and flies had to expose the same logic object twice. `Motor` diff --git a/docs/explanations/decisions/0025-bounded-data-logic-tier.md b/docs/explanations/decisions/0025-bounded-data-logic-tier.md new file mode 100644 index 0000000000..757811907b --- /dev/null +++ b/docs/explanations/decisions/0025-bounded-data-logic-tier.md @@ -0,0 +1,140 @@ +# 25. Add a bounded `DetectorDataLogic` tier + +Date: 2026-07-17 + +## Status + +Accepted + +## Context + +[ADR 0012](0012-detector-rewrite.md) split data production out into `DetectorDataLogic`. +After [ADR 0024](0024-standard-detector-is-a-standard-readable.md) removed the +single-event tier, one tier was left: a logic prepared for an unbounded number of +collections, publishing stream assets as it wrote. Four open issues bottom out in two +gaps. + +**The data logic is not told enough at prepare time.** Issue #1309 wants HDF chunks sized +from the scan: an Xspress at 400 Hz flushing at 2 Hz wants 200 frames per chunk. An +unbounded logic was told nothing about timing, so chunks were read back from the IOC or +hardcoded. What is needed is the frame *period*, since the requirement is a flush *rate*, +and it is known in step scans too. Rejected: a chunk size on a `TriggerInfo` subclass (as +Malcolm did) or computed by the trigger logic puts a data concern in the trigger path, +against [ADR 0012](0012-detector-rewrite.md)'s rule that a logic is handed only what it +needs. + +**Detectors that produce data but cannot write files.** Issues #1248 and #888 (a duplicate) +come from scalers and electrometers, which want `StandardDetector` but must emit event +pages, and `StandardDetector` inherited `WritesStreamAssets` unconditionally. + +**These are the same gap.** Both own a finite buffer that must be sized before acquisition, +so must be told *how many* collections to expect (a scaler's MCA length, a stats plugin's +`TSNumPoints`), and a finite buffer is naturally read as pages. So one new tier serves both, +and #1309's period rides along. It also lets a detector with no file writer publish a stats +time series (`TSTotal`) in a fly scan, where only the scalar was modelled. + +## Decision + +### One method that starts and describes + +A logic inherits one of `BoundedDataLogic` and `UnboundedDataLogic`, as whether a source +must be told how many collections to take is a property of its type. Its one abstract +`start_provider` arms the writer and returns a provider describing what it writes. The +unbounded logics start first; a bounded one is also told the datakeys they make and the +scan's `num_collections`, publishes the rest, or returns `None` if none are left. No two +providers may make the same datakey. + +Rejected: + +- **Describe, then start**, so the detector could choose providers before anything was + written. The provider had to carry what `start` needed, and Odin (frame shape known once + the file processor runs) and the PandA (`DATASETS` empty until capture) cannot describe + before writing. Telling a bounded logic what the unbounded providers make removes the + choice. +- **One `prepare_*` per tier**: it armed hardware to find out what it would get. + +Both kinds get the period per collection, not per exposure, because a writer after a chain +that averages exposures sees one frame per collection. + +### The tier is the type of the provider returned + +There is no declaration of tier: a logic returning a `StreamableDataProvider` publishes +stream assets, one returning a `PageableDataProvider` event pages, and one may return +either. `None` means "not this stage", such as a plugin switched off, and is not warned +about (#1364): a detector may carry more logics than one scan uses. + +A pageable provider's readings are derived from its pages rather than a second code path. + +### The provider holds the scan's view of the hardware + +A provider is given the `collections_written` signal it counts by, as it already was its +`flush_signal`. The split is by phase: the logic owns the lifecycle, and the provider what +one scan reads back. + +### The period is resolved in `prepare()` + +A `livetime` of 0 means "whatever is set on the detector", so `prepare()` fills it from +`default_trigger_info()` *after* preparing the trigger logic, when the hardware holds the +real values. That reads hardware state, so like the implicit prepare in `trigger()` it is +opt-in (`OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES`); otherwise a period of 0 means "use what +is on the hardware". + +### HDF chunks are sized from the flush period + +A chunk holds one flush period of frames, so a slow detector flushes small chunks and a fast +one packs many. + +### Unbounded providers carry over, bounded providers are re-made + +- An unbounded provider carries over to the next `prepare()` of a stage, so a step scan does + not reopen its file per point and a counting source is not restarted. It starts a new + `StreamResource` in the same file, as a run cannot refer to an earlier run's. +- It carries over only while the name, `collections_per_event`, period and flush period are + unchanged, as they are baked into a `StreamResource` and `num_frames_chunks` can only be + set before `capture`. Otherwise it is stopped before its logic starts again. +- It also carries over only if every unbounded provider has written the same whole number of + events. A `StreamDatum` indexes events in the file, so frames that are not a whole event, + or that one file has and another lacks, would point the next run's events at the wrong + frames. Starting again gives each a fresh count. +- Whether a logic serves is decided at the first `prepare()` of a stage, as one asked later + would count from 0 while a carried provider counts on. +- A bounded logic is restarted at every `prepare()` and every `trigger()` after the first + since it, never at `kickoff()`: the first point is armed by its `prepare()`, and re-arming + it would put a spurious stop/start on the arming control. One that served the first point + and returns `None` later raises, as the scan's datakeys are fixed. Each arm gives a new + `StreamResource`, so a logic wanting one per scan should be unbounded. + +### The progress baseline is read live, and erasing belongs to the data provider + +`trigger()` and `kickoff()` read every provider's count after re-arming. Whoever resets a +counter does it in `start_provider` and waits until it reads 0, so the baseline is never +stale. So a control that both clears the buffer and starts counting, like a MeasComp +CTR-08's `EraseStart`, is fired by the data logic, before `acquire()` arms; the acquire +logic keeps the stop control. The logics share one IO device: the split is of concerns, not +of hardware. + +### Logic objects fill exactly one role + +`DetectorLogic` takes each role by keyword, so an object fills only the role it is passed +as. + +### A detector produces stream assets or event pages, never both + +The bundler collects one or the other from a device, so a detector whose providers would +make both raises (the exception is in [ADR 0026](0026-detector-is-a-flyable.md)). The kind +is decided per prepare, so one detector class can be either, depending on its arguments. +Changing kind between runs is documented rather than enforced, as a Device cannot see run +boundaries. + +## Consequences + +- Out-of-tree data logics must be rewritten onto `start_provider`; there is no shim. +- `default_trigger_info()` should return the current `livetime` and `deadtime`, or the HDF + writer uses the chunk size set on the IOC. +- Detectors that cannot write files become expressible, closing #1248 and #888. +- The areaDetector file writers keep `always_enable`: `True` means the logic owns the + writer's `EnableCallbacks`, `False` (the default) that the user does, read at the first + `prepare()` of a stage. Detectors that always enabled the writer, as before, pass `True`. +- A device whose exposure count and buffer size are one PV has it written by both the + trigger and data logics, which agree only when `exposures_per_collection` is 1: latent, as + such devices do not average exposures. diff --git a/docs/explanations/decisions/0026-detector-is-a-flyable.md b/docs/explanations/decisions/0026-detector-is-a-flyable.md new file mode 100644 index 0000000000..7d8f3cbe91 --- /dev/null +++ b/docs/explanations/decisions/0026-detector-is-a-flyable.md @@ -0,0 +1,156 @@ +# 26. StandardDetector is a StandardFlyable + +Date: 2026-08-28 + +## Status + +Accepted + +## Context + +`StandardDetector` re-implemented `prepare`, `kickoff` and `complete` and threaded its own +state between them, while `StandardFlyable` already did that for every other flyer. The +two drifted apart: the detector guarded its stages by checking a context for `None`, so a +verb called too early failed with `AttributeError: 'NoneType' object has no attribute ...`. + +`StandardFlyable` requires a single `FlyableLogic` in a `logic` property +([ADR 0021](0021-single-logic-property.md)), and a detector had three logic objects +registered by `add_detector_logics()`. + +## Decision + +### One `DetectorLogic` holds the trigger, acquire and data logics + +`DetectorLogic` is what `logic` returns, and the orchestration (preparing the trigger +logic, making the data providers, arming, waiting) moves into it. It is a frozen dataclass +with no back-reference to its Device, so [ADR 0017](0017-standard-movable.md) holds +unqualified. + +```python +class MyDetector(StandardDetector): + def __init__(self, prefix: str, name: str = "") -> None: + self.driver = MyDriverIO(prefix) + super().__init__(name=name) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=MyTriggerLogic(self.driver), + acquire=MyAcquireLogic(self.driver), + ) +``` + +- `add_detector_logics()` is removed, not kept as a forwarder: a detector's logics are fixed + for its lifetime, and two ways to wire one Device is one too many. +- `StandardDetector` takes no `__init__` arguments, as they would interfere with multiple + inheritance. `logic` is an `abstract_cached_property`; assigning `self.logic` in + `__init__` cannot replace it, because `ABCMeta` checks abstract methods before `__init__` + runs. `DetectorLogic.with_device()` builds an ad-hoc detector, like any flyer's. +- Datakeys are named after the Device, whose name only exists once `init_devices` calls + `set_name`, so `StandardFlyable` passes the name to `prepare_to_fly`. + +### The Device binds the collect verb at `prepare()` + +A detector makes stream assets or event pages, never both +([ADR 0025](0025-bounded-data-logic-tier.md)). The bluesky bundler picks which to call +from the *static* type, so declaring both would silently drop the pages, and declaring +neither could not fly. So `StandardDetector.prepare()` overrides the flyer's and, when it +succeeds, binds the verb `DetectorLogic.produces` names as an instance attribute and +removes the other. A step scan restarts its bounded providers at each point, but they must +make the same datakeys, so it keeps the verb it was prepared with. + +This is a bundler limitation: a device knows what it produced, and deciding from that is a +few lines in `RunBundler.collect`. Once released, both verbs become plain methods and the +override goes. + +### Pageable and streamable data mix when the stream assets cover them + +[ADR 0025](0025-bounded-data-logic-tier.md) made carrying both kinds of data logic an +error, but it is useful: an areaDetector stats total can go into the HDF file as an +NDAttribute *and* be read from the plugin's buffer. So: + +- A bounded logic leaves out the datakeys the unbounded ones make, and sits the scan out, + unarmed, if that is all of them. +- A registered signal whose name a provider key has is dropped from `read()` and + `describe()` while prepared, and keeps its hint, since the replacement has its name. +- The provider key must have the signal's dtype and shape, optionally with a leading + `collections_per_event` dimension. Anything else is a `ValueError` at `describe()`: two + different things share a name, and one must be renamed. +- A registered signal that nothing suppresses is read once per event. + +This is a general `StandardReadable` hook mapping a datakey name to `None` or a replacing +`DataKey`, matched by name so a signal is found wherever it is registered. `None` also drops +the hint, as it is for a signal that is switched off. + +### The detector asks its logic which readables the data hides + +The core `DetectorSuppressLogic`, `DetectorLogic.suppress`, names the readables the +detector's data hides. It is asked at the first `prepare()` of a stage and the answer kept +in the prepare context, so the frozen logic holds no state. Each later `prepare()` asks +again and raises on a difference, before any data logic starts, so a run is never split +part way and no PV the user owns is written. + +### The detector flies with one `fly()` generator + +`FlyableLogic.fly` is one async generator +([ADR 0022](0022-fly-generator-and-ensure-stopped.md)). Progress is collections written +since `prepare_to_fly` noted the count (`ctx.unbounded_start_collections`) out of those +requested. `unstage()` cancels the fly, so no stop logic is needed beyond +`ensure_stopped`. + +A fly scan collects only what the data logics make: registered readables are read only by +`trigger()`. + +### An acquisition is a generator too + +`start_acquiring` and `wait_for_idle` had to hold what the first started on the logic, and +cancelling between them left it running. They become one async generator, `acquire()`, which +owns what it starts, so closing it at its `yield` ends it: + +- An externally triggered detector arms in `prepare()`, so it is waiting on its trigger + source when `kickoff()` returns, and the context holds it until `fly` or `trigger()` takes + it. It cannot be prepared again before then, so nothing replaces it. +- One still armed at `unstage()` is closed then, not at garbage collection; stopping the + detector stays the job of `ensure_stopped`. +- `ADAcquireLogic` writes `WaitForPlugins` (on by default) at every `stage()`, not as the + IOC booted: without it a step scan silently reads plugin scalars of the previous frame, + and the wait is free when a file writer is slowest anyway. `ADContAcqAcquireLogic` has + none, as its driver never stops acquiring. + +### One kickoff per prepare + +The detector was prepared for N events and kicked off a few at a time (`events_to_kickoff`), +but nothing drove that: the RunEngine kicks off once, as for every other flyer. It goes; +`kickoff()` requests the whole scan, and a second needs a new `prepare()`. A detector with +no data logic is still a valid flyer for its triggering. + +### One context per prepare, kept until the next `stage()` + +One `DetectorCtx` threads from `prepare_to_fly` to `fly`; what exists only once flying +lives in `fly`, so no field waits for kickoff as `| None`. The logic holds no state: the +next `prepare()` gets the last context as `previous_ctx`, which is how unbounded providers +carry over and bounded ones are restarted ([ADR 0025](0025-bounded-data-logic-tier.md)). +The trigger logic is not re-prepared per step point. + +- A failed `prepare()` leaves no context, so a retry is a first prepare of the stage: it + takes the hidden set afresh and starts every data logic, whatever the failed one got to. +- The context outlives `complete()`, so `collect_while_completing` can collect after it. +- A step scan's collect publishes up to the point `trigger()` waited for, never re-reading + the counter, so it publishes exactly what `trigger()` saw written. + +`trigger()` runs in a `TRIGGERING` state, so `unstage()` cancels it, `kickoff()` cannot +overlap it, and a failed one leaves the detector staged. It prepares implicitly if nothing +has been. + +## Consequences + +- A verb called out of order raises a `TransitionError` naming it and the state (`: + describe not allowed from STAGED, allowed from: ...`), not `Prepare not run`, and a second + `kickoff()` without a `prepare()` is rejected. +- A detector satisfies neither `WritesStreamAssets` nor `EventPageCollectable` until + prepared; the RunEngine only dispatches on them after prepare. +- `StandardDetector` no longer implements `complete`, `kickoff`, `stage` or `unstage`, so + lifecycle changes reach detectors too. It adds `__on_*` hooks for `read()`, `describe()`, + `hints`, suppression and `unstage()`, registered in lists rather than overriding the + verbs, so mix-ins combined on one Device do not shadow each other through the MRO. It + overrides only `prepare()`, to bind the collect verb. diff --git a/docs/explanations/plan-stubs.md b/docs/explanations/plan-stubs.md index 91010b5c8c..ebc2ef38dd 100644 --- a/docs/explanations/plan-stubs.md +++ b/docs/explanations/plan-stubs.md @@ -200,7 +200,7 @@ This code is to aid understanding rather than for copying. This will be document ### Fly scan nested within a step scan -After `stage` and `open_run` the detectors are `prepare`d for the entire scan. The inner loop `set`s the slow (software) motors to each point, followed by the hardware driven section of the scan. Just like the [](#pure-fly-scan), this consists of `prepare`, then `kickoff`, then `complete` of the fast (hardware) motors, `collect`ing from the detectors repeatedly until they are finished. Finally `close_run` and `unstage` do the cleanup. +After `stage` and `open_run` the inner loop `set`s the slow (software) motors to each point, followed by the hardware driven section of the scan. Just like the [](#pure-fly-scan), this consists of `prepare`, then `kickoff`, then `complete` of the detectors and fast (hardware) motors, `collect`ing from the detectors repeatedly until they are finished. A detector accepts one `kickoff` per `prepare`, so it is `prepare`d again for each hardware driven section. Finally `close_run` and `unstage` do the cleanup. ::::{tab-set} :sync-group: diagram-code @@ -213,9 +213,9 @@ After `stage` and `open_run` the detectors are `prepare`d for the entire scan. T :align: center flowchart TD stage["stage dets, motors"] --> open_run - open_run --> prepare_d["prepare dets"] - prepare_d --> set["set slow motors"] - set --> prepare_m["prepare fast motors"] + open_run --> set["set slow motors"] + set --> prepare_d["prepare dets"] + prepare_d --> prepare_m["prepare fast motors"] prepare_m --> kickoff["kickoff dets, fast motors"] kickoff --> complete["complete dets, fast motors"] complete --> collect["collect dets"] @@ -240,16 +240,17 @@ def fly_inside_step( exposure_time: float, readout_time: float, ) -> MsgGenerator: - yield from bps.prepare( - detector, - TriggerInfo( - livetime=exposure_time, - deadtime=readout_time, - number_of_events=exposures_per_row * len(columns), - ), - ) - for column in columns: + for column in columns: yield from bps.abs_set(column_motor, column) + # One kickoff per prepare, so the detector is prepared for each row + yield from bps.prepare( + detector, + TriggerInfo( + livetime=exposure_time, + deadtime=readout_time, + number_of_events=exposures_per_row, + ), + ) yield from bps.prepare( row_motor, FlyMotorInfo( diff --git a/docs/how-to/choose-right-baseclass.md b/docs/how-to/choose-right-baseclass.md index bbea3b6df6..7fa89dd5e0 100644 --- a/docs/how-to/choose-right-baseclass.md +++ b/docs/how-to/choose-right-baseclass.md @@ -10,9 +10,9 @@ When writing a new Device there are several base classes to choose from that wil There are some utility baseclasses that allow you to create a Device pre-populated with the right verbs to work in bluesky plans: -- [](#StandardReadable) allows you to compose the values of child Signals and Devices together so that you can `read()` the Device during a step scan. +- [](#StandardReadable) allows you to compose the values of child Signals and Devices together so that you can `read()` the Device during a step scan. What each child contributes can also be changed on a live Device, see [](./change-readable-format.md). - [](#StandardMovable) allows devices that move to a target value and wait for a readback to confirm arrival (motors, temperature controllers, piezo stages, and similar) to be implemented by composing a [](#MovableLogic) instance, rather than directly implementing the `Locatable`, `Stoppable`, and `Subscribable` protocols. See [](../explanations/when-to-extend-movable.md) for more details. -- [](#StandardDetector) allows file-writing detectors to be used within both step and fly scans, reporting periodic references to the data that has been written so far. Instances of [](#DetectorTriggerLogic), [](#DetectorAcquireLogic), and [](#DetectorDataLogic) are composed together to provide this functionality. This separation of concerns makes it easy to mix and match different trigger modes, acquisition strategies, and data outputs. +- [](#StandardDetector) allows detectors to be used within both step and fly scans, reporting data as it is acquired: references to file-written data, or the values themselves. Instances of [](#DetectorTriggerLogic), [](#DetectorAcquireLogic), and [](#DetectorDataLogic) are composed together to provide this functionality. This separation of concerns makes it easy to mix and match different trigger modes, acquisition strategies, and data outputs. - [](#StandardFlyable) allows actuators (like a motor controller) to be used within a fly scan, by composing a [](#FlyableLogic) instance rather than directly implementing the `Preparable`, `Flyable`, and `Stageable` protocols. Use it as a mix-in on a Device, or call [](#FlyableLogic.with_device) to wrap a bare logic object in an ephemeral flyer within a plan. ## Adding verbs via protocols diff --git a/docs/how-to/implement-ad-detector.md b/docs/how-to/implement-ad-detector.md index 469c51ef5f..8ffb3a39d4 100644 --- a/docs/how-to/implement-ad-detector.md +++ b/docs/how-to/implement-ad-detector.md @@ -41,9 +41,11 @@ To preserve hardware state in plans like [`bp.count`](#bluesky.plans.count) and [`bps.trigger_and_read`](#bluesky.plan_stubs.trigger_and_read) that call `trigger()` without a preceding `prepare()`, implement: - `default_trigger_info()` - Return the [](#TriggerInfo) to use for the implicit - prepare. For AD detectors call `await trigger_info_from_driver(self.driver.num_images)` to - read back the current `num_images` from the driver rather than resetting it to 1. You should also pass a process - plugin to `trigger_info_from_driver()` if your detector uses one to produce multiple exposures per collection. + prepare. For AD detectors call `await trigger_info_from_driver(self.driver)` to + read back the current `num_images`, `acquire_time` and `acquire_period` from the + driver rather than resetting them. Also pass the process plugin, + `trigger_info_from_driver(self.driver, self.process_plugin)`, if your detector uses + one to produce multiple exposures per collection. @@ -64,8 +66,7 @@ For example, for ADAravis: ## Use ADAcquireLogic or create custom Acquire Logic Most areaDetectors can use the standard [](#adcore.ADAcquireLogic) which handles starting and stopping acquisition via the driver's `acquire` signal. If your detector requires custom acquisition behaviour (e.g., waiting for a specific ready signal), create a [](#DetectorAcquireLogic) subclass with: -- `start_acquiring()` - Start acquisition -- `wait_for_idle()` - Wait until acquisition is complete +- `acquire()` - An async generator that starts acquisition, yields once armed, then waits until acquisition is complete - `ensure_stopped()` - Stop acquisition - `ensure_ready()` - (optional) Override if stage-time reset differs from `ensure_stopped` @@ -78,7 +79,7 @@ Now you need to make an [](#adcore.AreaDetector) subclass that uses your IO and The constructor parameters should include: - `prefix`: The PV prefix for the driver and plugins -- `*writer_factories`: Zero or more [](#adcore.ADWriterFactory) instances (e.g. `ADWriterFactory.hdf(path_provider)`); omit entirely to skip file writing +- `*writer_factories`: Zero or more [](#adcore.ADDataFactory) instances (e.g. `ADDataFactory.hdf(path_provider)`); omit entirely to skip file writing - `driver_suffix`: A PV suffix for the driver, defaulting to `"cam1:"` - `proc_suffix`: An optional PV suffix for a process plugin, which the detector creates with [](#adcore.plugins_with_proc) and gives to its trigger logic - `plugins`: An optional mapping of {`name`: [](#adcore.NDPluginBaseIO)} for additional plugins @@ -94,7 +95,7 @@ For example, for ADAravis: The `AreaDetector` baseclass will: - Store the driver as `self.driver` -- Call `add_detector_logics()` to register your trigger and acquire logic +- Build a [](#DetectorLogic) from your trigger and acquire logic and return it from `logic` - Create and register a data logic for file writing for each factory in `writer_factories` - Add configuration signals (driver.acquire_time, driver.acquire_period, and any you specify) - Store any plugins as attributes on the detector @@ -116,7 +117,7 @@ For example, for ADAravis: ## Add multiple data streams (optional) -The composition-based architecture makes it possible to add multiple data outputs to a detector. After creating the detector, you can call `add_detector_logics()` to add additional data sources: +The composition-based architecture makes it possible to give a detector several data outputs. Every [](#adcore.ADDataFactory) factory takes `always_enable`: `False` (the default) follows the plugin's `EnableCallbacks` PV, so a data output whose plugin is off sits out and another can serve the scan, and `True` switches the plugin on at the start of a scan. ### Reading stats plugins alongside file writing A plugin scalar needs no data logic: a detector is a [](#StandardReadable), so register @@ -125,7 +126,9 @@ the signal and it appears in `read()` alongside whatever the file writer produce ```python from ophyd_async.core import StandardReadableFormat as Format -det = adaravis.AravisDetector(prefix, path_provider, plugins={"stats": stats}) +det = adaravis.AravisDetector( + prefix, adcore.ADDataFactory.hdf(path_provider), plugins={"stats": stats} +) # Add stats total as a hinted signal in events det.set_readable_format(det.stats.total, Format.HINTED_UNCACHED_SIGNAL) ``` @@ -136,7 +139,7 @@ roi1 = adcore.NDROIIO("PREFIX:ROI1:") roi2 = adcore.NDROIIO("PREFIX:ROI2:") det = adaravis.AravisDetector( "PREFIX:", - adcore.ADWriterFactory.hdf( + adcore.ADDataFactory.hdf( path_provider, writer_suffix="HDF1:", writer_name="hdf1", @@ -147,7 +150,7 @@ det = adaravis.AravisDetector( color_mode_signal=driver.color_mode, ), ), - adcore.ADWriterFactory.hdf( + adcore.ADDataFactory.hdf( path_provider, writer_suffix="HDF2:", writer_name="hdf2", @@ -184,7 +187,7 @@ cb_plugin = adcore.NDCircularBuffIO("PREFIX:CB1:") det = adcore.AreaDetector( driver, "PREFIX:", - adcore.ADWriterFactory.hdf(path_provider), + adcore.ADDataFactory.hdf(path_provider), acquire_logic=adcore.ADContAcqAcquireLogic(driver, cb_plugin, num_frames_to_skip=1), trigger_logic=adcore.ADContAcqTriggerLogic(driver, cb_plugin, None), plugins={"cb1": cb_plugin}, @@ -193,9 +196,9 @@ det = adcore.AreaDetector( The [](#adcore.ADContAcqTriggerLogic) will: - Validate that a requested exposure time matches the driver's current exposure time -- Configure zero pre-trigger frames and the requested number of post-trigger frames, then arm circular buffer capture +- Configure the requested number of post-trigger frames -The [](#adcore.ADContAcqAcquireLogic) opens the circular buffer gate instead of starting the driver. Unstaging stops CB capture but leaves the driver acquiring continuously. +The [](#adcore.ADContAcqAcquireLogic) does the rest of the circular buffer set-up in its `ensure_ready`, at `stage()`: it checks the circular buffer reads directly from the driver, switches it on, configures zero pre-trigger frames, then arms circular buffer capture. At each acquisition it opens the circular buffer gate instead of starting the driver. Unstaging stops CB capture but leaves the driver acquiring continuously. ### Skip dirty frames @@ -265,7 +268,7 @@ You have now made a detector, and can import and create it like this: ```python from ophyd_async.epics import adaravis, adcore -det = adaravis.AravisDetector("PREFIX:", adcore.ADWriterFactory.hdf(path_provider)) +det = adaravis.AravisDetector("PREFIX:", adcore.ADDataFactory.hdf(path_provider)) ``` The detector will now support: diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md index 78afba8c51..48cf30ad97 100644 --- a/docs/reference/environment-variables.md +++ b/docs/reference/environment-variables.md @@ -10,15 +10,19 @@ effect on ophyd-async itself. ## `OPHYD_ASYNC_PRESERVE_DETECTOR_STATE` -Controls the implicit prepare performed by +Controls whether a [`StandardDetector`](#ophyd_async.core.StandardDetector) reads +hardware state back with +[`DetectorTriggerLogic.default_trigger_info()`](#ophyd_async.core.DetectorTriggerLogic.default_trigger_info). +It does so in two places: the implicit prepare performed by [`StandardDetector.trigger()`](#ophyd_async.core.StandardDetector.trigger) when -[`prepare()`](#ophyd_async.core.StandardDetector.prepare) has not been called since -the last `stage()`. +[`prepare()`](#ophyd_async.core.StandardFlyable.prepare) has not been called since +the last `stage()`, and `prepare()` with a `livetime` of 0, which reads back the +current livetime and deadtime to give the data logics the exposure period. | Value | Behaviour | |-------|-----------| -| `YES` | Calls [`DetectorTriggerLogic.default_trigger_info()`](#ophyd_async.core.DetectorTriggerLogic.default_trigger_info) to read current hardware state before the implicit prepare. Raises `RuntimeError` if `default_trigger_info()` is not implemented on the trigger logic. | -| anything else (default) | Uses a bare [`TriggerInfo()`](#ophyd_async.core.TriggerInfo), resetting detector state to defaults. | +| `YES` | Calls `default_trigger_info()` in both places. The implicit prepare raises `RuntimeError` if it is not implemented on the trigger logic. | +| anything else (default) | The implicit prepare uses a bare [`TriggerInfo()`](#ophyd_async.core.TriggerInfo), resetting detector state to defaults, and a `livetime` of 0 gives the data logics a period of 0. | See [ADR 0013](../explanations/decisions/0013-preserve-hardware-state-in-step-scan.md) for rationale. diff --git a/docs/tutorials/implementing-detectors.md b/docs/tutorials/implementing-detectors.md index 0c955d4d5d..0236f05850 100644 --- a/docs/tutorials/implementing-detectors.md +++ b/docs/tutorials/implementing-detectors.md @@ -179,7 +179,7 @@ It derives from [](#StandardDetector) which is a utility baseclass that implemen - [](#DetectorAcquireLogic) to start and stop acquisition and wait for it to complete - [](#DetectorDataLogic) to tell the detector to open a file, describe the datasets it will write, and emit StreamAsset documents as frames are written -In this case, we have three logic classes written just for this simulation, all taking a reference to the pattern generator that provides methods for both detector control and file writing. In other cases the detector control and filewriting may be handled by different sub-devices that talk to different parts of the control system. The job of the top level detector class is to take the arguments that the logic classes need, create the logic instances, and pass them to [](#StandardDetector.add_detector_logics). +In this case, we have three logic classes written just for this simulation, all taking a reference to the pattern generator that provides methods for both detector control and file writing. In other cases the detector control and filewriting may be handled by different sub-devices that talk to different parts of the control system. The job of the top level detector class is to take the arguments that the logic classes need and, in `logic`, create the logic instances and wire them into a [](#DetectorLogic) by role: `DetectorLogic(trigger=..., acquire=..., data=[...])`. Now let's look at the underlying classes that define the detector behavior: @@ -215,28 +215,46 @@ Next we have `BlobAcquireLogic`, a [](#DetectorAcquireLogic) subclass: Its job is to control the acquisition process on the detector, starting and stopping the collection of data: - `ensure_ready()` (inherited default) delegates to `ensure_stopped`, putting the detector into a known state at the start of each scan. -- `start_acquiring()` starts the acquisition process that has been prepared. In this case we create a background task that will write our simulation images to file. -- `wait_for_idle()` waits for that acquisition process to be complete. -- `ensure_stopped()` interrupts the acquisition process and waits for it to complete. Called from `unstage()` at the end of a scan. +- `acquire()` is an async generator for one acquisition. It starts the acquisition process that has been prepared, in this case a background task that will write our simulation images to file, yields once it is running, then waits for it to be complete. If the scan is cancelled the detector closes it at the `yield`, so the `finally` cancels the task. +- `ensure_stopped()` stops the detector at the end of a scan, called from `unstage()`. Here `acquire()` owns the only thing running, so there is nothing to do. ### `BlobDataLogic` -Then we have `BlobDataLogic`, a [](#DetectorDataLogic) subclass: +Then we have `BlobDataLogic`, an [](#UnboundedDataLogic) subclass: ```{literalinclude} ../../src/ophyd_async/sim/_blob_data_logic.py :language: python ``` Its job is to manage the file writing and data streaming: -- `prepare_unbounded()` tells the detector to open a file, and returns a [](#StreamableDataProvider) that describes the datasets that will be written and tracks write progress +- `start_provider()` works out where the file will go, opens it, and returns a [](#StreamableDataProvider) describing the datasets being written - `get_hinted_fields()` returns the data keys that are interesting to plot -- `stop()` tells the detector to close the file +- `ensure_stopped()` tells the detector to close the file -The [](#StreamableDataProvider) returned from `prepare_unbounded()` contains: -- `collections_written_signal` - a signal that tracks how many frames have been written +The [](#StreamableDataProvider) returned from `start_provider()` is made for the scan's collections per event and given `collections_written`, a signal that tracks how many frames have been written. It contains: - `make_datakeys()` - creates DataKey descriptions for each dataset - `make_stream_docs()` - emits StreamResource and StreamDatum documents as frames are written +#### Which kind of data a logic produces + +The detector starts each of its logics for the scan, and the type of provider that comes back says which kind of data it is. + +- a [](#StreamableDataProvider) publishes stream assets, like a file writer. The detector exposes `collect_asset_docs`, and `BlobDataLogic` is an example. +- a [](#PageableDataProvider) publishes event pages, like an areaDetector stats time series or a scaler's arrays, so the detector exposes `collect_pages` instead. Its `make_page()` carries on from where its last page got to; a step scan's `read()` takes the last event of the page made up to the end of the point. +- `None` means "not this stage", for instance a plugin that is switched off. The logic decides this without writing anything. This is not an error: a detector may carry more data logics than any one scan uses. + +Separately from that, a data logic is either a [](#BoundedDataLogic) or an [](#UnboundedDataLogic), whichever it inherits from (not both). An unbounded logic, like a file writer, keeps counting for as long as it is told to; its collections-written signal is read when each `prepare()` finishes. A bounded logic needs to be told how many collections to take, like a finite buffer: it is started after the unbounded ones and handed the datakeys their providers already make, and its provider must make none of them. It returns `None` if nothing is left to make, and raises for a scan with no fixed number of collections (`num_collections` of 0) if something is. At each point of a step scan after the first it is stopped and started again, so the buffer is cleared. Either kind may return either kind of provider. + +A source that produces one value per event, like a plugin scalar, needs no data logic at all: a detector is a [](#StandardReadable), so register the signal with [](#StandardReadable.set_readable_format). + +Both kinds of `start_provider()` are given the detector `name`, which a logic uses for its datakeys (adding its own suffix if it has one), and the fields they need from the scan's [](#TriggerInfo): `collections_per_event`, a `collection_period` and a `flush_period`, and the bounded one also `existing_datakeys` and `num_collections`. The `collection_period` is `exposures_per_collection * (livetime + deadtime)`, where `prepare()` reads the livetime and deadtime back from the trigger logic when the plan's [](#TriggerInfo) leaves `livetime` at 0 and [](#OPHYD_ASYNC_PRESERVE_DETECTOR_STATE) is `YES`; otherwise it is 0 for "use what is on the hardware". + +No two providers may make the same datakey: `prepare()` raises if they do. A datakey a provider makes replaces a registered signal of that name. + +```{note} +Which logics serve is decided at the first `prepare()` after `stage()` and holds until the next `stage()`, so it can change from one stage to the next: a detector whose file writer is switched off can serve its stats time series instead. Change it **between stages**, not inside one: a run's descriptor is emitted at its start, so switching part way through would leave the descriptor and the documents disagreeing. +``` + ## Conclusion We have seen how to make a [](#StandardDetector) and how the [](#DetectorTriggerLogic), [](#DetectorAcquireLogic), and [](#DetectorDataLogic) classes allow us to customise its behavior through composition. This separation of concerns makes it easy to: diff --git a/src/ophyd_async/core/__init__.py b/src/ophyd_async/core/__init__.py index 3583552b48..5875112f71 100644 --- a/src/ophyd_async/core/__init__.py +++ b/src/ophyd_async/core/__init__.py @@ -9,6 +9,9 @@ soft_command, ) from ._data_providers import ( + DataProvider, + EventPageDataProvider, + PageableDataProvider, StreamableDataProvider, StreamResourceDataProvider, StreamResourceInfo, @@ -22,12 +25,16 @@ ) from ._derived_signal_backend import Transform, merge_gathered_dicts from ._detector import ( + BoundedDataLogic, DetectorAcquireLogic, DetectorDataLogic, + DetectorLogic, + DetectorSuppressLogic, DetectorTrigger, DetectorTriggerLogic, StandardDetector, TriggerInfo, + UnboundedDataLogic, ) from ._device import ( Device, @@ -49,7 +56,12 @@ OnOff, YesNo, ) -from ._flyable import KICKED_OFF, FlyableLogic, FlyMotorInfo, StandardFlyable +from ._flyable import ( + KICKED_OFF, + FlyableLogic, + FlyMotorInfo, + StandardFlyable, +) from ._log import config_ophyd_async_logging, logger from ._mock_signal_backend import MockSignalBackend from ._mock_signal_utils import ( @@ -291,6 +303,10 @@ def __getattr__(name): "DetectorTriggerLogic", "DetectorAcquireLogic", "DetectorDataLogic", + "BoundedDataLogic", + "UnboundedDataLogic", + "DetectorLogic", + "DetectorSuppressLogic", "StandardDetector", # Path "PathInfo", @@ -304,6 +320,9 @@ def __getattr__(name): "AutoMaxIncrementingPathProvider", "UUIDFilenameProvider", # Data Providers + "DataProvider", + "EventPageDataProvider", + "PageableDataProvider", "StreamableDataProvider", "StreamResourceInfo", "StreamResourceDataProvider", diff --git a/src/ophyd_async/core/_data_providers.py b/src/ophyd_async/core/_data_providers.py index 8faa7ba069..c07df6881d 100644 --- a/src/ophyd_async/core/_data_providers.py +++ b/src/ophyd_async/core/_data_providers.py @@ -1,36 +1,195 @@ -from abc import abstractmethod -from collections.abc import AsyncIterator, Sequence +from abc import ABC, abstractmethod +from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence from typing import Any +import numpy as np from bluesky.protocols import StreamAsset from event_model import ComposeStreamResource, DataKey, StreamRange +from event_model.documents import PartialEventPage -from ._signal import SignalR, SignalW -from ._utils import ConfinedModel +from ._datatypes import Array1D +from ._signal import SignalR, SignalW, observe_value +from ._utils import ( + ConfinedModel, + _gather_or_cancel_all, + error_if_none, + get_dtype, +) +ARRAY_WAIT_TIMEOUT = 2.0 +"""How long `EventPageDataProvider` waits for an array to hold the collections +counted, in seconds. Longer than the one second between the posts of an +areaDetector time series.""" -class StreamableDataProvider: - collections_written_signal: SignalR[int] + +async def _wait_for_length(signal: SignalR[Array1D[Any]], needed: int) -> Array1D[Any]: + """Return the first value of `signal` holding at least `needed` collections.""" + held = 0 + kept = None + try: + async for value in observe_value(signal, done_timeout=ARRAY_WAIT_TIMEOUT): + held = len(value) + if held >= needed: + kept = value + break + except TimeoutError: + raise TimeoutError( + f"{signal.source} held {held} collections after {ARRAY_WAIT_TIMEOUT}s, " + f"but {needed} were needed" + ) from None + return error_if_none(kept, f"{signal.source} was not observed") + + +class DataProvider(ABC): + """What a [](#DetectorDataLogic) returns to publish the data it is writing. + + The two kinds, [](#StreamableDataProvider) and [](#PageableDataProvider), + differ in how the data is published; what they share is here. A provider + counts its events from an origin: collection 0 of `collections_written` + unless `start_new_resources` says otherwise. + + :param collections_written: counts the collections written so far, reset to + 0 by the data logic before it returns the provider + :param collections_per_event: how many collections make up one event + """ + + def __init__( + self, collections_written: SignalR[int], collections_per_event: int + ) -> None: + self.collections_written = collections_written + self.collections_per_event = collections_per_event @abstractmethod - async def make_datakeys(self, collections_per_event: int) -> dict[str, DataKey]: - """Return a DataKey for each Signal that produces a Reading. + async def make_datakeys(self) -> dict[str, DataKey]: + """Return a DataKey for each field, of shape `[collections_per_event, ...]`. Called before the first exposure is taken. + """ - :param collections_per_event: this should appear in the shape of each DataKey + @abstractmethod + def start_new_resources(self, origin: int) -> None: + """Count events from collection `origin`, with nothing yet published. + + Called on an unbounded provider at each `prepare()`, so each run has its + own stream resources. + """ + + +class StreamableDataProvider(DataProvider): + """Publishes its data as stream assets: a StreamResource and StreamDatums.""" + + @abstractmethod + def make_stream_docs(self, index: int) -> AsyncIterator[StreamAsset]: + """Make the StreamAsset documents for the events up to `index`. + + Emits the StreamResources the first time, then StreamDatums from where + the last call got to. + + :param index: the event to publish up to, counted from the origin """ - async def make_stream_docs( - self, collections_written: int, collections_per_event: int - ) -> AsyncIterator[StreamAsset]: - """Make StreamAsset documents up to the given index. - Default implementation is a no-op. Subclasses should override this - to emit actual StreamAsset documents. +class PageableDataProvider(DataProvider): + """Publishes its data as event pages. + + For a source read back as arrays rather than written to a file: an + areaDetector stats time series, or a scaler's MCA arrays. + """ + + @abstractmethod + async def make_page(self, index: int) -> PartialEventPage | None: + """Return one page of the events from where the last call got to. + + :param index: the event to publish up to + :return: the page, or `None` if there are no new events """ - while False: - yield + + +class EventPageDataProvider(PageableDataProvider): + """Emits a fixed-length array of collections as event pages. + + For a device that fills one array per field as it acquires, alongside an + array of per-collection timestamps: an areaDetector stats time series, or a + scaler's MCA arrays. Each array is sliced into `collections_per_event`-length + chunks, one per event, and the last collection of an event supplies that + event's timestamp. A page waits up to `ARRAY_WAIT_TIMEOUT` for each array to + hold every collection of its events, and raises `TimeoutError` naming the + signal if one does not. + + :param data: datakey (already suffixed) to the array signal that backs it + :param timestamps: the acquisition time of each collection + :param collections_written: counts the collections written so far + :param collections_per_event: how many collections make up one event + :param refresh: + awaited before each page's arrays are read, for a device that posts its + arrays less often than it counts collections: it asks the device to post + them, so they hold every collection counted + """ + + def __init__( + self, + data: Mapping[str, SignalR[Array1D[Any]]], + timestamps: SignalR[Array1D[np.float64]], + collections_written: SignalR[int], + collections_per_event: int, + refresh: Callable[[], Awaitable[None]] | None = None, + ) -> None: + super().__init__(collections_written, collections_per_event) + self.data = dict(data) + self.timestamps = timestamps + self.refresh = refresh + self.start_new_resources(0) + + def start_new_resources(self, origin: int) -> None: + self._origin = origin + # The events published by make_page + self._published = 0 + + async def make_datakeys(self) -> dict[str, DataKey]: + return { + datakey: DataKey( + source=signal.source, + shape=[self.collections_per_event], + dtype="array", + dtype_numpy=get_dtype( + error_if_none(signal.datatype, f"{signal.source} has no datatype") + ).str, + ) + for datakey, signal in self.data.items() + } + + async def make_page(self, index: int) -> PartialEventPage | None: + if index <= self._published: + return None + if self.refresh: + await self.refresh() + cpe = self.collections_per_event + needed = self._origin + index * cpe + # Wait for every array and the per-collection timestamps to hold the + # collections counted, all at once + *arrays_held, stamps = await _gather_or_cancel_all( + [ + *(_wait_for_length(signal, needed) for signal in self.data.values()), + _wait_for_length(self.timestamps, needed), + ] + ) + arrays = dict(zip(self.data, arrays_held, strict=True)) + starts = [self._origin + event * cpe for event in range(self._published, index)] + self._published = index + # One timestamp per event: the acquisition time of its last collection + event_times = [float(stamps[start + cpe - 1]) for start in starts] + data = { + datakey: [array[start : start + cpe] for start in starts] + for datakey, array in arrays.items() + } + return { + # TODO: numpy arrays are not in PartialEventPage's value type, but + # they round trip through the bundler fine; fix upstream in + # event-model rather than converting to lists here + "data": data, # type: ignore[typeddict-item] + "time": event_times, + "timestamps": dict.fromkeys(data, event_times), + } class StreamResourceInfo(ConfinedModel): @@ -63,8 +222,12 @@ class StreamResourceInfo(ConfinedModel): class StreamResourceDataProvider(StreamableDataProvider): """A helper class to make stream resource and datums for HDF datasets. - :param full_file_name: Absolute path to the file that has been written - :param datasets: Descriptions of each of the datasets that will appear in the file + :param uri: the file that is being written + :param resources: descriptions of each of the datasets that will appear in it + :param mimetype: the mimetype of the file + :param collections_written: counts the collections written so far + :param collections_per_event: how many collections make up one event + :param flush_signal: set before making stream docs, to flush the file """ def __init__( @@ -72,19 +235,23 @@ def __init__( uri: str, resources: Sequence[StreamResourceInfo], mimetype: str, - collections_written_signal: SignalR[int], + collections_written: SignalR[int], + collections_per_event: int, flush_signal: SignalW[bool] | None = None, ) -> None: + super().__init__(collections_written, collections_per_event) self.uri = uri self.resources = list(resources) - self.collections_written_signal = collections_written_signal + self.mimetype = mimetype self.flush_signal = flush_signal - self.last_emitted = 0 + self.start_new_resources(0) + + def start_new_resources(self, origin: int) -> None: bundler_composer = ComposeStreamResource() self.bundles = [ bundler_composer( - mimetype=mimetype, - uri=uri, + mimetype=self.mimetype, + uri=self.uri, data_key=resource.data_key, parameters={ "chunk_shape": resource.chunk_shape, @@ -95,38 +262,37 @@ def __init__( ) for resource in self.resources ] + self._resources_emitted = False + self._origin = origin + # The collections in the file that StreamDatums have been made for + self._published = origin - async def make_datakeys(self, collections_per_event: int) -> dict[str, DataKey]: - describe = { + async def make_datakeys(self) -> dict[str, DataKey]: + cpe = self.collections_per_event + return { resource.data_key: DataKey( source=resource.source or self.uri, - shape=[collections_per_event, *resource.shape], - dtype="array" - if collections_per_event > 1 or len(resource.shape) > 0 - else "number", + shape=[cpe, *resource.shape], + dtype="array" if cpe > 1 or len(resource.shape) > 0 else "number", dtype_numpy=resource.dtype_numpy, external="STREAM:", ) for resource in self.resources } - return describe - async def make_stream_docs( - self, collections_written: int, collections_per_event: int - ) -> AsyncIterator[StreamAsset]: + async def make_stream_docs(self, index: int) -> AsyncIterator[StreamAsset]: if self.flush_signal: await self.flush_signal.set(True) # TODO: fail if we get dropped frames - indices_written = collections_written // collections_per_event - if indices_written and not self.last_emitted: - for bundle in self.bundles: - yield "stream_resource", bundle.stream_resource_doc - - if indices_written > self.last_emitted: - indices: StreamRange = { - "start": self.last_emitted, - "stop": indices_written, - } - self.last_emitted = indices_written + cpe = self.collections_per_event + written = self._origin + index * cpe + # StreamDatum indices are events in the file, which may hold earlier runs + indices: StreamRange = {"start": self._published // cpe, "stop": written // cpe} + if indices["stop"] > indices["start"]: + if not self._resources_emitted: + self._resources_emitted = True + for bundle in self.bundles: + yield "stream_resource", bundle.stream_resource_doc for bundle in self.bundles: yield "stream_datum", bundle.compose_stream_datum(indices) + self._published = written diff --git a/src/ophyd_async/core/_detector.py b/src/ophyd_async/core/_detector.py index 6cf7f27a9e..0dc91ca6cd 100644 --- a/src/ophyd_async/core/_detector.py +++ b/src/ophyd_async/core/_detector.py @@ -1,51 +1,60 @@ """Module which defines abstract classes to work with detectors.""" import asyncio -import functools import os import time import warnings -from abc import ABC, abstractmethod -from collections.abc import AsyncIterator, Awaitable, Callable, Iterator, Sequence -from dataclasses import dataclass +from abc import abstractmethod +from collections.abc import ( + AsyncGenerator, + AsyncIterator, + Collection, + Iterator, + Mapping, + Sequence, +) +from contextlib import aclosing, asynccontextmanager +from dataclasses import dataclass, field from enum import Enum from functools import cached_property -from typing import cast +from typing import Any, Generic, TypeVar, cast from bluesky.protocols import ( Collectable, - Flyable, + HasHints, Hints, - Preparable, Reading, Stageable, + Status, StreamAsset, Triggerable, - WritesStreamAssets, ) from event_model import DataKey -from pydantic import Field, NonNegativeInt, PositiveInt, computed_field +from event_model.documents import PartialEventPage +from pydantic import Field, PositiveInt, computed_field -from ._data_providers import StreamableDataProvider -from ._readable import ( - StandardReadable, - StandardReadableFormat, - _config_signals, +from ._data_providers import ( + DataProvider, + PageableDataProvider, + StreamableDataProvider, ) +from ._device import Device +from ._flyable import KICKED_OFF, FlyableLogic, StandardFlyable +from ._readable import StandardReadable, StandardReadableFormat from ._settings import Settings -from ._signal import ( - SignalDict, - SignalR, - SignalRW, - observe_signals_value, - soft_signal_rw, -) -from ._status import AsyncStatus, WatchableAsyncStatus +from ._signal import SignalDict, SignalR, SignalRW, observe_signals_value +from ._standard_base import _State, _StateMachine +from ._status import WatchableAsyncStatus from ._utils import ( DEFAULT_TIMEOUT, ConfinedModel, WatcherUpdate, + _gather_or_cancel_all, + abstract_cached_property, error_if_none, + gather_dict, + get_unique, + merge_dicts, merge_gathered_dicts, ) @@ -96,8 +105,8 @@ class TriggerInfo(ConfinedModel): w) for each. """ - number_of_events: NonNegativeInt = Field(default=1) - """Number of bluesky events that will be emitted, (0 means infinite).""" + number_of_events: PositiveInt = Field(default=1) + """Number of bluesky events that will be emitted.""" exposure_timeout: float = Field( default_factory=lambda d: d["livetime"] + d["deadtime"] + DEFAULT_TIMEOUT, @@ -112,7 +121,18 @@ class TriggerInfo(ConfinedModel): ), gt=0, ) - """What is the maximum timeout on waiting for a collection""" + """The longest wait for progress before giving up: the most time allowed + between one collection being written and the next, not a total for the + scan.""" + + flush_period: float = Field(default=0.0, ge=0.0) + """How many seconds between flushes of the data being written. + + A data logic that buffers before it writes sizes that buffer from this and + the exposure period, so data appears at roughly this rate: an HDF chunk of + `flush_period / period` frames, say. Set it to the rate the plan collects at + so a fly scan sees each batch as it lands. 0 means leave whatever the + hardware is set to.""" @computed_field @cached_property @@ -195,14 +215,11 @@ async def default_trigger_info(self) -> TriggerInfo: raise NotImplementedError(self) -def _logic_supported(base_class, method) -> bool: +def _trigger_logic_supported(method) -> bool: # If the function that is bound in a subclass is the same as the function - # attached to the superclass, then the subclass has not overridden it, so - # this method is not supported by the subclass. - return method.__func__ is not getattr(base_class, method.__name__) - - -_trigger_logic_supported = functools.partial(_logic_supported, DetectorTriggerLogic) + # attached to DetectorTriggerLogic, then the subclass has not overridden it, + # so this method is not supported by the subclass. + return method.__func__ is not getattr(DetectorTriggerLogic, method.__name__) def _get_supported_triggers( @@ -218,52 +235,51 @@ def _get_supported_triggers( return supported_triggers -class DetectorAcquireLogic(ABC): - """Abstract base class for detector acquisition lifecycle hooks. +class DetectorAcquireLogic: + """Base class for detector acquisition lifecycle hooks. - Subclasses implement four hooks that are called at defined points in the - `[](#StandardDetector)` lifecycle: + Subclasses implement these hooks, called at defined points in the + [](#StandardDetector) lifecycle: - `ensure_ready` — called from `stage()` to put the detector into a known - idle state before a scan begins. - - `start_acquiring` — called from `prepare()`, `kickoff()`, or `trigger()` - to start the detector acquiring. - - `wait_for_idle` — called after the final collection completes to confirm - the detector has returned to idle. + idle state before a scan begins. Defaults to `ensure_stopped`. + - `acquire`: an async generator for one acquisition: start the detector + acquiring, yield once it is armed, then wait for it to return to idle. - `ensure_stopped` — called from `unstage()` to stop the detector and perform any end-of-scan cleanup. - - The default `ensure_ready` delegates to `ensure_stopped`, which is correct - for detectors where stage-time reset and scan-end teardown are identical. - Override `ensure_ready` when the two phases require different behaviour - (e.g. arming the detector once at stage time while keeping it armed across - multiple kickoff/complete cycles). """ async def ensure_ready(self): """Ensure the detector is idle before a scan. - Called from `stage()`. The default implementation delegates to - `ensure_stopped`, which is sufficient for detectors that perform the - same reset at stage time as at unstage time. Override this method when - a different action is required at stage time (for example, arming the - detector once so it is ready for multiple kickoff/complete cycles - without re-arming). + Called from `stage()`. Defaults to `ensure_stopped`; override it when + stage needs a different action, such as arming once for many + kickoff/complete cycles. """ await self.ensure_stopped() @abstractmethod - async def start_acquiring(self): - """Start the detector acquiring. - - Called from `prepare()` for external-trigger modes, and from - `kickoff()` / `trigger()` for internal-trigger modes. + def acquire(self) -> AsyncGenerator[None, None]: + """Start acquiring, yield once armed, then wait until idle. + + Write it as an async generator that yields once. It is entered in + `prepare()` for external triggering, and in `kickoff()` or `trigger()` + for internal triggering, then run to the end once the data is written. + Acquisition must stop by itself once the data is written: a free-running + detector is not supported. + If the scan is cancelled it is closed at its `yield`, so own whatever it + starts: hold a status with `async with`, or cancel a task in `finally`. + For example: + + ```python + async def acquire(self): + async with self.driver.acquire.set(True) as status: + await wait_for_value(self.driver.armed, True, timeout=10) + yield # armed + await status + ``` """ - @abstractmethod - async def wait_for_idle(self): - """Wait for the detector to return to idle after the final collection.""" - @abstractmethod async def ensure_stopped(self): """Stop the detector and perform end-of-scan cleanup. @@ -272,175 +288,1063 @@ async def ensure_stopped(self): """ -def _all_the_same(collections_written: set[int]) -> int: - """Ensure all collection counts are the same, raising an error if they differ. +#: The provider a data logic returns from `start_provider` +ProviderT = TypeVar("ProviderT", bound=DataProvider) - :param collections_written: Set of collection counts from different providers - :return: The single collection count value - :raises RuntimeError: If the set contains more than one distinct value - """ - if len(collections_written) != 1: - msg = ( - "Detectors have written different numbers of collections: " - + f"{collections_written}" - ) - raise RuntimeError(msg) - return collections_written.pop() +class DetectorDataLogic(Generic[ProviderT]): + """Base class for detector data logic: do not inherit from it directly. -async def _get_collections_written( - data_providers: Sequence[StreamableDataProvider], - reducer: Callable[[set[int]], int] = _all_the_same, -) -> int: - """Return a single collections_written value for the given providers. - - By default this function ensures all providers agree and returns that - single value. If `reducer` is provided it will be called with the set of - observed values and should return a single int to use. - """ - # Work out where all the streamable data providers are up to - collections_written = set( - await asyncio.gather( - *[sdp.collections_written_signal.get_value() for sdp in data_providers] - ) - ) - if collections_written: - # Let our reducer decide how to return a single int - return reducer(collections_written) - else: - # There are none, this is valid as we then don't use the value anywhere - # so just return 0 - return 0 + Inherit from [](#BoundedDataLogic) or [](#UnboundedDataLogic), which say + whether the data source needs to know how many collections to take. + At the first `prepare()` after `stage()` the detector starts every + unbounded data logic, then every bounded one, handing each bounded logic the + datakeys the unbounded providers already make. The logics that return a + provider serve the stage. At each later `prepare()` the bounded ones are + started again, and the unbounded ones only if their provider cannot carry + over (see [](#UnboundedDataLogic)). + The provider describes and publishes what is being written. -class DetectorDataLogic: - """Abstract base class for detector data logic and handling. + Whether the data is published as stream assets or event pages is the type + of the provider that is returned, not a declaration on the logic, so one + logic may produce either depending on the scan. - Implementations must override `prepare_unbounded`. A source that produces a - single value per event, like a plugin scalar, does not need a data logic at - all: register its signal with + A source that produces a single value per event, like a plugin scalar, needs + no data logic at all: register its signal with [](#StandardReadable.set_readable_format) instead. """ - #: Add this suffix to the detector name to specify the datakey. These need to be - #: different for each DetectorDataLogic added to a detector - datakey_suffix: str = "" + def __init_subclass__(cls, **kwargs) -> None: + super().__init_subclass__(**kwargs) + if "stop" in cls.__dict__: + raise TypeError( + f"{cls.__name__} defines stop, which DetectorDataLogic no longer " + "calls: stop -> ensure_stopped" + ) - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: - """Provider can work for an unbounded number of collections.""" - raise NotImplementedError(self) + async def ensure_ready(self) -> None: + """Ensure the data logic is idle before a scan. - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: - """Return the hinted streams.""" + Called from `stage()`. Defaults to `ensure_stopped`. + """ + await self.ensure_stopped() + + def get_hinted_fields(self, name: str) -> Sequence[str]: + """Return the datakeys to hint, of those this logic may produce. + + The detector hints only those its providers actually make. + + :param name: the detector name, which prefixes every datakey + """ return [] - async def stop(self) -> None: + async def ensure_stopped(self) -> None: """Stop taking data.""" return None -_data_logic_supported = functools.partial(_logic_supported, DetectorDataLogic) +class BoundedDataLogic(DetectorDataLogic[ProviderT]): + """A data logic whose source must be told how many collections to take. + For a finite buffer, like an areaDetector stats time series. It is started + after the unbounded logics, so it can leave out datakeys they already make. + At each step point after the first the detector stops it and starts a new + provider, so a buffer holding the last point's data is cleared. + """ -@dataclass -class _PrepareCtx: - trigger_info: TriggerInfo - streamable_data_providers: Sequence[StreamableDataProvider] - collections_written: int + @abstractmethod + async def start_provider( + self, + name: str, + existing_datakeys: Sequence[str], + num_collections: int, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> ProviderT | None: + """Start writing and return a new provider describing what is written. + + Return a [](#StreamableDataProvider) to publish stream assets, or a + [](#PageableDataProvider) to publish event pages, made for + `collections_per_event`. The provider must make none of + `existing_datakeys`. Return `None`, without writing, if there is nothing + left to make: this is not an error and is not warned about. + + :param name: the detector name, which prefixes every datakey + :param existing_datakeys: the datakeys the unbounded providers make + :param num_collections: how many collections the scan takes + :param collections_per_event: how many collections make up one event + :param collection_period: seconds per collection, + `exposures_per_collection * (livetime + deadtime)`, for sizing + chunks (0 means "use whatever is currently set on the hardware") + :param flush_period: how often to flush + """ + raise NotImplementedError(self) + + +class UnboundedDataLogic(DetectorDataLogic[ProviderT]): + """A data logic whose source takes data for as long as it is told to. + + For a file writer, or anything else that does not need to know how many + collections to take. Its provider carries over to the next `prepare()` of + the same stage if that is for the same collections per event, collection period + and flush period, and every unbounded provider has written the same whole + number of events, so it may already have written collections when a scan + starts. + """ + + @abstractmethod + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> ProviderT | None: + """Start writing and return a new provider describing what is written. + + Return a [](#StreamableDataProvider) to publish stream assets, or a + [](#PageableDataProvider) to publish event pages, made for + `collections_per_event`. Return `None`, without writing, to sit out the + scans of this stage, for instance a plugin that is switched off: this is + not an error and is not warned about. + + :param name: the detector name, which prefixes every datakey + :param collections_per_event: how many collections make up one event + :param collection_period: seconds per collection, + `exposures_per_collection * (livetime + deadtime)`, for sizing + chunks (0 means "use whatever is currently set on the hardware") + :param flush_period: how often to flush + """ + raise NotImplementedError(self) + + +def _merge_pages( + pages: Sequence[tuple[PageableDataProvider, PartialEventPage | None]], +) -> PartialEventPage | None: + """Merge one page per pageable provider into a single page. + + `None` if every provider returned `None`. Raises if only some did, if the + pages hold different numbers of events, or if a key appears twice. + """ + if all(page is None for _, page in pages): + return None + # A None page has length None, so a mix of None and real pages differs too + lengths = { + f"{type(provider).__name__}[{i}]": len(page["time"]) if page else None + for i, (provider, page) in enumerate(pages) + } + get_unique(lengths, "numbers of events") + made = [page for _, page in pages if page] + # Each event happened when the last of its providers' data arrived + times = zip(*(page["time"] for page in made), strict=True) + return { + "data": merge_dicts(page["data"] for page in made), + "timestamps": merge_dicts(page["timestamps"] for page in made), + "time": [max(t) for t in times], + } @dataclass -class _KickoffCtx: +class DetectorCtx: + """What `prepare()` set up, held until the next `stage()`.""" + + #: The plan's TriggerInfo trigger_info: TriggerInfo - data_providers: Sequence[StreamableDataProvider] - collections_written: int - collections_requested: int - is_last_kickoff: bool + #: The detector name, which prefixes every datakey + name: str + #: Seconds per collection: `exposures_per_collection` times the livetime + #: plus deadtime, or what the hardware has set when the livetime is 0 and + #: its state is preserved + collection_period: float + #: The bounded data logics of this stage: those that returned a provider + #: when last started + bounded_logics: Sequence[BoundedDataLogic[Any]] = () + #: The unbounded data logics of this stage: those that returned a provider + #: when last started + unbounded_logics: Sequence[UnboundedDataLogic[Any]] = () + #: The providers `bounded_logics` started for this prepare, one per logic. + #: Restarted at each step point after the first + bounded_providers: Sequence[DataProvider] = () + #: The providers `unbounded_logics` started, one per logic. Carried over from + #: the previous prepare if made for the same collections per event, + #: collection period and flush period, and all have written the same whole + #: number of events + unbounded_providers: Sequence[DataProvider] = () + #: The datakeys of every provider, the same at every prepare of the stage + datakeys: dict[str, DataKey] = field(default_factory=dict) + #: What `DetectorSuppressLogic.hidden_datakeys` returned at the first + #: prepare of the stage, the same at every prepare of it + hidden: set[str] = field(default_factory=set) + #: The collections every unbounded data logic had written at the end of + #: prepare() + unbounded_start_collections: int = 0 + #: How many trigger()s this prepare has served + triggers: int = 0 + #: The readings of `DetectorLogic.get_readings` at the current step point, + #: `None` until read + readings: dict[str, Reading] | None = None + #: The acquire logic's `acquire()`, run to its yield by prepare() for + #: external triggering + external_acquire_gen: AsyncGenerator[None, None] | None = None + + @property + def providers(self) -> list[DataProvider]: + """Every serving provider, the unbounded ones first.""" + return [*self.unbounded_providers, *self.bounded_providers] + + +async def _datakeys(providers: Sequence[DataProvider]) -> dict[str, DataKey]: + """The datakeys of every provider, raising on one made by more than one.""" + return await merge_gathered_dicts(p.make_datakeys() for p in providers) + + +async def _stop_logics(logics: Sequence[DetectorDataLogic[Any]]) -> None: + """Stop every logic in `logics`.""" + await _gather_or_cancel_all([dl.ensure_stopped() for dl in logics]) + + +async def _start_unbounded(ctx: DetectorCtx) -> None: + """Start `ctx.unbounded_logics`, keeping those that return a provider.""" + info = ctx.trigger_info + providers = await _gather_or_cancel_all( + [ + dl.start_provider( + ctx.name, + info.collections_per_event, + ctx.collection_period, + info.flush_period, + ) + for dl in ctx.unbounded_logics + ] + ) + started = list(zip(ctx.unbounded_logics, providers, strict=True)) + ctx.unbounded_logics = [dl for dl, p in started if p] + ctx.unbounded_providers = [p for _, p in started if p] + + +async def _start_bounded(ctx: DetectorCtx) -> None: + """Start `ctx.bounded_logics`, keeping those that return a provider. + + Their providers leave out the datakeys the unbounded providers make. + """ + info = ctx.trigger_info + existing_datakeys = list(await _datakeys(ctx.unbounded_providers)) + providers = await _gather_or_cancel_all( + [ + dl.start_provider( + ctx.name, + existing_datakeys, + info.number_of_collections, + info.collections_per_event, + ctx.collection_period, + info.flush_period, + ) + for dl in ctx.bounded_logics + ] + ) + started = list(zip(ctx.bounded_logics, providers, strict=True)) + ctx.bounded_logics = [dl for dl, p in started if p] + ctx.bounded_providers = [p for _, p in started if p] + + +async def _restart_bounded(ctx: DetectorCtx, when: str) -> None: + """Stop and start `ctx.bounded_logics`, then check the datakeys are unchanged. + + :param when: when this happens, for the error message + """ + await _stop_logics(ctx.bounded_logics) + await _start_bounded(ctx) + before = ctx.datakeys + ctx.datakeys = await _datakeys(ctx.providers) + _check_datakeys(ctx, before, when) + + +def _check_datakeys(ctx: DetectorCtx, before: Collection[str], when: str) -> None: + """Raise unless the serving providers make the datakeys `before`.""" + if missing_or_extra := sorted(set(before) ^ ctx.datakeys.keys()): + raise RuntimeError( + f"The data logics made {sorted(ctx.datakeys)} at {when}, but " + f"{sorted(before)} before it, which differ in {missing_or_extra}: " + "a data logic that serves a stage must make the same datakeys for " + "all of it, so unstage() and stage() to change what it makes" + ) + + +def _made_for(ctx: DetectorCtx) -> tuple[str, int, float, float]: + """What an unbounded provider was made for: it carries over if unchanged.""" + info = ctx.trigger_info + return ( + ctx.name, + info.collections_per_event, + ctx.collection_period, + info.flush_period, + ) + + +async def _unbounded_written(ctx: DetectorCtx) -> list[int]: + """The collections each unbounded provider has written.""" + return await asyncio.gather( + *(p.collections_written.get_value() for p in ctx.unbounded_providers) + ) + + +def _carries_over( + previous_ctx: DetectorCtx, ctx: DetectorCtx, written: Sequence[int] +) -> bool: + """Whether the unbounded providers of `previous_ctx` can serve `ctx`. + + They can if made for the same scan, and the collections each has `written` + are the same whole number of events. + """ + cpe = ctx.trigger_info.collections_per_event + return ( + _made_for(previous_ctx) == _made_for(ctx) + and len(set(written)) <= 1 + and all(n % cpe == 0 for n in written) + ) + + +def _bounded_origin(ctx: DetectorCtx) -> int: + """The events written before the bounded providers were last started. + + The step points before the current one, as each point restarts them, and 0 + in a fly scan. + """ + return max(ctx.triggers - 1, 0) + + +def _collections_offset(ctx: DetectorCtx) -> dict[SignalR[int], int]: + """The offset from each provider's `collections_written` to the scan count. + + Added to `collections_written` it gives the collections since `prepare()`: + positive for a bounded data logic, which only counts from the current step + point, and negative for an unbounded one, which counts from before + `prepare()`. + """ + cpe = ctx.trigger_info.collections_per_event + bounded = _bounded_origin(ctx) * cpe + unbounded = -ctx.unbounded_start_collections + return { + **{p.collections_written: bounded for p in ctx.bounded_providers}, + **{p.collections_written: unbounded for p in ctx.unbounded_providers}, + } + + +def _provider_ends(ctx: DetectorCtx, index: int) -> Iterator[tuple[DataProvider, int]]: + """Each serving provider, with event `index` counted from its origin.""" + for provider in ctx.bounded_providers: + yield provider, index - _bounded_origin(ctx) + for provider in ctx.unbounded_providers: + yield provider, index + + +async def _no_acquisition() -> AsyncGenerator[None, None]: + """The acquisition of a detector with no acquire logic.""" + yield + + +def _preserve_detector_state() -> bool: + # Opt-in: set OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES to have the detector + # read back current hardware state via default_trigger_info(). See ADR 0013. + # TODO: flip default to YES and remove this guard in a future PR once + # downstream code has had time to implement default_trigger_info(). + return os.environ.get("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "NO").upper() == "YES" + + +class DetectorSuppressLogic: + """Base class for saying which readables a detector's data hides.""" + + @abstractmethod + async def hidden_datakeys(self) -> set[str]: + """The names of the readables the detector's data hides. + + Called at the first `prepare()` after `stage()`, and again at each later + one, which raises if the result differs. Each name is that of a readable + the detector registers. + """ + + +def _raise_if_hidden_changed(name: str, before: set[str], now: set[str]) -> None: + changed = sorted(before ^ now) + if changed: + raise RuntimeError( + f"{name}: hidden datakeys changed since the first prepare() after " + f"stage(): {changed}. unstage() and stage() to use the new values." + ) + + +@dataclass(frozen=True, kw_only=True, eq=False) +class DetectorLogic(FlyableLogic[TriggerInfo, DetectorCtx]): + """Drive a detector through its trigger, acquire and data logics. + + The fields are configuration and never change; what a scan sets up is held + in the context, so `dataclasses.replace` makes an independent copy. + + :param trigger: sets up exposures; without one only internal triggering + with the hardware's current timing is supported + :param acquire: starts and stops acquisition + :param data: the data logics, each publishing one part of the data + :param suppress: says which registered readables the data hides + """ + + trigger: DetectorTriggerLogic | None = None + acquire: DetectorAcquireLogic | None = None + data: Sequence[BoundedDataLogic[Any] | UnboundedDataLogic[Any]] = () + suppress: DetectorSuppressLogic | None = None + + def __post_init__(self) -> None: + object.__setattr__(self, "data", tuple(self.data)) + for dl in self.data: + if isinstance(dl, BoundedDataLogic) == isinstance(dl, UnboundedDataLogic): + raise TypeError( + f"{dl!r} must inherit from exactly one of BoundedDataLogic and " + "UnboundedDataLogic" + ) + + @cached_property + def supported_triggers(self) -> set[DetectorTrigger]: + """The trigger types the trigger logic implements.""" + if self.trigger is None: + return {DetectorTrigger.INTERNAL} + return _get_supported_triggers(self.trigger) + + @cached_property + def supports_deadtime(self) -> bool: + """Whether the trigger logic can calculate a deadtime.""" + return self.trigger is not None and _trigger_logic_supported( + self.trigger.get_deadtime + ) + + def with_device(self, name: str = "") -> "StandardDetector": + """Wrap this logic in an ephemeral `StandardDetector` for use in a plan.""" + return _LogicDetector(self, name=name) + + def get_deadtime(self, config_values: SignalDict) -> float: + """Return the deadtime the trigger logic calculates from `config_values`.""" + trigger = error_if_none(self.trigger, "No trigger logic") + return trigger.get_deadtime(config_values) + + async def default_trigger_info(self) -> TriggerInfo: + """The TriggerInfo to prepare with when a plan calls trigger() directly. + + The trigger logic's `default_trigger_info()` if + [](#OPHYD_ASYNC_PRESERVE_DETECTOR_STATE) is `YES`, otherwise a bare + [`TriggerInfo()`](#TriggerInfo). + """ + info = TriggerInfo() + if _preserve_detector_state() and self.trigger is not None: + if not _trigger_logic_supported(self.trigger.default_trigger_info): + raise RuntimeError( + "OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES is set but the " + "trigger logic has no default_trigger_info(): implement it on " + f"{type(self.trigger).__name__} or unset the environment " + "variable" + ) + info = await self.trigger.default_trigger_info() + return info + + async def ensure_ready(self) -> None: + """Make every data logic and the detector ready, in parallel.""" + await _gather_or_cancel_all( + [ + *(dl.ensure_ready() for dl in self.data), + *([self.acquire.ensure_ready()] if self.acquire else []), + ] + ) + + async def prepare_to_fly( + self, name: str, value: TriggerInfo, previous_ctx: DetectorCtx | None + ) -> AsyncGenerator[DetectorCtx | WatcherUpdate, None]: + """Set the trigger logic up, yield the context, then start the data. + + The context holds `value`, the collection period, which is read back + from the trigger logic when `value` leaves the livetime at 0, and what + the suppress logic hid. That is asked before the trigger logic is + prepared, and raises at a later prepare if it differs from the first. An + externally triggered detector is armed last, after the data logics are + started so their buffers are sized. + """ + hidden = await self._hidden_datakeys(name, previous_ctx) + await self._prepare_trigger_logic(value) + # After preparing the trigger logic, as the hardware then holds the + # values read back + collection_period = await self._resolve_collection_period(value) + ctx = DetectorCtx( + trigger_info=value, + name=name, + collection_period=collection_period, + hidden=hidden, + ) + yield ctx + await self._start_providers(ctx, previous_ctx) + if value.trigger is not DetectorTrigger.INTERNAL: + ctx.external_acquire_gen = await self._arm() + + async def _hidden_datakeys( + self, name: str, previous_ctx: DetectorCtx | None + ) -> set[str]: + hidden = ( + set() if self.suppress is None else await self.suppress.hidden_datakeys() + ) + if previous_ctx is not None: + _raise_if_hidden_changed(name, previous_ctx.hidden, hidden) + return hidden + + async def _start_providers( + self, ctx: DetectorCtx, previous_ctx: DetectorCtx | None + ) -> None: + """Start the data providers for this scan. + + At the first `prepare()` of a stage every unbounded data logic is + started, then every bounded one, leaving out the datakeys the unbounded + providers make. The logics that return a provider serve the rest of the + stage. At a later `prepare()` the unbounded providers of `previous_ctx` + carry over if they were made for the same name, collections per event, + collection period and flush period, and have all written the same whole + number of events, and are restarted otherwise; the bounded ones are + always restarted. Every unbounded provider then starts + new stream resources at `ctx.unbounded_start_collections`, so each run + has its own. + """ + if previous_ctx is None: + ctx.unbounded_logics = [ + dl for dl in self.data if isinstance(dl, UnboundedDataLogic) + ] + ctx.bounded_logics = [ + dl for dl in self.data if isinstance(dl, BoundedDataLogic) + ] + await _start_unbounded(ctx) + await _start_bounded(ctx) + ctx.datakeys = await _datakeys(ctx.providers) + else: + ctx.unbounded_logics = previous_ctx.unbounded_logics + ctx.bounded_logics = previous_ctx.bounded_logics + ctx.datakeys = previous_ctx.datakeys + ctx.unbounded_providers = previous_ctx.unbounded_providers + written = await _unbounded_written(ctx) + if not _carries_over(previous_ctx, ctx, written): + await _stop_logics(ctx.unbounded_logics) + await _start_unbounded(ctx) + await _restart_bounded(ctx, "this prepare()") + # TODO: delete this check if the run bundler learns to collect both from + # one device (bluesky/bluesky#2111), with the prepare() override. See ADR 0026. + if all(self.produces(ctx)): + pageable = await _datakeys( + [p for p in ctx.providers if isinstance(p, PageableDataProvider)] + ) + raise TypeError( + f"The detector would produce {sorted(pageable)} as event pages and " + "the rest of its data as stream assets; bluesky collects only one " + "of them from a detector, so its stream assets must make these " + "datakeys too, or a data logic must not serve this scan" + ) + # Every unbounded data logic has written the same: they all started + # together, after stage() stopped them all, or carried over having done so + written = await _unbounded_written(ctx) + counts = { + f"{type(dl).__name__}[{i}]": n + for i, (dl, n) in enumerate(zip(ctx.unbounded_logics, written, strict=True)) + } + ctx.unbounded_start_collections = ( + get_unique(counts, "numbers of collections written") if counts else 0 + ) + for provider in ctx.unbounded_providers: + provider.start_new_resources(ctx.unbounded_start_collections) + + async def fly( + self, ctx: DetectorCtx + ) -> AsyncGenerator[KICKED_OFF | WatcherUpdate, None]: + """Start the fly scan, then report events written until it finishes.""" + info = ctx.trigger_info + events = info.number_of_events + async with self._acquiring(ctx): + yield KICKED_OFF + async for update in self._wait_for( + ctx, 0, events, divisor=info.collections_per_event + ): + yield update + + async def step(self, ctx: DetectorCtx) -> AsyncGenerator[WatcherUpdate, None]: + """Take one step point's exposures and wait for them to be written. + + At each point after the first the bounded providers are restarted, so a + buffer holding the last point's data is cleared. Unbounded providers + keep counting. + """ + info = ctx.trigger_info + if info.number_of_events != 1: + raise ValueError( + "trigger() is not supported for multiple events, the detector was " + f"prepared with number_of_events={info.number_of_events}: prepare " + "with number_of_events=1 for a step scan" + ) + if ctx.triggers and info.trigger is not DetectorTrigger.INTERNAL: + raise RuntimeError( + "An externally triggered detector must be prepared again before " + "each trigger()" + ) + # prepare() started the bounded providers for the first point + if ctx.triggers: + await _restart_bounded(ctx, f"step point {ctx.triggers + 1}") + ctx.triggers += 1 + ctx.readings = None + async with self._acquiring(ctx): + async for update in self._wait_for( + ctx, ctx.triggers - 1, ctx.triggers, divisor=1 + ): + yield update + + async def ensure_stopped(self) -> None: + """Stop every data logic and the detector.""" + await _gather_or_cancel_all( + [ + *(dl.ensure_stopped() for dl in self.data), + *([self.acquire.ensure_stopped()] if self.acquire else []), + ] + ) + + def get_hinted_fields(self, name: str, ctx: DetectorCtx | None = None) -> list[str]: + """The hinted fields of the data logics, each once. + + :param name: the detector name, which prefixes every datakey, used if + `ctx` is `None` + :param ctx: the prepared context, whose serving logics are asked for + `ctx.name` and whose datakeys filter their fields; every data logic + is asked if `None` + """ + if ctx is None: + return [field for dl in self.data for field in dl.get_hinted_fields(name)] + fields = ( + field + for dl in (*ctx.unbounded_logics, *ctx.bounded_logics) + for field in dl.get_hinted_fields(ctx.name) + if field in ctx.datakeys + ) + # A bounded logic may hint a datakey an unbounded provider makes + return list(dict.fromkeys(fields)) + + def produces(self, ctx: DetectorCtx) -> tuple[bool, bool]: + """Whether the serving providers make (stream assets, event pages).""" + return ( + any(isinstance(p, StreamableDataProvider) for p in ctx.providers), + any(isinstance(p, PageableDataProvider) for p in ctx.providers), + ) + + async def get_readings(self, ctx: DetectorCtx) -> dict[str, Reading]: + """Read the current step point's event from each pageable provider. + + Makes the page up to the end of the point and reads its last event. The + readings are kept in `ctx`, so a repeat read at the same point returns + them again. + """ + if ctx.readings is None: + page = await self._make_page(ctx, ctx.triggers) + ctx.readings = ( + { + key: Reading( + value=values[-1], timestamp=page["timestamps"][key][-1] + ) + for key, values in page["data"].items() + } + if page + else {} + ) + return ctx.readings + + async def get_written_index(self, ctx: DetectorCtx) -> int: + """The events every serving data logic has written since `prepare()`. + + In a step scan this counts every point, though the bounded data logics + only hold the current one. + """ + offsets = _collections_offset(ctx) + written = await gather_dict({s: s.get_value() for s in offsets}) + collections = min((written[s] + o for s, o in offsets.items()), default=0) + return collections // ctx.trigger_info.collections_per_event + + async def _collect_end(self, ctx: DetectorCtx, index: int | None) -> int: + """The event a collect publishes up to, counted since `prepare()`. + + `index` if given, else in a step scan the step point `trigger()` waited + for, or `get_written_index` in a fly scan. + """ + if index is None: + index = ctx.triggers or await self.get_written_index(ctx) + return index + + async def _make_page(self, ctx: DetectorCtx, index: int) -> PartialEventPage | None: + """One page merging each pageable provider's events up to event `index`. + + `None` if no provider has new events. + """ + providers = [ + (p, end) + for p, end in _provider_ends(ctx, index) + if isinstance(p, PageableDataProvider) + ] + pages = await asyncio.gather(*(p.make_page(end) for p, end in providers)) + return _merge_pages( + [(p, page) for (p, _), page in zip(providers, pages, strict=True)] + ) + + async def asset_docs( + self, ctx: DetectorCtx, index: int | None + ) -> AsyncIterator[StreamAsset]: + """The stream assets of each streamable provider up to event `index`. + + `None` collects a step scan to the end of its point, and a fly scan up to + `get_written_index`. + """ + for p, end in _provider_ends(ctx, await self._collect_end(ctx, index)): + if isinstance(p, StreamableDataProvider): + async for doc in p.make_stream_docs(end): + yield doc + + async def event_pages( + self, ctx: DetectorCtx, index: int | None + ) -> AsyncIterator[PartialEventPage]: + """One page merging each pageable provider's events up to event `index`. + + `None` collects a step scan to the end of its point, and a fly scan up to + `get_written_index`. + """ + if page := await self._make_page(ctx, await self._collect_end(ctx, index)): + yield page + + async def _wait_for( + self, ctx: DetectorCtx, start: int, end: int, divisor: int + ) -> AsyncIterator[WatcherUpdate[int]]: + """Wait until every serving data logic has written up to event `end`. + + Yields the collections written since event `start`, divided by + `divisor`. Events are counted since `prepare()`. + """ + offsets = _collections_offset(ctx) + # A detector with no data logic has nothing to wait for + if not offsets: + return + cpe = ctx.trigger_info.collections_per_event + target = (end - start) * cpe + # Until each signal reports, count it as at `start` + collections = dict.fromkeys(offsets, start * cpe) + start_time = time.monotonic() + async for signal, value in observe_signals_value( + *offsets, timeout=ctx.trigger_info.collection_timeout + ): + collections[signal] = value + offsets[signal] + done = min(collections.values()) - start * cpe + yield WatcherUpdate( + name=ctx.name, + current=done // divisor, + initial=0, + target=target // divisor, + unit="", + precision=0, + time_elapsed=time.monotonic() - start_time, + ) + if done >= target: + break + + async def _arm(self) -> AsyncGenerator[None, None]: + """Run the acquire logic's `acquire()` to its yield, when it is armed.""" + acquisition = self.acquire.acquire() if self.acquire else _no_acquisition() + try: + await anext(acquisition) + except StopAsyncIteration: + raise RuntimeError( + f"{type(self.acquire).__name__}.acquire() finished without yielding " + "once armed" + ) from None + return acquisition + + @asynccontextmanager + async def _acquiring(self, ctx: DetectorCtx) -> AsyncGenerator[None, None]: + """Take the acquisition while the data is waited for. + + Takes the one armed by `prepare()` for external triggering, else arms + one now. Runs it to the end after a clean exit, and closes it otherwise. + """ + acquisition = ctx.external_acquire_gen or await self._arm() + # Taken once: a later step point arms afresh + ctx.external_acquire_gen = None + async with aclosing(acquisition): + yield + async for _ in acquisition: + raise RuntimeError( + f"{type(self.acquire).__name__}.acquire() must yield only once" + ) + + async def _prepare_trigger_logic(self, value: TriggerInfo) -> None: + """Set the trigger logic up for `value`, raising if it cannot do it.""" + if value.trigger not in self.supported_triggers: + supported = ", ".join(sorted(t.name for t in self.supported_triggers)) + raise ValueError( + f"Trigger type {value.trigger} not supported, supported types are: " + f"[{supported}]" + ) + if not self.trigger: + if value.livetime or value.deadtime or value.exposures_per_collection != 1: + raise ValueError( + "No trigger logic, so cannot set livetime or deadtime " + "or exposures_per_collection" + ) + return + if _trigger_logic_supported(self.trigger.prepare_exposures_per_collection): + # Set up even for 1, to clear a previous setting + await self.trigger.prepare_exposures_per_collection( + value.exposures_per_collection + ) + elif value.exposures_per_collection != 1: + raise ValueError( + "Multiple exposures per collection not supported: the trigger " + "logic must implement prepare_exposures_per_collection()" + ) + match value.trigger: + case DetectorTrigger.INTERNAL: + await self.trigger.prepare_internal( + num=value.number_of_exposures, + livetime=value.livetime, + deadtime=value.deadtime, + ) + case DetectorTrigger.EXTERNAL_EDGE: + await self.trigger.prepare_edge( + num=value.number_of_exposures, livetime=value.livetime + ) + case DetectorTrigger.EXTERNAL_LEVEL: + await self.trigger.prepare_level(num=value.number_of_exposures) + + async def _resolve_collection_period(self, value: TriggerInfo) -> float: + # A livetime of 0 means "use whatever the detector currently has set", + # which a data logic sizing chunks or a buffer by the period needs to + # know. Reading it back is reading hardware state, so is opt-in. + period = value.livetime + value.deadtime + if ( + value.livetime == 0.0 + and _preserve_detector_state() + and self.trigger is not None + and _trigger_logic_supported(self.trigger.default_trigger_info) + ): + current = await self.trigger.default_trigger_info() + if current.livetime or current.deadtime: + period = current.livetime + current.deadtime + return value.exposures_per_collection * period + + +def _config_signals(device: Device) -> set[SignalR]: + """Return every `Signal` contributing to a Device's `read_configuration()`. + + Recurses into children registered as [](#StandardReadableFormat.CHILD), so + a Device that registers a sub-Device whole picks up whatever that + sub-Device declares as configuration. A Device that is not a + `StandardReadable` contributes nothing. + + This reads the registry rather than calling `read_configuration()`, so it + does no I/O and includes read-only `SignalR`s as well as `SignalRW`s. + """ + signals: set[SignalR] = set() + if not isinstance(device, StandardReadable): + return signals + for entry in device.get_readable_formats(): + match entry: + case (signal, StandardReadableFormat.CONFIG_SIGNAL): + signals.add(signal) + case (child, StandardReadableFormat.CHILD): + signals |= _config_signals(child) + case _: + pass + return signals class StandardDetector( StandardReadable, + StandardFlyable[TriggerInfo, DetectorCtx], Stageable, Triggerable, - Preparable, - Flyable, Collectable, - WritesStreamAssets, + HasHints, ): """Detector base class for step and fly scanning detectors. - Aggregates trigger, arm, reading or stream logic together. + Aggregates trigger, acquire and data logic together in a + [](#DetectorLogic), which a subclass builds in `logic`: + + ```python + class MyDetector(StandardDetector): + def __init__(self, prefix: str, name: str = "") -> None: + self.driver = MyDriverIO(prefix) + super().__init__(name=name) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=MyTriggerLogic(self.driver), + acquire=MyAcquireLogic(self.driver), + ) + ``` + + For an ad-hoc detector, [](#DetectorLogic.with_device) does the same thing + without a subclass. Signals read during a step scan are registered with [](#StandardReadable.set_readable_format), exactly as on any other `StandardReadable`; data produced by a `DetectorDataLogic` is added on top - of those in `read()` and `describe()`. - - `read()` and `describe()` require `prepare()` to have run, and raise - otherwise, so that a descriptor can never be emitted without the detector's - data keys. `trigger()` prepares implicitly, so a step scan never has to do - it explicitly. `read_configuration()` and `describe_configuration()` have no - such requirement. + of those in `read()` and `describe()`, replacing a registered signal of the + same name, shape and dtype. + + Every verb but `stage()`, `read_configuration()` and + `describe_configuration()` is for after `stage()`. A step scan triggers and + reads, and `trigger()` prepares implicitly if needed; a fly scan prepares, + kicks off, completes and collects. `read()` is for after a `trigger()`, the + collect verbs for after a `kickoff()`, and `describe()`, + `describe_collect()` and `get_index()` for after a `prepare()`. An + externally triggered detector is prepared again before each `trigger()`. + After any verb raises, `unstage()` and `stage()` it before using it again. + + `WritesStreamAssets` (and its `collect_asset_docs`) and + `EventPageCollectable` (and its `collect_pages`) are *not* inherited: which + of them applies depends on what the data logics produce for a given scan, so + whichever it is gets bound as an instance attribute by `prepare()`. """ - # Logic for the detector - _trigger_logic: DetectorTriggerLogic | None = None - _acquire_logic: DetectorAcquireLogic | None = None - _data_logics: Sequence[DetectorDataLogic] = () - # Context produced by prepare, used by trigger and kickoff - _prepare_ctx: _PrepareCtx | None = None - # Context produced by kickoff, used by complete - _kickoff_ctx: _KickoffCtx | None = None - # The triggers that are supported by the trigger logic - _supported_triggers: set[DetectorTrigger] = {DetectorTrigger.INTERNAL} - def __init__(self, *args, **kwargs) -> None: super().__init__(*args, **kwargs) - # Contribute to _StandardBase's fan out rather than replacing stage(), - # so the readable children registered on this detector still get staged - self._stage_funcs += (self._stage_detector,) - self._unstage_funcs += (self._unstage_detector,) # What the data logics produce, gathered alongside the registered # children, so the verb methods themselves need no overriding self._read_funcs += (self.__on_read,) self._describe_funcs += (self.__on_describe,) self._hints_funcs += (self.__on_hints,) + self._suppressed_datakeys_funcs += (self.__on_suppressed_datakeys,) + self._unstage_funcs += (self.__on_unstage,) - # Report the number of events for the next kickoff @cached_property - def events_to_kickoff(self) -> SignalRW[int]: - # TODO: only allow this to be revised down when trigger_info.number_of_events >1 - # and we have a reusable data provider - # requries https://github.com/bluesky/ophyd-async/issues/1119 - signal = soft_signal_rw(int) - # Name and parent this manually as `Device` doesn't know how to deal with cached - # properties - signal.parent = self - signal.set_name(f"{self.name}-events_to_kickoff") - return signal - - def add_detector_logics( - self, *logics: DetectorTriggerLogic | DetectorAcquireLogic | DetectorDataLogic - ) -> None: - """Add acquire, trigger or data logic to the detector. - - :param logic: The logic to add - """ - for logic in logics: - if isinstance(logic, DetectorTriggerLogic): - if self._trigger_logic is not None: - raise RuntimeError("Detector already has trigger logic") - self._trigger_logic = logic - # Store the triggers that are supported - self._supported_triggers = _get_supported_triggers(logic) - elif isinstance(logic, DetectorAcquireLogic): - if self._acquire_logic is not None: - raise RuntimeError("Detector already has acquire logic") - self._acquire_logic = logic - elif isinstance(logic, DetectorDataLogic): - self._data_logics = (*self._data_logics, logic) - else: - raise TypeError(f"Unknown logic type: {type(logic)}") + def _state(self) -> _StateMachine: + return super()._state | { + _State.PREPARED: {_State.TRIGGERING}, + _State.TRIGGERING: {_State.TRIGGERED, _State.STAGED}, + _State.TRIGGERED: {_State.TRIGGERING, _State.PREPARING}, + } + + @abstract_cached_property + @abstractmethod + def logic(self) -> DetectorLogic: + """The logic that drives this detector.""" + raise NotImplementedError + + async def __on_unstage(self) -> None: + # Close an acquisition prepare() armed and nothing took. This does not + # stop the detector, the acquire logic's ensure_stopped() does that: it + # only ends the generator's tasks now, rather than when it is garbage + # collected, which asyncio warns about + if self._ctx and self._ctx.external_acquire_gen: + await self._ctx.external_acquire_gen.aclose() + + def __on_suppressed_datakeys(self) -> Mapping[str, DataKey | None]: + """The datakeys of the serving providers and what the suppress logic hides.""" + if self._ctx is None: + return {} + return {**self._ctx.datakeys, **dict.fromkeys(self._ctx.hidden)} + + async def __on_describe(self) -> dict[str, DataKey]: + """What the data logics contribute to `describe()`. + + Every serving provider describes its shape for a step scan, whichever + way it goes on to produce the data. + """ + return self._prepared_ctx("describe").datakeys + + async def __on_read(self) -> dict[str, Reading]: + """What the data logics contribute to `read()`: pageable data only. + + Streamable providers produce their data through `collect_asset_docs`. + """ + self._state.check_one_of(_State.TRIGGERED, verb="read") + return await self.logic.get_readings(self._prepared_ctx("read")) + + def __on_hints(self) -> Iterator[Hints]: + """The data logics' hinted fields, alongside the children's.""" + if fields := self.logic.get_hinted_fields(self.name, self._ctx): + yield {"fields": fields} + + def prepare(self, value: TriggerInfo) -> WatchableAsyncStatus[Any]: + """Set up the scan as `StandardFlyable.prepare`, then bind the collect verb. + + The collect verb matching what the serving providers produce is bound as + a real instance attribute, so the bundler's isinstance checks + (`WritesStreamAssets` vs `EventPageCollectable`) see exactly one of them. + """ + # TODO: delete this override once the run bundler picks the collect verb + # from what a device produced (bluesky/bluesky#2111). See ADR 0026. + status = super().prepare(value) + # Added first, so it runs before whoever waits on the status + status.add_callback(self._bind_collect_verb) + return status + + def _bind_collect_verb(self, status: Status) -> None: + if not status.success: + return + # Those checks resolve with inspect.getattr_static on Python 3.12+, which + # does not call __getattr__, so a dynamic hook would be invisible; both + # names are reserved by Device, hence object.__setattr__. + streamable, pageable = self.logic.produces(self._prepared_ctx("prepare")) + for verb, method, wanted in ( + ("collect_asset_docs", self._collect_asset_docs, streamable), + ("collect_pages", self._collect_pages, pageable), + ): + if wanted: + object.__setattr__(self, verb, method) + elif verb in self.__dict__: + object.__delattr__(self, verb) + + @WatchableAsyncStatus.wrap + async def trigger(self) -> AsyncIterator[WatcherUpdate[int]]: + """Trigger a single exposure. + + If [`prepare()`](#StandardFlyable.prepare) has not been called since the + last `stage()`, an implicit prepare is performed. When + [](#OPHYD_ASYNC_PRESERVE_DETECTOR_STATE) is `YES` + [](#DetectorTriggerLogic.default_trigger_info) is called to read current + hardware state; otherwise a bare [`TriggerInfo()`](#TriggerInfo) is used. + `unstage()` cancels a running trigger, and a failed one leaves the detector + staged. + """ + if self._state.state is _State.STAGED: + info = await self.logic.default_trigger_info() + await self.prepare(info) + async with self._state.during( + _State.TRIGGERING, end=_State.TRIGGERED, on_error=_State.STAGED + ): + async for update in self.logic.step(self._prepared_ctx("trigger")): + yield update + + async def describe_collect(self) -> dict[str, DataKey]: + return self._prepared_ctx("describe_collect").datakeys + + async def _collect_asset_docs( + self, index: int | None = None + ) -> AsyncIterator[StreamAsset]: + # Bound as collect_asset_docs when there is a streamable provider + self._state.check_one_of( + _State.TRIGGERED, _State.FLYING, _State.FLOWN, verb="collect_asset_docs" + ) + async for doc in self.logic.asset_docs( + self._prepared_ctx("collect_asset_docs"), index + ): + yield doc + + async def _collect_pages( + self, index: int | None = None + ) -> AsyncIterator[PartialEventPage]: + # Bound as collect_pages when there is a pageable provider + self._state.check_one_of(_State.FLYING, _State.FLOWN, verb="collect_pages") + async for page in self.logic.event_pages( + self._prepared_ctx("collect_pages"), index + ): + yield page + + async def get_index(self) -> int: + return await self.logic.get_written_index(self._prepared_ctx("get_index")) # Back compat - delete before 1.0 def add_config_signals(self, *signals: SignalR) -> None: @@ -464,17 +1368,13 @@ async def get_trigger_deadtime( """Get supported trigger types and deadtime for the detector. The trigger logic is given the value of every signal this detector - reports as configuration, rather than nominating the ones it wants: - the values that determine deadtime are exactly the ones that ought to - be recorded for the scan anyway, so a second list to keep in step - earned nothing. + reports as configuration. :param settings: Optional settings to use when getting configuration values :return: Tuple of supported trigger types and deadtime in seconds """ - if self._trigger_logic and _trigger_logic_supported( - self._trigger_logic.get_deadtime - ): + deadtime = None + if self.logic.supports_deadtime: config_values = SignalDict() to_read: list[SignalR] = [] for sig in _config_signals(self): @@ -484,338 +1384,24 @@ async def get_trigger_deadtime( config_values[sig] = settings[cast(SignalRW, sig)] else: to_read.append(sig) - # Read live values concurrently: this is now every configuration - # signal rather than a handful the logic named, so doing it in - # series would be one round trip per signal + # Read live values concurrently for sig, value in zip( to_read, await asyncio.gather(*(sig.get_value() for sig in to_read)), strict=True, ): config_values[sig] = value - deadtime = self._trigger_logic.get_deadtime(config_values) - else: - deadtime = None - return self._supported_triggers, deadtime - - @AsyncStatus.wrap - async def _stage_detector(self) -> None: - """Make sure the detector is idle and ready to be used.""" - coros: list[Awaitable] = [data_logic.stop() for data_logic in self._data_logics] - if self._acquire_logic: - coros.append(self._acquire_logic.ensure_ready()) - await asyncio.gather(*coros) - self._prepare_ctx = None - self._kickoff_ctx = None - await self.events_to_kickoff.set(0) - - async def _update_prepare_context(self, trigger_info: TriggerInfo) -> None: - # The only thing that would stop us being able to reuse a provider is - # if the collections_per_event changes, as that would change the - # StreamResource shape. All other TriggerInfo parameters (exposures, livetime, - # etc.) don't affect the data provider configuration. - if ( - self._prepare_ctx - and self._prepare_ctx.trigger_info.collections_per_event - == trigger_info.collections_per_event - ): - # Reuse the existing data providers - streamable_data_providers = self._prepare_ctx.streamable_data_providers - else: - # Stop the existing providers if there is a context and make new ones - if self._prepare_ctx: - for data_logic in self._data_logics: - await data_logic.stop() - # Setup the data logic for the right number of collections - streamable_coros: list[Awaitable[StreamableDataProvider]] = [] - for dl in self._data_logics: - if not _data_logic_supported(dl.prepare_unbounded): - msg = f"DataLogic hasn't overridden any prepare_* methods {dl}" - raise RuntimeError(msg) - streamable_coros.append( - dl.prepare_unbounded(self.name + dl.datakey_suffix) - ) - streamable_data_providers = await asyncio.gather(*streamable_coros) - - # Stash the prepare context so we can use it in trigger/kickoff - self._prepare_ctx = _PrepareCtx( - trigger_info=trigger_info, - streamable_data_providers=streamable_data_providers, - collections_written=await _get_collections_written( - streamable_data_providers - ), - ) + deadtime = self.logic.get_deadtime(config_values) + return self.logic.supported_triggers, deadtime - async def _wait_for_index( - self, - data_providers: Sequence[StreamableDataProvider], - trigger_info: TriggerInfo, - initial_collections_written: int, - collections_requested: int, - wait_for_idle: bool, - watcher_divisor: int = 1, - ) -> AsyncIterator[WatcherUpdate]: - start_time = time.monotonic() - current_collections_written = { - dp.collections_written_signal: initial_collections_written - for dp in data_providers - } - target_collections_written = initial_collections_written + collections_requested - if data_providers: - async for sig, value in observe_signals_value( - *current_collections_written.keys(), - timeout=trigger_info.collection_timeout, - ): - current_collections_written[sig] = value - collections_written = min(current_collections_written.values()) - yield WatcherUpdate( - name=self.name, - current=collections_written // watcher_divisor, - initial=initial_collections_written // watcher_divisor, - target=target_collections_written // watcher_divisor, - unit="", - precision=0, - time_elapsed=time.monotonic() - start_time, - ) - if collections_written >= target_collections_written: - break - if self._acquire_logic and wait_for_idle: - await self._acquire_logic.wait_for_idle() - @AsyncStatus.wrap - async def prepare(self, value: TriggerInfo) -> None: - """Prepare the detector for a number of triggers. +class _LogicDetector(StandardDetector): + """A concrete `StandardDetector` that wraps a logic object.""" - :param value: TriggerInfo describing how to trigger the detector - """ - if self._trigger_logic and _trigger_logic_supported( - self._trigger_logic.prepare_exposures_per_collection - ): - # If we can do multiple exposures per collection then set it up - # even if there was only 1 requested to clear previous settings - await self._trigger_logic.prepare_exposures_per_collection( - value.exposures_per_collection - ) - elif value.exposures_per_collection != 1: - raise ValueError( - f"Multiple exposures per collection not supported by {self}" - ) - # Setup the trigger logic for the right number of exposures - if value.trigger not in self._supported_triggers: - format_triggers = ", ".join( - sorted(t.name for t in self._supported_triggers) - ) - raise ValueError( - f"Trigger type {value.trigger} not supported by '{self.name}', " - f"supported types are: [{format_triggers}]" - ) - if self._trigger_logic: - match value.trigger: - case DetectorTrigger.INTERNAL: - await self._trigger_logic.prepare_internal( - num=value.number_of_exposures, - livetime=value.livetime, - deadtime=value.deadtime, - ) - case DetectorTrigger.EXTERNAL_EDGE: - await self._trigger_logic.prepare_edge( - num=value.number_of_exposures, - livetime=value.livetime, - ) - case DetectorTrigger.EXTERNAL_LEVEL: - await self._trigger_logic.prepare_level( - num=value.number_of_exposures, - ) - elif value.livetime != 0.0 or value.deadtime != 0.0: - raise ValueError( - f"Detector {self.name} has no trigger logic, so cannot set livetime or " - "deadtime" - ) - # NOTE: this section must come after preparing the trigger logic as we may - # use parameters from it to determine datatype for the streams - await self._update_prepare_context(value) - # Tell people how many collections we will acquire for - await self.events_to_kickoff.set(value.number_of_events) - # External triggering can start acquiring now - if self._acquire_logic and value.trigger != DetectorTrigger.INTERNAL: - await self._acquire_logic.start_acquiring() - - @WatchableAsyncStatus.wrap - async def trigger(self) -> AsyncIterator[WatcherUpdate[int]]: - """Trigger a single exposure. + def __init__(self, logic: DetectorLogic, name: str = "") -> None: + self._logic = logic + super().__init__(name=name) - If [`prepare()`](#StandardDetector.prepare) has not been called since - the last `stage()`, an implicit prepare is - performed. When [](#OPHYD_ASYNC_PRESERVE_DETECTOR_STATE) is `YES` - [](#DetectorTriggerLogic.default_trigger_info) is called to read current - hardware state; otherwise a bare [`TriggerInfo()`](#TriggerInfo) is - used. - """ - if self._prepare_ctx is None: - # Opt-in: set OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES to have - # trigger() read back current hardware state (e.g. num_images) via - # default_trigger_info() instead of always falling back to TriggerInfo(). - # See ADR 0013 for rationale. - # TODO: flip default to YES and remove this guard in a future PR once - # downstream code has had time to implement default_trigger_info(). - preserve_state = ( - os.environ.get("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "NO").upper() - == "YES" - ) - if preserve_state and self._trigger_logic is not None: - if not _trigger_logic_supported( - self._trigger_logic.default_trigger_info - ): - raise RuntimeError( - f"OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES is set but " - f"'{self.name}' has no default_trigger_info() - implement " - "default_trigger_info() on your DetectorTriggerLogic subclass " - "or unset the environment variable." - ) - trigger_info = await self._trigger_logic.default_trigger_info() - else: - trigger_info = TriggerInfo() - await self.prepare(trigger_info) - else: - # Check the one that was provided is suitable for triggering - trigger_info = self._prepare_ctx.trigger_info - if trigger_info.number_of_events != 1: - msg = ( - "trigger() is not supported for multiple events, the detector was " - f"prepared with number_of_events={trigger_info.number_of_events}." - ) - raise ValueError(msg) - # Ensure the data provider is still usable - await self._update_prepare_context(trigger_info) - ctx = error_if_none(self._prepare_ctx, "Prepare should have been run") - # Start the detector acquiring and wait for it to finish. - if self._acquire_logic: - await self._acquire_logic.start_acquiring() - async for update in self._wait_for_index( - data_providers=ctx.streamable_data_providers, - trigger_info=ctx.trigger_info, - initial_collections_written=ctx.collections_written, - collections_requested=ctx.trigger_info.collections_per_event, - watcher_divisor=1, - wait_for_idle=True, - ): - yield update - - @AsyncStatus.wrap - async def kickoff(self): - ctx = error_if_none(self._prepare_ctx, "Prepare not called") - if not ctx.streamable_data_providers: - raise ValueError( - f"Detector {self.name} is not streamable, so cannot kickoff" - ) - collections_written, events_to_kickoff = await asyncio.gather( - _get_collections_written(ctx.streamable_data_providers), - self.events_to_kickoff.get_value(), - ) - collections_requested = ( - events_to_kickoff * ctx.trigger_info.collections_per_event - ) - last_requested_collection = collections_written + collections_requested - last_expected_collection = ( - ctx.collections_written + ctx.trigger_info.number_of_collections - ) - if last_requested_collection > last_expected_collection: - msg = ( - f"Kickoff requested {collections_written}:{last_requested_collection}, " - f"but detector was only prepared up to {last_expected_collection}" - ) - raise RuntimeError(msg) - self._kickoff_ctx = _KickoffCtx( - trigger_info=ctx.trigger_info, - data_providers=ctx.streamable_data_providers, - collections_written=collections_written, - collections_requested=collections_requested, - is_last_kickoff=last_requested_collection == last_expected_collection, - ) - # External triggering has already started; internal starts now - if self._acquire_logic and ctx.trigger_info.trigger == DetectorTrigger.INTERNAL: - await self._acquire_logic.start_acquiring() - - @WatchableAsyncStatus.wrap - async def complete(self): - ctx = error_if_none(self._kickoff_ctx, "Kickoff not called") - async for update in self._wait_for_index( - data_providers=ctx.data_providers, - trigger_info=ctx.trigger_info, - initial_collections_written=ctx.collections_written, - collections_requested=ctx.collections_requested, - wait_for_idle=ctx.is_last_kickoff, - watcher_divisor=ctx.trigger_info.collections_per_event, - ): - yield update - - async def __on_read(self) -> dict[str, Reading]: - """What the data logics contribute to `read()`: nothing, but checks prepare. - - Raising when nothing has been prepared is deliberate, and predates - registered children being readable at all: a detector that has not been - prepared has no data keys. In practice `trigger()` prepares implicitly, - so `trigger_and_read` never reaches this. - """ - error_if_none(self._prepare_ctx, "Prepare not run") - return {} - - async def __on_describe(self) -> dict[str, DataKey]: - """What the data logics contribute to `describe()`. - - Raises when nothing has been prepared, as a `describe()` that quietly - succeeded would emit a descriptor missing the detector's data. - Streamable providers describe their shape for a step scan, but produce - their data through collect_asset_docs rather than read(). - """ - ctx = error_if_none(self._prepare_ctx, "Prepare not run") - return await merge_gathered_dicts( - sdp.make_datakeys(ctx.trigger_info.collections_per_event) - for sdp in ctx.streamable_data_providers - ) - - async def describe_collect(self) -> dict[str, DataKey]: - ctx = error_if_none(self._prepare_ctx, "Prepare not run") - # Only streamable data providers produce data during collect - coros = [ - dp.make_datakeys(ctx.trigger_info.collections_per_event) - for dp in ctx.streamable_data_providers - ] - return await merge_gathered_dicts(coros) - - def __on_hints(self) -> Iterator[Hints]: - """The data logics' hinted fields, alongside the children's.""" - for dl in self._data_logics: - if fields := dl.get_hinted_fields(self.name + dl.datakey_suffix): - yield {"fields": list(fields)} - - async def collect_asset_docs( - self, index: int | None = None - ) -> AsyncIterator[StreamAsset]: - # Collect stream datum documents for all indices written. - ctx = error_if_none(self._prepare_ctx, "Prepare not called") - if index is None: - # The index is optional, and provided for fly scans, if there is - # more than one detector to make sure they collect in step - index = await self.get_index() - for data_provider in ctx.streamable_data_providers: - async for doc in data_provider.make_stream_docs( - collections_written=index * ctx.trigger_info.collections_per_event, - collections_per_event=ctx.trigger_info.collections_per_event, - ): - yield doc - - async def get_index(self) -> int: - ctx = error_if_none(self._prepare_ctx, "Prepare not called") - min_collections_written = await _get_collections_written( - ctx.streamable_data_providers, reducer=min - ) - return min_collections_written // ctx.trigger_info.collections_per_event - - @AsyncStatus.wrap - async def _unstage_detector(self) -> None: - """Stop the detector and file writing.""" - coros: list[Awaitable] = [data_logic.stop() for data_logic in self._data_logics] - if self._acquire_logic: - coros.append(self._acquire_logic.ensure_stopped()) - await asyncio.gather(*coros) + @cached_property + def logic(self) -> DetectorLogic: + return self._logic diff --git a/src/ophyd_async/core/_readable.py b/src/ophyd_async/core/_readable.py index 55c8e0f3a3..ce2ea5b307 100644 --- a/src/ophyd_async/core/_readable.py +++ b/src/ophyd_async/core/_readable.py @@ -4,15 +4,17 @@ from collections.abc import ( Awaitable, Callable, + Collection, Generator, Iterator, + Mapping, MutableMapping, Sequence, ) from contextlib import contextmanager from enum import Enum from functools import cached_property -from typing import Literal +from typing import Literal, cast from bluesky.protocols import HasHints, Hints, Reading from event_model import DataKey @@ -21,7 +23,7 @@ from ._protocol import AsyncConfigurable, AsyncReadable, AsyncStageable from ._signal import SignalR, walk_devices from ._standard_base import _StandardBase -from ._utils import merge_gathered_dicts +from ._utils import merge_dicts, merge_gathered_dicts class StandardReadableFormat(Enum): @@ -113,9 +115,30 @@ def __iter__(self) -> Iterator[Device]: def __len__(self) -> int: return len(self._entries) - def entries(self) -> list[ReadableEntry]: - """A snapshot of each child and its format, in registration order.""" - return list(self._entries.values()) + def entries(self, without: Collection[str] = ()) -> list[ReadableEntry]: + """A snapshot of each child and its format, in registration order. + + :param without: names of signals to leave out + """ + return [ + entry + for entry in self._entries.values() + if entry[1] is Format.CHILD or entry[0].name not in without + ] + + +def _switched_off_names(suppressed: Mapping[str, DataKey | None]) -> set[str]: + """The names that are switched off, which have no replacement.""" + return {name for name, datakey in suppressed.items() if datakey is None} + + +def _without_fields(hints: Hints, names: Collection[str]) -> Hints: + """`hints` with `names` left out of its fields.""" + fields = hints.get("fields") + if isinstance(fields, str) or not isinstance(fields, Sequence): + # Absent or malformed: hints reports the malformed ones + return hints + return cast(Hints, {**hints, "fields": [n for n in fields if n not in names]}) class _SealDefaultFormatsMeta(type(_StandardBase)): # type: ignore[misc] @@ -181,6 +204,14 @@ def _readables(self) -> _Readables: Callable[[], Awaitable[dict[str, DataKey]]], ... ] = () _hints_funcs: tuple[Callable[[], Iterator[Hints]], ...] = () + # The names of registered children that are not read for now, by name so a + # child's own children are covered. A DataKey means something else provides + # that name instead: the child is dropped from read() and describe() after + # checking it against the DataKey, and kept in hints. None means the name is + # switched off: it is dropped from read(), describe() and hints. + _suppressed_datakeys_funcs: tuple[ + Callable[[], Mapping[str, DataKey | None]], ... + ] = () def __init__(self, *args, **kwargs) -> None: super().__init__(*args, **kwargs) @@ -258,10 +289,21 @@ async def __on_read_configuration(self) -> dict[str, Reading]: pass return await merge_gathered_dicts(coros) + def _suppressed_datakeys(self) -> dict[str, DataKey | None]: + return merge_dicts(func() for func in self._suppressed_datakeys_funcs) + async def __on_describe(self) -> dict[str, DataKey]: - """What the registered children contribute to `describe()`.""" + """What the registered children contribute to `describe()`. + + Every signal with a replacement is described too, to check the + replacement can stand in for it. + """ coros = [] - for entry in self._readables.entries(): + suppressed = self._suppressed_datakeys() + replacements = { + name: datakey for name, datakey in suppressed.items() if datakey is not None + } + for entry in self._readables.entries(without=_switched_off_names(suppressed)): match entry: case (device, Format.CHILD): if isinstance(device, AsyncReadable): @@ -275,12 +317,29 @@ async def __on_describe(self) -> dict[str, DataKey]: coros.append(signal.describe()) case (_, Format.CONFIG_SIGNAL): pass - return await merge_gathered_dicts(coros) + datakeys = await merge_gathered_dicts(coros) + # A replacement needs the signal's dtype, and its shape or that shape with + # one extra leading dimension, e.g. one per collection of an event + mismatched = [] + for key in sorted(datakeys.keys() & replacements.keys()): + signal, data = datakeys[key], replacements[key] + same_dtype = signal.get("dtype_numpy") == data.get("dtype_numpy") + same_shape = signal["shape"] in (data["shape"], data["shape"][1:]) + if not (same_dtype and same_shape): + mismatched.append(key) + if mismatched: + raise ValueError( + f"{self.name}: duplicate datakeys {mismatched}: registered signals " + "with a different shape or dtype from the data that would replace " + "them, so rename one of them" + ) + return {k: v for k, v in datakeys.items() if k not in suppressed} async def __on_read(self) -> dict[str, Reading]: """What the registered children contribute to `read()`.""" coros = [] - for entry in self._readables.entries(): + suppressed = self._suppressed_datakeys() + for entry in self._readables.entries(without=suppressed): match entry: case (device, Format.CHILD): if isinstance(device, AsyncReadable): @@ -294,15 +353,18 @@ async def __on_read(self) -> dict[str, Reading]: coros.append(signal.read(cached=False)) case (_, Format.CONFIG_SIGNAL): pass - return await merge_gathered_dicts(coros) + readings = await merge_gathered_dicts(coros) + # entries() leaves out suppressed signals, but not those read by a child + return {k: v for k, v in readings.items() if k not in suppressed} def __on_hints(self) -> Iterator[Hints]: """What the registered children contribute to `hints`.""" - for entry in self._readables.entries(): + switched_off = _switched_off_names(self._suppressed_datakeys()) + for entry in self._readables.entries(without=switched_off): match entry: case (device, Format.CHILD): if isinstance(device, HasHints): - yield device.hints + yield _without_fields(device.hints, switched_off) case ( signal, Format.HINTED_SIGNAL | Format.HINTED_UNCACHED_SIGNAL, @@ -519,44 +581,12 @@ def add_readables( self.set_readable_format(device, format) -def _config_signals(device: Device) -> set[SignalR]: - """Return every `Signal` contributing to a Device's `read_configuration()`. - - Recurses into children registered as [](#StandardReadableFormat.CHILD), so - a Device that registers a sub-Device whole picks up whatever that - sub-Device declares as configuration. A Device that is not a - `StandardReadable` contributes nothing. - - Unlike `walk_config_signals` this reads the registry rather than calling - `read_configuration()`, so it does no I/O and includes read-only `SignalR`s - as well as `SignalRW`s (`ADBaseIO.model` is a `SignalR`, and it is exactly - the signal an areaDetector trigger logic needs for deadtime). - """ - signals: set[SignalR] = set() - if not isinstance(device, StandardReadable): - return signals - for entry in device.get_readable_formats(): - match entry: - case (signal, Format.CONFIG_SIGNAL): - signals.add(signal) - case (child, Format.CHILD): - signals |= _config_signals(child) - case _: - pass - return signals - - -#: Reserved key under which [](#store_settings) writes readable formats. Chosen -#: so that it cannot be a dotted attribute path: `<` cannot start a Python -#: identifier, so no attribute assignment can produce a colliding key, and -#: unlike `*FORMATS*` it needs no quoting in yaml. +#: Reserved key under which [](#store_settings) writes readable formats, which +#: cannot be a dotted attribute path. READABLE_FORMATS_KEY = "" #: The key standing in for the root Device itself in a [](#ReadableFormats), -#: where every other key is a dotted attribute path. Not a valid Python -#: identifier, so no attribute assignment can produce a colliding path, and -#: unlike `""` it does not make yaml fall back to its hard-to-read explicit -#: key syntax. +#: where every other key is a dotted attribute path. ROOT_DEVICE_KEY = "" #: The changes to the readable formats of a Device tree since construction, in diff --git a/src/ophyd_async/epics/adandor.py b/src/ophyd_async/epics/adandor.py index 95d4c7963c..02dbbd0173 100644 --- a/src/ophyd_async/epics/adandor.py +++ b/src/ophyd_async/epics/adandor.py @@ -17,7 +17,7 @@ from ophyd_async.epics.adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -36,7 +36,6 @@ ] _MIN_DEAD_TIME = 0.1 -_MAX_NUM_IMAGE = 999_999 class Andor2TriggerMode(StrictEnum): @@ -76,19 +75,17 @@ def get_deadtime(self, config_values: SignalDict) -> float: async def prepare_internal(self, num: int, livetime: float, deadtime: float): await self.driver.trigger_mode.set(Andor2TriggerMode.INTERNAL) - await prepare_exposures(self.driver, num or _MAX_NUM_IMAGE, livetime, deadtime) + await prepare_exposures(self.driver, num, livetime, deadtime) async def prepare_edge(self, num: int, livetime: float): await self.driver.trigger_mode.set(Andor2TriggerMode.EXT_TRIGGER) - await prepare_exposures(self.driver, num or _MAX_NUM_IMAGE, livetime) + await prepare_exposures(self.driver, num, livetime) async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class AndorDetector(AreaDetector[Andor2DriverIO]): @@ -108,7 +105,7 @@ class AndorDetector(AreaDetector[Andor2DriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix="cam1:", proc_suffix: str | None = None, plugins: dict[str, NDPluginBaseIO] | None = None, diff --git a/src/ophyd_async/epics/adaravis.py b/src/ophyd_async/epics/adaravis.py index 26595e279c..ed215a1da8 100644 --- a/src/ophyd_async/epics/adaravis.py +++ b/src/ophyd_async/epics/adaravis.py @@ -19,7 +19,7 @@ from .adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -87,9 +87,7 @@ async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class AravisDetector(AreaDetector[AravisDriverIO]): @@ -112,7 +110,7 @@ class AravisDetector(AreaDetector[AravisDriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix: str = "cam1:", proc_suffix: str | None = None, override_deadtime: float | None = None, diff --git a/src/ophyd_async/epics/adcore/__init__.py b/src/ophyd_async/epics/adcore/__init__.py index be4307c00e..5c2a17cfea 100644 --- a/src/ophyd_async/epics/adcore/__init__.py +++ b/src/ophyd_async/epics/adcore/__init__.py @@ -5,6 +5,7 @@ from ._acquire_logic import ADAcquireLogic, ADContAcqAcquireLogic from ._data_logic import ( + ADDataFactory, ADHDFDataLogic, ADMultipartDataLogic, ADWriterFactory, @@ -96,6 +97,7 @@ "NDArrayDescription", "ADHDFDataLogic", "ADMultipartDataLogic", + "ADDataFactory", "ADWriterFactory", # Detector "AreaDetector", diff --git a/src/ophyd_async/epics/adcore/_acquire_logic.py b/src/ophyd_async/epics/adcore/_acquire_logic.py index 6c6b18e231..83180bad15 100644 --- a/src/ophyd_async/epics/adcore/_acquire_logic.py +++ b/src/ophyd_async/epics/adcore/_acquire_logic.py @@ -1,8 +1,8 @@ import asyncio +from collections.abc import AsyncGenerator from ophyd_async.core import ( DEFAULT_TIMEOUT, - AsyncStatus, DetectorAcquireLogic, EnableDisable, SignalR, @@ -25,7 +25,7 @@ class ADAcquireLogic(DetectorAcquireLogic): Whether the detector should wait for its plugins to drain before it reports acquisition as complete. `WaitForPlugins` makes the driver's `AcquireBusy` wait for the number of queued arrays to reach zero, and - `wait_for_idle` blocks on the `acquire` put callback, so turning it on + `acquire` waits for the `acquire` put callback, so turning it on is what makes a step scan read plugin scalars for the frame it just took rather than the one before. Defaults to on, since racing is silent and the wait costs nothing when a file writer is the slowest @@ -48,7 +48,6 @@ def __init__( self.driver_armed_signal = driver.acquire self.wait_for_plugins = wait_for_plugins self.timeout = timeout - self.acquire_status: AsyncStatus | None = None async def ensure_ready(self): # Stop first: WaitForPlugins is latched and has no _RBV, so set it once @@ -59,19 +58,17 @@ async def ensure_ready(self): await self.ensure_stopped() await self.driver.wait_for_plugins.set(self.wait_for_plugins) - async def start_acquiring(self): - self.acquire_status = await set_and_wait_for_other_value( + async def acquire(self) -> AsyncGenerator[None, None]: + async with await set_and_wait_for_other_value( set_signal=self.driver.acquire, set_value=True, match_signal=self.driver_armed_signal, match_value=True, wait_for_set_completion=False, timeout=self.timeout, - ) - - async def wait_for_idle(self): - if self.acquire_status: - await self.acquire_status + ) as status: + yield + await status await wait_for_good_state( self.driver.detector_state, {ADState.IDLE, ADState.ABORTED}, @@ -91,8 +88,8 @@ class ADContAcqAcquireLogic(DetectorAcquireLogic): Deliberately has no `wait_for_plugins` equivalent to [](#ADAcquireLogic). That setting gates the driver's `AcquireBusy` going to 0 at the end of an acquisition, and here the driver is always acquiring, so there is no such - transition to wait on. `wait_for_idle` waits on the circular buffer's - capture instead. + transition to wait on. `acquire` waits on the circular buffer's trigger + instead. :param driver: The continuously acquiring detector driver :param cb_plugin: The circular buffer plugin gating downstream plugins @@ -149,7 +146,7 @@ async def ensure_ready(self): timeout=DEFAULT_TIMEOUT, ) - async def start_acquiring(self): + async def acquire(self) -> AsyncGenerator[None, None]: num_frames_to_skip = self.num_frames_to_skip if num_frames_to_skip: initial_count, acquire_period = await asyncio.gather( @@ -163,8 +160,7 @@ async def start_acquiring(self): timeout=num_frames_to_skip * acquire_period + DEFAULT_TIMEOUT, ) await self.cb_plugin.trigger_.set(True) - - async def wait_for_idle(self): + yield num_images, acquire_period = await asyncio.gather( self.cb_plugin.post_count.get_value(), self.acquire_period_signal.get_value(), diff --git a/src/ophyd_async/epics/adcore/_data_logic.py b/src/ophyd_async/epics/adcore/_data_logic.py index bc11d4cc8c..97c76cff41 100644 --- a/src/ophyd_async/epics/adcore/_data_logic.py +++ b/src/ophyd_async/epics/adcore/_data_logic.py @@ -1,5 +1,6 @@ import asyncio -from collections.abc import Callable, Sequence +import warnings +from collections.abc import Awaitable, Callable, Sequence from dataclasses import dataclass from pathlib import PureWindowsPath from typing import Any, Generic @@ -8,15 +9,17 @@ import numpy as np from ophyd_async.core import ( - DetectorDataLogic, + DEFAULT_TIMEOUT, + BoundedDataLogic, EnableDisable, PathInfo, PathProvider, SignalR, - StreamableDataProvider, StreamResourceDataProvider, StreamResourceInfo, + UnboundedDataLogic, set_and_wait_for_value, + wait_for_value, ) from ophyd_async.epics.core import stop_busy_record @@ -28,8 +31,9 @@ NDArrayBaseIO, NDFileHDF5IO, NDPluginBaseIO, + NDPluginBaseIOT, NDPluginFileIO, - NDPluginFileIOT, + plugin_is_enabled, ) from ._ndattribute import NDAttributeDataType, NDAttributePvDbrType @@ -144,8 +148,14 @@ async def prepare_file_paths( await writer.num_capture.set(0) +async def _start_capture(writer: NDPluginFileIO) -> None: + await set_and_wait_for_value(writer.capture, True, wait_for_set_completion=False) + # Capturing resets NumCaptured, which the detector reads as its baseline + await wait_for_value(writer.num_captured, 0, timeout=DEFAULT_TIMEOUT) + + @dataclass -class ADHDFDataLogic(DetectorDataLogic): +class ADHDFDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): """Data logic for AreaDetector HDF5 writer plugin. :param array_description: Signals describing the NDArray shape and data type. @@ -164,73 +174,88 @@ class ADHDFDataLogic(DetectorDataLogic): plugins: Sequence[NDPluginBaseIO] = () datakey_suffix: str = "" hinted: bool = True + #: False sits out the scans of a stage if the plugin's EnableCallbacks PV is + #: off at the first prepare. True serves every scan. Starting switches it on. + always_enable: bool = False - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: - # Work out where to write + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider | None: + if not self.always_enable and not await plugin_is_enabled(self.writer): + return None + datakey_name = name + self.datakey_suffix path_info = self.path_provider(datakey_name) - # Determine number of frames that will be saved per HDF chunk. - # On a fresh IOC startup, this is set to zero until the first capture, - # so if it is zero, set it to 1. - frames_per_chunk = await self.writer.num_frames_chunks.get_value() - if frames_per_chunk == 0: - frames_per_chunk = 1 - await self.writer.num_frames_chunks.set(frames_per_chunk) - # Setup the HDF writer - await asyncio.gather( - self.writer.chunk_size_auto.set(True), - self.writer.num_extra_dims.set(0), - self.writer.lazy_open.set(True), - self.writer.swmr_mode.set(True), - self.writer.xml_file_name.set(""), - self.writer.enable_callbacks.set(EnableDisable.ENABLE), - prepare_file_paths( - path_info=path_info, file_template="%s%s.h5", writer=self.writer + # Size the HDF chunk to hold a flush period of collections, e.g. 400 Hz + # (2.5 ms) with a 0.5 s flush period gives 200 frames per chunk (#1309). + # Otherwise use the chunk size set on the IOC, forcing a fresh-startup 0 + # to 1. + if flush_period > 0 and collection_period > 0: + frames_per_chunk = max(1, round(flush_period / collection_period)) + else: + frames_per_chunk = await self.writer.num_frames_chunks.get_value() or 1 + main_dataset, ndattribute_dtype_sources = await asyncio.gather( + get_ndarray_resource_info( + array_description=self.array_description, + data_key=datakey_name, + parameters={"dataset": "/entry/data/data"}, + frames_per_chunk=frames_per_chunk, + ), + get_ndattribute_dtype_source( + (self.driver, *self.plugins) if self.driver else (*self.plugins,) ), - ) - # Start capturing - await set_and_wait_for_value( - self.writer.capture, True, wait_for_set_completion=False - ) - # Return a provider that reflects what we have made - main_dataset = await get_ndarray_resource_info( - array_description=self.array_description, - data_key=datakey_name, - parameters={"dataset": "/entry/data/data"}, - frames_per_chunk=frames_per_chunk, - ) - ndattribute_dtype_sources = await get_ndattribute_dtype_source( - (self.driver, *self.plugins) if self.driver else (*self.plugins,) ) ndattribute_datasets = [ StreamResourceInfo( - data_key=name, + data_key=attr_name, shape=(), # NDAttributes appear to always be configured with # this chunk size chunk_shape=(16384,), dtype_numpy=dtype_numpy, source=source, - parameters={"dataset": f"/entry/instrument/NDAttributes/{name}"}, + parameters={"dataset": f"/entry/instrument/NDAttributes/{attr_name}"}, ) - for name, (dtype_numpy, source) in ndattribute_dtype_sources.items() + for attr_name, (dtype_numpy, source) in ndattribute_dtype_sources.items() ] + # Write back the chunk size the main dataset is described with, so the + # file matches the chunk_shape in its stream resource either way + await self.writer.num_frames_chunks.set(frames_per_chunk) + # Setup the HDF writer + coros: list[Awaitable] = [ + self.writer.chunk_size_auto.set(True), + self.writer.num_extra_dims.set(0), + self.writer.lazy_open.set(True), + self.writer.swmr_mode.set(True), + self.writer.xml_file_name.set(""), + prepare_file_paths( + path_info=path_info, file_template="%s%s.h5", writer=self.writer + ), + self.writer.enable_callbacks.set(EnableDisable.ENABLE), + ] + await asyncio.gather(*coros) + await _start_capture(self.writer) return StreamResourceDataProvider( uri=f"{path_info.directory_uri}{path_info.filename}.h5", resources=[main_dataset] + ndattribute_datasets, mimetype="application/x-hdf5", - collections_written_signal=self.writer.num_captured, + collections_written=self.writer.num_captured, + collections_per_event=collections_per_event, flush_signal=self.writer.flush_now, ) - async def stop(self) -> None: + async def ensure_stopped(self) -> None: await stop_busy_record(self.writer.capture) - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: - return [datakey_name] if self.hinted else [] + def get_hinted_fields(self, name: str) -> Sequence[str]: + return [name + self.datakey_suffix] if self.hinted else [] @dataclass -class ADMultipartDataLogic(DetectorDataLogic): +class ADMultipartDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): """Data logic for multipart AreaDetector file writers (e.g. JPEG, TIFF). :param array_description: Signals describing the NDArray shape and data type. @@ -241,6 +266,10 @@ class ADMultipartDataLogic(DetectorDataLogic): Mimetype for the written files (e.g. "multipart/related;type=image/jpeg"). :param datakey_suffix: Suffix to append to the data key for the main dataset :param hinted: Include the image in hints. Set False for BestEffortCallback. + :param always_enable: + False sits out the scans of a stage if the plugin's EnableCallbacks PV + is off at the first prepare. True serves every scan. Starting switches + it on. """ array_description: NDArrayDescription @@ -250,10 +279,26 @@ class ADMultipartDataLogic(DetectorDataLogic): mimetype: str datakey_suffix: str = "" hinted: bool = True + always_enable: bool = False - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: - # Work out where to write + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider | None: + # A multipart writer writes one file per frame, so there is nothing to + # size from the periods + if not self.always_enable and not await plugin_is_enabled(self.writer): + return None + datakey_name = name + self.datakey_suffix path_info = self.path_provider(datakey_name) + main_dataset = await get_ndarray_resource_info( + array_description=self.array_description, + data_key=datakey_name, + parameters={"template": path_info.filename + "_{:06d}" + self.extension}, + ) # Setup the file writer await asyncio.gather( self.writer.lazy_open.set(True), @@ -262,43 +307,36 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: file_template="%s%s_%6.6d" + self.extension, writer=self.writer, ), + self.writer.enable_callbacks.set(EnableDisable.ENABLE), ) - # Start capturing - await set_and_wait_for_value( - self.writer.capture, True, wait_for_set_completion=False - ) - # Return a provider that reflects what we have made - main_dataset = await get_ndarray_resource_info( - array_description=self.array_description, - data_key=datakey_name, - parameters={"template": path_info.filename + "_{:06d}" + self.extension}, - ) + await _start_capture(self.writer) return StreamResourceDataProvider( uri=path_info.directory_uri, resources=[main_dataset], mimetype=self.mimetype, - collections_written_signal=self.writer.num_captured, + collections_written=self.writer.num_captured, + collections_per_event=collections_per_event, ) - async def stop(self) -> None: + async def ensure_stopped(self) -> None: await stop_busy_record(self.writer.capture) - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: - return [datakey_name] if self.hinted else [] + def get_hinted_fields(self, name: str) -> Sequence[str]: + return [name + self.datakey_suffix] if self.hinted else [] @dataclass -class ADWriterFactory(Generic[NDPluginFileIOT]): - """Factory that creates a file-writer plugin and its matching data logic. +class ADDataFactory(Generic[NDPluginBaseIOT]): + """Factory that creates a plugin and its matching data logic. - Construct using the classmethods `hdf`, `jpeg`, or `tiff`, then pass one + Construct using the static methods `hdf`, `jpeg` or `tiff`, then pass one or more instances to `AreaDetector` as positional `*writer_factories` - arguments. When the detector is initialised `__call__` is invoked with - the detector's PV `prefix`, its `driver`, and the flat list of extra - `plugins`; it returns the writer device and the corresponding - `DetectorDataLogic`. + arguments. When the detector is initialised + `__call__` is invoked with the detector's PV `prefix`, its `driver`, and the + flat list of extra `plugins`; it returns the writer device and the + corresponding data logic. - :param writer_cls: Concrete `NDPluginFileIO` subclass to instantiate. + :param writer_cls: Concrete `NDPluginBaseIO` subclass to instantiate. :param writer_suffix: PV suffix appended to *prefix* to form the writer's PV prefix. :param writer_name: Attribute name under which the writer device is stored on the @@ -313,11 +351,11 @@ class ADWriterFactory(Generic[NDPluginFileIOT]): depends on signals that are only available once the driver has been constructed (e.g. inside a detector subclass that creates its own driver). :param data_logic_factory: - Callable ``(writer, array_description, driver, plugins) → DetectorDataLogic`` + Callable `(writer, array_description, driver, plugins) → data logic` that builds the data logic given the already-constructed writer. """ - writer_cls: type[NDPluginFileIOT] + writer_cls: type[NDPluginBaseIOT] writer_suffix: str writer_name: str datakey_suffix: str @@ -325,8 +363,8 @@ class ADWriterFactory(Generic[NDPluginFileIOT]): NDArrayDescription | Callable[[ADBaseIO], NDArrayDescription] | None ) data_logic_factory: Callable[ - [NDPluginFileIOT, NDArrayDescription, ADBaseIO, Sequence[NDPluginBaseIO]], - DetectorDataLogic, + [NDPluginBaseIOT, NDArrayDescription, ADBaseIO, Sequence[NDPluginBaseIO]], + BoundedDataLogic | UnboundedDataLogic, ] def __call__( @@ -334,7 +372,7 @@ def __call__( prefix: str, driver: ADBaseIO, plugins: Sequence[NDPluginBaseIO], - ) -> tuple[NDPluginFileIOT, DetectorDataLogic]: + ) -> tuple[NDPluginBaseIOT, BoundedDataLogic | UnboundedDataLogic]: """Instantiate the writer plugin and build the data logic. :param prefix: EPICS PV prefix for the detector (same as `AreaDetector.prefix`). @@ -371,7 +409,8 @@ def hdf( | Callable[[ADBaseIO], NDArrayDescription] | None = None, hinted: bool = True, - ) -> "ADWriterFactory[NDFileHDF5IO]": + always_enable: bool = False, + ) -> "ADDataFactory[NDFileHDF5IO]": """Create a factory for an HDF5 file writer. :param path_provider: Provides file path information for each acquisition. @@ -387,8 +426,12 @@ def hdf( when the shape/type comes from a plugin rather than the main driver (e.g. an ROI plugin). :param hinted: Include the image in hints. Set False for BestEffortCallback. + :param always_enable: False (the default) sits out the scans of a stage + if the plugin's EnableCallbacks PV is off at the first prepare, so + another data logic can serve them. True serves every scan. Starting + switches the plugin on either way. """ - return ADWriterFactory( + return ADDataFactory( writer_cls=NDFileHDF5IO, writer_suffix=writer_suffix, writer_name=writer_name, @@ -402,6 +445,7 @@ def hdf( plugins=list(plugins), datakey_suffix=datakey_suffix, hinted=hinted, + always_enable=always_enable, ), ) @@ -415,7 +459,8 @@ def jpeg( | Callable[[ADBaseIO], NDArrayDescription] | None = None, hinted: bool = True, - ) -> "ADWriterFactory[NDPluginFileIO]": + always_enable: bool = False, + ) -> "ADDataFactory[NDPluginFileIO]": """Create a factory for a JPEG file writer. :param path_provider: Provides file path information for each acquisition. @@ -430,8 +475,12 @@ def jpeg( Pass an `NDArrayDescription` or a callable ``(driver) → NDArrayDescription`` when the shape/type comes from a plugin. :param hinted: Include the image in hints. Set False for BestEffortCallback. + :param always_enable: False (the default) sits out the scans of a stage + if the plugin's EnableCallbacks PV is off at the first prepare, so + another data logic can serve them. True serves every scan. Starting + switches the plugin on either way. """ - return ADWriterFactory( + return ADDataFactory( writer_cls=NDPluginFileIO, writer_suffix=writer_suffix, writer_name=writer_name, @@ -446,6 +495,7 @@ def jpeg( mimetype="multipart/related;type=image/jpeg", datakey_suffix=datakey_suffix, hinted=hinted, + always_enable=always_enable, ) ), ) @@ -460,7 +510,8 @@ def tiff( | Callable[[ADBaseIO], NDArrayDescription] | None = None, hinted: bool = True, - ) -> "ADWriterFactory[NDPluginFileIO]": + always_enable: bool = False, + ) -> "ADDataFactory[NDPluginFileIO]": """Create a factory for a TIFF file writer. :param path_provider: Provides file path information for each acquisition. @@ -475,8 +526,12 @@ def tiff( Pass an `NDArrayDescription` or a callable ``(driver) → NDArrayDescription`` when the shape/type comes from a plugin. :param hinted: Include the image in hints. Set False for BestEffortCallback. + :param always_enable: False (the default) sits out the scans of a stage + if the plugin's EnableCallbacks PV is off at the first prepare, so + another data logic can serve them. True serves every scan. Starting + switches the plugin on either way. """ - return ADWriterFactory( + return ADDataFactory( writer_cls=NDPluginFileIO, writer_suffix=writer_suffix, writer_name=writer_name, @@ -491,6 +546,38 @@ def tiff( mimetype="multipart/related;type=image/tiff", datakey_suffix=datakey_suffix, hinted=hinted, + always_enable=always_enable, ) ), ) + + +class ADWriterFactory(ADDataFactory[NDPluginBaseIOT]): + """Deprecated name of `ADDataFactory`.""" + + def __post_init__(self) -> None: + _warn_writer_factory() + + @staticmethod + def hdf(*args, **kwargs) -> ADDataFactory[NDFileHDF5IO]: + _warn_writer_factory() + return ADDataFactory.hdf(*args, **kwargs) + + @staticmethod + def jpeg(*args, **kwargs) -> ADDataFactory[NDPluginFileIO]: + _warn_writer_factory() + return ADDataFactory.jpeg(*args, **kwargs) + + @staticmethod + def tiff(*args, **kwargs) -> ADDataFactory[NDPluginFileIO]: + _warn_writer_factory() + return ADDataFactory.tiff(*args, **kwargs) + + +def _warn_writer_factory() -> None: + warnings.warn( + DeprecationWarning( + "'ADWriterFactory' is deprecated, use 'ADDataFactory' instead" + ), + stacklevel=3, + ) diff --git a/src/ophyd_async/epics/adcore/_detector.py b/src/ophyd_async/epics/adcore/_detector.py index 2a1cb72b2e..5bf7759d3b 100644 --- a/src/ophyd_async/epics/adcore/_detector.py +++ b/src/ophyd_async/epics/adcore/_detector.py @@ -1,16 +1,20 @@ from collections.abc import Mapping, Sequence +from functools import cached_property from typing import Generic from ophyd_async.core import ( + BoundedDataLogic, DetectorAcquireLogic, + DetectorLogic, DetectorTriggerLogic, SignalR, StandardDetector, + UnboundedDataLogic, ) from ophyd_async.core import StandardReadableFormat as Format from ._acquire_logic import ADContAcqAcquireLogic -from ._data_logic import ADWriterFactory +from ._data_logic import ADDataFactory from ._io import ( ADBaseIO, ADBaseIOT, @@ -27,7 +31,7 @@ def __init__( self, driver: ADBaseIOT, prefix: str | None = None, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, acquire_logic: DetectorAcquireLogic | None = None, trigger_logic: DetectorTriggerLogic | None = None, plugins: Mapping[str, NDPluginBaseIO] | None = None, @@ -37,11 +41,10 @@ def __init__( self.driver = driver if plugins is not None: for plugin_name, plugin in plugins.items(): - setattr(self, plugin_name, plugin) - if trigger_logic: - self.add_detector_logics(trigger_logic) - if acquire_logic: - self.add_detector_logics(acquire_logic) + self._set_unique(plugin_name, plugin) + self._trigger_logic = trigger_logic + self._acquire_logic = acquire_logic + data_logics: list[BoundedDataLogic | UnboundedDataLogic] = [] if writer_factories: if prefix is None: raise ValueError("prefix is required when writer_factories are given") @@ -54,8 +57,9 @@ def __init__( plugin_list = list(plugins.values()) if plugins else [] for factory in writer_factories: writer, data_logic = factory(prefix, driver, plugin_list) - setattr(self, factory.writer_name, writer) - self.add_detector_logics(data_logic) + self._set_unique(factory.writer_name, writer) + data_logics.append(data_logic) + self._data_logics = tuple(data_logics) # The driver is always wired up, so register it whole and let ADBaseIO # declare which of its signals are configuration. Plugins are # deliberately *not* registered: a detector normally carries far more @@ -66,6 +70,22 @@ def __init__( self.set_readable_format(signal, Format.CONFIG_SIGNAL) super().__init__(name=name) + def _set_unique(self, attr: str, value: NDPluginBaseIO) -> None: + if attr in self.__dict__ or hasattr(type(self), attr): + raise ValueError( + f"Cannot add {type(value).__name__} as attribute '{attr}': " + f"{type(self).__name__} already has an attribute with that name" + ) + setattr(self, attr, value) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=self._trigger_logic, + acquire=self._acquire_logic, + data=self._data_logics, + ) + def get_plugin( self, name: str, plugin_type: type[NDPluginBaseIOT] = NDPluginBaseIO ) -> NDPluginBaseIOT: @@ -119,12 +139,13 @@ class ContAcqDetector(AreaDetector[ADBaseIOT]): :param driver_cls: IO class for the driver, defaults to ADBaseIO :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param cb_suffix: Suffix for the circular buffer plugin PV, defaults to "CB1:" + :param cb_plugin_name: Attribute name of the circular buffer plugin :param proc_suffix: Suffix for the NDProcessIO plugin PV, if any, defaults to None :param num_frames_to_skip: Incoming frames to skip before each CB gate opening, defaults to 0 :param plugins: Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + proc_suffix instead. So is the name cb_plugin_name. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ @@ -132,10 +153,11 @@ class ContAcqDetector(AreaDetector[ADBaseIOT]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_cls: type[ADBaseIOT] = ADBaseIO, driver_suffix: str = "cam1:", cb_suffix: str = "CB1:", + cb_plugin_name: str = "cb", proc_suffix: str | None = None, num_frames_to_skip: int = 0, plugins: dict[str, NDPluginBaseIO] | None = None, @@ -143,15 +165,21 @@ def __init__( name: str = "", ) -> None: driver = driver_cls(prefix + driver_suffix) - cb_plugin = NDCircularBuffIO(prefix + cb_suffix) + cb_plugin = NDCircularBuffIO(prefix + cb_suffix, name=cb_plugin_name) plugins, proc = plugins_with_proc(plugins, prefix, proc_suffix) + if cb_plugin_name in plugins: + msg = ( + "Pass cb_plugin_name to name the circular buffer plugin, " + f"not plugins[{cb_plugin_name!r}]" + ) + raise ValueError(msg) super().__init__( driver, prefix, *writer_factories, acquire_logic=ADContAcqAcquireLogic(driver, cb_plugin, num_frames_to_skip), trigger_logic=ADContAcqTriggerLogic(driver, cb_plugin, proc), - plugins={**plugins, "cb": cb_plugin}, + plugins={**plugins, cb_plugin_name: cb_plugin}, config_sigs=config_sigs, name=name, ) diff --git a/src/ophyd_async/epics/adcore/_io.py b/src/ophyd_async/epics/adcore/_io.py index 65ec984eed..38d3d7ef51 100644 --- a/src/ophyd_async/epics/adcore/_io.py +++ b/src/ophyd_async/epics/adcore/_io.py @@ -169,6 +169,17 @@ class NDPluginBaseIO(NDArrayBaseIO): queue_size: A[SignalRW[int], PvSuffix.rbv("QueueSize")] +async def plugin_is_enabled(plugin: NDPluginBaseIO) -> bool: + """Whether a plugin is switched on, so is receiving NDArrays. + + The nearest thing to "is this plugin in the chain": a plugin with callbacks + disabled certainly is not, though an enabled one may still have its + `nd_array_port` pointed somewhere else. Parsing the whole plugin graph was + rejected in ADR 0023 as too large. + """ + return await plugin.enable_callbacks.get_value() is EnableDisable.ENABLE + + class NDROIIO(StandardReadable, NDPluginBaseIO): """Plugin for taking a region of an NDArray. diff --git a/src/ophyd_async/epics/adcore/_trigger_logic.py b/src/ophyd_async/epics/adcore/_trigger_logic.py index efb511d12d..536daa9a04 100644 --- a/src/ophyd_async/epics/adcore/_trigger_logic.py +++ b/src/ophyd_async/epics/adcore/_trigger_logic.py @@ -2,13 +2,11 @@ from dataclasses import dataclass from ophyd_async.core import ( - DetectorTrigger, - EnableDisable, - SignalR, - TriggerInfo, + DetectorTriggerLogic as _DetectorTriggerLogic, ) from ophyd_async.core import ( - DetectorTriggerLogic as _DetectorTriggerLogic, + EnableDisable, + TriggerInfo, ) from ._io import ( @@ -82,30 +80,47 @@ async def prepare_filter( ) -async def trigger_info_from_driver( - num_images_signal: SignalR[int], - process_plugin: NDProcessIO | None = None, -) -> TriggerInfo: - """Default TriggerInfo for AD detectors, reading num_images from the driver. +async def _exposures_per_collection_from_filter( + process_plugin: NDProcessIO | None, +) -> int: + """Read the exposures per collection from the process plugin's filter. - :param num_images_signal: SignalR for the number of images to acquire - :param process_plugin: Optional NDProcessIO plugin for exposures per collection + :param process_plugin: The NDProcessIO plugin instance, if any + :return: `num_filter` if the filter is enabled, otherwise 1 + :raises ValueError: If the filter is enabled with `num_filter` below 1 """ - num_images = await num_images_signal.get_value() - exposures_per_collection = 1 + if process_plugin is None or not await process_plugin.enable_filter.get_value(): + return 1 + num_filter = await process_plugin.num_filter.get_value() + if num_filter < 1: + raise ValueError( + f"{process_plugin.name}: filter enabled with " + f"num_filter={num_filter}, set it to 1 or more" + ) + return num_filter - if process_plugin is not None and await process_plugin.enable_filter.get_value(): - exposures_per_collection = await process_plugin.num_filter.get_value() - if exposures_per_collection < 1: - raise ValueError( - f"{process_plugin.name}: filter enabled with ", - f"num_filter={exposures_per_collection}, set it to 1 or more", - ) +async def trigger_info_from_driver( + driver: ADBaseIO, process_plugin: NDProcessIO | None = None +) -> TriggerInfo: + """Read the default TriggerInfo for AD detectors back from the driver. + + :param driver: The driver to read num_images, acquire_time and acquire_period from + :param process_plugin: The NDProcessIO plugin filtering exposures, if any + """ + num_images, livetime, period = await asyncio.gather( + driver.num_images.get_value(), + driver.acquire_time.get_value(), + driver.acquire_period.get_value(), + ) + exposures_per_collection = await _exposures_per_collection_from_filter( + process_plugin + ) return TriggerInfo( collections_per_event=max(1, num_images // exposures_per_collection), exposures_per_collection=exposures_per_collection, - trigger=DetectorTrigger.INTERNAL, + livetime=livetime, + deadtime=max(0.0, period - livetime), ) @@ -149,6 +164,17 @@ async def default_trigger_info(self) -> TriggerInfo: # Read post_count (not driver.num_images) because the CB plugin's # post_count is what governs how many frames are buffered per trigger # in continuous-acquisition mode, not the driver's num_images. - return await trigger_info_from_driver( - self.cb_plugin.post_count, self.process_plugin + post_count, livetime, period = await asyncio.gather( + self.cb_plugin.post_count.get_value(), + self.driver.acquire_time.get_value(), + self.driver.acquire_period.get_value(), + ) + exposures_per_collection = await _exposures_per_collection_from_filter( + self.process_plugin + ) + return TriggerInfo( + collections_per_event=max(1, post_count // exposures_per_collection), + exposures_per_collection=exposures_per_collection, + livetime=livetime, + deadtime=max(0.0, period - livetime), ) diff --git a/src/ophyd_async/epics/adkinetix.py b/src/ophyd_async/epics/adkinetix.py index bca7aa36d4..e84320951a 100644 --- a/src/ophyd_async/epics/adkinetix.py +++ b/src/ophyd_async/epics/adkinetix.py @@ -19,7 +19,7 @@ from .adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, plugins_with_proc, @@ -88,9 +88,7 @@ async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, num_filter=exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class KinetixDetector(AreaDetector[KinetixDriverIO]): @@ -110,7 +108,7 @@ class KinetixDetector(AreaDetector[KinetixDriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix: str = "cam1:", proc_suffix: str | None = None, plugins: dict[str, NDPluginBaseIO] | None = None, diff --git a/src/ophyd_async/epics/admerlin.py b/src/ophyd_async/epics/admerlin.py index c6a09295ef..33b783b381 100644 --- a/src/ophyd_async/epics/admerlin.py +++ b/src/ophyd_async/epics/admerlin.py @@ -22,7 +22,7 @@ ADAcquireLogic, ADBaseDataType, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -113,9 +113,7 @@ async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class MerlinDetector(AreaDetector[MerlinDriverIO]): @@ -135,7 +133,7 @@ class MerlinDetector(AreaDetector[MerlinDriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix="cam1:", proc_suffix: str | None = None, plugins: dict[str, NDPluginBaseIO] | None = None, diff --git a/src/ophyd_async/epics/adpilatus.py b/src/ophyd_async/epics/adpilatus.py index a6cd11ab87..6bbe70d5a2 100644 --- a/src/ophyd_async/epics/adpilatus.py +++ b/src/ophyd_async/epics/adpilatus.py @@ -21,7 +21,7 @@ from .adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -41,8 +41,6 @@ "PilatusReadoutTime", ] -_MAX_NUM_IMAGE = 999_999 - class PilatusTriggerMode(StrictEnum): """Trigger modes for ADPilatus detector.""" @@ -115,23 +113,21 @@ def get_deadtime(self, config_values: SignalDict) -> float: async def prepare_internal(self, num: int, livetime: float, deadtime: float): await self.driver.trigger_mode.set(PilatusTriggerMode.INTERNAL) - await prepare_exposures(self.driver, num or _MAX_NUM_IMAGE, livetime, deadtime) + await prepare_exposures(self.driver, num, livetime, deadtime) async def prepare_edge(self, num: int, livetime: float): await self.driver.trigger_mode.set(self.edge_trigger_mode) - await prepare_exposures(self.driver, num or _MAX_NUM_IMAGE, livetime) + await prepare_exposures(self.driver, num, livetime) async def prepare_level(self, num: int): await self.driver.trigger_mode.set(PilatusTriggerMode.EXT_ENABLE) - await prepare_exposures(self.driver, num or _MAX_NUM_IMAGE) + await prepare_exposures(self.driver, num) async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class PilatusDetector(AreaDetector[PilatusDriverIO]): @@ -156,7 +152,7 @@ class PilatusDetector(AreaDetector[PilatusDriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, readout_time: PilatusReadoutTime = PilatusReadoutTime.PILATUS3, edge_trigger_mode: PilatusTriggerMode = PilatusTriggerMode.EXT_TRIGGER, driver_suffix="cam1:", diff --git a/src/ophyd_async/epics/adsimdetector.py b/src/ophyd_async/epics/adsimdetector.py index ad412452e9..e1243e54c4 100644 --- a/src/ophyd_async/epics/adsimdetector.py +++ b/src/ophyd_async/epics/adsimdetector.py @@ -14,7 +14,7 @@ from .adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -44,9 +44,7 @@ async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class SimDetector(AreaDetector[ADBaseIO]): @@ -66,7 +64,7 @@ class SimDetector(AreaDetector[ADBaseIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix="cam1:", proc_suffix: str | None = None, plugins: dict[str, NDPluginBaseIO] | None = None, diff --git a/src/ophyd_async/epics/advimba.py b/src/ophyd_async/epics/advimba.py index 9567ee5c30..7cb053d3a0 100644 --- a/src/ophyd_async/epics/advimba.py +++ b/src/ophyd_async/epics/advimba.py @@ -22,7 +22,7 @@ from .adcore import ( ADAcquireLogic, ADBaseIO, - ADWriterFactory, + ADDataFactory, AreaDetector, NDPluginBaseIO, NDProcessIO, @@ -134,9 +134,7 @@ async def prepare_exposures_per_collection(self, exposures_per_collection: int): await prepare_filter(self.process_plugin, exposures_per_collection) async def default_trigger_info(self): - return await trigger_info_from_driver( - self.driver.num_images, self.process_plugin - ) + return await trigger_info_from_driver(self.driver, self.process_plugin) class VimbaDetector(AreaDetector[VimbaDriverIO]): @@ -159,7 +157,7 @@ class VimbaDetector(AreaDetector[VimbaDriverIO]): def __init__( self, prefix: str, - *writer_factories: ADWriterFactory, + *writer_factories: ADDataFactory, driver_suffix="cam1:", override_deadtime: float | None = None, proc_suffix: str | None = None, diff --git a/src/ophyd_async/fastcs/eiger/_acquire_logic.py b/src/ophyd_async/fastcs/eiger/_acquire_logic.py index b3f84af3c9..f5a4c6c66d 100644 --- a/src/ophyd_async/fastcs/eiger/_acquire_logic.py +++ b/src/ophyd_async/fastcs/eiger/_acquire_logic.py @@ -1,3 +1,5 @@ +from collections.abc import AsyncGenerator + from ophyd_async.core import ( DEFAULT_TIMEOUT, DetectorAcquireLogic, @@ -15,10 +17,9 @@ def __init__( self.detector = detector self.arm_when_ready = arm_when_ready - async def start_acquiring(self): + async def acquire(self) -> AsyncGenerator[None, None]: await self.arm_when_ready.trigger() - - async def wait_for_idle(self): + yield await wait_for_value(self.detector.state, "idle", timeout=DEFAULT_TIMEOUT) async def ensure_stopped(self): diff --git a/src/ophyd_async/fastcs/eiger/_detector.py b/src/ophyd_async/fastcs/eiger/_detector.py index f993114c40..f7718f2d78 100644 --- a/src/ophyd_async/fastcs/eiger/_detector.py +++ b/src/ophyd_async/fastcs/eiger/_detector.py @@ -1,4 +1,7 @@ +from functools import cached_property + from ophyd_async.core import ( + DetectorLogic, PathProvider, SignalR, StandardDetector, @@ -34,17 +37,22 @@ def __init__( # Need to do this first so the type hints are filled in connector = fastcs_connector(prefix, self) - self.add_detector_logics( - EigerTriggerLogic(self.detector), - EigerAcquireLogic(self.detector, self.arm_when_ready), - odin.OdinDataLogic( - path_provider=path_provider, - odin=self.od, - detector_bit_depth=self.detector.bit_depth_image, - pixel_mask=self.pixel_mask, - ), - ) - + self._path_provider = path_provider self.set_readable_format(self.pixel_mask, StandardReadableFormat.CONFIG_SIGNAL) super().__init__(name=name, connector=connector) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=EigerTriggerLogic(self.detector), + acquire=EigerAcquireLogic(self.detector, self.arm_when_ready), + data=[ + odin.OdinDataLogic( + path_provider=self._path_provider, + odin=self.od, + detector_bit_depth=self.detector.bit_depth_image, + pixel_mask=self.pixel_mask, + ) + ], + ) diff --git a/src/ophyd_async/fastcs/jungfrau/_acquire_logic.py b/src/ophyd_async/fastcs/jungfrau/_acquire_logic.py index 28ff8cd031..f34767fda4 100644 --- a/src/ophyd_async/fastcs/jungfrau/_acquire_logic.py +++ b/src/ophyd_async/fastcs/jungfrau/_acquire_logic.py @@ -1,3 +1,5 @@ +from collections.abc import AsyncGenerator + from ophyd_async.core import DEFAULT_TIMEOUT, DetectorAcquireLogic, wait_for_value from ._io import DetectorStatus, JungfrauDriverIO, PedestalMode @@ -7,10 +9,9 @@ class JungfrauAcquireLogic(DetectorAcquireLogic): def __init__(self, detector: JungfrauDriverIO) -> None: self.detector = detector - async def start_acquiring(self): + async def acquire(self) -> AsyncGenerator[None, None]: await self.detector.acquisition_start.trigger() - - async def wait_for_idle(self): + yield await wait_for_value( self.detector.detector_status, DetectorStatus.IDLE, timeout=DEFAULT_TIMEOUT ) diff --git a/src/ophyd_async/fastcs/jungfrau/_detector.py b/src/ophyd_async/fastcs/jungfrau/_detector.py index 0adb10db3c..d05fbbd58e 100644 --- a/src/ophyd_async/fastcs/jungfrau/_detector.py +++ b/src/ophyd_async/fastcs/jungfrau/_detector.py @@ -1,4 +1,11 @@ -from ophyd_async.core import PathProvider, StandardDetector, soft_signal_rw +from functools import cached_property + +from ophyd_async.core import ( + DetectorLogic, + PathProvider, + StandardDetector, + soft_signal_rw, +) from ophyd_async.fastcs import odin from ophyd_async.fastcs.core import fastcs_connector @@ -29,13 +36,19 @@ def __init__( self.acquisition_type = soft_signal_rw( AcquisitionType, AcquisitionType.STANDARD ) - self.add_detector_logics( - JungfrauTriggerLogic(self.detector, self.acquisition_type), - JungfrauAcquireLogic(self.detector), - odin.OdinDataLogic( - path_provider=path_provider, - odin=self.odin, - detector_bit_depth=self.detector.bit_depth, - ), - ) + self._path_provider = path_provider super().__init__(name=name) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=JungfrauTriggerLogic(self.detector, self.acquisition_type), + acquire=JungfrauAcquireLogic(self.detector), + data=[ + odin.OdinDataLogic( + path_provider=self._path_provider, + odin=self.odin, + detector_bit_depth=self.detector.bit_depth, + ) + ], + ) diff --git a/src/ophyd_async/fastcs/odin/_data_logic.py b/src/ophyd_async/fastcs/odin/_data_logic.py index 282e707f6f..172379e56e 100644 --- a/src/ophyd_async/fastcs/odin/_data_logic.py +++ b/src/ophyd_async/fastcs/odin/_data_logic.py @@ -5,20 +5,19 @@ from ophyd_async.core import ( DEFAULT_TIMEOUT, - DetectorDataLogic, PathProvider, SignalR, SignalRW, - StreamableDataProvider, StreamResourceDataProvider, StreamResourceInfo, + UnboundedDataLogic, wait_for_value, ) from ._io import OdinIO -class OdinDataLogic(DetectorDataLogic): +class OdinDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): def __init__( self, path_provider: PathProvider, @@ -31,9 +30,17 @@ def __init__( self.detector_bit_depth = detector_bit_depth self.pixel_mask = pixel_mask - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider: + # Odin sizes its own chunks and writes for as long as it is told to, so + # the periods are not used here yet. # Work out where to write - path_info = self.path_provider(datakey_name) + path_info = self.path_provider(name) # Get the current bit depth datatype = f"uint{await self.detector_bit_depth.get_value()}" # Setup the HDF writer @@ -62,12 +69,13 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: # Must ensure frames_written reset # See issue: https://github.com/DiamondLightSource/fastcs-odin/issues/107 await wait_for_value(self.odin.fp.frames_written, 0, timeout=DEFAULT_TIMEOUT) - # Return a provider that reflects what we have made + # The frame shape is only readable from the file processor once it is + # writing data_shape = await asyncio.gather( self.odin.fp.data_dims_0.get_value(), self.odin.fp.data_dims_1.get_value() ) resource = StreamResourceInfo( - data_key=datakey_name, + data_key=name, shape=data_shape, chunk_shape=(1, *data_shape), dtype_numpy=np.dtype(datatype).str, @@ -78,15 +86,16 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: uri=f"{path_info.directory_uri}{filename}_000001.h5", resources=[resource], mimetype="application/x-hdf5", - collections_written_signal=self.odin.fp.frames_written, + collections_written=self.odin.fp.frames_written, + collections_per_event=collections_per_event, ) - async def stop(self) -> None: + async def ensure_stopped(self) -> None: await asyncio.gather( self.odin.fp.stop_writing.trigger(), self.odin.mw.stop_.trigger(), ) - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: + def get_hinted_fields(self, name: str) -> Sequence[str]: # The main dataset is always hinted - return [datakey_name] + return [name] diff --git a/src/ophyd_async/fastcs/panda/_acquire_logic.py b/src/ophyd_async/fastcs/panda/_acquire_logic.py index b594b2c1a7..726888c56c 100644 --- a/src/ophyd_async/fastcs/panda/_acquire_logic.py +++ b/src/ophyd_async/fastcs/panda/_acquire_logic.py @@ -1,3 +1,5 @@ +from collections.abc import AsyncGenerator + from ophyd_async.core import ( DetectorAcquireLogic, set_and_wait_for_other_value, @@ -12,12 +14,10 @@ class PandaAcquireLogic(DetectorAcquireLogic): def __init__(self, pcap: PcapBlock) -> None: self.pcap = pcap - async def start_acquiring(self): + async def acquire(self) -> AsyncGenerator[None, None]: await set_and_wait_for_other_value(self.pcap.arm, True, self.pcap.active, True) - - async def wait_for_idle(self): - # TODO: https://github.com/PandABlocks/PandABlocks-FPGA/issues/262 - pass + # TODO: wait for idle, https://github.com/PandABlocks/PandABlocks-FPGA/issues/262 + yield async def ensure_stopped(self): await set_and_wait_for_other_value( diff --git a/src/ophyd_async/fastcs/panda/_data_logic.py b/src/ophyd_async/fastcs/panda/_data_logic.py index ef8ec675e2..e4e51d820f 100644 --- a/src/ophyd_async/fastcs/panda/_data_logic.py +++ b/src/ophyd_async/fastcs/panda/_data_logic.py @@ -4,11 +4,12 @@ import numpy as np from ophyd_async.core import ( - DetectorDataLogic, + DEFAULT_TIMEOUT, PathProvider, - StreamableDataProvider, StreamResourceDataProvider, StreamResourceInfo, + UnboundedDataLogic, + wait_for_value, ) from ._block import DataBlock, PandaCaptureMode @@ -16,7 +17,7 @@ logger = logging.getLogger("ophyd_async") -class PandaHDFDataLogic(DetectorDataLogic): +class PandaHDFDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): def __init__( self, path_provider: PathProvider, @@ -25,15 +26,24 @@ def __init__( self.path_provider = path_provider self.data_block = data_block - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: - # Work out where to write - path_info = self.path_provider(datakey_name) + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider: + # TODO: size the chunk from `collection_period` once the IOC exposes the + # chunk-size signal (see the chunk_shape TODO below). + path_info = self.path_provider(name) # Set create dir depth first to guarantee that callback when setting # directory path has correct value await self.data_block.create_directory.set(path_info.create_dir_depth) - # Setup the HDF writer + # Setup the HDF writer. The PandA flushes on a timer rather than by + # chunk size, so it takes the flush period directly; 0 means it flushes + # only when its buffer fills. await asyncio.gather( - self.data_block.flush_period.set(0), + self.data_block.flush_period.set(flush_period), self.data_block.hdf_directory.set(str(path_info.directory_path)), self.data_block.hdf_file_name.set( f"{path_info.filename}.h5", @@ -48,12 +58,14 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: ) # Start capturing await self.data_block.capture.set(True) - # Load data from the datasets PV on the panda, update internal - # representation of datasets that the panda will write. + # Capturing resets NumCaptured, which the detector reads as its baseline + await wait_for_value(self.data_block.num_captured, 0, timeout=DEFAULT_TIMEOUT) + # The datasets the PandA writes: PandABlocks-ioc only fills the table + # once capturing has started capture_table = await self.data_block.datasets.get_value() if len(capture_table) == 0: logger.warning( - f"PandA {datakey_name} DATASETS table is empty! " + f"PandA {name} DATASETS table is empty! " "No stream resource docs will be generated. " "Make sure captured positions have their corresponding " "*:DATASET PV set to a scientifically relevant name." @@ -74,8 +86,9 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: uri=f"{path_info.directory_uri}{path_info.filename}.h5", resources=resources, mimetype="application/x-hdf5", - collections_written_signal=self.data_block.num_captured, + collections_written=self.data_block.num_captured, + collections_per_event=collections_per_event, ) - async def stop(self) -> None: + async def ensure_stopped(self) -> None: await self.data_block.capture.set(False) diff --git a/src/ophyd_async/fastcs/panda/_detector.py b/src/ophyd_async/fastcs/panda/_detector.py index c365d56fa8..f9e90207ea 100644 --- a/src/ophyd_async/fastcs/panda/_detector.py +++ b/src/ophyd_async/fastcs/panda/_detector.py @@ -1,6 +1,8 @@ from __future__ import annotations -from ophyd_async.core import PathProvider, StandardDetector +from functools import cached_property + +from ophyd_async.core import DetectorLogic, PathProvider, StandardDetector from ophyd_async.fastcs.core import fastcs_connector from ._acquire_logic import PandaAcquireLogic @@ -23,9 +25,13 @@ def __init__( error_hint = f"Is PandABlocks-ioc at least version {MINIMUM_PANDA_IOC}?" # This has to be first so we make self.pcap connector = fastcs_connector(prefix, self, error_hint) - self.add_detector_logics( - PandaTriggerLogic(self.pcap), - PandaAcquireLogic(self.pcap), - PandaHDFDataLogic(path_provider, self.data), - ) + self._path_provider = path_provider super().__init__(name=name, connector=connector) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=PandaTriggerLogic(self.pcap), + acquire=PandaAcquireLogic(self.pcap), + data=[PandaHDFDataLogic(self._path_provider, self.data)], + ) diff --git a/src/ophyd_async/sim/_blob_acquire_logic.py b/src/ophyd_async/sim/_blob_acquire_logic.py index 5a2e5c0c39..02a84ad432 100644 --- a/src/ophyd_async/sim/_blob_acquire_logic.py +++ b/src/ophyd_async/sim/_blob_acquire_logic.py @@ -1,6 +1,6 @@ -import asyncio +from collections.abc import AsyncGenerator -from ophyd_async.core import DetectorAcquireLogic, TriggerInfo, cancel_and_wait +from ophyd_async.core import AsyncStatus, DetectorAcquireLogic, TriggerInfo from ._pattern_generator import PatternGenerator @@ -9,19 +9,13 @@ class BlobAcquireLogic(DetectorAcquireLogic): def __init__(self, pattern_generator: PatternGenerator): self.pattern_generator = pattern_generator self.trigger_info: TriggerInfo | None = None - self.task: asyncio.Task | None = None - async def start_acquiring(self): - # Start a background process off writing the images to file - self.task = asyncio.create_task(self.pattern_generator.write_images_to_file()) - - async def wait_for_idle(self): - # Wait for the background task to complete - if self.task: - await self.task + async def acquire(self) -> AsyncGenerator[None, None]: + # Write the images to file in the background, until done or closed + async with AsyncStatus(self.pattern_generator.write_images_to_file()) as status: + yield + await status async def ensure_stopped(self): - # Stop the background task and wait for it to finish - if self.task: - task, self.task = self.task, None - await cancel_and_wait(task) + # acquire() owns the writing task, so there is nothing else to stop + pass diff --git a/src/ophyd_async/sim/_blob_data_logic.py b/src/ophyd_async/sim/_blob_data_logic.py index 01073f0f62..a773335aa4 100644 --- a/src/ophyd_async/sim/_blob_data_logic.py +++ b/src/ophyd_async/sim/_blob_data_logic.py @@ -3,11 +3,10 @@ import numpy as np from ophyd_async.core import ( - DetectorDataLogic, PathProvider, - StreamableDataProvider, StreamResourceDataProvider, StreamResourceInfo, + UnboundedDataLogic, ) from ._pattern_generator import DATA_PATH, SUM_PATH, PatternGenerator @@ -16,7 +15,7 @@ HEIGHT = 240 -class BlobDataLogic(DetectorDataLogic): +class BlobDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): def __init__( self, path_provider: PathProvider, @@ -25,15 +24,21 @@ def __init__( self.path_provider = path_provider self.pattern_generator = pattern_generator - async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: - # Work out where to write - path_info = self.path_provider(datakey_name) - # Open the file - write_path = path_info.directory_path / f"{path_info.filename}.h5" - self.pattern_generator.open_file(write_path, WIDTH, HEIGHT) - # Return a provider that reflects what we have made + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider: + # The sim blob writer uses a fixed chunk shape and writes for as long as + # it is told to, so neither period is needed + path_info = self.path_provider(name) + self.pattern_generator.open_file( + path_info.directory_path / f"{path_info.filename}.h5", WIDTH, HEIGHT + ) data_resource = StreamResourceInfo( - data_key=datakey_name, + data_key=name, shape=(HEIGHT, WIDTH), # NDAttributes appear to always be configured with # this chunk size @@ -42,7 +47,7 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: parameters={"dataset": DATA_PATH}, ) sum_resource = StreamResourceInfo( - data_key=f"{datakey_name}-sum", + data_key=f"{name}-sum", shape=(), # NDAttributes appear to always be configured with # this chunk size @@ -54,12 +59,13 @@ async def prepare_unbounded(self, datakey_name: str) -> StreamableDataProvider: uri=f"{path_info.directory_uri}{path_info.filename}.h5", resources=[data_resource, sum_resource], mimetype="application/x-hdf5", - collections_written_signal=self.pattern_generator.images_written, + collections_written=self.pattern_generator.images_written, + collections_per_event=collections_per_event, ) - async def stop(self) -> None: + async def ensure_stopped(self) -> None: self.pattern_generator.close_file() - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: + def get_hinted_fields(self, name: str) -> Sequence[str]: # The main dataset is always hinted - return [datakey_name] + return [name] diff --git a/src/ophyd_async/sim/_blob_detector.py b/src/ophyd_async/sim/_blob_detector.py index 6760e088a1..932b6ba5e0 100644 --- a/src/ophyd_async/sim/_blob_detector.py +++ b/src/ophyd_async/sim/_blob_detector.py @@ -1,6 +1,12 @@ from collections.abc import Sequence +from functools import cached_property -from ophyd_async.core import PathProvider, SignalR, StandardDetector +from ophyd_async.core import ( + DetectorLogic, + PathProvider, + SignalR, + StandardDetector, +) from ophyd_async.core import StandardReadableFormat as Format from ._blob_acquire_logic import BlobAcquireLogic @@ -20,13 +26,20 @@ def __init__( name: str = "", ) -> None: self.pattern_generator = pattern_generator or PatternGenerator() - self.add_detector_logics( - BlobTriggerLogic(pattern_generator=self.pattern_generator), - BlobAcquireLogic(pattern_generator=self.pattern_generator), - BlobDataLogic( - path_provider=path_provider, pattern_generator=self.pattern_generator - ), - ) for signal in config_sigs: self.set_readable_format(signal, Format.CONFIG_SIGNAL) + self._path_provider = path_provider super().__init__(name=name) + + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + trigger=BlobTriggerLogic(pattern_generator=self.pattern_generator), + acquire=BlobAcquireLogic(pattern_generator=self.pattern_generator), + data=[ + BlobDataLogic( + path_provider=self._path_provider, + pattern_generator=self.pattern_generator, + ) + ], + ) diff --git a/tests/container_tests/test_adsim_system.py b/tests/container_tests/test_adsim_system.py index b9a3c3f426..0a2048890a 100644 --- a/tests/container_tests/test_adsim_system.py +++ b/tests/container_tests/test_adsim_system.py @@ -34,6 +34,7 @@ import bluesky.plan_stubs as bps import bluesky.plans as bp +import bluesky.preprocessors as bpp import pytest from aioca import purge_channel_caches from bluesky.run_engine import RunEngine @@ -105,7 +106,9 @@ def adsim(RE: RunEngine, shared_tmp_path: Path) -> AreaDetector: process_plugin = NDProcessIO(f"{prefix}-DI-CAM-01:PROC:") adsim = SimDetector( f"{prefix}-DI-CAM-01:", - adcore.ADWriterFactory.hdf(provider, writer_suffix="HDF5:"), + adcore.ADDataFactory.hdf( + provider, writer_suffix="HDF5:", always_enable=True + ), driver_suffix="DET:", plugins={"proc1": process_plugin}, ) @@ -145,15 +148,18 @@ def apply_baseline_settings(adsim: SimDetector) -> MsgGenerator[None]: def test_prepare_is_idempotent_and_sets_exposure_time( RE: RunEngine, adsim: SimDetector, bl01t_di_cam_01: None ) -> None: - def prepare_then_count() -> MsgGenerator[None]: + @bpp.stage_decorator([adsim]) + @bpp.run_decorator() + def prepare_then_trigger() -> MsgGenerator[None]: + # prepare() must come after stage() yield from bps.prepare( adsim, TriggerInfo(livetime=0.2), wait=True, ) - yield from bp.count([adsim]) + yield from bps.trigger_and_read([adsim]) - RE(prepare_then_count()) + RE(prepare_then_trigger()) actual_exposure_time: float = RE(bps.rd(adsim.driver.acquire_time)).plan_result assert actual_exposure_time == 0.2 diff --git a/tests/unit_tests/core/test_cancel_swallow.py b/tests/unit_tests/core/test_cancel_swallow.py index 866617e4da..9d08fb227b 100644 --- a/tests/unit_tests/core/test_cancel_swallow.py +++ b/tests/unit_tests/core/test_cancel_swallow.py @@ -1,13 +1,22 @@ import asyncio from collections.abc import Callable, Coroutine +from contextlib import suppress from typing import Any from unittest.mock import Mock +import numpy as np import pytest -from ophyd_async.core import AsyncStatus, soft_signal_rw +from ophyd_async.core import ( + Array1D, + AsyncStatus, + EventPageDataProvider, + soft_signal_r_and_setter, + soft_signal_rw, +) # The waits these cleanups cancel are internal, with no public way to slow them down +from ophyd_async.core import _data_providers as data_providers_module # noqa: PLC2701 from ophyd_async.core import _signal as signal_module # noqa: PLC2701 from ophyd_async.sim._blob_acquire_logic import BlobAcquireLogic # noqa: PLC2701 @@ -31,13 +40,13 @@ async def _status_aexit(cancelling: asyncio.Event, _: pytest.MonkeyPatch): await asyncio.sleep(0) # let the status task start -async def _blob_ensure_stopped(cancelling: asyncio.Event, _: pytest.MonkeyPatch): +async def _blob_acquire_close(cancelling: asyncio.Event, _: pytest.MonkeyPatch): pattern_generator = Mock() pattern_generator.write_images_to_file = lambda: _slow_to_cancel(cancelling) - logic = BlobAcquireLogic(pattern_generator) - await logic.start_acquiring() + acquisition = BlobAcquireLogic(pattern_generator).acquire() + await anext(acquisition) await asyncio.sleep(0) # let the task start - await logic.ensure_stopped() + await acquisition.aclose() async def _set_and_wait(cancelling: asyncio.Event, monkeypatch: pytest.MonkeyPatch): @@ -53,8 +62,42 @@ async def wait_for_value(self, signal, timeout): await signal_module.set_and_wait_for_value(sig, 1, timeout=0.1) +async def _make_page_cancelled( + cancelling: asyncio.Event, monkeypatch: pytest.MonkeyPatch +): + started = asyncio.Event() + data, _ = soft_signal_r_and_setter(Array1D[np.float64]) + timestamps, _ = soft_signal_r_and_setter(Array1D[np.float64]) + + # The data wait finishes cleaning up however often it is cancelled, the + # timestamps wait ends at once, so the page is left waiting for the data one + async def wait_for_length(signal, needed): + started.set() + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + if signal is data: + cancelling.set() + cleanup = asyncio.ensure_future(asyncio.sleep(0.2)) + while not cleanup.done(): + with suppress(asyncio.CancelledError): + await asyncio.shield(cleanup) + raise + + monkeypatch.setattr(data_providers_module, "_wait_for_length", wait_for_length) + provider = EventPageDataProvider( + {"det-stats": data}, timestamps, soft_signal_rw(int), 1 + ) + page = asyncio.create_task(provider.make_page(1)) + await started.wait() + # The first cancel starts the cleanup; cancelling the caller cancels it again + page.cancel() + await page + + @pytest.mark.parametrize( - "cleanup", [_status_aexit, _blob_ensure_stopped, _set_and_wait] + "cleanup", + [_status_aexit, _blob_acquire_close, _set_and_wait, _make_page_cancelled], ) async def test_cancelling_caller_during_cleanup_waits_for_it_then_raises( cleanup: Callable[[asyncio.Event, pytest.MonkeyPatch], Coroutine[Any, Any, None]], diff --git a/tests/unit_tests/core/test_data_providers.py b/tests/unit_tests/core/test_data_providers.py new file mode 100644 index 0000000000..401103134c --- /dev/null +++ b/tests/unit_tests/core/test_data_providers.py @@ -0,0 +1,113 @@ +import asyncio +from collections.abc import Callable + +import numpy as np +import pytest + +from ophyd_async.core import ( + Array1D, + EventPageDataProvider, + soft_signal_r_and_setter, + soft_signal_rw, +) + + +def make_provider(collections_per_event: int) -> EventPageDataProvider: + """A 5-collection buffer, filled, with one timestamp per collection.""" + data, _ = soft_signal_r_and_setter( + Array1D[np.float64], np.array([10.0, 11.0, 12.0, 13.0, 14.0]) + ) + timestamps, _ = soft_signal_r_and_setter( + Array1D[np.float64], np.array([100.0, 101.0, 102.0, 103.0, 104.0]) + ) + return EventPageDataProvider( + {"det-stats": data}, timestamps, soft_signal_rw(int), collections_per_event + ) + + +@pytest.mark.parametrize( + "collections_per_event,expected_data,expected_times", + [ + # step scan: one event holds the whole 5-collection buffer, timed by its + # last collection + (5, [[10.0, 11.0, 12.0, 13.0, 14.0]], [104.0]), + # fly scan: five events of one collection each, timed one by one + ( + 1, + [[10.0], [11.0], [12.0], [13.0], [14.0]], + [100.0, 101.0, 102.0, 103.0, 104.0], + ), + ], +) +async def test_slices_the_arrays_into_events( + collections_per_event: int, + expected_data: list[list[float]], + expected_times: list[float], +): + provider = make_provider(collections_per_event) + page = await provider.make_page(5 // collections_per_event) + assert page is not None + np.testing.assert_array_equal(page["data"]["det-stats"], expected_data) + assert page["time"] == expected_times + + +async def test_datakeys_take_dtype_from_the_signal(): + datakeys = await make_provider(5).make_datakeys() + assert list(datakeys) == ["det-stats"] + assert datakeys["det-stats"]["shape"] == [5] + assert datakeys["det-stats"]["dtype"] == "array" + assert datakeys["det-stats"]["dtype_numpy"] == " tuple[EventPageDataProvider, Callable[[int], None]]: + """Arrays holding the first `held` of 5 collections, and a way to post the rest.""" + values = np.array([10.0, 11.0, 12.0, 13.0, 14.0]) + times = np.array([100.0, 101.0, 102.0, 103.0, 104.0]) + data, set_data = soft_signal_r_and_setter(Array1D[np.float64], values[:held]) + timestamps, set_times = soft_signal_r_and_setter(Array1D[np.float64], times[:held]) + + def post(count: int) -> None: + set_data(values[:count]) + set_times(times[:count]) + + return EventPageDataProvider( + {"det-stats": data}, timestamps, soft_signal_rw(int), 1 + ), post + + +async def test_page_waits_for_arrays_shorter_than_the_count(): + provider, post = make_lagging_provider(held=2) + + async def post_later(): + await asyncio.sleep(0.1) + post(5) + + posting = asyncio.create_task(post_later()) + page = await provider.make_page(5) + await posting + + assert page is not None + np.testing.assert_array_equal( + page["data"]["det-stats"], [[10.0], [11.0], [12.0], [13.0], [14.0]] + ) + assert page["time"] == [100.0, 101.0, 102.0, 103.0, 104.0] + + +async def test_page_raises_if_the_arrays_never_catch_up(monkeypatch): + monkeypatch.setattr("ophyd_async.core._data_providers.ARRAY_WAIT_TIMEOUT", 0.1) + provider, _ = make_lagging_provider(held=2) + + with pytest.raises(TimeoutError, match="held 2 collections .* 5 were needed"): + await provider.make_page(5) diff --git a/tests/unit_tests/core/test_detector_use_cases.py b/tests/unit_tests/core/test_detector_use_cases.py new file mode 100644 index 0000000000..f0e57368e6 --- /dev/null +++ b/tests/unit_tests/core/test_detector_use_cases.py @@ -0,0 +1,1811 @@ +"""Use-case tests for the detector core (spec: detector-logic-spec.md). + +Written from the spec for the detector-core rewrite. The strict xfails are +behaviour the spec wants that something outside ophyd-async still prevents. + +The fakes in this file stand in for the three kinds of device a detector drives: +a device-side `Source` (counter plus data), an acquire logic that fills it, and +data logics that publish it as readables, event pages or stream assets. +""" + +import asyncio +import re +from collections import defaultdict +from collections.abc import AsyncIterator, Callable, Sequence +from functools import cached_property +from typing import Any, cast + +import bluesky.plan_stubs as bps +import bluesky.preprocessors as bpp +import numpy as np +import pytest +from bluesky import RunEngine +from event_model import DataKey +from event_model.documents import PartialEventPage + +from ophyd_async.core import ( + KICKED_OFF, + Array1D, + AsyncStatus, + BoundedDataLogic, + DetectorAcquireLogic, + DetectorDataLogic, + DetectorLogic, + DetectorTrigger, + DetectorTriggerLogic, + FlyableLogic, + PageableDataProvider, + SignalR, + StandardDetector, + StandardFlyable, + StreamResourceDataProvider, + StreamResourceInfo, + TransitionError, + TriggerInfo, + UnboundedDataLogic, + soft_signal_rw, +) +from ophyd_async.core import StandardReadableFormat as Format + +# --------------------------------------------------------------------------- +# Fakes +# --------------------------------------------------------------------------- + + +class Source: + """The device side: a counter of collections written, and their data. + + Each collection written holds the next number from a counter that never + resets, so every point of a scan has distinguishable data. + """ + + def __init__(self) -> None: + self.written = soft_signal_rw(int) + self.data: list[int] = [] + self._next = 0 + + async def produce(self, collections: int) -> None: + for _ in range(collections): + self.data.append(self._next) + self._next += 1 + await self.written.set(len(self.data)) + + async def rearm(self) -> None: + self.data.clear() + await self.written.set(0) + + +class InternalTrigger(DetectorTriggerLogic): + async def prepare_internal(self, num: int, livetime: float, deadtime: float): ... + + async def default_trigger_info(self) -> TriggerInfo: + return TriggerInfo() + + +class ExternalTrigger(InternalTrigger): + async def prepare_edge(self, num: int, livetime: float): ... + + +class FillingAcquire(DetectorAcquireLogic): + """Every acquire() produces `per_start` collections in each source.""" + + def __init__(self, *sources: Source, per_start: int = 1) -> None: + self.sources = sources + self.per_start = per_start + self.idle_waits = 0 + + async def acquire(self): + for source in self.sources: + await source.produce(self.per_start) + yield + self.idle_waits += 1 + + async def ensure_stopped(self): ... + + +class BoomError(Exception): + """What a fake raises when a test tells it to.""" + + +class FakeStream(StreamResourceDataProvider): + def __init__(self, *args: Any, owner: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self.owner = owner + + async def make_stream_docs(self, index): + if self.owner.fail: + raise BoomError("make_stream_docs") + async for doc in super().make_stream_docs(index): + yield doc + + +class FakePageable(PageableDataProvider): + """Event pages straight from a source, one value per collection. + + Every datakey holds the same values. `make_page` carries on from the last + page. + """ + + def __init__( + self, + datakey_names: Sequence[str], + source: Source, + collections_per_event: int, + shape_tail: tuple[int, ...] = (), + dtype_numpy: str = " None: + super().__init__(source.written, collections_per_event) + self.datakey_names = datakey_names + self.source = source + self.owner = owner + self.shape_tail = shape_tail + self.dtype_numpy = dtype_numpy + self.start_new_resources(0) + + def start_new_resources(self, origin: int) -> None: + self.origin = origin + self.published = 0 + + async def make_datakeys(self) -> dict[str, DataKey]: + return { + datakey_name: DataKey( + source="fake", + shape=[self.collections_per_event, *self.shape_tail], + dtype="array", + dtype_numpy=self.dtype_numpy, + ) + for datakey_name in self.datakey_names + } + + def _page(self, first: int, last: int) -> PartialEventPage: + cpe = self.collections_per_event + events = range(first, last) + starts = [self.origin + e * cpe for e in events] + values = [self.source.data[start : start + cpe] for start in starts] + times = [float(e) for e in events] + return { + "data": dict.fromkeys(self.datakey_names, values), + "time": times, + "timestamps": dict.fromkeys(self.datakey_names, times), + } + + async def make_page(self, index: int) -> PartialEventPage | None: + if self.owner and self.owner.fail: + raise BoomError("make_page") + if index <= self.published: + return None + page = self._page(self.published, index) + self.published = index + return page + + +class _FakeData: + """Publishes a `Source` as event pages or stream assets. + + :param bounded: a finite buffer: it needs a finite N (raises for N=0 if it + has datakeys left to make) and is cleared at each start + :param suffix: the suffix of the hinted datakey + :param also: the suffixes of further datakeys holding the same data + """ + + bounded: bool + + def __init__( + self, + source: Source, + *, + publication: str, + suffix: str = "", + also: Sequence[str] = (), + shape_tail: tuple[int, ...] = (), + dtype_numpy: str = " None: + self.source = source + self.publication = publication + self.datakey_suffix = suffix + self.also = also + self.shape_tail = shape_tail + self.dtype_numpy = dtype_numpy + self.tmp_path = tmp_path + self.hints = hints + self.periods: list[float] = [] + self.starts = 0 + self.stops = 0 + self.fail = False + self.fail_start = False + self.sit_out = False + #: What was done to this logic, in order + self.log: list[str] = [] + + @property + def collections_written(self) -> SignalR[int]: + return self.source.written + + async def _start( + self, + name: str, + existing_datakeys: Sequence[str], + num_collections: int, + collections_per_event: int, + collection_period: float, + ) -> Any: + self.periods.append(collection_period) + datakey_names = [ + datakey_name + for datakey_name in ( + name + suffix for suffix in (self.datakey_suffix, *self.also) + ) + if datakey_name not in existing_datakeys + ] + if self.sit_out or not datakey_names: + return None + self.starts += 1 + self.log.append("start") + if self.fail_start: + raise BoomError("start") + if self.bounded: + # A finite buffer is armed empty + await self.source.rearm() + if self.publication == "stream": + return FakeStream( + uri=f"file://localhost/{self.tmp_path.as_posix().lstrip('/')}/f.h5", + resources=[ + StreamResourceInfo( + data_key=datakey_name, + shape=self.shape_tail, + chunk_shape=(1, *self.shape_tail), + dtype_numpy=self.dtype_numpy, + parameters={"dataset": "/data"}, + ) + for datakey_name in datakey_names + ], + mimetype="application/x-hdf5", + collections_written=self.source.written, + collections_per_event=collections_per_event, + owner=self, + ) + return FakePageable( + datakey_names, + self.source, + collections_per_event, + shape_tail=self.shape_tail, + dtype_numpy=self.dtype_numpy, + owner=self, + ) + + async def ensure_stopped(self) -> None: + self.stops += 1 + self.log.append("stop") + + def get_hinted_fields(self, name: str) -> Sequence[str]: + datakey_name = name + self.datakey_suffix + return self.hints(datakey_name) if self.hints else [datakey_name] + + +class FakeBoundedData(_FakeData, BoundedDataLogic[Any]): + bounded = True + + async def start_provider( + self, + name: str, + existing_datakeys: Sequence[str], + num_collections: int, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> Any: + return await self._start( + name, + existing_datakeys, + num_collections, + collections_per_event, + collection_period, + ) + + +class FakeUnboundedData(_FakeData, UnboundedDataLogic[Any]): + bounded = False + + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> Any: + # An unbounded source takes data for as long as it is told to + return await self._start(name, (), 1, collections_per_event, collection_period) + + +def FakeData( # noqa: N802 + source: Source, *, bounded: bool, **kwargs: Any +) -> FakeBoundedData | FakeUnboundedData: + return (FakeBoundedData if bounded else FakeUnboundedData)(source, **kwargs) + + +def make_detector( + name: str = "det", + *data: _FakeData, + acquire: DetectorAcquireLogic | None = None, + trigger: DetectorTriggerLogic | None = None, + readable: SignalR | None = None, +) -> StandardDetector: + det = DetectorLogic( + trigger=trigger or InternalTrigger(), acquire=acquire, data=data + ).with_device(name) + if readable is not None: + det.counts = readable + det.set_readable_format(readable, Format.HINTED_SIGNAL) + return det + + +def run_plan(RE: RunEngine, plan) -> dict[str, list[dict]]: + docs: dict[str, list[dict]] = defaultdict(list) + RE(plan, lambda name, doc: docs[name].append(doc)) + return docs + + +def step_scan(det: StandardDetector, points: int, info: TriggerInfo | None = None): + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def plan(): + if info is not None: + # prepare() comes after stage(), which resets the detector + yield from bps.prepare(det, info, wait=True) + for _ in range(points): + yield from bps.trigger_and_read([det]) + + return plan() + + +def fly_scan(det: StandardDetector, info: TriggerInfo): + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def plan(): + yield from bps.prepare(det, info, wait=True) + yield from bps.declare_stream(det, name="primary", collect=True) + yield from bps.kickoff(det, wait=True) + yield from bps.collect_while_completing( + flyers=[det], dets=[det], flush_period=0.05 + ) + + return plan() + + +# --------------------------------------------------------------------------- +# Use cases 1, 2, 3, 5, 11 and the four quadrants of D2, through the RunEngine +# --------------------------------------------------------------------------- + + +def test_uc1_readables_only_step_scan_needs_no_prepare(RE: RunEngine): + """Use case 1: no data logic, trigger() prepares implicitly.""" + counts = soft_signal_rw(int, 42, name="det-counts") + acquire = FillingAcquire() + det = make_detector("det", acquire=acquire, readable=counts) + + docs = run_plan(RE, step_scan(det, points=2)) + + assert [e["data"] for e in docs["event"]] == [{"det-counts": 42}] * 2 + assert set(docs["descriptor"][0]["data_keys"]) == {"det-counts"} + assert docs["stream_resource"] == [] + assert acquire.idle_waits == 2 + + +def test_uc2_streamable_step_scan_with_readables(RE: RunEngine, tmp_path): + """Use case 2: one StreamResource, one StreamDatum per point, Events too.""" + source = Source() + counts = soft_signal_rw(int, 7, name="det-counts") + det = make_detector( + "det", + FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path), + acquire=FillingAcquire(source), + readable=counts, + ) + + docs = run_plan(RE, step_scan(det, points=3)) + + (resource,) = docs["stream_resource"] + assert resource["data_key"] == "det" + assert [ + (d["indices"]["start"], d["indices"]["stop"]) for d in docs["stream_datum"] + ] == [ + (0, 1), + (1, 2), + (2, 3), + ] + assert {d["stream_resource"] for d in docs["stream_datum"]} == {resource["uid"]} + assert [e["data"] for e in docs["event"]] == [{"det-counts": 7}] * 3 + keys = docs["descriptor"][0]["data_keys"] + assert keys["det"]["external"] == "STREAM:" + assert "external" not in keys["det-counts"] + + +def test_uc3_streamable_fly_scan_makes_assets_and_no_events(RE: RunEngine, tmp_path): + """Use case 3: stream assets only, covering all N events.""" + source = Source() + det = make_detector( + "det", + FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path), + acquire=FillingAcquire(source, per_start=6), + ) + + docs = run_plan( + RE, fly_scan(det, TriggerInfo(number_of_events=3, collections_per_event=2)) + ) + + assert docs["event"] == [] and docs["event_page"] == [] + (resource,) = docs["stream_resource"] + assert resource["data_key"] == "det" + stops = [d["indices"]["stop"] for d in docs["stream_datum"]] + assert stops[-1] == 3 + assert docs["stop"][0]["num_events"] == {"primary": 3} + + +def test_two_runs_in_one_stage_each_have_their_own_stream_resource( + RE: RunEngine, tmp_path +): + source = Source() + data = FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path) + det = make_detector("det", data, acquire=FillingAcquire(source)) + + @bpp.stage_decorator([det]) + def plan(): + for _ in range(2): + yield from bpp.run_wrapper( + bpp.pchain( + bps.prepare(det, TriggerInfo(), wait=True), + bps.trigger_and_read([det]), + ) + ) + + runs: list[dict[str, list[dict]]] = [] + + def collect(name: str, doc: dict) -> None: + if name == "start": + runs.append(defaultdict(list)) + runs[-1][name].append(doc) + + RE(plan(), collect) + + # The provider carries over, so the same file with the indices carrying on + assert data.log.count("start") == 1 + for run, index in zip(runs, (0, 1), strict=True): + (resource,) = run["stream_resource"] + (datum,) = run["stream_datum"] + assert datum["stream_resource"] == resource["uid"] + assert datum["indices"] == {"start": index, "stop": index + 1} + assert runs[0]["stream_resource"][0]["uid"] != runs[1]["stream_resource"][0]["uid"] + + +def test_uc5_pageable_step_scan_with_several_collections_per_event(RE: RunEngine): + """Use case 5 (and 4's buffer): one array reading of shape [N] per point. + + The finite buffer is cleared between points, so each point holds its own N. + """ + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=True), + acquire=FillingAcquire(source, per_start=3), + ) + + docs = run_plan(RE, step_scan(det, 2, TriggerInfo(collections_per_event=3))) + + assert docs["descriptor"][0]["data_keys"]["det"]["shape"] == [3] + assert [e["data"]["det"] for e in docs["event"]] == [[0, 1, 2], [3, 4, 5]] + + +def test_uc4_pageable_fly_scan_collects_event_pages(RE: RunEngine): + """Use case 4: N events come out of collect_pages as event pages.""" + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=True), + acquire=FillingAcquire(source, per_start=4), + ) + + docs = run_plan(RE, fly_scan(det, TriggerInfo(number_of_events=4))) + + (page,) = docs["event_page"] + assert page["data"]["det"] == [[0], [1], [2], [3]] + assert docs["stop"][0]["num_events"] == {"primary": 4} + + +@pytest.mark.parametrize( + "publication,bounded", + [ + ("stream", False), + ("stream", True), + ("page", True), + ("page", False), + ], +) +def test_d2_all_four_bound_by_publication_quadrants_step_scan( + RE: RunEngine, tmp_path, publication: str, bounded: bool +): + """Bound and publication are separate axes (D2): every quadrant step-scans. + + An unbounded pageable source (a monitor-fed accumulator) is never re-armed + and its count keeps rising, so the progress baseline of each point must be + read live; a baseline of 0 reads stale data. + """ + source = Source() + det = make_detector( + "det", + FakeData(source, publication=publication, bounded=bounded, tmp_path=tmp_path), + acquire=FillingAcquire(source, per_start=1), + ) + + docs = run_plan(RE, step_scan(det, 3)) + + if publication == "page": + assert [e["data"]["det"] for e in docs["event"]] == [[0], [1], [2]] + else: + # A bounded provider is re-made at each point, so it starts a new stream + # resource each time; an unbounded one carries on in the same one + stops = [1, 1, 1] if bounded else [1, 2, 3] + assert [d["indices"]["stop"] for d in docs["stream_datum"]] == stops + assert len(docs["stream_resource"]) == (3 if bounded else 1) + assert docs["event"][0]["data"] == {} + + +@pytest.mark.parametrize("publication", ["stream", "page"]) +async def test_d2_progress_baseline_is_read_live_for_unbounded_sources( + publication: str, tmp_path +): + """A source that already holds collections (e.g. left running) is not stale.""" + source = Source() + await source.produce(5) + det = make_detector( + "det", + FakeData(source, publication=publication, bounded=False, tmp_path=tmp_path), + acquire=FillingAcquire(source, per_start=1), + ) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + await det.trigger() + # Events since prepare(), from the 5 already there + assert await det.get_index() == 2 + + +# --------------------------------------------------------------------------- +# Bounded logics are re-armed at each step point after the first (D2, D3) +# --------------------------------------------------------------------------- + + +async def test_d3_bounded_logic_is_restarted_at_each_point_after_the_first(): + source = Source() + data = FakeData(source, publication="page", bounded=True) + det = make_detector("det", data, acquire=FillingAcquire(source)) + await det.stage() + await det.prepare(TriggerInfo()) + + await det.trigger() + assert data.starts == 1 + await det.trigger() + await det.trigger() + assert data.starts == 3 + + +async def test_d3_step_scan_index_counts_since_prepare_across_points(tmp_path): + """get_index counts every point; a bounded provider only holds its own.""" + bounded_source, open_source = Source(), Source() + det = make_detector( + "det", + FakeData( + bounded_source, + publication="stream", + bounded=True, + suffix="-b", + tmp_path=tmp_path, + ), + FakeData(open_source, publication="stream", bounded=False, tmp_path=tmp_path), + acquire=FillingAcquire(bounded_source, open_source), + ) + await det.stage() + await det.prepare(TriggerInfo()) + points = [] + for _ in range(3): + await det.trigger() + index = await det.get_index() + docs = [d async for d in det.collect_asset_docs(index)] # type: ignore[attr-defined] + points.append( + ( + index, + [d["data_key"] for name, d in docs if name == "stream_resource"], + [ + (d["indices"]["start"], d["indices"]["stop"]) + for name, d in docs + if name == "stream_datum" + ], + ) + ) + # The bounded provider is restarted at each point, so has a new resource + # holding one event, while the unbounded one carries on in its first + assert points == [ + (1, ["det-b", "det"], [(0, 1), (0, 1)]), + (2, ["det-b"], [(0, 1), (1, 2)]), + (3, ["det-b"], [(0, 1), (2, 3)]), + ] + + +async def test_d3_unbounded_logic_is_never_restarted_and_fly_never_rearms(): + source = Source() + bounded = FakeData(source, publication="page", bounded=True) + other = Source() + unbounded = FakeData(other, publication="page", bounded=False, suffix="-u") + det = make_detector( + "det", bounded, unbounded, acquire=FillingAcquire(source, other, per_start=2) + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=2)) + await det.kickoff() + await det.complete() + assert (bounded.starts, unbounded.starts) == (1, 1) + await det.prepare(TriggerInfo()) + await det.trigger() + await det.trigger() + # The unbounded provider carried over to the second prepare + assert (bounded.starts, unbounded.starts) == (3, 1) + + +async def test_d3_mixed_bounded_and_unbounded_pageable_steps_repeatedly(): + """One baseline per bound group: the bounded one resets, the other counts on.""" + bounded_source, open_source = Source(), Source() + await open_source.produce(3) + bounded = FakeData(bounded_source, publication="page", bounded=True, suffix="-b") + unbounded = FakeData(open_source, publication="page", bounded=False, suffix="-u") + det = make_detector( + "det", + bounded, + unbounded, + acquire=FillingAcquire(bounded_source, open_source), + ) + await det.stage() + await det.prepare(TriggerInfo()) + values = [] + for _ in range(3): + await det.trigger() + reading = await det.read() + values.append((reading["det-b"]["value"], reading["det-u"]["value"])) + assert values == [([0], [3]), ([1], [4]), ([2], [5])] + + +async def test_d3_fly_scan_with_an_unbounded_source_non_zero_at_kickoff(): + bounded_source, open_source = Source(), Source() + await open_source.produce(4) + det = make_detector( + "det", + FakeData(bounded_source, publication="page", bounded=True, suffix="-b"), + FakeData(open_source, publication="page", bounded=False, suffix="-u"), + acquire=FillingAcquire(bounded_source, open_source, per_start=3), + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=3)) + await det.kickoff() + await det.complete() + # Events since prepare() + assert await det.get_index() == 3 + (page,) = [p async for p in det.collect_pages()] # type: ignore[attr-defined] + assert page["data"]["det-b"] == [[0], [1], [2]] + assert page["data"]["det-u"] == [[4], [5], [6]] + + +class _BoundedAndUnbounded(BoundedDataLogic[Any], UnboundedDataLogic[Any]): ... + + +@pytest.mark.parametrize("dl", [DetectorDataLogic(), _BoundedAndUnbounded()]) +def test_d3_data_must_be_a_bounded_or_an_unbounded_logic(dl): + with pytest.raises(TypeError, match="must inherit from exactly one of"): + DetectorLogic(data=[dl]) + + +# --------------------------------------------------------------------------- +# Lifecycle state machine (spec §3), every verb from every state +# --------------------------------------------------------------------------- + +STATES = ["U", "S", "P", "T", "K", "C"] + + +class GatedAcquire(FillingAcquire): + """Runs acquire() to its end only once `done` is set, so a fly stays FLYING.""" + + def __init__(self, *sources: Source) -> None: + super().__init__(*sources) + self.done = asyncio.Event() + + async def acquire(self): + async for _ in super().acquire(): + yield + await self.done.wait() + + +async def reach(state: str, tmp_path) -> StandardDetector: + source = Source() + acquire = GatedAcquire(source) + det = make_detector( + "det", + FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path), + acquire=acquire, + ) + # "K" is kicked off with the fly still running + if state != "K": + acquire.done.set() + if state == "U": + return det + await det.stage() + if state == "S": + return det + await det.prepare(TriggerInfo()) + if state == "T": + await det.trigger() + elif state in "KC": + await det.kickoff() + if state == "C": + await det.complete() + return det + + +async def call(det: StandardDetector, verb: str) -> None: + if verb == "prepare": + await det.prepare(TriggerInfo()) + elif verb == "complete": + # Let a running fly finish + cast(GatedAcquire, det.logic.acquire).done.set() + await asyncio.wait_for(det.complete(), 2) + elif verb == "collect_asset_docs": + _ = [doc async for doc in det.collect_asset_docs()] # type: ignore[attr-defined] + else: + await asyncio.wait_for(getattr(det, verb)(), 2) + + +# (state, verb) -> does the verb succeed. From the §3 table. +TABLE: dict[tuple[str, str], bool] = {} +for _verb, _row in { + "prepare": "FTFTFT", + "trigger": "FTTTFF", + "kickoff": "FFTFFF", + "complete": "FFFFTT", + "read": "FFFTFF", + "describe": "FFTTTT", + "describe_collect": "FFTTTT", + "get_index": "FFTTTT", +}.items(): + for _state, _ok in zip(STATES, _row, strict=True): + TABLE[(_state, _verb)] = _ok == "T" +for _state in "PTKC": + # Allowed after a trigger() too: bluesky's read() collects stream assets + TABLE[(_state, "collect_asset_docs")] = _state in "TKC" + + +#: The states in which a verb may read the prepared context +_PREPARED = "FLOWN, FLYING, PREPARED, TRIGGERED, TRIGGERING" + +# What each verb that fails from a state says, as the start of the message +MESSAGES: dict[tuple[str, str], str] = { + ("U", "prepare"): "cannot go to PREPARING from UNSTAGED", + ("P", "prepare"): "cannot go to PREPARING from PREPARED", + ("K", "prepare"): "cannot go to PREPARING from FLYING", + ("U", "trigger"): "cannot go to TRIGGERING from UNSTAGED", + ("K", "trigger"): "cannot go to TRIGGERING from FLYING", + ("C", "trigger"): "cannot go to TRIGGERING from FLOWN", + ("U", "kickoff"): "cannot go to FLYING from UNSTAGED", + ("S", "kickoff"): "cannot go to FLYING from STAGED", + ("T", "kickoff"): "cannot go to FLYING from TRIGGERED", + ("K", "kickoff"): "cannot go to FLYING from FLYING", + ("C", "kickoff"): "cannot go to FLYING from FLOWN", + ("U", "complete"): "cannot complete from UNSTAGED, allowed only after kickoff()", + ("S", "complete"): "cannot complete from STAGED, allowed only after kickoff()", + ("P", "complete"): "cannot complete from PREPARED, allowed only after kickoff()", + ("T", "complete"): "cannot complete from TRIGGERED, allowed only after kickoff()", + ("U", "read"): "det: read not allowed from UNSTAGED, allowed from: TRIGGERED", + ("S", "read"): "det: read not allowed from STAGED, allowed from: TRIGGERED", + ("P", "read"): "det: read not allowed from PREPARED, allowed from: TRIGGERED", + ("K", "read"): "det: read not allowed from FLYING, allowed from: TRIGGERED", + ("C", "read"): "det: read not allowed from FLOWN, allowed from: TRIGGERED", + ( + "U", + "describe", + ): f"det: describe not allowed from UNSTAGED, allowed from: {_PREPARED}", + ( + "S", + "describe", + ): f"det: describe not allowed from STAGED, allowed from: {_PREPARED}", + ( + "U", + "describe_collect", + ): f"det: describe_collect not allowed from UNSTAGED, allowed from: {_PREPARED}", + ( + "S", + "describe_collect", + ): f"det: describe_collect not allowed from STAGED, allowed from: {_PREPARED}", + ( + "U", + "get_index", + ): f"det: get_index not allowed from UNSTAGED, allowed from: {_PREPARED}", + ( + "S", + "get_index", + ): f"det: get_index not allowed from STAGED, allowed from: {_PREPARED}", + ("P", "collect_asset_docs"): ( + "det: collect_asset_docs not allowed from PREPARED, allowed from: " + "FLOWN, FLYING, TRIGGERED" + ), +} + + +def _rows(): + for (state, verb), ok in TABLE.items(): + id = f"{state}-{verb}" + yield pytest.param(state, verb, ok, id=id) + + +@pytest.mark.parametrize("state,verb,ok", _rows()) +async def test_lifecycle_verb_from_state(state, verb, ok, tmp_path): + det = await reach(state, tmp_path) + if ok: + await call(det, verb) + else: + with pytest.raises(RuntimeError, match=re.escape(MESSAGES[(state, verb)])): + await call(det, verb) + # Ends a fly still running in "K" + await det.unstage() + + +async def test_a_prepare_refused_mid_fly_leaves_the_collect_verb_bound(tmp_path): + det = await reach("K", tmp_path) + with pytest.raises(RuntimeError, match="cannot go to PREPARING from FLYING"): + await det.prepare(TriggerInfo()) + cast(GatedAcquire, det.logic.acquire).done.set() + await det.complete() + assert [doc async for doc in det.collect_asset_docs()] # type: ignore[attr-defined] + await det.unstage() + + +@pytest.mark.parametrize("state", STATES) +async def test_restage_returns_to_staged_from_any_state(state, tmp_path): + det = await reach(state, tmp_path) + await det.unstage() + await det.stage() + with pytest.raises(TransitionError, match="describe not allowed from STAGED"): + await det.describe() + await det.prepare(TriggerInfo()) + await det.describe() + + +@pytest.mark.parametrize("state", STATES) +async def test_unstage_from_any_state_is_never_an_error(state, tmp_path): + det = await reach(state, tmp_path) + await det.unstage() + await det.unstage() + + +@pytest.mark.parametrize("state", ["S", "P", "T", "K", "C"]) +async def test_unstage_leads_to_unstaged(state, tmp_path): + det = await reach(state, tmp_path) + await det.unstage() + with pytest.raises(RuntimeError, match="from UNSTAGED"): + await det.prepare(TriggerInfo()) + + +class _Flyer(StandardFlyable[int, None]): + @cached_property + def logic(self) -> FlyableLogic[int, None]: + return _NoopFlyableLogic() + + +class _NoopFlyableLogic(FlyableLogic[int, None]): + async def prepare_to_fly( + self, name: str, value: int, previous_ctx: None + ) -> AsyncIterator[None]: + yield None + + async def fly(self, ctx: None) -> AsyncIterator[KICKED_OFF]: + yield KICKED_OFF + + async def ensure_stopped(self) -> None: ... + + +@pytest.mark.parametrize("verb", ["prepare", "kickoff", "complete"]) +async def test_r10_standard_flyable_verbs_raise_before_stage(verb: str): + flyer = _Flyer(name="flyer") + with pytest.raises(RuntimeError, match="from UNSTAGED"): + await (flyer.prepare(1) if verb == "prepare" else getattr(flyer, verb)()) + await flyer.stage() + await flyer.prepare(1) + await flyer.unstage() + await flyer.unstage() # a no-op + with pytest.raises(RuntimeError, match="from UNSTAGED"): + await flyer.prepare(1) + + +async def test_trigger_after_complete_says_to_prepare_again(tmp_path): + det = await reach("C", tmp_path) + with pytest.raises( + TransitionError, + match="cannot go to TRIGGERING from FLOWN, allowed: PREPARING", + ): + await det.trigger() + + +async def test_external_trigger_second_trigger_without_prepare_raises(): + source = Source() + det = make_detector( + "det", trigger=ExternalTrigger(), acquire=FillingAcquire(source) + ) + await det.stage() + await det.prepare( + TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE, exposure_timeout=0.1) + ) + await det.trigger() + with pytest.raises(RuntimeError, match="prepare"): + await det.trigger() + + +async def test_internal_trigger_repeats_on_one_prepare(): + acquire = FillingAcquire() + det = make_detector("det", acquire=acquire) + await det.stage() + await det.prepare(TriggerInfo()) + for _ in range(5): + await det.trigger() + assert acquire.idle_waits == 5 + + +class HeldAcquire(DetectorAcquireLogic): + """Arms with a status that only ensure_stopped() finishes, like a driver.""" + + def __init__(self) -> None: + self.statuses: list[AsyncStatus] = [] + self.closed = 0 + self.armed = asyncio.Event() + self._stopped = asyncio.Event() + + async def ensure_ready(self): + self._stopped.clear() + + async def acquire(self): + try: + async with AsyncStatus(self._stopped.wait()) as status: + self.statuses.append(status) + self.armed.set() + yield + await status + finally: + self.closed += 1 + + async def ensure_stopped(self): + self._stopped.set() + + +async def test_unstage_closes_the_acquisition_an_external_prepare_armed(): + acquire = HeldAcquire() + det = make_detector("det", trigger=ExternalTrigger(), acquire=acquire) + await det.stage() + await det.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE)) + assert acquire.armed.is_set() + assert acquire.closed == 0 + + await det.unstage() + assert acquire.closed == 1 + assert acquire.statuses[0].done + + +class ArmLoggingAcquire(FillingAcquire): + """Writes "arm" to `log` once armed.""" + + def __init__(self, log: list[str], *sources: Source) -> None: + super().__init__(*sources) + self.log = log + + async def acquire(self): + async for _ in super().acquire(): + self.log.append("arm") + yield + + +async def test_an_external_prepare_starts_the_data_before_arming(): + source = Source() + data = FakeData(source, bounded=False, publication="page") + det = make_detector( + "det", + data, + trigger=ExternalTrigger(), + acquire=ArmLoggingAcquire(data.log, source), + ) + await det.stage() + data.log.clear() + + await det.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE)) + assert data.log == ["start", "arm"] + await det.unstage() + + +async def test_the_last_prepare_is_forgotten_at_stage_not_unstage(): + source = Source() + serving = FakeData(source, bounded=False, publication="page") + sitting_out = FakeData(source, bounded=False, publication="page", suffix="-b") + sitting_out.sit_out = True + det = make_detector("det", serving, sitting_out) + # Before a prepare, every data logic's fields are hinted + await det.stage() + assert det.hints == {"fields": ["det", "det-b"]} + + await det.prepare(TriggerInfo()) + assert det.hints == {"fields": ["det"]} + await det.unstage() + assert det.hints == {"fields": ["det"]} + + await det.stage() + assert det.hints == {"fields": ["det", "det-b"]} + + +class StopData(FakeBoundedData): + """Its `ensure_stopped` raises if `stop_mode` is "fail", runs until + cancelled if "hang".""" + + stop_mode = "" + stopping = False + + async def ensure_stopped(self) -> None: + await super().ensure_stopped() + if self.stop_mode == "fail": + raise BoomError("stop") + self.stopping = self.stop_mode == "hang" + try: + if self.stopping: + await asyncio.Event().wait() + finally: + self.stopping = False + + +# trigger() restarts a bounded logic at each point after the first +@pytest.mark.parametrize("verb", ["stage", "trigger", "unstage"]) +async def test_a_failed_stop_leaves_no_other_stop_running(verb: str): + source, other_source = Source(), Source() + failing = StopData(source, publication="page") + hanging = StopData(other_source, publication="page", suffix="-h") + det = make_detector( + "det", failing, hanging, acquire=FillingAcquire(source, other_source) + ) + if verb != "stage": + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + failing.stop_mode, hanging.stop_mode = "fail", "hang" + + with pytest.raises(BoomError, match="stop"): + await getattr(det, verb)() + + assert not hanging.stopping + + +async def test_unstage_cancels_a_running_trigger_that_kickoff_cannot_overlap(): + acquire = HeldAcquire() + det = make_detector( + "det", FakeData(Source(), publication="page", bounded=False), acquire=acquire + ) + await det.stage() + # Prepares implicitly, arms, then waits for data that never comes + status = det.trigger() + await acquire.armed.wait() + with pytest.raises(RuntimeError, match="cannot go to FLYING from TRIGGERING"): + await det.kickoff() + await det.unstage() + with pytest.raises(asyncio.CancelledError): + await status + assert acquire.closed == 1 + assert acquire.statuses[0].done + + +# --------------------------------------------------------------------------- +# Use cases 10 and 11 +# --------------------------------------------------------------------------- + + +async def test_a_bounded_logic_publishes_only_the_keys_unbounded_do_not_make(): + unbounded_source, bounded_source = Source(), Source() + det = make_detector( + "det", + FakeData(unbounded_source, publication="page", bounded=False, suffix="-u"), + FakeData( + bounded_source, publication="page", bounded=True, suffix="-u", also=["-b"] + ), + acquire=FillingAcquire(unbounded_source, bounded_source, per_start=2), + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=2)) + await det.kickoff() + await det.complete() + + (page,) = [p async for p in det.collect_pages()] # type: ignore[attr-defined] + assert page["data"] == {"det-u": [[0], [1]], "det-b": [[0], [1]]} + # Hinted fields are those the providers make + assert det.hints == {"fields": ["det-u"]} + + +@pytest.mark.parametrize( + "bounded, again", + [ + # A bounded logic is restarted at the next step point + (True, lambda det: det.trigger()), + # An unbounded one at a prepare() for a different scan + (False, lambda det: det.prepare(TriggerInfo(collections_per_event=2))), + ], +) +async def test_a_data_logic_that_served_the_stage_must_serve_all_of_it( + bounded: bool, again +): + source = Source() + data = FakeData(source, publication="page", bounded=bounded) + det = make_detector("det", data, acquire=FillingAcquire(source)) + await det.stage() + await det.trigger() + + data.sit_out = True + with pytest.raises(RuntimeError, match=r"made \[\] at .*, but \['det'\] before it"): + await again(det) + + +async def test_uc11_no_data_no_readables_steps_and_flies(): + acquire = FillingAcquire() + det = make_detector("det", acquire=acquire) + await det.stage() + + await det.trigger() + await det.trigger() + assert acquire.idle_waits == 2 + assert await det.read() == {} + + # A fly has nothing to collect, so completes once acquisition has run + await det.prepare(TriggerInfo(number_of_events=3)) + await det.kickoff() + await det.complete() + assert acquire.idle_waits == 3 + assert not hasattr(det, "collect_asset_docs") + assert not hasattr(det, "collect_pages") + + +# --------------------------------------------------------------------------- +# Data-set rules (§5) +# --------------------------------------------------------------------------- + + +async def test_u5_read_is_idempotent_for_pageable_data(): + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=True), + acquire=FillingAcquire(source, per_start=2), + ) + await det.stage() + await det.prepare(TriggerInfo(collections_per_event=2)) + await det.trigger() + + first = await det.read() + second = await det.read() + + assert first["det"]["value"] == [0, 1] + assert second == first + + +async def test_u5_collect_pages_takes_an_index(): + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=True), + acquire=FillingAcquire(source, per_start=5), + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=5)) + await det.kickoff() + + first = [p async for p in det.collect_pages(index=2)] # type: ignore[attr-defined] + rest = [p async for p in det.collect_pages(index=5)] # type: ignore[attr-defined] + + assert [p["data"]["det"] for p in first] == [[[0], [1]]] + assert [p["data"]["det"] for p in rest] == [[[2], [3], [4]]] + + +async def test_uc12b_multiple_pageable_on_one_detector_merge_into_one_page(): + """Use case 12(b), written as if the bundler limits were fixed.""" + a, b = Source(), Source() + det = make_detector( + "det", + FakeData(a, publication="page", bounded=True, suffix="-a"), + FakeData(b, publication="page", bounded=True, suffix="-b"), + acquire=FillingAcquire(a, b, per_start=3), + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=3)) + await det.kickoff() + await det.complete() + + pages = [p async for p in det.collect_pages()] # type: ignore[attr-defined] + + (page,) = pages + assert page["data"] == {"det-a": [[0], [1], [2]], "det-b": [[0], [1], [2]]} + assert set(page["timestamps"]) == {"det-a", "det-b"} + + +@pytest.mark.xfail( + strict=True, + reason="bluesky's RunBundler collects stream assets XOR event pages per " + "device, so prepare() raises on the mix; passes once that goes", +) +async def test_uc12a_mixed_pageable_and_streamable_on_one_detector(tmp_path): + """Use case 12(a), written as if the bundler limits were fixed.""" + stream_source, page_source = Source(), Source() + det = make_detector( + "det", + FakeData(stream_source, publication="stream", bounded=False, tmp_path=tmp_path), + FakeData(page_source, publication="page", bounded=True, suffix="-stats"), + acquire=FillingAcquire(stream_source, page_source, per_start=2), + ) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=2)) + assert set(await det.describe_collect()) == {"det", "det-stats"} + await det.kickoff() + await det.complete() + + docs = [d async for d in det.collect_asset_docs()] # type: ignore[attr-defined] + pages = [p async for p in det.collect_pages()] # type: ignore[attr-defined] + + assert [name for name, _ in docs] == ["stream_resource", "stream_datum"] + (page,) = pages + assert page["data"] == {"det-stats": [[0], [1]]} + + +class _BreachingData(FakeUnboundedData): + """Its provider's pages drop their last event, are `None`, or page `det` too.""" + + breach = "" + + async def start_provider(self, *args: Any) -> Any: + provider = await super().start_provider(*args) + make_page = provider.make_page + + async def breaching_make_page(index: int) -> PartialEventPage | None: + page = await make_page(index) + if page and self.breach == "short": + page = { + "data": {k: v[:-1] for k, v in page["data"].items()}, + "timestamps": {k: v[:-1] for k, v in page["timestamps"].items()}, + "time": page["time"][:-1], + } + elif self.breach == "none": + page = None + elif page and self.breach == "extra_key": + page["data"]["det"] = page["data"]["det-b"] + return page + + provider.make_page = breaching_make_page + return provider + + +@pytest.mark.parametrize( + "breach, error, match", + [ + ( + "short", + TypeError, + r"Differing numbers of events: FakePageable\[0\] has 1, " + r"FakePageable\[1\] has 0", + ), + ( + "none", + TypeError, + r"Differing numbers of events: FakePageable\[0\] has 1, " + r"FakePageable\[1\] has None", + ), + ( + "extra_key", + ValueError, + "Duplicate keys found while merging dictionaries: det", + ), + ], +) +async def test_pageable_providers_must_page_each_event_of_their_own_keys( + breach: str, error: type[Exception], match: str +): + source = Source() + breaching = _BreachingData(source, publication="page", suffix="-b") + breaching.breach = breach + det = make_detector( + "det", + FakeData(source, publication="page", bounded=False), + breaching, + acquire=FillingAcquire(source), + ) + await det.stage() + await det.trigger() + with pytest.raises(error, match=match): + await det.read() + + +def _reading_detector(suppressor: _FakeData, source: Source, value: Any): + reading = soft_signal_rw(Array1D[np.int64], value, name="det-reading") + return make_detector( + "det", suppressor, acquire=FillingAcquire(source, per_start=3), readable=None + ), reading + + +@pytest.mark.parametrize("publication", ["page", "stream"]) +async def test_readable_is_suppressed_by_pageable_and_streamable_keys( + publication, tmp_path +): + source = Source() + data = FakeData( + source, + publication=publication, + bounded=False, + suffix="-reading", + shape_tail=(), + dtype_numpy=" TriggerInfo: + return TriggerInfo(livetime=0.1, deadtime=0.02) + + +@pytest.mark.parametrize( + "env,expected_period", + [ + ("YES", pytest.approx(0.12)), + (None, 0.0), + ], +) +async def test_u12_zero_livetime_readback_is_gated_by_the_env_var( + monkeypatch: pytest.MonkeyPatch, env, expected_period +): + if env is None: + monkeypatch.delenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", raising=False) + else: + monkeypatch.setenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", env) + data = FakeData(Source(), publication="page", bounded=False) + det = make_detector( + "det", data, trigger=ReadbackTrigger(), acquire=FillingAcquire() + ) + await det.stage() + + await det.prepare(TriggerInfo(number_of_events=1)) + + assert data.periods == [expected_period] + + +# --------------------------------------------------------------------------- +# collect_pages from each state (U1): only after kickoff() +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "state,error", + [ + ( + "P", + "det: collect_pages not allowed from PREPARED, allowed from: FLOWN, FLYING", + ), + ( + "T", + "det: collect_pages not allowed from TRIGGERED, " + "allowed from: FLOWN, FLYING", + ), + ("K", None), + ("C", None), + ], +) +async def test_collect_pages_from_each_state(state: str, error: str | None): + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=False), + acquire=FillingAcquire(source), + ) + await det.stage() + await det.prepare(TriggerInfo()) + if state == "T": + await det.trigger() + elif state in "KC": + await det.kickoff() + if state == "C": + await det.complete() + + collect = det.collect_pages() # type: ignore[attr-defined] + if error is None: + assert [page async for page in collect] != [] + else: + with pytest.raises(RuntimeError, match=re.escape(error)): + _ = [page async for page in collect] + + +# --------------------------------------------------------------------------- +# Verifier findings 2 and 4 to 6: re-asking, notes, failed prepare and start +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("publication", ["stream", "page"]) +async def test_an_unbounded_provider_carries_over_without_its_logic_being_asked( + publication: str, tmp_path +): + source = Source() + unbounded = FakeData( + source, publication=publication, bounded=False, tmp_path=tmp_path + ) + det = make_detector("det", unbounded, acquire=FillingAcquire(source)) + await det.stage() + unbounded.log.clear() + info = TriggerInfo(livetime=0.1, deadtime=0.01) + await det.prepare(info) + await det.trigger() + await det.prepare(info) + await det.trigger() + await det.prepare(info) + + assert unbounded.log == ["start"] + assert set(await det.describe_collect()) == {"det"} + + +async def test_a_carried_pageable_provider_first_page_starts_at_the_new_baseline(): + source = Source() + det = make_detector( + "det", + FakeData(source, publication="page", bounded=False), + acquire=FillingAcquire(source), + ) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + await det.prepare(TriggerInfo()) + await det.kickoff() + await det.complete() + + # The first run's collection is not repeated in the second run's page + pages = [page async for page in det.collect_pages()] # type: ignore[attr-defined] + assert [page["data"] for page in pages] == [{"det": [[1]]}] + + +class NewFileData(FakeUnboundedData): + """A file writer: each start opens a new file, counting from 0.""" + + async def start_provider(self, *args: Any, **kwargs: Any) -> Any: + await self.source.rearm() + return await super().start_provider(*args, **kwargs) + + +@pytest.mark.parametrize( + "extra, info, indices", + [ + # Run 1 wrote one whole event in each file, so run 2 carries on after it + ((0, 0), TriggerInfo(collections_per_event=2), {"start": 1, "stop": 2}), + # Half an event more in each file + ((1, 1), TriggerInfo(collections_per_event=2), {"start": 0, "stop": 1}), + # A whole event more in one file only + ((2, 0), TriggerInfo(collections_per_event=2), {"start": 0, "stop": 1}), + # Run 2 takes a different number of collections per event + ( + (0, 0), + TriggerInfo(collections_per_event=1, number_of_events=2), + {"start": 0, "stop": 2}, + ), + ], + ids=["carried-over", "ragged", "different-origins", "changed-cpe"], +) +async def test_unbounded_providers_carry_over_only_from_a_whole_shared_event( + extra: tuple[int, int], info: TriggerInfo, indices: dict[str, int], tmp_path +): + sources = Source(), Source() + det = make_detector( + "det", + NewFileData(sources[0], publication="stream", tmp_path=tmp_path), + NewFileData(sources[1], publication="stream", tmp_path=tmp_path, suffix="-s"), + acquire=FillingAcquire(*sources, per_start=2), + ) + await det.stage() + await det.prepare(TriggerInfo(collections_per_event=2)) + await det.kickoff() + await det.complete() + # Frames written between the runs, say by an external trigger + for source, frames in zip(sources, extra, strict=True): + await source.produce(frames) + await det.prepare(info) + await det.kickoff() + await det.complete() + + docs = [doc async for doc in det.collect_asset_docs()] # type: ignore[attr-defined] + assert [doc["indices"] for name, doc in docs if name == "stream_datum"] == [ + indices, + indices, + ] + + +async def test_a_changed_scan_stops_the_old_provider_before_making_the_new(tmp_path): + sources = Source(), Source() + unbounded = FakeData( + sources[0], publication="stream", bounded=False, tmp_path=tmp_path + ) + steady = FakeData( + sources[1], publication="stream", bounded=False, tmp_path=tmp_path, suffix="-s" + ) + det = make_detector( + "det", unbounded, steady, acquire=FillingAcquire(*sources, per_start=2) + ) + await det.stage() + for data in ( + unbounded, + steady, + ): + data.log.clear() + await det.prepare(TriggerInfo(livetime=0.1)) + assert unbounded.log == steady.log == ["start"] + + # Each of what a provider is made for, in turn + for changed in ( + TriggerInfo(livetime=0.2), + TriggerInfo(livetime=0.2, flush_period=2.0), + TriggerInfo(livetime=0.2, flush_period=2.0, collections_per_event=2), + ): + await det.trigger() + unbounded.log.clear() + steady.log.clear() + await det.prepare(changed) + assert unbounded.log == ["stop", "start"] + assert steady.log == ["stop", "start"] + + +async def test_a_settings_change_takes_effect_at_the_next_stage_not_prepare(tmp_path): + # Stands for a plugin's EnableCallbacks being switched off in the Settings + source = Source() + plugin = FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path) + det = make_detector("det", plugin, acquire=FillingAcquire(source)) + await det.stage() + info = TriggerInfo() + await det.prepare(info) + await det.trigger() + + plugin.sit_out = True + await det.prepare(info) + assert set(await det.describe_collect()) == {"det"} + + await det.unstage() + assert plugin.log[-1] == "stop" + await det.stage() + await det.prepare(info) + assert await det.describe_collect() == {} + + +@pytest.mark.parametrize( + "verb", + [ + "stage", + "unstage", + "trigger", + "kickoff", + "complete", + "read", + "collect_pages", + "collect_asset_docs", + ], +) +async def test_f4_exceptions_from_logic_propagate(verb: str, tmp_path): + class Failing(FillingAcquire): + failing = "" + + async def ensure_ready(self): + if self.failing == "stage": + raise BoomError(self.failing) + + async def ensure_stopped(self): + if self.failing == "unstage": + raise BoomError(self.failing) + + async def acquire(self): + if self.failing in ("trigger", "kickoff"): + raise BoomError(self.failing) + async for _ in super().acquire(): + yield + if self.failing == "complete": + raise BoomError(self.failing) + + source = Source() + acquire = Failing(source) + publication = "stream" if verb == "collect_asset_docs" else "page" + data = FakeData(source, publication=publication, bounded=False, tmp_path=tmp_path) + det = make_detector("det", data, acquire=acquire) + # Before kickoff, as fly() runs on to the end of acquire() as soon as it can + acquire.failing = verb + if verb != "stage": + await det.stage() + if verb not in ("stage", "unstage"): + await det.prepare(TriggerInfo(number_of_events=1)) + if verb == "read": + await det.trigger() + if verb in ("complete", "collect_pages", "collect_asset_docs"): + await det.kickoff() + data.fail = verb in ("read", "collect_pages", "collect_asset_docs") + + with pytest.raises(BoomError): + if verb == "collect_pages": + _ = [page async for page in det.collect_pages()] # type: ignore[attr-defined] + elif verb == "collect_asset_docs": + _ = [doc async for doc in det.collect_asset_docs()] # type: ignore[attr-defined] + else: + await getattr(det, verb)() + + +class YieldsTwice(FillingAcquire): + async def acquire(self): + yield + yield + + +async def test_acquire_that_yields_twice_raises(): + det = make_detector("det", acquire=YieldsTwice()) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=1)) + + with pytest.raises(RuntimeError, match=r"YieldsTwice\.acquire\(\) must yield only"): + await det.trigger() + + +@pytest.mark.parametrize( + "bounded_fails, with_bounded, keys", + [ + (False, False, ["det"]), + (True, True, ["det", "det-b"]), + (False, True, ["det", "det-b"]), + ], +) +async def test_prepare_after_a_failed_start_is_a_first_prepare( + bounded_fails: bool, with_bounded: bool, keys: list[str], tmp_path +): + source, bounded_source = Source(), Source() + unbounded = FakeData(source, publication="stream", bounded=False, tmp_path=tmp_path) + data = [unbounded] + if with_bounded: + data.append( + FakeData( + bounded_source, + publication="stream", + bounded=True, + tmp_path=tmp_path, + suffix="-b", + ) + ) + failing = data[-1] if bounded_fails else unbounded + det = make_detector("det", *data, acquire=FillingAcquire(source, bounded_source)) + await det.stage() + failing.fail_start = True + with pytest.raises(BoomError, match="start"): + await det.prepare(TriggerInfo()) + + failing.fail_start = False + await det.prepare(TriggerInfo()) + + assert list(await det.describe_collect()) == keys + + +@pytest.mark.parametrize("publication", ["page", "stream"]) +@pytest.mark.parametrize("cpe", [1, 3]) +async def test_scalar_readable_is_suppressed_by_a_key_of_any_collections_per_event( + publication: str, cpe: int, tmp_path +): + """A readable gives one value per event, so [cpe] suppresses it, not just [].""" + source = Source() + data = FakeData( + source, + publication=publication, + bounded=False, + suffix="-reading", + tmp_path=tmp_path, + ) + det = make_detector("det", data, acquire=FillingAcquire(source, per_start=cpe)) + det.reading = soft_signal_rw(int, 9, name="det-reading") + det.set_readable_format(det.reading, Format.HINTED_SIGNAL) + await det.stage() + await det.prepare(TriggerInfo(collections_per_event=cpe)) + await det.trigger() + + description = await det.describe() + + assert description["det-reading"]["shape"] == [cpe] + reading = await det.read() + if publication == "page": + assert reading["det-reading"]["value"] == list(range(cpe)) + else: + assert reading == {} + + +def test_data_logic_with_the_removed_stop_raises(): + with pytest.raises(TypeError, match=r"StaleData defines stop, which .* -> ensure"): + type("StaleData", (BoundedDataLogic,), {"stop": lambda self: None}) diff --git a/tests/unit_tests/core/test_readable.py b/tests/unit_tests/core/test_readable.py index 40ef5f3387..71c72baf5e 100644 --- a/tests/unit_tests/core/test_readable.py +++ b/tests/unit_tests/core/test_readable.py @@ -3,6 +3,7 @@ import pytest from bluesky.protocols import HasHints +from event_model import DataKey from ophyd_async.core import ( Device, @@ -12,6 +13,7 @@ SignalR, StandardReadable, TransitionError, + set_mock_attr, soft_signal_r_and_setter, soft_signal_rw, ) @@ -553,3 +555,81 @@ def test_set_readable_format_rejects_non_signal_for_signal_format(): sr.child = Device(name="child") with pytest.raises(TypeError, match="is not a SignalR"): sr.set_readable_format(sr.child, Format.CONFIG_SIGNAL) + + +class _Suppressing(StandardReadable): + """A readable that is told which names to suppress, as a detector would.""" + + def __init__(self, name: str = ""): + self.suppressed: dict[str, DataKey | None] = {} + super().__init__(name) + self._suppressed_datakeys_funcs += (lambda: self.suppressed,) + + +class _Pair(StandardReadable): + """A child that reads two hinted signals, to be suppressed from its parent.""" + + def __init__(self, name: str = ""): + with self.add_children_as_readables(Format.HINTED_SIGNAL): + self.one = soft_signal_rw(int, 1) + self.two = soft_signal_rw(int, 2) + super().__init__(name) + + +async def _suppressing(suppressed: dict[str, DataKey | None]) -> _Suppressing: + sr = _Suppressing("sr") + sr.pair = _Pair() + sr.sig = soft_signal_rw(int, 3) + sr.set_readable_format(sr.pair, Format.CHILD) + sr.set_readable_format(sr.sig, Format.HINTED_SIGNAL) + sr.set_name("sr") + await sr.connect(mock=True) + sr.suppressed = suppressed + return sr + + +async def _datakeys() -> dict[str, DataKey]: + return await (await _suppressing({})).describe() + + +@pytest.mark.parametrize("suppressed_name", ["sr-sig", "sr-pair-two"]) +async def test_a_switched_off_name_is_dropped_from_read_describe_and_hints( + suppressed_name: str, +): + sr = await _suppressing({suppressed_name: None}) + + expected = {"sr-pair-one", "sr-pair-two", "sr-sig"} - {suppressed_name} + assert set(await sr.read()) == expected + assert set(await sr.describe()) == expected + assert set(sr.hints["fields"]) == expected + + +@pytest.mark.parametrize("suppressed_name", ["sr-sig", "sr-pair-two"]) +async def test_a_replaced_name_is_dropped_from_read_and_describe_but_kept_in_hints( + suppressed_name: str, +): + replacement = (await _datakeys())[suppressed_name] + sr = await _suppressing({suppressed_name: replacement}) + + expected = {"sr-pair-one", "sr-pair-two", "sr-sig"} - {suppressed_name} + assert set(await sr.read()) == expected + assert set(await sr.describe()) == expected + assert set(sr.hints["fields"]) == {"sr-pair-one", "sr-pair-two", "sr-sig"} + + +async def test_a_replacement_of_a_different_dtype_is_a_duplicate_datakey(): + replacement = (await _datakeys())["sr-sig"] + replacement["dtype_numpy"] = " StreamableDataProvider: + self.shape = shape + self.dtype_numpy = dtype_numpy + + async def start_provider( + self, + name: str, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> StreamResourceDataProvider: resource = StreamResourceInfo( - data_key=datakey_name, - shape=(10, 15), - chunk_shape=(1, 10, 15), - dtype_numpy="|u1", + data_key=name + self.datakey_suffix, + shape=self.shape, + chunk_shape=(1, *self.shape), + dtype_numpy=self.dtype_numpy, parameters={"dataset": "/data"}, ) - provider = StreamResourceDataProvider( + return StreamResourceDataProvider( uri=f"file://localhost/{self.tmp_path.as_posix().lstrip('/')}/test.h5", resources=[resource], mimetype="application/x-hdf5", - collections_written_signal=self.collections_written, + collections_written=self.collections_written, + collections_per_event=collections_per_event, ) - return provider - async def stop(self) -> None: + async def ensure_stopped(self) -> None: self.stop_count += 1 - def get_hinted_fields(self, datakey_name: str) -> Sequence[str]: - return [datakey_name] + def get_hinted_fields(self, name: str) -> Sequence[str]: + return [name + self.datakey_suffix] async def test_stream_resource_data_provider_1d_single_collection_is_array(): @@ -174,10 +195,11 @@ async def test_stream_resource_data_provider_1d_single_collection_is_array(): ) ], mimetype="application/x-hdf5", - collections_written_signal=soft_signal_rw(int), + collections_written=soft_signal_rw(int), + collections_per_event=1, ) - assert await provider.make_datakeys(collections_per_event=1) == { + assert await provider.make_datakeys() == { "det-spectrum": { "dtype": "array", "dtype_numpy": " None: + self.published = 0 + + async def make_datakeys(self) -> dict[str, DataKey]: + return { + self.datakey_name: DataKey( + source="mock", + shape=[self.collections_per_event], + dtype="array", + dtype_numpy=" PartialEventPage: + new = range(events) + return { + "data": { + self.datakey_name: [[0] * self.collections_per_event for _ in new] + }, + "time": [0.0 for _ in new], + "timestamps": {self.datakey_name: [0.0 for _ in new]}, + } + + async def make_page(self, index: int) -> PartialEventPage | None: + if index <= self.published: + return None + page = self._page(index - self.published) + self.published = index + return page + + +class BoundedOnlyDataLogic(BoundedDataLogic[MockPageableProvider]): + """Produces bounded data held in a finite buffer, sized when it is armed.""" + + def __init__(self, datakey_suffix: str = ""): + self.datakey_suffix = datakey_suffix + self.collections_written = soft_signal_rw(int) + self.prepare_calls: list[tuple[int, float]] = [] + self.stop_count = 0 + self.providers: list[MockPageableProvider] = [] + + async def start_provider( + self, + name: str, + existing_datakeys: Sequence[str], + num_collections: int, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> MockPageableProvider | None: + datakey_name = name + self.datakey_suffix + if datakey_name in existing_datakeys: + return None + self.prepare_calls.append((num_collections, collection_period)) + # A real buffer clears its progress counter when armed + await self.collections_written.set(0) + provider = MockPageableProvider( + datakey_name, self.collections_written, collections_per_event + ) + self.providers.append(provider) + return provider + + async def ensure_stopped(self) -> None: + self.stop_count += 1 + + def get_hinted_fields(self, name: str) -> Sequence[str]: + return [name + self.datakey_suffix] + + # Parameterized Tests for Logic Combinations @@ -216,9 +320,8 @@ async def test_get_trigger_deadtime( trigger_logic, expected_triggers, expected_deadtime ): """Test get_trigger_deadtime with various trigger logic implementations.""" - det = StandardDetector() - if trigger_logic: - det.add_detector_logics(trigger_logic) + det = DetectorLogic(trigger=trigger_logic).with_device() + await det.stage() if isinstance(trigger_logic, DeadtimeTriggerLogic): # A logic that needs a signal declares it as configuration; the logic # no longer nominates signals separately @@ -234,10 +337,9 @@ async def test_get_trigger_deadtime_with_settings(): deadtime_signal = soft_signal_rw(float, 0.02) # Create detector with DeadtimeTriggerLogic - det = StandardDetector() + det = DetectorLogic(trigger=DeadtimeTriggerLogic(deadtime_signal)).with_device() + await det.stage() det.sig = deadtime_signal - tl = DeadtimeTriggerLogic(deadtime_signal) - det.add_detector_logics(tl) det.set_readable_format(deadtime_signal, Format.CONFIG_SIGNAL) # Verify initial deadtime from signal @@ -266,9 +368,9 @@ async def test_get_trigger_deadtime_with_settings(): ) async def test_prepare_trigger_types(trigger_type): """Test each trigger type is properly delegated to trigger logic.""" - det = StandardDetector() trigger_logic = AllTriggerTypesLogic() - det.add_detector_logics(trigger_logic) + det = DetectorLogic(trigger=trigger_logic).with_device() + await det.stage() trigger_info = TriggerInfo( trigger=trigger_type, livetime=0.5, deadtime=0.1, number_of_events=10 @@ -282,8 +384,8 @@ async def test_prepare_trigger_types(trigger_type): async def test_prepare_unsupported_trigger_type(): """Test that preparing with unsupported trigger type raises error.""" - det = StandardDetector() - det.add_detector_logics(JustInternalTriggerLogic()) + det = DetectorLogic(trigger=JustInternalTriggerLogic()).with_device() + await det.stage() with pytest.raises(ValueError, match="Trigger type.*EXTERNAL_EDGE not supported"): await det.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE)) @@ -320,9 +422,9 @@ async def test_trigger_info_calculations( @pytest.mark.parametrize("exposures_per_collection", [1, 2, 5, 10]) async def test_exposures_per_collection(exposures_per_collection): """Test exposure averaging configuration.""" - det = StandardDetector() tl = AveragingTriggerLogic() - det.add_detector_logics(tl) + det = DetectorLogic(trigger=tl).with_device() + await det.stage() await det.prepare( TriggerInfo( @@ -337,8 +439,8 @@ async def test_exposures_per_collection(exposures_per_collection): async def test_exposures_per_collection_not_supported(): """Test that exposures_per_collection > 1 fails without supporting logic.""" - det = StandardDetector() - det.add_detector_logics(JustInternalTriggerLogic()) # Doesn't support averaging + det = DetectorLogic(trigger=JustInternalTriggerLogic()).with_device() + await det.stage() with pytest.raises( ValueError, match="Multiple exposures per collection not supported" @@ -356,11 +458,11 @@ async def test_exposures_per_collection_not_supported(): ) async def test_arm_timing(trigger_type, arm_timing, tmp_path): """Verify detector is armed at the correct time based on trigger type.""" - det = StandardDetector() tl = AllTriggerTypesLogic() al = MockAcquireLogic() dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(tl, al, dl) + det = DetectorLogic(trigger=tl, acquire=al, data=[dl]).with_device() + await det.stage() # Prepare the detector await det.prepare(TriggerInfo(trigger=trigger_type, number_of_events=2)) @@ -385,25 +487,40 @@ async def test_arm_timing(trigger_type, arm_timing, tmp_path): # External triggers should still be armed from prepare assert al.arm_count == 1 assert al.armed is True + await det.unstage() -async def test_trigger_arms_detector(tmp_path): - """Test that trigger() arms the detector when arm logic is present.""" - det = StandardDetector() +@pytest.mark.parametrize( + "trigger_type,arms_at_trigger", + [ + (DetectorTrigger.INTERNAL, True), + (DetectorTrigger.EXTERNAL_EDGE, False), + (DetectorTrigger.EXTERNAL_LEVEL, False), + ], +) +async def test_trigger_arms_only_an_internally_triggered_detector( + trigger_type, arms_at_trigger, tmp_path +): + """trigger() arms the detector, unless prepare() already did. + + An externally triggered detector was armed by prepare() and is sitting + waiting on its trigger source, so arming it again would restart it. + """ al = MockAcquireLogic() dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(JustInternalTriggerLogic(), al, dl) + det = DetectorLogic( + trigger=AllTriggerTypesLogic(), acquire=al, data=[dl] + ).with_device() + await det.stage() - await det.prepare(TriggerInfo()) + await det.prepare(TriggerInfo(trigger=trigger_type)) + assert al.arm_count == (0 if arms_at_trigger else 1) + assert al.armed is not arms_at_trigger - # Should not be armed yet - assert al.armed is False - assert al.arm_count == 0 - - # Trigger should arm it status = det.trigger() # Give it a moment to arm await wait_for_pending_wakeups(raise_if_exceeded=False) + # Armed either way by now, but only ever once assert al.armed is True assert al.arm_count == 1 @@ -412,11 +529,43 @@ async def test_trigger_arms_detector(tmp_path): await status +class NoYieldAcquireLogic(MockAcquireLogic): + async def acquire(self): + return + yield + + +@pytest.mark.parametrize("entry", ["trigger", "fly", "external_fly"]) +async def test_acquire_that_never_yields_raises(tmp_path, entry): + dl = StreamableOnlyDataLogic(tmp_path) + det = DetectorLogic( + acquire=NoYieldAcquireLogic(), data=[dl], trigger=AllTriggerTypesLogic() + ).with_device() + await det.stage() + external = entry == "external_fly" + info = TriggerInfo( + trigger=DetectorTrigger.EXTERNAL_EDGE if external else DetectorTrigger.INTERNAL, + livetime=0.1, + ) + with pytest.raises( + RuntimeError, + match=r"NoYieldAcquireLogic\.acquire\(\) finished without yielding", + ): + await det.prepare(info) + if entry == "trigger": + status = det.trigger() + else: + await det.kickoff() + status = det.complete() + await wait_for_pending_wakeups() + await dl.collections_written.set(1) + await status + + async def test_arm_logic_called_on_stage(): """Test that acquire logic is stopped on stage.""" - det = StandardDetector() al = MockAcquireLogic() - det.add_detector_logics(al) + det = DetectorLogic(acquire=al).with_device() al.armed = True # Simulate being armed await det.stage() @@ -427,26 +576,26 @@ async def test_arm_logic_called_on_stage(): async def test_describe_before_prepare_raises(tmp_path): """Test that describe() fails before prepare().""" - det = StandardDetector() - det.add_detector_logics(StreamableOnlyDataLogic(tmp_path)) + det = DetectorLogic(data=[StreamableOnlyDataLogic(tmp_path)]).with_device() + await det.stage() - with pytest.raises(RuntimeError, match="Prepare not run"): + with pytest.raises(TransitionError, match="not allowed from STAGED"): await det.describe() async def test_describe_collect_before_prepare_raises(tmp_path): """Test that describe_collect() fails before prepare().""" - det = StandardDetector() - det.add_detector_logics(StreamableOnlyDataLogic(tmp_path)) + det = DetectorLogic(data=[StreamableOnlyDataLogic(tmp_path)]).with_device() + await det.stage() - with pytest.raises(RuntimeError, match="Prepare not run"): + with pytest.raises(TransitionError, match="not allowed from STAGED"): await det.describe_collect() async def test_trigger_after_multi_event_prepare_raises(): """Test that trigger() after prepare with multiple events fails.""" - det = StandardDetector() - det.add_detector_logics(JustInternalTriggerLogic()) + det = DetectorLogic(trigger=JustInternalTriggerLogic()).with_device() + await det.stage() await det.prepare(TriggerInfo(number_of_events=5)) @@ -459,15 +608,14 @@ async def test_preserve_detector_state_requires_default_trigger_info( ): """Test PRESERVE_DETECTOR_STATE=YES errors without default_trigger_info.""" monkeypatch.setenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "YES") - det = StandardDetector(name="mydet") + det = DetectorLogic(trigger=AllTriggerTypesLogic()).with_device("mydet") # AllTriggerTypesLogic intentionally does not implement default_trigger_info - det.add_detector_logics(AllTriggerTypesLogic()) await det.stage() with pytest.raises( RuntimeError, - match="OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES is set but 'mydet' has no " - "default_trigger_info\\(\\)", + match="OPHYD_ASYNC_PRESERVE_DETECTOR_STATE=YES is set but the " + "trigger logic has no default_trigger_info\\(\\)", ): await det.trigger() @@ -479,12 +627,12 @@ async def test_preserve_detector_state_no_trigger_logic_falls_back( trigger() silently falls back to a bare TriggerInfo() rather than raising. A detector with no trigger logic has no hardware state to preserve.""" monkeypatch.setenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "YES") - det = StandardDetector(name="nodet") + det = DetectorLogic().with_device("nodet") await det.stage() - # Should not raise — no trigger logic means nothing to preserve + # Should not raise: no trigger logic means nothing to preserve await det.trigger() - assert det._prepare_ctx is not None - assert det._prepare_ctx.trigger_info == TriggerInfo() + # ...and it was prepared, implicitly + assert await det.describe() == {} async def test_preserve_detector_state_multi_collection_watcher_and_assets( @@ -503,9 +651,10 @@ async def default_trigger_info(self) -> TriggerInfo: return TriggerInfo(collections_per_event=5) monkeypatch.setenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "YES") - det = StandardDetector(name="multidet") dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(MultiCollectionTriggerLogic(), dl) + det = DetectorLogic(trigger=MultiCollectionTriggerLogic(), data=[dl]).with_device( + "multidet" + ) await det.stage() # Collect watcher updates via watch() callback @@ -529,11 +678,6 @@ async def default_trigger_info(self) -> TriggerInfo: assert all(update["target"] == 5 for update in updates) -@pytest.mark.xfail( - reason="collect_asset_docs() checks ``dl.collections_written`` again after " - "``trigger()``", - strict=True, -) async def test_collect_asset_docs_uses_trigger_observed_event( tmp_path, ): @@ -542,12 +686,14 @@ async def test_collect_asset_docs_uses_trigger_observed_event( It could be the case that ``dl.collections_written`` changes between ``trigger()`` and ``collect_asset_docs``.""" - det = StandardDetector(name="det") dl = StreamableOnlyDataLogic(tmp_path) dl.collections_written, set_collections_written = soft_signal_r_and_setter( int, getter=lambda: 0 ) - det.add_detector_logics(JustInternalTriggerLogic(), dl) + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "det" + ) + await det.stage() await det.prepare(TriggerInfo(collections_per_event=5, exposure_timeout=0.1)) status = det.trigger() @@ -561,66 +707,60 @@ async def test_collect_asset_docs_uses_trigger_observed_event( async def test_trigger_times_out_on_collection_timeout(tmp_path): - det = StandardDetector(name="det") - det.add_detector_logics( - JustInternalTriggerLogic(), StreamableOnlyDataLogic(tmp_path) - ) + det = DetectorLogic( + trigger=JustInternalTriggerLogic(), data=[StreamableOnlyDataLogic(tmp_path)] + ).with_device("det") + await det.stage() await det.prepare(TriggerInfo(exposure_timeout=100, collection_timeout=0.1)) with pytest.raises(TimeoutError): await det.trigger() -async def test_kickoff_respects_prepare_bounds(tmp_path): - """Test that multiple kickoff() calls respect prepared bounds.""" - det = StandardDetector() - tl = JustInternalTriggerLogic() +async def test_one_kickoff_per_prepare(tmp_path): + """A prepare serves one kickoff/complete cycle; the next needs a new prepare.""" dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(tl, dl) + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "foo" + ) + await det.stage() - # Prepare for 5 events await det.prepare(TriggerInfo(number_of_events=5)) - - # Update collections_written signal to simulate data being written - - # First kickoff for 3 events - await det.events_to_kickoff.set(3) await det.kickoff() - await dl.collections_written.set(3) - # Second kickoff for 2 events should work (total = 5) - await det.events_to_kickoff.set(2) - await det.kickoff() + # A second kickoff without an intervening complete is rejected + with pytest.raises(RuntimeError, match="cannot go to FLYING from FLYING"): + await det.kickoff() + + status = det.complete() await dl.collections_written.set(5) + await status - # Third kickoff should fail (would exceed 5) - await det.events_to_kickoff.set(1) - with pytest.raises( - RuntimeError, - match="Kickoff requested 5:6, but detector was only prepared up to 5", - ): + # ...and so is one after complete, until prepare() runs again + with pytest.raises(RuntimeError, match="cannot go to FLYING from FLOWN"): await det.kickoff() -async def test_stage_resets_state(): - """Test that stage() resets detector state.""" - det = StandardDetector() - det.add_detector_logics(JustInternalTriggerLogic()) +async def test_restage_resets_state(): + """Test that unstage() then stage() resets detector state.""" + det = DetectorLogic(trigger=JustInternalTriggerLogic()).with_device() + await det.stage() await det.prepare(TriggerInfo(number_of_events=5)) - await det.events_to_kickoff.set(3) - # Stage should reset everything + # Restaging should reset everything + await det.unstage() await det.stage() - assert det._prepare_ctx is None - assert det._kickoff_ctx is None - assert await det.events_to_kickoff.get_value() == 0 + with pytest.raises(TransitionError, match="describe not allowed from STAGED"): + await det.describe() + with pytest.raises(TransitionError, match="cannot go to FLYING from STAGED"): + await det.kickoff() async def test_hints_from_single_data_logic(tmp_path): """Test that hints come from data logic.""" - det = StandardDetector(name="bar") - det.add_detector_logics(StreamableOnlyDataLogic(tmp_path)) + det = DetectorLogic(data=[StreamableOnlyDataLogic(tmp_path)]).with_device("bar") + await det.stage() await det.prepare(TriggerInfo()) @@ -629,10 +769,10 @@ async def test_hints_from_single_data_logic(tmp_path): async def test_hints_from_multiple_data_logics(tmp_path): """Test that hints are aggregated from multiple data logics.""" - det = StandardDetector(name="bar") dl1 = StreamableOnlyDataLogic(tmp_path) dl2 = StreamableOnlyDataLogic(tmp_path, datakey_suffix="-extra") - det.add_detector_logics(dl1, dl2) + det = DetectorLogic(data=[dl1, dl2]).with_device("bar") + await det.stage() await det.prepare(TriggerInfo()) @@ -654,14 +794,13 @@ async def test_config_signals_in_describe_configuration( signal_type, initial_value, expected_dtype, expected_dtype_numpy, expected_shape ): """Test that added config signals appear in describe_configuration.""" - det = StandardDetector() + det = DetectorLogic().with_device() + await det.stage() signal = soft_signal_rw( signal_type, initial_value=initial_value, name="test-config" ) det.set_readable_format(signal, Format.CONFIG_SIGNAL) - await det.stage() - # Check describe_configuration config_desc = await det.describe_configuration() assert config_desc == { @@ -682,38 +821,23 @@ async def test_config_signals_in_describe_configuration( ) -async def test_kickoff_without_streamable_data_raises(): - """Test that kickoff() without streamable data fails.""" - det = StandardDetector(name="foo") - det.add_detector_logics(JustInternalTriggerLogic()) - - # A detector with no data logic can still be triggered for a step scan - await det.prepare(TriggerInfo()) - await det.trigger() # This works - - # ...but there is nothing to collect, so it cannot be flown - await det.prepare(TriggerInfo(number_of_events=5)) - with pytest.raises( - ValueError, match="Detector foo is not streamable, so cannot kickoff" - ): - await det.kickoff() - - async def test_streamable_supports_both_step_and_fly(tmp_path): """Test that streamable data logic supports both step and fly scanning.""" - det = StandardDetector(name="foo") tl = JustInternalTriggerLogic() dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(tl, dl) + det = DetectorLogic(trigger=tl, data=[dl]).with_device("foo") + await det.stage() # Step scan should work status = det.trigger() # Yield so detector can get collections written, then set it so we complete await wait_for_pending_wakeups(raise_if_exceeded=False) await dl.collections_written.set(1) - await status + await wait_for_pending_wakeups(raise_if_exceeded=False) + assert status.done assert status.success docs = [doc async for doc in det.collect_asset_docs()] + sr = docs[:1] assert docs == [ ( "stream_resource", @@ -743,17 +867,21 @@ async def test_streamable_supports_both_step_and_fly(tmp_path): await det.prepare(TriggerInfo(number_of_events=5)) await det.kickoff() status = det.complete() - # Check that setting collections written will only give the first 4 + # Check that setting collections written will only give the first 4, in a + # new resource for the new prepare await dl.collections_written.set(5) docs = [doc async for doc in det.collect_asset_docs()] - assert docs == [ + new_sr = docs[:1] + assert new_sr == [(name, {**doc, "uid": ANY}) for name, doc in sr] + assert new_sr[0][1]["uid"] != sr[0][1]["uid"] + assert docs[1:] == [ ( "stream_datum", { "descriptor": "", "indices": {"start": 1, "stop": 5}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": ANY, + "stream_resource": new_sr[0][1]["uid"], "uid": ANY, }, ), @@ -779,7 +907,8 @@ async def test_streamable_supports_both_step_and_fly(tmp_path): async def test_read_returns_correct_values(): """Test that read() returns the values of registered signals.""" - det = StandardDetector(name="det") + det = DetectorLogic().with_device("det") + await det.stage() det.counts, _ = soft_signal_r_and_setter(int, 42, name="det-counts") det.set_readable_format(det.counts, Format.HINTED_SIGNAL) @@ -789,7 +918,8 @@ async def test_read_returns_correct_values(): async def test_detector_with_no_logics(): """Test that detector works with no logics for basic internal triggering.""" - det = StandardDetector() + det = DetectorLogic().with_device() + await det.stage() # Should support only INTERNAL triggering triggers, deadtime = await det.get_trigger_deadtime() @@ -798,56 +928,58 @@ async def test_detector_with_no_logics(): # Can prepare with INTERNAL trigger but not external await det.prepare(TriggerInfo()) - - with pytest.raises(ValueError, match="Trigger type.*not supported"): + await det.trigger() + with pytest.raises(ValueError, match="Trigger type.*EXTERNAL_EDGE not supported"): await det.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE)) -async def test_detector_without_trigger_logic_cannot_set_timing_or_exteral_triggering(): - """Test that detector without trigger logic cannot set livetime/deadtime/trigger.""" - det = StandardDetector(name="foo") - msg = "Detector foo has no trigger logic, so " - - with pytest.raises(ValueError, match=msg + "cannot set livetime or deadtime"): - await det.prepare(TriggerInfo(livetime=0.5)) - - with pytest.raises(ValueError, match=msg + "cannot set livetime or deadtime"): - await det.prepare(TriggerInfo(deadtime=0.1)) - - -async def test_cannot_add_two_trigger_logics(): - """Test that adding two trigger logics raises an error.""" - det = StandardDetector() - tl1 = JustInternalTriggerLogic() - tl2 = AllTriggerTypesLogic() - - det.add_detector_logics(tl1) - - with pytest.raises(RuntimeError, match="Detector already has trigger logic"): - det.add_detector_logics(tl2) +@pytest.mark.parametrize( + "info", + [ + TriggerInfo(livetime=0.5), + TriggerInfo(deadtime=0.1), + TriggerInfo(exposures_per_collection=2), + ], +) +async def test_detector_without_trigger_logic_cannot_set_timing(info: TriggerInfo): + det = DetectorLogic().with_device("foo") + await det.stage() + with pytest.raises(ValueError, match="No trigger logic, so cannot set livetime"): + await det.prepare(info) -async def test_cannot_add_two_arm_logics(): - """Test that adding two acquire logics raises an error.""" - det = StandardDetector() - al1 = MockAcquireLogic() - al2 = MockAcquireLogic() +async def test_errors_from_logic_calls_keep_their_message(): + det = DetectorLogic().with_device("foo") + await det.stage() + with pytest.raises(ValueError) as exc_info: + await det.prepare(TriggerInfo(livetime=0.5)) + assert exc_info.value.args == ( + "No trigger logic, so cannot set livetime or deadtime " + "or exposures_per_collection", + ) - det.add_detector_logics(al1) - with pytest.raises(RuntimeError, match="Detector already has acquire logic"): - det.add_detector_logics(al2) +async def test_detector_logic_takes_keywords_only(): + """Each logic's role is named, so none can be given positionally.""" + with pytest.raises(TypeError, match="positional"): + DetectorLogic(JustInternalTriggerLogic()) # type: ignore[misc] -async def test_add_unknown_logic_type_raises(): - """Test that adding an unknown logic type raises TypeError.""" - det = StandardDetector() +async def test_replaced_detector_logic_does_not_share_prepared_data(tmp_path): + """A copy made with dataclasses.replace has its own prepared state.""" + logic = DetectorLogic( + trigger=JustInternalTriggerLogic(), data=[StreamableOnlyDataLogic(tmp_path)] + ) + det = logic.with_device("det") + copy = dataclasses.replace(logic).with_device("copy") + await det.stage() + await copy.stage() - class UnknownLogic: - pass + await det.prepare(TriggerInfo()) - with pytest.raises(TypeError, match="Unknown logic type"): - det.add_detector_logics(UnknownLogic()) + assert list(await det.describe()) == ["det"] + with pytest.raises(TransitionError, match="describe not allowed from STAGED"): + await copy.describe() @pytest.mark.parametrize("initial_shutter_closed", [True, False]) @@ -868,18 +1000,15 @@ def __init__(self, shutter_closed: bool): async def ensure_ready(self): pass # don't touch the shutter at stage time - async def start_acquiring(self): + async def acquire(self): self.shutter_closed = False # open shutter when acquiring - - async def wait_for_idle(self): - pass + yield async def ensure_stopped(self): self.shutter_closed = True # close shutter at end of scan - det = StandardDetector() al = ShutterAcquireLogic(initial_shutter_closed) - det.add_detector_logics(al) + det = DetectorLogic(acquire=al).with_device() await det.stage() assert ( @@ -887,86 +1016,207 @@ async def ensure_stopped(self): ) # stage() must not touch the shutter await det.trigger() - assert al.shutter_closed is False # start_acquiring() opens the shutter + assert al.shutter_closed is False # acquire() opens the shutter await det.unstage() assert al.shutter_closed is True # unstage() must close the shutter -async def test_data_logic_with_no_prepare_methods_raises(): - """Test error when DataLogic doesn't override any prepare methods.""" +async def test_bounded_step_scan_reads_derived_from_page(): + """A bounded buffer describes as an array and read() derives one reading.""" + dl = BoundedOnlyDataLogic() + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "foo" + ) + await det.stage() + + ti = TriggerInfo( + number_of_events=1, collections_per_event=5, livetime=0.1, deadtime=0.0 + ) + await det.prepare(ti) + desc = await det.describe() + assert desc["foo"]["shape"] == [5] + # prepare sized the buffer for 5 collections at the 0.1s period + assert dl.prepare_calls == [(5, pytest.approx(0.1))] + + status = det.trigger() + await wait_for_pending_wakeups(raise_if_exceeded=False) + await dl.collections_written.set(5) + await status + assert status.success + # prepare() armed the buffer for this point, so trigger() does not re-arm it + assert len(dl.providers) == 1 + assert dl.stop_count == 1 + reading = await det.read() + assert reading["foo"]["value"] == [0, 0, 0, 0, 0] + + # The next point stops, re-makes and re-starts the logic, as the buffer + # holds the last point's data, and counts from the re-armed buffer's 0 + status = det.trigger() + await wait_for_pending_wakeups(raise_if_exceeded=False) + assert len(dl.providers) == 2 + assert dl.stop_count == 2 + assert not status.done + await dl.collections_written.set(5) + await status + assert status.success + assert dl.prepare_calls == [(5, pytest.approx(0.1))] * 2 + + +async def test_bounded_fly_scan_accumulates_across_kickoffs(): + """A bounded buffer is armed once at prepare and not re-armed per kickoff.""" + dl = BoundedOnlyDataLogic() + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "foo" + ) + await det.stage() + + await det.prepare( + TriggerInfo(number_of_events=10, collections_per_event=1, livetime=0.1) + ) + assert dl.prepare_calls == [(10, pytest.approx(0.1))] + + await det.kickoff() + status = det.complete() + # The buffer fills as the scan runs, and pages come out as it goes + await dl.collections_written.set(5) + pages = [page async for page in det.collect_pages()] # type: ignore[attr-defined] + assert pages[0]["data"]["foo"] == [[0]] * 5 + assert not status.done + + await dl.collections_written.set(10) + pages = [page async for page in det.collect_pages()] # type: ignore[attr-defined] + assert pages[0]["data"]["foo"] == [[0]] * 5 + await status + # kickoff() never re-arms: the buffer was only ever armed once + assert dl.prepare_calls == [(10, pytest.approx(0.1))] + + +@pytest.mark.parametrize( + "requested_livetime,default,expected_period", + [ + # livetime unset -> resolved from the trigger logic's current state + (0.0, TriggerInfo(livetime=0.1, deadtime=0.02), 0.12), + # livetime given -> used as-is, no readback substitution + (0.05, TriggerInfo(livetime=0.1, deadtime=0.02), 0.05), + # livetime unset but trigger logic has nothing set -> stays 0 + (0.0, TriggerInfo(), 0.0), + ], +) +async def test_prepare_resolves_zero_livetime_for_bounded_period( + requested_livetime, default, expected_period, monkeypatch: pytest.MonkeyPatch +): + """A livetime of 0 is filled in from the trigger logic before sizing a buffer. + + Only when hardware state is preserved: test_u12 in test_detector_use_cases + covers the variable being unset. + """ + monkeypatch.setenv("OPHYD_ASYNC_PRESERVE_DETECTOR_STATE", "YES") + + class PeriodTriggerLogic(DetectorTriggerLogic): + async def prepare_internal(self, num: int, livetime: float, deadtime: float): + pass + + async def default_trigger_info(self) -> TriggerInfo: + return default + + dl = BoundedOnlyDataLogic() + det = DetectorLogic(trigger=PeriodTriggerLogic(), data=[dl]).with_device("foo") + await det.stage() + + await det.prepare( + TriggerInfo( + number_of_events=1, collections_per_event=3, livetime=requested_livetime + ) + ) + assert dl.prepare_calls == [(3, pytest.approx(expected_period))] + + +async def test_data_logic_with_no_start_provider_raises(): + """Test error when a DataLogic does not implement start_provider.""" - class EmptyDataLogic(DetectorDataLogic): - pass + class EmptyDataLogic(UnboundedDataLogic): + async def start_provider(self, *args): + return await super().start_provider(*args) # type: ignore[safe-super] - det = StandardDetector() - det.add_detector_logics(EmptyDataLogic()) + det = DetectorLogic(data=[EmptyDataLogic()]).with_device() + await det.stage() - with pytest.raises(RuntimeError, match="hasn't overridden any prepare_\\* methods"): + with pytest.raises(NotImplementedError): await det.prepare(TriggerInfo()) async def test_unstage_disarms_detector(): """Test that unstage() calls disarm on the detector.""" - det = StandardDetector() al = MockAcquireLogic() - det.add_detector_logics(al) + det = DetectorLogic(acquire=al).with_device() + await det.stage() al.armed = True await det.unstage() - assert al.disarm_count == 1 + # Once on stage(), once on unstage() + assert al.disarm_count == 2 assert al.armed is False -async def test_prepare_stops_data_logic_when_recreating_providers(tmp_path): - """Test that prepare() calls stop() on data logic when recreating providers.""" - det = StandardDetector(name="det") +@pytest.mark.parametrize( + "second_collections_per_event,stops", [(2, 0), (3, 1)], ids=["kept", "remade"] +) +async def test_prepare_stops_data_logic_when_recreating_providers( + tmp_path, second_collections_per_event: int, stops: int +): + """A streamable provider is kept while it would be made the same, else remade.""" dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(JustInternalTriggerLogic(), dl) + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "det" + ) + await det.stage() + staged_stops = dl.stop_count - # First prepare with collections_per_event=2 - await det.prepare(TriggerInfo(number_of_events=3, collections_per_event=2)) - assert dl.stop_count == 0 # No stop on first prepare + await det.prepare(TriggerInfo(collections_per_event=2)) + assert dl.stop_count == staged_stops # No stop on first prepare + status = det.trigger() + await wait_for_pending_wakeups(raise_if_exceeded=False) + await dl.collections_written.set(2) + await status - # Second prepare with different collections_per_event triggers recreation - await det.prepare(TriggerInfo(number_of_events=3, collections_per_event=3)) - assert dl.stop_count == 1 # stop() should have been called + await det.prepare(TriggerInfo(collections_per_event=second_collections_per_event)) + assert dl.stop_count == staged_stops + stops async def test_different_collections_written_raises(tmp_path): """Test that different collections_written values from providers raises error.""" - det = StandardDetector(name="det") dl1 = StreamableOnlyDataLogic(tmp_path) - dl2 = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(JustInternalTriggerLogic(), dl1, dl2) - - await det.prepare(TriggerInfo(number_of_events=5)) - - # Set different collection counts for each data logic + dl2 = StreamableOnlyDataLogic(tmp_path, datakey_suffix="-2") + det = DetectorLogic( + trigger=JustInternalTriggerLogic(), data=[dl1, dl2] + ).with_device("det") + await det.stage() + # Neither start_provider() resets its count await dl1.collections_written.set(3) await dl2.collections_written.set(5) - # Should raise RuntimeError when collect_asset_docs tries to validate with pytest.raises( - RuntimeError, + TypeError, match=re.escape( - "Detectors have written different numbers of collections: {3, 5}" + "Differing numbers of collections written: " + "StreamableOnlyDataLogic[0] has 3, StreamableOnlyDataLogic[1] has 5" ), ): - await det.kickoff() + await det.prepare(TriggerInfo(number_of_events=5)) async def test_data_logic_and_registered_signal(tmp_path): """A registered signal and a data logic both describe(), only one collects.""" - det = StandardDetector(name="det") + det = DetectorLogic( + trigger=JustInternalTriggerLogic(), data=[StreamableOnlyDataLogic(tmp_path)] + ).with_device("det") + await det.stage() det.counts, _ = soft_signal_r_and_setter(int, 42, name="det-counts") det.set_readable_format(det.counts, Format.HINTED_SIGNAL) - det.add_detector_logics( - JustInternalTriggerLogic(), StreamableOnlyDataLogic(tmp_path) - ) - await det.prepare(TriggerInfo()) + await det.prepare(TriggerInfo(number_of_events=3)) # Should have data from the data logic and the registered signal description = await det.describe() @@ -997,20 +1247,24 @@ async def test_data_logic_and_registered_signal(tmp_path): }, } # Should be able to do fly scanning (has streamable logic) - await det.prepare(TriggerInfo(number_of_events=3)) await det.kickoff() + await det.unstage() async def test_collect_asset_docs_with_explicit_index(tmp_path): """Test collect_asset_docs() with explicitly provided index.""" - det = StandardDetector(name="det") dl = StreamableOnlyDataLogic(tmp_path) - det.add_detector_logics(JustInternalTriggerLogic(), dl) + det = DetectorLogic(trigger=JustInternalTriggerLogic(), data=[dl]).with_device( + "det" + ) + await det.stage() await det.prepare(TriggerInfo(number_of_events=5, collections_per_event=2)) + await det.kickoff() # Collect with explicit index (not relying on get_index) docs = [doc async for doc in det.collect_asset_docs(index=3)] + await det.unstage() # Should emit docs for 3 events assert docs == [ @@ -1046,7 +1300,8 @@ async def test_child_readable_config_signals_in_describe_configuration(): config_sig = soft_signal_rw(float, initial_value=1.5, name="child-exposure") child.set_readable_format(config_sig, Format.CONFIG_SIGNAL) - det = StandardDetector(name="det") + det = DetectorLogic().with_device("det") + await det.stage() det.set_readable_format(child, Format.CHILD) await det.prepare(TriggerInfo()) @@ -1063,10 +1318,15 @@ async def test_child_readable_read_signals_in_read(tmp_path): read_sig = soft_signal_rw(int, initial_value=99, name="child-counts") child.set_readable_format(read_sig, Format.HINTED_SIGNAL) - det = StandardDetector(name="det") + dl = StreamableOnlyDataLogic(tmp_path) + det = DetectorLogic(data=[dl]).with_device("det") + await det.stage() det.set_readable_format(child, Format.CHILD) - det.add_detector_logics(StreamableOnlyDataLogic(tmp_path)) await det.prepare(TriggerInfo()) + status = det.trigger() + await wait_for_pending_wakeups(raise_if_exceeded=False) + await dl.collections_written.set(1) + await status # The data logic describes its stream, the child describes its signal desc = await det.describe() @@ -1084,9 +1344,9 @@ async def test_child_readable_hints_merged(tmp_path): hinted_sig = soft_signal_rw(float, name="child-intensity") child.set_readable_format(hinted_sig, Format.HINTED_SIGNAL) - det = StandardDetector(name="det") + det = DetectorLogic(data=[StreamableOnlyDataLogic(tmp_path)]).with_device("det") + await det.stage() det.set_readable_format(child, Format.CHILD) - det.add_detector_logics(StreamableOnlyDataLogic(tmp_path)) assert "fields" in det.hints assert "child-intensity" in det.hints["fields"] @@ -1115,19 +1375,32 @@ async def test_trigger_logic_not_implemented_errors(): async def test_data_logic_not_implemented_errors(): """Test NotImplementedError for unimplemented DetectorDataLogic methods.""" - logic = DetectorDataLogic() + class Bounded(BoundedDataLogic[DataProvider]): + async def start_provider(self, *args): + return await super().start_provider(*args) # type: ignore[safe-super] + + class Unbounded(UnboundedDataLogic[DataProvider]): + async def start_provider(self, *args): + return await super().start_provider(*args) # type: ignore[safe-super] + + with pytest.raises(NotImplementedError): + await Bounded().start_provider("test", [], 1, 1, 0, 0) with pytest.raises(NotImplementedError): - await logic.prepare_unbounded("test") + await Unbounded().start_provider("test", 1, 0, 0) + logic = Bounded() - # stop() should not raise (has default implementation) - await logic.stop() # Should pass + # These have default implementations, so should not raise + assert logic.get_hinted_fields("test") == [] + await logic.ensure_ready() + await logic.ensure_stopped() async def test_detector_readable_format_changes_at_runtime(): # A StandardDetector is a StandardReadable, so a plugin signal can be moved # between configuration and hinted reads without redefining the Device. - det = StandardDetector(name="det") + det = DetectorLogic().with_device("det") + await det.stage() det.temperature, _ = soft_signal_r_and_setter(float, 20.0, name="temperature") name = det.temperature.name @@ -1146,8 +1419,222 @@ async def test_detector_readable_format_changes_at_runtime(): async def test_detector_add_config_signals_is_deprecated(): - det = StandardDetector(name="det") + det = DetectorLogic().with_device("det") + await det.stage() signal, _ = soft_signal_r_and_setter(float, 1.0, name="sig") with pytest.deprecated_call(): det.add_config_signals(signal) assert set(await det.read_configuration()) == {signal.name} + + +async def test_streamable_logic_looks_like_writes_stream_assets(tmp_path): + """An unbounded logic exposes collect_asset_docs, and only that. + + The bluesky bundler picks up stream assets via a structural isinstance + against WritesStreamAssets (a hasattr check for collect_asset_docs). A + detector must look like exactly one of WritesStreamAssets / + EventPageCollectable so the bundler routes it down a single path. + """ + det = DetectorLogic(data=[StreamableOnlyDataLogic(tmp_path)]).with_device() + await det.stage() + + # Which verb applies follows what the logics will produce, so it is not + # known until they have been asked + assert not isinstance(det, WritesStreamAssets) + await det.prepare(TriggerInfo()) + + assert isinstance(det, WritesStreamAssets) + assert not isinstance(det, EventPageCollectable) + assert hasattr(det, "collect_asset_docs") + assert not hasattr(det, "collect_pages") + + +async def test_bounded_logic_looks_like_event_page_collectable(): + """A bounded logic exposes collect_pages, and only that.""" + det = DetectorLogic(data=[BoundedOnlyDataLogic()]).with_device() + await det.stage() + await det.prepare(TriggerInfo()) + + assert isinstance(det, EventPageCollectable) + assert not isinstance(det, WritesStreamAssets) + assert hasattr(det, "collect_pages") + assert not hasattr(det, "collect_asset_docs") + + +async def test_no_data_logic_looks_like_neither(): + """A detector with no data logic emits neither stream assets nor event pages.""" + det = DetectorLogic(trigger=JustInternalTriggerLogic()).with_device() + await det.stage() + + assert not isinstance(det, WritesStreamAssets) + assert not isinstance(det, EventPageCollectable) + assert not hasattr(det, "collect_asset_docs") + assert not hasattr(det, "collect_pages") + + +async def test_missing_attribute_raises_standard_attribute_error(): + """__getattr__ falls through to a normal AttributeError for other names.""" + det = DetectorLogic().with_device() + await det.stage() + with pytest.raises( + AttributeError, + match=r"object has no attribute 'does_not_exist'", + ): + det.does_not_exist # noqa: B018 + + +async def test_unsuppressed_bounded_keys_raise(tmp_path): + """A finite buffer the stream assets do not cover cannot be combined with them.""" + det = DetectorLogic( + data=[ + BoundedOnlyDataLogic(datakey_suffix="-stats"), + StreamableOnlyDataLogic(tmp_path), + ] + ).with_device("det") + await det.stage() + + # Only knowable once the logics have said what they would make + with pytest.raises( + TypeError, + match=( + r"would produce \['det-stats'\] as event pages and the rest of its " + r"data as stream assets" + ), + ): + await det.prepare(TriggerInfo()) + + +async def test_suppressed_bounded_logic_sits_the_scan_out(tmp_path): + """A finite buffer whose keys are all written durably gives way to the file.""" + bounded = BoundedOnlyDataLogic() + # The same quantity as the buffer: one integer per collection + streamable = StreamableOnlyDataLogic(tmp_path, shape=(), dtype_numpy=" None: + self.hidden = hidden + + async def hidden_datakeys(self) -> set[str]: + return set(self.hidden) + + +async def test_suppress_logic_hides_readables_until_unstage(): + hidden = soft_signal_rw(int, 1, name="det-hidden") + also_hidden = soft_signal_rw(int, 2, name="det-also-hidden") + kept = soft_signal_rw(int, 3, name="det-kept") + suppress = SetSuppressLogic({"det-hidden", "det-also-hidden"}) + det = DetectorLogic(suppress=suppress).with_device("det") + for signal in (hidden, also_hidden, kept): + det.set_readable_format(signal, Format.HINTED_SIGNAL) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + + description = await det.describe() + + assert set(description) == {"det-kept"} + + # The same result at a later prepare is no change + await det.prepare(TriggerInfo()) + await det.trigger() + + # A different result is, until the next stage takes it + suppress.hidden = {"det-hidden"} + with pytest.raises( + RuntimeError, + match=re.escape( + "det: hidden datakeys changed since the first prepare() after " + "stage(): ['det-also-hidden']. unstage() and stage() to use the new values." + ), + ): + await det.prepare(TriggerInfo()) + await det.unstage() + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + assert set(await det.describe()) == {"det-also-hidden", "det-kept"} + + +class _HangingLogic: + """A data logic whose start_provider never returns, recording if it runs.""" + + running = False + + async def start_provider(self, *args): + self.running = True + try: + await asyncio.Event().wait() + finally: + self.running = False + + +class _FailingLogic: + async def start_provider(self, *args): + raise ValueError("path does not exist") + + +class HangingUnboundedLogic(_HangingLogic, UnboundedDataLogic): ... + + +class FailingUnboundedLogic(_FailingLogic, UnboundedDataLogic): ... + + +class HangingBoundedLogic(_HangingLogic, BoundedDataLogic): ... + + +class FailingBoundedLogic(_FailingLogic, BoundedDataLogic): ... + + +@pytest.mark.parametrize( + "hanging,failing", + [ + (HangingUnboundedLogic, FailingUnboundedLogic), + (HangingBoundedLogic, FailingBoundedLogic), + ], + ids=["unbounded", "bounded"], +) +async def test_failed_start_provider_stops_the_others_starting(hanging, failing): + """Nothing a failed prepare() started is still running after it raises.""" + hanging_logic = hanging() + det = DetectorLogic(data=[hanging_logic, failing()]).with_device("det") + await det.stage() + + with pytest.raises(ValueError, match="path does not exist"): + await det.prepare(TriggerInfo()) + + assert not hanging_logic.running + await det.unstage() + + +async def test_hidden_datakeys_are_asked_before_the_trigger_logic_is_prepared(): + calls: list[str] = [] + + class RecordingTriggerLogic(JustInternalTriggerLogic): + async def prepare_internal(self, num: int, livetime: float, deadtime: float): + calls.append("prepare_internal") + + class RecordingSuppressLogic(DetectorSuppressLogic): + async def hidden_datakeys(self) -> set[str]: + calls.append("hidden_datakeys") + return set() + + det = DetectorLogic( + trigger=RecordingTriggerLogic(), suppress=RecordingSuppressLogic() + ).with_device("det") + await det.stage() + + await det.prepare(TriggerInfo()) + + assert calls == ["hidden_datakeys", "prepare_internal"] diff --git a/tests/unit_tests/core/test_trigger_info.py b/tests/unit_tests/core/test_trigger_info.py index 0fc3850171..56b85021f0 100644 --- a/tests/unit_tests/core/test_trigger_info.py +++ b/tests/unit_tests/core/test_trigger_info.py @@ -61,8 +61,8 @@ { "number_of_events": -100, }, - "number_of_events\n Input should be greater than or " - "equal to 0 [type=greater_than_equal, input_value=-100, input_type=int]", + "number_of_events\n Input should be greater than 0 " + "[type=greater_than, input_value=-100, input_type=int]", ), ], ) diff --git a/tests/unit_tests/epics/adcore/test_acquire_logic.py b/tests/unit_tests/epics/adcore/test_acquire_logic.py index 33987751a8..f628748f2a 100644 --- a/tests/unit_tests/epics/adcore/test_acquire_logic.py +++ b/tests/unit_tests/epics/adcore/test_acquire_logic.py @@ -6,6 +6,7 @@ from ophyd_async.core import ( callback_on_mock_put, + get_mock, init_devices, set_callback_filter, set_mock_value, @@ -18,14 +19,18 @@ async def adbase_detector() -> adcore.AreaDetector[adcore.ADBaseIO]: driver = adcore.ADBaseIO("PREFIX:DRV:") async with init_devices(mock=True): - det = adcore.AreaDetector(driver=driver) - det.add_detector_logics(adcore.ADAcquireLogic(driver)) + det = adcore.AreaDetector( + driver=driver, acquire_logic=adcore.ADAcquireLogic(driver) + ) return det async def test_acquire_logic_trigger_internal_calls_acquire( adbase_detector: adcore.AreaDetector[adcore.ADBaseIO], ): + await adbase_detector.stage() + # Only the calls after stage() are of interest + get_mock(adbase_detector).reset_mock() await adbase_detector.trigger() assert_has_calls( adbase_detector.driver, @@ -38,14 +43,16 @@ async def test_acquire_logic_trigger_internal_calls_acquire( async def test_acquire_logic_when_arming_times_out(): driver = adcore.ADBaseIO("PREFIX:DRV:") async with init_devices(mock=True): - det = adcore.AreaDetector(driver=driver) - det.add_detector_logics(adcore.ADAcquireLogic(driver, timeout=0.02)) + det = adcore.AreaDetector( + driver=driver, acquire_logic=adcore.ADAcquireLogic(driver, timeout=0.02) + ) async def sleep_for_a_bit(value): await asyncio.sleep(0.02) callback_on_mock_put(det.driver.acquire, sleep_for_a_bit) + await det.stage() with pytest.raises( TimeoutError, match=re.escape( @@ -60,11 +67,13 @@ async def sleep_for_a_bit(value): async def test_acquire_logic_wait_for_idle_in_bad_state(): driver = adcore.ADBaseIO("PREFIX:DRV:") async with init_devices(mock=True): - det = adcore.AreaDetector(driver=driver) - det.add_detector_logics(adcore.ADAcquireLogic(driver, timeout=0.05)) + det = adcore.AreaDetector( + driver=driver, acquire_logic=adcore.ADAcquireLogic(driver, timeout=0.05) + ) set_mock_value(driver.detector_state, adcore.ADState.ERROR) + await det.stage() with pytest.raises(ValueError) as exc_info: await det.trigger() @@ -74,7 +83,7 @@ async def test_acquire_logic_wait_for_idle_in_bad_state(): assert "ADState.ABORTED" in error_msg -async def test_start_acquiring_driver_and_ensure_status_timing( +async def test_acquire_driver_and_ensure_status_timing( adbase_detector: adcore.AreaDetector[adcore.ADBaseIO], ): """This test ensures the camera has time to return to a good state. @@ -97,12 +106,15 @@ async def complete_acquire() -> None: adcore.ADState.IDLE, ) + await adbase_detector.stage() await asyncio.gather(adbase_detector.trigger(), complete_acquire()) async def test_acquire_logic_disarm( adbase_detector: adcore.AreaDetector[adcore.ADBaseIO], ): + await adbase_detector.stage() + get_mock(adbase_detector).reset_mock() await adbase_detector.unstage() assert_has_calls( adbase_detector.driver, @@ -112,18 +124,20 @@ async def test_acquire_logic_disarm( ) -async def test_start_acquiring_driver_and_ensure_status_disconnected(): +async def test_acquire_driver_and_ensure_status_disconnected(): """This test ensures the function behaves gracefully if no detector states are available. """ driver = adcore.ADBaseIO("PREFIX:DRV:") async with init_devices(mock=True): - det = adcore.AreaDetector(driver=driver) - det.add_detector_logics(adcore.ADAcquireLogic(driver, timeout=0.1)) + det = adcore.AreaDetector( + driver=driver, acquire_logic=adcore.ADAcquireLogic(driver, timeout=0.1) + ) set_callback_filter(driver.detector_state, lambda v: None) + await det.stage() with pytest.raises(asyncio.TimeoutError) as exc: await det.trigger() assert ( diff --git a/tests/unit_tests/epics/adcore/test_cont_acq_detector.py b/tests/unit_tests/epics/adcore/test_cont_acq_detector.py index 1657152176..23e06f0b70 100644 --- a/tests/unit_tests/epics/adcore/test_cont_acq_detector.py +++ b/tests/unit_tests/epics/adcore/test_cont_acq_detector.py @@ -39,10 +39,11 @@ async def test_cont_acq_controller_invalid_trigger_mode( cont_acq_detector: adcore.AreaDetector[adcore.ADBaseIO], ): trigger_info = TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE) + await cont_acq_detector.stage() with pytest.raises( ValueError, match=re.escape( - "Trigger type DetectorTrigger.EXTERNAL_EDGE not supported by 'det', " + "Trigger type DetectorTrigger.EXTERNAL_EDGE not supported, " "supported types are: [INTERNAL]" ), ): @@ -58,6 +59,7 @@ async def test_cont_acq_controller_not_in_continuous_mode( cont_acq_detector: adcore.AreaDetector[adcore.ADBaseIO], ): set_mock_value(cont_acq_detector.driver.image_mode, adcore.ADImageMode.SINGLE) + await cont_acq_detector.stage() with pytest.raises(RuntimeError, match=ERROR_MESSAGE): await cont_acq_detector.trigger() @@ -66,6 +68,7 @@ async def test_cont_acq_controller_not_acquiring( cont_acq_detector: adcore.AreaDetector[adcore.ADBaseIO], ): set_mock_value(cont_acq_detector.driver.acquire, False) + await cont_acq_detector.stage() with pytest.raises(RuntimeError, match=ERROR_MESSAGE): await cont_acq_detector.trigger() @@ -73,6 +76,7 @@ async def test_cont_acq_controller_not_acquiring( async def test_cont_acq_controller_invalid_exposure_time( cont_acq_detector: adcore.AreaDetector[adcore.ADBaseIO], ): + await cont_acq_detector.stage() with pytest.raises( ValueError, match=re.escape( @@ -126,7 +130,7 @@ async def test_cont_acq_skips_dirty_frames( async with init_devices(mock=True): det = adcore.ContAcqDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), proc_suffix=proc_suffix, num_frames_to_skip=num_frames_to_skip, ) @@ -288,3 +292,75 @@ async def test_cont_acq_requires_cb_fed_by_driver( match="CB plugin NDArrayPort is 'PROC1', but must be the driver port 'DRV1'", ): await cont_acq_detector.stage() + + +@pytest.fixture +async def cont_acq_hdf_detector( + static_path_provider: StaticPathProvider, +) -> adcore.AreaDetector[adcore.ADBaseIO]: + async with init_devices(mock=True): + det = adcore.ContAcqDetector( + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) + ) + set_mock_value(det.driver.image_mode, adcore.ADImageMode.CONTINUOUS) + set_mock_value(det.driver.acquire_time, 0.8) + set_mock_value(det.driver.acquire_period, 1.0) + set_mock_value(det.driver.acquire, True) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + return det + + +async def test_uc9_continuous_acquisition_step_then_fly_scan( + cont_acq_hdf_detector: adcore.AreaDetector[adcore.ADBaseIO], +): + """Use case 9: the driver stays acquiring, the buffer captures N per trigger. + + Streamable and bounded: stream assets for step and fly, the buffer sized by + the prepare that precedes each. + """ + det = cont_acq_hdf_detector + cb = det.get_plugin("cb", adcore.NDCircularBuffIO) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + captured = 0 + + frames_per_capture = 0 + + def soft_trigger(value: bool) -> bool | None: + # Each soft trigger of the circular buffer writes this many frames, + # then the busy record resets + nonlocal captured + if value: + captured += frames_per_capture + set_mock_value(hdf.num_captured, captured) + return False + return None + + callback_on_mock_put(cb.trigger_, soft_trigger) + + # Step scan: two points of 2 frames each, one StreamResource + await det.stage() + frames_per_capture = 2 + await det.prepare(TriggerInfo(collections_per_event=2)) + await det.trigger() + await det.trigger() + assert await cb.post_count.get_value() == 2 + docs = [doc async for doc in det.collect_asset_docs()] # pyright: ignore[reportAttributeAccessIssue] + assert [name for name, _ in docs] == [ + "stream_resource", + "stream_datum", + ] + assert docs[1][1]["indices"] == {"start": 0, "stop": 2} + + # Fly scan: 3 events of 2 frames, captured in one go + frames_per_capture = 6 + await det.prepare(TriggerInfo(number_of_events=3, collections_per_event=2)) + assert await cb.post_count.get_value() == 6 + await det.kickoff() + await det.complete() + docs = [doc async for doc in det.collect_asset_docs()] # pyright: ignore[reportAttributeAccessIssue] + assert [d["indices"] for n, d in docs if n == "stream_datum"] == [ + {"start": 2, "stop": 5} + ] + await det.unstage() diff --git a/tests/unit_tests/epics/adcore/test_data_logic.py b/tests/unit_tests/epics/adcore/test_data_logic.py index 9e940744b3..6224c9a0c2 100644 --- a/tests/unit_tests/epics/adcore/test_data_logic.py +++ b/tests/unit_tests/epics/adcore/test_data_logic.py @@ -1,3 +1,4 @@ +import asyncio import os from pathlib import Path, PurePath, PurePosixPath, PureWindowsPath from unittest.mock import ANY, call @@ -10,10 +11,16 @@ StaticFilenameProvider, StaticPathProvider, TriggerInfo, + get_mock, init_devices, set_mock_value, ) from ophyd_async.epics import adcore, adsimdetector +from ophyd_async.epics.adcore import ( + ADHDFDataLogic, + ADMultipartDataLogic, + NDArrayDescription, +) from ophyd_async.testing import assert_has_calls @@ -24,7 +31,7 @@ async def hdf_det( async with init_devices(mock=True): detector = adsimdetector.SimDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), plugins={"stats": adcore.NDStatsIO("PREFIX:STATS:")}, ) set_mock_value(detector.driver.array_size_x, 1024) @@ -33,7 +40,103 @@ async def hdf_det( return detector +async def test_start_provider_opens_the_file_and_starts_capture( + hdf_det: adcore.AreaDetector[adcore.ADBaseIO], + static_path_provider: StaticPathProvider, +): + writer = hdf_det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(writer.file_path_exists, True) + logic = ADHDFDataLogic( + array_description=NDArrayDescription( + shape_signals=[hdf_det.driver.array_size_y, hdf_det.driver.array_size_x], + data_type_signal=hdf_det.driver.data_type, + color_mode_signal=hdf_det.driver.color_mode, + ), + path_provider=static_path_provider, + driver=hdf_det.driver, + writer=writer, + ) + + assert await writer.capture.get_value() is False + + made = await logic.start_provider("det", 1, 0.1, 0) + assert made is not None + assert await writer.capture.get_value() is True + assert await writer.file_name.get_value() != "" + + +@pytest.mark.parametrize( + "always_enable,plugin_enabled,makes_provider", + [ + # The default serves, whatever the plugin was set to + (True, EnableDisable.DISABLE, True), + # Following the plugin, a disabled one writes nothing... + (False, EnableDisable.DISABLE, False), + # ...and an enabled one still writes + (False, EnableDisable.ENABLE, True), + ], +) +async def test_hdf_follows_the_plugin_when_not_enabling_it( + hdf_det: adcore.AreaDetector[adcore.ADBaseIO], + static_path_provider: StaticPathProvider, + always_enable: bool, + plugin_enabled: EnableDisable, + makes_provider: bool, +): + writer = hdf_det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(writer.file_path_exists, True) + set_mock_value(writer.enable_callbacks, plugin_enabled) + logic = ADHDFDataLogic( + array_description=NDArrayDescription( + shape_signals=[hdf_det.driver.array_size_y, hdf_det.driver.array_size_x], + data_type_signal=hdf_det.driver.data_type, + color_mode_signal=hdf_det.driver.color_mode, + ), + path_provider=static_path_provider, + driver=hdf_det.driver, + writer=writer, + always_enable=always_enable, + ) + + made = await logic.start_provider("det", 1, 0.1, 0) + assert (made is not None) is makes_provider + # Starting switches the plugin on either way, and sitting out writes nothing + assert await writer.capture.get_value() is makes_provider + assert await writer.enable_callbacks.get_value() is ( + EnableDisable.ENABLE if makes_provider else EnableDisable.DISABLE + ) + + +@pytest.mark.parametrize("kind", ["hdf", "jpeg", "tiff"]) +@pytest.mark.parametrize( + "plugin_enabled,makes_provider", + [ + (EnableDisable.DISABLE, False), + (EnableDisable.ENABLE, True), + ], +) +async def test_a_writer_follows_its_plugin_by_default( + hdf_det: adcore.AreaDetector[adcore.ADBaseIO], + static_path_provider: StaticPathProvider, + kind: str, + plugin_enabled: EnableDisable, + makes_provider: bool, +): + factory = getattr(adcore.ADDataFactory, kind)(static_path_provider) + writer, logic = factory("PREFIX:", hdf_det.driver, []) + async with init_devices(mock=True): + writer.set_name(kind) + await writer.connect(mock=True) + set_mock_value(writer.file_path_exists, True) + set_mock_value(writer.enable_callbacks, plugin_enabled) + + made = await logic.start_provider("det", 1, 0.1, 0) + + assert (made is not None) is makes_provider + + async def test_hdf_writer_file_not_found(hdf_det: adcore.AreaDetector[adcore.ADBaseIO]): + await hdf_det.stage() with pytest.raises( FileNotFoundError, match=r"Path .* doesn't exist or not writable!" ): @@ -46,7 +149,7 @@ async def test_hdf_writer_passes_parent_name_to_path_provider(tmp_path: Path): ) async with init_devices(mock=True): det = adsimdetector.SimDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(pp), name="sim_detector" + "PREFIX:", adcore.ADDataFactory.hdf(pp), name="sim_detector" ) writer = det.get_plugin("hdf", adcore.NDPluginFileIO) @@ -65,6 +168,9 @@ async def test_prepare_hdf( ): writer = hdf_det.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) + await hdf_det.stage() + # Only the calls after stage() are of interest + get_mock(hdf_det).reset_mock() await hdf_det.prepare(TriggerInfo(number_of_events=3)) assert_has_calls( hdf_det, @@ -95,8 +201,8 @@ async def test_prepare_hdf( @pytest.mark.parametrize( "factory_cls,extension", [ - (adcore.ADWriterFactory.jpeg, ".jpg"), - (adcore.ADWriterFactory.tiff, ".tiff"), + (adcore.ADDataFactory.jpeg, ".jpg"), + (adcore.ADDataFactory.tiff, ".tiff"), ], ) async def test_prepare_multipart( @@ -107,11 +213,15 @@ async def test_prepare_multipart( name = factory_cls.__name__ writer = det.get_plugin(name, adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) + await det.stage() + # Only the calls after stage() are of interest + get_mock(det).reset_mock() await det.prepare(TriggerInfo(number_of_events=3)) assert_has_calls( writer, [ call.lazy_open.put(True), + call.enable_callbacks.put(EnableDisable.ENABLE), call.create_directory.put(0), call.file_path.put(f"{static_path_provider().directory_path}{os.sep}"), call.file_name.put("ophyd_async_tests"), @@ -125,12 +235,67 @@ async def test_prepare_multipart( ) +@pytest.mark.parametrize( + "flush_period,livetime,deadtime,exposures_per_collection,expected_frames_per_chunk", + [ + # 0.5s flush / 0.01s period -> 50 frames per chunk (#1309) + (0.5, 0.01, 0.0, 1, 50), + # 0.5s flush / 0.05s period -> 10 frames per chunk + (0.5, 0.04, 0.01, 1, 10), + # A sub-frame flush period floors to a single frame per chunk + (0.001, 0.05, 0.0, 1, 1), + # 5 exposures of 0.01s make each frame written, so 0.5s holds 10 + (0.5, 0.01, 0.0, 5, 10), + ], +) +async def test_hdf_chunk_sized_from_flush_period( + static_path_provider: StaticPathProvider, + flush_period: float, + livetime: float, + deadtime: float, + exposures_per_collection: int, + expected_frames_per_chunk: int, +): + """A TriggerInfo flush_period sizes the HDF chunk from the collection period.""" + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.hdf(static_path_provider), + # Averages the exposures of each collection into one frame + proc_suffix="PROC:", + ) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + set_mock_value(det.driver.data_type, adcore.ADBaseDataType.UINT16) + writer = det.get_plugin("hdf", adcore.NDPluginFileIO) + set_mock_value(writer.file_path_exists, True) + await det.stage() + await det.prepare( + TriggerInfo( + livetime=livetime, + deadtime=deadtime, + exposures_per_collection=exposures_per_collection, + number_of_events=3, + flush_period=flush_period, + ) + ) + assert await writer.num_frames_chunks.get_value() == expected_frames_per_chunk + await det.kickoff() + (sr, *_) = [doc async for doc in det.collect_asset_docs(3)] + await det.unstage() + assert sr[1]["parameters"]["chunk_shape"] == ( + expected_frames_per_chunk, + 768, + 1024, + ) + + @pytest.mark.parametrize( "factory_cls,is_hdf", [ - (adcore.ADWriterFactory.hdf, True), - (adcore.ADWriterFactory.jpeg, False), - (adcore.ADWriterFactory.tiff, False), + (adcore.ADDataFactory.hdf, True), + (adcore.ADDataFactory.jpeg, False), + (adcore.ADDataFactory.tiff, False), ], ) async def test_can_specify_different_uri_and_path( @@ -151,7 +316,9 @@ async def test_can_specify_different_uri_and_path( set_mock_value(writer.file_path_exists, True) await det.stage() await det.prepare(TriggerInfo()) + await det.kickoff() docs = [doc async for doc in det.collect_asset_docs(1)] + await det.unstage() # Make sure we set the write path to the directory_path attr from our path info assert await writer.file_path.get_value() == str(tmp_path) + os.sep @@ -185,9 +352,9 @@ async def test_can_specify_different_uri_and_path( @pytest.mark.parametrize( "factory_cls,is_hdf", [ - (adcore.ADWriterFactory.hdf, True), - (adcore.ADWriterFactory.jpeg, False), - (adcore.ADWriterFactory.tiff, False), + (adcore.ADDataFactory.hdf, True), + (adcore.ADDataFactory.jpeg, False), + (adcore.ADDataFactory.tiff, False), ], ) async def test_can_override_uri_with_different_path_semantics( @@ -208,7 +375,9 @@ async def test_can_override_uri_with_different_path_semantics( set_mock_value(writer.file_path_exists, True) await det.stage() await det.prepare(TriggerInfo()) + await det.kickoff() docs = [doc async for doc in det.collect_asset_docs(1)] + await det.unstage() assert await writer.file_path.get_value() == f"{write_path}{expected_separator}" assert len(docs) == 2 @@ -241,6 +410,7 @@ async def test_stats_describe_raises_error_with_dbr_native( """, ) + await hdf_det.stage() with pytest.raises( RuntimeError, match="NDAttribute mydetector-Temperature has dbrtype DBR_NATIVE," @@ -270,6 +440,7 @@ async def test_describe_different_color_modes( writer = hdf_det.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) set_mock_value(hdf_det.driver.color_mode, color_mode) + await hdf_det.stage() if shape is RuntimeError: # Expected to fail for this mode with pytest.raises( @@ -295,6 +466,7 @@ async def test_3d_dataset_shape(hdf_det: adcore.AreaDetector[adcore.ADBaseIO]): writer = hdf_det.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) set_mock_value(hdf_det.driver.array_size_z, 10) + await hdf_det.stage() await hdf_det.prepare(TriggerInfo()) describe = await hdf_det.describe() assert describe["detector"] == { @@ -304,3 +476,56 @@ async def test_3d_dataset_shape(hdf_det: adcore.AreaDetector[adcore.ADBaseIO]): "shape": [1, 10, 768, 1024], "source": ANY, } + + +async def _start_waits_for_a_zero_counter(logic, counter, previous: int = 5) -> None: + set_mock_value(counter, previous) + starting = asyncio.create_task(logic.start_provider("det", 1, 0.1, 0)) + await asyncio.sleep(0.1) + # The IOC has not yet reset NumCaptured, so the detector's baseline would + # be the previous scan's count + assert not starting.done() + set_mock_value(counter, 0) + assert await asyncio.wait_for(starting, timeout=1) is not None + + +async def test_hdf_start_waits_for_num_captured_to_read_zero( + hdf_det: adcore.AreaDetector[adcore.ADBaseIO], + static_path_provider: StaticPathProvider, +): + writer = hdf_det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(writer.file_path_exists, True) + logic = ADHDFDataLogic( + array_description=NDArrayDescription( + shape_signals=[hdf_det.driver.array_size_y, hdf_det.driver.array_size_x], + data_type_signal=hdf_det.driver.data_type, + color_mode_signal=hdf_det.driver.color_mode, + ), + path_provider=static_path_provider, + driver=hdf_det.driver, + writer=writer, + ) + await _start_waits_for_a_zero_counter(logic, writer.num_captured) + + +async def test_multipart_start_waits_for_num_captured_to_read_zero( + static_path_provider: StaticPathProvider, +): + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", adcore.ADDataFactory.tiff(static_path_provider) + ) + writer = det.get_plugin("tiff", adcore.NDPluginFileIO) + set_mock_value(writer.file_path_exists, True) + logic = ADMultipartDataLogic( + array_description=NDArrayDescription( + shape_signals=[det.driver.array_size_y, det.driver.array_size_x], + data_type_signal=det.driver.data_type, + color_mode_signal=det.driver.color_mode, + ), + path_provider=static_path_provider, + writer=writer, + extension=".tiff", + mimetype="multipart/related;type=image/tiff", + ) + await _start_waits_for_a_zero_counter(logic, writer.num_captured) diff --git a/tests/unit_tests/epics/adcore/test_detector.py b/tests/unit_tests/epics/adcore/test_detector.py index f08d0fa592..d3eca62921 100644 --- a/tests/unit_tests/epics/adcore/test_detector.py +++ b/tests/unit_tests/epics/adcore/test_detector.py @@ -13,9 +13,9 @@ @pytest.mark.parametrize( "factory", [ - adcore.ADWriterFactory.hdf, - adcore.ADWriterFactory.jpeg, - adcore.ADWriterFactory.tiff, + adcore.ADDataFactory.hdf, + adcore.ADDataFactory.jpeg, + adcore.ADDataFactory.tiff, ], ) @pytest.mark.parametrize("hinted", [True, False]) @@ -41,9 +41,9 @@ def test_writer_image_hints(static_path_provider: StaticPathProvider, factory, h @pytest.mark.parametrize( "factory", [ - adcore.ADWriterFactory.hdf, - adcore.ADWriterFactory.jpeg, - adcore.ADWriterFactory.tiff, + adcore.ADDataFactory.hdf, + adcore.ADDataFactory.jpeg, + adcore.ADDataFactory.tiff, ], ) def test_writer_image_hints_enabled_by_default( @@ -74,7 +74,7 @@ def test_area_detector_requires_prefix_when_factories_given( adcore.AreaDetector( driver, None, - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), name="det", ) @@ -91,12 +91,44 @@ def test_area_detector_rejects_duplicate_writer_names( adcore.AreaDetector( driver, "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider, writer_name="hdf"), - adcore.ADWriterFactory.hdf(static_path_provider, writer_name="hdf"), + adcore.ADDataFactory.hdf(static_path_provider, writer_name="hdf"), + adcore.ADDataFactory.hdf(static_path_provider, writer_name="hdf"), name="det", ) +def test_plugin_named_like_a_factory_attribute_raises( + static_path_provider: StaticPathProvider, +): + driver = adcore.ADBaseIO("PREFIX:DRV:") + + with pytest.raises(ValueError, match=r"attribute 'stats'"): + adcore.AreaDetector( + driver, + "PREFIX:", + adcore.ADDataFactory.hdf(static_path_provider, writer_name="stats"), + plugins={"stats": adcore.NDStatsIO("PREFIX:STATS:")}, + name="det", + ) + + +def test_plugin_named_like_the_driver_raises(): + driver = adcore.ADBaseIO("PREFIX:DRV:") + + with pytest.raises(ValueError, match=r"attribute 'driver'"): + adcore.AreaDetector( + driver, plugins={"driver": adcore.NDStatsIO("PREFIX:S:")}, name="det" + ) + + +def test_deprecated_writer_factory_still_builds_the_new_factories( + static_path_provider: StaticPathProvider, +): + with pytest.warns(DeprecationWarning, match="ADWriterFactory"): + hdf = adcore.ADWriterFactory.hdf(static_path_provider) + assert isinstance(hdf, adcore.ADDataFactory) + + # --------------------------------------------------------------------------- # AreaDetector.get_plugin guards # --------------------------------------------------------------------------- @@ -151,14 +183,15 @@ async def test_get_ndarray_resource_info_undefined_datatype( det = adcore.AreaDetector( adcore.ADBaseIO("PREFIX:DRV:"), "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), name="det", ) set_mock_value(det.driver.data_type, adcore.ADBaseDataType.UNDEFINED) set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + await det.stage() with pytest.raises( ValueError, - match=r"^mock\+ca://PREFIX:DRV:DataType_RBV is blank, this is not supported$", + match=r"^mock\+ca://PREFIX:DRV:DataType_RBV is blank, this is not supported", ): await det.prepare(TriggerInfo()) @@ -170,14 +203,15 @@ async def test_get_ndarray_resource_info_unsupported_color_mode( det = adcore.AreaDetector( adcore.ADBaseIO("PREFIX:DRV:"), "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), name="det", ) set_mock_value(det.driver.color_mode, adcore.ADBaseColorMode.BAYER) set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + await det.stage() with pytest.raises( RuntimeError, - match=r"^Unsupported ColorMode Bayer! Only Mono and RGB1 are supported\.$", + match=r"^Unsupported ColorMode Bayer! Only Mono and RGB1 are supported\.", ): await det.prepare(TriggerInfo()) @@ -188,3 +222,15 @@ def test_plugins_proc_is_reserved_for_proc_suffix(proc_suffix: str | None): adcore.plugins_with_proc( {"proc": adcore.NDProcessIO("PREFIX:PROC:")}, "PREFIX:", proc_suffix ) + + +@pytest.mark.parametrize("cb_plugin_name", ["cb", "buffer"]) +def test_cont_acq_detector_plugins_cannot_reuse_the_cb_plugin_name(cb_plugin_name): + with pytest.raises( + ValueError, match=r"Pass cb_plugin_name to name the circular buffer plugin" + ): + adcore.ContAcqDetector( + "PREFIX:", + cb_plugin_name=cb_plugin_name, + plugins={cb_plugin_name: adcore.NDStatsIO("PREFIX:STAT:")}, + ) diff --git a/tests/unit_tests/epics/adcore/test_use_cases.py b/tests/unit_tests/epics/adcore/test_use_cases.py index 648947b478..dd664fb4c9 100644 --- a/tests/unit_tests/epics/adcore/test_use_cases.py +++ b/tests/unit_tests/epics/adcore/test_use_cases.py @@ -2,14 +2,16 @@ import pytest -from ophyd_async.core import StandardReadableFormat as Format from ophyd_async.core import ( + EnableDisable, StaticPathProvider, + TransitionError, TriggerInfo, callback_on_mock_put, init_devices, set_mock_value, ) +from ophyd_async.core import StandardReadableFormat as Format from ophyd_async.epics import adaravis, adcore, adsimdetector from ophyd_async.epics.core import epics_signal_r from ophyd_async.testing import assert_reading @@ -22,7 +24,7 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( async with init_devices(mock=True): det = adsimdetector.SimDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), plugins={"stats": stat}, ) temp = epics_signal_r(float, "SAMP:TEMP:RBV") @@ -71,8 +73,7 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( assert {"det-driver-acquire_period", "det-driver-acquire_time"} <= config.keys() assert config["det-driver-acquire_period"]["value"] == 0.1 assert config["det-driver-acquire_time"]["value"] == 0.05 - # Check we need to call prepare or trigger before we can describe - with pytest.raises(RuntimeError, match="Prepare not run"): + with pytest.raises(TransitionError, match="describe not allowed from STAGED"): await det.describe() # Need to tell it the path is valid set_mock_value(writer.file_path_exists, True) @@ -188,14 +189,18 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( "det-stats-max_value", } <= readings.keys() docs = [doc async for doc in det.collect_asset_docs()] - assert docs == [ + # Each prepare starts new resources in the same file, the indices carrying on + new_sr, sd = docs[:3], docs[3:] + assert new_sr == [(name, {**doc, "uid": ANY}) for name, doc in sr] + assert {doc["uid"] for _, doc in new_sr}.isdisjoint(doc["uid"] for _, doc in sr) + assert sd == [ ( "stream_datum", { "descriptor": "", "indices": {"start": 1, "stop": 2}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": sr[i][1]["uid"], + "stream_resource": new_sr[i][1]["uid"], "uid": ANY, }, ) @@ -213,7 +218,7 @@ async def test_step_scan_tiff_detector( ): async with init_devices(mock=True): det = adsimdetector.SimDetector( - "PREFIX:", adcore.ADWriterFactory.tiff(static_path_provider) + "PREFIX:", adcore.ADDataFactory.tiff(static_path_provider) ) writer = det.get_plugin("tiff", adcore.NDPluginFileIO) @@ -279,7 +284,7 @@ async def test_step_scan_tiff_detector( async def test_flyscan_aravis_detector(static_path_provider: StaticPathProvider): async with init_devices(mock=True): det = adaravis.AravisDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = det.get_plugin("hdf", adcore.NDPluginFileIO) @@ -341,6 +346,7 @@ async def test_flyscan_aravis_detector(static_path_provider: StaticPathProvider) # Make 2 and 1/3 events and check they are produced set_mock_value(writer.num_captured, 7) docs = [doc async for doc in det.collect_asset_docs()] + sr = docs[:1] assert docs == [ ( "stream_resource", @@ -386,24 +392,27 @@ async def test_flyscan_aravis_detector(static_path_provider: StaticPathProvider) # Check that another kickoff without prepare is not allowed with pytest.raises( RuntimeError, - match="Kickoff requested 15:30, but detector was only prepared up to 15", + match="cannot go to FLYING from FLOWN", ): await det.kickoff() # But preparing again is ok await det.prepare(TriggerInfo(collections_per_event=3, number_of_events=3)) await det.kickoff() - # Completing adds indexes to the same resource + # Completing starts a new resource in the same file, the indices carrying on status = det.complete() set_mock_value(writer.num_captured, 24) docs = [doc async for doc in det.collect_asset_docs()] - assert docs == [ + new_sr = docs[:1] + assert new_sr == [(name, {**doc, "uid": ANY}) for name, doc in sr] + assert new_sr[0][1]["uid"] != sr[0][1]["uid"] + assert docs[1:] == [ ( "stream_datum", { "descriptor": "", "indices": {"start": 5, "stop": 8}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": ANY, + "stream_resource": new_sr[0][1]["uid"], "uid": ANY, }, ), @@ -420,7 +429,7 @@ async def test_2_rois_with_hdf(tmp_path): async with init_devices(mock=True): det = adaravis.AravisDetector( "PREFIX:", - adcore.ADWriterFactory.hdf( + adcore.ADDataFactory.hdf( path_provider, writer_suffix="HDF1:", writer_name="hdf1", @@ -431,7 +440,7 @@ async def test_2_rois_with_hdf(tmp_path): color_mode_signal=driver.color_mode, ), ), - adcore.ADWriterFactory.hdf( + adcore.ADDataFactory.hdf( path_provider, writer_suffix="HDF2:", writer_name="hdf2", @@ -590,11 +599,11 @@ async def test_step_scan_keep_numimages( async with init_devices(mock=True): det = adsimdetector.SimDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), plugins={"stats": stat}, ) - # Plugins are not registered automatically; opt this one in - det.set_readable_format(stat, Format.CHILD) + # Plugins are not registered automatically, and the stats readables give one + # value per event so cannot be used with 42 collections per event set_mock_value(det.driver.acquire_period, 0.1) set_mock_value(det.driver.acquire_time, 0.05) @@ -607,8 +616,8 @@ async def test_step_scan_keep_numimages( assert {"det-driver-acquire_period", "det-driver-acquire_time"} <= config.keys() assert config["det-driver-acquire_period"]["value"] == 0.1 assert config["det-driver-acquire_time"]["value"] == 0.05 - # Check we need to call prepare or trigger before we can describe - with pytest.raises(RuntimeError, match="Prepare not run"): + await det.stage() + with pytest.raises(TransitionError, match="describe not allowed from STAGED"): await det.describe() # Need to tell it the path is valid set_mock_value(writer.file_path_exists, True) @@ -630,8 +639,7 @@ async def test_step_scan_keep_numimages( "source": uri, } assert "det" in det.hints["fields"] - reading = await det.read() - assert "det-stats-total" in reading + assert await det.read() == {} sr, sd = [doc async for doc in det.collect_asset_docs()] assert sr == ( "stream_resource", @@ -658,26 +666,37 @@ async def test_step_scan_keep_numimages( }, ) - # Check we can prepare and change the exposure + # Check we can prepare and change the exposure. Changing the exposure changes + # the frame period, which (per ADR 0025) determines the HDF chunk shape, so a + # new StreamResource is opened rather than the old file being reused. Here the + # chunk shape is unchanged because this writer leaves flush_period unset and so + # reads the chunk size back from the IOC, but the period change still starts a + # fresh file. + # Starting a capture resets the IOC's NumCaptured, which start() waits for + callback_on_mock_put( + writer.capture, lambda v: set_mock_value(writer.num_captured, 0) if v else None + ) await det.prepare( TriggerInfo(collections_per_event=42, livetime=0.01, collection_timeout=0.1) ) callback_on_mock_put( - det.driver.acquire, lambda v: set_mock_value(writer.num_captured, 84) + det.driver.acquire, lambda v: set_mock_value(writer.num_captured, 42) ) assert await det.driver.acquire_time.get_value() == 0.01 await det.trigger() readings = await det.read() - assert "det-stats-total" in readings - docs = [doc async for doc in det.collect_asset_docs()] - (doc,) = docs - assert doc == ( + assert "det-stats-total" not in readings + new_sr, new_sd = [doc async for doc in det.collect_asset_docs()] + assert new_sr[0] == "stream_resource" + assert new_sr[1]["uid"] != sr[1]["uid"] + assert new_sr[1]["parameters"]["chunk_shape"] == (1, 768, 1024) + assert new_sd == ( "stream_datum", { "descriptor": "", - "indices": {"start": 1, "stop": 2}, + "indices": {"start": 0, "stop": 1}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": sr[1]["uid"], + "stream_resource": new_sr[1]["uid"], "uid": ANY, }, ) @@ -697,7 +716,7 @@ async def test_kinetix_step_scan_with_averaging_filter( det = adkinetix.KinetixDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), proc_suffix="PROC:", ) @@ -788,3 +807,78 @@ async def test_kinetix_step_scan_with_averaging_filter( }, ), ] + + +# Ruling 1: a once-per-event readable is shadowed by a same-named higher tier at +# any collections_per_event, and read once per event when nothing shadows it + + +@pytest.mark.parametrize("cpe", [1, 2]) +async def test_readable_with_no_data_logic_is_read_once_per_event(cpe: int): + stat = adcore.NDStatsIO("PREFIX:STATS:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stat}, name="det") + det.set_readable_format(stat.total, Format.UNCACHED_SIGNAL) + await det.stage() + await det.prepare(TriggerInfo(number_of_events=1, collections_per_event=cpe)) + + assert (await det.describe())["det-stats-total"]["shape"] == [] + + +@pytest.mark.parametrize("on_at_stage", [True, False]) +async def test_hdf_by_settings_follows_its_plugin_at_stage_for_the_whole_stage( + tmp_path, on_at_stage: bool +): + """Switching a writer's plugin mid-stage changes nothing until restaged.""" + path_provider = StaticPathProvider( + lambda datakey_name=None: str(datakey_name), tmp_path + ) + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + *( + adcore.ADDataFactory.hdf( + path_provider, + writer_suffix=f"HDF{i}:", + writer_name=f"hdf{i}", + datakey_suffix=f"-roi{i}", + always_enable=False, + ) + for i in (1, 2) + ), + ) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + hdf1, hdf2 = (det.get_plugin(f"hdf{i}", adcore.NDFileHDF5IO) for i in (1, 2)) + for hdf in (hdf1, hdf2): + set_mock_value(hdf.file_path_exists, True) + set_mock_value(hdf1.enable_callbacks, EnableDisable.ENABLE) + frames = 0 + + def acquire(value: bool) -> None: + nonlocal frames + if value: + frames += 3 + for hdf in (hdf1, hdf2): + set_mock_value(hdf.num_captured, frames) + + callback_on_mock_put(det.driver.acquire, acquire) + served = {"det-roi1", "det-roi2"} if on_at_stage else {"det-roi1"} + info = TriggerInfo(number_of_events=3) + + def enable_hdf2(enabled: bool) -> None: + value = EnableDisable.ENABLE if enabled else EnableDisable.DISABLE + set_mock_value(hdf2.enable_callbacks, value) + + enable_hdf2(on_at_stage) + await det.stage() + for flip in (False, True): + if flip: + enable_hdf2(not on_at_stage) + await det.prepare(info) + await det.kickoff() + await det.complete() + assert set(await det.describe_collect()) == served + # Events since prepare() + assert await det.get_index() == info.number_of_events + await det.unstage() diff --git a/tests/unit_tests/epics/test_adandor.py b/tests/unit_tests/epics/test_adandor.py index 36784c8289..3c5d83eeeb 100644 --- a/tests/unit_tests/epics/test_adandor.py +++ b/tests/unit_tests/epics/test_adandor.py @@ -7,6 +7,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -20,7 +21,7 @@ async def test_adandor( ) -> adandor.AndorDetector: async with init_devices(mock=True): detector = adandor.AndorDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -46,6 +47,9 @@ async def test_deadtime( async def test_prepare_external_edge( test_adandor: adandor.AndorDetector, ): + await test_adandor.stage() + # Only the calls after stage() are of interest + get_mock(test_adandor).reset_mock() await test_adandor.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -65,20 +69,6 @@ async def test_prepare_external_edge( ) -async def test_prepare_forever( - test_adandor: adandor.AndorDetector, -): - await test_adandor.prepare(TriggerInfo(number_of_events=0)) - assert_has_calls( - test_adandor.driver, - [ - call.trigger_mode.put(adandor.Andor2TriggerMode.INTERNAL), - call.image_mode.put(adcore.ADImageMode.MULTIPLE), - call.num_images.put(999_999), - ], - ) - - @pytest.mark.parametrize("num_images,expected_first_dim", [(3, 3), (0, 1)]) async def test_trigger_uses_num_images( test_adandor: adandor.AndorDetector, diff --git a/tests/unit_tests/epics/test_adaravis.py b/tests/unit_tests/epics/test_adaravis.py index 551e76cb6a..693eef163e 100644 --- a/tests/unit_tests/epics/test_adaravis.py +++ b/tests/unit_tests/epics/test_adaravis.py @@ -8,6 +8,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -21,7 +22,7 @@ async def test_adaravis( ) -> adaravis.AravisDetector: async with init_devices(mock=True): detector = adaravis.AravisDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -51,9 +52,22 @@ async def test_deadtime( assert deadtime == actual_deadtime +async def test_deadtime_inputs_dropped_from_configuration_are_not_found( + test_adaravis: adaravis.AravisDetector, +): + set_mock_value(test_adaravis.driver.model, "Mako G-125") + test_adaravis.driver.set_readable_format(test_adaravis.driver.model, None) + test_adaravis.set_readable_format(test_adaravis.driver, None) + with pytest.raises(KeyError): + await test_adaravis.get_trigger_deadtime() + + async def test_prepare_external_edge( test_adaravis: adaravis.AravisDetector, ): + await test_adaravis.stage() + # Only the calls after stage() are of interest + get_mock(test_adaravis).reset_mock() await test_adaravis.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -77,6 +91,9 @@ async def test_prepare_external_edge( async def test_prepare_internal( test_adaravis: adaravis.AravisDetector, ): + await test_adaravis.stage() + # Only the calls after stage() are of interest + get_mock(test_adaravis).reset_mock() await test_adaravis.prepare(TriggerInfo(number_of_events=11)) assert_has_calls( test_adaravis.driver, diff --git a/tests/unit_tests/epics/test_adkinetix.py b/tests/unit_tests/epics/test_adkinetix.py index 2260f33715..a502148dea 100644 --- a/tests/unit_tests/epics/test_adkinetix.py +++ b/tests/unit_tests/epics/test_adkinetix.py @@ -7,6 +7,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -20,7 +21,7 @@ async def test_adkinetix( ) -> adkinetix.KinetixDetector: async with init_devices(mock=True): detector = adkinetix.KinetixDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -50,6 +51,9 @@ async def test_deadtime( async def test_prepare_external_edge( test_adkinetix: adkinetix.KinetixDetector, ): + await test_adkinetix.stage() + # Only the calls after stage() are of interest + get_mock(test_adkinetix).reset_mock() await test_adkinetix.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -72,6 +76,9 @@ async def test_prepare_external_edge( async def test_prepare_external_level( test_adkinetix: adkinetix.KinetixDetector, ): + await test_adkinetix.stage() + # Only the calls after stage() are of interest + get_mock(test_adkinetix).reset_mock() await test_adkinetix.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_LEVEL, @@ -92,6 +99,9 @@ async def test_prepare_external_level( async def test_prepare_internal( test_adkinetix: adkinetix.KinetixDetector, ): + await test_adkinetix.stage() + # Only the calls after stage() are of interest + get_mock(test_adkinetix).reset_mock() await test_adkinetix.prepare(TriggerInfo(number_of_events=2, livetime=0.3)) assert_has_calls( test_adkinetix.driver, diff --git a/tests/unit_tests/epics/test_admerlin.py b/tests/unit_tests/epics/test_admerlin.py index 34d1286398..cc77d2968d 100644 --- a/tests/unit_tests/epics/test_admerlin.py +++ b/tests/unit_tests/epics/test_admerlin.py @@ -7,6 +7,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -20,7 +21,7 @@ async def test_admerlin( ) -> admerlin.MerlinDetector: async with init_devices(mock=True): detector = admerlin.MerlinDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -62,6 +63,9 @@ async def test_data_type_inferred_from_counter_depth( async def test_prepare_external_edge( test_admerlin: admerlin.MerlinDetector, ): + await test_admerlin.stage() + # Only the calls after stage() are of interest + get_mock(test_admerlin).reset_mock() await test_admerlin.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, diff --git a/tests/unit_tests/epics/test_adpilatus.py b/tests/unit_tests/epics/test_adpilatus.py index 70641c51db..28d2e62c97 100644 --- a/tests/unit_tests/epics/test_adpilatus.py +++ b/tests/unit_tests/epics/test_adpilatus.py @@ -8,6 +8,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -21,7 +22,7 @@ async def test_adpilatus( ) -> adpilatus.PilatusDetector: async with init_devices(mock=True): detector = adpilatus.PilatusDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) set_mock_value(detector.driver.armed, True) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) @@ -66,7 +67,7 @@ async def test_deadtime(readout_time: adpilatus.PilatusReadoutTime, tmp_path): async with init_devices(mock=True): pilatus = adpilatus.PilatusDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(path_provider), + adcore.ADDataFactory.hdf(path_provider), readout_time=readout_time, ) trigger_modes, deadtime = await pilatus.get_trigger_deadtime() @@ -87,6 +88,7 @@ async def test_times_out_if_not_armed( assert isinstance(acquire_logic, adcore.ADAcquireLogic) acquire_logic.timeout = 0.01 + await test_adpilatus.stage() with pytest.raises(TimeoutError): await test_adpilatus.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_EDGE)) @@ -94,6 +96,9 @@ async def test_times_out_if_not_armed( async def test_prepare_external_edge( test_adpilatus: adpilatus.PilatusDetector, ): + await test_adpilatus.stage() + # Only the calls after stage() are of interest + get_mock(test_adpilatus).reset_mock() await test_adpilatus.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -119,12 +124,15 @@ async def test_prepare_external_edge_with_mult_trigger_mode( async with init_devices(mock=True): detector = adpilatus.PilatusDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider), edge_trigger_mode=adpilatus.PilatusTriggerMode.MULT_TRIGGER, ) set_mock_value(detector.driver.armed, True) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) + await detector.stage() + # Only the calls after stage() are of interest + get_mock(detector).reset_mock() await detector.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -165,7 +173,7 @@ def test_edge_trigger_mode_rejects_non_edge_modes( ): adpilatus.PilatusDetector( "PREFIX:", - adcore.ADWriterFactory.hdf(path_provider), + adcore.ADDataFactory.hdf(path_provider), edge_trigger_mode=invalid_mode, ) @@ -173,6 +181,9 @@ def test_edge_trigger_mode_rejects_non_edge_modes( async def test_prepare_external_level( test_adpilatus: adpilatus.PilatusDetector, ): + await test_adpilatus.stage() + # Only the calls after stage() are of interest + get_mock(test_adpilatus).reset_mock() await test_adpilatus.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_LEVEL, @@ -190,20 +201,6 @@ async def test_prepare_external_level( ) -async def test_prepare_forever( - test_adpilatus: adpilatus.PilatusDetector, -): - await test_adpilatus.prepare(TriggerInfo(number_of_events=0)) - assert_has_calls( - test_adpilatus.driver, - [ - call.trigger_mode.put(adpilatus.PilatusTriggerMode.INTERNAL), - call.image_mode.put(adcore.ADImageMode.MULTIPLE), - call.num_images.put(999_999), - ], - ) - - @pytest.mark.parametrize("num_images,expected_first_dim", [(7, 7), (0, 1)]) async def test_trigger_uses_num_images( test_adpilatus: adpilatus.PilatusDetector, diff --git a/tests/unit_tests/epics/test_adsimdetector.py b/tests/unit_tests/epics/test_adsimdetector.py index 4c6ff7cc48..fb8b59fd96 100644 --- a/tests/unit_tests/epics/test_adsimdetector.py +++ b/tests/unit_tests/epics/test_adsimdetector.py @@ -6,6 +6,7 @@ DetectorTrigger, StaticPathProvider, TriggerInfo, + get_mock, init_devices, set_mock_value, ) @@ -19,7 +20,7 @@ async def test_adsimdetector( ) -> adsimdetector.SimDetector: async with init_devices(mock=True): detector = adsimdetector.SimDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -43,6 +44,9 @@ async def test_deadtime( async def test_prepare_internal( test_adsimdetector: adsimdetector.SimDetector, ): + await test_adsimdetector.stage() + # Only the calls after stage() are of interest + get_mock(test_adsimdetector).reset_mock() await test_adsimdetector.prepare(TriggerInfo(number_of_events=11)) assert_has_calls( test_adsimdetector.driver, diff --git a/tests/unit_tests/epics/test_advimba.py b/tests/unit_tests/epics/test_advimba.py index 610579365b..e44ed1b435 100644 --- a/tests/unit_tests/epics/test_advimba.py +++ b/tests/unit_tests/epics/test_advimba.py @@ -8,6 +8,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -21,7 +22,7 @@ async def test_advimba( ) -> advimba.VimbaDetector: async with init_devices(mock=True): detector = advimba.VimbaDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) writer = detector.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -58,6 +59,9 @@ async def test_deadtime( async def test_prepare_external_level( test_advimba: advimba.VimbaDetector, ): + await test_advimba.stage() + # Only the calls after stage() are of interest + get_mock(test_advimba).reset_mock() await test_advimba.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_LEVEL, @@ -80,6 +84,9 @@ async def test_prepare_external_level( async def test_prepare_external_edge( test_advimba: advimba.VimbaDetector, ): + await test_advimba.stage() + # Only the calls after stage() are of interest + get_mock(test_advimba).reset_mock() await test_advimba.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_EDGE, @@ -104,6 +111,9 @@ async def test_prepare_external_edge( async def test_prepare_internal( test_advimba: advimba.VimbaDetector, ): + await test_advimba.stage() + # Only the calls after stage() are of interest + get_mock(test_advimba).reset_mock() await test_advimba.prepare(TriggerInfo(number_of_events=11)) assert_has_calls( test_advimba.driver, diff --git a/tests/unit_tests/epics/test_default_trigger_info.py b/tests/unit_tests/epics/test_default_trigger_info.py index 8904c99918..10b09c814d 100644 --- a/tests/unit_tests/epics/test_default_trigger_info.py +++ b/tests/unit_tests/epics/test_default_trigger_info.py @@ -50,7 +50,7 @@ async def _make_detector( async with init_devices(mock=True): det = detector_cls( "PREFIX:", - adcore.ADWriterFactory.hdf(path_provider), + adcore.ADDataFactory.hdf(path_provider), proc_suffix=proc_suffix, ) set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) diff --git a/tests/unit_tests/fastcs/eiger/test_eiger_detector.py b/tests/unit_tests/fastcs/eiger/test_eiger_detector.py index 2570f72f28..6eb3914db1 100644 --- a/tests/unit_tests/fastcs/eiger/test_eiger_detector.py +++ b/tests/unit_tests/fastcs/eiger/test_eiger_detector.py @@ -7,6 +7,7 @@ StaticFilenameProvider, StaticPathProvider, TriggerInfo, + get_mock, init_devices, set_mock_value, ) @@ -29,6 +30,9 @@ def detector(RE, tmp_path): async def test_prepare_internal_calls_correct_parameters( detector: EigerDetector, tmp_path ): + await detector.stage() + # Only the calls after stage() are of interest + get_mock(detector).reset_mock() await detector.prepare( TriggerInfo( number_of_events=10, diff --git a/tests/unit_tests/fastcs/jungfrau/test_controller.py b/tests/unit_tests/fastcs/jungfrau/test_controller.py index 2505ea80ad..39da8f837e 100644 --- a/tests/unit_tests/fastcs/jungfrau/test_controller.py +++ b/tests/unit_tests/fastcs/jungfrau/test_controller.py @@ -41,7 +41,7 @@ def jungfrau(RE: RunEngine, tmp_path): [ ( TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL), - "Trigger type DetectorTrigger.EXTERNAL_LEVEL not supported by 'jungfrau', " + "Trigger type DetectorTrigger.EXTERNAL_LEVEL not supported, " "supported types are: [EXTERNAL_EDGE, INTERNAL]", ), ( @@ -54,6 +54,7 @@ def jungfrau(RE: RunEngine, tmp_path): async def test_prepare_val_error_on_bad_trigger_info( trigger_info: TriggerInfo, match: str, jungfrau: JungfrauDetector ): + await jungfrau.stage() with pytest.raises( ValueError, match=re.escape(match), @@ -63,6 +64,7 @@ async def test_prepare_val_error_on_bad_trigger_info( async def test_prepare_warn_on_small_exposure(jungfrau: JungfrauDetector, caplog): bad_trigger_info = TriggerInfo(livetime=1e-6) + await jungfrau.stage() with caplog.at_level(logging.WARNING): await jungfrau.prepare(bad_trigger_info) assert "Exposure time shorter than 2μs is not recommended" in caplog.messages @@ -70,6 +72,7 @@ async def test_prepare_warn_on_small_exposure(jungfrau: JungfrauDetector, caplog async def test_arm(jungfrau: JungfrauDetector): mock = get_mock(jungfrau.detector) + await jungfrau.stage() await jungfrau.prepare(TriggerInfo()) mock.reset_mock() callback_on_mock_execute( @@ -81,6 +84,8 @@ async def test_arm(jungfrau: JungfrauDetector): async def test_disarm(jungfrau: JungfrauDetector): + await jungfrau.stage() + get_mock(jungfrau).reset_mock() await jungfrau.unstage() assert_has_calls( jungfrau.detector, @@ -103,6 +108,9 @@ async def test_signals_set_in_pedestal_mode(jungfrau: JungfrauDetector): ) mock = get_mock(jungfrau.detector) mock.reset_mock() + await jungfrau.stage() + # Only the calls after stage() are of interest + get_mock(jungfrau).reset_mock() await jungfrau.prepare(good_trigger_info) assert_has_calls( jungfrau.detector, @@ -121,6 +129,9 @@ async def test_signals_set_in_standard_internal_mode(jungfrau: JungfrauDetector) collections_per_event=10, trigger=DetectorTrigger.INTERNAL, ) + await jungfrau.stage() + # Only the calls after stage() are of interest + get_mock(jungfrau).reset_mock() await jungfrau.prepare(good_trigger_info) assert_has_calls( jungfrau.detector, @@ -140,6 +151,9 @@ async def test_signals_set_in_standard_external_mode(jungfrau: JungfrauDetector) trigger=DetectorTrigger.EXTERNAL_EDGE, deadtime=1, ) + await jungfrau.stage() + # Only the calls after stage() are of interest + get_mock(jungfrau).reset_mock() await jungfrau.prepare(good_trigger_info) assert_has_calls( jungfrau.detector, @@ -155,6 +169,7 @@ async def test_signals_set_in_standard_external_mode(jungfrau: JungfrauDetector) async def test_error_in_pedestal_and_external_modes(jungfrau: JungfrauDetector): await jungfrau.acquisition_type.set(AcquisitionType.PEDESTAL) + await jungfrau.stage() with pytest.raises( ValueError, match="Jungfrau must be triggered internally while in pedestal mode.", @@ -176,6 +191,7 @@ async def test_prepare_val_error_if_pedestal_mode_and_odd_number_of_events( trigger_info = TriggerInfo(livetime=1e-3, deadtime=1, collections_per_event=15) + await jungfrau.stage() with pytest.raises( ValueError, match=re.escape( diff --git a/tests/unit_tests/fastcs/odin/test_odin_io.py b/tests/unit_tests/fastcs/odin/test_odin_io.py index aba2db062b..bc4db9c0bc 100644 --- a/tests/unit_tests/fastcs/odin/test_odin_io.py +++ b/tests/unit_tests/fastcs/odin/test_odin_io.py @@ -1,4 +1,5 @@ import asyncio +from functools import cached_property from pathlib import Path from unittest.mock import call @@ -6,11 +7,13 @@ from bluesky import RunEngine from ophyd_async.core import ( + DetectorLogic, StandardDetector, StaticFilenameProvider, StaticPathProvider, TriggerInfo, callback_on_mock_execute, + get_mock, init_devices, set_mock_value, soft_signal_rw, @@ -27,11 +30,15 @@ def __init__(self, tmp_path: Path, name="", connector=None): path_provider = StaticPathProvider(StaticFilenameProvider("filename"), tmp_path) self.odin = OdinIO(connector=fastcs_connector("PREFIX:")) self.bit_depth = soft_signal_rw(int, BIT_DEPTH) - self.add_detector_logics( - OdinDataLogic(path_provider, self.odin, self.bit_depth) - ) + self._path_provider = path_provider super().__init__(name, connector) + @cached_property + def logic(self) -> DetectorLogic: + return DetectorLogic( + data=[OdinDataLogic(self._path_provider, self.odin, self.bit_depth)] + ) + @pytest.fixture def odin_det(RE: RunEngine, tmp_path) -> OdinDet: @@ -44,6 +51,7 @@ async def test_describe_gives_detector_shape(odin_det: OdinDet, tmp_path): set_mock_value(odin_det.odin.writing, True) set_mock_value(odin_det.odin.fp.data_dims_1, 1024) set_mock_value(odin_det.odin.fp.data_dims_0, 768) + await odin_det.stage() await odin_det.prepare(TriggerInfo()) description = await odin_det.describe() assert description == { @@ -62,6 +70,8 @@ async def test_describe_gives_detector_shape(odin_det: OdinDet, tmp_path): async def test_when_closed_then_data_capture_turned_off(odin_det: OdinDet): + await odin_det.stage() + get_mock(odin_det).reset_mock() await odin_det.unstage() assert_has_calls( odin_det, @@ -78,6 +88,8 @@ async def test_wait_for_active_and_file_names_before_capture_then_wait_for_writi odin: OdinIO = odin_det.odin ev = asyncio.Event() callback_on_mock_execute(odin_det.odin.fp.start_writing, lambda: ev.set()) + await odin_det.stage() + get_mock(odin_det).reset_mock() # Start it preparing status = odin_det.prepare(TriggerInfo(number_of_events=15)) # Wait for start_writing to be called @@ -110,5 +122,6 @@ async def test_wait_for_active_and_file_names_before_capture_then_wait_for_writi @pytest.mark.timeout(15) async def test_hinted_fields(odin_det: OdinDet): set_mock_value(odin_det.odin.writing, True) + await odin_det.stage() await odin_det.prepare(TriggerInfo()) assert odin_det.hints == {"fields": ["det"]} diff --git a/tests/unit_tests/fastcs/panda/test_hdf_panda.py b/tests/unit_tests/fastcs/panda/test_hdf_panda.py index 76c423f7cc..3367d91996 100644 --- a/tests/unit_tests/fastcs/panda/test_hdf_panda.py +++ b/tests/unit_tests/fastcs/panda/test_hdf_panda.py @@ -1,5 +1,6 @@ from __future__ import annotations +import asyncio import logging from unittest.mock import ANY, call @@ -11,6 +12,7 @@ StaticPathProvider, TriggerInfo, callback_on_mock_put, + get_mock, init_devices, set_mock_value, ) @@ -20,7 +22,11 @@ PandaCaptureMode, PandaHdf5DatasetType, ) -from ophyd_async.testing import assert_configuration, assert_has_calls +from ophyd_async.testing import ( + assert_configuration, + assert_has_calls, + wait_for_pending_wakeups, +) PANDA_DETECTOR_NAME = "mock_panda" @@ -78,6 +84,7 @@ async def test_open_returns_correct_descriptors_and_resources( hdf_panda: HDFPanda, table: DatasetTable, caplog, tmp_path ): set_mock_value(hdf_panda.data.datasets, table) + await hdf_panda.stage() with caplog.at_level(logging.WARNING): await hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) description = await hdf_panda.describe() @@ -99,6 +106,7 @@ async def test_open_returns_correct_descriptors_and_resources( "external": "STREAM:", } + await hdf_panda.kickoff() set_mock_value(hdf_panda.data.num_captured, 7) docs = [doc async for doc in hdf_panda.collect_asset_docs()] sr = docs[: len(table)] @@ -134,6 +142,9 @@ async def test_open_returns_correct_descriptors_and_resources( async def test_open_sets_correct_signals(hdf_panda: HDFPanda, tmp_path): + await hdf_panda.stage() + # Only the calls after stage() are of interest + get_mock(hdf_panda).reset_mock() await hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) assert_has_calls( hdf_panda, @@ -150,6 +161,8 @@ async def test_open_sets_correct_signals(hdf_panda: HDFPanda, tmp_path): async def test_close_sets_correct_signals(hdf_panda: HDFPanda): + await hdf_panda.stage() + get_mock(hdf_panda).reset_mock() await hdf_panda.unstage() assert_has_calls(hdf_panda.pcap, [call.arm.put(False)]) assert_has_calls(hdf_panda.data, [call.capture.put(False)]) @@ -161,6 +174,7 @@ async def test_flyscan_documents(hdf_panda: HDFPanda, tmp_path): assert config_description == {} await assert_configuration(hdf_panda, {}) # Prepare for N frames + await hdf_panda.stage() await hdf_panda.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_LEVEL, @@ -276,7 +290,7 @@ async def test_flyscan_documents(hdf_panda: HDFPanda, tmp_path): # Check that another kickoff without prepare is not allowed with pytest.raises( RuntimeError, - match="Kickoff requested 15:30, but detector was only prepared up to 15", + match="cannot go to FLYING from FLOWN", ): await hdf_panda.kickoff() # But preparing again is ok @@ -288,19 +302,22 @@ async def test_flyscan_documents(hdf_panda: HDFPanda, tmp_path): ) ) await hdf_panda.kickoff() - # Completing adds indexes to the same resource + # Completing starts new resources in the same file, the indices carrying on status = hdf_panda.complete() set_mock_value(hdf_panda.data.num_captured, 24) set_mock_value(hdf_panda.pcap.active, False) docs = [doc async for doc in hdf_panda.collect_asset_docs()] - assert docs == [ + new_sr = docs[:2] + assert new_sr == [(name, {**doc, "uid": ANY}) for name, doc in sr] + assert {doc["uid"] for _, doc in new_sr}.isdisjoint(doc["uid"] for _, doc in sr) + assert docs[2:] == [ ( "stream_datum", { "descriptor": "", "indices": {"start": 5, "stop": 8}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": sr[0][1]["uid"], + "stream_resource": new_sr[0][1]["uid"], "uid": ANY, }, ), @@ -310,7 +327,7 @@ async def test_flyscan_documents(hdf_panda: HDFPanda, tmp_path): "descriptor": "", "indices": {"start": 5, "stop": 8}, "seq_nums": {"start": 0, "stop": 0}, - "stream_resource": sr[1][1]["uid"], + "stream_resource": new_sr[1][1]["uid"], "uid": ANY, }, ), @@ -319,6 +336,7 @@ async def test_flyscan_documents(hdf_panda: HDFPanda, tmp_path): async def test_get_indices_written(hdf_panda: HDFPanda): + await hdf_panda.stage() await hdf_panda.prepare( TriggerInfo( trigger=DetectorTrigger.EXTERNAL_LEVEL, @@ -334,6 +352,7 @@ async def test_get_indices_written(hdf_panda: HDFPanda): async def test_oserror_when_hdf_dir_does_not_exist(hdf_panda: HDFPanda): set_mock_value(hdf_panda.data.directory_exists, False) + await hdf_panda.stage() with pytest.raises(OSError): await hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) @@ -342,3 +361,93 @@ async def test_deadtime(hdf_panda: HDFPanda): supported_triggers, deadtime = await hdf_panda.get_trigger_deadtime() assert deadtime == 8e-9 assert supported_triggers == {DetectorTrigger.EXTERNAL_LEVEL} + + +async def test_uc8_step_scan_stages_first_and_emits_stream_assets( + hdf_panda: HDFPanda, tmp_path +): + """Use case 8, step: a trigger() gives one StreamDatum per dataset. + + Written for the rewrite's lifecycle: stage() comes first. The PandA is + externally triggered, so prepare() arms it and trigger() waits for the point. + """ + await hdf_panda.stage() + + await hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) + status = hdf_panda.trigger() + await wait_for_pending_wakeups(raise_if_exceeded=False) + assert not status.done + # The external trigger fires and the PandA captures the point + set_mock_value(hdf_panda.data.num_captured, 1) + await status + docs = [doc async for doc in hdf_panda.collect_asset_docs()] # pyright: ignore[reportAttributeAccessIssue] + assert [(n, d.get("data_key")) for n, d in docs[:2]] == [ + ("stream_resource", "x"), + ("stream_resource", "y"), + ] + assert [d["indices"] for n, d in docs[2:]] == [{"start": 0, "stop": 1}] * 2 + assert await hdf_panda.read() == {} + await hdf_panda.unstage() + + +def test_uc8_fly_scan_through_the_run_engine(RE, hdf_panda: HDFPanda): + """Use case 8, fly: stream assets covering N events, no Events.""" + import bluesky.plan_stubs as bps # noqa: PLC0415 + import bluesky.preprocessors as bpp # noqa: PLC0415 + + @bpp.stage_decorator([hdf_panda]) + @bpp.run_decorator() + def plan(): + yield from bps.prepare( + hdf_panda, + TriggerInfo( + trigger=DetectorTrigger.EXTERNAL_LEVEL, + collections_per_event=3, + number_of_events=2, + ), + wait=True, + ) + yield from bps.declare_stream(hdf_panda, name="primary", collect=True) + yield from bps.kickoff(hdf_panda, wait=True) + # The external triggers fire and the PandA captures everything + set_mock_value(hdf_panda.data.num_captured, 6) + set_mock_value(hdf_panda.pcap.active, False) + yield from bps.collect_while_completing( + flyers=[hdf_panda], dets=[hdf_panda], flush_period=0.05 + ) + + docs: dict[str, list] = {} + RE(plan(), lambda n, d: docs.setdefault(n, []).append(d)) + + assert [r["data_key"] for r in docs["stream_resource"]] == ["x", "y"] + assert "event" not in docs + assert [d["indices"]["stop"] for d in docs["stream_datum"]][-2:] == [2, 2] + + +async def test_prepare_waits_for_num_captured_to_read_zero(hdf_panda: HDFPanda): + await hdf_panda.stage() + set_mock_value(hdf_panda.data.num_captured, 5) + + preparing = hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) + await asyncio.sleep(0.1) + # NumCaptured has not been reset by the capture yet + assert not preparing.done + set_mock_value(hdf_panda.data.num_captured, 0) + await asyncio.wait_for(preparing, timeout=1) + + +async def test_datasets_filled_only_once_capture_starts_are_described( + hdf_panda: HDFPanda, +): + # PandABlocks-ioc fills DATASETS when capture starts, not before + set_mock_value(hdf_panda.data.datasets, TABLES[0]) + callback_on_mock_put( + hdf_panda.data.capture, + lambda value, wait=True: ( + set_mock_value(hdf_panda.data.datasets, TABLES[1]) if value else None + ), + ) + await hdf_panda.stage() + await hdf_panda.prepare(TriggerInfo(trigger=DetectorTrigger.EXTERNAL_LEVEL)) + + assert list(await hdf_panda.describe()) == ["x"] diff --git a/tests/unit_tests/plan_stubs/test_ndattributes.py b/tests/unit_tests/plan_stubs/test_ndattributes.py index 8246e48c10..8695c67415 100644 --- a/tests/unit_tests/plan_stubs/test_ndattributes.py +++ b/tests/unit_tests/plan_stubs/test_ndattributes.py @@ -10,7 +10,7 @@ def test_setup_ndstats_raises_type_error(RE, static_path_provider: StaticPathProvider): with init_devices(mock=True): det = adsimdetector.SimDetector( - "PREFIX:", adcore.ADWriterFactory.hdf(static_path_provider) + "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) ) with pytest.raises( AttributeError,