diff --git a/CHANGELOG.md b/CHANGELOG.md index 5071fa5038..0e2c9b86d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,12 @@ - `SimMotor.stage()` now calls `SimMotorLogic.ensure_ready()`, which calls `ensure_stopped()` and so stops the simulated motion, waits for it to finish, and sets the setpoint to the current readback. `unstage()` cancels a fly in progress, then calls `ensure_stopped()`. The simulated motion runs in the task of the verb that moves, so `stop()` during a `prepare()` or fly, or `stage()` during a `set()`, cancels that verb, which raises `CancelledError`. - The progress of `StandardFlyable.prepare()` now comes from the `WatcherUpdate`s the logic's `prepare_to_fly` yields, replacing the heuristic that watched setpoint writes (#1448). - `WatchableAsyncStatus` closes the async generator it drives as soon as it is cancelled or fails, rather than when the generator is collected, and a watcher that raises is logged and ignored rather than failing the status. +- Each statistic of an `NDStatsIO` has one name, that of its scalar signal (e.g. `det-stats-total`), as a readable, as a stats time series and as an HDF NDAttribute, so the highest of these serving a scan suppresses the others, and the hint of the scalar stays. The plugin's `EnableCallbacks` and `compute_statistics`/`compute_centroid` PVs at the first `prepare()` after `stage()` decide which statistics are used: the scalars of those switched off are left out of `read()`, `describe()` and `hints`, `AreaDetector.prepare()` raises, naming what changed, if any of these switches has changed since the first `prepare()` after `stage()`, and a stats time series publishes every statistic switched on that has a read format on the plugin. +- `AreaDetector` registers every plugin as a `CHILD` readable, so the signals a plugin declares as read, configuration or hinted appear in the detector's `read()`, `read_configuration()` and `hints`. Opt a plugin out with `set_readable_format(plugin, None)`. This covers the plugins in `plugins`, which are all the detector's plugins including those it creates (the circular buffer and process plugin), and the plugin each writer factory creates. Every factory's `build` is given all of them. A logic writes only the switches of the plugins it was built to own; the switches of every other plugin it only reads. +- `setup_ndstats_sum` is removed: list the stats plugin in `ADDataFactory.hdf(ndattribute_sources=[stats])` and the HDF writer writes its NDAttributes, named after their scalars, so `det-sum` is now `det-stats-total`. A listed plugin's `nd_attributes_file` is overwritten when the HDF writer serves a scan; `setup_ndattributes` on another plugin or the driver works as before. +- `AreaDetector` hides the read signals of a disabled plugin: at the first `prepare()` after `stage()` it walks the readable tree from the detector and asks each `NDPluginBaseIO` it finds for its `get_enabled_params()`, through a `DetectorSuppressLogic`, and keeps the answer until the next `stage()`. It leaves out of `read()`, `describe()` and `hints` each read signal of a plugin that is not in the answer, and `prepare()` raises if a plugin now answers differently. A plugin that is not a `StandardReadable`, such as a trigger logic's process plugin or an acquire logic's circular buffer, is not asked. +- `NDROIStatIO.stage()` names each channel `-`, with characters other than letters, digits and underscores replaced by underscores and the channel number used for an empty name, and raises if two would be the same. This changes the datakey of every ROI stat signal. A channel that is not used leaves `read()`, `describe()` and `hints`, and a `Name` changed after `stage()` takes effect at the next one. +- `adcore.prepare_filter` sets `enable_filter` to False for a single exposure, rather than filtering one, and switches the process plugin on as before. - A `StandardReadable` that a detector suppresses with its data keeps the hints of the signals it drops from `read()` and `describe()`, as the data has the same name. A data logic that hints such a name now makes `hints` raise. - `StandardReadable` now keeps one format per registered child, keyed by the child, so registering a child again replaces its format instead of adding a second contribution, and the order of `hints` is the order children were first registered. The private `_describe_config_funcs`, `_read_config_funcs` and `_has_hints` tuples are gone: a subclass that added to them now adds a hook to `_describe_configuration_funcs`, `_read_configuration_funcs` or `_hints_funcs` (see ADR 0023). - The deprecated `ConfigSignal` and `HintedSignal` aliases are removed from `ophyd_async.core`; use `StandardReadableFormat.CONFIG_SIGNAL`, `HINTED_SIGNAL` or `HINTED_UNCACHED_SIGNAL`. `StandardReadable.add_readables()` is deprecated in favour of `set_readable_format()` on each device. @@ -48,14 +54,18 @@ - `StandardDetector` exposes `collect_asset_docs()` or `collect_pages()`, whichever the serving providers produce, as an instance attribute bound by `prepare()`, so the bluesky run bundler asks for the right one. - A registered signal with the same name as a datakey a serving provider makes is left out of `read()`, `describe()` and `hints`, so the provider's data replaces it; `describe()` raises `ValueError` if the two differ in dtype or shape beyond one leading dimension. - `TriggerInfo.number_of_events` must now be positive. -- `ADHDFDataLogic` sizes its HDF chunks from `TriggerInfo.flush_period`: when it and the collection period are both positive, `NumFramesChunks` is set to the frames collected in a flush period, `round(flush_period / collection_period)`, at least 1 (#1309). Otherwise the value on the IOC is used, 0 taken as 1. `NumFramesChunks` is now written back at every `prepare()` that starts the HDF writer, where it used to be written only to change 0 to 1. -- `adcore.ADWriterFactory` is renamed `ADDataFactory`; the old name still works with a `DeprecationWarning`. Its factories take `always_enable`, default `False`: the plugin's `EnableCallbacks` PV is followed and the plugin sits out the scans of a stage if it is off at the first `prepare()`, so another data logic can serve them. The HDF writer used to be switched on whatever the PV said, so pass `always_enable=True` to keep that. +- `ADHDFDataLogic` sizes its HDF chunks from `TriggerInfo.flush_period`: when it and the collection period are both positive, `NumFramesChunks` is set to the frames collected in a flush period, `round(flush_period / collection_period)`, at least 1 (#1309), and at most what fits in a 4 MiB chunk. Otherwise the value on the IOC is used, 0 taken as 1. `NumFramesChunks` is now written back at every `prepare()` that starts the HDF writer, where it used to be written only to change 0 to 1. +- `adcore.ADWriterFactory` is renamed `ADDataFactory`; the old name still works with a `DeprecationWarning`. `ADDataFactory` is built with keywords only: `build`, a callable `(prefix, driver, plugins) -> (named plugin or None, data logic)`, in place of `writer_cls`, `writer_suffix` and `writer_name`, which `hdf`, `jpeg` and `tiff` still take. Its factories take `always_enable`, default `False`: the plugin's `EnableCallbacks` PV is followed and the plugin sits out the scans of a stage if it is off at the first `prepare()`, so another data logic can serve them. The HDF writer used to be switched on whatever the PV said, so pass `always_enable=True` to keep that. `ADHDFDataLogic` and `ADMultipartDataLogic` with it switch their plugin on at `stage()`, and no longer write `EnableCallbacks` at each `prepare()`. - `StreamResourceDataProvider` takes `collections_written` (main's keyword was `collections_written_signal`), the signal counting the collections written, and `collections_per_event`. - `StreamableDataProvider.collections_written_signal` is now `DataProvider.collections_written`, an attribute every provider has, the signal counting the collections written. ### Added +- `NDParam`, a scalar a plugin publishes with its asyn parameter, address and time series array, and the `ts_*` array signals of every statistic on `NDStatsIO`. +- `ADDataFactory.hdf(ndattribute_sources=)`, the plugins whose NDAttributes the HDF writer writes. A source must be in the detector's `plugins`, as one outside it is never connected. The NDAttributes of other plugins and the driver are parsed from the XML the user set, with the names in it. +- `NDPluginBaseIO.get_enabled_params()`, the `NDParam`s whose switches are on for the coming scan; those with a read format on the plugin are published, overridden by `NDStatsIO` and `NDROIStatIO`. - `DetectorSuppressLogic`, set as `DetectorLogic.suppress`, returns from `hidden_datakeys()` the names of the readables the detector's data hides. `DetectorLogic.prepare_to_fly` asks it at the first `prepare()` after `stage()`, keeps the answer in the context and raises at a later `prepare()` if it differs. +- `NDROIStatIO` stamps the `min_value`, `max_value`, `mean_value`, `total` and `net` of each channel that is used and has a read format, named after the scalar. `NDROIStatIO` is a `StandardReadable` whose channels are registered as `CHILD` readables, so there is no need to register each channel. - `KICKED_OFF`, the sentinel `FlyableLogic.fly` yields once the scan is up to speed. - `watch_readback()` reports the progress of a move by watching a `MovableLogic` readback, for use from `FlyableLogic` hooks. - `SimMotor.prepare()` and `SimMotor.complete()` report progress to watchers, as `Motor` does. @@ -67,9 +77,13 @@ - `StandardReadable.set_readable_format()`, `reset_readable_formats()`, `get_readable_formats()` and `get_readable_format_changes()` (which yield each child with its format), to change how a child contributes to `read()`, `read_configuration()` and `hints` at runtime, the equivalent of `Kind` in ophyd v1 (see "How to change what a Device reports at runtime"). - `ReadableEntry`, `ReadableFormats`, `walk_readable_formats()`, `apply_readable_formats()` and `READABLE_FORMATS_KEY` in `ophyd_async.core`, and a `readable_formats` field on `Settings`, so formats are stored and retrieved with the signal values. - pyright now runs with `reportMatchNotExhaustive`. -- `BoundedDataLogic`, `UnboundedDataLogic`, `DetectorLogic`, `DataProvider`, `PageableDataProvider` and `EventPageDataProvider` in `ophyd_async.core`, and `ADDataFactory` in `ophyd_async.epics.adcore`. -- `EventPageDataProvider(refresh=)`, an awaitable callable awaited before each page's arrays are read. - `TriggerInfo.flush_period`, how many seconds between flushes of the data being written, default 0 for whatever the hardware is set to. Each data logic's `start_provider()` is given it. +- `BoundedDataLogic`, `UnboundedDataLogic`, `DetectorLogic`, `DataProvider`, `PageableDataProvider` and `EventPageDataProvider` in `ophyd_async.core`, and `ADDataFactory` and `NDStatsTSAcquireMode` in `ophyd_async.epics.adcore`. +- `adcore.TimeSeriesDataLogic` and the `ADDataFactory.time_series(plugin)` factory, a `BoundedDataLogic` that publishes the time series of an `NDPluginStats` or `NDPluginROIStat` as event pages, for an areaDetector that writes no file. It needs a fixed number of collections, as the buffer is sized before acquisition, and sits out when the plugin is switched off at the first `prepare()` after `stage()`, so a detector can switch between its file writer and its time series from one stage to the next. It never switches the plugin on itself; for an `NDStatsIO` it does switch on the embedded `TS:` time series plugin, which the IOC starts disabled. For an `NDROIStatIO` it publishes one array per used channel per scalar with a read format, named after the scalar (e.g. `det-roistat-Blue-total`), which it suppresses. +- `NDROIStatTSControl`, the `ts_control`, `ts_num_points`, `ts_current_point` and `ts_acquiring` signals of `NDROIStatIO`, and the `ts_*` arrays of `NDROIStatNIO`, which `TimeSeriesDataLogic` uses. +- `adcore.NDStatsTimeSeriesIO`, the `TS:` sub-plugin of `NDStatsIO`, available as `NDStatsIO.ts_plugin`; `TimeSeriesDataLogic` switches it on. +- `NDROIStatNIO.net` and `NDROIStatNIO.ts_net`: `net` has no read format, as on `NDStatsIO`, so give it one to read, stamp or publish the `TSNet` time series of a channel. +- `EventPageDataProvider(refresh=)`, an awaitable callable awaited before each page's arrays are read. The stats and ROI stat time series use it to ask the plugin to post its arrays (`NDStatsIO.ts_read`, or `NDROIStatTSControl.READ`), which it otherwise does only about once a second while acquiring, so a page collected mid-scan no longer reads arrays shorter than the count. Both time series raise `ValueError` at `prepare()` if the plugin's arrays (`ts_timestamp_nelm`, the waveforms' `NELM`) hold fewer points than the scan needs. ### Fixed diff --git a/docs/explanations/decisions/0024-standard-detector-is-a-standard-readable.md b/docs/explanations/decisions/0024-standard-detector-is-a-standard-readable.md index 93f0faac31..abaf8757c5 100644 --- a/docs/explanations/decisions/0024-standard-detector-is-a-standard-readable.md +++ b/docs/explanations/decisions/0024-standard-detector-is-a-standard-readable.md @@ -6,6 +6,9 @@ Date: 2026-10-30 Accepted +Partly superseded by [ADR 0027](0027-areadetector-statistic-names.md), under which +`AreaDetector` registers every plugin. + ## Context [ADR 0012](0012-detector-rewrite.md) gave `StandardDetector` its own ways to put a signal diff --git a/docs/explanations/decisions/0027-areadetector-statistic-names.md b/docs/explanations/decisions/0027-areadetector-statistic-names.md new file mode 100644 index 0000000000..7aa9a9e4e3 --- /dev/null +++ b/docs/explanations/decisions/0027-areadetector-statistic-names.md @@ -0,0 +1,125 @@ +# 27. One name per areaDetector statistic, chosen by the plugin's switches + +Date: 2026-11-03 + +## Status + +Accepted + +## Context + +[ADR 0025](0025-bounded-data-logic-tier.md) added a bounded tier so an areaDetector that +writes no file can publish a stats time series, and sized HDF chunks from the flush +period. [ADR 0026](0026-detector-is-a-flyable.md) let the data a detector makes suppress +the readables it replaces. An areaDetector statistic can reach the documents in three +ways: the scalar read once per event, its time series, and an NDAttribute in the HDF file. +Fly scanning with an HDF writer and a time series together was #1364. + +## Decision + +### Stats go via NDAttributes with a file writer, via the time series without + +An HDF writer already pulls NDAttributes into the file, so only a detector that writes no +file needs `TimeSeriesDataLogic`. A plugin has one time series control shared by about +twenty arrays, so one logic covers them all rather than several contending over it. It +serves `NDStatsIO` and `NDROIStatIO` through one inner logic per plugin type, as the two +differ only in how they arm, post and stop. + +### HDF chunks are capped at 4 MiB + +A chunk holds one flush period of frames, so a fast detector with small frames (an Xspress3 +MCA) packs many. The cap keeps a fast detector with large frames near what a filesystem +reads well. 4 MiB is a first guess for GPFS, left as an open question to reviewers. + +The period is `exposures_per_collection * (livetime + deadtime)`, so with a `livetime` of 0 a +chunk is sized from the deadtime alone. A period of 0 sizes nothing: the chunk is the +`NumFramesChunks` the IOC holds, or 1 if that is 0. + +### One name per statistic, and the plugin's switches decide which are used + +Suppression only works if every tier names a quantity the same. A statistic read per event, +as a time series, or as an HDF NDAttribute is named after the scalar signal +(`det-stats-total`), the name a user already gets by registering it, so the highest tier +serving a scan suppresses the others unconfigured. `setup_ndstats_sum` goes. + +Which statistics are used is decided by the plugin's PVs: they make the IOC produce the data +and are captured by settings, and switches in Python could only disagree with them. + +- `NDPluginBaseIO.get_enabled_params()` returns the `NDParam`s a plugin's own switches have + on (`ComputeStatistics`, an ROI channel's `Use`), or `None` if they gate none of its read + signals. Overrides leave `EnableCallbacks` to the callers, which check it once. +- `published_params(plugin)`, the enabled params of an enabled plugin that have a read + format, is what both the HDF writer's NDAttributes and the time series publish, so each + tier publishes what the others would. The IO classes hold signals and static facts; + `AreaDetector` and the data logics decide. +- `AreaDetector`'s `DetectorSuppressLogic` ([ADR 0026](0026-detector-is-a-flyable.md)) + walks the detector's *readable* tree through its `CHILD` formats, not the Device tree, + so a stats plugin registered under an ROI is found. A disabled plugin hides all its read + signals, as it produces nothing; an enabled one hides those its params leave out. +- Only a plugin whose switches no logic writes may gate its readables, so the hidden set + can be read before the trigger logic is prepared. These are the plugins a logic is + passed by reference, stats and ROI stat plugins, whose switches are only read. The + process plugin and circular buffer are not `StandardReadable`, so declare nothing to read + or hide and are not asked. +- The stats time series logic always enables the `TS:` time series plugin embedded in the + stats plugin it reads, when it arms, as it is the logic's internal machinery. It writes no + other switch of the stats plugin. +- A plugin's `NDArrayPort` is not checked nor the chain upstream walked: it can cross IOCs + over PVA, and the wiring is the user's business. The one exception is the circular buffer + of `ADContAcqAcquireLogic`, which must read directly from the driver, else `stage()` raises. + +Only the plugin registered as a readable hints a statistic, whichever tier serves it. + +A logic writes only the switches of the plugins it was built to own: the circular buffer +its acquire logic is given, the process plugin its trigger logic is given, its writer, and +the `TS:` sub-plugin. Every other plugin's switches it only reads, to decide what to +publish. + +- `plugins` on `AreaDetector` is all the detector's plugins, by attribute name: those the + caller passes and those the detector creates (the circular buffer, and the process plugin + of `proc_suffix`, which a detector gives to its trigger logic too). It holds no `None`: a + detector without a plugin leaves its name out. No logic writes the `EnableCallbacks`, + `ComputeStatistics` or ROI channel `Use` of a stats or ROI stat plugin. +- The writers a factory creates (HDF, TIFF, JPEG) are the detector's own too. Its logics + always switch on the circular buffer and the process plugin, and a writer's + `always_enable` says whether its logic switches it on (`True`) or sits out the scans of a + stage if the user left it off at the first `prepare()` (`False`, the default). A detector + with no writer is valid: its data is read from the plugins. +- Every plugin of the detector is an attribute and registered as a readable; + `set_readable_format(plugin, None)` opts one out. This replaces the per-plugin opt-in of + [ADR 0024](0024-standard-detector-is-a-standard-readable.md), as a plugin that is not read + cannot hint the statistics the other tiers publish. The factories are given all of them, + so the HDF writer parses the NDAttributes of the process plugin. +- Data factories take plugins by reference, which must be in `plugins`: a plugin outside it + is never connected, so `ADDataFactory.hdf(ndattribute_sources=)` and + `ADDataFactory.time_series` raise `ValueError` when the detector is made. + +The HDF writer stays free of stats knowledge: `ADDataFactory.hdf(ndattribute_sources=)` +lists the plugins it stamps, so two writers stamp only their own ROIs. NDAttributes from the +user's XML keep their names, as they may be about something outside the detector. + +An ROI stat plugin is a source and a readable whose used channels are its children. Each +channel is named at `stage()` as `-`, sanitised, or by number if +`Name` is empty, so the names say what it measures. Naming at `stage()` keeps asking for +params free of side effects; a clash raises there. + +### Time series pages are read only once the arrays hold them + +Both plugins post their arrays only when full, on Stop or Read, or about once a second, so a +page read between posts would be short: + +- The provider is given a refresh that requests a Read before each page, posting the arrays + so far without stopping the buffer. It lives on the provider, which decides when a page is + read. +- A Read can complete before the waveforms process, so the provider `observe_value`s each + array until it holds the page, for at most `ARRAY_WAIT_TIMEOUT`. +- The arrays hold `NCHANS` elements, so `start_provider` checks the timestamp waveform's + `NELM` and raises at `prepare()` for a longer scan. `TSNumPoints` cannot serve: only the + ROI stat plugin clamps it to `NCHANS`. + +## Consequences + +- Fly scanning an areaDetector with an HDF writer and a time series works, closing #1364. +- `setup_ndstats_sum` is removed, and its `det-sum` is now `det-stats-total`. +- A statistic is switched on and off with the plugin's PVs, not with arguments to the data + logics. diff --git a/docs/how-to/implement-ad-detector.md b/docs/how-to/implement-ad-detector.md index 8ffb3a39d4..7d8b8ac339 100644 --- a/docs/how-to/implement-ad-detector.md +++ b/docs/how-to/implement-ad-detector.md @@ -81,8 +81,8 @@ The constructor parameters should include: - `prefix`: The PV prefix for the driver and plugins - `*writer_factories`: Zero or more [](#adcore.ADDataFactory) instances (e.g. `ADDataFactory.hdf(path_provider)`); omit entirely to skip file writing - `driver_suffix`: A PV suffix for the driver, defaulting to `"cam1:"` -- `proc_suffix`: An optional PV suffix for a process plugin, which the detector creates with [](#adcore.plugins_with_proc) and gives to its trigger logic -- `plugins`: An optional mapping of {`name`: [](#adcore.NDPluginBaseIO)} for additional plugins +- `proc_suffix`: If your detector can filter exposures, an optional PV suffix for the [](#adcore.NDProcessIO) it creates, and gives to its trigger logic and to `AreaDetector` in `plugins`. Build the plugins and the process plugin together with [](#adcore.plugins_with_proc), which raises `ValueError` if the caller's `plugins` already has a plugin called `proc` +- `plugins`: An optional mapping of {`name`: [](#adcore.NDPluginBaseIO)} for the other plugins of the detector - `config_sigs`: Additional signals to report in configuration (beyond the standard acquire_time and acquire_period) - `name`: An optional name for the device - Any detector-specific override parameters for your trigger logic @@ -98,7 +98,8 @@ The `AreaDetector` baseclass will: - Build a [](#DetectorLogic) from your trigger and acquire logic and return it from `logic` - Create and register a data logic for file writing for each factory in `writer_factories` - Add configuration signals (driver.acquire_time, driver.acquire_period, and any you specify) -- Store any plugins as attributes on the detector +- Store any plugins as attributes on the detector, and register each as a readable, so the signals it declares appear in `read()`, `read_configuration()` and `hints` (opt one out with `det.set_readable_format(plugin, None)`). This covers the `plugins`, those you pass and those the detector creates (the process plugin, the circular buffer of `ContAcqDetector`), and the plugin each writer factory creates +- Pass every one of those plugins to the `build` of each factory, so the HDF writer can parse their NDAttributes ## Declare the public interface @@ -117,20 +118,53 @@ For example, for ADAravis: ## Add multiple data streams (optional) -The composition-based architecture makes it possible to give a detector several data outputs. Every [](#adcore.ADDataFactory) factory takes `always_enable`: `False` (the default) follows the plugin's `EnableCallbacks` PV, so a data output whose plugin is off sits out and another can serve the scan, and `True` switches the plugin on at the start of a scan. +The composition-based architecture makes it possible to give a detector several data outputs. The file writer factories ([](#adcore.ADDataFactory.hdf), `jpeg` and `tiff`) take `always_enable`: `False` (the default) follows the plugin's `EnableCallbacks` PV, so a writer whose plugin is off sits out and another output can serve the scan, and `True` switches the plugin on at `stage()`. A stats time series has no `always_enable`: its plugin's switches decide, as below. A writer with `always_enable=True` is switched on once, at `stage()`; the factory does not write `EnableCallbacks` again at `prepare()`. ### Reading stats plugins alongside file writing -A plugin scalar needs no data logic: a detector is a [](#StandardReadable), so register -the signal and it appears in `read()` alongside whatever the file writer produces. +A plugin in `plugins` is registered as a readable, so a detector with a stats plugin needs no +data logic to read it: the plugin declares which of its signals are read and hinted, and they +appear in `read()` alongside whatever the file writer produces. ```python from ophyd_async.core import StandardReadableFormat as Format +stats = adcore.NDStatsIO("PREFIX:STATS:") det = adaravis.AravisDetector( prefix, adcore.ADDataFactory.hdf(path_provider), plugins={"stats": stats} ) -# Add stats total as a hinted signal in events -det.set_readable_format(det.stats.total, Format.HINTED_UNCACHED_SIGNAL) +# Read a statistic the plugin does not read by default +det.set_readable_format(stats.sigma, Format.UNCACHED_SIGNAL) +``` + +Each statistic of an [](#adcore.NDStatsIO) has one name, that of its scalar signal (`det-stats-total` here), however it is published: + +- as the scalar, read once per event, when nothing else serves it +- as a stats time series ([](#adcore.ADDataFactory.time_series)), for a detector that writes no file +- as an NDAttribute in the HDF file, for each plugin listed in the `ndattribute_sources` of [](#adcore.ADDataFactory.hdf) + +The time series and the HDF writer take the plugin by reference, so make it first and put it in `plugins`, else constructing the detector raises `ValueError` naming it. Every plugin listed in `ndattribute_sources` must sit upstream of the HDF writer in the areaDetector plugin chain, or the writer never sees its attributes. Only the factories of file writers make a plugin of their own. + +```python +stats = adcore.NDStatsIO("PREFIX:STATS:") +det = adaravis.AravisDetector( + prefix, + adcore.ADDataFactory.hdf(path_provider, ndattribute_sources=[stats]), + adcore.ADDataFactory.time_series(stats), + plugins={"stats": stats}, +) +``` + +The highest of these that serves a scan replaces the others, and the plugin's hints stay. Which statistics are used is set on the plugin, not in Python: a statistic is used if the plugin's `EnableCallbacks` and its group's compute switch (`compute_statistics` or `compute_centroid`) are on at the first `prepare()` after `stage()`, and it has a read format on the plugin. A scalar whose statistic is off is left out of `read()`, `describe()` and `hints`. + +The first `prepare()` after `stage()` asks each plugin what it publishes and hides the read signals it does not, and each later `prepare()` raises if a plugin's answer has changed since, so set the switches that decide it before staging, for example in a settings file. The `nd_array_port` of a plugin and the chain of plugins that feeds it are not checked: how they are wired at `stage()` is up to you, except that `stage()` raises unless the circular buffer of a continuously acquiring detector reads directly from the driver. The circular buffer of a continuously acquiring detector is switched on by its acquire logic at `stage()`. The process plugin of a detector with a `proc_suffix` is held by its trigger logic, and `prepare_filter` switches it on. An [](#adcore.NDROIStatIO) works the same way, with the `use` and `Name` of each channel deciding what is published: its channels are readable children of the plugin, named `-` at `stage()`, so a channel called Blue has the total `det-roistat-Blue-total`. A `Name` changed after that takes effect at the next `stage()`. Its time series ([](#adcore.ADDataFactory.time_series)) works like the stats time series, publishing each used channel's statistics under the same names. + +```python +roistat = adcore.NDROIStatIO("PREFIX:ROISTAT:", num_channels=3) +det = adaravis.AravisDetector( + prefix, + adcore.ADDataFactory.time_series(roistat), + plugins={"roistat": roistat}, +) ``` ### Multiple HDF writers for different ROIs diff --git a/docs/tutorials/implementing-detectors.md b/docs/tutorials/implementing-detectors.md index 0236f05850..482a1ca0d3 100644 --- a/docs/tutorials/implementing-detectors.md +++ b/docs/tutorials/implementing-detectors.md @@ -240,7 +240,7 @@ The [](#StreamableDataProvider) returned from `start_provider()` is made for the The detector starts each of its logics for the scan, and the type of provider that comes back says which kind of data it is. - a [](#StreamableDataProvider) publishes stream assets, like a file writer. The detector exposes `collect_asset_docs`, and `BlobDataLogic` is an example. -- a [](#PageableDataProvider) publishes event pages, like an areaDetector stats time series or a scaler's arrays, so the detector exposes `collect_pages` instead. Its `make_page()` carries on from where its last page got to; a step scan's `read()` takes the last event of the page made up to the end of the point. +- a [](#PageableDataProvider) publishes event pages, like an areaDetector stats time series or a scaler's arrays, so the detector exposes `collect_pages` instead. Its `make_page()` carries on from where its last page got to; a step scan's `read()` takes the last event of the page made up to the end of the point. See [](#TimeSeriesDataLogic). - `None` means "not this stage", for instance a plugin that is switched off. The logic decides this without writing anything. This is not an error: a detector may carry more data logics than any one scan uses. Separately from that, a data logic is either a [](#BoundedDataLogic) or an [](#UnboundedDataLogic), whichever it inherits from (not both). An unbounded logic, like a file writer, keeps counting for as long as it is told to; its collections-written signal is read when each `prepare()` finishes. A bounded logic needs to be told how many collections to take, like a finite buffer: it is started after the unbounded ones and handed the datakeys their providers already make, and its provider must make none of them. It returns `None` if nothing is left to make, and raises for a scan with no fixed number of collections (`num_collections` of 0) if something is. At each point of a step scan after the first it is stopped and started again, so the buffer is cleared. Either kind may return either kind of provider. diff --git a/src/ophyd_async/epics/adandor.py b/src/ophyd_async/epics/adandor.py index 02dbbd0173..03cc7cdb36 100644 --- a/src/ophyd_async/epics/adandor.py +++ b/src/ophyd_async/epics/adandor.py @@ -96,8 +96,8 @@ class AndorDetector(AreaDetector[Andor2DriverIO]): :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/adaravis.py b/src/ophyd_async/epics/adaravis.py index ed215a1da8..e52a23f5ce 100644 --- a/src/ophyd_async/epics/adaravis.py +++ b/src/ophyd_async/epics/adaravis.py @@ -101,8 +101,8 @@ class AravisDetector(AreaDetector[AravisDriverIO]): If provided, this value is used for deadtime instead of looking up based on camera model. :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/adcore/__init__.py b/src/ophyd_async/epics/adcore/__init__.py index 5c2a17cfea..a0067fc7d6 100644 --- a/src/ophyd_async/epics/adcore/__init__.py +++ b/src/ophyd_async/epics/adcore/__init__.py @@ -10,6 +10,7 @@ ADMultipartDataLogic, ADWriterFactory, NDArrayDescription, + TimeSeriesDataLogic, ) from ._detector import AreaDetector, ContAcqDetector, plugins_with_proc from ._io import ( @@ -32,6 +33,7 @@ NDFileHDF5Compression, NDFileHDF5IO, NDFileIO, + NDParam, NDPluginBaseIO, NDPluginFileIO, NDProcessFilterCallbacks, @@ -39,7 +41,10 @@ NDProcessIO, NDROIStatIO, NDROIStatNIO, + NDROIStatTSControl, NDStatsIO, + NDStatsTimeSeriesIO, + NDStatsTSAcquireMode, ) from ._ndattribute import ( NDAttributeDataType, @@ -48,7 +53,7 @@ NDAttributePvDbrType, ndattributes_to_xml, ) -from ._plan_stubs import setup_ndattributes, setup_ndstats_sum +from ._plan_stubs import setup_ndattributes from ._trigger_logic import ( ADContAcqTriggerLogic, prepare_exposures, @@ -66,9 +71,13 @@ "ADBaseIO", "NDPluginBaseIO", "NDROIIO", + "NDParam", "NDStatsIO", + "NDStatsTimeSeriesIO", + "NDStatsTSAcquireMode", "NDROIStatNIO", "NDROIStatIO", + "NDROIStatTSControl", "NDCBFlushOnSoftTrgMode", "NDCircularBuffIO", "ADFileWriteMode", @@ -99,6 +108,7 @@ "ADMultipartDataLogic", "ADDataFactory", "ADWriterFactory", + "TimeSeriesDataLogic", # Detector "AreaDetector", "ContAcqDetector", @@ -110,5 +120,4 @@ "NDAttributeParam", "ndattributes_to_xml", "setup_ndattributes", - "setup_ndstats_sum", ] diff --git a/src/ophyd_async/epics/adcore/_acquire_logic.py b/src/ophyd_async/epics/adcore/_acquire_logic.py index 83180bad15..fe64abe523 100644 --- a/src/ophyd_async/epics/adcore/_acquire_logic.py +++ b/src/ophyd_async/epics/adcore/_acquire_logic.py @@ -27,10 +27,8 @@ class ADAcquireLogic(DetectorAcquireLogic): `AcquireBusy` wait for the number of queued arrays to reach zero, and `acquire` waits for the `acquire` put callback, so turning it on is what makes a step scan read plugin scalars for the frame it just - took rather than the one before. Defaults to on, since racing is - silent and the wait costs nothing when a file writer is the slowest - thing in the chain anyway. Turn it off for a detector whose plugin - chain must not gate acquisition. + took rather than the one before. Turn it off for a detector whose + plugin chain must not gate acquisition. :param timeout: How long to wait for the driver to arm and to return to idle. """ @@ -51,10 +49,7 @@ def __init__( async def ensure_ready(self): # Stop first: WaitForPlugins is latched and has no _RBV, so set it once - # per scan while the driver is known to be idle rather than on the - # latency path before every arm. Always written, never left at whatever - # the IOC happened to boot with, so a scan does not depend on who - # touched the PV last. + # per scan while the driver is idle. Always written. await self.ensure_stopped() await self.driver.wait_for_plugins.set(self.wait_for_plugins) @@ -85,11 +80,8 @@ class ADContAcqAcquireLogic(DetectorAcquireLogic): The CB plugin must read directly from the driver, so `stage()` raises if its `NDArrayPort` is not the driver's port. - Deliberately has no `wait_for_plugins` equivalent to [](#ADAcquireLogic). - That setting gates the driver's `AcquireBusy` going to 0 at the end of an - acquisition, and here the driver is always acquiring, so there is no such - transition to wait on. `acquire` waits on the circular buffer's trigger - instead. + There is no `wait_for_plugins` as in [](#ADAcquireLogic): the driver is + always acquiring, so `acquire` waits on the circular buffer's trigger. :param driver: The continuously acquiring detector driver :param cb_plugin: The circular buffer plugin gating downstream plugins diff --git a/src/ophyd_async/epics/adcore/_data_logic.py b/src/ophyd_async/epics/adcore/_data_logic.py index 97c76cff41..264e68e003 100644 --- a/src/ophyd_async/epics/adcore/_data_logic.py +++ b/src/ophyd_async/epics/adcore/_data_logic.py @@ -1,26 +1,32 @@ import asyncio +import math import warnings -from collections.abc import Awaitable, Callable, Sequence +from collections.abc import Awaitable, Callable, Iterable, Iterator, Sequence from dataclasses import dataclass from pathlib import PureWindowsPath -from typing import Any, Generic +from typing import Any from xml.etree import ElementTree as ET import numpy as np from ophyd_async.core import ( DEFAULT_TIMEOUT, + Array1D, BoundedDataLogic, + Device, EnableDisable, + EventPageDataProvider, PathInfo, PathProvider, SignalR, + StandardReadable, StreamResourceDataProvider, StreamResourceInfo, UnboundedDataLogic, set_and_wait_for_value, wait_for_value, ) +from ophyd_async.core import StandardReadableFormat as Format from ophyd_async.epics.core import stop_busy_record from ._io import ( @@ -30,12 +36,20 @@ ADFileWriteMode, NDArrayBaseIO, NDFileHDF5IO, + NDParam, NDPluginBaseIO, - NDPluginBaseIOT, NDPluginFileIO, - plugin_is_enabled, + NDROIStatIO, + NDROIStatTSControl, + NDStatsIO, + NDStatsTSAcquireMode, +) +from ._ndattribute import ( + NDAttributeDataType, + NDAttributeParam, + NDAttributePvDbrType, + ndattributes_to_xml, ) -from ._ndattribute import NDAttributeDataType, NDAttributePvDbrType @dataclass @@ -90,31 +104,139 @@ async def get_ndarray_resource_info( async def get_ndattribute_dtype_source( elements: Sequence[NDArrayBaseIO], + sources: Sequence[NDPluginBaseIO] = (), ) -> dict[str, tuple[str, str]]: - nd_attribute_xmls = await asyncio.gather( - *[x.nd_attributes_file.get_value() for x in elements] + """The dtype and source of each NDAttribute stamped on a frame. + + :param elements: parse the NDAttributes in the `nd_attributes_file` of each + :param sources: write the NDAttributes of each to its `nd_attributes_file` + """ + per_element = await asyncio.gather( + *(_parse_ndattribute_dtype_source(element) for element in elements), + *(_write_ndattribute_dtype_source(source) for source in sources), ) + return {name: value for found in per_element for name, value in found.items()} + + +def _check_in_plugins( + used: Sequence[NDPluginBaseIO], plugins: Sequence[NDPluginBaseIO] +) -> None: + """Raise if a plugin taken by reference is not in the detector's `plugins`. + + :param used: the plugins a data logic takes by reference + :param plugins: the detector's plugins, whose children also count + """ + in_plugins = list(_device_tree(plugins)) + for plugin in used: + if not any(plugin is device for device in in_plugins): + raise ValueError( + f"{plugin.name or type(plugin).__name__} is not in the detector's " + "plugins: add it to `plugins=`" + ) + + +def _device_tree(devices: Iterable[Device]) -> Iterator[Device]: + """The devices, and the children of each, down the Device tree.""" + for device in devices: + yield device + yield from _device_tree(child for _, child in device.children()) + + +# The formats that put a signal in read(), rather than in configuration +_READ_FORMATS = ( + Format.HINTED_SIGNAL, + Format.UNCACHED_SIGNAL, + Format.HINTED_UNCACHED_SIGNAL, +) + + +def _read_signals(device: Device) -> Iterator[SignalR]: + """The signals a device registers to be read, down its readable children.""" + if isinstance(device, StandardReadable): + for entry in device.get_readable_formats(): + match entry: + case (child, Format.CHILD): + yield from _read_signals(child) + case (signal, format) if format in _READ_FORMATS: + yield signal + case _: + pass + + +async def published_params(plugin: NDPluginBaseIO) -> list[NDParam]: + """The enabled params of an enabled plugin whose signal it registers to be read. + + These are what `ADHDFDataLogic` writes as NDAttributes and the time series + data logic publishes. + """ + enabled, params = await asyncio.gather( + plugin_is_enabled(plugin), plugin.get_enabled_params() + ) + read = set(_read_signals(plugin)) + return [param for param in params or () if enabled and param.signal in read] + + +async def plugin_is_enabled(plugin: NDPluginBaseIO) -> bool: + """Whether a plugin is switched on, so is receiving NDArrays. + + The nearest thing to "is this plugin in the chain": a plugin with callbacks + disabled certainly is not, though an enabled one may still have its + `nd_array_port` pointed somewhere else. + """ + return await plugin.enable_callbacks.get_value() is EnableDisable.ENABLE + + +async def _write_ndattribute_dtype_source( + source: NDPluginBaseIO, +) -> dict[str, tuple[str, str]]: + ndattributes = [ + NDAttributeParam( + name=param.signal.name, + param=param.param, + datatype=NDAttributeDataType.DOUBLE, + addr=param.addr, + ) + for param in await published_params(source) + ] + xml = ndattributes_to_xml(ndattributes) + capacity = await source.nd_attributes_file_nelm.get_value() + if len(xml) > capacity: + raise ValueError( + f"The NDAttributes XML for {source.nd_attributes_file.source} is " + f"{len(xml)} characters but the waveform holds {capacity}: " + f"raise XMLSIZE for this plugin in the IOC" + ) + await source.nd_attributes_file.set(xml) + return { + ndattribute.name: (ndattribute.datatype.value, "") + for ndattribute in ndattributes + } + + +async def _parse_ndattribute_dtype_source( + element: NDArrayBaseIO, +) -> dict[str, tuple[str, str]]: + maybe_xml = await element.nd_attributes_file.get_value() ndattribute_dtypes: dict[str, tuple[str, str]] = {} - for maybe_xml in nd_attribute_xmls: - # This is the check that ADCore does to see if it is an XML string - # rather than a filename to parse - if "" in maybe_xml: - root = ET.fromstring(maybe_xml) - for child in root: - if child.attrib.get("type", "EPICS_PV") == "EPICS_PV": - dbrtype = child.attrib.get("dbrtype", "DBR_NATIVE") - if dbrtype == "DBR_NATIVE": - raise RuntimeError( - f"NDAttribute {child.attrib['name']} has dbrtype " - "DBR_NATIVE, which is not supported" - ) - dtype_numpy = NDAttributePvDbrType[dbrtype].value - source = "ca://" + child.attrib["source"] - else: - datatype = child.attrib.get("datatype", "INT") - dtype_numpy = NDAttributeDataType[datatype].value - source = "" - ndattribute_dtypes[child.attrib["name"]] = (dtype_numpy, source) + # This is the check that ADCore does to see if it is an XML string + # rather than a filename to parse + if "" in maybe_xml: + root = ET.fromstring(maybe_xml) + for child in root: + if child.attrib.get("type", "EPICS_PV") == "EPICS_PV": + dbrtype = child.attrib.get("dbrtype", "DBR_NATIVE") + if dbrtype == "DBR_NATIVE": + raise RuntimeError( + f"NDAttribute {child.attrib['name']} has dbrtype " + "DBR_NATIVE, which is not supported" + ) + dtype_numpy = NDAttributePvDbrType[dbrtype].value + source = "ca://" + child.attrib["source"] + else: + datatype = child.attrib.get("datatype", "INT") + dtype_numpy = NDAttributeDataType[datatype].value + source = "" + ndattribute_dtypes[child.attrib["name"]] = (dtype_numpy, source) return ndattribute_dtypes @@ -148,6 +270,28 @@ async def prepare_file_paths( await writer.num_capture.set(0) +async def _ensure_writer_ready(writer: NDPluginFileIO, always_enable: bool) -> None: + """Stop the writer, and switch it on if `always_enable`. Called on stage().""" + await stop_busy_record(writer.capture) + if always_enable: + await writer.enable_callbacks.set(EnableDisable.ENABLE) + + +# The largest HDF chunk a flush period may size. TODO: check with reviewers that +# 4 MiB suits GPFS, and whether it should be a parameter +_MAX_CHUNK_BYTES = 4 * 1024 * 1024 + + +def _frames_per_flush( + flush_period: float, collection_period: float, frame: StreamResourceInfo +) -> int: + """The frames a flush period holds, capped so a chunk is `_MAX_CHUNK_BYTES`.""" + # Made with one frame per chunk, so the chunk shape is the frame's + frame_bytes = math.prod(frame.chunk_shape) * np.dtype(frame.dtype_numpy).itemsize + wanted = round(flush_period / collection_period) + return max(1, min(wanted, _MAX_CHUNK_BYTES // frame_bytes)) + + async def _start_capture(writer: NDPluginFileIO) -> None: await set_and_wait_for_value(writer.capture, True, wait_for_set_completion=False) # Capturing resets NumCaptured, which the detector reads as its baseline @@ -162,7 +306,11 @@ class ADHDFDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): :param path_provider: Callable that provides path information for file writing. :param writer: The NDFileHDFIO plugin instance. :param driver: The AreaDetector driver instance to extract NDAttributes from. - :param plugins: Additional NDPluginBaseIO instances to extract NDAttributes from. + :param plugins: Every plugin of the detector, including those it creates, + to parse NDAttributes from, in the XML the user set on each. + :param ndattribute_sources: Plugins to write the NDAttributes of, named after + what they publish. A plugin in `plugins` that is also listed here is + written, not parsed. :param datakey_suffix: Suffix to append to the data key for the main dataset :param hinted: Include the image in hints. Set False for BestEffortCallback. """ @@ -172,12 +320,17 @@ class ADHDFDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): writer: NDFileHDF5IO driver: NDArrayBaseIO | None = None plugins: Sequence[NDPluginBaseIO] = () + ndattribute_sources: Sequence[NDPluginBaseIO] = () datakey_suffix: str = "" hinted: bool = True - #: False sits out the scans of a stage if the plugin's EnableCallbacks PV is - #: off at the first prepare. True serves every scan. Starting switches it on. + #: False sits out the scans of a stage if the plugin is off at the first + #: prepare() of the stage. True switches the plugin on (EnableCallbacks) on + #: stage(), so it serves every scan. always_enable: bool = False + async def ensure_ready(self) -> None: + await _ensure_writer_ready(self.writer, self.always_enable) + async def start_provider( self, name: str, @@ -189,25 +342,31 @@ async def start_provider( return None datakey_name = name + self.datakey_suffix path_info = self.path_provider(datakey_name) - # Size the HDF chunk to hold a flush period of collections, e.g. 400 Hz - # (2.5 ms) with a 0.5 s flush period gives 200 frames per chunk (#1309). - # Otherwise use the chunk size set on the IOC, forcing a fresh-startup 0 - # to 1. - if flush_period > 0 and collection_period > 0: - frames_per_chunk = max(1, round(flush_period / collection_period)) - else: - frames_per_chunk = await self.writer.num_frames_chunks.get_value() or 1 + elements = (self.driver, *self.plugins) if self.driver else (*self.plugins,) + parsed = [ + element + for element in elements + if not any(element is source for source in self.ndattribute_sources) + ] main_dataset, ndattribute_dtype_sources = await asyncio.gather( get_ndarray_resource_info( array_description=self.array_description, data_key=datakey_name, parameters={"dataset": "/entry/data/data"}, - frames_per_chunk=frames_per_chunk, - ), - get_ndattribute_dtype_source( - (self.driver, *self.plugins) if self.driver else (*self.plugins,) ), + get_ndattribute_dtype_source(parsed, self.ndattribute_sources), ) + # Size the HDF chunk to hold a flush period of collections, e.g. 400 Hz + # (2.5 ms) with a 0.5 s flush period gives 200 frames per chunk (#1309). + # Otherwise use the chunk size set on the IOC, forcing a fresh-startup 0 + # to 1. + if flush_period > 0 and collection_period > 0: + frames_per_chunk = _frames_per_flush( + flush_period, collection_period, main_dataset + ) + else: + frames_per_chunk = await self.writer.num_frames_chunks.get_value() or 1 + main_dataset.chunk_shape = (frames_per_chunk, *main_dataset.chunk_shape[1:]) ndattribute_datasets = [ StreamResourceInfo( data_key=attr_name, @@ -234,7 +393,6 @@ async def start_provider( prepare_file_paths( path_info=path_info, file_template="%s%s.h5", writer=self.writer ), - self.writer.enable_callbacks.set(EnableDisable.ENABLE), ] await asyncio.gather(*coros) await _start_capture(self.writer) @@ -267,9 +425,9 @@ class ADMultipartDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): :param datakey_suffix: Suffix to append to the data key for the main dataset :param hinted: Include the image in hints. Set False for BestEffortCallback. :param always_enable: - False sits out the scans of a stage if the plugin's EnableCallbacks PV - is off at the first prepare. True serves every scan. Starting switches - it on. + False sits out the scans of a stage if the plugin is off at the first + prepare() of the stage. True switches the plugin on (EnableCallbacks) on + stage(), so it serves every scan. """ array_description: NDArrayDescription @@ -281,6 +439,9 @@ class ADMultipartDataLogic(UnboundedDataLogic[StreamResourceDataProvider]): hinted: bool = True always_enable: bool = False + async def ensure_ready(self) -> None: + await _ensure_writer_ready(self.writer, self.always_enable) + async def start_provider( self, name: str, @@ -307,7 +468,6 @@ async def start_provider( file_template="%s%s_%6.6d" + self.extension, writer=self.writer, ), - self.writer.enable_callbacks.set(EnableDisable.ENABLE), ) await _start_capture(self.writer) return StreamResourceDataProvider( @@ -325,79 +485,202 @@ def get_hinted_fields(self, name: str) -> Sequence[str]: return [name + self.datakey_suffix] if self.hinted else [] -@dataclass -class ADDataFactory(Generic[NDPluginBaseIOT]): - """Factory that creates a plugin and its matching data logic. - - Construct using the static methods `hdf`, `jpeg` or `tiff`, then pass one - or more instances to `AreaDetector` as positional `*writer_factories` - arguments. When the detector is initialised - `__call__` is invoked with the detector's PV `prefix`, its `driver`, and the - flat list of extra `plugins`; it returns the writer device and the - corresponding data logic. - - :param writer_cls: Concrete `NDPluginBaseIO` subclass to instantiate. - :param writer_suffix: PV suffix appended to *prefix* to form the writer's PV prefix. - :param writer_name: - Attribute name under which the writer device is stored on the - `AreaDetector` instance. Each static method defaults to its own name - (``"hdf"``, ``"jpeg"``, ``"tiff"``); override when passing multiple - factories so each writer gets a distinct name. - :param datakey_suffix: Suffix appended to the datakey name in stream resources. - :param array_description: - Override the array shape/type description built from the driver. - May be an `NDArrayDescription` instance or a callable - ``(driver) → NDArrayDescription``; use a callable when the description - depends on signals that are only available once the driver has been - constructed (e.g. inside a detector subclass that creates its own driver). - :param data_logic_factory: - Callable `(writer, array_description, driver, plugins) → data logic` - that builds the data logic given the already-constructed writer. +async def _check_capacity(capacity: SignalR[int], num_collections: int) -> None: + """Raise if the time series arrays hold fewer points than the scan needs.""" + points = await capacity.get_value() + if points < num_collections: + raise ValueError( + f"{capacity.source}: the scan needs {num_collections} points, " + f"but the plugin's arrays hold {points}" + ) + + +class TimeSeriesDataLogic(BoundedDataLogic[EventPageDataProvider]): + """Data logic publishing the time series of a stats or ROI stat plugin. + + For detectors that write no file: the plugin holds each statistic in a + fixed-length buffer that must be sized before acquisition, so this logic + sizes it from the scan's number of collections. The plugin has one time-series + control shared across many arrays, so one logic covers every array. + + It publishes the time series of each of the plugin's `published_params` + under the name of its scalar signal, so the plugin's switches and read + formats decide what is published, and this logic never writes them. + It hints nothing: a hinted scalar of a plugin registered as a readable stays + in the hints while its series replaces it. + + The buffer is sized and armed in `start_provider`, which the detector calls + again at each step point after the first. Arming erases the arrays and resets + the current point before the detector's acquire logic drives the camera; its + frames then feed the time series via NDArray callbacks. `start_provider` + raises `ValueError` if the plugin's arrays hold fewer points than the scan + needs. Before each page the plugin is asked to post its arrays up to the + current point. A detector that writes a file should pull stats into the file + as NDAttributes instead (see `ADHDFDataLogic`). + + For an `NDStatsIO`, arming writes the fixed-length mode and then 1 to + `ts_acquire`, and `ts_read` posts the arrays. For an `NDROIStatIO` it + writes ERASE_START to `ts_control`, the plugin stops acquiring by itself + when the buffer is full, and READ posts the arrays. Each ROI channel has its + own timestamp array, and that of the first channel published is used. + + :param plugin: the stats or ROI stat plugin whose time series to read """ - writer_cls: type[NDPluginBaseIOT] - writer_suffix: str - writer_name: str - datakey_suffix: str - array_description: ( - NDArrayDescription | Callable[[ADBaseIO], NDArrayDescription] | None - ) - data_logic_factory: Callable[ - [NDPluginBaseIOT, NDArrayDescription, ADBaseIO, Sequence[NDPluginBaseIO]], - BoundedDataLogic | UnboundedDataLogic, - ] + def __init__(self, plugin: NDStatsIO | NDROIStatIO) -> None: + self.plugin = plugin + match plugin: + case NDStatsIO(): + self._logic = _StatsTimeSeriesLogic(plugin) + case NDROIStatIO(): + self._logic = _ROIStatTimeSeriesLogic(plugin) - def __call__( + async def start_provider( self, - prefix: str, - driver: ADBaseIO, - plugins: Sequence[NDPluginBaseIO], - ) -> tuple[NDPluginBaseIOT, BoundedDataLogic | UnboundedDataLogic]: - """Instantiate the writer plugin and build the data logic. - - :param prefix: EPICS PV prefix for the detector (same as `AreaDetector.prefix`). - :param driver: The detector driver, used to read array shape/type metadata. - :param plugins: Additional plugins whose NDAttribute XML files should be - included in HDF5 metadata. - :return: ``(writer, data_logic)`` tuple ready to attach to the detector. - """ - writer = self.writer_cls(prefix + self.writer_suffix) - if callable(self.array_description): - array_description = self.array_description(driver) - elif self.array_description is not None: - array_description = self.array_description - else: - array_description = NDArrayDescription( - shape_signals=[ - driver.array_size_z, - driver.array_size_y, - driver.array_size_x, - ], - data_type_signal=driver.data_type, - color_mode_signal=driver.color_mode, - ) - data_logic = self.data_logic_factory(writer, array_description, driver, plugins) - return writer, data_logic + name: str, + existing_datakeys: Sequence[str], + num_collections: int, + collections_per_event: int, + collection_period: float, + flush_period: float, + ) -> EventPageDataProvider | None: + # The buffer is sized by count, not rate, so neither period is used + params = await published_params(self.plugin) + arrays = { + param.signal.name: param.ts_signal + for param in params + if param.ts_signal is not None + and param.signal.name not in existing_datakeys + } + if not arrays: + return None + timestamp, capacity = self._logic.timestamp_and_capacity(params) + await _check_capacity(capacity, num_collections) + await self._logic.arm(num_collections) + # The IOC resets the count within the put, but its monitor may arrive + # after the put's readback, and the detector reads the count next + await wait_for_value(self.plugin.ts_current_point, 0, timeout=DEFAULT_TIMEOUT) + return EventPageDataProvider( + arrays, + timestamp, + self.plugin.ts_current_point, + collections_per_event, + self._logic.post_arrays, + ) + + async def ensure_stopped(self) -> None: + await self._logic.stop() + + +@dataclass(frozen=True) +class _StatsTimeSeriesLogic: + """How an `NDStatsIO` arms, posts and stops its time series. + + Arming also enables the embedded time series plugin, which the IOC starts + disabled. + """ + + stats: NDStatsIO + + def timestamp_and_capacity( + self, params: Sequence[NDParam] + ) -> tuple[SignalR[Array1D[np.float64]], SignalR[int]]: + return self.stats.ts_timestamp, self.stats.ts_timestamp_nelm + + async def arm(self, num_collections: int) -> None: + """Size the buffer, then erase the arrays and start filling them.""" + # Fixed-length mode before arming. Writing 1 to ts_acquire clears the + # arrays and resets ts_current_point to 0. The put completes when the + # buffer is full, so do not wait for it. + # The embedded time series plugin ignores the stats until it is enabled + await asyncio.gather( + self.stats.ts_plugin.enable_callbacks.set(EnableDisable.ENABLE), + self.stats.ts_num_points.set(num_collections), + self.stats.ts_acquire_mode.set(NDStatsTSAcquireMode.FIXED_LENGTH), + ) + await set_and_wait_for_value( + self.stats.ts_acquire, True, wait_for_set_completion=False + ) + + async def post_arrays(self) -> None: + """Ask the plugin to post its arrays, which it does about once a second.""" + await self.stats.ts_read.set(True) + + async def stop(self) -> None: + await stop_busy_record(self.stats.ts_acquire) + + +@dataclass(frozen=True) +class _ROIStatTimeSeriesLogic: + """How an `NDROIStatIO` arms, posts and stops its time series.""" + + roistat: NDROIStatIO + + def timestamp_and_capacity( + self, params: Sequence[NDParam] + ) -> tuple[SignalR[Array1D[np.float64]], SignalR[int]]: + # Each channel has its own timestamp array: use the first published + first_channel = self.roistat.channels[params[0].addr + 1] + return first_channel.ts_timestamp, first_channel.ts_timestamp_nelm + + async def arm(self, num_collections: int) -> None: + """Size the buffer, then erase the arrays and start filling them.""" + # The plugin keeps the current point when resized, so size it first and + # then erase and start, which resets the current point to 0 + await self.roistat.ts_num_points.set(num_collections) + await self.roistat.ts_control.set(NDROIStatTSControl.ERASE_START) + + async def post_arrays(self) -> None: + """Ask the plugin to post its arrays, which it does about once a second.""" + await self.roistat.ts_control.set(NDROIStatTSControl.READ) + + async def stop(self) -> None: + await self.roistat.ts_control.set(NDROIStatTSControl.STOP) + await wait_for_value(self.roistat.ts_acquiring, False, timeout=DEFAULT_TIMEOUT) + + +def _array_description( + driver: ADBaseIO, + override: NDArrayDescription | Callable[[ADBaseIO], NDArrayDescription] | None, +) -> NDArrayDescription: + """The `override`, or the shape and type read from the `driver`.""" + if callable(override): + description = override(driver) + elif override is not None: + description = override + else: + description = NDArrayDescription( + shape_signals=[ + driver.array_size_z, + driver.array_size_y, + driver.array_size_x, + ], + data_type_signal=driver.data_type, + color_mode_signal=driver.color_mode, + ) + return description + + +@dataclass(kw_only=True) +class ADDataFactory: + """Factory that builds a data logic, and the plugin it needs if any. + + Construct using the static methods `hdf`, `jpeg`, `tiff` or `time_series`, + then pass one or more instances to `AreaDetector` as positional + `*writer_factories` arguments, which calls `build` when it is initialised. + + :param build: + Callable `(prefix, driver, plugins) → (named_plugin, data_logic)`, + given the detector's PV prefix, its driver and every plugin it has, + those the user passes and those it creates, where + `named_plugin` is the attribute name and plugin the logic needs, or + `None` if the logic takes the plugins it uses by reference. + """ + + build: Callable[ + [str, ADBaseIO, Sequence[NDPluginBaseIO]], + tuple[tuple[str, NDPluginBaseIO] | None, BoundedDataLogic | UnboundedDataLogic], + ] @staticmethod def hdf( @@ -410,7 +693,8 @@ def hdf( | None = None, hinted: bool = True, always_enable: bool = False, - ) -> "ADDataFactory[NDFileHDF5IO]": + ndattribute_sources: Sequence[NDPluginBaseIO] = (), + ) -> "ADDataFactory": """Create a factory for an HDF5 file writer. :param path_provider: Provides file path information for each acquisition. @@ -427,27 +711,35 @@ def hdf( (e.g. an ROI plugin). :param hinted: Include the image in hints. Set False for BestEffortCallback. :param always_enable: False (the default) sits out the scans of a stage - if the plugin's EnableCallbacks PV is off at the first prepare, so - another data logic can serve them. True serves every scan. Starting - switches the plugin on either way. + if the plugin's EnableCallbacks PV is off at the first prepare() of + the stage, so another data logic can serve them. True switches the + plugin on at stage, so it serves every scan. + :param ndattribute_sources: The plugins whose NDAttributes this writer + writes into the file, as datasets named after what each publishes, + e.g. `det-stats-total`. Each must be in the detector's `plugins`, + else `ValueError` is raised when the detector is made. The + NDAttributes of the driver and of the detector's plugins that are + not listed are parsed from their `nd_attributes_file` as the user + set it. """ - return ADDataFactory( - writer_cls=NDFileHDF5IO, - writer_suffix=writer_suffix, - writer_name=writer_name, - datakey_suffix=datakey_suffix, - array_description=array_description, - data_logic_factory=lambda writer, desc, driver, plugins: ADHDFDataLogic( - array_description=desc, + + def build(prefix, driver, plugins): + _check_in_plugins(ndattribute_sources, plugins) + writer = NDFileHDF5IO(prefix + writer_suffix) + data_logic = ADHDFDataLogic( + array_description=_array_description(driver, array_description), path_provider=path_provider, writer=writer, driver=driver, plugins=list(plugins), + ndattribute_sources=ndattribute_sources, datakey_suffix=datakey_suffix, hinted=hinted, always_enable=always_enable, - ), - ) + ) + return (writer_name, writer), data_logic + + return ADDataFactory(build=build) @staticmethod def jpeg( @@ -460,7 +752,7 @@ def jpeg( | None = None, hinted: bool = True, always_enable: bool = False, - ) -> "ADDataFactory[NDPluginFileIO]": + ) -> "ADDataFactory": """Create a factory for a JPEG file writer. :param path_provider: Provides file path information for each acquisition. @@ -476,29 +768,26 @@ def jpeg( when the shape/type comes from a plugin. :param hinted: Include the image in hints. Set False for BestEffortCallback. :param always_enable: False (the default) sits out the scans of a stage - if the plugin's EnableCallbacks PV is off at the first prepare, so - another data logic can serve them. True serves every scan. Starting - switches the plugin on either way. + if the plugin's EnableCallbacks PV is off at the first prepare() of + the stage, so another data logic can serve them. True switches the + plugin on at stage, so it serves every scan. """ - return ADDataFactory( - writer_cls=NDPluginFileIO, - writer_suffix=writer_suffix, - writer_name=writer_name, - datakey_suffix=datakey_suffix, - array_description=array_description, - data_logic_factory=lambda writer, desc, driver, plugins: ( - ADMultipartDataLogic( - array_description=desc, - path_provider=path_provider, - writer=writer, - extension=".jpg", - mimetype="multipart/related;type=image/jpeg", - datakey_suffix=datakey_suffix, - hinted=hinted, - always_enable=always_enable, - ) - ), - ) + + def build(prefix, driver, plugins): + writer = NDPluginFileIO(prefix + writer_suffix) + data_logic = ADMultipartDataLogic( + array_description=_array_description(driver, array_description), + path_provider=path_provider, + writer=writer, + extension=".jpg", + mimetype="multipart/related;type=image/jpeg", + datakey_suffix=datakey_suffix, + hinted=hinted, + always_enable=always_enable, + ) + return (writer_name, writer), data_logic + + return ADDataFactory(build=build) @staticmethod def tiff( @@ -511,7 +800,7 @@ def tiff( | None = None, hinted: bool = True, always_enable: bool = False, - ) -> "ADDataFactory[NDPluginFileIO]": + ) -> "ADDataFactory": """Create a factory for a TIFF file writer. :param path_provider: Provides file path information for each acquisition. @@ -527,49 +816,66 @@ def tiff( when the shape/type comes from a plugin. :param hinted: Include the image in hints. Set False for BestEffortCallback. :param always_enable: False (the default) sits out the scans of a stage - if the plugin's EnableCallbacks PV is off at the first prepare, so - another data logic can serve them. True serves every scan. Starting - switches the plugin on either way. + if the plugin's EnableCallbacks PV is off at the first prepare() of + the stage, so another data logic can serve them. True switches the + plugin on at stage, so it serves every scan. """ - return ADDataFactory( - writer_cls=NDPluginFileIO, - writer_suffix=writer_suffix, - writer_name=writer_name, - datakey_suffix=datakey_suffix, - array_description=array_description, - data_logic_factory=lambda writer, desc, driver, plugins: ( - ADMultipartDataLogic( - array_description=desc, - path_provider=path_provider, - writer=writer, - extension=".tiff", - mimetype="multipart/related;type=image/tiff", - datakey_suffix=datakey_suffix, - hinted=hinted, - always_enable=always_enable, - ) - ), - ) + def build(prefix, driver, plugins): + writer = NDPluginFileIO(prefix + writer_suffix) + data_logic = ADMultipartDataLogic( + array_description=_array_description(driver, array_description), + path_provider=path_provider, + writer=writer, + extension=".tiff", + mimetype="multipart/related;type=image/tiff", + datakey_suffix=datakey_suffix, + hinted=hinted, + always_enable=always_enable, + ) + return (writer_name, writer), data_logic -class ADWriterFactory(ADDataFactory[NDPluginBaseIOT]): - """Deprecated name of `ADDataFactory`.""" + return ADDataFactory(build=build) - def __post_init__(self) -> None: - _warn_writer_factory() + @staticmethod + def time_series(plugin: NDStatsIO | NDROIStatIO) -> "ADDataFactory": + """Create a factory for the time series of a stats or ROI stat plugin. + + Unlike `hdf`, `jpeg` and `tiff`, this makes no plugin and writes no file: + it reads the time series of the plugin you give it, a fixed-length buffer + read back as event pages, for detectors that cannot write assets (see + `TimeSeriesDataLogic`). Each series is named after its scalar signal, e.g. + `det-stats-total` or `det-roistat-Blue-total`, and the plugin's + switches decide which are published. + + :param plugin: The stats or ROI stat plugin, which must be in the + detector's `plugins`, else `ValueError` is raised when the detector + is made. The same plugin can be listed in + `hdf(ndattribute_sources=)`. + """ + + def build(prefix, driver, plugins): + _check_in_plugins([plugin], plugins) + return None, TimeSeriesDataLogic(plugin) + + return ADDataFactory(build=build) + + +class ADWriterFactory(ADDataFactory): + """Deprecated name of `ADDataFactory`.""" @staticmethod - def hdf(*args, **kwargs) -> ADDataFactory[NDFileHDF5IO]: + def hdf(*args, **kwargs) -> ADDataFactory: _warn_writer_factory() return ADDataFactory.hdf(*args, **kwargs) @staticmethod - def jpeg(*args, **kwargs) -> ADDataFactory[NDPluginFileIO]: + def jpeg(*args, **kwargs) -> ADDataFactory: _warn_writer_factory() return ADDataFactory.jpeg(*args, **kwargs) @staticmethod - def tiff(*args, **kwargs) -> ADDataFactory[NDPluginFileIO]: + def tiff(*args, **kwargs) -> ADDataFactory: _warn_writer_factory() return ADDataFactory.tiff(*args, **kwargs) diff --git a/src/ophyd_async/epics/adcore/_detector.py b/src/ophyd_async/epics/adcore/_detector.py index 5bf7759d3b..86ca9edd8f 100644 --- a/src/ophyd_async/epics/adcore/_detector.py +++ b/src/ophyd_async/epics/adcore/_detector.py @@ -1,4 +1,6 @@ -from collections.abc import Mapping, Sequence +import asyncio +from collections.abc import Callable, Iterator, Mapping, Sequence +from dataclasses import dataclass from functools import cached_property from typing import Generic @@ -6,15 +8,18 @@ BoundedDataLogic, DetectorAcquireLogic, DetectorLogic, + DetectorSuppressLogic, DetectorTriggerLogic, + ReadableEntry, SignalR, StandardDetector, + StandardReadable, UnboundedDataLogic, ) from ophyd_async.core import StandardReadableFormat as Format from ._acquire_logic import ADContAcqAcquireLogic -from ._data_logic import ADDataFactory +from ._data_logic import ADDataFactory, _read_signals, plugin_is_enabled from ._io import ( ADBaseIO, ADBaseIOT, @@ -26,7 +31,62 @@ from ._trigger_logic import ADContAcqTriggerLogic +def _readable_plugins( + entries: Iterator[ReadableEntry], +) -> Iterator[NDPluginBaseIO]: + """The plugins registered as CHILD readables, down the readable tree.""" + for entry in entries: + match entry: + case (StandardReadable() as child, Format.CHILD): + if isinstance(child, NDPluginBaseIO): + yield child + yield from _readable_plugins(child.get_readable_formats()) + case _: + pass + + +@dataclass(frozen=True) +class _PluginSuppressLogic(DetectorSuppressLogic): + """Hides the read signals of each readable plugin that its switches have off. + + :param get_readable_formats: the detector's, the top of the readable tree + that holds the plugins to ask + """ + + get_readable_formats: Callable[[], Iterator[ReadableEntry]] + + async def hidden_datakeys(self) -> set[str]: + plugins = set(_readable_plugins(self.get_readable_formats())) + hidden = await asyncio.gather(*(_hidden(plugin) for plugin in plugins)) + return set().union(*hidden) + + +async def _hidden(plugin: NDPluginBaseIO) -> set[str]: + """All the read signals of a disabled plugin, else those not enabled.""" + enabled, params = await asyncio.gather( + plugin_is_enabled(plugin), plugin.get_enabled_params() + ) + read = {signal.name for signal in _read_signals(plugin)} + if not enabled: + hidden = read + elif params is None: + hidden = set() + else: + hidden = read - {param.signal.name for param in params} + return hidden + + class AreaDetector(StandardDetector, Generic[ADBaseIOT]): + """An areaDetector driver, with its plugins and the data logics that serve it. + + Every plugin of the detector is an attribute and registered as a `CHILD` + readable, as is each writer a factory makes. The factories are given all the + plugins, to parse their NDAttributes. + + :param plugins: All the detector's plugins by attribute name, those the + caller passes and those the detector creates, such as its process plugin. + """ + def __init__( self, driver: ADBaseIOT, @@ -44,28 +104,34 @@ def __init__( self._set_unique(plugin_name, plugin) self._trigger_logic = trigger_logic self._acquire_logic = acquire_logic + plugin_list = list(plugins.values()) if plugins else [] + writers: list[NDPluginBaseIO] = [] data_logics: list[BoundedDataLogic | UnboundedDataLogic] = [] if writer_factories: if prefix is None: raise ValueError("prefix is required when writer_factories are given") - names = [f.writer_name for f in writer_factories] + made = [ + factory.build(prefix, driver, plugin_list) + for factory in writer_factories + ] + named_writers = [named for named, _ in made if named is not None] + names = [attr for attr, _ in named_writers] if len(names) != len(set(names)): duplicates = sorted({n for n in names if names.count(n) > 1}) raise ValueError( f"Duplicate writer_name(s) in writer_factories: {duplicates}" ) - plugin_list = list(plugins.values()) if plugins else [] - for factory in writer_factories: - writer, data_logic = factory(prefix, driver, plugin_list) - self._set_unique(factory.writer_name, writer) - data_logics.append(data_logic) + for attr, writer in named_writers: + self._set_unique(attr, writer) + writers.append(writer) + data_logics = [data_logic for _, data_logic in made] self._data_logics = tuple(data_logics) # The driver is always wired up, so register it whole and let ADBaseIO - # declare which of its signals are configuration. Plugins are - # deliberately *not* registered: a detector normally carries far more - # plugins than are wired into the chain, so opt in per plugin with - # `set_readable_format(det.stats, Format.CHILD)` instead. + # declare which of its signals are configuration. So is each of the + # plugins and writers, whose own formats declare what they read. self.set_readable_format(self.driver, Format.CHILD) + for plugin in (*plugin_list, *writers): + self.set_readable_format(plugin, Format.CHILD) for signal in config_sigs: self.set_readable_format(signal, Format.CONFIG_SIGNAL) super().__init__(name=name) @@ -84,6 +150,7 @@ def logic(self) -> DetectorLogic: trigger=self._trigger_logic, acquire=self._acquire_logic, data=self._data_logics, + suppress=_PluginSuppressLogic(self.get_readable_formats), ) def get_plugin( @@ -144,8 +211,8 @@ class ContAcqDetector(AreaDetector[ADBaseIOT]): :param num_frames_to_skip: Incoming frames to skip before each CB gate opening, defaults to 0 :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. So is the name cb_plugin_name. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. So is the name cb_plugin_name. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/adcore/_io.py b/src/ophyd_async/epics/adcore/_io.py index 38d3d7ef51..c5833ee0b4 100644 --- a/src/ophyd_async/epics/adcore/_io.py +++ b/src/ophyd_async/epics/adcore/_io.py @@ -1,16 +1,26 @@ +import asyncio +import re +from collections import Counter +from collections.abc import Sequence +from dataclasses import dataclass from typing import Annotated as A from typing import TypeVar +import numpy as np + from ophyd_async.core import ( + Array1D, DeviceVector, EnableDisable, SignalR, SignalRW, + SignalW, StandardReadable, StrictEnum, SubsetEnum, SupersetEnum, TriggerableCommand, + gather_dict, non_zero, ) from ophyd_async.core import StandardReadableFormat as Format @@ -60,6 +70,8 @@ class NDArrayBaseIO(EpicsDevice): port_name: A[SignalR[str], PvSuffix("PortName_RBV")] unique_id: A[SignalR[int], PvSuffix("UniqueId_RBV")] nd_attributes_file: A[SignalRW[str], PvSuffix("NDAttributesFile")] + # The capacity of the NDAttributesFile waveform (the IOC's XMLSIZE) + nd_attributes_file_nelm: A[SignalR[int], PvSuffix("NDAttributesFile.NELM")] # The array_size_x/y/z signals are used when outputting an NDArray # (i.e. from a driver) @@ -156,6 +168,22 @@ class ADBaseIO(StandardReadable, NDArrayBaseIO): NDPluginBaseIOT = TypeVar("NDPluginBaseIOT", bound="NDPluginBaseIO") +@dataclass(frozen=True) +class NDParam: + """A scalar an NDPlugin computes, in each form the plugin produces it. + + :param signal: the scalar value for the last frame, its name is the datakey + :param param: the asyn parameter, the source of an NDAttribute + :param addr: the asyn address of the parameter + :param ts_signal: the value for each frame of the plugin's time series, if any + """ + + signal: SignalR[float] + param: str + addr: int = 0 + ts_signal: SignalR[Array1D[np.float64]] | None = None + + class NDPluginBaseIO(NDArrayBaseIO): """Base class from which plugins are derived. @@ -168,16 +196,15 @@ class NDPluginBaseIO(NDArrayBaseIO): nd_array_address: A[SignalRW[int], PvSuffix.rbv("NDArrayAddress")] queue_size: A[SignalRW[int], PvSuffix.rbv("QueueSize")] + async def get_enabled_params(self) -> Sequence[NDParam] | None: + """The params this plugin's switches have on, when it is enabled. -async def plugin_is_enabled(plugin: NDPluginBaseIO) -> bool: - """Whether a plugin is switched on, so is receiving NDArrays. - - The nearest thing to "is this plugin in the chain": a plugin with callbacks - disabled certainly is not, though an enabled one may still have its - `nd_array_port` pointed somewhere else. Parsing the whole plugin graph was - rejected in ADR 0023 as too large. - """ - return await plugin.enable_callbacks.get_value() is EnableDisable.ENABLE + `None`, as here, means its switches gate none of its read signals. + Otherwise `AreaDetector`'s suppress logic hides the read signals not among + them, and `published_params` keeps those with a read format. Only a plugin + whose switches no logic writes may gate its read signals. + """ + return None class NDROIIO(StandardReadable, NDPluginBaseIO): @@ -198,6 +225,26 @@ class NDROIIO(StandardReadable, NDPluginBaseIO): bin_z: A[SignalRW[int], PvSuffix.rbv("BinZ")] +class NDStatsTimeSeriesIO(EpicsDevice): + """The NDPluginTimeSeries that NDPluginStats embeds, at the `TS:` sub-prefix. + + Its `enable_callbacks` is separate from the stats plugin's own. + """ + + enable_callbacks: A[SignalRW[EnableDisable], PvSuffix.rbv("EnableCallbacks")] + + +class NDStatsTSAcquireMode(StrictEnum): + """Whether an NDPluginStats time series stops when full or wraps around. + + Mirrors the `TSAcquireMode` record in + ADCore/db/NDPluginTimeSeries.template. + """ + + FIXED_LENGTH = "Fixed length" + CIRCULAR_BUFFER = "Circ. buffer" + + class NDStatsIO(StandardReadable, NDPluginBaseIO): """Plugin for computing statistics from an image or ROI within an image. @@ -226,6 +273,43 @@ class NDStatsIO(StandardReadable, NDPluginBaseIO): sigma: A[SignalR[float], PvSuffix("Sigma_RBV")] total: A[SignalR[float], PvSuffix("Total_RBV"), Format.HINTED_UNCACHED_SIGNAL] net: A[SignalR[float], PvSuffix("Net_RBV")] + # Time series. NDStats delegates its time series to an embedded + # NDPluginTimeSeries instance, so the controls live under an inner "TS:" + # prefix while the per-statistic arrays are named by NDStats itself at the + # plugin's own prefix. ts_num_points sizes the buffer, ts_acquire erases and + # starts it (writing 1 clears the arrays and resets ts_current_point to 0), + # ts_current_point reports progress, writing 1 to ts_read posts the arrays + # while acquiring, each ts_ holds the series of that statistic and + # ts_timestamp holds the per-point acquisition times. The embedded plugin + # takes no points while ts_plugin.enable_callbacks is Disable, which is the + # IOC's default. + # These are not registered as readables: the scalars serve read(), and the + # arrays are consumed by TimeSeriesDataLogic as event pages. + ts_plugin: A[NDStatsTimeSeriesIO, PvSuffix("TS:")] + ts_acquire: A[SignalRW[bool], PvSuffix("TS:TSAcquire"), EpicsOptions(wait=non_zero)] + ts_read: A[SignalW[bool], PvSuffix("TS:TSRead")] + ts_num_points: A[SignalRW[int], PvSuffix("TS:TSNumPoints")] + ts_current_point: A[SignalR[int], PvSuffix("TS:TSCurrentPoint")] + ts_acquiring: A[SignalR[bool], PvSuffix("TS:TSAcquiring")] + ts_acquire_mode: A[SignalRW[NDStatsTSAcquireMode], PvSuffix.rbv("TS:TSAcquireMode")] + ts_min_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMinValue")] + ts_min_x: A[SignalR[Array1D[np.float64]], PvSuffix("TSMinX")] + ts_min_y: A[SignalR[Array1D[np.float64]], PvSuffix("TSMinY")] + ts_max_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMaxValue")] + ts_max_x: A[SignalR[Array1D[np.float64]], PvSuffix("TSMaxX")] + ts_max_y: A[SignalR[Array1D[np.float64]], PvSuffix("TSMaxY")] + ts_mean_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMeanValue")] + ts_sigma: A[SignalR[Array1D[np.float64]], PvSuffix("TSSigma")] + ts_total: A[SignalR[Array1D[np.float64]], PvSuffix("TSTotal")] + ts_net: A[SignalR[Array1D[np.float64]], PvSuffix("TSNet")] + ts_centroid_x: A[SignalR[Array1D[np.float64]], PvSuffix("TSCentroidX")] + ts_centroid_y: A[SignalR[Array1D[np.float64]], PvSuffix("TSCentroidY")] + ts_sigma_x: A[SignalR[Array1D[np.float64]], PvSuffix("TSSigmaX")] + ts_sigma_y: A[SignalR[Array1D[np.float64]], PvSuffix("TSSigmaY")] + ts_sigma_xy: A[SignalR[Array1D[np.float64]], PvSuffix("TSSigmaXY")] + ts_timestamp: A[SignalR[Array1D[np.float64]], PvSuffix("TSTimestamp")] + # How many points the time series arrays hold (the waveforms' NELM) + ts_timestamp_nelm: A[SignalR[int], PvSuffix("TSTimestamp.NELM")] # Centroid statistics compute_centroid: A[ SignalRW[bool], PvSuffix.rbv("ComputeCentroid"), Format.CONFIG_SIGNAL @@ -254,6 +338,40 @@ class NDStatsIO(StandardReadable, NDPluginBaseIO): hist_min: A[SignalRW[float], PvSuffix.rbv("HistMin"), Format.CONFIG_SIGNAL] hist_max: A[SignalRW[float], PvSuffix.rbv("HistMax"), Format.CONFIG_SIGNAL] + async def get_enabled_params(self) -> Sequence[NDParam]: + """The statistics of the groups computed.""" + basic, centroid = await asyncio.gather( + self.compute_statistics.get_value(), + self.compute_centroid.get_value(), + ) + computed: list[NDParam] = [] + if basic: + computed += [ + NDParam(self.min_value, "MIN_VALUE", ts_signal=self.ts_min_value), + NDParam(self.min_x, "MIN_X", ts_signal=self.ts_min_x), + NDParam(self.min_y, "MIN_Y", ts_signal=self.ts_min_y), + NDParam(self.max_value, "MAX_VALUE", ts_signal=self.ts_max_value), + NDParam(self.max_x, "MAX_X", ts_signal=self.ts_max_x), + NDParam(self.max_y, "MAX_Y", ts_signal=self.ts_max_y), + NDParam(self.mean_value, "MEAN_VALUE", ts_signal=self.ts_mean_value), + NDParam(self.sigma, "SIGMA_VALUE", ts_signal=self.ts_sigma), + NDParam(self.total, "TOTAL", ts_signal=self.ts_total), + NDParam(self.net, "NET", ts_signal=self.ts_net), + ] + if centroid: + computed += [ + NDParam( + self.centroid_x, "CENTROIDX_VALUE", ts_signal=self.ts_centroid_x + ), + NDParam( + self.centroid_y, "CENTROIDY_VALUE", ts_signal=self.ts_centroid_y + ), + NDParam(self.sigma_x, "SIGMAX_VALUE", ts_signal=self.ts_sigma_x), + NDParam(self.sigma_y, "SIGMAY_VALUE", ts_signal=self.ts_sigma_y), + NDParam(self.sigma_xy, "SIGMAXY_VALUE", ts_signal=self.ts_sigma_xy), + ] + return computed + class NDROIStatNIO(StandardReadable, EpicsDevice): """Defines the parameters for a single ROI used for statistics calculation. @@ -278,24 +396,117 @@ class NDROIStatNIO(StandardReadable, EpicsDevice): SignalR[float], PvSuffix("MeanValue_RBV"), Format.HINTED_UNCACHED_SIGNAL ] total: A[SignalR[float], PvSuffix("Total_RBV"), Format.HINTED_UNCACHED_SIGNAL] + net: A[SignalR[float], PvSuffix("Net_RBV")] + # Time series of this ROI, filled while the plugin's time series acquires. + # These are not registered as readables: the scalars serve read(), and the + # arrays are consumed by TimeSeriesDataLogic as event pages. + ts_min_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMinValue")] + ts_max_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMaxValue")] + ts_mean_value: A[SignalR[Array1D[np.float64]], PvSuffix("TSMeanValue")] + ts_total: A[SignalR[Array1D[np.float64]], PvSuffix("TSTotal")] + ts_net: A[SignalR[Array1D[np.float64]], PvSuffix("TSNet")] + ts_timestamp: A[SignalR[Array1D[np.float64]], PvSuffix("TSTimestamp")] + # How many points the time series arrays hold (the waveforms' NELM) + ts_timestamp_nelm: A[SignalR[int], PvSuffix("TSTimestamp.NELM")] + + +class NDROIStatTSControl(StrictEnum): + """What a write to the time series control of an `NDROIStatIO` does. + + Mirrors the `TSControl` record in ADCore/db/NDROIStat.template. + """ + ERASE_START = "Erase/Start" + START = "Start" + STOP = "Stop" + READ = "Read" + ERASE = "Erase" -class NDROIStatIO(NDPluginBaseIO): + +class NDROIStatIO(StandardReadable, NDPluginBaseIO): """Plugin for calculating basic statistics for multiple ROIs. Each ROI is implemented as an instance of NDROIStatNIO, - and the collection of ROIs is held as a DeviceVector. + and the collection of ROIs is held as a DeviceVector. Each channel is + registered as a [](#StandardReadableFormat.CHILD) readable, so it reads the + scalars of its own formats. + + `stage()` renames each channel to + `-`, with characters other than letters, digits and + underscores in the PV replaced by underscores, or the channel number if the + PV is empty. This mirrors the interface provided by ADCore/db/NDROIStat.template. See HTML docs at https://areadetector.github.io/areaDetector/ADCore/NDPluginROIStat.html """ + # Time series control, shared by every channel. Writing ERASE_START to + # ts_control erases the arrays of every channel, resets ts_current_point to 0 + # and starts acquiring, which stops by itself after ts_num_points frames. + ts_control: A[SignalRW[NDROIStatTSControl], PvSuffix("TSControl")] + ts_num_points: A[SignalRW[int], PvSuffix("TSNumPoints")] + ts_current_point: A[SignalR[int], PvSuffix("TSCurrentPoint")] + ts_acquiring: A[SignalR[bool], PvSuffix("TSAcquiring")] + def __init__(self, prefix, num_channels=8, with_pvi=False, name=""): - self.channels = DeviceVector( - {i: NDROIStatNIO(f"{prefix}{i}:") for i in range(1, num_channels + 1)} - ) + self._stage_funcs += (self.__on_stage,) + with self.add_children_as_readables(): + self.channels = DeviceVector( + {i: NDROIStatNIO(f"{prefix}{i}:") for i in range(1, num_channels + 1)} + ) super().__init__(prefix, with_pvi, name) + async def __on_stage(self) -> None: + """Name each channel after its Name PV, raising if two would clash.""" + pv_names = await gather_dict( + {number: c.name_.get_value() for number, c in self.channels.items()} + ) + names = { + number: re.sub(r"[^A-Za-z0-9_]", "_", pv_name) if pv_name else str(number) + for number, pv_name in pv_names.items() + } + counts = Counter(names.values()) + clashes = sorted(name for name, count in counts.items() if count > 1) + if clashes: + raise ValueError( + f"{self.name}: more than one channel is named {clashes}, " + "so change the Name PV of all but one" + ) + if self.name: + for number, name in names.items(): + self.channels[number].set_name( + f"{self.name}{self._child_name_separator}{name}", + child_name_separator=self._child_name_separator, + ) + + async def get_enabled_params(self) -> Sequence[NDParam]: + """The params of each used channel.""" + uses = await gather_dict( + {number: c.use.get_value() for number, c in self.channels.items()} + ) + used = [number for number, use in uses.items() if use] + return [ + param + for number in used + for param in self._channel_params(number, self.channels[number]) + ] + + def _channel_params(self, number: int, channel: NDROIStatNIO) -> list[NDParam]: + """The params of a channel, at its asyn address.""" + addr = number - 1 + return [ + NDParam(channel.min_value, "ROISTAT_MIN_VALUE", addr, channel.ts_min_value), + NDParam(channel.max_value, "ROISTAT_MAX_VALUE", addr, channel.ts_max_value), + NDParam( + channel.mean_value, + "ROISTAT_MEAN_VALUE", + addr, + channel.ts_mean_value, + ), + NDParam(channel.total, "ROISTAT_TOTAL", addr, channel.ts_total), + NDParam(channel.net, "ROISTAT_NET", addr, channel.ts_net), + ] + class NDCBFlushOnSoftTrgMode(StrictEnum): ON_NEW_IMAGE = "OnNewImage" diff --git a/src/ophyd_async/epics/adcore/_ndattribute.py b/src/ophyd_async/epics/adcore/_ndattribute.py index 571196eabe..d90facfe85 100644 --- a/src/ophyd_async/epics/adcore/_ndattribute.py +++ b/src/ophyd_async/epics/adcore/_ndattribute.py @@ -52,31 +52,30 @@ def ndattributes_to_xml( root = ET.Element("Attributes") for ndattribute in ndattributes: if isinstance(ndattribute, NDAttributeParam): - ET.SubElement( - root, - "Attribute", - name=ndattribute.name, - type="PARAM", - source=ndattribute.param, - addr=str(ndattribute.addr), - datatype=ndattribute.datatype.name, - description=ndattribute.description, - ) + fields = { + "name": ndattribute.name, + "type": "PARAM", + "source": ndattribute.param, + "addr": str(ndattribute.addr), + "datatype": ndattribute.datatype.name, + } elif isinstance(ndattribute, NDAttributePv): - ET.SubElement( - root, - "Attribute", - name=ndattribute.name, - type="EPICS_PV", + fields = { + "name": ndattribute.name, + "type": "EPICS_PV", # Strip protocol prefix if present - source=ndattribute.signal.source.split("://")[-1], - dbrtype=ndattribute.dbrtype.name, - description=ndattribute.description, - ) + "source": ndattribute.signal.source.split("://")[-1], + "dbrtype": ndattribute.dbrtype.name, + } else: raise ValueError( f"Invalid type for ndattributes: {type(ndattribute)}. " "Expected NDAttributePv or NDAttributeParam." ) + # An empty description is left out: the plugin's XML waveform is only + # 256 to 1024 characters, and each one costs 15 of them + if ndattribute.description: + fields["description"] = ndattribute.description + ET.SubElement(root, "Attribute", fields) xml_text = ET.tostring(root, encoding="unicode") return xml_text diff --git a/src/ophyd_async/epics/adcore/_plan_stubs.py b/src/ophyd_async/epics/adcore/_plan_stubs.py index 82dab00448..a23a0dc511 100644 --- a/src/ophyd_async/epics/adcore/_plan_stubs.py +++ b/src/ophyd_async/epics/adcore/_plan_stubs.py @@ -3,10 +3,8 @@ import bluesky.plan_stubs as bps from bluesky.utils import plan -from ._detector import AreaDetector -from ._io import NDArrayBaseIO, NDStatsIO +from ._io import NDArrayBaseIO from ._ndattribute import ( - NDAttributeDataType, NDAttributeParam, NDAttributePv, ndattributes_to_xml, @@ -24,22 +22,3 @@ def setup_ndattributes( xml, wait=True, ) - - -@plan -def setup_ndstats_sum(detector: AreaDetector, stats_name: str = "stats"): - """Set up nd stats sum nd attribute for a detector.""" - stats = detector.get_plugin(stats_name, NDStatsIO) - yield from ( - setup_ndattributes( - stats, - [ - NDAttributeParam( - name=f"{detector.name}-sum", - param="TOTAL", - datatype=NDAttributeDataType.DOUBLE, - description="Sum of the array", - ) - ], - ) - ) diff --git a/src/ophyd_async/epics/adcore/_trigger_logic.py b/src/ophyd_async/epics/adcore/_trigger_logic.py index 536daa9a04..5d3708f093 100644 --- a/src/ophyd_async/epics/adcore/_trigger_logic.py +++ b/src/ophyd_async/epics/adcore/_trigger_logic.py @@ -6,6 +6,7 @@ ) from ophyd_async.core import ( EnableDisable, + SignalR, TriggerInfo, ) @@ -53,7 +54,8 @@ async def prepare_filter( """Prepare the AD process plugin for a given number of images to filter. :param process_plugin: The NDProcessIO plugin instance (optional) - :param num_filter: Number of images to filter (default: 1) + :param num_filter: Number of images to filter (default: 1). 1 switches the + filter off. :param filter_type: Type of filter to apply (optional) """ if process_plugin is None: @@ -62,6 +64,14 @@ async def prepare_filter( return raise ValueError("Process plugin must be provided to setup filtering!") + # A single exposure needs no filter, and must not keep an earlier one's + if num_filter == 1: + await asyncio.gather( + process_plugin.enable_filter.set(False), + process_plugin.enable_callbacks.set(EnableDisable.ENABLE), + ) + return + # Clear any already ingested frames await process_plugin.reset_filter.trigger() @@ -101,15 +111,19 @@ async def _exposures_per_collection_from_filter( async def trigger_info_from_driver( - driver: ADBaseIO, process_plugin: NDProcessIO | None = None + driver: ADBaseIO, + process_plugin: NDProcessIO | None = None, + num_images_signal: SignalR[int] | None = None, ) -> TriggerInfo: """Read the default TriggerInfo for AD detectors back from the driver. :param driver: The driver to read num_images, acquire_time and acquire_period from :param process_plugin: The NDProcessIO plugin filtering exposures, if any + :param num_images_signal: The signal to read the number of images from, + instead of the driver's `num_images` """ num_images, livetime, period = await asyncio.gather( - driver.num_images.get_value(), + (num_images_signal or driver.num_images).get_value(), driver.acquire_time.get_value(), driver.acquire_period.get_value(), ) @@ -164,17 +178,8 @@ async def default_trigger_info(self) -> TriggerInfo: # Read post_count (not driver.num_images) because the CB plugin's # post_count is what governs how many frames are buffered per trigger # in continuous-acquisition mode, not the driver's num_images. - post_count, livetime, period = await asyncio.gather( - self.cb_plugin.post_count.get_value(), - self.driver.acquire_time.get_value(), - self.driver.acquire_period.get_value(), - ) - exposures_per_collection = await _exposures_per_collection_from_filter( - self.process_plugin - ) - return TriggerInfo( - collections_per_event=max(1, post_count // exposures_per_collection), - exposures_per_collection=exposures_per_collection, - livetime=livetime, - deadtime=max(0.0, period - livetime), + return await trigger_info_from_driver( + self.driver, + self.process_plugin, + num_images_signal=self.cb_plugin.post_count, ) diff --git a/src/ophyd_async/epics/adkinetix.py b/src/ophyd_async/epics/adkinetix.py index e84320951a..a66a7d8cbf 100644 --- a/src/ophyd_async/epics/adkinetix.py +++ b/src/ophyd_async/epics/adkinetix.py @@ -99,8 +99,8 @@ class KinetixDetector(AreaDetector[KinetixDriverIO]): :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/admerlin.py b/src/ophyd_async/epics/admerlin.py index 33b783b381..1ebd3beccb 100644 --- a/src/ophyd_async/epics/admerlin.py +++ b/src/ophyd_async/epics/admerlin.py @@ -124,8 +124,8 @@ class MerlinDetector(AreaDetector[MerlinDriverIO]): :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/adpilatus.py b/src/ophyd_async/epics/adpilatus.py index 6bbe70d5a2..e43dd0036e 100644 --- a/src/ophyd_async/epics/adpilatus.py +++ b/src/ophyd_async/epics/adpilatus.py @@ -143,8 +143,8 @@ class PilatusDetector(AreaDetector[PilatusDriverIO]): :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/adsimdetector.py b/src/ophyd_async/epics/adsimdetector.py index e1243e54c4..7e268e306d 100644 --- a/src/ophyd_async/epics/adsimdetector.py +++ b/src/ophyd_async/epics/adsimdetector.py @@ -55,8 +55,8 @@ class SimDetector(AreaDetector[ADBaseIO]): :param driver_suffix: Suffix for the driver PV, defaults to "cam1:" :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/src/ophyd_async/epics/advimba.py b/src/ophyd_async/epics/advimba.py index 7cb053d3a0..b83217bf02 100644 --- a/src/ophyd_async/epics/advimba.py +++ b/src/ophyd_async/epics/advimba.py @@ -148,8 +148,8 @@ class VimbaDetector(AreaDetector[VimbaDriverIO]): based on camera model. :param proc_suffix: If provided, an NDProcessIO plugin is created at this suffix :param plugins: - Additional areaDetector plugins to include. "proc" is reserved: use - proc_suffix instead. + areaDetector plugins to add, each registered as a readable. "proc" is + reserved: use proc_suffix instead. :param config_sigs: Additional signals to include in configuration :param name: Name for the detector device """ diff --git a/tests/container_tests/test_adsim_system.py b/tests/container_tests/test_adsim_system.py index 0a2048890a..9444cbaf1b 100644 --- a/tests/container_tests/test_adsim_system.py +++ b/tests/container_tests/test_adsim_system.py @@ -28,13 +28,17 @@ import os import sys +from collections import defaultdict from pathlib import Path from typing import Any from unittest.mock import ANY, patch +from urllib.parse import urlparse import bluesky.plan_stubs as bps import bluesky.plans as bp import bluesky.preprocessors as bpp +import h5py +import numpy as np import pytest from aioca import purge_channel_caches from bluesky.run_engine import RunEngine @@ -50,6 +54,7 @@ ) from ophyd_async.core import ( + EnableDisable, StaticFilenameProvider, StaticPathProvider, TriggerInfo, @@ -68,6 +73,9 @@ TIMEOUT = 60.0 # allow extra time for docker compose +# The ROIStat plugin is on its own prefix in example-services' bl01t-di-cam-01 IOC +ROISTAT_PREFIX = "BL01T-EA-TST-02:" + # Applies to every test here, not just the one that used to carry it: they all # need the IOC, and the IOC needs services that are not set up on Windows. pytestmark = pytest.mark.skipif( @@ -116,6 +124,59 @@ def adsim(RE: RunEngine, shared_tmp_path: Path) -> AreaDetector: return adsim +@pytest.fixture +def adsimstat(RE: RunEngine) -> AreaDetector: + """A writer-less SimDetector whose only data logic is the STAT time series. + + The stats plugin is given to the factory by reference, and listed in `plugins` + so it is in the detector's tree. It writes no file: the NDPluginStats time + series is a fixed-length buffer read back as event pages, so this exercises + the bounded data-logic tier against a real IOC (the + `ADDataFactory.time_series` factory and NDStatsIO's `TS:*` signals). + """ + stats = adcore.NDStatsIO("BL01T-DI-CAM-01:STAT:") + with init_devices(): + adsimstat = SimDetector( + "BL01T-DI-CAM-01:", + adcore.ADDataFactory.time_series(stats), + driver_suffix="DET:", + plugins={"stats": stats}, + ) + + return adsimstat + + +@pytest.fixture +def adsimroi(RE: RunEngine, shared_tmp_path: Path) -> AreaDetector: + """A SimDetector with an HDF writer and the ROIStat time series. + + The ROIStat plugin is in `plugins` so it is in the detector's tree, and is + given to both factories by reference: the writer stamps its NDAttributes into + the file as `ndattribute_sources`, and the time series reads its `TS*` arrays. + Which of them serves the statistics is decided by the HDF plugin's + EnableCallbacks at the first prepare() (`always_enable=False`). Note its prefix + is not the camera's: example-services puts the ROIStat plugin on + BL01T-EA-TST-02. + """ + provider = StaticPathProvider(StaticFilenameProvider("adsimroi"), shared_tmp_path) + roistat = adcore.NDROIStatIO(f"{ROISTAT_PREFIX}ROISTAT:", num_channels=4) + with init_devices(): + adsimroi = SimDetector( + "BL01T-DI-CAM-01:", + adcore.ADDataFactory.hdf( + provider, + writer_suffix="HDF5:", + always_enable=False, + ndattribute_sources=[roistat], + ), + adcore.ADDataFactory.time_series(roistat), + driver_suffix="DET:", + plugins={"roistat": roistat}, + ) + + return adsimroi + + @pytest.fixture(autouse=True) def reset_adsim_to_baseline(RE: RunEngine, adsim: SimDetector) -> None: """Put the detector back to a known state before every test. @@ -360,6 +421,184 @@ def test_software_triggering( ] +@pytest.mark.timeout(TIMEOUT + 15.0) +def test_stats_time_series_step_scan( + RE: RunEngine, adsimstat: SimDetector, bl01t_di_cam_01: None +) -> None: + """A writer-less stats detector arms the TS buffer, acquires, and reads the + per-frame Total series back as a single event holding the whole array. + + This drives the real STAT plugin's `TS:*` records end to end: prepare sizes + and arms `TS:TSNumPoints`/`TS:TSAcquire`, the sim driver takes the frames, + and read() slices `TSTotal` (timestamped by `TSTimestamp`) into a + single 5-element reading named after the `Total_RBV` scalar. + """ + collections = 5 + stats = adsimstat.get_plugin("stats", adcore.NDStatsIO) + # The plugin's switches at the first prepare() decide which statistics are published + RE( + bps.mv( + stats.enable_callbacks, + EnableDisable.ENABLE, + stats.compute_statistics, + True, + ) + ) + + @bpp.stage_decorator([adsimstat]) + @bpp.run_decorator() + def prepare_then_count() -> MsgGenerator[None]: + # prepare() must come after stage(), which resets the detector + yield from bps.prepare( + adsimstat, + TriggerInfo(collections_per_event=collections, livetime=0.1), + wait=True, + ) + yield from bps.trigger_and_read([adsimstat]) + + named_docs: list[tuple[str, DocumentType]] = [] + RE(prepare_then_count(), lambda name, doc: named_docs.append((name, doc))) + + # A bounded logic exposes collect_pages, never collect_asset_docs. + assert hasattr(adsimstat, "collect_pages") + assert not hasattr(adsimstat, "collect_asset_docs") + + descriptor = next(doc for name, doc in named_docs if name == "descriptor") + data_key = descriptor["data_keys"]["adsimstat-stats-total"] + assert data_key["shape"] == [collections] + assert data_key["dtype"] == "array" + assert data_key["dtype_numpy"] == "= 0 + + +@pytest.mark.timeout(TIMEOUT + 30.0) +@pytest.mark.parametrize("hdf_enabled", [True, False]) +def test_roistat_is_served_by_the_hdf_file_or_the_time_series( + RE: RunEngine, adsimroi: SimDetector, bl01t_di_cam_01: None, hdf_enabled: bool +) -> None: + """The HDF plugin's EnableCallbacks picks who serves the ROIStat statistics. + + On, the HDF writer stamps each used channel's NDAttributes into the file and + the time series sits out: the documents are stream resources and datums, with + no event pages. Off, the time series serves them as event pages and there is + no file. Either way a key is `--` for each used + channel, and an unused channel publishes nothing. + + Relies on ADCore/db/NDROIStat.template and NDROIStatN.template (read, not run + here): `Name`, `Use`, `MinX`.. are per channel; `TSControl` Erase/Start sizes + and arms a fixed-length buffer of `TSNumPoints` that the plugin fills one + point per array it processes; `TSControl` Read posts the arrays so far. + """ + frames = 5 + roistat = adsimroi.get_plugin("roistat", adcore.NDROIStatIO) + hdf = adsimroi.get_plugin("hdf", adcore.NDFileHDF5IO) + # The switches at the first prepare() decide who serves, and the Name PVs name + # the keys, so + # all of it is settled before the stage that records it. Channel 3 is unused. + left, centre, unused = (roistat.channels[number] for number in (1, 2, 3)) + RE( + bps.mv( + roistat.enable_callbacks, + EnableDisable.ENABLE, + hdf.enable_callbacks, + EnableDisable.ENABLE if hdf_enabled else EnableDisable.DISABLE, + # The ROIStat plugin stamps its NDAttributes on the arrays it passes + # on, so the file only receives them downstream of it. The baseline + # puts the port back before the next test. + hdf.nd_array_port, + "DET.ROISTAT", + left.name_, + "Left", + left.use, + True, + left.min_x, + 0, + left.min_y, + 0, + left.size_x, + 512, + left.size_y, + 1024, + centre.name_, + "Centre", + centre.use, + True, + centre.min_x, + 256, + centre.min_y, + 256, + centre.size_x, + 512, + centre.size_y, + 512, + unused.use, + False, + roistat.channels[4].use, + False, + ) + ) + pixels = {"Left": 512 * 1024, "Centre": 512 * 512} + stats = ("min_value", "max_value", "mean_value", "total") + keys = [f"adsimroi-roistat-{name}-{stat}" for name in pixels for stat in stats] + + @bpp.stage_decorator([adsimroi]) + @bpp.run_decorator() + def fly() -> MsgGenerator[None]: + yield from bps.prepare( + adsimroi, TriggerInfo(number_of_events=frames, livetime=0.1), wait=True + ) + yield from bps.declare_stream(adsimroi, name="primary", collect=True) + yield from bps.kickoff(adsimroi, wait=True) + yield from bps.collect_while_completing( + flyers=[adsimroi], dets=[adsimroi], flush_period=0.5 + ) + + docs: dict[str, list[Any]] = defaultdict(list) + RE(fly(), lambda name, doc: docs[name].append(doc)) + + assert docs["stop"][0]["exit_status"] == "success" + (descriptor,) = docs["descriptor"] + if hdf_enabled: + # The file serves them: stream resources, in the file, one value a frame + assert docs["event_page"] == [] + resources = {doc["data_key"]: doc for doc in docs["stream_resource"]} + assert sorted(resources) == sorted(["adsimroi", *keys]) + frames_indexed = max(doc["indices"]["stop"] for doc in docs["stream_datum"]) + assert frames_indexed == frames + (path,) = {urlparse(doc["uri"]).path for doc in resources.values()} + with h5py.File(path) as hdf_file: + for key in keys: + dataset = hdf_file[resources[key]["parameters"]["dataset"]] + assert isinstance(dataset, h5py.Dataset) + values = np.asarray(dataset[:]) + assert values.shape == (frames,) + assert np.isfinite(values).all() + else: + # The time series serves them: pages of one reading per frame, no file + assert docs["stream_resource"] == [] + assert sorted(descriptor["data_keys"]) == sorted(keys) + assert docs["event"] == [] + pages = docs["event_page"] + assert sum(len(page["seq_num"]) for page in pages) == frames + series = { + key: np.concatenate([np.ravel(page["data"][key]) for page in pages]) + for key in keys + } + for name, size in pixels.items(): + prefix = f"adsimroi-roistat-{name}-" + smallest, largest, mean, total = (series[prefix + stat] for stat in stats) + assert (smallest <= mean).all() + assert (mean <= largest).all() + assert total == pytest.approx(mean * size, rel=1e-3) + + def run_plan_and_get_documents( RE: RunEngine, plan: MsgGenerator[Any] ) -> list[DocumentType]: diff --git a/tests/unit_tests/epics/adcore/test_cont_acq_detector.py b/tests/unit_tests/epics/adcore/test_cont_acq_detector.py index 23e06f0b70..aefa41f708 100644 --- a/tests/unit_tests/epics/adcore/test_cont_acq_detector.py +++ b/tests/unit_tests/epics/adcore/test_cont_acq_detector.py @@ -6,6 +6,7 @@ from ophyd_async.core import ( DetectorTrigger, + EnableDisable, StaticPathProvider, TriggerInfo, callback_on_mock_put, @@ -143,6 +144,12 @@ async def test_cont_acq_skips_dirty_frames( set_mock_value(writer.file_path_exists, True) set_mock_value(cb.capture, True) set_mock_value(cb.pre_count, 2) + if proc_suffix: + # The process plugin is the detector's, so it is switched on whatever it was + set_mock_value( + det.get_plugin("proc", adcore.NDProcessIO).enable_callbacks, + EnableDisable.DISABLE, + ) async def set_pre_count(value: int) -> int: # ADCore refuses PreCount changes while capture is active. @@ -207,6 +214,9 @@ async def receive_frame(counter: int) -> None: await det.unstage() assert not await cb.capture.get_value() assert await det.driver.acquire.get_value() + if proc_suffix: + proc = det.get_plugin("proc", adcore.NDProcessIO) + assert await proc.enable_callbacks.get_value() is EnableDisable.ENABLE @pytest.mark.parametrize("cancel", [False, True]) @@ -364,3 +374,18 @@ def soft_trigger(value: bool) -> bool | None: {"start": 2, "stop": 5} ] await det.unstage() + + +async def test_cont_acq_stage_switches_the_circular_buffer_on( + cont_acq_detector: adcore.AreaDetector[adcore.ADBaseIO], +): + set_mock_value(cont_acq_detector.cb.enable_callbacks, EnableDisable.DISABLE) + await cont_acq_detector.stage() + assert ( + await cont_acq_detector.cb.enable_callbacks.get_value() is EnableDisable.ENABLE + ) + + await cont_acq_detector.prepare(TriggerInfo()) + await cont_acq_detector.trigger() + await cont_acq_detector.prepare(TriggerInfo()) + await cont_acq_detector.trigger() diff --git a/tests/unit_tests/epics/adcore/test_data_logic.py b/tests/unit_tests/epics/adcore/test_data_logic.py index 6224c9a0c2..d7ba6ab103 100644 --- a/tests/unit_tests/epics/adcore/test_data_logic.py +++ b/tests/unit_tests/epics/adcore/test_data_logic.py @@ -98,9 +98,10 @@ async def test_hdf_follows_the_plugin_when_not_enabling_it( always_enable=always_enable, ) + await logic.ensure_ready() made = await logic.start_provider("det", 1, 0.1, 0) assert (made is not None) is makes_provider - # Starting switches the plugin on either way, and sitting out writes nothing + # Staging switches the plugin on only if told to, and sitting out writes nothing assert await writer.capture.get_value() is makes_provider assert await writer.enable_callbacks.get_value() is ( EnableDisable.ENABLE if makes_provider else EnableDisable.DISABLE @@ -123,9 +124,11 @@ async def test_a_writer_follows_its_plugin_by_default( makes_provider: bool, ): factory = getattr(adcore.ADDataFactory, kind)(static_path_provider) - writer, logic = factory("PREFIX:", hdf_det.driver, []) + named_writer, logic = factory.build("PREFIX:", hdf_det.driver, []) + assert named_writer is not None + attr, writer = named_writer async with init_devices(mock=True): - writer.set_name(kind) + writer.set_name(attr) await writer.connect(mock=True) set_mock_value(writer.file_path_exists, True) set_mock_value(writer.enable_callbacks, plugin_enabled) @@ -184,7 +187,6 @@ async def test_prepare_hdf( call.hdf.lazy_open.put(True), call.hdf.swmr_mode.put(True), call.hdf.xml_file_name.put(""), - call.hdf.enable_callbacks.put(EnableDisable.ENABLE), call.hdf.create_directory.put(0), call.hdf.file_path.put(f"{static_path_provider().directory_path}{os.sep}"), call.hdf.file_name.put("ophyd_async_tests"), @@ -221,7 +223,6 @@ async def test_prepare_multipart( writer, [ call.lazy_open.put(True), - call.enable_callbacks.put(EnableDisable.ENABLE), call.create_directory.put(0), call.file_path.put(f"{static_path_provider().directory_path}{os.sep}"), call.file_name.put("ophyd_async_tests"), @@ -236,27 +237,37 @@ async def test_prepare_multipart( @pytest.mark.parametrize( - "flush_period,livetime,deadtime,exposures_per_collection,expected_frames_per_chunk", + "size_x,size_y,flush_period,livetime,deadtime,exposures_per_collection," + "expected_frames_per_chunk", [ # 0.5s flush / 0.01s period -> 50 frames per chunk (#1309) - (0.5, 0.01, 0.0, 1, 50), + (256, 16, 0.5, 0.01, 0.0, 1, 50), # 0.5s flush / 0.05s period -> 10 frames per chunk - (0.5, 0.04, 0.01, 1, 10), + (256, 16, 0.5, 0.04, 0.01, 1, 10), + # 0.5s flush / 0.03s period -> 16.7 frames, rounded to 17 + (256, 16, 0.5, 0.03, 0.0, 1, 17), # A sub-frame flush period floors to a single frame per chunk - (0.001, 0.05, 0.0, 1, 1), + (256, 16, 0.001, 0.05, 0.0, 1, 1), # 5 exposures of 0.01s make each frame written, so 0.5s holds 10 - (0.5, 0.01, 0.0, 5, 10), + (256, 16, 0.5, 0.01, 0.0, 5, 10), + # 1.5 MiB frames: a 4 MiB chunk holds 2 of the 50 + (1024, 768, 0.5, 0.01, 0.0, 1, 2), + # 8 MiB frames: still one per chunk + (2048, 2048, 0.5, 0.01, 0.0, 1, 1), ], ) async def test_hdf_chunk_sized_from_flush_period( static_path_provider: StaticPathProvider, + size_x: int, + size_y: int, flush_period: float, livetime: float, deadtime: float, exposures_per_collection: int, expected_frames_per_chunk: int, ): - """A TriggerInfo flush_period sizes the HDF chunk from the collection period.""" + """A TriggerInfo flush_period sizes the HDF chunk from the collection period, + capped at 4 MiB.""" async with init_devices(mock=True): det = adsimdetector.SimDetector( "PREFIX:", @@ -264,8 +275,8 @@ async def test_hdf_chunk_sized_from_flush_period( # Averages the exposures of each collection into one frame proc_suffix="PROC:", ) - set_mock_value(det.driver.array_size_x, 1024) - set_mock_value(det.driver.array_size_y, 768) + set_mock_value(det.driver.array_size_x, size_x) + set_mock_value(det.driver.array_size_y, size_y) set_mock_value(det.driver.data_type, adcore.ADBaseDataType.UINT16) writer = det.get_plugin("hdf", adcore.NDPluginFileIO) set_mock_value(writer.file_path_exists, True) @@ -285,8 +296,8 @@ async def test_hdf_chunk_sized_from_flush_period( await det.unstage() assert sr[1]["parameters"]["chunk_shape"] == ( expected_frames_per_chunk, - 768, - 1024, + size_y, + size_x, ) @@ -529,3 +540,75 @@ async def test_multipart_start_waits_for_num_captured_to_read_zero( mimetype="multipart/related;type=image/tiff", ) await _start_waits_for_a_zero_counter(logic, writer.num_captured) + + +async def test_hdf_writes_the_ndattributes_of_its_sources_and_parses_the_others( + static_path_provider: StaticPathProvider, +): + listed = adcore.NDStatsIO("PREFIX:LISTED:") + unlisted = adcore.NDStatsIO("PREFIX:UNLISTED:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.hdf( + static_path_provider, ndattribute_sources=[listed] + ), + plugins={"listed": listed, "unlisted": unlisted}, + name="det", + ) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + # Opt the unlisted plugin out of being read, so only the file could name it + det.set_readable_format(unlisted, None) + for stats in (listed, unlisted): + set_mock_value(stats.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(stats.compute_statistics, True) + set_mock_value(listed.nd_attributes_file_nelm, 2048) + # Names that the unlisted plugin was told to stamp are kept as they are + unlisted_attributes = [ + adcore.NDAttributeParam( + name="free-form", param="TOTAL", datatype=adcore.NDAttributeDataType.DOUBLE + ) + ] + set_mock_value( + unlisted.nd_attributes_file, adcore.ndattributes_to_xml(unlisted_attributes) + ) + await det.stage() + await det.prepare(TriggerInfo()) + + datakeys = set(await det.describe()) + + assert "det-listed-total" in datakeys + assert "free-form" in datakeys + assert not [key for key in datakeys if key.startswith("det-unlisted")] + # The listed plugin's file was written, the unlisted plugin's left as set + assert "det-listed-total" in await listed.nd_attributes_file.get_value() + assert await unlisted.nd_attributes_file.get_value() == ( + adcore.ndattributes_to_xml(unlisted_attributes) + ) + + +async def test_hdf_raises_if_the_ndattributes_xml_does_not_fit_the_plugin( + static_path_provider: StaticPathProvider, +): + stats = adcore.NDStatsIO("PREFIX:STATS:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.hdf(static_path_provider, ndattribute_sources=[stats]), + plugins={"stats": stats}, + name="det", + ) + set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + set_mock_value(stats.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(stats.compute_statistics, True) + set_mock_value(stats.nd_attributes_file_nelm, 100) + await det.stage() + + with pytest.raises( + ValueError, + match=r"PREFIX:STATS:NDAttributesFile is \d+ characters but the waveform " + r"holds 100: raise XMLSIZE", + ): + await det.prepare(TriggerInfo()) diff --git a/tests/unit_tests/epics/adcore/test_detector.py b/tests/unit_tests/epics/adcore/test_detector.py index d3eca62921..2a837fbaf3 100644 --- a/tests/unit_tests/epics/adcore/test_detector.py +++ b/tests/unit_tests/epics/adcore/test_detector.py @@ -1,7 +1,7 @@ import pytest +from ophyd_async.core import StandardReadableFormat as Format from ophyd_async.core import ( - StandardReadableFormat, StaticPathProvider, TriggerInfo, init_devices, @@ -20,21 +20,31 @@ ) @pytest.mark.parametrize("hinted", [True, False]) def test_writer_image_hints(static_path_provider: StaticPathProvider, factory, hinted): - driver = adcore.ADBaseIO("PREFIX:DRV:") stats = adcore.NDStatsIO("PREFIX:STAT:") - det = adcore.AreaDetector( - driver, + det_with_stats = adcore.AreaDetector( + adcore.ADBaseIO("PREFIX:DRV:"), "PREFIX:", factory(static_path_provider, datakey_suffix="-image", hinted=hinted), plugins={"stats": stats}, name="det", ) + det = adcore.AreaDetector( + adcore.ADBaseIO("PREFIX:DRV2:"), + "PREFIX:", + factory(static_path_provider, datakey_suffix="-image", hinted=hinted), + name="det2", + ) # No hinted fields means no hints at all - assert det.hints == ({"fields": ["det-image"]} if hinted else {}) - det.set_readable_format(stats.total, StandardReadableFormat.HINTED_SIGNAL) - # Readable-format signals are hinted ahead of the data logics' fields - assert det.hints == { - "fields": ["det-stats-total"] + (["det-image"] if hinted else []) + assert det.hints == ({"fields": ["det2-image"]} if hinted else {}) + # A plugin in plugins= is a readable, hinted ahead of the data logics' fields + assert det_with_stats.hints == { + "fields": [ + "det-stats-min_value", + "det-stats-max_value", + "det-stats-mean_value", + "det-stats-total", + *(["det-image"] if hinted else []), + ] } @@ -121,6 +131,99 @@ def test_plugin_named_like_the_driver_raises(): ) +def test_stats_time_series_makes_no_plugin_of_its_own(): + stats = adcore.NDStatsIO("PREFIX:STATS:") + + det = adcore.AreaDetector( + adcore.ADBaseIO("PREFIX:DRV:"), + "PREFIX:", + adcore.ADDataFactory.time_series(stats), + plugins={"stats": stats}, + name="det", + ) + + assert [name for name, _ in det.children()] == ["driver", "stats"] + + +def test_the_detector_registers_every_plugin_and_writer_as_a_child_readable( + static_path_provider: StaticPathProvider, +): + stats = adcore.NDStatsIO("PREFIX:STATS:") + + det = adcore.ContAcqDetector( + "PREFIX:", + adcore.ADDataFactory.hdf(static_path_provider), + proc_suffix="PROC:", + plugins={"stats": stats}, + name="det", + ) + + children = [device for device, format in det.get_readable_formats()] + assert children == [det.driver, stats, det.proc, det.cb, det.hdf] + assert all(format is Format.CHILD for _, format in det.get_readable_formats()) + + +def test_the_factories_are_given_the_plugins_of_the_user_and_the_detector( + static_path_provider: StaticPathProvider, +): + stats = adcore.NDStatsIO("PREFIX:STATS:") + seen = [] + hdf = adcore.ADDataFactory.hdf(static_path_provider) + + def build(prefix, driver, plugins): + seen.extend(plugins) + return hdf.build(prefix, driver, plugins) + + det = adcore.ContAcqDetector( + "PREFIX:", + adcore.ADDataFactory(build=build), + proc_suffix="PROC:", + plugins={"stats": stats}, + name="det", + ) + + assert seen == [stats, det.proc, det.cb] + + +@pytest.mark.parametrize( + "make_factory", + [ + lambda path_provider, plugin: adcore.ADDataFactory.hdf( + path_provider, ndattribute_sources=[plugin] + ), + lambda path_provider, plugin: adcore.ADDataFactory.time_series(plugin), + ], + ids=["hdf-ndattribute-sources", "time-series"], +) +def test_a_plugin_taken_by_reference_must_be_in_plugins( + static_path_provider: StaticPathProvider, make_factory +): + stats = adcore.NDStatsIO("PREFIX:STATS:", name="stats") + + with pytest.raises(ValueError, match=r"stats is not .* add it to `plugins=`"): + adcore.AreaDetector( + adcore.ADBaseIO("PREFIX:DRV:"), + "PREFIX:", + make_factory(static_path_provider, stats), + name="det", + ) + + +def test_area_detector_accepts_an_ndattribute_source_below_a_plugin( + static_path_provider: StaticPathProvider, +): + roi = adcore.NDROIIO("PREFIX:ROI:") + roi.stat = adcore.NDStatsIO("PREFIX:STATS:") + + adcore.AreaDetector( + adcore.ADBaseIO("PREFIX:DRV:"), + "PREFIX:", + adcore.ADDataFactory.hdf(static_path_provider, ndattribute_sources=[roi.stat]), + plugins={"roi": roi}, + name="det", + ) + + def test_deprecated_writer_factory_still_builds_the_new_factories( static_path_provider: StaticPathProvider, ): diff --git a/tests/unit_tests/epics/adcore/test_prepare_exposures.py b/tests/unit_tests/epics/adcore/test_prepare_exposures.py index eea704defc..ed389d4378 100644 --- a/tests/unit_tests/epics/adcore/test_prepare_exposures.py +++ b/tests/unit_tests/epics/adcore/test_prepare_exposures.py @@ -113,7 +113,7 @@ async def test_prepare_exposures( ("exposures_per_collection", "filter_type", "expected_filter"), [ # None keeps the filter type pre-configured in the IOC (AVERAGE) - (1, None, adcore.NDProcessFilterType.AVERAGE), + (2, None, adcore.NDProcessFilterType.AVERAGE), (5, adcore.NDProcessFilterType.AVERAGE, adcore.NDProcessFilterType.AVERAGE), (10, adcore.NDProcessFilterType.SUM, adcore.NDProcessFilterType.SUM), ( @@ -150,6 +150,28 @@ async def test_prepare_filter( assert await process_plugin.filter_type.get_value() == expected_filter +@pytest.mark.parametrize( + ("num_filter", "filter_left_on"), + [ + (5, True), + # One exposure needs no filter, and clears an earlier one + (1, False), + ], +) +async def test_prepare_filter_switches_the_plugin_on( + process_plugin: adcore.NDProcessIO, + num_filter: int, + filter_left_on: bool, +): + set_mock_value(process_plugin.enable_callbacks, EnableDisable.DISABLE) + set_mock_value(process_plugin.enable_filter, not filter_left_on) + + await adcore.prepare_filter(process_plugin, num_filter) + + assert await process_plugin.enable_callbacks.get_value() is EnableDisable.ENABLE + assert await process_plugin.enable_filter.get_value() is filter_left_on + + async def test_prepare_filter_without_plugin(): await adcore.prepare_filter(None, num_filter=1) with pytest.raises(ValueError, match="Process plugin must be provided"): diff --git a/tests/unit_tests/epics/adcore/test_time_series.py b/tests/unit_tests/epics/adcore/test_time_series.py new file mode 100644 index 0000000000..414d9bc2da --- /dev/null +++ b/tests/unit_tests/epics/adcore/test_time_series.py @@ -0,0 +1,380 @@ +import asyncio +from collections import defaultdict +from typing import get_type_hints +from unittest.mock import call + +import bluesky.plan_stubs as bps +import bluesky.preprocessors as bpp +import numpy as np +import pytest +from bluesky import RunEngine + +from ophyd_async.core import ( + Array1D, + DetectorAcquireLogic, + DetectorLogic, + DetectorTriggerLogic, + EnableDisable, + EventPageDataProvider, + SignalR, + StandardDetector, + TriggerInfo, + callback_on_mock_put, + get_mock_put, + init_devices, + non_zero, + set_mock_put_proceeds, + set_mock_value, +) +from ophyd_async.core import StandardReadableFormat as Format +from ophyd_async.epics import adcore +from ophyd_async.epics.adcore import ( + NDStatsTSAcquireMode, + TimeSeriesDataLogic, +) +from ophyd_async.epics.core import EpicsOptions +from ophyd_async.testing import assert_has_calls + + +@pytest.fixture +async def stats() -> adcore.NDStatsIO: + async with init_devices(mock=True): + plugin = adcore.NDStatsIO("PREFIX:STATS:", name="det-stats") + # The IOC's arrays hold NCHANS points, 2048 by default + set_mock_value(plugin.ts_timestamp_nelm, 2048) + # Switched on, so it publishes its basic statistics + set_mock_value(plugin.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(plugin.compute_statistics, True) + return plugin + + +# The statistics NDStatsIO gives a read format, so publishes with switches on +PUBLISHED = [ + "det-stats-min_value", + "det-stats-min_x", + "det-stats-min_y", + "det-stats-max_value", + "det-stats-max_x", + "det-stats-max_y", + "det-stats-mean_value", + "det-stats-total", +] + + +async def test_start_sizes_and_arms_the_buffer(stats: adcore.NDStatsIO): + set_mock_value(stats.ts_plugin.enable_callbacks, EnableDisable.DISABLE) + # Left wrapping by the user, so it would never fill and stop + set_mock_value(stats.ts_acquire_mode, NDStatsTSAcquireMode.CIRCULAR_BUFFER) + logic = TimeSeriesDataLogic(stats) + provider = await logic.start_provider("det-stats", [], 5, 5, 0.1, 0) + assert isinstance(provider, EventPageDataProvider) + assert await stats.ts_num_points.get_value() == 5 + assert await stats.ts_acquire_mode.get_value() == NDStatsTSAcquireMode.FIXED_LENGTH + # ts_acquire=1 arms and clears the buffer + assert await stats.ts_acquire.get_value() is True + # The embedded time series plugin is the logic's own, and takes no points + # while it is disabled + assert await stats.ts_plugin.enable_callbacks.get_value() == EnableDisable.ENABLE + # Each series is named after its scalar + datakeys = await provider.make_datakeys() + assert list(datakeys) == PUBLISHED + assert datakeys["det-stats-total"]["shape"] == [5] + + +@pytest.mark.parametrize("points,raises", [(2048, False), (2049, True)]) +async def test_start_raises_for_a_scan_longer_than_the_buffer( + stats: adcore.NDStatsIO, points: int, raises: bool +): + logic = TimeSeriesDataLogic(stats) + + start = logic.start_provider("det", [], points, 1, 0.1, 0) + + if raises: + with pytest.raises( + ValueError, match="needs 2049 points, but the plugin's arrays hold 2048" + ): + await start + else: + assert await start is not None + + +async def test_every_time_series_can_be_published(stats: adcore.NDStatsIO): + """Every ts_* array but the timestamps is a param's, so can be published.""" + hints = get_type_hints(adcore.NDStatsIO) + arrays = { + signal + for attr, signal in stats.children() + if attr.startswith("ts_") + and attr != "ts_timestamp" + and hints[attr] == SignalR[Array1D[np.float64]] + } + # Compute every group and give every scalar a read format + set_mock_value(stats.compute_centroid, True) + for attr, scalar in stats.children(): + if f"ts_{attr}" in dict(stats.children()): + stats.set_readable_format(scalar, Format.UNCACHED_SIGNAL) + + params = await stats.get_enabled_params() + + assert arrays == {param.ts_signal for param in params} + + +@pytest.mark.parametrize("reset_late", [False, True]) +async def test_start_leaves_the_progress_counter_at_zero( + stats: adcore.NDStatsIO, reset_late: bool +): + """The count a step point is measured from is 0 once start_provider returns. + + A buffer that still holds a previous scan's points reports them until it is + armed; the IOC resets TSCurrentPoint inside the TSAcquire write (ADCore + NDPluginTimeSeries::writeInt32 -> acquireReset), but its monitor may arrive + after the put's readback. + """ + set_mock_value(stats.ts_current_point, 5) + loop = asyncio.get_running_loop() + + def reset(value: bool): + if value and reset_late: + loop.call_later(0.01, set_mock_value, stats.ts_current_point, 0) + elif value: + set_mock_value(stats.ts_current_point, 0) + + callback_on_mock_put(stats.ts_acquire, reset) + logic = TimeSeriesDataLogic(stats) + + assert await logic.start_provider("det", [], 5, 1, 0.1, 0) is not None + + assert await stats.ts_current_point.get_value() == 0 + + +def test_ts_acquire_is_declared_as_a_busy_record(): + """TSAcquire completes its put only when the buffer is full, so only a put + of 1 may wait for completion, as for the other areaDetector busy records.""" + (options,) = [ + meta + for meta in get_type_hints(adcore.NDStatsIO, include_extras=True)[ + "ts_acquire" + ].__metadata__ + if isinstance(meta, EpicsOptions) + ] + assert options.wait is non_zero + + +async def test_start_does_not_wait_for_the_buffer_to_fill(stats: adcore.NDStatsIO): + logic = TimeSeriesDataLogic(stats) + # The put of 1 to TSAcquire does not complete until the buffer is full + set_mock_put_proceeds(stats.ts_acquire, False) + + await asyncio.wait_for(logic.start_provider("det", [], 5, 1, 0.1, 0), timeout=1) + + assert await stats.ts_acquire.get_value() is True + set_mock_put_proceeds(stats.ts_acquire, True) + await logic.ensure_stopped() + assert_has_calls(stats.ts_acquire, [call.put(True), call.put(False)]) + assert await stats.ts_acquire.get_value() is False + + +@pytest.mark.parametrize( + "plugin_enabled,compute_statistics,compute_centroid,expected_datakeys", + [ + # Switched on, it publishes every statistic with a read format... + (EnableDisable.ENABLE, True, False, PUBLISHED), + # ...including a centroid given one, when its group is computed + (EnableDisable.ENABLE, True, True, [*PUBLISHED, "det-stats-centroid_x"]), + (EnableDisable.ENABLE, False, True, ["det-stats-centroid_x"]), + # Nothing is computed, so it sits out + (EnableDisable.ENABLE, False, False, None), + # A disabled plugin computes nothing, so it sits out + (EnableDisable.DISABLE, True, True, None), + ], +) +async def test_publishes_what_the_switches_leave_on( + stats: adcore.NDStatsIO, + plugin_enabled: EnableDisable, + compute_statistics: bool, + compute_centroid: bool, + expected_datakeys: list[str] | None, +): + stats.set_readable_format(stats.centroid_x, Format.UNCACHED_SIGNAL) + set_mock_value(stats.enable_callbacks, plugin_enabled) + set_mock_value(stats.compute_statistics, compute_statistics) + set_mock_value(stats.compute_centroid, compute_centroid) + logic = TimeSeriesDataLogic(stats) + + made = await logic.start_provider("det", [], 5, 1, 0.1, 0) + + if expected_datakeys is None: + assert made is None + # Sitting out arms nothing + assert await stats.ts_acquire.get_value() is False + else: + assert made is not None + assert list(await made.make_datakeys()) == expected_datakeys + # The switches are the plugin's to decide, never written by this logic + switches = [ + stats.enable_callbacks, + stats.compute_statistics, + stats.compute_centroid, + ] + for switch in switches: + get_mock_put(switch).assert_not_called() + + +@pytest.mark.parametrize( + "existing_datakeys,expected_datakeys", + [ + # Nothing made elsewhere, so it publishes every array + ([], PUBLISHED), + # Only the arrays the unbounded providers do not make + ([*PUBLISHED[1:], "det"], PUBLISHED[:1]), + # Nothing left to make, so it sits out + (PUBLISHED, None), + ], +) +async def test_publishes_only_the_arrays_not_already_made( + stats: adcore.NDStatsIO, + existing_datakeys: list[str], + expected_datakeys: list[str] | None, +): + logic = TimeSeriesDataLogic(stats) + + made = await logic.start_provider("det", existing_datakeys, 5, 1, 0.1, 0) + + if expected_datakeys is None: + assert made is None + # Sitting out arms nothing + assert await stats.ts_acquire.get_value() is False + else: + assert made is not None + assert list(await made.make_datakeys()) == expected_datakeys + + +async def test_ensure_stopped_stops_the_time_series(stats: adcore.NDStatsIO): + logic = TimeSeriesDataLogic(stats) + set_mock_value(stats.ts_acquire, True) + await logic.ensure_stopped() + assert await stats.ts_acquire.get_value() is False + + +async def test_ensure_stopped_waits_for_the_roi_stat_time_series_to_stop(): + async with init_devices(mock=True): + roistat = adcore.NDROIStatIO("PREFIX:ROISTAT:", num_channels=1) + set_mock_value(roistat.ts_acquiring, True) + loop = asyncio.get_running_loop() + + def stop(value: adcore.NDROIStatTSControl): + if value == adcore.NDROIStatTSControl.STOP: + # The plugin reports it has stopped after the put returns + loop.call_later(0.01, set_mock_value, roistat.ts_acquiring, False) + + callback_on_mock_put(roistat.ts_control, stop) + + await TimeSeriesDataLogic(roistat).ensure_stopped() + + assert await roistat.ts_acquiring.get_value() is False + + +class _JustInternal(DetectorTriggerLogic): + async def prepare_internal(self, num: int, livetime: float, deadtime: float): ... + + async def default_trigger_info(self) -> TriggerInfo: + return TriggerInfo() + + +class _FillStatsAcquireLogic(DetectorAcquireLogic): + """Fills the mock time series when acquisition starts.""" + + def __init__(self, stats: adcore.NDStatsIO, values: np.ndarray, times: np.ndarray): + self.stats = stats + self.values = values + self.times = times + + async def acquire(self): + for param in await self.stats.get_enabled_params(): + assert param.ts_signal is not None + set_mock_value(param.ts_signal, self.values) + set_mock_value(self.stats.ts_timestamp, self.times) + set_mock_value(self.stats.ts_current_point, len(self.values)) + yield + + async def ensure_stopped(self): ... + + +@pytest.fixture +def stats_detector(stats: adcore.NDStatsIO) -> StandardDetector: + """A writer-less detector whose only data logic is a stats time series.""" + values = np.array([5.0, 6.0, 7.0, 8.0]) + times = np.array([200.0, 201.0, 202.0, 203.0]) + return DetectorLogic( + trigger=_JustInternal(), + acquire=_FillStatsAcquireLogic(stats, values, times), + data=[TimeSeriesDataLogic(stats)], + ).with_device("det") + + +def test_step_scan_through_run_engine(RE: RunEngine, stats_detector: StandardDetector): + """A writer-less detector describes and reads as internal data. + + Driven through the RunEngine so the descriptor its datakeys make is the one + a real scan would emit, and so has to pass event-model validation. + """ + det = stats_detector + docs: dict[str, list] = defaultdict(list) + + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def plan(): + # prepare() comes after stage(), which resets the detector + yield from bps.prepare(det, TriggerInfo(collections_per_event=4), wait=True) + yield from bps.trigger_and_read([det]) + + RE(plan(), lambda name, doc: docs[name].append(doc)) + + (descriptor,) = docs["descriptor"] + assert list(descriptor["data_keys"]) == PUBLISHED + total = descriptor["data_keys"]["det-stats-total"] + assert total["shape"] == [4] + assert "external" not in total + (event,) = docs["event"] + np.testing.assert_array_equal( + event["data"]["det-stats-total"], [5.0, 6.0, 7.0, 8.0] + ) + assert event["timestamps"]["det-stats-total"] == 203.0 + + +def test_fly_scan_emits_event_pages(RE: RunEngine, stats_detector: StandardDetector): + """A writer-less detector collects its buffer as event pages.""" + det = stats_detector + docs: dict[str, list] = defaultdict(list) + + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def plan(): + yield from bps.prepare(det, TriggerInfo(number_of_events=4), wait=True) + yield from bps.declare_stream(det, name="primary") + yield from bps.kickoff(det, wait=True) + yield from bps.collect_while_completing( + flyers=[det], dets=[det], flush_period=0.05 + ) + + RE(plan(), lambda name, doc: docs[name].append(doc)) + + (page,) = docs["event_page"] + np.testing.assert_array_equal( + page["data"]["det-stats-total"], [[5.0], [6.0], [7.0], [8.0]] + ) + assert docs["stop"][0]["num_events"] == {"primary": 4} + + +async def test_step_scan_read_derives_single_reading(stats_detector: StandardDetector): + """A step-scan read derives one reading holding the whole buffer, timed by + the last point's ts_timestamp.""" + det = stats_detector + await det.stage() + await det.prepare(TriggerInfo(collections_per_event=4)) + await det.trigger() + reading = await det.read() + total = reading["det-stats-total"] + np.testing.assert_array_equal(total["value"], [5.0, 6.0, 7.0, 8.0]) + assert total["timestamp"] == 203.0 + await det.unstage() diff --git a/tests/unit_tests/epics/adcore/test_use_cases.py b/tests/unit_tests/epics/adcore/test_use_cases.py index dd664fb4c9..023ea2d48d 100644 --- a/tests/unit_tests/epics/adcore/test_use_cases.py +++ b/tests/unit_tests/epics/adcore/test_use_cases.py @@ -1,19 +1,34 @@ +import asyncio +from collections import defaultdict +from typing import Annotated as A +from typing import get_type_hints from unittest.mock import ANY +from xml.etree import ElementTree as ET +import bluesky.plan_stubs as bps +import bluesky.preprocessors as bpp +import numpy as np import pytest +from bluesky import RunEngine from ophyd_async.core import ( + Array1D, + Device, EnableDisable, + SignalR, + StandardReadable, StaticPathProvider, TransitionError, TriggerInfo, callback_on_mock_put, + get_mock_put, init_devices, + mock_puts_blocked, set_mock_value, ) from ophyd_async.core import StandardReadableFormat as Format from ophyd_async.epics import adaravis, adcore, adsimdetector -from ophyd_async.epics.core import epics_signal_r +from ophyd_async.epics.core import PvSuffix, epics_signal_r from ophyd_async.testing import assert_reading @@ -24,19 +39,14 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( async with init_devices(mock=True): det = adsimdetector.SimDetector( "PREFIX:", - adcore.ADDataFactory.hdf(static_path_provider), + adcore.ADDataFactory.hdf(static_path_provider, ndattribute_sources=[stat]), plugins={"stats": stat}, ) temp = epics_signal_r(float, "SAMP:TEMP:RBV") # Plugins are not registered automatically; opt this one in det.set_readable_format(stat, Format.CHILD) + # The HDF writer stamps the stats plugin's NDAttributes; the driver stamps these ndattributes = [ - adcore.NDAttributeParam( - name="det-sum", - param="TOTAL", - datatype=adcore.NDAttributeDataType.DOUBLE, - description="Sum of the array", - ), adcore.NDAttributePv( name="sample-temp", signal=temp, @@ -44,7 +54,12 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( description="Temperature of the sample", ), ] - set_mock_value(stat.nd_attributes_file, adcore.ndattributes_to_xml(ndattributes)) + set_mock_value( + det.driver.nd_attributes_file, adcore.ndattributes_to_xml(ndattributes) + ) + set_mock_value(stat.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(stat.compute_statistics, True) + set_mock_value(stat.nd_attributes_file_nelm, 2048) writer = det.get_plugin("hdf", adcore.NDFileHDF5IO) set_mock_value(det.driver.acquire_period, 0.1) set_mock_value(det.driver.acquire_time, 0.05) @@ -94,7 +109,8 @@ async def test_step_scan_hdf_detector_with_stats_and_temp( "shape": [1, 768, 1024], "source": uri, }, - "det-sum": { + # The HDF file serves the stats, under the names of their scalars + "det-stats-total": { "dtype": "number", "dtype_numpy": " None: + """Set the PVs that switch the basic statistics of a stats plugin.""" + set_mock_value( + stats.enable_callbacks, EnableDisable.ENABLE if on else EnableDisable.DISABLE + ) + set_mock_value(stats.compute_statistics, on) + + +def _time_series_arrays(stats: adcore.NDStatsIO) -> list[Device]: + """The array of each statistic of a stats plugin, not its timestamps.""" + hints = get_type_hints(adcore.NDStatsIO) + return [ + signal + for attr, signal in stats.children() + if attr.startswith("ts_") + and attr != "ts_timestamp" + and hints[attr] == SignalR[Array1D[np.float64]] + ] + + +def _fill_time_series( + det, plugins: list[adcore.NDStatsIO], points: int, also=lambda: None +): + """Behave like the IOC: arming a time series erases it, acquiring counts the + points, and the arrays are posted only when TSRead asks for them, as for a + page collected between the plugin's once-a-second posts.""" + + def erase(value): + if value: + for plugin in plugins: + set_mock_value(plugin.ts_current_point, 0) + + def count(value): + if value: + for plugin in plugins: + set_mock_value(plugin.ts_current_point, points) + also() + + def post(plugin: adcore.NDStatsIO): + for array in _time_series_arrays(plugin): + set_mock_value( + array, # pyright: ignore[reportArgumentType] + np.arange(points, dtype=np.float64), + ) + set_mock_value( + plugin.ts_timestamp, # pyright: ignore[reportArgumentType] + 100.0 + np.arange(points), + ) + + for plugin in plugins: + set_mock_value(plugin.ts_timestamp_nelm, 2048) + callback_on_mock_put(plugin.ts_acquire, erase) + callback_on_mock_put(plugin.ts_read, lambda _, plugin=plugin: post(plugin)) + callback_on_mock_put(det.driver.acquire, count) + + +def _fly_plan(det, info: TriggerInfo): + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def plan(): + yield from bps.prepare(det, info, wait=True) + yield from bps.declare_stream(det, name="primary", collect=True) + yield from bps.kickoff(det, wait=True) + yield from bps.collect_while_completing( + flyers=[det], dets=[det], flush_period=0.05 + ) + + return plan() + + +def _fly( + RE: RunEngine, + det, + series: list[adcore.NDStatsIO], + points: int, + hdf: adcore.NDFileHDF5IO | None = None, +) -> dict[str, list]: + also = lambda: None # noqa: E731 + if hdf is not None: + also = lambda: set_mock_value(hdf.num_captured, points) # noqa: E731 + _fill_time_series(det, series, points=points, also=also) + docs: dict[str, list] = defaultdict(list) + RE( + _fly_plan(det, TriggerInfo(number_of_events=points)), + lambda n, d: docs[n].append(d), + ) + return docs + + +async def _superset_detector( + path_provider: StaticPathProvider, always_enable: bool = False +): + """Use case 6: the readables, time series and HDF NDAttributes of one stats + plugin, with its basic statistics switched on.""" + stats = adcore.NDStatsIO("PREFIX:STAT:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.hdf( + path_provider, always_enable=always_enable, ndattribute_sources=[stats] + ), + adcore.ADDataFactory.time_series(stats), + plugins={"stats": stats}, + ) + _switch_stats(stats, on=True) + set_mock_value(stats.nd_attributes_file_nelm, 2048) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + set_mock_value(det.get_plugin("hdf", adcore.NDFileHDF5IO).file_path_exists, True) + return det + + +# Use cases A and B: a detector with a stats plugin in plugins= only, then with its +# time series. A plugin in plugins= is a readable, so its hints come with it. + + +async def test_uc_a_detector_with_readable_stats(RE: RunEngine): + stats = adcore.NDStatsIO("PREFIX:STAT:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stats}, name="det") + _switch_stats(stats, on=True) + set_mock_value(stats.total, 12.0) + + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def step_plan(): + yield from bps.trigger_and_read([det]) + + docs: dict[str, list] = defaultdict(list) + RE(step_plan(), lambda name, doc: docs[name].append(doc)) + + (descriptor,) = docs["descriptor"] + assert list(descriptor["data_keys"]) == STATS_KEYS + assert descriptor["hints"]["det"]["fields"] == STATS_HINTS + (event,) = docs["event"] + assert event["data"]["det-stats-total"] == 12.0 + + +async def test_uc_a_a_plugin_opts_out_of_being_read(): + stats = adcore.NDStatsIO("PREFIX:STAT:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stats}, name="det") + _switch_stats(stats, on=True) + det.set_readable_format(stats, None) + + await det.stage() + await det.prepare(TriggerInfo()) + + assert not [key for key in await det.describe() if "stats" in key] + assert det.hints == {} + + +def test_uc_b_detector_with_a_stats_time_series(RE: RunEngine): + stats = adcore.NDStatsIO("PREFIX:STAT:") + with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.time_series(stats), + plugins={"stats": stats}, + name="det", + ) + _switch_stats(stats, on=True) + + docs = _fly(RE, det, [stats], points=3) + + (descriptor,) = docs["descriptor"] + assert list(descriptor["data_keys"]) == STATS_KEYS + # The time series replaces the readable, whose hints stay + assert descriptor["hints"]["det"]["fields"] == STATS_HINTS + (page,) = docs["event_page"] + assert page["data"]["det-stats-total"] == [[0.0], [1.0], [2.0]] + # The arrays were posted for the page + get_mock_put(stats.ts_read).assert_called_once_with(True) + # It made no plugin of its own + assert [name for name, _ in det.children() if "stat" in name] == ["stats"] + + +@pytest.mark.parametrize( + "always_enable,hdf_enabled,hdf_serves", + [ + # By its PV, HDF on: the file serves, every time series suppressed + (False, EnableDisable.ENABLE, True), + # By its PV, HDF off: the time series serves + (False, EnableDisable.DISABLE, False), + # Always enabled, HDF serves whatever its PV was + (True, EnableDisable.DISABLE, True), + ], +) +async def test_uc6_the_hdf_switch_picks_the_file_or_the_time_series( + RE: RunEngine, + static_path_provider: StaticPathProvider, + always_enable: bool, + hdf_enabled: EnableDisable, + hdf_serves: bool, +): + det = await _superset_detector(static_path_provider, always_enable) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(hdf.enable_callbacks, hdf_enabled) + + docs = _fly(RE, det, [det.get_plugin("stats", adcore.NDStatsIO)], 3, hdf=hdf) + + # The registered stats plugin hints the statistics, whoever serves them + (descriptor,) = docs["descriptor"] + assert descriptor["hints"]["det"]["fields"] == [ + *STATS_HINTS, + *(["det"] if hdf_serves else []), + ] + if hdf_serves: + assert docs["event_page"] == [] + keys = [doc["data_key"] for doc in docs["stream_resource"]] + assert keys == ["det", *STATS_KEYS] + else: + assert docs["stream_resource"] == [] + (page,) = docs["event_page"] + assert list(page["data"]) == STATS_KEYS + assert page["data"]["det-stats-total"] == [[0.0], [1.0], [2.0]] + + +async def test_uc6_hdf_enabled_the_file_serves_every_stats_quantity( + static_path_provider: StaticPathProvider, +): + det = await _superset_detector(static_path_provider) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(hdf.enable_callbacks, EnableDisable.ENABLE) + await det.stage() + callback_on_mock_put( + det.driver.acquire, lambda v: set_mock_value(hdf.num_captured, 1) + ) + await det.trigger() + + description = await det.describe() + + assert description["det-stats-total"]["external"] == "STREAM:" + assert "det-stats-total" not in await det.read() + + +@pytest.fixture +async def two_roi_detector(tmp_path): + """Use case 7: two HDF writers and two time series, one per ROI.""" + path_provider = StaticPathProvider( + lambda datakey_name=None: str(datakey_name), tmp_path + ) + stats1 = adcore.NDStatsIO("PREFIX:STAT1:") + stats2 = adcore.NDStatsIO("PREFIX:STAT2:") + async with init_devices(mock=True): + det = adaravis.AravisDetector( + "PREFIX:", + adcore.ADDataFactory.hdf( + path_provider, + writer_suffix="HDF1:", + writer_name="hdf1", + datakey_suffix="-roi1", + always_enable=False, + ), + adcore.ADDataFactory.hdf( + path_provider, + writer_suffix="HDF2:", + writer_name="hdf2", + datakey_suffix="-roi2", + always_enable=False, + ), + adcore.ADDataFactory.time_series(stats1), + adcore.ADDataFactory.time_series(stats2), + plugins={"stats1": stats1, "stats2": stats2}, + ) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + for name in ("hdf1", "hdf2"): + plugin = det.get_plugin(name, adcore.NDFileHDF5IO) + set_mock_value(plugin.enable_callbacks, EnableDisable.DISABLE) + set_mock_value(plugin.file_path_exists, True) + return det + + +@pytest.mark.parametrize( + "stats1_on,stats2_on", + [(True, True), (False, True)], +) +def test_uc7_hdf_disabled_the_time_series_switched_on_serve_in_one_page( + RE: RunEngine, two_roi_detector, stats1_on: bool, stats2_on: bool +): + det = two_roi_detector + series = [det.get_plugin(n, adcore.NDStatsIO) for n in ("stats1", "stats2")] + _switch_stats(series[0], stats1_on) + _switch_stats(series[1], stats2_on) + + docs = _fly(RE, det, series, points=2) + + assert docs["stream_resource"] == [] + (page,) = docs["event_page"] + totals = [ + f"det-{name}-total" + for name, on in (("stats1", stats1_on), ("stats2", stats2_on)) + if on + ] + assert [key for key in page["data"] if key.endswith("-total")] == totals + for total in totals: + assert page["data"][total] == [[0.0], [1.0]] + + +# Ruling 1: a once-per-event readable is suppressed by a same-named higher tier at +# any collections_per_event, and read once per event when nothing suppresses it + + +@pytest.mark.parametrize("cpe", [1, 2]) +@pytest.mark.parametrize("hdf_enabled", [False, True]) +@pytest.mark.parametrize("bounded", [False, True]) +async def test_readable_stats_total_is_suppressed_by_any_higher_tier( + static_path_provider: StaticPathProvider, + bounded: bool, + hdf_enabled: bool, + cpe: int, +): + """readable total < time series total (bounded) < HDF NDAttribute (unbounded).""" + stats = adcore.NDStatsIO("PREFIX:STAT:") + factories: list[adcore.ADDataFactory] = [ + adcore.ADDataFactory.hdf( + static_path_provider, always_enable=False, ndattribute_sources=[stats] + ) + ] + if bounded: + factories.append(adcore.ADDataFactory.time_series(stats)) + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", *factories, plugins={"stats": stats}, name="det" + ) + _switch_stats(stats, on=True) + set_mock_value(stats.ts_timestamp_nelm, 2048) + set_mock_value(stats.nd_attributes_file_nelm, 2048) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value( + hdf.enable_callbacks, + EnableDisable.ENABLE if hdf_enabled else EnableDisable.DISABLE, + ) + set_mock_value(hdf.file_path_exists, True) + set_mock_value(det.driver.array_size_x, 1024) + set_mock_value(det.driver.array_size_y, 768) + await det.stage() + info = TriggerInfo(number_of_events=1, collections_per_event=cpe) + + await det.prepare(info) + total = (await det.describe())["det-stats-total"] + if bounded or hdf_enabled: + # Suppressed: the key now describes a [cpe] collections, not one value + assert total["shape"] == [cpe] + else: + assert total["shape"] == [] @pytest.mark.parametrize("cpe", [1, 2]) @@ -818,18 +1197,113 @@ async def test_readable_with_no_data_logic_is_read_once_per_event(cpe: int): stat = adcore.NDStatsIO("PREFIX:STATS:") async with init_devices(mock=True): det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stat}, name="det") - det.set_readable_format(stat.total, Format.UNCACHED_SIGNAL) + _switch_stats(stat, on=True) await det.stage() await det.prepare(TriggerInfo(number_of_events=1, collections_per_event=cpe)) assert (await det.describe())["det-stats-total"]["shape"] == [] -@pytest.mark.parametrize("on_at_stage", [True, False]) -async def test_hdf_by_settings_follows_its_plugin_at_stage_for_the_whole_stage( - tmp_path, on_at_stage: bool +@pytest.mark.parametrize( + "enabled,computed,published", + [ + (EnableDisable.ENABLE, True, True), + # The plugin computes nothing while disabled + (EnableDisable.DISABLE, True, False), + (EnableDisable.ENABLE, False, False), + ], +) +async def test_stats_scalar_follows_its_switches_at_first_prepare( + enabled: EnableDisable, computed: bool, published: bool ): - """Switching a writer's plugin mid-stage changes nothing until restaged.""" + """The switches at the first prepare() decide which scalars are read.""" + stat = adcore.NDStatsIO("PREFIX:STATS:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stat}, name="det") + set_mock_value(stat.enable_callbacks, enabled) + set_mock_value(stat.compute_statistics, computed) + + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + + assert ("det-stats-total" in await det.describe()) is published + assert ("det-stats-total" in await det.read()) is published + assert ("det-stats-total" in det.hints.get("fields", [])) is published + await det.unstage() + + +@pytest.mark.parametrize( + "changed", + [ + lambda stat: set_mock_value(stat.compute_statistics, False), + lambda stat: set_mock_value(stat.enable_callbacks, EnableDisable.DISABLE), + ], + ids=["compute", "callbacks"], +) +async def test_prepare_raises_for_a_switch_changed_since_the_first_prepare(changed): + stat = adcore.NDStatsIO("PREFIX:STATS:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stat}, name="det") + _switch_stats(stat, on=True) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + changed(stat) + + with pytest.raises( + RuntimeError, match=r"changed since the first prepare.*det-stats-total" + ): + await det.prepare(TriggerInfo()) + + # Staging again takes the new value, and nothing was written to the PV + await det.unstage() + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + assert "det-stats-total" not in await det.read() + get_mock_put(stat.compute_statistics).assert_not_called() + + +async def test_retry_after_hidden_set_change_is_a_fresh_first_prepare(): + stat = adcore.NDStatsIO("PREFIX:STATS:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", plugins={"stats": stat}, name="det") + _switch_stats(stat, on=True) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + set_mock_value(stat.compute_statistics, False) + + with pytest.raises(RuntimeError, match=r"changed since the first prepare"): + await det.prepare(TriggerInfo()) + # The failed prepare() left no recording, so the retry is a first prepare + # that takes the changed value as its new baseline + await det.prepare(TriggerInfo()) + + +@pytest.mark.parametrize("exposures_per_collection", [1, 2]) +async def test_the_process_plugin_is_switched_on_by_the_acquire_logic( + exposures_per_collection: int, +): + """The detector made the process plugin, so prepare() switches it on.""" + async with init_devices(mock=True): + det = adsimdetector.SimDetector("PREFIX:", proc_suffix="PROC:", name="det") + proc = det.get_plugin("proc", adcore.NDProcessIO) + set_mock_value(proc.enable_callbacks, EnableDisable.DISABLE) + + await det.stage() + await det.prepare(TriggerInfo(exposures_per_collection=exposures_per_collection)) + await det.trigger() + + assert await proc.enable_callbacks.get_value() is EnableDisable.ENABLE + + +@pytest.mark.parametrize("on_at_first_prepare", [True, False]) +async def test_hdf_by_settings_follows_its_plugin_at_first_prepare( + tmp_path, on_at_first_prepare: bool +): + """The writer's plugin at the first prepare() decides whether the writer serves.""" path_provider = StaticPathProvider( lambda datakey_name=None: str(datakey_name), tmp_path ) @@ -863,22 +1337,594 @@ def acquire(value: bool) -> None: set_mock_value(hdf.num_captured, frames) callback_on_mock_put(det.driver.acquire, acquire) - served = {"det-roi1", "det-roi2"} if on_at_stage else {"det-roi1"} + served = {"det-roi1", "det-roi2"} if on_at_first_prepare else {"det-roi1"} info = TriggerInfo(number_of_events=3) def enable_hdf2(enabled: bool) -> None: value = EnableDisable.ENABLE if enabled else EnableDisable.DISABLE set_mock_value(hdf2.enable_callbacks, value) - enable_hdf2(on_at_stage) + enable_hdf2(on_at_first_prepare) await det.stage() - for flip in (False, True): - if flip: - enable_hdf2(not on_at_stage) - await det.prepare(info) - await det.kickoff() - await det.complete() - assert set(await det.describe_collect()) == served - # Events since prepare() - assert await det.get_index() == info.number_of_events + await det.prepare(info) + await det.kickoff() + await det.complete() + assert set(await det.describe_collect()) == served + # Events since prepare() + assert await det.get_index() == info.number_of_events + await det.unstage() + + +# Use case 8: a stats plugin per ROI, each stamped into the file of its own HDF +# writer. A writer stamps only the NDAttribute sources listed for it. + + +def _publish_only_total(stats: adcore.NDStatsIO) -> None: + """Leave the total as the only statistic with a read format.""" + for signal, format in stats.get_readable_formats(): + if signal is not stats.total and format is not Format.CONFIG_SIGNAL: + stats.set_readable_format(signal, None) + + +def _roi_array_description(roi: adcore.NDROIIO): + return lambda driver: adcore.NDArrayDescription( + shape_signals=(roi.size_y, roi.size_x), + data_type_signal=driver.data_type, + color_mode_signal=driver.color_mode, + ) + + +# Neither stats plugin is a plugins= entry: each is registered on its own ROI, +# so the suppress logic must walk the readable tree to hide roi2's switched-off +# total +@pytest.mark.parametrize("roi2_computes_stats", [True, False]) +async def test_uc8_2_rois_each_with_stats_and_hdf( + RE: RunEngine, tmp_path, roi2_computes_stats: bool +): + path_provider = StaticPathProvider( + lambda datakey_name=None: str(datakey_name), tmp_path + ) + rois = [adcore.NDROIIO(f"PREFIX:ROI{i}:") for i in (1, 2)] + for i, roi in enumerate(rois, start=1): + roi.stat = adcore.NDStatsIO(f"PREFIX:STAT{i}:") + _publish_only_total(roi.stat) + roi1, roi2 = rois + async with init_devices(mock=True): + det = adaravis.AravisDetector( + "PREFIX:", + *( + adcore.ADDataFactory.hdf( + path_provider, + writer_suffix=f"HDF{i}:", + writer_name=f"hdf{i}", + datakey_suffix=f"-roi{i}", + array_description=_roi_array_description(roi), + ndattribute_sources=[roi.stat], + ) + for i, roi in enumerate(rois, start=1) + ), + plugins={"roi1": roi1, "roi2": roi2}, + ) + hdfs = [det.get_plugin(f"hdf{i}", adcore.NDFileHDF5IO) for i in (1, 2)] + for roi in rois: + roi.set_readable_format(roi.stat, Format.CHILD) + for roi in rois: + _switch_stats(roi.stat, on=True) + set_mock_value(roi.stat.nd_attributes_file_nelm, 2048) + set_mock_value(roi.size_x, 40) + set_mock_value(roi.size_y, 30) + set_mock_value(roi2.stat.compute_statistics, roi2_computes_stats) + for hdf in hdfs: + set_mock_value(hdf.file_path_exists, True) + + def capture_three_frames(value: bool) -> None: + if value: + for hdf in hdfs: + set_mock_value(hdf.num_captured, 3) + + callback_on_mock_put(det.driver.acquire, capture_three_frames) + docs: dict[str, list] = defaultdict(list) + + RE( + _fly_plan(det, TriggerInfo(number_of_events=3)), + lambda name, doc: docs[name].append(doc), + ) + + # Each writer's file has its own ROI and the totals of its own stats plugin + uris = {doc["data_key"]: doc["uri"] for doc in docs["stream_resource"]} + expected = { + "det-roi1": ANY, + "det-roi1-stat-total": uris["det-roi1"], + "det-roi2": ANY, + } + if roi2_computes_stats: + expected["det-roi2-stat-total"] = uris["det-roi2"] + assert uris == expected + assert uris["det-roi1"] != uris["det-roi2"] + (descriptor,) = docs["descriptor"] + assert set(descriptor["data_keys"]) == set(uris) + # The totals stay hinted by the stats plugins that the files replace + expected_hints = ["det-roi1-stat-total"] + if roi2_computes_stats: + expected_hints.append("det-roi2-stat-total") + assert descriptor["hints"]["det"]["fields"] == [ + *expected_hints, + "det-roi1", + "det-roi2", + ] + + +# Use case 9: an ROIStat plugin, modelled on the test-rig spectroscopy plans +# (DRV -> ROISTAT -> HDF, three channels named Blue, Green and Red, only their +# totals published). The HDF writer lists the plugin as an NDAttribute source, +# so when it serves the totals are in its file, and when it sits out they are +# read from the channels. + +CHANNEL_NAMES = {1: "Blue", 2: "Green", 3: "Red"} + + +async def _roistat_detector( + tmp_path, hdf_on: bool, channel_names: dict[int, str] = CHANNEL_NAMES +): + path_provider = StaticPathProvider( + lambda datakey_name=None: str(datakey_name), tmp_path + ) + roistat = adcore.NDROIStatIO("PREFIX:ROISTAT:", num_channels=len(channel_names)) + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + adcore.ADDataFactory.hdf( + path_provider, always_enable=False, ndattribute_sources=[roistat] + ), + plugins={"roistat": roistat}, + name="det", + ) + set_mock_value(roistat.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(roistat.nd_attributes_file_nelm, 2048) + for number, name in channel_names.items(): + channel = roistat.channels[number] + set_mock_value(channel.name_, name) + set_mock_value(channel.use, True) + set_mock_value(channel.total, 10.0 * number) + for scalar in (channel.min_value, channel.max_value, channel.mean_value): + channel.set_readable_format(scalar, None) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value( + hdf.enable_callbacks, EnableDisable.ENABLE if hdf_on else EnableDisable.DISABLE + ) + set_mock_value(hdf.file_path_exists, True) + set_mock_value(det.driver.array_size_x, 64) + set_mock_value(det.driver.array_size_y, 48) + return det, roistat, hdf + + +def _scan_roistat_detector(RE: RunEngine, det, hdf: adcore.NDFileHDF5IO, hdf_on: bool): + """Run a fly scan if the HDF writer serves it, otherwise a step scan.""" + docs: dict[str, list] = defaultdict(list) + if hdf_on: + callback_on_mock_put( + det.driver.acquire, + lambda value: set_mock_value(hdf.num_captured, 3) if value else None, + ) + plan = _fly_plan(det, TriggerInfo(number_of_events=3)) + else: + + @bpp.stage_decorator([det]) + @bpp.run_decorator() + def step_plan(): + yield from bps.trigger_and_read([det]) + + plan = step_plan() + RE(plan, lambda name, doc: docs[name].append(doc)) + return docs + + +@pytest.mark.parametrize("hdf_on", [True, False]) +async def test_uc9_roistat_totals_are_in_the_file_or_read_from_the_channels( + RE: RunEngine, tmp_path, hdf_on: bool +): + det, roistat, hdf = await _roistat_detector(tmp_path, hdf_on) + # Before stage the channels have their default names + assert roistat.channels[1].total.name == "det-roistat-channels-1-total" + + docs = _scan_roistat_detector(RE, det, hdf, hdf_on) + + totals = [f"det-roistat-{name}-total" for name in CHANNEL_NAMES.values()] + (descriptor,) = docs["descriptor"] + assert set(descriptor["data_keys"]) >= set(totals) + # Hinted whether they are in the file or read, along with the image if filed + assert descriptor["hints"]["det"]["fields"] == ( + [*totals, "det"] if hdf_on else totals + ) + if hdf_on: + assert {doc["data_key"] for doc in docs["stream_resource"]} == {"det", *totals} + assert all(key in descriptor["data_keys"] for key in totals) + assert ( + descriptor["data_keys"]["det-roistat-Blue-total"]["external"] == "STREAM:" + ) + # The plugin was told to stamp each total from its own channel's address + xml = await roistat.nd_attributes_file.get_value() + stamped = [ + ( + a.attrib["name"], + a.attrib["source"], + a.attrib["addr"], + a.attrib["datatype"], + ) + for a in ET.fromstring(xml) + ] + assert stamped == [ + ("det-roistat-Blue-total", "ROISTAT_TOTAL", "0", "DOUBLE"), + ("det-roistat-Green-total", "ROISTAT_TOTAL", "1", "DOUBLE"), + ("det-roistat-Red-total", "ROISTAT_TOTAL", "2", "DOUBLE"), + ] + else: + assert docs["stream_resource"] == [] + (event,) = docs["event"] + assert {key: event["data"][key] for key in totals} == { + "det-roistat-Blue-total": 10.0, + "det-roistat-Green-total": 20.0, + "det-roistat-Red-total": 30.0, + } + + +@pytest.mark.parametrize("hdf_on", [True, False]) +async def test_uc9_a_roistat_channel_that_is_not_used_disappears( + RE: RunEngine, tmp_path, hdf_on: bool +): + det, roistat, hdf = await _roistat_detector(tmp_path, hdf_on) + set_mock_value(roistat.channels[2].use, False) + + docs = _scan_roistat_detector(RE, det, hdf, hdf_on) + + (descriptor,) = docs["descriptor"] + assert "det-roistat-Blue-total" in descriptor["data_keys"] + assert "det-roistat-Red-total" in descriptor["data_keys"] + assert not [key for key in descriptor["data_keys"] if "Green" in key] + assert not [key for key in descriptor["hints"]["det"]["fields"] if "Green" in key] + if hdf_on: + assert not [ + doc for doc in docs["stream_resource"] if "Green" in doc["data_key"] + ] + else: + (event,) = docs["event"] + assert not [key for key in event["data"] if "Green" in key] + # Its name was still taken from the PV + assert roistat.channels[2].total.name == "det-roistat-Green-total" + + +async def test_uc9_roistat_channel_names_are_sanitised_at_stage( + tmp_path, +): + names = {1: "Ice Cream", 2: "", 3: "r.o/i-3", 4: "keep_Case9"} + det, roistat, _ = await _roistat_detector(tmp_path, False, names) + + await det.stage() + + assert [roistat.channels[number].name for number in names] == [ + "det-roistat-Ice_Cream", + "det-roistat-2", + "det-roistat-r_o_i_3", + "det-roistat-keep_Case9", + ] + assert roistat.channels[1].total.name == "det-roistat-Ice_Cream-total" + + +async def test_uc9_roistat_channels_with_the_same_sanitised_name_raise_at_stage( + tmp_path, +): + det, _, _ = await _roistat_detector(tmp_path, False, {1: "A B", 2: "A_B", 3: "C"}) + + with pytest.raises(ValueError, match=r"more than one channel is named \['A_B'\]"): + await det.stage() + + +async def test_uc9_a_roistat_channel_renamed_after_stage_is_renamed_at_the_next( + tmp_path, +): + det, roistat, _ = await _roistat_detector(tmp_path, False) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + set_mock_value(roistat.channels[3].name_, "Infrared") + + await det.prepare(TriggerInfo()) + assert roistat.channels[3].total.name == "det-roistat-Red-total" + + await det.unstage() + await det.stage() + assert roistat.channels[3].total.name == "det-roistat-Infrared-total" + + +async def test_uc9_prepare_raises_for_a_roistat_channel_used_since_the_first( + tmp_path, +): + det, roistat, _ = await _roistat_detector(tmp_path, False) + set_mock_value(roistat.channels[3].use, False) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + set_mock_value(roistat.channels[3].use, True) + + with pytest.raises( + RuntimeError, match=r"changed since the first prepare.*det-roistat-Red-total" + ): + await det.prepare(TriggerInfo()) + + +# Use case 10: the time series of an ROIStat plugin, for a detector that writes no +# file. Each used channel publishes one array per statistic with a read format, +# named after the scalar its channel already reads, which it suppresses. + + +async def _roistat_time_series_detector( + path_provider: StaticPathProvider | None = None, +): + """Three channels named Blue, Green and Red, all used; with an HDF writer + stamping the plugin's NDAttributes if given a path provider.""" + roistat = adcore.NDROIStatIO("PREFIX:ROISTAT:", num_channels=len(CHANNEL_NAMES)) + hdf_factories = ( + [ + adcore.ADDataFactory.hdf( + path_provider, always_enable=False, ndattribute_sources=[roistat] + ) + ] + if path_provider + else [] + ) + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", + *hdf_factories, + adcore.ADDataFactory.time_series(roistat), + plugins={"roistat": roistat}, + name="det", + ) + set_mock_value(roistat.enable_callbacks, EnableDisable.ENABLE) + set_mock_value(roistat.nd_attributes_file_nelm, 2048) + for number, name in CHANNEL_NAMES.items(): + set_mock_value(roistat.channels[number].ts_timestamp_nelm, 2048) + set_mock_value(roistat.channels[number].name_, name) + set_mock_value(roistat.channels[number].use, True) + return det, roistat + + +def _fly_roistat_time_series( + RE: RunEngine, det, roistat: adcore.NDROIStatIO, points: int, also=lambda: None +) -> dict[str, list]: + """Behave like the IOC: ERASE_START resets the count, acquiring counts the + points, and READ posts each channel's arrays, filled with + 10 * channel number + frame number, as for a page collected between the + plugin's once-a-second posts. The timestamps differ per channel, which the + IOC's do not, to show which channel's are used.""" + + def post(): + for number, channel in roistat.channels.items(): + for array in [ + channel.ts_min_value, + channel.ts_max_value, + channel.ts_mean_value, + channel.ts_total, + channel.ts_net, + ]: + set_mock_value( + array, # pyright: ignore[reportArgumentType] + 10.0 * number + np.arange(points), + ) + set_mock_value( + channel.ts_timestamp, # pyright: ignore[reportArgumentType] + 100.0 * number + np.arange(points), + ) + + def control(value): + if value == adcore.NDROIStatTSControl.ERASE_START: + set_mock_value(roistat.ts_current_point, 0) + elif value == adcore.NDROIStatTSControl.READ: + post() + + def count(value): + if value: + set_mock_value(roistat.ts_current_point, points) + also() + + callback_on_mock_put(roistat.ts_control, control) + callback_on_mock_put(det.driver.acquire, count) + docs: dict[str, list] = defaultdict(list) + RE( + _fly_plan(det, TriggerInfo(number_of_events=points)), + lambda name, doc: docs[name].append(doc), + ) + return docs + + +async def test_uc10_roistat_time_series_publishes_each_used_channel(RE: RunEngine): + det, roistat = await _roistat_time_series_detector() + # Green is not used, Red publishes only its total, and Blue its net too + set_mock_value(roistat.channels[2].use, False) + red = roistat.channels[3] + for scalar in (red.min_value, red.max_value, red.mean_value): + red.set_readable_format(scalar, None) + blue = roistat.channels[1] + blue.set_readable_format(blue.net, Format.UNCACHED_SIGNAL) + + docs = _fly_roistat_time_series(RE, det, roistat, points=3) + + blue_keys = [ + f"det-roistat-Blue-{stat}" + for stat in ("min_value", "max_value", "mean_value", "total", "net") + ] + keys = [*blue_keys, "det-roistat-Red-total"] + (descriptor,) = docs["descriptor"] + # The arrays replace the scalars of the same name, and the unused channel is gone + assert list(descriptor["data_keys"]) == keys + total = descriptor["data_keys"]["det-roistat-Blue-total"] + assert total["source"] == "mock+ca://PREFIX:ROISTAT:1:TSTotal" + assert total["shape"] == [1] + net = descriptor["data_keys"]["det-roistat-Blue-net"] + assert net["source"] == "mock+ca://PREFIX:ROISTAT:1:TSNet" + # The channels hint their scalars, whoever serves them, and net is not hinted + hinted = [key for key in keys if key != "det-roistat-Blue-net"] + assert descriptor["hints"]["det"]["fields"] == hinted + assert docs["event"] == [] + (page,) = docs["event_page"] + assert list(page["data"]) == keys + assert page["data"]["det-roistat-Blue-total"] == [[10.0], [11.0], [12.0]] + assert page["data"]["det-roistat-Blue-net"] == [[10.0], [11.0], [12.0]] + assert page["data"]["det-roistat-Red-total"] == [[30.0], [31.0], [32.0]] + # Every array is timed by the first used channel's timestamps + assert page["timestamps"]["det-roistat-Red-total"] == [100.0, 101.0, 102.0] + # The buffer was sized to the scan, then erased and started, its arrays + # posted for the page, and stopped on unstage + assert get_mock_put(roistat.ts_num_points).call_args_list[-1].args == (3,) + assert [ + call.args[0] for call in get_mock_put(roistat.ts_control).call_args_list + ] == [ + adcore.NDROIStatTSControl.STOP, + adcore.NDROIStatTSControl.ERASE_START, + adcore.NDROIStatTSControl.READ, + adcore.NDROIStatTSControl.STOP, + ] + + +async def test_uc10_roistat_time_series_is_erased_only_once_sized(): + det, roistat = await _roistat_time_series_detector() + callback_on_mock_put( + roistat.ts_control, + lambda value: set_mock_value(roistat.ts_current_point, 0), + ) + await det.stage() + controls = get_mock_put(roistat.ts_control) + controls.reset_mock() + + with mock_puts_blocked(roistat.ts_num_points): + status = det.prepare(TriggerInfo(number_of_events=3)) + while not get_mock_put(roistat.ts_num_points).called: + await asyncio.sleep(0) + await asyncio.sleep(0.01) + # The plugin keeps its current point when resized, so not erased yet + assert not controls.called + await status + + assert [call.args[0] for call in controls.call_args_list] == [ + adcore.NDROIStatTSControl.ERASE_START + ] + + +@pytest.mark.parametrize("points,raises", [(2048, False), (2049, True)]) +async def test_uc10_prepare_raises_for_a_scan_longer_than_the_roistat_buffer( + points: int, raises: bool +): + det, roistat = await _roistat_time_series_detector() + # The arrays hold NCHANS points + set_mock_value(roistat.channels[1].ts_timestamp_nelm, 2048) + await det.stage() + + prepare = det.prepare(TriggerInfo(number_of_events=points)) + + if raises: + with pytest.raises( + ValueError, match="needs 2049 points, but the plugin's arrays hold 2048" + ): + await prepare + else: + await prepare + + +@pytest.mark.parametrize( + "enable_callbacks,used,serves", + [ + (EnableDisable.ENABLE, True, True), + # Off at the first prepare(): the time series sits out, and the scalars go + (EnableDisable.DISABLE, True, False), + (EnableDisable.ENABLE, False, False), + ], +) +async def test_uc10_roistat_time_series_follows_the_switches_at_first_prepare( + enable_callbacks: EnableDisable, used: bool, serves: bool +): + det, roistat = await _roistat_time_series_detector() + set_mock_value(roistat.enable_callbacks, enable_callbacks) + for channel in roistat.channels.values(): + set_mock_value(channel.use, used) + callback_on_mock_put( + roistat.ts_control, + lambda value: set_mock_value(roistat.ts_current_point, 0), + ) + + await det.stage() + await det.prepare(TriggerInfo(number_of_events=2)) + + shapes = { + key: datakey["shape"] + for key, datakey in (await det.describe()).items() + if "roistat" in key + } + assert shapes == ( + { + f"det-roistat-{name}-{stat}": [1] + for name in CHANNEL_NAMES.values() + for stat in ("min_value", "max_value", "mean_value", "total") + } + if serves + else {} + ) + erase_starts = [ + call + for call in get_mock_put(roistat.ts_control).call_args_list + if call.args[0] == adcore.NDROIStatTSControl.ERASE_START + ] + assert len(erase_starts) == (1 if serves else 0) + + +@pytest.mark.parametrize("hdf_enabled", [True, False]) +async def test_uc10_the_hdf_switch_picks_the_file_or_the_roistat_time_series( + RE: RunEngine, + static_path_provider: StaticPathProvider, + hdf_enabled: bool, +): + det, roistat = await _roistat_time_series_detector(static_path_provider) + hdf = det.get_plugin("hdf", adcore.NDFileHDF5IO) + set_mock_value(hdf.file_path_exists, True) + set_mock_value(det.driver.array_size_x, 64) + set_mock_value(det.driver.array_size_y, 48) + set_mock_value( + hdf.enable_callbacks, + EnableDisable.ENABLE if hdf_enabled else EnableDisable.DISABLE, + ) + for channel in roistat.channels.values(): + for scalar in (channel.min_value, channel.max_value, channel.mean_value): + channel.set_readable_format(scalar, None) + + docs = _fly_roistat_time_series( + RE, det, roistat, points=2, also=lambda: set_mock_value(hdf.num_captured, 2) + ) + + totals = [f"det-roistat-{name}-total" for name in CHANNEL_NAMES.values()] + if hdf_enabled: + assert docs["event_page"] == [] + keys = [doc["data_key"] for doc in docs["stream_resource"]] + assert keys == ["det", *totals] + else: + assert docs["stream_resource"] == [] + (page,) = docs["event_page"] + assert list(page["data"]) == totals + + +class _CustomPlugin(StandardReadable, adcore.NDPluginBaseIO): + value: A[SignalR[float], PvSuffix("Value_RBV"), Format.HINTED_UNCACHED_SIGNAL] + + +async def test_a_custom_readable_plugin_keeps_its_readable_after_prepare(): + custom = _CustomPlugin("PREFIX:CUSTOM:") + async with init_devices(mock=True): + det = adsimdetector.SimDetector( + "PREFIX:", plugins={"custom": custom}, name="det" + ) + set_mock_value(custom.value, 3.0) + await det.stage() + await det.prepare(TriggerInfo()) + await det.trigger() + + assert "det-custom-value" in await det.describe() + assert (await det.read())["det-custom-value"]["value"] == 3.0 diff --git a/tests/unit_tests/epics/test_areadetector_subclass_naming.py b/tests/unit_tests/epics/test_areadetector_subclass_naming.py index 496d71cb4f..576bb98a6c 100644 --- a/tests/unit_tests/epics/test_areadetector_subclass_naming.py +++ b/tests/unit_tests/epics/test_areadetector_subclass_naming.py @@ -42,7 +42,8 @@ def check_name(name: str, device: Device): # remove trailing underscore from name, # used to resolve clashes with Bluesky terms name = name[:-1] if name.endswith("_") else name - assert inflection.underscore(pv) == name + # A field of a record, such as TSTimestamp.NELM, is named record_field + assert inflection.underscore(pv.replace(".", "_")) == name elif isinstance(device, DeviceVector): return else: diff --git a/tests/unit_tests/epics/test_default_trigger_info.py b/tests/unit_tests/epics/test_default_trigger_info.py index 10b09c814d..fe6a4fd9da 100644 --- a/tests/unit_tests/epics/test_default_trigger_info.py +++ b/tests/unit_tests/epics/test_default_trigger_info.py @@ -136,5 +136,6 @@ async def test_trigger_uses_num_images_and_process_plugin( shape_0 = await _trigger_with_preserved_state(det, expected_collections) assert shape_0 == expected_collections - assert await proc.num_filter.get_value() == expected_exposures + # Filtering is switched off, not set to 1 exposure, when it is disabled + assert await proc.num_filter.get_value() == num_filter assert await num_signal.get_value() == expected_collections * expected_exposures diff --git a/tests/unit_tests/plan_stubs/test_ndattributes.py b/tests/unit_tests/plan_stubs/test_ndattributes.py index 8695c67415..329ee55f2a 100644 --- a/tests/unit_tests/plan_stubs/test_ndattributes.py +++ b/tests/unit_tests/plan_stubs/test_ndattributes.py @@ -2,23 +2,11 @@ import pytest -from ophyd_async.core import StaticPathProvider, init_devices -from ophyd_async.epics import adcore, adsimdetector +from ophyd_async.core import init_devices +from ophyd_async.epics import adcore from ophyd_async.epics.core import epics_signal_r -def test_setup_ndstats_raises_type_error(RE, static_path_provider: StaticPathProvider): - with init_devices(mock=True): - det = adsimdetector.SimDetector( - "PREFIX:", adcore.ADDataFactory.hdf(static_path_provider) - ) - with pytest.raises( - AttributeError, - match="det has no plugin named 'stats'", - ): - RE(adcore.setup_ndstats_sum(det)) - - async def test_nd_attributes_plan_stub(RE): async with init_devices(mock=True): stat = adcore.NDStatsIO("PREFIX:STATS:") @@ -67,3 +55,11 @@ async def test_nd_attributes_plan_stub_gives_correct_error(RE, arg): + "Expected NDAttributePv or NDAttributeParam.", ): RE(adcore.setup_ndattributes(stat, [arg])) + + +def test_empty_description_is_left_out_of_the_xml(): + param = adcore.NDAttributeParam( + name="det-sum", param="sum", datatype=adcore.NDAttributeDataType.DOUBLE + ) + xml = ET.fromstring(adcore.ndattributes_to_xml([param])) + assert "description" not in xml[0].attrib