Make readable format a runtime setting keyed by Device

CHANGELOG.md

@@ -27,6 +27,9 @@
 - `SimMotor.stage()` now calls `SimMotorLogic.ensure_ready()`, which calls `ensure_stopped()` and so stops the simulated motion, waits for it to finish, and sets the setpoint to the current readback. `unstage()` cancels a fly in progress, then calls `ensure_stopped()`. The simulated motion runs in the task of the verb that moves, so `stop()` during a `prepare()` or fly, or `stage()` during a `set()`, cancels that verb, which raises `CancelledError`.
 - The progress of `StandardFlyable.prepare()` now comes from the `WatcherUpdate`s the logic's `prepare_to_fly` yields, replacing the heuristic that watched setpoint writes (#1448).
 - `WatchableAsyncStatus` closes the async generator it drives as soon as it is cancelled or fails, rather than when the generator is collected, and a watcher that raises is logged and ignored rather than failing the status.
+- `StandardReadable` now keeps one format per registered child, keyed by the child, so registering a child again replaces its format instead of adding a second contribution, and the order of `hints` is the order children were first registered. The private `_describe_config_funcs`, `_read_config_funcs` and `_has_hints` tuples are gone: a subclass that added to them now adds a hook to `_describe_configuration_funcs`, `_read_configuration_funcs` or `_hints_funcs` (see ADR 0023).
+- The deprecated `ConfigSignal` and `HintedSignal` aliases are removed from `ophyd_async.core`; use `StandardReadableFormat.CONFIG_SIGNAL`, `HINTED_SIGNAL` or `HINTED_UNCACHED_SIGNAL`. `StandardReadable.add_readables()` is deprecated in favour of `set_readable_format()` on each device.
+- `store_settings()` writes the readable format changes of the Device tree under the reserved key `<READABLE_FORMATS>`, and `apply_settings()` resets the tree to its class formats before applying them. Settings files stored before this have no such key and leave formats alone. `retrieve_settings()` warns about and drops any other `<...>` key.
 
 ### Added
 
@@ -38,6 +41,9 @@
 - `cancel_and_wait()` cancels tasks and waits for all of them to end, even through a cancel of its caller, then raises the error of the first task, in the order given, that ended with one. If its caller was cancelled, it raises `CancelledError` instead.
 - `plugins_with_proc()` adds the process plugin made from a `proc_suffix` to a detector's `plugins`, and raises `ValueError` if `plugins` already has a `"proc"`. Every areaDetector vendor detector and `ContAcqDetector` uses it, so `"proc"` is reserved: pass `proc_suffix` instead.
 - `Logic`, the base of `MovableLogic` and `FlyableLogic`, with the abstract `ensure_stopped()`. `FlyableLogic` adds `ensure_ready()`, which defaults to it.
+- `StandardReadable.set_readable_format()`, `reset_readable_formats()`, `get_readable_formats()` and `get_readable_format_changes()` (which yield each child with its format), to change how a child contributes to `read()`, `read_configuration()` and `hints` at runtime, the equivalent of `Kind` in ophyd v1 (see "How to change what a Device reports at runtime").
+- `ReadableEntry`, `ReadableFormats`, `walk_readable_formats()`, `apply_readable_formats()` and `READABLE_FORMATS_KEY` in `ophyd_async.core`, and a `readable_formats` field on `Settings`, so formats are stored and retrieved with the signal values.
+- pyright now runs with `reportMatchNotExhaustive`.
 
 ### Fixed
 

docs/explanations/decisions/0023-runtime-readable-format.md

@@ -0,0 +1,125 @@
+# 23. Make readable format a runtime setting keyed by Device
+
+Date: 2026-08-18
+
+## Status
+
+Accepted
+
+## Context
+
+In ophyd v1 a `Component`'s `kind` could be changed at runtime, so one Device could serve
+several techniques: a monochromator's energy is configuration for one and a hinted read for
+another. `StandardReadable` had no equivalent: `add_readables` appended bound methods into
+tuples that did not record which device each came from, so nothing could be removed or
+re-formatted afterwards (#1394).
+
+In #1395 `StandardDetector` had its own `add_config_signals`, and reviewers would not have
+areaDetector plugins registered automatically: a detector carries far more plugins than are
+wired into its chain, so reporting all of them hides which values mean anything. Per-plugin
+opt-in after construction is the primitive #1394 asks for.
+
+Three designs were considered:
+
+- **Chain parsing**, registering what the plugin graph wires up. Rejected as too large, and
+  it fails once plugins cross IOCs, as every plugin would have to be exposed to ophyd.
+- **One registry for formats and logic objects.** Rejected: a format is keyed by child,
+  unbounded in number and merged by union, where a logic is keyed by role, one per role, and
+  a conflict is an error, so the registry would branch on entry kind at every insert.
+  Changing a format alters `describe()`; changing a logic, the protocols the Device
+  satisfies.
+- **A format registry keyed by device**, leaving logic objects alone. Chosen.
+
+## Decision
+
+### The registry is a `{device: format}` dict
+
+`StandardReadable` maps each registered child to its format, in registration order, and
+derives each verb's callables on demand, so the verbs always see the current registry.
+`set_readable_format(device, format)` sets one entry, and `None` removes it.
+
+- Keying by `Device` is identity keying, which a registry of children needs: `Device` has no
+  `__eq__`, and `DeviceVector` and `DeviceMap` hash by `id()`.
+- A tuple of pairs was tried for immutability, but rebuilding it per registration made
+  construction O(n²): 77 ms for 2000 signals against 13 ms for the dict.
+- It is a `cached_property`, because `AreaDetector` and `add_children_as_readables` register
+  children *before* `super().__init__()`, which would discard anything `__init__` assigned.
+- Only a `SignalR` takes a signal format, so the verbs get entries already typed.
+- Registering a device twice replaces its format, as the old double contribution was an
+  accident.
+- The deprecated `ConfigSignal` and `HintedSignal` markers go: they compared equal to their
+  target, which only worked while `add_readables` matched at registration.
+
+### Runtime changes can be undone, and the baseline is sealed by a metaclass
+
+@oliwenmandiamond pointed out that a runtime change makes the class definition misleading,
+and two safeguards were proposed: class-declared formats immutable unless opted out, or a
+separate `StandardDynamicReadable`. Both were rejected, as they block the case this ADR
+exists for: #1394's monochromator *declares* `energy` as configuration, and locking it would
+make a beamline get the class changed to switch technique. @jwlodek added that ophyd v1 has
+always been mutable, NSLS-II codebases rely on it, and differences between the libraries are
+a cost.
+
+Adopted instead, from both: `StandardReadable.reset_readable_formats` puts a Device back
+to what its class declared, so a plan can undo a retune without recording what it changed.
+Its baseline is the class, so it discards a stored technique too.
+
+The baseline is sealed by a metaclass. Formats are declared in annotations, by registration
+before `super().__init__()` (all 24 in-tree Devices) and by registration after it (legal,
+though unused). `type.__call__` is the only point after all three; a snapshot at the end of
+`__init__` was measured to miss the third, so the reset would silently drop a declared
+child. `Device` already has a metaclass, so this adds no machinery.
+
+### Format changes apply between runs, not within one
+
+A change part way through a run would make `describe()` disagree with the descriptor emitted
+at its start. This is documented rather than enforced, as a Device cannot see run
+boundaries.
+
+### Formats are stored with settings, under a reserved key
+
+A technique change wants values and formats together, so they share one file and one
+`apply_settings`. Values stay flat, and one reserved key holds the format changes:
+
+```yaml
+energy: 7.0
+<READABLE_FORMATS>:
+  <ROOT_DEVICE>:
+    energy: HINTED_SIGNAL
+    temperature: null
+```
+
+- No version marker: a file without the key changes no formats.
+- `<` cannot begin a Python identifier, so no attribute path collides with a reserved key.
+  Unlike `*FORMATS*` it needs no YAML quoting, and `<ROOT_DEVICE>` avoids the `? '' :` form
+  PyYAML emits for an empty key. `retrieve_settings` warns and drops an unknown `<...>` key,
+  so a newer file loads in an older version.
+- Formats are keyed by owner, then child, because a format belongs to a pair: one child can
+  be registered on two owners with different formats. Recording the owner needs no "direct
+  child" rule, so `driver.acquire_time` and `DeviceVector` elements work.
+- Formats are stored even by a config-only `store_settings`, as they decide which signals
+  are configuration.
+
+### Only changes are stored, and applying resets to the class defaults first
+
+`walk_readable_formats` stores how each `StandardReadable` differs from its construction,
+and `apply_readable_formats` resets the tree to its class defaults, then applies that.
+Rejected:
+
+- **Store everything, apply by replacing**: an old file silently unregisters a signal added
+  in a later version, and undoes a release's change of default.
+- **Store everything, apply by merging**: cannot return from technique B to A, as what B
+  changed and A does not mention stays.
+
+Every path is resolved before anything changes, so a bad path cannot half apply.
+`Settings.partition` copies the formats to both halves, because plans such as
+`apply_panda_settings` apply only the halves, and a copy on one would let the plan decide
+whether they are restored. Formats go after values, so a failed write leaves them untouched.
+
+## Consequences
+
+- A Device can switch, save and restore techniques at runtime, closing #1394.
+- `add_readables` keeps working but gains "set" semantics, so it is deprecated in favour of
+  `set_readable_format`, which says what it does.
+- A runtime change to a device registered outside its owner has no stable path, so
+  `walk_readable_formats` warns and skips it.

docs/how-to/change-readable-format.md

@@ -0,0 +1,88 @@
+# How to change what a Device reports at runtime
+
+A [](#StandardReadable) decides what appears in `read()`, `read_configuration()` and
+`hints` from the [](#StandardReadableFormat) each of its children is registered with.
+Normally that is fixed when the Device class is written, but it can also be changed on a
+live Device with [](#StandardReadable.set_readable_format).
+
+This is the equivalent of changing a `Component`'s `kind` in ophyd v1.
+
+## Changing one signal
+
+A beamline with two techniques might scan a monochromator's energy in one and hold it
+fixed in the other. The same Device serves both:
+
+```python
+from ophyd_async.core import StandardReadableFormat as Format
+
+# Scanning technique: energy is a hinted read
+mono.set_readable_format(mono.energy, Format.HINTED_SIGNAL)
+
+# Fixed technique: energy is configuration, recorded once per run
+mono.set_readable_format(mono.energy, Format.CONFIG_SIGNAL)
+
+# Not interesting for this technique at all
+mono.set_readable_format(mono.energy, None)
+```
+
+[](#StandardReadable.get_readable_formats) yields each child Device that is currently registered
+with its format, so a child that does not contribute is simply absent:
+
+```python
+for child, format in mono.get_readable_formats():
+    print(child.name, format)
+```
+
+## Putting it back
+
+[](#StandardReadable.reset_readable_formats) undoes every runtime change, back to
+how the class declared things: annotations, `add_children_as_readables`, and
+anything the Device's own `__init__` registered:
+
+```python
+mono.set_readable_format(mono.energy, Format.HINTED_SIGNAL)  # retune for a technique
+...
+mono.reset_readable_formats()  # back to the class defaults
+```
+
+So a plan that retunes a Device can put it back without having to record what it
+changed. It resets that Device only, not its children. To do a whole tree:
+
+```python
+from ophyd_async.core import apply_readable_formats
+
+apply_readable_formats(detector, {})
+```
+
+```{note}
+The baseline is what the Device class declares, so this also discards formats that came
+from a stored settings file (see [](./store-and-retrieve.md)).
+```
+
+```{note}
+Change formats **between runs**, not inside one. A run's descriptor is emitted when the
+run starts, and [](#StandardReadableFormat.HINTED_SIGNAL) begins monitoring in `stage()`,
+so a format changed part way through a run would leave `describe()` and `read()`
+disagreeing.
+```
+
+## Opting a detector plugin in
+
+A [](#StandardDetector) is a [](#StandardReadable), so the same call works on one. This
+matters most for areaDetector, where a detector often carries many more plugins than are
+wired into its chain, so plugins are never registered automatically, and you opt in to
+the ones that are actually producing meaningful values:
+
+```python
+det.set_readable_format(det.stats.total, Format.HINTED_UNCACHED_SIGNAL)
+```
+
+Data produced by a [](#DetectorDataLogic) is added on top of these, so a detector can
+write a file *and* report a plugin scalar in a step scan.
+
+```{seealso}
+[](./store-and-retrieve.md) for saving a set of formats alongside signal values and
+loading them back, and
+[](../explanations/decisions/0023-runtime-readable-format.md) for why the registry is
+keyed by Device.
+```

docs/how-to/store-and-retrieve.md

@@ -43,3 +43,26 @@ def load_panda(panda1: HDFPanda):
     settings = yield from retrieve_settings(provider, "yaml_file_name", panda1)
     yield from apply_settings_if_different(settings, apply_panda_settings)
 ```
+
+## Storing readable formats too
+
+As well as signal values, [](#store_settings) records how each child contributes to the Device's bluesky verbs (which signals are hinted, which are configuration) and [](#apply_settings) restores that alongside the values. The two change on the same cadence, so they live in one file and are applied by one call. This means a stored file can switch a Device between techniques as well as putting its values back; see [](./change-readable-format.md) for changing a format directly.
+
+Values stay flat in the yaml. Formats go under the reserved [](#READABLE_FORMATS_KEY), and only what has changed since the Device was constructed is recorded, so the file stays short and can be edited by hand:
+
+```yaml
+energy: 7.0
+temperature: 20.0
+<READABLE_FORMATS>:
+  <ROOT_DEVICE>:
+    energy: HINTED_SIGNAL
+    temperature: null
+  stats:
+    total: HINTED_UNCACHED_SIGNAL
+```
+
+The outer key of each formats entry is the [](#StandardReadable) the child is registered on, with `<ROOT_DEVICE>` standing for the Device you stored. The inner key is the child's path from that owner. The owner is recorded rather than worked out from the child's path, because the same child can be registered on more than one Device with a different format each time. A null means the child has been dropped.
+
+Applying first resets every Device in the tree to the formats its class declares, then applies the changes in the file. So a file stored with nothing changed (`<READABLE_FORMATS>: {}`) puts the class defaults back, and a signal added to a Device class after the file was stored keeps the format the class gives it.
+
+A file with no `<READABLE_FORMATS>` key changes no formats, so settings files saved before formats were storable keep working untouched. A file with *only* that key switches technique without writing a single value to hardware.

pyproject.toml

@@ -92,6 +92,7 @@ version_file = "src/ophyd_async/_version.py"
 [tool.pyright]
 typeCheckingMode = "standard"
 reportMissingImports = false  # Ignore missing stubs in imported modules
+reportMatchNotExhaustive = true
 
 [tool.pytest.ini_options]
 # Run pytest with all our checkers, and don't spam us with massive tracebacks on error

src/ophyd_async/core/__init__.py

@@ -99,10 +99,13 @@
     Watcher,
 )
 from ._readable import (
-    ConfigSignal,
-    HintedSignal,
+    READABLE_FORMATS_KEY,
+    ReadableEntry,
+    ReadableFormats,
     StandardReadable,
     StandardReadableFormat,
+    apply_readable_formats,
+    walk_readable_formats,
 )
 from ._settings import Settings, SettingsProvider
 from ._signal import (
@@ -279,6 +282,11 @@ def __getattr__(name):
     # Readable
     "StandardReadable",
     "StandardReadableFormat",
+    "ReadableEntry",
+    "ReadableFormats",
+    "walk_readable_formats",
+    "apply_readable_formats",
+    "READABLE_FORMATS_KEY",
     # Detector
     "DetectorTrigger",
     "TriggerInfo",
@@ -343,8 +351,6 @@ def __getattr__(name):
     "DerivedSignalFactory",
     "merge_gathered_dicts",
     # Back compat - delete before 1.0
-    "ConfigSignal",
-    "HintedSignal",
     # Standard enums
     "EnabledDisabled",
     "EnableDisable",