Fix auxiliary scan ordering

docs/explanations/beamlines.md

@@ -0,0 +1,8 @@
+# Beamlines
+
+```{toctree}
+:maxdepth: 1
+:glob:
+
+i15-1 <beamlines/i15-1/i15-1>
+```

docs/explanations/beamlines/i15-1/auxiliary_scans.md

@@ -0,0 +1,25 @@
+# i15-1 Auxiliary Scans
+
+I15-1 PDF experiments require some scans to be run before hand, for use during analysis. These are added to the queue automatically by the i15-1 queue plugin. For each experiment, we need:
+
+- an air scan, at room temperature
+- an empty capillary scan, matching the capillary type and temperature range to the experiment
+- a standard sample scan, at room temperature, matching the capillary type to the experiment
+
+These must happen in this order, as the analysis process is layered - i.e, the empty capillary scan needs the air scan to be processed, the standard sample needs the empty capillary, and therefore the air, and the actual experiment needs all three. Additionally, to save time, we can merge these auxiliary scans together, to avoid queuing up three additional scans for every experiment. For example, experiments can always share an air scan (as it's always the same); empty capillary scans can be shared as long as the temperature range spans the range of the experiment, and the capillary type is the same; and standard sample scans can be shared as long as the capillary type is the same.
+
+The above requirements lead to the following logic:
+
+- An experiment will require the three auxiliary scans outlined above, the details of which will be a function of the experiment parameters. See `I151Converter._get_required_auxiliary_scans`
+- An `AuxiliaryScan` has a `requires` field, which details the other required auxiliary scans that must proceed it. The `AuxiliaryScan` objects produced by `I151Converter._get_required_auxiliary_scans` have each other as requirements. The empty capillary scan will be required by the standard sample scan, and the air scan will be required by both the other two.
+
+To add auxiliary scans, we build a new queue that includes the tasks along with their auxiliary scans. For each task in the queue, we will:
+
+- Gather the required auxiliary scans
+- For each required auxiliary scan:
+- Check whether a matching auxiliary scan already exists in the queue or tiled. For an existing scan to be considered suitable, all of the required auxiliary scan's own required auxiliary scans must also be required by the existing scan.
+- If a suitable existing does not exist, we will see if we can combine the required auxiliary scan with an existing scan. This, for example, could involve expanding the temperature range of a queued auxiliary scan. Combining these auxiliary scans also combines the requirements of both scans, so no requirement information is lost, but we also don't duplicate similar requirements. We then go to the beginning of this checklist, and repeat for the required auxiliary scans of the combined auxiliary scan, inserting the results before the combined scan in the queue.
+- If we cannot combine the required auxiliary scan, we will add the auxiliary scan to the queue. However, right before we do this, we will go through the required auxiliary scans of the auxiliary scan we added, and repeat this checklist for them. This ensures the auxiliary scan is added behind its requirements.
+- Once all the recursive adding of required auxiliary scans is complete, we add the task itself to the new queue.
+
+![Auxiliary scan logic flowchart](./i15-1_aux_scans.drawio.png)

docs/explanations/beamlines/i15-1/i15-1.md

@@ -0,0 +1,8 @@
+# i15-1
+
+```{toctree}
+:maxdepth: 1
+:glob:
+
+Auxiliary Scans <auxiliary_scans>
+```

docs/reference/rest_api.json

@@ -1072,29 +1072,9 @@
       "OIDCConfig": {
         "properties": {
           "well_known_url": {
-            "anyOf": [
-              {
-                "type": "string"
-              },
-              {
-                "type": "null"
-              }
-            ],
+            "type": "string",
             "title": "Well Known Url",
-            "description": "URL to fetch OIDC config from the provider",
-            "deprecated": true
-          },
-          "issuer": {
-            "anyOf": [
-              {
-                "type": "string"
-              },
-              {
-                "type": "null"
-              }
-            ],
-            "title": "Issuer",
-            "description": "URL of OIDC provider"
+            "description": "URL to fetch OIDC config from the provider"
           },
           "client_id": {
             "type": "string",
@@ -1117,6 +1097,7 @@
         "additionalProperties": false,
         "type": "object",
         "required": [
+          "well_known_url",
           "client_id"
         ],
         "title": "OIDCConfig"

src/daq_queuing_service/plugins/i15_1/auxiliary.py

@@ -1,8 +1,10 @@
+from collections.abc import Sequence
 from enum import StrEnum
 from pathlib import Path
+from typing import Self
 
 from daq_config_server.models.i15_1.standards_puck import StandardsPin
-from pydantic import BaseModel, ConfigDict, computed_field
+from pydantic import BaseModel, ConfigDict, Field, computed_field
 
 from daq_queuing_service.task_queue.task import Experiment
 
@@ -32,6 +34,25 @@ def get_background_temperatures(
     )
 
 
+def combine_auxiliary_requirements(
+    required_auxiliaries_1: Sequence["AuxiliaryScan"],
+    required_auxiliaries_2: list["AuxiliaryScan"],
+) -> list["AuxiliaryScan"]:
+    result: list[AuxiliaryScan] = []
+    remaining = required_auxiliaries_2.copy()
+    for aux1 in required_auxiliaries_1:
+        for aux2 in remaining:
+            if combined := aux1.attempt_to_combine_with(aux2):
+                result.append(combined)
+                remaining.remove(aux2)
+
+                break
+        else:
+            result.append(aux1)
+
+    return result + remaining
+
+
 class AuxiliaryScan(BaseModel):
     """Auxiliary scans are scans added automatically by the queue, such as background
     scans and standard sample calibration scans. They are needed for analysis of user
@@ -43,6 +64,7 @@ class AuxiliaryScan(BaseModel):
     pin: StandardsPin | None
     time_per_pdf: float
     list_of_temperatures: list[int] | None = None
+    requires: list[Self] = Field(default_factory=lambda: [], exclude=True)
 
     @computed_field
     @property
@@ -59,7 +81,9 @@ def is_suitable(
         temperature_step: int = DEFAULT_TEMPERATURE_STEP,
     ) -> bool:
         """Determine if this background is suitable compared to an experiment's required
-        background.
+        background. This includes checking that scans in required_background.requires
+        have a suitable pairing in self.requires, to ensure nested requirements are
+        taken into account.
 
         Args:
             required_background (AuxiliaryScan): The required background
@@ -75,6 +99,10 @@ def is_suitable(
         elif self.list_of_temperatures:
             return False
 
+        for required in required_background.requires:
+            if not any(r.is_suitable(required) for r in self.requires):
+                return False
+
         return (
             self.instrument_session == required_background.instrument_session
             and self.pin == required_background.pin
@@ -109,7 +137,8 @@ def attempt_to_combine_with(
         temperature_step: int = DEFAULT_TEMPERATURE_STEP,
     ) -> "AuxiliaryScan | None":
         """Creates a background that combines the requirements of this background object
-        and a provided required background, if possible.
+        and a provided required background, if possible. Anything that previously deemed
+        this scan as suitable should also deem the combined scan suitable.
 
         Args:
             required_background (AuxiliaryScan): The required background
@@ -142,6 +171,9 @@ def attempt_to_combine_with(
             pin=self.pin,
             time_per_pdf=max(self.time_per_pdf, required_background.time_per_pdf),
             list_of_temperatures=list_of_temperatures,
+            requires=combine_auxiliary_requirements(
+                self.requires, required_background.requires
+            ),
         )
         assert result.is_suitable(required_background)
         return result
@@ -167,6 +199,9 @@ def from_experiment(cls, experiment: Experiment) -> "AuxiliaryScan":
             list_of_temperatures=experiment.experiment_definition.data.get(
                 "list_of_temperatures"
             ),
+            requires=experiment.experiment_definition.data.get(
+                "required_auxiliary_scans", []
+            ),
         )
 
 

src/daq_queuing_service/plugins/i15_1/i15_1_converter.py

@@ -1,3 +1,4 @@
+from dataclasses import dataclass, replace
 from functools import cached_property
 from typing import Any
 
@@ -52,9 +53,16 @@
 )
 
 
-def _filter_auxiliary_scans(tasks: list[Task]) -> list[tuple[int, AuxiliaryScan]]:
+@dataclass(frozen=True)
+class QueuedAuxiliary:
+    index: int
+    task_id: str
+    auxiliary: AuxiliaryScan
+
+
+def _filter_auxiliary_scans(tasks: list[Task]) -> list[QueuedAuxiliary]:
     return [
-        (i, AuxiliaryScan.from_experiment(task.experiment))
+        QueuedAuxiliary(i, task.id, AuxiliaryScan.from_experiment(task.experiment))
         for i, task in enumerate(tasks)
         if isinstance(task.experiment, Experiment)
         and is_auxiliary_experiment(task.experiment)
@@ -200,9 +208,7 @@ def _add_required_auxiliary_scans(
         for task in tasks:
             with add_session_to_logger(task.experiment.instrument_session):
                 experiment = task.experiment
-                if isinstance(experiment, Experiment) and not is_auxiliary_experiment(
-                    experiment
-                ):
+                if isinstance(experiment, Experiment):
                     instrument_session = experiment.instrument_session
 
                     required_auxiliary_scans = self._get_required_auxiliary_scans(
@@ -241,8 +247,8 @@ def _ensure_auxiliary_in_queue_or_tiled(
 
         queued_auxiliary_scans = _filter_auxiliary_scans(new_tasks)
         if any(
-            queued_auxiliary.is_suitable(required_auxiliary)
-            for _, queued_auxiliary in queued_auxiliary_scans
+            queued_auxiliary.auxiliary.is_suitable(required_auxiliary)
+            for queued_auxiliary in queued_auxiliary_scans
         ):
             return new_tasks
 
@@ -260,6 +266,7 @@ def _ensure_auxiliary_in_queue_or_tiled(
         )
         return self._add_or_replace_auxiliary_scan(
             required_auxiliary,
+            current_task,
             new_tasks,
             queued_auxiliary_scans,
             instrument_session,
@@ -268,32 +275,66 @@ def _ensure_auxiliary_in_queue_or_tiled(
     def _add_or_replace_auxiliary_scan(
         self,
         required_auxiliary: AuxiliaryScan,
+        current_task: TaskWithPosition | None,
         new_tasks: list[Task],
-        queued_auxiliary_scans: list[tuple[int, AuxiliaryScan]],
+        queued_auxiliary_scans: list[QueuedAuxiliary],
         instrument_session: str,
     ) -> list[Task]:
-        combined_auxiliary_scan = None
-        index = None
-
-        for i, auxiliary_scan in queued_auxiliary_scans:
-            if combined_auxiliary_scan := auxiliary_scan.attempt_to_combine_with(
-                required_auxiliary
-            ):
-                index = i
-                break
+        # It is assumed every scan in queued_auxiliary_scans has already added their
+        # requirements before for them
+        combined_auxiliary = self._find_combinable_auxiliary(
+            required_auxiliary, queued_auxiliary_scans
+        )
 
         aux_experiment = self._construct_auxiliary_experiment(
-            combined_auxiliary_scan or required_auxiliary, instrument_session
+            combined_auxiliary.auxiliary if combined_auxiliary else required_auxiliary,
+            instrument_session,
         )
-        if index is None:
-            new_tasks.append(Task(experiment=aux_experiment))
+
+        if combined_auxiliary is None:
+            aux_task = Task(experiment=aux_experiment)
+            prefix = new_tasks
+            suffix = []
+            additional_required = required_auxiliary.requires
         else:
-            new_tasks[index] = Task(experiment=aux_experiment)
+            aux_task = Task(experiment=aux_experiment, id=combined_auxiliary.task_id)
+            prefix = new_tasks[: combined_auxiliary.index]
+            suffix = new_tasks[combined_auxiliary.index + 1 :]
+            additional_required = combined_auxiliary.auxiliary.requires
+
+        self._tiled_auxiliary_scans[aux_task.id] = {}
+        for required in additional_required:
+            prefix = self._ensure_auxiliary_in_queue_or_tiled(
+                required,
+                current_task,
+                prefix,
+                aux_task.id,
+                instrument_session,
+            )
+        new_tasks = prefix + [aux_task] + suffix
+
         return new_tasks
 
+    def _find_combinable_auxiliary(
+        self,
+        required_auxiliary: AuxiliaryScan,
+        queued_auxiliary_scans: list[QueuedAuxiliary],
+    ):
+        for queued_auxiliary in queued_auxiliary_scans:
+            if (
+                combined_auxiliary_scan
+                := queued_auxiliary.auxiliary.attempt_to_combine_with(
+                    required_auxiliary
+                )
+            ):
+                return replace(queued_auxiliary, auxiliary=combined_auxiliary_scan)
+
     def _get_required_auxiliary_scans(
         self, experiment: Experiment
     ) -> list[AuxiliaryScan]:
+        if is_auxiliary_experiment(experiment):
+            return experiment.experiment_definition.data["required_auxiliary_scans"]
+
         time_per_pdf = experiment.experiment_definition.data["time_per_pdf"]
         list_of_temperatures = experiment.experiment_definition.data.get(
             "list_of_temperatures"
@@ -302,34 +343,40 @@ def _get_required_auxiliary_scans(
             list_of_temperatures = get_background_temperatures(list_of_temperatures)
 
         assert experiment.sample
-        return [
-            AuxiliaryScan(
-                instrument_session=experiment.instrument_session,
-                pin=None,
-                time_per_pdf=time_per_pdf,
-            ),
-            AuxiliaryScan(
-                instrument_session=experiment.instrument_session,
-                pin=StandardsPin(
-                    capillary=experiment.sample.data["capillary"], contents=None
-                ),
-                time_per_pdf=time_per_pdf,
-                list_of_temperatures=list_of_temperatures,
+
+        air = AuxiliaryScan(
+            instrument_session=experiment.instrument_session,
+            pin=None,
+            time_per_pdf=time_per_pdf,
+        )
+        empty_capillary = AuxiliaryScan(
+            instrument_session=experiment.instrument_session,
+            pin=StandardsPin(
+                capillary=experiment.sample.data["capillary"], contents=None
             ),
-            AuxiliaryScan(
-                instrument_session=experiment.instrument_session,
-                pin=StandardsPin(
-                    capillary=experiment.sample.data["capillary"], contents="Silicon"
-                ),
-                time_per_pdf=time_per_pdf,
+            time_per_pdf=time_per_pdf,
+            list_of_temperatures=list_of_temperatures,
+            requires=[air],
+        )
+        standard_sample = AuxiliaryScan(
+            instrument_session=experiment.instrument_session,
+            pin=StandardsPin(
+                capillary=experiment.sample.data["capillary"], contents="Silicon"
             ),
-        ]
+            time_per_pdf=time_per_pdf,
+            requires=[air, empty_capillary],
+        )
+
+        return [standard_sample, empty_capillary, air]
 
     def _construct_auxiliary_experiment(
         self, auxiliary_scan: AuxiliaryScan, instrument_session: str
     ) -> Experiment:
         LOGGER.debug(f"Constructing experiment for auxiliary scan: {auxiliary_scan}")
-        data: dict[str, Any] = {"time_per_pdf": auxiliary_scan.time_per_pdf}
+        data: dict[str, Any] = {
+            "time_per_pdf": auxiliary_scan.time_per_pdf,
+            "required_auxiliary_scans": auxiliary_scan.requires,
+        }
         if auxiliary_scan.list_of_temperatures:
             settle_time = (
                 self._queue_params.blower_auxiliary_collection_params.settle_time_s