ESRF id31 new detector default and Q plot in all SIXC reference frames - #66
Merged
Conversation
for live data analysis and working with hdf5 files where new data is still added, the hdf5 locking must be turned off
As ESRF fastscans contain two separate hdf5 nodes, the data must be treated separately when loading a segmented scan. Requirement: class name includes 'BlissScan'
the max image or sum image are automatically deactivated when the user changes the image using the slider below the central plot
fix a bug that causes the Qt icon to be deactivated even though the max/sum image is shown in the central plot
this commit allows the refreshing of hdf5 files while new scans in the same dataset are still performed - without causing broken link errors at ESRF beamline id31
The Pilatus 4 4M detector is now set as default, the fallback geometry is more universal and the ch8153 beamtime has been added to the backend selection
using the pyFAI FiberIntegrator package, grazing incidence detector images can now easily be converted from pixel coordinates to their Q values in units of inverse Angstrom
Resolve conflicts in favour of master's formatting with the ID31 beamline changes re-applied. Add QAlpha/FiberIntegrator equivalence tests and document both reciprocal-space routes.
Make the reciprocal-space conversion exact for arbitrary azimuthal references, use orGUI diffractometer conventions for single crystals instead of the 2d FiberIntegrator. - add the Q_alpha, Q_lab, Q_omega, Q_chi, Q_phi and Q_cryst frames with a selector next to the Q-plot action, defaulting to Q_alpha - add the _qconversion_cpp extension, which converts a 6.2 megapixel Pilatus 6M image in ~50 ms instead of ~740 ms, with a numpy fallback - reuse the pyFAI integrator while the conversion is unchanged, keyed on the collapsed conversion coefficients, so stepping through images no longer resets the integrator for every image - keep the Q-plot switched on and rebuild it in place when the image, the max/sum selection, the frame, the machine angles, the energy or the orientation matrix change - pass a legend rather than an image item to the alpha slider - take the manual backend selection from backends.default_beamtime
Finnjek
reviewed
Aug 3, 2026
Finnjek
left a comment
Collaborator
There was a problem hiding this comment.
Looks very nice! I have tested the Q conversion with several ID31 datasets. Very fast conversion compared to the old pyFAI method. However I have found some minor issues that may need to be addressed before merging to main:
- the cursor live values shown under the plot are wrong when in Q-Plot mode. This was the same in the old pyFAI conversion and is because the X and Y value are linked to the Q_parallel and Q_perp values instead of pixels. We should change the labels 'X' and 'Y' and give correct values for HKL, delta and gamma when in Q-plot mode.
- When toggling the Q-Plot button, the plot orientation (x-origin and y-origin) sometimes doesn't roll back to its original value.
- Also, the Q-Plot image always has its origin in the bottom left corner, even if the detector image is differently oriented. We could however leave it like this, as this is much clearer and only very few (crazy) people do experiments in downward scattering orientation.
- The new id31 backend class has no 'BlissScan' in its name and is therefore not recognized by the isID31 option in orGUI.py, for example when loading segmented scans.
- Maybe replace the button icon that is currently text 'Q-plot' to a thumbnail of a capital Q with a vector arrow to have a uniform style?
Addresses the review of #66. - The position readout treated the Q-plot axes as detector pixels, so the X/Y labels were wrong and HKL, del, gam and Q were nonsense. The axes are now labelled q_par/q_perp per frame, and the other fields come from inverting the projection through the Ewald condition, so a Q-plot point reports the same values as the pixel it was rebinned from. Positions no pixel maps to show "------". The reported Q is the reconstructed vector, unrefracted like the axes it has to match. - Leaving the Q-plot did not restore the plot orientation: the x-axis origin was never recorded, and the y-axis restore was tied to the maximum/sum image. Both are now restored unconditionally, and a failed conversion no longer leaves the plot half-converted. - The Q-plot ordinate follows the geometry instead of always pointing up, so an inverted setup such as ID31's is no longer mirrored with respect to the detector image. - The toolbar action gains an icon instead of being the only text button.
Addresses the review of #66. Add the optional Scan.listScans, which returns the scan identifiers, bare or as (identifier, label) pairs. Withoutit the loader applies the backend's own parse_h5_node to every root entry, keeps what yields a scanno and collapses repeated numbers. BREAKING CHANGE: custom backends relying on the old beamtime-id list must implement listScans, or ensure parse_h5_node raises for non-scan entries.
Owner
Author
|
Thanks, I completely forgot about the issue with the x y pixel conversion for Q. Unsure about the Q-plot orientation behavior. The solution now tries to adapt automatically, but this might cause problems. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the ESRF ID31 beamtime work onto master, together with a reworked
experimental reciprocal-space (Q) display.
masteris fully contained in this branch, so it merges without conflicts.Breaking changes
HDF5_USE_FILE_LOCKING=False),so files still open for writing by the acquisition system can be read. A
manually set environment variable still wins, and
--hdflocking/-lrestores the previous behaviour.
instead of a hardcoded list of beamtime ids. This fixes the
id31_default_p4backend, which the old list did not cover, and makes new ID31 beamtimes work
without code changes. Custom backends whose class name does not contain
BlissScanare no longer treated as ID31-style even if their beamtime id waspreviously listed.
ID31 beamline support
BlissScan_EBS_p4backend for the Pilatus4 detector, an example backend underexamples/backend/ID31_EBS_p4_backend.py, and thech8153beamtime.id31_default_p4; the fallback geometry describes aPilatus4 4M CdTe. The manual backend selector now follows
backends.default_beamtimeinstead of a second hardcoded default.toolbar icons no longer get out of sync.
Experimental Q-plot
A
Q-plottoolbar action displays the image in reciprocal space, with adrop-down selecting the frame:
Q_alpha(default),Q_lab,Q_omega,Q_chi,Q_phiandQ_cryst. It lives inorgui.app.qconversion—deliberately in the application layer, not in
orgui.datautils.xrayutils, so itis not mistaken for production reciprocal-space code, and it is marked
experimental in the UI, the module and the docs.
Why it does not simply use pyFAI's fiber units. pyFAI's
sample_orientationis the EXIF dihedral group acting on the image array, so itonly spans quarter turns, whereas orGUI's azimuthal reference is continuous. An
implementation based on that flag is correct only at exact multiples of 90
degrees — at an azimuth of 45 degrees it is off by several inverse Angstrom. The
azimuth is therefore passed through
tilt_angle, and because pyFAI composes theincidence and tilt rotations about fixed axes while in orGUI the azimuth rotates
the alpha axis itself, orGUI supplies
FiberIntegratorwith its ownUnitFiberdefinitions. pyFAI still performs the rebinning.Performance. The conversion collapses into a single affine relation and is
evaluated by a new pybind11 extension,
_qconversion_cpp. A 6.2 megapixelPilatus 6M image converts in ~50 ms instead of ~740 ms, bit-exact against the
numpy reference; a numpy fallback is used when the extension is not built. The
pyFAI integrator is reused while the conversion is unchanged, so stepping
through images no longer resets it per image.
numexpris deliberately not used: version 2.11.0 returns wrong results for asmall fraction of multi-threaded evaluations of this kind of expression
(reproducible with no orGUI code involved, ~3/200 evaluations).
Other GUI changes
selection, the frame, the machine angles, the energy or the orientation matrix
change.
HKLas one bracketed triplet and the momentumtransfer of the selected frame as
Q[alpha],Q[lab], … with five decimals.Field size hints are matched to their content so the triplets are not elided.
dialog now restores the configuration that was active when the dialog opened.
Both dialogs apply every edit immediately, so restoring the widgets alone left
the discarded configuration active.
Testing
New tests cover the conversion against
QAlphaper pixel for flat and tilteddetectors at azimuths that are deliberately not multiples of 90 degrees, that
Q_phireproducesanglesToHklandQ_crystreproducesB·H, that thecompiled kernel matches the numpy fallback, the integrator cache does not serve
stale coordinates, and the dialog-cancel and readout behaviour.
Notes for reviewers
meson.buildgains_qconversion_cpp, so a rebuild is needed for the fastpath. Without it the numpy fallback runs automatically (~135 ms for 6 Mpixel).
Q_omegaframe is mathematically indistinguishable from `Q_