Repository navigation
edge: draft BIL sheets from OME-Zarr stores and voxel manifests - #5
Merged
Merged
Conversation
Two new scigantic-edge record formats reach submission_draft(s): - ome-zarr: step sizes from the level-0 scale in the axes' unit (the _um table now reads NGFF unit names, micro sign, Greek mu, plurals and British spellings), channels/slices/size from the shape, dimensionorder from the axes names (CZYX, TCZYX). - voxel-manifest: Instrument sheet from the manifest of an Allen Institute voxel acquisition (instrument name, objective, laser lines, emission bands, camera). Light-sheet is claimed only when the instrument's name says so; the template instrument is a simulator. A voxel acquisition is one store per volume and channel under <root>/volumes/NNNNNN/<profile>/<channel>.ome.zarr. Stores collapse into one draft per manifest: by the store's manifest sidecar, by the manifest's storePaths, or by the /volumes/ prefix when no manifest is indexed yet; otherwise a store groups under its parent directory like any other file. channels is the count of distinct channel ids across the volumes, not the sizeC of one store. A new draft key, acquisition, carries store and volume counts, channel names and the manifest's status, revision and timestamps. fixtures/edge/voxel_run1_index.jsonl is the edge decoder's index over a real headless run of voxel's simulated instrument; the exaSPIM fixture is hand-written from two public tiles' metadata. The prior is unchanged: its bil-image fields already line up with what an ome-zarr record says.
…malformed records Found by running the scigantic-edge binary over its own fixture stores and feeding the index through the bridge, then at scale and under fuzzing. - index.jsonl is an append log. In -watch mode a store re-records as its shards land and a voxel manifest.json re-records on every status change, so read_index now yields only the last line per source, as the box's own resume does. An 8,117-line watch log of a 200-volume, 4-channel run drafted 8,000 stores, 40 slices (the first partial record) and status "running" at revision 0; it now drafts 800 stores, 400 slices, "completed", rev 49. - A plain container group (AIND's SPIM.ome.zarr) joins the draft of the stores it holds. It used to start an empty draft of its own, so submission_draft() raised on every public AIND v2 acquisition. - One reference store (has a shape, most complete, finest voxel) supplies the whole Image row. Filling blanks from the next store paired the 561 nm store's 30.08 um voxel with the 488 nm store's 3548 px width on the 2026-08-04 public exaSPIM acquisition. Disagreements across stores are reported in acquisition.mismatch, and store states in storeStates. - AIND's tile_..._ch_<channel> store names count tiles and channels. - An OME-Zarr scale with no unit, or a non-length unit, leaves the step sizes blank instead of calling it micrometres. - Every record is normalised on entry; list fields, NaN and infinity are type-checked. 30,000 random malformed indexes raise nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
scigantic-edge now decodes OME-Zarr stores and Allen Institute
voxelacquisition manifests (Scigantic/scigantic-edge#8, merged). This teaches the bridge both record formats, sosubmission_draft()can fill BIL's Image and Instrument sheets from an acquisition written as OME-Zarr.ome-zarr: step sizes from the level-0 scale in the axes' declared unit, channels, slices and size from the shape, dimension order from the axes.voxel-manifest: Instrument sheet from the manifest (instrument, objective, laser lines, emission bands, camera). Light-sheet is claimed only when the instrument says so.storePaths, or by the/volumes/prefix before the manifest is indexed). A newacquisitionblock carries store, volume and channel counts, store states, cross-store mismatches and the manifest's status, revision and timestamps.Stress test
The second commit is what the stress test found and fixed. Each has a test.
SPIM.ome.zarrgroup + tiles), indexed by the real edge binarysubmission_draft()raisedacquisition.mismatchAlso: AIND
tile_..._ch_<n>names count tiles and channels, and an OME-Zarr scale with no unit is left blank rather than called micrometres.tests/test_edge.py: 23 pass. mypy clean. The live suite was not run: the BIL download host is still not answering, as with the 0.6.0 release.