Skip to content

standards write speed, UoA collection and metadata fixes - #346

Merged
kujaku11 merged 5 commits into
IAGA-DVI-DataStandards:patchesfrom
bvkay:perf-update
Sep 11, 2026
Merged

kujaku11 merged 5 commits into
IAGA-DVI-DataStandards:patchesfrom
bvkay:perf-update

Conversation

@bvkay

@bvkay bvkay commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Depends on IAGA-DVI-DataStandards/mt-io#6: item 2 imports UoACollection from there, so CI against the released mt-io will fail on that import until #6 merges. The other items are independent of it.

  1. Bulk add_rows and one standards summary: the ~600-row standards summary table was written one h5py setitem per row, twice per file (file init and add_survey) and add_row formatted every row into a debug f-string even with logging off. Rows now write as one block and the summary array is built once per process and reused.

  2. Split PR6-24 stations into runs: the UoA client now builds its file list through UoACollection (added in mt-io#6), so multi-run EDL stations come in as separate runs instead of one concatenated block.

  3. Scan for data in blocks: has_data ran np.nonzero over entire channel datasets, so closing a file read every channel in full. It now scans ~4M-sample blocks and exits at the first nonzero.

  4. Client metadata fixes: in phoenix.py the station update_metadata() call sat outside the station loop so only the last station was finalised and in metronix.py the survey update_metadata was referenced without parentheses.

@bvkay
bvkay requested a review from kujaku11 August 27, 2026 06:24
@kujaku11
kujaku11 changed the base branch from master to patches September 11, 2026 23:04

@kujaku11 kujaku11 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bvkay Thanks for the updates, these look like updates for efficiency.

@kujaku11 kujaku11 added the release:patch patch update label Sep 11, 2026
@kujaku11
kujaku11 merged commit 5986cbd into IAGA-DVI-DataStandards:patches Sep 11, 2026
5 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:patch patch update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants