Skip to content

Add read_plate_dataset(): Parquet reader for the plate dataset - #81

Closed
vcjdeboer wants to merge 6 commits into
develfrom
st-engine/parquet-reader
Closed

vcjdeboer wants to merge 6 commits into
develfrom
st-engine/parquet-reader

Conversation

@vcjdeboer

@vcjdeboer vcjdeboer commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Before this PR leaves draft

  • Re-run R CMD check --no-manual on the head commit. It was skipped for 54868e8, 400f7bc, 2915b2a and c261d83 on Vincent's word ("skip the tarball rebuild"); last pass was on e9906ed.
  • Re-run the webR load check (library(seahtrue) and read_plate_dataset() under webR 0.6.0). It was skipped for 54868e8, 400f7bc, 2915b2a and c261d83; last pass was on e9906ed.
  • The parser's plate_dataset_schema_v1.json is on swamp_seahtrue's main branch, and its sha256 still equals PLATE_DATASET_SCHEMA_V1_SHA256.

Summary

Adds an exported Parquet reader for the plate dataset, read_plate_dataset(). A plate dataset is the Parquet file the lab's .asyr parser writes for one plate, one row per well per tick.

The reader:

  • Reads the file as data only. Nothing from the file is evaluated or deserialised. Arrow and R metadata in the file are ignored (use_arrow_metadata = FALSE), and every column is reduced to a plain atomic vector.

  • Rejects an unknown or missing plate_dataset_schema_version. Only "1" is known.

  • Refuses anything outside schema version 1:

    • missing, extra, reordered or mistyped columns;
    • values that should be constant per plate, well, measurement or tick but are not;
    • a non-finite value in any double column, or a non-finite O2 (mmHg) or pH after conversion;
    • anything other than 96 wells named A01..H12;
    • a missing well or tick;
    • a plate id that is not a lowercase sha256;
    • a date_run that is not ISO 8601 UTC.

    File size, row count, uncompressed size and string length are all capped. Refusal messages quote file values only truncated and escaped.

  • Converts emissions. O2 emission becomes O2 (mmHg) by Stern-Volmer with F0 and the uncorrected Ksv. pH emission becomes pH on a line through the calibration pH at the pH target emission.

  • Keeps every well. flagged_well and plate_flagged_well are manual flags a person set while looking at the data, both carried as data. Their one effect: a Background well flagged in either is left out of the background average. When every Background well is flagged, or there is none, the background columns are missing.

  • Returns seahtrue's plate structure with our names and runs validate_preprocessed() itself.

    • assay_info carries plate_dataset_schema_version and plate_dataset_schema_sha256, so callers can check which schema the plate was read with without using internals.
    • rate_data has zero rows: no OCR or ECAR yet.
  • Adds nanoparquet (>= 0.5.1) to Imports. It has no dependencies and runs under webR; arrow does not.

The schema is a copy of the parser's plate_dataset_schema_v1.json: 34 columns, sha256 8ffe7fdd…4c5b, parser commit f6d95e1. The parser is still in review, so this PR stays a draft.

Checks

  • devtools::test() at c261d83: 171 expectations pass, 0 failures, 89 of them for the new reader on synthetic plate datasets. No real plate data is in this repository.
  • At e9906ed:
    • R CMD check --no-manual: 0 errors, 0 warnings, 2 notes. One note is new: Imports now has 21 non-default packages.
    • webR 0.6.0 (R 4.6.0): the package loads with library(seahtrue) and the reader works.
    • Both real plates the parser produced then read and passed validation locally.
  • Both checks are still to be re-run for c261d83 (see the checklist at the top).

Do not merge or tag from this PR; Vincent decides the release.

🤖 Generated with Claude Code

vcjdeboer and others added 2 commits October 3, 2026 14:17
New exported read_plate_dataset(path, date_processed, max_file_bytes)
reads one plate dataset (the Parquet file the parser writes for one
plate, one row per well per tick) at plate dataset schema version 1:

- reads the file as data only: no eval, parse or unserialize of file
  content, Arrow/R metadata in the file ignored, attributes dropped;
- refuses an unknown or missing plate_dataset_schema_version, any
  missing, extra, reordered or mistyped column, and any value outside
  the schema's rules (one value per plate/well/measurement/tick, 96
  wells A01..H12, every well at every tick, lowercase sha256 plate id,
  ISO 8601 UTC date_run), with size, row and string-length caps;
- converts O2 emissions to O2 (mmHg) (Stern-Volmer, uncorrected Ksv)
  and pH emissions to pH (line through the calibration pH at the target
  emission);
- returns seahtrue's plate structure with our names and runs
  validate_preprocessed() itself; assay_info carries
  plate_dataset_schema_version and plate_dataset_schema_sha256.

Adds nanoparquet (>= 0.5.1) to Imports; it also runs under webR.
Tests use synthetic plate datasets only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ooter-cap tests

- Refuse a non-finite value (Inf, -Inf, NaN) in any double column, and a
  non-finite O2_mmHg, pH_em_corr_corr or pH after conversion (a zero
  emission, F0, Ksv or calibration emission), so validation never passes
  over NaN silently.
- Schema v1 copy follows the parser's updated file (sha256 8ffe7fdd...,
  parser commit f6d95e1): new required BOOLEAN column plate_flagged_well
  after flagged_well. Both manual flags are carried in raw_data as data;
  no well is dropped or excluded by either, and background wells are
  averaged whether flagged or not.
- Tests for the footer row-count cap, rows read differing from the
  footer, and the uncompressed-size cap (footer reader mocked), for the
  non-finite refusals, and for the two flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vcjdeboer

Copy link
Copy Markdown
Owner Author

Revise round (Vincent at acceptance: "4 revise, F2 ok"), commit 54868e8:

  • Non-finite values are refused: Inf, -Inf or NaN in any double column, and a non-finite O2 (mmHg) or pH after conversion. A test covers both.
  • Tests now cover the footer row-count cap, rows read differing from the footer, and the uncompressed-size cap, using a mocked footer reader.
  • The schema v1 copy follows the parser's updated file (sha256 8ffe7fdd…, parser commit f6d95e1). It adds a required BOOLEAN column, plate_flagged_well. Both manual well flags are carried as data: no well is dropped or excluded, and background wells are averaged whether flagged or not.

devtools::test(): 157 expectations, 0 failures.

On Vincent's word ("skip the tarball rebuild"), R CMD check and the webR load check were not re-run this round. Both must be re-run once before this PR leaves draft.

🤖 Generated with Claude Code

vcjdeboer and others added 3 commits October 3, 2026 19:17
…round

Vincent: "flagged background should be excluded!". A "Background" well
flagged in either flagged_well or plate_flagged_well is left out of the
background average; a plate whose every Background well is flagged is
refused rather than given a background from nothing. The flags exclude
nothing else: every well stays in raw_data, with both flag columns as
data.

Tests: a background well flagged under each flag column is left out (the
flagged well, H01, changes the mean, so the test tells exclusion from no
exclusion); the all-flagged refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vincent: "accept no refusal". A plate whose every Background well is
flagged (in either flag column) now gets missing background columns, as
a plate with no Background well does, instead of being refused. The flag
rule itself is unchanged: a Background well flagged in either column is
left out of the background average.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eft out

Code review CR4-F3: the flag tests now also expect pH_em_corr_bkg to
leave out the flagged well H01 (whose pH emission differs from the other
three Background wells), under each flag column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vcjdeboer
vcjdeboer changed the base branch from main to devel October 3, 2026 18:53
NEWS.md: keep the development-version entry for read_plate_dataset() above devel's 1.7.2 entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vcjdeboer vcjdeboer closed this Oct 3, 2026
@vcjdeboer
vcjdeboer deleted the st-engine/parquet-reader branch October 3, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant