Skip to content

TDMS: multi-segment files fail to read #974

Description

@d-chambers

A TDMS file is a sequence of segments, and DASCore reads only the first one correctly. Every file with more than one segment fails.

Two things go wrong in dascore/io/tdms/utils.py:

  1. _get_data appends each following segment on axis 1 (channels), while data_node is (samples, channels) and the sample count it returns is summed along axis 0. Two segments of 1000 samples on 1152 channels come back as (1000, 2304) with a stated length of 2000.
  2. The time coordinate counts bytes from the end of the header to the end of the file, so every segment’s lead-in and metadata is counted as data. For the two-segment file above that gives 2027 samples where the file holds 2000 — and scan uses the same number, so an index built from it is wrong too.

Reproducing, with the example file written twice (a TDMS file grows by appending segments, and this file declares its segment length exactly):

import dascore as dc
from pathlib import Path
from dascore.utils.downloader import fetch

raw = Path(fetch("iDAS005_tdms_example.626.tdms")).read_bytes()
path = Path("two_segment.tdms")
path.write_bytes(raw + raw)

dc.spool(path)[0]
# CoordDataError: Data array has a shape of (1000, 2304) which doesnt match
# the coordinate manager shape of (2027, 1152).

Note sample_tdms_file_v4713.tdms cannot stand in for the example above: its lead-in claims 146 MB for a 1 MB file, so anything appended to it reads as part of the first segment.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions