Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
865216a
Fixing issue in CDB considering the channel id an int
pscheidler May 16, 2025
79083ec
Updating all IDs to UInt
pscheidler May 17, 2025
86dadd4
Adding Python 3.13 to the tests
pscheidler May 17, 2025
c55276a
Update idelib/schemata/mide_ide.xml
pscheidler May 20, 2025
1a8b1df
Update .github/workflows/unit-tests.yml
pscheidler May 20, 2025
7293c76
Merge pull request #159 from MideTechnology/tester/python_3-13
pscheidler May 21, 2025
a4254b1
Merge pull request #158 from MideTechnology/bugfix/channel_id_uint
pscheidler May 21, 2025
8955b8d
doc python version fix
StokesMIDE Jun 23, 2025
28beb1e
Added non-sync-related changes
StokesMIDE Jul 2, 2025
7f6aa0c
Some missed changes
StokesMIDE Jul 2, 2025
292fa14
Removed sync-related schema changes (will be in other PR)
StokesMIDE Jul 2, 2025
c04cc7d
Removed timestamp offset changes
StokesMIDE Jul 2, 2025
ada205f
removed a bit more sync stuff
StokesMIDE Jul 2, 2025
7cd2e02
Merge branch 'feature/non-sync' into develop
StokesMIDE Jul 7, 2025
b2861c9
Removed Sphinx fail-on-warning
StokesMIDE Jul 7, 2025
f6fc081
Updated versions in publish-to-pypi GHA
StokesMIDE Jul 7, 2025
6c0b884
Test update for `ebmlite` 3.4.0
StokesMIDE Jul 7, 2025
a425045
Feature/es 851 util docs (#162)
StokesMIDE Jul 7, 2025
782367b
Basic sync functionality (#160)
StokesMIDE Aug 7, 2025
6c79e0d
docstrings and cleanup (#163)
StokesMIDE Sep 23, 2025
458a0f0
Added a couple of elements for future userdata use (#164)
StokesMIDE Sep 24, 2025
c6f4f18
Feature/sync (#167)
StokesMIDE Oct 20, 2025
f08c465
Added Python 3.14 to unit tests and PyPI classifiers; updated docs UR…
StokesMIDE Oct 28, 2025
09ef446
Fix for export w/ UTC timestamps
StokesMIDE Nov 18, 2025
c7f08fa
Schema change: added `<EscapedCharacters>` to `<SerialCommandInterface>`
StokesMIDE Jan 27, 2026
f9400fe
Fix for Numpy >2.2.1; removed workaround for Python < 3.7)
StokesMIDE Jan 27, 2026
e073316
Merge branch 'feature/HDSP-983_EscapeControlCharacters' into develop
StokesMIDE Jan 27, 2026
39ab572
Added basic `idesync` utility documentation (#171)
StokesMIDE Jan 28, 2026
ac85ca6
Version bump (3.4.0)
StokesMIDE Jan 28, 2026
51a24e2
Merge branch 'main' into release/v3.4.0
StokesMIDE Jan 28, 2026
775bbe6
Fix reading files when only the first block has 1 sample (#165)
XanthanGum Jan 28, 2026
c4e6bc4
Merge branch 'develop' into release/v3.4.0
StokesMIDE Jan 28, 2026
f7d1bf4
type hints
StokesMIDE Jan 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/publish-to-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ jobs:

steps:
- uses: actions/checkout@master

- name: Set up Python 3.12
uses: actions/setup-python@v3
uses: actions/setup-python@v4

with:
python-version: 3.12

Expand Down
9 changes: 3 additions & 6 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,7 @@ jobs:
fail-fast: false
matrix:
os: [windows-latest, ubuntu-latest, macos-latest]
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
exclude:
- os: ubuntu-latest
python-version: '3.6'
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14']

env:
OS: ${{ matrix.os }}
Expand All @@ -35,8 +32,8 @@ jobs:
- run: python -m pip install .[test,docs]

- run: python -m pytest ./testing/ --cov=idelib --cov-report=xml -n auto
- run: sphinx-build -W -b html docs docs/html
- run: sphinx-build -W -b doctest docs docs/doctest
- run: sphinx-build -b html docs docs/html
- run: sphinx-build -b doctest docs docs/doctest

- uses: codecov/codecov-action@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ docs/_build/
docs/html/
docs/latex/
docs/doctest/
docs/doctrees/

# PyBuilder
target/
Expand Down
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,18 @@

# _idelib_ README

_idelib_ is a Python API for accessing [enDAQ's](http://endaq.com) IDE recordings. The IDE format is
an [EBML](http://matroska-org.github.io/libebml/specs.html) encoded file using a
custom schema. This library utilizes our
[ebmlite](https://github.com/MideTechnology/ebmlite) to parse the files, and
provides classes that make reading data simple.
_idelib_ is the core Python API for accessing the contents of [enDAQ](http://endaq.com) ``.IDE`` recording files.
It provides a means of easily accessing time series sensor data, with all necessary calibration applied, as
well as recording metadata.

The package also contains the command-line utilities ``ideexport`` and ``ideinfo`` for use outside of Python.


## IDE File Basics

### What's an IDE file?

An IDE file is a read-only hierarchical data format that stores recording
An IDE file is an [EBML](http://matroska-org.github.io/libebml/specs.html)-based, read-only hierarchical data format that stores recording
information generated by an [enDAQ sensor device](http://endaq.com/collections/endaq-sensors-shock-vibration-s-series). It contains both time-indexed
data from several different kinds of sensors (like acceleration, pressure,
temperature, etc.), as well as metadata about the recording device (like device
Expand Down
3 changes: 3 additions & 0 deletions docs/api_ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,6 @@
:caption: Contents:

idelib/dataset
idelib/importer
idelib/sync
idelib/util
6 changes: 5 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@
# documentation root, use os.path.abspath to make it absolute, like shown here.
#

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

import idelib

# -- Project information -----------------------------------------------------
Expand Down Expand Up @@ -204,4 +208,4 @@
# -- Options for intersphinx extension ---------------------------------------

# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'python': ('https://docs.python.org/2', None)}
intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
2 changes: 1 addition & 1 deletion docs/ide_basics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ IDE File Basics
What's an IDE file?
-------------------

An IDE file is a read-only hierarchical data format that stores recording information generated by an enDAQ sensor device. It contains both time-indexed data from several different kinds of sensors (like acceleration, pressure, temperature, etc.), as well as metadata about the recording device (like device serial number, model number, device name, etc.) and recording settings.
An IDE file is a read-only, `EBML <http://matroska-org.github.io/libebml/specs.html>`_-based hierarchical data format that stores recording information generated by an enDAQ sensor device. It contains both time-indexed data from several different kinds of sensors (like acceleration, pressure, temperature, etc.), as well as metadata about the recording device (like device serial number, model number, device name, etc.) and recording settings.


.. _dataset_desc:
Expand Down
4 changes: 2 additions & 2 deletions docs/idelib/dataset.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
dataset.py
============
``idelib.dataset``
==================

.. automodule:: idelib.dataset

Expand Down
10 changes: 10 additions & 0 deletions docs/idelib/importer.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
``idelib.importer``
===================

.. automodule:: idelib.importer

.. autofunction:: idelib.importer.importFile

.. autofunction:: idelib.importer.openFile

.. autofunction:: idelib.importer.readData
6 changes: 6 additions & 0 deletions docs/idelib/sync.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
``idelib.sync``
===============

.. automodule:: idelib.sync
:members:

5 changes: 5 additions & 0 deletions docs/idelib/util.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
``idelib.util``
===============

.. automodule:: idelib.util
:members:
6 changes: 5 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
`idelib`: a Python Library for accessing enDAQ sensor recordings
================================================================

`idelib` is a Python API for accessing `enDAQ <https://endaq.com/>`_'s IDE recordings. The IDE format is an `EBML <https://github.com/ietf-wg-cellar/ebml-specification>`_ encoded file using a custom schema. This library utilizes our `ebmlite <https://github.com/MideTechnology/ebmlite>`_ to parse the files, and provides classes that make reading data simple.
`idelib` is the core Python API for accessing the contents of `enDAQ <https://endaq.com/>`_ IDE recordings.
It provides a means of easily accessing time series sensor data, with all necessary calibration applied, as
well as recording metadata.

The package also contains the command-line utilities ``ideexport`` and ``ideinfo`` for use outside of Python.

.. toctree::
:maxdepth: 2
:caption: Contents:

ide_basics
utilities
api_ref


Expand Down
132 changes: 132 additions & 0 deletions docs/utilities.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
IDE File Command-Line Utilities
===============================

``idelib`` provides some useful command-line utilities for converting IDE data to other formats,
and for viewing general information about an IDE file. These are automatically installed when
installing the ``idelib`` package.

Getting Started
---------------

Installing
''''''''''

The utilities are part of the ``idelib`` package, so they are installed along with it. The common
way to install is via ``pip`` (which is typically bundled with Python)::

pip install idelib


Running the scripts
'''''''''''''''''''

These utilities can be run in two ways: via an executable script, or as a Python module.

Running as a Python module
~~~~~~~~~~~~~~~~~~~~~~~~~~

You can run the utilities by running Python with the ``-m`` argument, followed by the name of the
submodule and any arguments, e.g.:

* ``python -m idelib.tools.ideexport --help``
* ``python -m idelib.tools.ideinfo --help``
* ``python -m idelib.tools.idesync --help``

While verbose, this is the most reliable way to run the utilities.

Running the executable
~~~~~~~~~~~~~~~~~~~~~~

When ``idelib`` is installed, it will create an executable file for each utility. The location
of the scripts varies by operating system (Linux/Windows/MacOS/etc.) and by Python environment (e.g., standard Python from
`python.org <https://python.org>`_ or `conda <https://anaconda.org/anaconda/conda>`_), but it should
be in your 'path', so running it is done by typing its name, followed by any arguments:

* ``ideexport --help``
* ``ideinfo --help``
* ``idesync --help``

Unfortunately, there are many factors specific to your computer and Python setup that can interfere
with this. Most commonly, the executables are somewhere outside of your 'path' and could not be found,
or security/authorization issues prevented the executables' creation. In either case, the result is
the standard 'not found' error for your OS.


``ideexport``
-------------

``ideexport`` converts one or more IDE files into text (CSV, etc.) or Matlab MAT v5 files.

.. code-block::

usage: ideexport.py [-h] [-o OUTPUT] [-t {csv,mat,txt}] [-c CHANNEL] [-m] [-u] [-n] [-r] [-d {comma,tab,pipe}] [-f] FILENAME.IDE [FILENAME.IDE ...]

Batch IDE Conversion Utility v3.4.0 - Copyright (c) 2026 Midé Technology

positional arguments:
FILENAME.IDE The source .IDE file(s) to convert. Wildcards permitted.

options:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
The output path to which to save the exported files. Defaults to the same location as the source file.
-t {csv,mat,txt}, --type {csv,mat,txt}
The type of file to export.
-c CHANNEL, --channel CHANNEL
Export the specific channel. Can be used multiple times. If not used, all channels will export.
-m, --removemean Remove the mean from accelerometer data.
-u, --utc Write timestamps as UTC 'Unix epoch' time.
-n, --names Include channel names in exported filenames.
-s, --sync Apply recordings' sychronization data (if present in the file).

Text Export Options (CSV, TXT, etc.):
-r, --headers Write 'header' information (column names) as the first row of text-based export.
-d {comma,tab,pipe}, --delimiter {comma,tab,pipe}
The delimiting character, for exporting non-CSV text-based files.
-f, --isoformat Write timestamps as ISO-formatted UTC.

``idesync``
-----------

``idesync`` generates synchronization metadata, synchronizing one or more files to a 'reference'
recording. This information, which is inserted into the recording, can be applied when exporting
with ``ideexport --sync``.

.. code-block::

usage: idesync.py [-h] [-g] [-i] FILENAME.IDE [FILENAME.IDE ...]

Batch IDE Synchronization Utility v3.4.0b1 - Copyright (c) 2025 Midé Technology

positional arguments:
FILENAME.IDE The recording to which to synchronize the others, and/or adjustits starting time if --gps is used.
FILENAME.IDE Recordings to sync to the reference.

options:
-h, --help show this help message and exit
-g, --gps Update the reference recording's start time using its GPS/GNSS data. Cannot be used in conjunction wtih --inherit.
-i, --inherit If the reference recording has been synced to another, sync the other datasets to that (rather than the reference itself). Cannot be used in conjunction with --gps.

Note: The applied synchronization is not currently displayed when viewing recordings in *enDAQ Lab*
(verion 3.1.1 or earlier).


``ideinfo``
-----------

``ideinfo`` gathers information about an IDE file, and either displays the text on the console or
writes it to a file.

.. code-block::

usage: ideinfo.py [-h] [-o OUTPUT] FILENAME.IDE [FILENAME.IDE ...]

IDE Info Viewer v3.4.0 - Copyright (c) 2026 Midé Technology

positional arguments:
FILENAME.IDE The source .IDE file(s) to convert. Wildcards permitted.

options:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
The output path to which to save the info text.
10 changes: 7 additions & 3 deletions idelib/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,24 @@
"""

__author__ = "David Randall Stokes"
__copyright__ = "Copyright (c) 2024 Midé Technology"
__copyright__ = "Copyright (c) 2026 Midé Technology"

__maintainer__ = "Midé Technology"
__email__ = "help@mide.com"

__version__ = '3.3.0'
__version__ = '3.4.0'

__status__ = "Production/Stable"

from .importer import importFile
import logging
logger = logging.getLogger(__name__)
logging.basicConfig(format="%(asctime)s %(levelname)s: %(message)s")

# Add EBML schema path to ebmlite search paths
import ebmlite

SCHEMA_PATH = "{idelib}/schemata"
if SCHEMA_PATH not in ebmlite.SCHEMA_PATH:
ebmlite.SCHEMA_PATH.insert(0, SCHEMA_PATH)

from .importer import importFile
16 changes: 5 additions & 11 deletions idelib/attributes.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,23 @@
"""

import datetime
import sys
import time
from typing import Any, Union

from ebmlite import loadSchema

# Dictionaries in Python 3.7+ are explicitly insert-ordered in all
# implementations. If older, continue to use `collections.OrderedDict`.
if sys.hexversion < 0x03070000:
from collections import OrderedDict as Dict
else:
Dict = dict

# ==============================================================================
#
# ==============================================================================

def decode_attributes(data, withTypes=False):
def decode_attributes(data: list[dict], withTypes: bool = False) -> dict:
""" Convert a set of Attributes (as a list of dictionaries containing an
`AttributeName` and one of the Attribute value elements (`IntAttribute`,
`FloatAttribute`, etc.) to a proper dictionary. Attributes are tagged
as 'multiple,' so they become lists when the EBML is parsed.
"""
result = Dict()
result = {}
for atts in data:
k = atts.pop('AttributeName', None)
if k is None:
Expand All @@ -41,7 +35,7 @@ def decode_attributes(data, withTypes=False):
return result


def encode_attributes(data):
def encode_attributes(data: Union[dict, tuple[str, Any]]) -> list[dict]:
""" Construct a set of `Attribute` dicts from a dictionary or list of
tuples/lists. Each value should be either a simple data type or a
tuple containing the value and the specific value element name
Expand Down Expand Up @@ -86,7 +80,7 @@ def encode_attributes(data):
return result


def build_attributes(data):
def build_attributes(data: Union[dict, tuple[str, Any]]) -> bytes:
""" Construct `Attribute` EBML from dictionary or list of key/value pairs.
Each value should be either a simple data type or a tuple containing
the value and the specific value element name (`IntAttribute`,
Expand Down
Loading