NDI-Python

Python implementation of the Neuroscience Data Interface (NDI) — a framework for managing, querying, and analyzing neuroscience experimental data.

NDI provides a unified interface for working with multi-modal neuroscience data (electrophysiology, imaging, stimulation) across different acquisition systems, with built-in support for time synchronization, document-based metadata, and cloud storage.

Features

Document management — JSON-schema-backed documents for experiments, subjects, probes, epochs, and more
Database — SQLite-backed storage with rich querying (regex, numeric, dependency graphs)
Time synchronization — Clock types, time mappings, and sync graphs for aligning data across devices
DAQ readers — Built-in readers for Intan, Blackrock, CED Spike2, and SpikeGadgets formats
Element/Probe hierarchy — Represent electrodes, optical fibers, and other recording devices
Session management — Directory-backed sessions with epoch discovery and file navigation
App framework — Extensible application framework with spike extraction, sorting, and stimulus analysis
Calculator framework — Reusable computation pipelines (tuning curves, orientation selectivity)
Cloud API — REST client for NDI Cloud with sync, upload/download, and DOI administration
Ontology providers — 13 providers (OLS, NCBITaxon, PubChem, RRID, UniProt, and more)
Schema validation — JSON Schema validation with superclass chain walking
OpenMINDS integration — Convert openMINDS metadata objects to NDI documents

Installation

git clone https://github.com/Waltham-Data-Science/NDI-python.git
cd NDI-python
python -m venv venv
source venv/bin/activate  # Linux/macOS (venv\Scripts\activate on Windows)
python ndi_install.py

The installer clones all dependencies, installs packages, and validates your setup. Run python -m ndi check at any time to verify your installation.

Updating

python ndi_install.py --update

Tutorials

python tutorials/tutorial_67f723d574f5f79c6062389d.py   # Dabrowska dataset
python tutorials/tutorial_682e7772cdf3f24938176fac.py   # Jess Haley dataset

See tutorials/README.md for full setup and cloud credentials.

Dependencies

NDI-Python requires these VH-Lab packages (installed automatically by ndi_install.py):

Package	Repository	Purpose
DID-python	VH-Lab/DID-python	Document database backend (SQLite, queries)
vhlab-toolbox-python	VH-Lab/vhlab-toolbox-python	Data utilities, file formats, signal processing

Additional dependencies (installed automatically): numpy, networkx, jsonschema, requests, scipy, pandas, matplotlib.

Manual installation (advanced)

git clone https://github.com/Waltham-Data-Science/NDI-python.git
cd NDI-python
python -m venv venv
source venv/bin/activate

# Clone vhlab-toolbox-python (not yet on PyPI)
git clone https://github.com/VH-Lab/vhlab-toolbox-python.git ~/.ndi/tools/vhlab-toolbox-python
export PYTHONPATH="$HOME/.ndi/tools/vhlab-toolbox-python:$PYTHONPATH"

# Install NDI-python (DID-python is resolved automatically via pip)
pip install -e ".[dev,tutorials]"
pip install scipy pandas matplotlib opencv-python-headless portalocker openminds

Quick Start

from ndi import Document, Query, DirSession
from ndi.session import MockSession

# Create a document with schema validation
doc = Document('base')
print(f"Document ID: {doc.id}")

# Query documents using Pythonic operators
q = Query('base.name') == 'my_experiment'
q_type = Query('').isa('subject')
q_combined = q & q_type

# Use MockSession for quick experimentation
with MockSession('test') as session:
    session.database_add(Document('base'))
    results = session.database_search(Query.all())
    print(f"Found {len(results)} documents")

# Use DirSession for persistent, directory-backed sessions
session = DirSession('my_experiment', '/path/to/data')
session.database_add(doc)
results = session.database_search(Query('').isa('base'))

Cloud API

from ndi.cloud.api.datasets import get_published_datasets

# Option 1: Set env vars and call without a client
#   export NDI_CLOUD_USERNAME="you@example.com"
#   export NDI_CLOUD_PASSWORD="your-password"
datasets = get_published_datasets()

# Option 2: Pass an explicit client
from ndi.cloud import CloudClient, login
config = login('you@example.com', 'your-password')
client = CloudClient(config)
datasets = get_published_datasets(client=client)

All ndi.cloud.api.* functions accept an optional client parameter. If omitted, a client is built automatically from environment variables.

Package Structure

src/ndi/
├── Core
│   ├── document.py            Document class with JSON schema loading
│   ├── query.py               Query builder with operator overloading (==, &, |)
│   ├── database.py            SQLite database backend
│   ├── ido.py                 Unique identifier generation (UUID-based)
│   ├── validate.py            JSON Schema validation with superclass chains
│   └── common/                PathConstants, timestamp, logging
│
├── Data Acquisition
│   ├── daq/                   DAQ system abstraction and readers
│   │   ├── reader/mfdaq/      Intan, Blackrock, CED Spike2, SpikeGadgets
│   │   └── metadatareader/    NewStim, NielsenLab metadata readers
│   ├── epoch/                 Epoch, EpochSet, EpochProbeMap
│   └── file/                  FileNavigator, EpochDirNavigator
│
├── Neural Elements
│   ├── element/               Element base class and utilities
│   ├── probe/                 Probe, ProbeTimeseries, ProbeTimeseriesMFDAQ
│   ├── element_timeseries.py  ElementTimeseries (data access)
│   ├── neuron.py              Neuron class
│   └── subject.py             Subject class
│
├── Sessions & Datasets
│   ├── session/               Session, DirSession, MockSession, SessionTable
│   ├── dataset.py             Multi-session Dataset container
│   └── cache.py               FIFO/LIFO cache implementations
│
├── Applications
│   ├── app/                   App framework, SpikeExtractor, SpikeSorter
│   │   └── stimulus/          StimulusDecoder, TuningResponse
│   ├── calc/                  Calculator framework
│   │   ├── example/           SimpleCalc
│   │   └── stimulus/          TuningCurveCalc, TuningFit
│   └── calculator.py          Calculator base class with run loop
│
├── Cloud & Sync
│   ├── cloud/                 CloudClient, CloudConfig
│   │   ├── api/               REST endpoints (datasets, documents, files, users)
│   │   ├── sync/              Sync engine (push, pull, index management)
│   │   └── admin/             DOI generation, Crossref submission
│   └── ontology/              13 ontology providers with LRU cache
│
├── Utilities
│   ├── fun/                   Utility functions (doc, epoch, file, data, stimulus)
│   ├── mock/                  Mock data generators for testing
│   ├── database_fun.py        Database search/export utilities
│   ├── database_ingestion.py  File ingestion/expulsion system
│   ├── openminds_convert.py   OpenMINDS object conversion
│   └── documentservice.py     DocumentService mixin
│
└── Shared Data (ndi_common/)
    ├── database_documents/    84 JSON document schemas
    ├── schema_documents/      JSON Schema validation files
    ├── probe/                 Probe type → class mapping
    └── ...                    Configs, ontology, vocabulary data

MATLAB Migration

This package is a complete Python port of the MATLAB NDI codebase. See MATLAB_MAPPING.md for the full function-by-function mapping.

Key API Differences

Concept	MATLAB	Python
Query creation	`ndi.query('field', 'exact_string', 'val')`	`Query('field') == 'val'`
Type query	`ndi.query('', 'isa', 'subject')`	`Query('').isa('subject')`
Combine queries	`ndi.query(q1, '&', q2)`	`q1 & q2`
Method names	`camelCase` (e.g., `setSessionId`)	`snake_case` (e.g., `set_session_id`)
Properties	`doc.id()` (method call)	`doc.id` (property access)
Session ID	`session.id()`	`session.id()` (still a method)

Architecture

                    DocumentService (mixin)
                    ├── Subject
                    └── Element ← Ido + EpochSet + DocumentService
                        ├── Probe (measurement devices)
                        │   └── ProbeTimeseries
                        │       ├── ProbeTimeseriesMFDAQ
                        │       └── ProbeTimeseriesStimulator
                        ├── ElementTimeseries (data access)
                        └── Neuron

App (DocumentService)
├── MarkGarbage
├── SpikeExtractor
├── SpikeSorter
├── StimulusDecoder
├── TuningResponse
└── Calculator (App + AppDoc)
    ├── SimpleCalc
    ├── TuningCurveCalc
    └── TuningFit (abstract)

Session (abstract)
├── DirSession (directory-backed)
└── MockSession (in-memory, for testing)

Dataset (multi-session container)

Development

Running Tests

# Run all tests (excludes symmetry tests by default)
pytest tests/ -v

# Run specific test module
pytest tests/test_document.py -v

# Run with coverage
pytest tests/ --cov=src/ndi --cov-report=term-missing

Cross-Language Symmetry Tests

Symmetry tests verify that NDI-python and NDI-matlab produce identical artifacts. They live in tests/symmetry/ and are excluded from the default test run because they require artifacts that may have been generated by a prior MATLAB session.

# Step 1 — Generate Python artifacts
pytest tests/symmetry/make_artifacts/ -v

# Step 2 — (Optional) Run MATLAB makeArtifacts to generate MATLAB artifacts
#   In MATLAB:  results = runtests('ndi.symmetry.makeArtifacts');

# Step 3 — Verify artifacts from both languages
pytest tests/symmetry/read_artifacts/ -v

# Run all symmetry tests at once (make + read)
pytest tests/symmetry/ -v

Tests that cannot find their expected artifact directory are skipped (not failed), so the suite runs safely on machines with only one language installed. See docs/developer_notes/symmetry_tests.md for the full framework description.

Code Quality

CI enforces formatting and lint on every push/PR:

# Format code (must pass `black --check` in CI)
black src/ tests/

# Lint (must pass `ruff check` in CI)
ruff check src/ tests/

# Auto-fix lint issues
ruff check --fix src/ tests/

# Type check (optional, not yet enforced in CI)
mypy src/ndi/

Building Documentation

pip install -e ".[docs]"
mkdocs build
mkdocs serve  # Local preview at http://127.0.0.1:8000

Test Coverage

1,704 tests passing across 30+ test files (Python 3.10, 3.11, 3.12)
Covers all modules: core, DAQ, time, session, app, cloud, ontology, validation
~71% line coverage across src/ndi/

License

This project is licensed under CC BY-NC-SA 4.0.

Contact Brandeis University Office of Technology Licensing for commercial licensing.

Acknowledgments

VH-Lab at Brandeis University — original MATLAB NDI codebase
Audri Bhowmick / Waltham Data Science — Python port
NDI Cloud — ndi-cloud.com for cloud storage and sharing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NDI-Python

Features

Installation

Updating

Tutorials

Dependencies

Quick Start

Cloud API

Package Structure

MATLAB Migration

Key API Differences

Architecture

Development

Running Tests

Cross-Language Symmetry Tests

Code Quality

Building Documentation

Test Coverage

License

Acknowledgments

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

NDI-Python

Features

Installation

Updating

Tutorials

Dependencies

Quick Start

Cloud API

Package Structure

MATLAB Migration

Key API Differences

Architecture

Development

Running Tests

Cross-Language Symmetry Tests

Code Quality

Building Documentation

Test Coverage

License

Acknowledgments