Merge pull request #22 from Waltham-Data-Science/claude/update-dev-docs-pCV94

Establish Bridge Protocol for MATLAB-Python parity

Author: stevevanhooser

.cursorrules

@@ -0,0 +1,18 @@
+# NDI Cursor Rules
+
+## 1. Project Context
+This is the NDI (Neuroscience Data Interface) Python port. It is a "Lead-Follow" project where the MATLAB codebase is the Source of Truth.
+
+## 2. Core Instructions
+Before proposing or writing any code, you MUST read and adhere to the following project-specific files:
+
+- **Entry Point:** Read `AGENTS.md` at the root for a high-level overview of your role.
+- **Porting Logic:** Read `docs/developer_notes/PYTHON_PORTING_GUIDE.md` for technical implementation rules (Pydantic, Naming, etc.).
+- **Parity Principles:** Read `docs/developer_notes/ndi_xlang_principles.md` for indexing vs. counting rules and data handling.
+- **The Contract:** Locate and read the `ndi_matlab_python_bridge.yaml` in the directory you are currently working in.
+
+## 3. Strict Naming Policy
+Do not attempt to convert MATLAB camelCase to snake_case. Maintain the casing defined in the bridge YAML files.
+
+## 4. Active Maintenance
+If a function is missing from the bridge YAML, add it based on the MATLAB source and notify the user: "INTERFACE UPDATE: I have modified the bridge contract for [Function Name]."

AGENTS.md

@@ -0,0 +1,38 @@
+# AGENTS.md
+
+## 1. Role & Mission
+
+You are an AI Developer for the NDI (Neuroscience Data Interface) project. Your mission is to maintain 1:1 functional and semantic parity between the mature MATLAB (Source of Truth) codebase and the Python port.
+
+## 2. The Mandatory Knowledge Base
+
+Before proposing, writing, or refactoring any code, you MUST read the following files in order:
+
+1. **Porting Protocol:** `docs/developer_notes/PYTHON_PORTING_GUIDE.md`
+   - Focus: Technical workflow, naming rules, Pydantic validation, and linting requirements.
+
+2. **Universal Principles:** `docs/developer_notes/ndi_xlang_principles.md`
+   - Focus: High-level logic rules (e.g., 0-vs-1 indexing, Semantic Parity for scientific counting, and NumPy usage).
+
+## 3. The Local Contract: The Bridge File
+
+Every sub-package contains a file named `ndi_matlab_python_bridge.yaml`.
+
+- **Rule 1: Consult the Bridge First.** This file defines the exact names, input arguments, and output tuples for that namespace.
+- **Rule 2: Active Maintenance.** If a function or class exists in MATLAB but is missing from the bridge file, you must:
+  1. Analyze the MATLAB `.m` file.
+  2. Add the new entry to the `ndi_matlab_python_bridge.yaml`.
+  3. **Notify the User:** You must state: "INTERFACE UPDATE: I have modified the bridge contract for [Function Name] to reflect the MATLAB source."
+- **Rule 3: Strict Naming.** You are forbidden from "Pythonizing" names (e.g., changing `ListAllDocuments` to `list_all_documents`) unless the bridge file explicitly instructs you to do so in the `decision_log`.
+
+## 4. Technical Constraints
+
+- **Validation:** All public API functions must use the `@pydantic.validate_call` decorator.
+- **Counting:** Any user-facing concept (Epochs, Channels, Trials) uses 1-based counting in Python to match MATLAB.
+- **Internal Access:** Use 0-based indexing for internal Python data structures (lists, NumPy arrays).
+- **Formatting:** Code must pass `black` and `ruff check --fix` before completion.
+
+## 5. Directory Mapping Reference
+
+- **MATLAB Source:** `VH-Lab/NDI-matlab` (GitHub)
+- **Python Target:** `src/ndi/[namespace]/` (Mirrors MATLAB `+namespace/`)

PYTHON_PORTING_GUIDE.md

@@ -0,0 +1,55 @@
+# NDI MATLAB to Python Porting Guide
+
+## 1. The Core Philosophy: Lead-Follow Architecture
+
+The MATLAB codebase is the **Source of Truth**. The Python version is a "faithful mirror." When a conflict arises between "Pythonic" style and MATLAB symmetry, **symmetry wins**.
+
+- **Lead-Follow:** MATLAB defines the logic, hierarchy, and naming.
+- **The Contract:** Every package contains an `ndi_matlab_python_bridge.yaml`. This file is the binding contract for function names, arguments, and return types for that specific namespace.
+
+## 2. Naming & Discovery (The Mirror Rule)
+
+Function and class names must match MATLAB exactly.
+
+- **Naming Source:** Refer to the local `ndi_matlab_python_bridge.yaml`.
+- **Missing Entries:** If a function is not in the bridge file, refer to the MATLAB source to determine the name, add the entry to the bridge file, and notify the user of the addition for their review.
+- **Case Preservation:** Use `ListAllDocuments`, not `list_all_documents`. Use `savetofile`, not `save_to_file`.
+- **Directory Parity:** Python file paths must mirror MATLAB `+namespace` paths (e.g., `+ndi/+cloud` → `src/ndi/cloud/`).
+
+## 3. The Porting Workflow (The Bridge Protocol)
+
+To port or update a function, agents must follow these steps:
+
+1. **Check the Bridge:** Open the `ndi_matlab_python_bridge.yaml` in the target package.
+2. **Sync the Interface:** If the function is missing or outdated, update the YAML entry first based on the MATLAB `.m` file.
+3. **Implement:** Write the Python code to satisfy the `input_arguments` and `output_arguments` defined in the YAML.
+4. **Log & Notify:** Document intentional divergences in the YAML's `decision_log`. Explicitly tell the user what changes were made to the bridge file so they can review the contract.
+
+## 4. Input Validation: Pydantic is Mandatory
+
+To replicate the robustness of the MATLAB `arguments` block, use Pydantic for all public-facing API functions.
+
+- **Decorator:** Use the `@pydantic.validate_call` decorator on all functions.
+- **Type Mirroring:**
+  - MATLAB `double`/`numeric` → Python `float | int`
+  - MATLAB `char`/`string` → Python `str`
+  - MATLAB `{member1, member2}` → Python `Literal["member1", "member2"]`
+- **Union Types:** Implement multiple allowed types as a Type Union (e.g., `str | int`).
+- **Coercion:** Allow Pydantic's default casting (e.g., allowing a string `"1"` to satisfy a `bool` type).
+- **Arbitrary Types:** For types like `numpy.ndarray`, use `config=ConfigDict(arbitrary_types_allowed=True)`.
+
+## 5. Multiple Returns (Outputs)
+
+MATLAB allows multiple return values natively. In Python, these must be returned as a **tuple** in the exact order defined in the `output_arguments` section of the bridge YAML.
+
+## 6. Code Style & Linting
+
+All Python code must pass formatting and linting before being committed.
+
+- **Black:** The sole code formatter. Use default line length (88).
+- **Ruff:** The primary linter. Run `ruff check --fix` before committing.
+
+## 7. Error Handling & Documentation
+
+- **Hard Fails:** If a MATLAB function throws an error, the Python version must raise a corresponding Exception (`ValueError`, `TypeError`, or `NDIError`).
+- **Docstring Symmetry:** Include the original MATLAB documentation in the Python docstring. Add a "Python-specific Notes" section at the bottom for library-specific details.

docs/developer_notes/PYTHON_PORTING_GUIDE.md

@@ -0,0 +1,83 @@
+# The NDI Bridge Protocol: YAML Specification
+
+# =============================================================================
+# 1. File Purpose & Placement
+# =============================================================================
+# Name:     ndi_matlab_python_bridge.yaml
+# Location: One file per sub-package directory
+#           (e.g., src/ndi/session/ndi_matlab_python_bridge.yaml).
+# Role:     This is the Primary Contract. It defines how MATLAB names and types
+#           map to Python. If a function is not here, it does not officially
+#           exist in the Python port.
+
+# =============================================================================
+# 2. Standard Header
+# =============================================================================
+# Every bridge file must begin with this metadata to orient the agent:
+
+project_metadata:
+  bridge_version: "1.1"
+  naming_policy: "Strict MATLAB Mirror"
+  indexing_policy: "Semantic Parity (1-based for user concepts, 0-based for internal data)"
+
+# =============================================================================
+# 3. The "Active Maintenance" Instruction
+# =============================================================================
+# When an agent or developer works on a function:
+#
+# 1. Check:  Does the function/class exist in the YAML?
+# 2. Add/Update: If it is missing or the MATLAB signature has changed,
+#    update the YAML first.
+# 3. Notify: The agent MUST explicitly tell the user:
+#    "I have updated the ndi_matlab_python_bridge.yaml to include
+#    [Function Name]. Please review the interface contract."
+
+# =============================================================================
+# 4. Structure for Classes and Functions
+# =============================================================================
+
+# --- Example: Class ---
+# - name: NDI_Session            # Exact MATLAB Name
+#   type: class
+#   matlab_path: "+ndi/Session.m"
+#   python_path: "ndi/session.py"
+#
+#   properties:
+#     - name: reference
+#       type_matlab: "char"
+#       type_python: "str"
+#       decision_log: "Mirroring property name exactly."
+#
+#   methods:
+#     - name: get_epoch_data
+#       input_arguments:
+#         - name: epoch_id
+#           type_python: "int"
+#           decision_log: "Semantic Parity: User provides 1-based ID."
+
+# --- Example: Standalone Function ---
+# - name: ListAllDocuments
+#   type: function
+#   matlab_path: "+ndi/+cloud/ListAllDocuments.m"
+#   python_path: "ndi/cloud/ListAllDocuments.py"
+#   input_arguments:
+#     - name: DatasetID
+#       type_matlab: "string | numeric"
+#       type_python: "str | int"
+#   output_arguments:
+#     - name: docs
+#       type_matlab: "struct array"
+#       type_python: "list[dict]"
+#   decision_log: "Python returns a list of dictionaries to mimic MATLAB struct arrays."
+
+# =============================================================================
+# 5. Summary of Field Rules
+# =============================================================================
+# Field             | Rule
+# ------------------|----------------------------------------------------------
+# name              | Strict Match. Case-sensitive match to the MATLAB .m file.
+# input_arguments   | Used to generate Pydantic @validate_call.
+# output_arguments  | Defines the order of the Return Tuple.
+# type_python       | Use Python 3.10+ syntax (e.g., str | int).
+# decision_log      | Mandatory for any divergence (e.g., "Shifted to
+#                   | 0-indexing for internal array access").