[did2 #3a] Make compat layer sufficient to flip the V_delta normalization gate

[stevevanhooser]

Tracking the work needed before [did2 #3] / #776 can be safely closed by flipping the NDI_DID2_NORMALIZE_ON_READ gate. PR #800 attempted the flip and revealed two compounding problems that the existing alias-table compat layer (#779 / #780 / #781) does not cover. PR #800 was closed without merging; the gate stays OFF until these are addressed.

Background

PR #799 added the alias-table compat triad — ndi.compat.augmentRead on read, ndi.compat.reconcileWrite on write, ndi.compat.translateQueryPaths on search — and was designed to make legacy did_v1 callers keep working once stored bodies normalise to V_delta. The triad is data-driven by ndi.compat.fieldAliases, which lists per-field mappings between V_delta canonical paths and did_v1 legacy paths.

That works for fields where v1 and V_delta both exist with different names (e.g., probe_location.ontology_name ↔ probe_location.location.node). PR #800's gate flip surfaced two cases where the triad is insufficient.

Problem 1 — depends_on struct-array schema fragility

ndi.compat.augmentRead mirrors depends_on(k).value into depends_on(k).id on every entry with a non-empty value. Because MATLAB struct arrays require a uniform field set, this grows the array's schema from {name, value} to {name, value, id}.

Every code path that later constructs a fresh struct('name', ..., 'value', ...) (only two fields) and merges it with that augmented array trips MATLAB:heterogeneousStrucAssignment.

PR #799 commit 76ba9a5 fixed the two sites that the test suite immediately hit (ndi.document/set_dependency_value line 733 and the plus() append branch at line 624) by routing through a new ndi.compat.dependsOnAppend helper. That fix is incomplete — at least the following sites still create {name, value} entries and may merge them into augmented arrays:

• src/ndi/+ndi/+calc/+stimulus/tuningcurve.m:525
• src/ndi/+ndi/+mock/+fun/stimulus_response.m:112
• The if any(tf) branch in ndi.document/plus line 621 (direct entry overwrite, which I explicitly chose not to fix in [did2 #6] ndi.document read-time legacy-alias augmentation #799)
• Various *.docs.parameter.examples/*.m.txt (less likely to be test-exercised, but still rotting examples)

Any of these can fail at runtime once the gate is ON.

Problem 2 — Fields dropped entirely by V_delta migrators

The compat layer mirrors V_delta → legacy. But at least one v1 field has no V_delta counterpart:

src/did/+did2/+convert/+migrators/daqreader_ndr.m
  "The v1 `ndi_daqreader_ndr_class` field has no V_delta counterpart and is dropped."

Legacy callers like ndi.database.fun.ndi_document2ndi_object (line 40) read obj_struct.ndi_<parent>_class to reconstruct the underlying MATLAB object via eval(). After gate-flip, the field is gone and the lookup errors:

Error: Unrecognized field name "ndi_daqreader_ndr_class".
  Error in ndi.database.fun.ndi_document2ndi_object (line 40)
    obj_string = getfield(obj_struct,['ndi_' obj_parent_string '_class']);
  Error in ndi.daq.system (line 60)
    obj.daqreader = ndi.database.fun.ndi_document2ndi_object(daqreader_doc, session);

Reproduction: ndi.symmetry.makeArtifacts.session.ingestionIntanNDR/testIngestionIntanNDRArtifacts on PR #800.

The alias model fundamentally cannot synthesize a field that V_delta dropped. There is nothing to mirror.

Audit needed

ndi.database.fun.ndi_document2ndi_object is called from at least 9 sites: probes, elements, daq systems, daq readers, daq metadata readers, filenavigators, syncgraphs. For each MATLAB class that uses the <class>.ndi_<class>_class reconstruction pattern, we need to know what the V_delta migrator does to that field:

• Pass-through unchanged (e.g., element.ndi_element_class per src/did/+did2/+convert/+migrators/element.m): legacy callers still work; no compat work needed.
• Renamed (e.g., daqmetadatareader.ndi_daqmetadatareader_class -> reader_class): add a row to ndi.compat.fieldAliases and the triad handles it.
• Dropped (e.g., daqreader_ndr.ndi_daqreader_ndr_class): structural decision required — see below.

Quick survey targets: daqreader, daqreader_ndr, daqsystem, daqmetadatareader, element, filenavigator, syncgraph, and any other class instantiated via eval() from a document body.

Structural decisions for dropped fields

For every "dropped" field surfaced by the audit, pick one of:

1. Restore the field in V_delta as optional. Update the migrator to preserve the legacy class string; declare the field as optional in the V_delta schema (did-schema). Augmentation then becomes identity. Cleanest if the field carries information V_delta otherwise loses.
2. Make the reader V_delta-aware. Update ndi.database.fun.ndi_document2ndi_object to derive the MATLAB constructor class from document_class.class_name (or another V_delta field) via a per-class mapping. Avoids polluting V_delta with v1-only metadata. Bespoke per-class.
3. Synthesize via augmentation. Extend ndi.compat.augmentRead to inject the legacy field from a hardcoded mapping. Functionally similar to (2) but lives in the compat layer rather than the consumer.

Likely a mix: option (1) where the class string carries real information that V_delta should preserve; option (2) where the v1 class string is just class(obj) reconstruction metadata that V_delta can derive.

Definition of done

• All depends_on write sites audited and migrated to a safe helper, or augmentation reworked to not extend struct-array schemas.
• All ndi_<class>_class lookups have a resolution path under V_delta (per the per-class decisions above).
• PR re-attempting the gate flip turns CI green on Vnext end-to-end.

Dependencies / blocks

• Blocks [did2 #3] / [did2 #3] Route v1→V_delta normalization through ndi.database (not session/dataset) #776 closeout.
• Blocks [did2 #12] / [did2 #12] Corpus end-to-end tests for the migration path #785 (corpus end-to-end tests) — those depend on normalization being on.
• Blocks [did2 #13] / [did2 #13] Cross-version cloud test: old NDI + new NDI on the same dataset #786 (cross-version cloud test) — same reason.
• Does NOT block [did2 #14] / [did2 #14] Migration guide (user-facing docs) #787 (migration guide) or [did2 #15] / [did2 #15] Release notes, deprecation timeline, version bump #788 (release notes) — those can proceed in parallel.

[stevevanhooser]

Paused pending two V_delta schema decisions.

Audit (above) surfaced two upstream design questions that, if resolved, materially simplify the compat work on this issue. Rather than touch the NDI-matlab compat layer twice, holding here until both land:

1. waltham-data-science/did-schema#49 — rename depends_on.value → depends_on.document_id in V_delta. Resolves "what's the canonical name" before we lock the alias in ndi.compat.fieldAliases.
2. waltham-data-science/did-schema#50 — restore daqmetadatareader.tab_separated_file_parameter in V_delta as optional. Eliminates one of the "dropped field" cases without needing an NDI-side workaround.

Once both schema issues + their did-matlab follow-ons land, this issue resumes with:

• (A) add the daqmetadatareader.reader_class ← ndi_daqmetadatareader_class row to ndi.compat.fieldAliases,
• (B) hardcode 'ndi.daq.reader.mfdaq.ndr' in ndi.database.fun.ndi_document2ndi_object for daqreader_ndr docs (the constant-value dropped field),
• (C) decide depends_on.id mirror strategy (keep + full audit vs. drop the mirror).

Both (A) and (C) will be revisited in light of the depends_on.document_id rename, so doing them now would create rework.

---

Generated by Claude Code