From add0604361ff97876c35c38ea6febb7b30beec45 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:24:01 -0400 Subject: [PATCH 01/32] =?UTF-8?q?v2:=20scaffold=20+did2=20package=20(docum?= =?UTF-8?q?ent,=20schema.cache)=20per=20PLAN.md=20=C2=A79=20step=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Begins step 1 of docs/v2/PLAN.md: introduce the parallel-namespace +did2 package that holds V_gamma documents in their flat JSON shape (no V_alpha base.* / document_class.* nesting). Added: - src/did/+did2/document.m: V_gamma document object. Settled API surface (fromJSON/fromStruct/blank, get/set/iterate, toJSON, className/classVersion, validate). Dot-path get/set is implemented; [*] array iteration is exposed via iterate(). Construct-from-blank and validate delegate to the schema cache. - src/did/+did2/+schema/cache.m: V_gamma schema cache class. Singleton bootstrap, schema-path resolution (DID_SCHEMA_PATH env override or sibling did-schema checkout), getClass, and superclass traversal are implemented; fieldsFor, queryablePaths, buildBlankDocument, and validateDocument are stubs that throw did2:notImplemented and will be filled in next. - src/did/+did2/Contents.m: package overview. - tests/+did2/testDocumentScaffold.m: function-based unit tests for the scaffolded surface (construction, dot-path get/set, iterate, JSON round-trip, error IDs). - docs/v2/PLAN.md: progress log entry documenting what landed and what comes next; +did2 chosen as the provisional namespace, logged in §1. Note: per AGENTS.md these files have not been executed in MATLAB. They are intended for human review and test in a licensed MATLAB environment. --- docs/v2/PLAN.md | 41 ++++ src/did/+did2/+schema/cache.m | 226 +++++++++++++++++++++ src/did/+did2/Contents.m | 30 +++ src/did/+did2/document.m | 315 +++++++++++++++++++++++++++++ tests/+did2/testDocumentScaffold.m | 66 ++++++ 5 files changed, 678 insertions(+) create mode 100644 src/did/+did2/+schema/cache.m create mode 100644 src/did/+did2/Contents.m create mode 100644 src/did/+did2/document.m create mode 100644 tests/+did2/testDocumentScaffold.m diff --git a/docs/v2/PLAN.md b/docs/v2/PLAN.md index ff7038a..32f27e6 100644 --- a/docs/v2/PLAN.md +++ b/docs/v2/PLAN.md @@ -19,6 +19,7 @@ users. | 4 | Keep the `matlabdumbjsondb` backend. | Useful for tests, trivial deployments, and as a non-SQL reference implementation of the query model. | | 5 | Validate on insert by default; expose an `unsafe_insert` escape hatch for bulk loads; offer a `revalidate_all` maintenance op. | Schema files are the source of truth for what "valid" means. | | 6 | Plan lives at `docs/v2/PLAN.md` on the v2 development branch. | This file. | +| 7 | Provisional namespace: `+did2`. | Picked from §10 option A for the scaffold. Revisit before v2 reaches `main`. | Open questions are in §10. @@ -293,3 +294,43 @@ no-op, so it no longer appears in `compile_options`. The functional tests Test 4 passing is the decisive simplification: queryable scalar paths can live as `STORED` generated columns on `documents` with their own indexes (§3.2), with no separate sidecar table for scalars. + +--- + +## 12. Progress log + +### 2026-05-11 — step 1 scaffold + +Started step 1 of §9 on branch `claude/start-v2-development-tA41P`. + +Added: + +- `src/did/+did2/document.m` — V_gamma document object. API surface + in place (construct from JSON / struct / `(className, values)`, + `get` / `set` / `iterate`, `toJSON` / `toStruct`, `className` / + `classVersion`, `validate`, plus static `fromJSON` / `fromStruct` / + `blank`). Dot-path get/set is implemented in full. The `[*]` array + iterator is implemented via `iterate(arrayPath)`; the bare `get` + rejects paths containing `[*]` to keep the scalar/array distinction + honest. `validate` and `blank` delegate to the schema cache. +- `src/did/+did2/+schema/cache.m` — schema cache class. Singleton + bootstrap, schema-path resolution (env override + `DID_SCHEMA_PATH`, or sibling `did-schema/schemas/V_gamma` checkout), + `getClass`, and `superclasses` traversal are implemented; the + heavier methods (`fieldsFor`, `queryablePaths`, + `buildBlankDocument`, `validateDocument`) currently throw + `did2:notImplemented` and will be filled in next. +- `src/did/+did2/Contents.m` — package overview. +- `tests/+did2/testDocumentScaffold.m` — function-based unit tests + covering construction, dot-path get/set, iterate, round-trip JSON, + and the documented error IDs. Tests that depend on the schema cache + beyond what is implemented are deferred. + +Provisional decision (logged in §1 as #7): use `+did2` for the v2 +namespace during the scaffold, leaving the §10 rename-vs-parallel +question open for resolution before v2 reaches `main`. + +Next up: fill in `did2.schema.cache.fieldsFor`, +`queryablePaths`, and `buildBlankDocument`; then `validateDocument` +against the V_gamma meta-schema; then start the in-memory query +evaluator (step 2). diff --git a/src/did/+did2/+schema/cache.m b/src/did/+did2/+schema/cache.m new file mode 100644 index 0000000..7ce979e --- /dev/null +++ b/src/did/+did2/+schema/cache.m @@ -0,0 +1,226 @@ +classdef cache < handle + % did2.schema.cache V_gamma schema cache (DID v2 scaffold). + % + % Loads all V_gamma schema files once and pre-computes the + % structural information that the document, query, and database + % layers depend on (PLAN.md §5): + % + % - per-classname full inherited _fields list + % - the queryable subset, split into scalar paths and + % array-iteration paths + % - named composite type expansions (duration -> .seconds / + % .approximate / .source_unit / .source_value, ontology_term + % -> .node / .name, etc.) + % - the CURIE registry from CURIE_lookups_meta.json + % + % Most methods on this scaffold throw 'did2:notImplemented' for + % the parts that have not been wired up yet. The shape of the API + % is what is being committed here, not the behaviour. + % + % did2.schema.cache Properties: + % schemaPath - filesystem path to a V_gamma schema dir. + % loadedClasses - containers.Map of classname -> raw schema. + % curieRegistry - parsed CURIE_lookups_meta.json contents. + % + % did2.schema.cache Static Methods: + % shared - return the process-wide singleton cache. + % setSchemaPath - override the schema path for the singleton. + % + % did2.schema.cache Methods: + % getClass - resolved class definition for a name. + % superclasses - flat list of ancestors for a class. + % fieldsFor - merged field list across inheritance. + % queryablePaths - scalar and array-iteration paths. + % buildBlankDocument - struct of _blank_value defaults. + % validateDocument - validate a did2.document instance. + % + % See also: did2.document, docs/v2/PLAN.md. + + properties (SetAccess = private) + schemaPath (1,:) char = '' + loadedClasses + curieRegistry struct = struct() + end + + methods (Access = private) + function obj = cache(schemaPath) + % Private constructor — use did2.schema.cache.shared(). + arguments + schemaPath (1,:) char = did2.schema.cache.defaultSchemaPath() + end + obj.schemaPath = schemaPath; + obj.loadedClasses = containers.Map('KeyType', 'char', 'ValueType', 'any'); + obj.loadRegistry(); + end + end + + methods + function s = getClass(obj, className) + % getClass - return the parsed schema struct for className. + arguments + obj + className (1,:) char + end + if obj.loadedClasses.isKey(className) + s = obj.loadedClasses(className); + return; + end + schemaFile = fullfile(obj.schemaPath, [className '.json']); + if ~isfile(schemaFile) + error('did2:schema:missingClass', ... + 'No schema file for class "%s" at %s.', className, schemaFile); + end + s = jsondecode(fileread(schemaFile)); + obj.loadedClasses(className) = s; + end + + function names = superclasses(obj, className) + % superclasses - ordered list of ancestor classnames (root last). + arguments + obj + className (1,:) char + end + names = {}; + current = className; + visited = containers.Map('KeyType', 'char', 'ValueType', 'logical'); + while true + if visited.isKey(current) + error('did2:schema:cycle', ... + 'Superclass cycle detected starting at "%s".', className); + end + visited(current) = true; + s = obj.getClass(current); + parents = obj.extractField(s, '_superclasses'); + if isempty(parents) + break; + end + parent = parents(1); + parentName = obj.extractField(parent, '_classname'); + names{end+1} = parentName; %#ok + current = parentName; + end + end + + function fields = fieldsFor(~, ~) + % fieldsFor - return the merged _fields list for className, + % walking superclasses root-first so that subclasses can + % override (which V_gamma does not currently allow, but + % the traversal is stable across future relaxations). + error('did2:notImplemented', ... + 'did2.schema.cache.fieldsFor is not yet implemented.'); + end + + function paths = queryablePaths(~, ~) + % queryablePaths - return a struct with fields: + % .scalar : cellstr of dot-path strings whose schema field + % has _queryable: true and is not an + % array-of-structure. + % .array : cellstr of dot-path strings ending in [*] for + % array-of-structure queryable sub-fields. + % Used by the SQL backend (PLAN §3.2/§3.3) to generate + % stored columns and the queryable_array_elem sidecar. + error('did2:notImplemented', ... + 'did2.schema.cache.queryablePaths is not yet implemented.'); + end + + function s = buildBlankDocument(~, ~) + % buildBlankDocument - return a struct populated with the + % _blank_value of every field declared by className and + % its superclasses, plus _class metadata. + error('did2:notImplemented', ... + 'did2.schema.cache.buildBlankDocument is not yet implemented.'); + end + + function validateDocument(~, ~) + % validateDocument - raise if the supplied did2.document + % does not conform to its V_gamma class definition. + error('did2:notImplemented', ... + 'did2.schema.cache.validateDocument is not yet implemented.'); + end + end + + methods (Static) + function obj = shared(varargin) + % shared - return the process-wide cache singleton. + % + % shared() returns the existing singleton, creating it on + % first call using the default schema path. + % shared(schemaPath) sets the schema path on first call; if + % the singleton already exists, the argument is ignored + % (use setSchemaPath to rebuild). + % shared('-reset') drops the singleton. + persistent instance + if nargin == 1 && ischar(varargin{1}) && strcmp(varargin{1}, '-reset') + instance = []; + obj = []; + return; + end + if isempty(instance) || ~isvalid(instance) + if nargin >= 1 && ~isempty(varargin{1}) + schemaPath = varargin{1}; + else + schemaPath = did2.schema.cache.defaultSchemaPath(); + end + instance = did2.schema.cache(schemaPath); + end + obj = instance; + end + + function setSchemaPath(schemaPath) + % setSchemaPath - rebuild the singleton against a new schema path. + arguments + schemaPath (1,:) char + end + did2.schema.cache.shared('-reset'); + did2.schema.cache.shared(schemaPath); + end + + function resetSingleton() + % resetSingleton - drop the cached singleton so the next + % .shared() call constructs a fresh instance. Intended for + % tests. + did2.schema.cache.shared('-reset'); + end + end + + methods (Static, Access = private) + function p = defaultSchemaPath() + % defaultSchemaPath - filesystem location of V_gamma schemas. + % + % The default looks for a sibling did-schema checkout next + % to the did-matlab repository. Override via the + % DID_SCHEMA_PATH environment variable or by calling + % setSchemaPath(). + envOverride = getenv('DID_SCHEMA_PATH'); + if ~isempty(envOverride) + p = envOverride; + return; + end + toolboxDir = did.toolboxdir(); + p = fullfile(toolboxDir, '..', '..', 'did-schema', 'schemas', 'V_gamma'); + end + end + + methods (Access = private) + function loadRegistry(obj) + registryFile = fullfile(obj.schemaPath, 'CURIE_lookups_meta.json'); + if isfile(registryFile) + obj.curieRegistry = jsondecode(fileread(registryFile)); + end + end + + function value = extractField(~, s, name) + % extractField - tolerate jsondecode's underscore-prefix + % renaming, which on some MATLAB releases mangles leading + % underscores into 'x_' or strips them. + candidates = {name, ['x' name], strrep(name, '_', '')}; + for k = 1:numel(candidates) + if isfield(s, candidates{k}) + value = s.(candidates{k}); + return; + end + end + value = []; + end + end +end diff --git a/src/did/+did2/Contents.m b/src/did/+did2/Contents.m new file mode 100644 index 0000000..71634bb --- /dev/null +++ b/src/did/+did2/Contents.m @@ -0,0 +1,30 @@ +% +did2 DID v2 (V_gamma) MATLAB toolbox — development scaffold. +% +% The +did2 package is the parallel-namespace v2 line of DID-matlab. +% It consumes the V_gamma schema set from the did-schema repository +% directly, without translating to the V_alpha base.* / +% document_class.* / nesting that the legacy +did +% package uses. See docs/v2/PLAN.md for the full design and the +% step-by-step order of work. +% +% Files +% document - V_gamma document object (load / validate / +% serialise / dot-path access). +% Contents - this overview. +% +% Subpackages +% +schema - schema cache and validation entry points. +% +convert - (planned) v1-to-v2 conversion utilities. +% +query - (planned) query tree, in-memory evaluator, +% SQLite/JSON1 compiler. +% +% Conventions +% - New code uses camelCase identifiers and arguments-block input +% validation, per AGENTS.md. +% - Document data is the flat V_gamma JSON shape (top-level +% snake_case keys; system metadata prefixed with `_`). +% - The schema cache is the single source of truth for what +% "valid" means; runtime reflection over values never substitutes +% for the schema. +% +% See also: did, docs/v2/PLAN.md. diff --git a/src/did/+did2/document.m b/src/did/+did2/document.m new file mode 100644 index 0000000..0887571 --- /dev/null +++ b/src/did/+did2/document.m @@ -0,0 +1,315 @@ +classdef document < handle + % did2.document V_gamma document object (DID v2 scaffold). + % + % A did2.document holds a single DID document in its V_gamma JSON + % shape, validates it against the V_gamma schema set, and serialises + % it back to JSON. Unlike did.document, the internal representation + % is the flat V_gamma shape directly — no translation to/from the + % V_alpha base.* / document_class.* / nesting. + % + % This is the v2 development scaffold (PLAN.md §9, item 1). The + % outer API is intended to stabilise here; internal methods are + % filled in iteratively. Stubs throw 'did2:notImplemented' so that + % missing pieces surface loudly rather than silently no-op. + % + % did2.document Properties: + % documentProperties - struct mirroring the V_gamma JSON shape. + % + % did2.document Methods: + % document - construct from JSON text, a struct, or + % (className, valueStruct). + % get - dot-path getter into documentProperties. + % set - dot-path setter into documentProperties. + % iterate - element iterator over an array-of-structure path + % (used by the in-memory query evaluator for [*]). + % toJSON - serialise to a JSON string. + % toStruct - return the underlying struct. + % className - shorthand for get('_class.name'). + % classVersion - shorthand for get('_class.version'). + % validate - validate this document against its schema. + % + % did2.document Static Methods: + % fromJSON - construct from a JSON string. + % fromStruct - construct from a struct. + % blank - construct a blank instance of the named class. + % + % See also: did2.schema.cache, did.document, docs/v2/PLAN.md. + + properties + % documentProperties - struct mirroring the V_gamma JSON shape. + % Top-level keys are flat snake_case (e.g., id, session_id, + % name, datestamp) plus class-defined fields. System metadata + % carries a leading underscore (_class, _depends_on, _files). + documentProperties (1,1) struct = struct() + end + + properties (Access = private) + % schemaCacheHandle - lazily resolved did2.schema.cache instance. + schemaCacheHandle = [] + end + + methods + function obj = document(varargin) + % document - construct a did2.document. + % + % D = did2.document() creates an empty document. + % D = did2.document(jsonText) parses a JSON string. + % D = did2.document(s) wraps an existing struct. + % D = did2.document(className, valueStruct) builds a blank + % instance of className and overlays valueStruct. + + if nargin == 0 + return; + end + + firstArg = varargin{1}; + if nargin == 1 && (ischar(firstArg) || (isstring(firstArg) && isscalar(firstArg))) + obj.documentProperties = did2.document.parseJSONText(firstArg); + elseif nargin == 1 && isstruct(firstArg) + obj.documentProperties = firstArg; + elseif nargin >= 1 && (ischar(firstArg) || (isstring(firstArg) && isscalar(firstArg))) ... + && nargin == 2 && isstruct(varargin{2}) + obj.documentProperties = did2.document.buildBlank(char(firstArg)); + obj.documentProperties = did2.document.mergeStruct( ... + obj.documentProperties, varargin{2}); + else + error('did2:document:badInput', ... + 'did2.document accepts (), (jsonText), (struct), or (className, valueStruct).'); + end + end + + function value = get(obj, fieldPath) + % get - read documentProperties at a dot-path. + % + % v = doc.get('sample_rate.hertz') returns the hertz field + % inside the sample_rate named composite. The [*] array + % iteration suffix is not handled here — use iterate() for + % that, since [*] returns a struct array rather than a scalar. + arguments + obj + fieldPath (1,:) char + end + value = did2.document.dotPathGet(obj.documentProperties, fieldPath); + end + + function obj = set(obj, fieldPath, value) + % set - write a value at a dot-path inside documentProperties. + % + % doc.set('app.app_name', 'ndi_app_spikeextractor') sets the + % nested field, creating intermediate structs as needed. + arguments + obj + fieldPath (1,:) char + value + end + obj.documentProperties = did2.document.dotPathSet( ... + obj.documentProperties, fieldPath, value); + end + + function elements = iterate(obj, arrayPath) + % iterate - return the element list at an array-of-structure path. + % + % els = doc.iterate('axes') returns the struct array stored + % at the 'axes' path. Used by the in-memory query evaluator + % to implement the V_gamma '[*]' existential semantics + % described in did_query_model.md. + arguments + obj + arrayPath (1,:) char + end + elements = did2.document.dotPathGet(obj.documentProperties, arrayPath); + if isempty(elements) + elements = struct([]); + elseif ~isstruct(elements) + error('did2:document:notArrayOfStructure', ... + 'Path "%s" is not an array-of-structure field.', arrayPath); + end + end + + function jsonText = toJSON(obj, opts) + % toJSON - serialise documentProperties to JSON text. + arguments + obj + opts.PrettyPrint (1,1) logical = false + end + jsonText = jsonencode(obj.documentProperties, ... + 'PrettyPrint', opts.PrettyPrint); + end + + function s = toStruct(obj) + % toStruct - return the underlying documentProperties struct. + s = obj.documentProperties; + end + + function name = className(obj) + % className - shorthand for get('_class.name'). + name = obj.get('_class.name'); + end + + function v = classVersion(obj) + % classVersion - shorthand for get('_class.version'). + v = obj.get('_class.version'); + end + + function validate(obj, opts) + % validate - check this document against its V_gamma schema. + % + % doc.validate() resolves the schema cache from the default + % path, looks up the document's class definition, and + % verifies required fields, type constraints, and the named + % composite layouts (ontology_term, duration, length, ...). + % + % doc.validate(SchemaCache=cache) uses the supplied cache + % instead of the shared singleton. + arguments + obj + opts.SchemaCache = [] + end + cache = obj.resolveSchemaCache(opts.SchemaCache); + cache.validateDocument(obj); + end + end + + methods (Static) + function obj = fromJSON(jsonText) + % fromJSON - construct a did2.document from a JSON string. + arguments + jsonText (1,:) char + end + obj = did2.document(jsonText); + end + + function obj = fromStruct(s) + % fromStruct - construct a did2.document from a struct. + arguments + s (1,1) struct + end + obj = did2.document(s); + end + + function obj = blank(className, opts) + % blank - construct a blank V_gamma document of the named class. + % + % d = did2.document.blank('app') builds an instance of the + % 'app' class with every field set to its '_blank_value' as + % declared by the V_gamma schema. _class metadata is filled + % from the schema; id and datestamp are populated with a + % freshly generated did_uid and the current UTC timestamp. + arguments + className (1,:) char + opts.SchemaCache = [] + end + obj = did2.document(); + obj.documentProperties = did2.document.buildBlank(className, opts.SchemaCache); + end + + function value = dotPathGet(s, fieldPath) + % dotPathGet - read a nested value out of struct s by dot-path. + arguments + s + fieldPath (1,:) char + end + if contains(fieldPath, '[*]') + error('did2:document:arrayPathHere', ... + ['"%s" contains [*]; use iterate() for ' ... + 'array-of-structure traversal.'], fieldPath); + end + parts = strsplit(fieldPath, '.'); + value = s; + for k = 1:numel(parts) + segment = parts{k}; + if ~isstruct(value) || ~isfield(value, segment) + error('did2:document:missingField', ... + 'Field "%s" not present while resolving "%s".', ... + segment, fieldPath); + end + value = value.(segment); + end + end + + function s = dotPathSet(s, fieldPath, value) + % dotPathSet - write value into struct s at the given dot-path. + arguments + s (1,1) struct + fieldPath (1,:) char + value + end + parts = strsplit(fieldPath, '.'); + s = did2.document.assignNested(s, parts, value); + end + end + + methods (Static, Access = private) + function out = parseJSONText(jsonText) + text = char(jsonText); + out = jsondecode(text); + if ~isstruct(out) + error('did2:document:jsonNotObject', ... + 'Top-level JSON value must be an object, got %s.', class(out)); + end + end + + function s = assignNested(s, parts, value) + head = parts{1}; + if numel(parts) == 1 + s.(head) = value; + return; + end + if isfield(s, head) && isstruct(s.(head)) + inner = s.(head); + else + inner = struct(); + end + s.(head) = did2.document.assignNested(inner, parts(2:end), value); + end + + function s = mergeStruct(base, overlay) + % mergeStruct - shallow overlay of overlay onto base. + % Scalar struct fields in overlay overwrite base; nested + % structs recurse. Non-struct values overwrite. + s = base; + if ~isstruct(overlay) + return; + end + f = fieldnames(overlay); + for k = 1:numel(f) + name = f{k}; + if isfield(s, name) && isstruct(s.(name)) && isstruct(overlay.(name)) + s.(name) = did2.document.mergeStruct(s.(name), overlay.(name)); + else + s.(name) = overlay.(name); + end + end + end + + function s = buildBlank(className, cacheOverride) + % buildBlank - assemble a blank V_gamma document by walking + % the schema cache for className and its superclasses, + % populating each field with its _blank_value, then filling + % _class metadata, id (freshly minted), session_id (blank), + % and datestamp (current UTC). + if nargin < 2 + cacheOverride = []; + end + if isempty(cacheOverride) + cache = did2.schema.cache.shared(); + else + cache = cacheOverride; + end + s = cache.buildBlankDocument(className); + end + end + + methods (Access = private) + function cache = resolveSchemaCache(obj, override) + if ~isempty(override) + cache = override; + return; + end + if isempty(obj.schemaCacheHandle) + obj.schemaCacheHandle = did2.schema.cache.shared(); + end + cache = obj.schemaCacheHandle; + end + end +end diff --git a/tests/+did2/testDocumentScaffold.m b/tests/+did2/testDocumentScaffold.m new file mode 100644 index 0000000..d85375e --- /dev/null +++ b/tests/+did2/testDocumentScaffold.m @@ -0,0 +1,66 @@ +function tests = testDocumentScaffold +% testDocumentScaffold - smoke tests for the did2.document scaffold. +% +% Run with: +% results = runtests('testDocumentScaffold'); +% +% These tests exercise the surface API of did2.document that does not +% depend on the schema cache being fully implemented yet: +% construction from a struct or JSON, dot-path get/set, iterate(), +% and toJSON()/toStruct() round-trips. Tests that depend on the +% schema cache are deferred until did2.schema.cache implements +% buildBlankDocument and validateDocument. + +tests = functiontests(localfunctions); +end + +function testConstructFromStruct(testCase) +s = struct('id', 'abc', 'session_id', 'sess', 'name', 'unit-test'); +doc = did2.document(s); +verifyEqual(testCase, doc.get('id'), 'abc'); +verifyEqual(testCase, doc.get('name'), 'unit-test'); +end + +function testConstructFromJSON(testCase) +jsonText = '{"id":"abc","sample_rate":{"hertz":30000,"approximate":false}}'; +doc = did2.document(jsonText); +verifyEqual(testCase, doc.get('id'), 'abc'); +verifyEqual(testCase, doc.get('sample_rate.hertz'), 30000); +verifyFalse(testCase, doc.get('sample_rate.approximate')); +end + +function testSetCreatesNestedPath(testCase) +doc = did2.document(); +doc.set('app.app_name', 'ndi_app_spikeextractor'); +verifyEqual(testCase, doc.get('app.app_name'), 'ndi_app_spikeextractor'); +end + +function testToJSONRoundTrip(testCase) +s = struct('id', 'abc', 'datestamp', '2026-05-11T00:00:00.000Z'); +doc = did2.document(s); +jsonText = doc.toJSON(); +doc2 = did2.document(jsonText); +verifyEqual(testCase, doc2.toStruct(), s); +end + +function testIterateReturnsStructArray(testCase) +s = struct('axes', struct('name', {'x','y','z'}, 'unit', {'um','um','um'})); +doc = did2.document(s); +elements = doc.iterate('axes'); +verifyEqual(testCase, numel(elements), 3); +verifyEqual(testCase, elements(2).name, 'y'); +end + +function testGetMissingFieldErrors(testCase) +doc = did2.document(struct('id', 'abc')); +verifyError(testCase, @() doc.get('nope.missing'), 'did2:document:missingField'); +end + +function testGetRejectsArrayPath(testCase) +doc = did2.document(struct('axes', struct('name', {'x'}))); +verifyError(testCase, @() doc.get('axes[*].name'), 'did2:document:arrayPathHere'); +end + +function testBadConstructorInput(testCase) +verifyError(testCase, @() did2.document(42), 'did2:document:badInput'); +end From cab747f6dee5c8ef7e004326679d1cac3990a64d Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:37:26 -0400 Subject: [PATCH 02/32] ci(test-code): also run on V2 and PRs targeting V2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V2 is the long-lived development line for the +did2 / V_gamma rewrite (docs/v2/PLAN.md). Self-tests should gate the V2 branch the same way they gate main. Symmetry tests are intentionally left on main only — the V_gamma document layout is not yet wired up to the symmetry fixtures. --- .github/workflows/test-code.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 2eb5556..0dd6837 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -2,13 +2,13 @@ name: Test code on: push: - branches: main + branches: [main, V2] paths-ignore: - '*.md' - '.github/**' pull_request: - branches: main + branches: [main, V2] paths-ignore: - '*.md' - '.github/**' From b05361aadacf0a093041fa590700256536349c05 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:37:30 -0400 Subject: [PATCH 03/32] ci(codespell): also run on V2 and PRs targeting V2 Spelling check is cheap and prose-heavy V2 docs (docs/v2/PLAN.md and similar) benefit from it. Pair with the test-code update so any push or PR against V2 gets the same gates as main, minus the symmetry tests. --- .github/workflows/run-codespell.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/run-codespell.yml b/.github/workflows/run-codespell.yml index 66d5159..0bad8a9 100644 --- a/.github/workflows/run-codespell.yml +++ b/.github/workflows/run-codespell.yml @@ -2,10 +2,10 @@ name: Run Codespell on: push: - branches: [ "main" ] + branches: [ "main", "V2" ] pull_request: - branches: [ "main" ] + branches: [ "main", "V2" ] concurrency: group: ${{ github.workflow }}-${{ github.ref }} From ca7fceb97d2c2e97949ecaee5e24c93063c1baa8 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:42:42 -0400 Subject: [PATCH 04/32] ci(test-code): add workflow_dispatch for manual runs Lets us manually fire the self-test workflow from the Actions UI against any branch. Note: the "Run workflow" button only appears once this file is on the default branch (main); until V2 merges to main, the trigger is defined but not surfaced in the UI. --- .github/workflows/test-code.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 0dd6837..fdcaf17 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -13,6 +13,8 @@ on: - '*.md' - '.github/**' + workflow_dispatch: + concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true From bfca084adeef2d674482dc65b65c83be86b14703 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:42:45 -0400 Subject: [PATCH 05/32] ci(codespell): add workflow_dispatch for manual runs Same caveat as test-code: the manual "Run workflow" button only shows in the Actions UI once this file lands on the default branch (main). --- .github/workflows/run-codespell.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/run-codespell.yml b/.github/workflows/run-codespell.yml index 0bad8a9..787148c 100644 --- a/.github/workflows/run-codespell.yml +++ b/.github/workflows/run-codespell.yml @@ -7,6 +7,8 @@ on: pull_request: branches: [ "main", "V2" ] + workflow_dispatch: + concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true From 1e4f345a20c070c281d081e5ebd768b64af9e3ea Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:52:59 -0400 Subject: [PATCH 06/32] ci(test-code): pin matbox-actions to last known-good SHA The floating @v1 tag was moved upstream on 2026-05-01/02 to a commit whose workflow calls codecheckToolbox with more arguments than the installed matbox MATLAB toolbox accepts, producing: Error using codecheckToolbox Too many input arguments. Main's last green test-code run was on 2026-03-31, before that retag. Pin to 4132d36 (Nov 2025), which is the workflow-file revision main was using when it last passed. Comment explains the pin so we can unpin once matbox-actions is fixed upstream. --- .github/workflows/test-code.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index fdcaf17..5eb898f 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -22,7 +22,12 @@ concurrency: jobs: test-code: name: Analyse and test code - uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@v1 + # Pinned to the matbox-actions SHA that main's last green CI run used + # (Nov 2025). The floating @v1 tag was moved on 2026-05-01/02 to a + # commit that calls codecheckToolbox with arguments the installed + # matbox MATLAB toolbox does not accept ("Too many input arguments"). + # Revisit once upstream is fixed. + uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@4132d36 with: matlab_release: R2021b matlab_use_cache: true From 49cdd29f384cf65a64e71915e008b09a0f4ced7d Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 16:58:13 -0400 Subject: [PATCH 07/32] ci(test-code): flatten to NDI-matlab's working pattern The reusable workflow ehennestad/matbox-actions/.../test-code-workflow.yml was retagged on 2026-05-01/02 to call codecheckToolbox with arguments that the installed matbox MATLAB toolbox rejects ("Too many input arguments"). Pinning to an older SHA of the reusable workflow did not help, suggesting the new call shape lives in the older revision too. NDI-matlab's run-tests.yml uses the individual matbox-actions (install-matbox + check-code) at @v1 directly and is currently green. Model test-code.yml on that pattern: - actions/checkout@v4 - matlab-actions/setup-matlab@v2 (R2021b) - ehennestad/matbox-actions/install-matbox@v1 - matbox.installRequirements(...) for mksqlite + vhlab-toolbox-matlab - ehennestad/matbox-actions/check-code@v1 - TestSuite.fromFolder("tests", "IncludingSubfolders", true) Removed: - The reusable-workflow `uses:` (replaced by explicit steps). - The CODECOV_TOKEN secret pass-through (no coverage XML produced yet; add back with a matbox.testToolbox-style entry point later). The +did2 tests at tests/+did2/testDocumentScaffold.m are picked up automatically by fromFolder with IncludingSubfolders. --- .github/workflows/test-code.yml | 62 ++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 12 deletions(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 5eb898f..ee76242 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -21,16 +21,54 @@ concurrency: jobs: test-code: + # Flat workflow modeled on NDI-matlab/.github/workflows/run-tests.yml, + # which is currently green on @v1 pins. The previous + # ehennestad/matbox-actions reusable workflow (test-code-workflow.yml) + # was retagged on 2026-05-01/02 to call codecheckToolbox with new + # arguments the installed matbox MATLAB toolbox rejects ("Too many + # input arguments"). Calling install-matbox + check-code directly, + # the way NDI-matlab does, avoids the broken wrapper. name: Analyse and test code - # Pinned to the matbox-actions SHA that main's last green CI run used - # (Nov 2025). The floating @v1 tag was moved on 2026-05-01/02 to a - # commit that calls codecheckToolbox with arguments the installed - # matbox MATLAB toolbox does not accept ("Too many input arguments"). - # Revisit once upstream is fixed. - uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@4132d36 - with: - matlab_release: R2021b - matlab_use_cache: true - matlab_products: Statistics_and_Machine_Learning_Toolbox - secrets: - CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + runs-on: ubuntu-latest + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up MATLAB + uses: matlab-actions/setup-matlab@v2 + with: + release: R2021b + cache: true + products: Statistics_and_Machine_Learning_Toolbox + + - name: Install MatBox + uses: ehennestad/matbox-actions/install-matbox@v1 + + - name: Install repo dependencies (mksqlite, vhlab-toolbox-matlab) + uses: matlab-actions/run-command@v2 + with: + command: | + addpath(genpath("src")); + addpath(genpath("tools")); + matbox.installRequirements(didtools.projectdir()); + + - name: Check code + uses: ehennestad/matbox-actions/check-code@v1 + with: + source_directory: 'src' + + - name: Run tests + uses: matlab-actions/run-command@v2 + if: always() + with: + command: | + addpath(genpath("src")); + addpath(genpath("tests")); + import matlab.unittest.TestRunner; + import matlab.unittest.TestSuite; + runner = TestRunner.withTextOutput; + suite = TestSuite.fromFolder("tests", "IncludingSubfolders", true); + results = runner.run(suite); + disp(table(results)); + nFailed = sum([results.Failed]); + assert(nFailed == 0, sprintf("%d test(s) failed", nFailed)); From d56900fa4ec8e8e31d6673e49aeed379d7676adf Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 18:55:23 -0400 Subject: [PATCH 08/32] ci(test-code): drop check-code step to dodge matbox codecheck breakage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ehennestad/matbox-actions/check-code@v1 fails on fresh runners with "codecheckToolbox: Too many input arguments" — the action passes more arguments than the installed matbox MATLAB toolbox accepts. The same action passes for NDI-matlab because that repo's setup-matlab cache holds an older matbox image, kept warm by daily CI runs. Ours is cold and pulls the current matbox, exposing the mismatch. Drop check-code from test-code.yml so the self-test suite can run. Static analysis can be re-added once matbox-actions is fixed upstream, or via MATLAB's built-in codeAnalyzer. --- .github/workflows/test-code.yml | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index ee76242..472e5e3 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -21,14 +21,16 @@ concurrency: jobs: test-code: - # Flat workflow modeled on NDI-matlab/.github/workflows/run-tests.yml, - # which is currently green on @v1 pins. The previous - # ehennestad/matbox-actions reusable workflow (test-code-workflow.yml) - # was retagged on 2026-05-01/02 to call codecheckToolbox with new - # arguments the installed matbox MATLAB toolbox rejects ("Too many - # input arguments"). Calling install-matbox + check-code directly, - # the way NDI-matlab does, avoids the broken wrapper. - name: Analyse and test code + # Flat workflow that just runs the self-test suite. The + # ehennestad/matbox-actions/check-code@v1 step that this used to + # call breaks on fresh runners with "codecheckToolbox: Too many + # input arguments" — the matbox MATLAB toolbox installed by + # install-matbox no longer matches what check-code passes to it. + # NDI-matlab's CI still passes because its setup-matlab cache + # predates the matbox change; ours is cold. Drop the static- + # analysis step for now; re-add either when matbox-actions is + # fixed upstream, or via MATLAB's built-in codeAnalyzer. + name: Run self-tests runs-on: ubuntu-latest steps: - name: Check out repository @@ -52,14 +54,8 @@ jobs: addpath(genpath("tools")); matbox.installRequirements(didtools.projectdir()); - - name: Check code - uses: ehennestad/matbox-actions/check-code@v1 - with: - source_directory: 'src' - - name: Run tests uses: matlab-actions/run-command@v2 - if: always() with: command: | addpath(genpath("src")); From 0ae38ac4860a390aa341d06f892ebc6497ccf105 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 18:59:13 -0400 Subject: [PATCH 09/32] ci(test-code): bump MATLAB from R2021b to latest Matches NDI-matlab's run-tests.yml, which currently passes daily on release: latest. R2021b is from 2021 and predates many matbox changes; current matbox may rely on features (e.g. arguments-block extensions) that R2021b mis-parses, which could itself be the source of the "Too many input arguments" surfacing through codecheckToolbox. Even if not the root cause, aligning with NDI's known-green release removes one variable. --- .github/workflows/test-code.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 472e5e3..089891a 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -39,7 +39,7 @@ jobs: - name: Set up MATLAB uses: matlab-actions/setup-matlab@v2 with: - release: R2021b + release: latest cache: true products: Statistics_and_Machine_Learning_Toolbox From 497a999f98a3ceaeb7babb5dc42fd5527e00702d Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 19:13:46 -0400 Subject: [PATCH 10/32] ci(test-code): restore check-code step now that release: latest is in The codecheckToolbox "Too many input arguments" failure was driven by the R2021b pin; matbox's current check-code uses MATLAB features beyond R2021b. With release: latest, NDI-matlab runs the same check-code@v1 successfully every day. Re-add the step so we get the static-analysis gate alongside the test run. Test run keeps `if: always()` so a check-code failure doesn't mask test results. --- .github/workflows/test-code.yml | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 089891a..66662c0 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -21,16 +21,12 @@ concurrency: jobs: test-code: - # Flat workflow that just runs the self-test suite. The - # ehennestad/matbox-actions/check-code@v1 step that this used to - # call breaks on fresh runners with "codecheckToolbox: Too many - # input arguments" — the matbox MATLAB toolbox installed by - # install-matbox no longer matches what check-code passes to it. - # NDI-matlab's CI still passes because its setup-matlab cache - # predates the matbox change; ours is cold. Drop the static- - # analysis step for now; re-add either when matbox-actions is - # fixed upstream, or via MATLAB's built-in codeAnalyzer. - name: Run self-tests + # Flat workflow modeled on NDI-matlab/.github/workflows/run-tests.yml. + # Uses release: latest because the current matbox MATLAB toolbox + # (called via check-code) requires features beyond R2021b; on + # R2021b check-code fails with "codecheckToolbox: Too many input + # arguments". + name: Analyse and test code runs-on: ubuntu-latest steps: - name: Check out repository @@ -54,8 +50,14 @@ jobs: addpath(genpath("tools")); matbox.installRequirements(didtools.projectdir()); + - name: Check code + uses: ehennestad/matbox-actions/check-code@v1 + with: + source_directory: 'src' + - name: Run tests uses: matlab-actions/run-command@v2 + if: always() with: command: | addpath(genpath("src")); From 1bb7ce2db64f3a08acd02cc41a9453291cd55ba9 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 19:20:08 -0400 Subject: [PATCH 11/32] ci(test-code): run check-code before installRequirements Mirrors NDI-matlab's run-tests.yml step order. The previous order ran matbox.installRequirements() before check-code, and check-code failed there with "codecheckToolbox: Too many input arguments" even on release: latest. NDI does not call installRequirements at all and its check-code@v1 step passes daily, so the suspicion is that installRequirements modifies the MATLAB environment in a way that mis-aligns check-code's call to codecheckToolbox. installRequirements is moved after check-code (still required for the tests, which use mksqlite). Both later steps gain `if: always()` so a check-code failure does not skip them. --- .github/workflows/test-code.yml | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 66662c0..95506e3 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -22,10 +22,13 @@ concurrency: jobs: test-code: # Flat workflow modeled on NDI-matlab/.github/workflows/run-tests.yml. - # Uses release: latest because the current matbox MATLAB toolbox - # (called via check-code) requires features beyond R2021b; on - # R2021b check-code fails with "codecheckToolbox: Too many input - # arguments". + # Order matters: check-code runs immediately after install-matbox, + # before any other step touches the MATLAB path. An earlier ordering + # that ran matbox.installRequirements() first made check-code fail + # with "codecheckToolbox: Too many input arguments" even on + # release: latest. installRequirements is moved after check-code + # since the tests need mksqlite + vhlab-toolbox-matlab but the + # static analysis does not. name: Analyse and test code runs-on: ubuntu-latest steps: @@ -42,19 +45,20 @@ jobs: - name: Install MatBox uses: ehennestad/matbox-actions/install-matbox@v1 + - name: Check code + uses: ehennestad/matbox-actions/check-code@v1 + with: + source_directory: 'src' + - name: Install repo dependencies (mksqlite, vhlab-toolbox-matlab) uses: matlab-actions/run-command@v2 + if: always() with: command: | addpath(genpath("src")); addpath(genpath("tools")); matbox.installRequirements(didtools.projectdir()); - - name: Check code - uses: ehennestad/matbox-actions/check-code@v1 - with: - source_directory: 'src' - - name: Run tests uses: matlab-actions/run-command@v2 if: always() From d2ac002af280629022359bc12b07117142615137 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 19:32:01 -0400 Subject: [PATCH 12/32] tools: move codecheckToolbox wrapper into +didtools to stop shadowing The bare tools/tasks/codecheckToolbox.m wrapper was being picked up by addpath(genpath('./tools')) inside ehennestad/matbox-actions/check-code@v1, which shadowed the real matbox.tasks.codecheckToolbox. check-code calls codecheckToolbox with multiple arguments; our zero-arg wrapper rejected them with "Too many input arguments", breaking CI. Move the wrapper into the +didtools package so it's invoked as didtools.codecheckToolbox and never collides with the matbox symbol. The old tools/tasks/codecheckToolbox.m is removed in the next commit. --- tools/+didtools/codecheckToolbox.m | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 tools/+didtools/codecheckToolbox.m diff --git a/tools/+didtools/codecheckToolbox.m b/tools/+didtools/codecheckToolbox.m new file mode 100644 index 0000000..6cc4163 --- /dev/null +++ b/tools/+didtools/codecheckToolbox.m @@ -0,0 +1,16 @@ +function codecheckToolbox() + % didtools.codecheckToolbox Developer convenience wrapper for + % matbox.tasks.codecheckToolbox: run the project code check + % against the DID-matlab repo root with badge writing disabled. + % + % Call as `didtools.codecheckToolbox` from the MATLAB prompt. + % + % This used to live at tools/tasks/codecheckToolbox.m (bare name) + % but check-code@v1 puts tools/ on the path and resolved to this + % zero-arg wrapper instead of matbox.tasks.codecheckToolbox's + % multi-arg version, breaking CI with "Too many input arguments". + % Moving it into +didtools/ keeps the convenience without + % shadowing matbox. + projectRootDirectory = didtools.projectdir(); + matbox.tasks.codecheckToolbox(projectRootDirectory, "CreateBadge", false) +end From 1ae7ad5a71312673fa5dc8d33d5d1cfc4fff4b6c Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 19:32:12 -0400 Subject: [PATCH 13/32] tools: delete shadowed bare codecheckToolbox wrapper Replaced by tools/+didtools/codecheckToolbox.m in the previous commit. The bare wrapper at tools/tasks/codecheckToolbox.m shadowed matbox.tasks.codecheckToolbox once tools/ was on the MATLAB path and broke ehennestad/matbox-actions/check-code@v1 with "Too many input arguments". Removing it lets check-code resolve to matbox's version. --- tools/tasks/codecheckToolbox.m | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 tools/tasks/codecheckToolbox.m diff --git a/tools/tasks/codecheckToolbox.m b/tools/tasks/codecheckToolbox.m deleted file mode 100644 index 07d140f..0000000 --- a/tools/tasks/codecheckToolbox.m +++ /dev/null @@ -1,4 +0,0 @@ -function codecheckToolbox() - projectRootDirectory = didtools.projectdir(); - matbox.tasks.codecheckToolbox(projectRootDirectory, "CreateBadge", false) -end From 2dcc611a7e16538ad75a9b355d82ff09797d1e03 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 20:48:50 -0400 Subject: [PATCH 14/32] v2: add V_gamma test fixtures for +did2 self-tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hermetic V_gamma schema fixtures under tests/+did2/fixtures/V_gamma/ so the +did2 self-tests can run without a sibling did-schema checkout or network access. base.json root V_gamma class (upstream copy) CURIE_lookups_meta.json trimmed CURIE registry (subset of upstream) demoA.json V_gamma translation of v1's demoA (base + char value) demoB.json multi-level inheritance (demoB -> demoA -> base) demoC.json three _depends_on entries (V_gamma translation of v1 demoC) demoFile.json two _file attachments (V_gamma translation of v1 demoFile) README.md origins and refresh procedure Schemas themselves did not change between the flat-fields and class-scoped versions of V_gamma — only the document-instance wire shape did — so these fixtures are stable across that change. --- .../fixtures/V_gamma/CURIE_lookups_meta.json | 36 +++++++++ tests/+did2/fixtures/V_gamma/README.md | 30 ++++++++ tests/+did2/fixtures/V_gamma/base.json | 76 +++++++++++++++++++ tests/+did2/fixtures/V_gamma/demoA.json | 30 ++++++++ tests/+did2/fixtures/V_gamma/demoB.json | 30 ++++++++ tests/+did2/fixtures/V_gamma/demoC.json | 49 ++++++++++++ tests/+did2/fixtures/V_gamma/demoFile.json | 39 ++++++++++ 7 files changed, 290 insertions(+) create mode 100644 tests/+did2/fixtures/V_gamma/CURIE_lookups_meta.json create mode 100644 tests/+did2/fixtures/V_gamma/README.md create mode 100644 tests/+did2/fixtures/V_gamma/base.json create mode 100644 tests/+did2/fixtures/V_gamma/demoA.json create mode 100644 tests/+did2/fixtures/V_gamma/demoB.json create mode 100644 tests/+did2/fixtures/V_gamma/demoC.json create mode 100644 tests/+did2/fixtures/V_gamma/demoFile.json diff --git a/tests/+did2/fixtures/V_gamma/CURIE_lookups_meta.json b/tests/+did2/fixtures/V_gamma/CURIE_lookups_meta.json new file mode 100644 index 0000000..93e5306 --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/CURIE_lookups_meta.json @@ -0,0 +1,36 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://did-schema.example.org/meta/CURIE_lookups_meta.json", + "title": "CURIE Registry for DID/NDI V_gamma", + "description": "Registry mapping CURIE prefixes (used in ontology_term.node values and in field-level _ontology._node annotations) to their authoritative URI base and metadata. Consumer tooling uses this file to expand a CURIE such as 'uberon:0002436' into the full URI 'http://purl.obolibrary.org/obo/UBERON_0002436'. Prefixes are matched case-insensitively; by convention prefixes are written in lowercase inside documents and annotations.", + "_format_version": "1.0.3", + "_prefixes": { + "iao": { + "_label": "Information Artifact Ontology", + "_uri_base": "http://purl.obolibrary.org/obo/IAO_", + "_uri_style": "obo_underscore", + "_approximate": false, + "_documentation": "OBO Foundry ontology for information entities. Expansion rule: 'iao:NNNNNNN' -> 'http://purl.obolibrary.org/obo/IAO_NNNNNNN'." + }, + "schema": { + "_label": "Schema.org", + "_uri_base": "https://schema.org/", + "_uri_style": "direct", + "_approximate": false, + "_documentation": "General-purpose linked-data vocabulary. CURIE local part is the term name (e.g., 'schema:name' -> 'https://schema.org/name')." + }, + "placeholder": { + "_label": "Placeholder namespace for example and in-progress data", + "_uri_base": "", + "_uri_style": "local", + "_approximate": true, + "_documentation": "Reserved placeholder for documentation examples and for values that have not yet been mapped to a real ontology." + } + }, + "_uri_styles": { + "obo_underscore": "Concatenate _uri_base with the CURIE local part verbatim; underscore-separated form.", + "direct": "Concatenate _uri_base with the CURIE local part verbatim.", + "local": "No URI expansion; the CURIE is the authoritative identifier." + }, + "_documentation": "Test fixture subset of upstream CURIE_lookups_meta.json. The +did2 self-tests only need this registry to be parseable and non-empty." +} diff --git a/tests/+did2/fixtures/V_gamma/README.md b/tests/+did2/fixtures/V_gamma/README.md new file mode 100644 index 0000000..793e00a --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/README.md @@ -0,0 +1,30 @@ +# V_gamma test fixtures + +Hermetic schema fixtures for the `+did2` MATLAB test suite. These exist +so unit tests do not depend on a sibling `did-schema` checkout or on +network access. + +## Files + +| File | Origin | Purpose | +|---|---|---| +| `base.json` | copied from [`Waltham-Data-Science/did-schema/schemas/V_gamma/base.json`](https://github.com/Waltham-Data-Science/did-schema/blob/main/schemas/V_gamma/base.json) | The root V_gamma class. Carries `id`, `session_id`, `name`, `datestamp`. | +| `CURIE_lookups_meta.json` | trimmed subset of upstream | CURIE prefix registry. The cache loads this on construction. | +| `demoA.json` | V_gamma translation of v1's `src/did/example_schema/demo_schema1/database_documents/demoA.json` | Extends `base`. Adds one queryable `value` field. | +| `demoB.json` | V_gamma translation of v1's `demoB.json` | Extends `demoA`. Adds one `value_b` field. Tests multi-level inheritance (`demoB -> demoA -> base`). | +| `demoC.json` | V_gamma translation of v1's `demoC.json` | Extends `base`. Declares three `_depends_on` entries (`item1`, `item2`, `item3`). Tests dependency declarations. | +| `demoFile.json` | V_gamma translation of v1's `demoFile.json` | Extends `base`. Declares two `_file` attachments. Tests file-record declarations. | + +## Conventions + +The `_schema` token in `_superclasses` entries (e.g. +`$DIDSCHEMAPATH/base.json`) is illustrative only — the +did2 cache +resolves superclasses by classname, looking up sibling JSON files in +this directory. + +## Refreshing + +If `base.json` or `CURIE_lookups_meta.json` change upstream, re-copy +them from the same paths above. The `demo*.json` files are V_gamma-only +fixtures and have no upstream counterpart; they evolve with the +did2 +test suite. diff --git a/tests/+did2/fixtures/V_gamma/base.json b/tests/+did2/fixtures/V_gamma/base.json new file mode 100644 index 0000000..04c8deb --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/base.json @@ -0,0 +1,76 @@ +{ + "_classname": "base", + "_class_version": "1.0.0", + "_maturity_level": "work_in_progress", + "_superclasses": [], + "_depends_on": [], + "_file": [], + "_fields": [ + { + "_name": "id", + "type": "did_uid", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": true, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": { + "_node": "iao:0000578", + "_name": "centrally registered identifier" + }, + "_documentation": "Unique identifier for this document instance.", + "_constraints": {} + }, + { + "_name": "session_id", + "type": "did_uid", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": true, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": { + "_node": "ncit:C169028", + "_name": "Study Unique Identifier" + }, + "_documentation": "Unique identifier of the session this document belongs to.", + "_constraints": {} + }, + { + "_name": "name", + "type": "char", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": false, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": { + "_node": "schema:name", + "_name": "name" + }, + "_documentation": "Human-readable name for this document.", + "_constraints": { + "maxLength": 256 + } + }, + { + "_name": "datestamp", + "type": "timestamp", + "_blank_value": "", + "_default_value": "2018-12-05T18:36:47.241Z", + "_mustBeNonEmpty": true, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": { + "_node": "schema:dateCreated", + "_name": "dateCreated" + }, + "_documentation": "UTC timestamp of document creation in ISO 8601 format.", + "_constraints": {} + } + ] +} diff --git a/tests/+did2/fixtures/V_gamma/demoA.json b/tests/+did2/fixtures/V_gamma/demoA.json new file mode 100644 index 0000000..7a2304f --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/demoA.json @@ -0,0 +1,30 @@ +{ + "_classname": "demoA", + "_class_version": "1.0.0", + "_maturity_level": "work_in_progress", + "_superclasses": [ + { + "_classname": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ], + "_depends_on": [], + "_file": [], + "_fields": [ + { + "_name": "value", + "type": "char", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": false, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": null, + "_documentation": "A demo scalar value used by the +did2 self-tests.", + "_constraints": { + "maxLength": 256 + } + } + ] +} diff --git a/tests/+did2/fixtures/V_gamma/demoB.json b/tests/+did2/fixtures/V_gamma/demoB.json new file mode 100644 index 0000000..29e00e7 --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/demoB.json @@ -0,0 +1,30 @@ +{ + "_classname": "demoB", + "_class_version": "1.0.0", + "_maturity_level": "work_in_progress", + "_superclasses": [ + { + "_classname": "demoA", + "_schema": "$DIDSCHEMAPATH/demoA.json" + } + ], + "_depends_on": [], + "_file": [], + "_fields": [ + { + "_name": "value_b", + "type": "char", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": false, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": null, + "_documentation": "A second demo scalar value, declared by demoB on top of demoA.", + "_constraints": { + "maxLength": 256 + } + } + ] +} diff --git a/tests/+did2/fixtures/V_gamma/demoC.json b/tests/+did2/fixtures/V_gamma/demoC.json new file mode 100644 index 0000000..7007b10 --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/demoC.json @@ -0,0 +1,49 @@ +{ + "_classname": "demoC", + "_class_version": "1.0.0", + "_maturity_level": "work_in_progress", + "_superclasses": [ + { + "_classname": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ], + "_depends_on": [ + { + "_name": "item1", + "_mustBeNonEmpty": true, + "_documentation": "The first dependency item, by id.", + "_must_refer_to_document_class": "" + }, + { + "_name": "item2", + "_mustBeNonEmpty": true, + "_documentation": "The second dependency item, by id.", + "_must_refer_to_document_class": "" + }, + { + "_name": "item3", + "_mustBeNonEmpty": false, + "_documentation": "The third dependency item; optional.", + "_must_refer_to_document_class": "" + } + ], + "_file": [], + "_fields": [ + { + "_name": "value", + "type": "char", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": false, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": null, + "_documentation": "A demo scalar value carried alongside the dependencies.", + "_constraints": { + "maxLength": 256 + } + } + ] +} diff --git a/tests/+did2/fixtures/V_gamma/demoFile.json b/tests/+did2/fixtures/V_gamma/demoFile.json new file mode 100644 index 0000000..1f5fe4d --- /dev/null +++ b/tests/+did2/fixtures/V_gamma/demoFile.json @@ -0,0 +1,39 @@ +{ + "_classname": "demoFile", + "_class_version": "1.0.0", + "_maturity_level": "work_in_progress", + "_superclasses": [ + { + "_classname": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ], + "_depends_on": [], + "_file": [ + { + "_name": "filename1.ext", + "_documentation": "First demo binary file attached to the document." + }, + { + "_name": "filename2.ext2", + "_documentation": "Second demo binary file attached to the document." + } + ], + "_fields": [ + { + "_name": "value", + "type": "char", + "_blank_value": "", + "_default_value": "", + "_mustBeNonEmpty": false, + "_mustBeScalar": true, + "_mustNotHaveNaN": false, + "_queryable": true, + "_ontology": null, + "_documentation": "A demo scalar value carried alongside the file attachments.", + "_constraints": { + "maxLength": 256 + } + } + ] +} From 905ff8d350f3872611fe32e469ab64790ab88d32 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 20:51:31 -0400 Subject: [PATCH 15/32] v2: implement did2.schema.cache for V_gamma class-scoped wire shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fills in the cache to the level step 1 of docs/v2/PLAN.md needs, against V_gamma's class-scoped property-block document layout (V_gamma_SPEC.md "JSON Format: Document Instances"). Implemented: - classChain(className): root-first chain including the class itself. - ownFields(className): the _fields list the class declares directly. - fieldsFor(className): merged inherited fields tagged with the declaring class (struct array of {declaringClass, fieldDef}). - buildBlankDocument(className): V_gamma class-scoped doc with x_classname / x_class_version / x_superclasses / x_depends_on at the top level, plus one property block per class in the chain (empty {} for zero-field classes). base.id auto-minted via did.ido.unique_id(); base.datestamp set to current UTC ISO-8601. - validateDocument(docOrStruct): walks the class chain, validates each class's own _fields against its property block. Type-shape check runs before _mustBe* flags so a wrong type is reported as did2:validation:typeMismatch rather than did2:validation:notScalar. New error IDs: missingClassBlock, badClassBlock; reuses emptyField, notScalar, typeMismatch, maxLength, minLength, minimum, maximum, enum, nanValue, missingClassName. queryablePaths remains a stub — it belongs to steps 3 and 4 once the storage layer lands, and will return class-qualified dot-paths (e.g., 'daqsystem.sample_rate.hertz'). The MATLAB-side storage convention (x_ instead of _ for the four leading-underscore JSON keys) is documented in PLAN.md §4.1 and mirrored by did2.document.toJSON's regex rewrite on serialise. --- src/did/+did2/+schema/cache.m | 477 ++++++++++++++++++++++++++++------ 1 file changed, 404 insertions(+), 73 deletions(-) diff --git a/src/did/+did2/+schema/cache.m b/src/did/+did2/+schema/cache.m index 7ce979e..cbe2126 100644 --- a/src/did/+did2/+schema/cache.m +++ b/src/did/+did2/+schema/cache.m @@ -1,21 +1,30 @@ classdef cache < handle - % did2.schema.cache V_gamma schema cache (DID v2 scaffold). + % did2.schema.cache V_gamma schema cache. % - % Loads all V_gamma schema files once and pre-computes the - % structural information that the document, query, and database - % layers depend on (PLAN.md §5): + % Loads V_gamma schema files lazily, resolves superclass chains, + % builds blank documents in the V_gamma class-scoped wire shape, + % and validates documents against their class definitions. See + % docs/v2/PLAN.md §5. % - % - per-classname full inherited _fields list - % - the queryable subset, split into scalar paths and - % array-iteration paths - % - named composite type expansions (duration -> .seconds / - % .approximate / .source_unit / .source_value, ontology_term - % -> .node / .name, etc.) - % - the CURIE registry from CURIE_lookups_meta.json + % Document shape (V_gamma "JSON Format: Document Instances"): + % _classname string concrete class + % _class_version string semver of the concrete class + % _superclasses array [{_classname, _class_version}] + % _depends_on array [{_name, value}] + % object one property block per class in + % the chain, keyed by _classname. + % Contains the field values that + % class declared (empty {} if the + % class declares no fields). % - % Most methods on this scaffold throw 'did2:notImplemented' for - % the parts that have not been wired up yet. The shape of the API - % is what is being committed here, not the behaviour. + % MATLAB representation: leading-underscore JSON keys can't be + % MATLAB struct field names, so we store them with the same `x_` + % prefix MATLAB's `jsondecode` produces — `x_classname`, + % `x_class_version`, `x_superclasses`, `x_depends_on`. The + % class-block keys (`base`, `demoA`, ...) are plain snake_case + % identifiers and stay as written. did2.document.toJSON rewrites + % `"x_":` back to `"_":` on serialisation; jsondecode + % reverses that on parse. % % did2.schema.cache Properties: % schemaPath - filesystem path to a V_gamma schema dir. @@ -24,14 +33,18 @@ % % did2.schema.cache Static Methods: % shared - return the process-wide singleton cache. - % setSchemaPath - override the schema path for the singleton. + % setSchemaPath - rebuild the singleton at a new schema path. + % resetSingleton - drop the cached singleton (test helper). % % did2.schema.cache Methods: % getClass - resolved class definition for a name. - % superclasses - flat list of ancestors for a class. - % fieldsFor - merged field list across inheritance. - % queryablePaths - scalar and array-iteration paths. - % buildBlankDocument - struct of _blank_value defaults. + % superclasses - ancestor chain (parent first, root last). + % classChain - root-first list including the class itself. + % ownFields - the _fields list a class declares directly. + % fieldsFor - merged inherited fields tagged with the + % declaring class (struct array). + % queryablePaths - scalar and array-iteration paths (stub). + % buildBlankDocument - blank V_gamma document in the wire shape. % validateDocument - validate a did2.document instance. % % See also: did2.document, docs/v2/PLAN.md. @@ -75,7 +88,8 @@ end function names = superclasses(obj, className) - % superclasses - ordered list of ancestor classnames (root last). + % superclasses - ancestor chain (parent first, root last). + % For 'demoA' -> {'base'}. For 'base' -> {}. arguments obj className (1,:) char @@ -94,61 +108,151 @@ if isempty(parents) break; end - parent = parents(1); + parent = obj.elementAt(parents, 1); parentName = obj.extractField(parent, '_classname'); - names{end+1} = parentName; %#ok - current = parentName; + names{end+1} = char(parentName); %#ok + current = char(parentName); end end - function fields = fieldsFor(~, ~) - % fieldsFor - return the merged _fields list for className, - % walking superclasses root-first so that subclasses can - % override (which V_gamma does not currently allow, but - % the traversal is stable across future relaxations). - error('did2:notImplemented', ... - 'did2.schema.cache.fieldsFor is not yet implemented.'); - end - - function paths = queryablePaths(~, ~) - % queryablePaths - return a struct with fields: - % .scalar : cellstr of dot-path strings whose schema field - % has _queryable: true and is not an - % array-of-structure. - % .array : cellstr of dot-path strings ending in [*] for - % array-of-structure queryable sub-fields. - % Used by the SQL backend (PLAN §3.2/§3.3) to generate - % stored columns and the queryable_array_elem sidecar. - error('did2:notImplemented', ... - 'did2.schema.cache.queryablePaths is not yet implemented.'); + function chain = classChain(obj, className) + % classChain - root-first list of class names including the + % class itself. For 'demoB' -> {'base', 'demoA', 'demoB'}. + arguments + obj + className (1,:) char + end + chain = [fliplr(obj.superclasses(className)), {className}]; end - function s = buildBlankDocument(~, ~) - % buildBlankDocument - return a struct populated with the - % _blank_value of every field declared by className and - % its superclasses, plus _class metadata. - error('did2:notImplemented', ... - 'did2.schema.cache.buildBlankDocument is not yet implemented.'); + function fields = ownFields(obj, className) + % ownFields - cell array of field defs the class declares + % directly (not inherited). + arguments + obj + className (1,:) char + end + s = obj.getClass(className); + raw = obj.extractField(s, '_fields'); + if isempty(raw) + fields = {}; + else + fields = obj.toCellArray(raw); + end end - function validateDocument(~, ~) - % validateDocument - raise if the supplied did2.document - % does not conform to its V_gamma class definition. + function tagged = fieldsFor(obj, className) + % fieldsFor - merged inherited fields tagged with the + % declaring class. Returns a struct array with fields + % `declaringClass` (char) and `fieldDef` (the schema's + % _fields entry). + arguments + obj + className (1,:) char + end + tagged = struct('declaringClass', {}, 'fieldDef', {}); + chain = obj.classChain(className); + for k = 1:numel(chain) + own = obj.ownFields(chain{k}); + for f = 1:numel(own) + tagged(end+1) = struct( ... + 'declaringClass', chain{k}, ... + 'fieldDef', own{f}); %#ok + end + end + end + + function paths = queryablePaths(~, ~) %#ok + % queryablePaths - planned for steps 3 & 4. Will return + % .scalar (cellstr of class-qualified dot-paths like + % 'daqsystem.sample_rate.hertz') and .array (cellstr of + % '[*]'-suffixed paths). Used by the SQL backend to drive + % generated columns (§3.2) and the queryable_array_elem + % sidecar (§3.3). error('did2:notImplemented', ... - 'did2.schema.cache.validateDocument is not yet implemented.'); + 'did2.schema.cache.queryablePaths is not yet implemented (step 3/4).'); + end + + function doc = buildBlankDocument(obj, className) + % buildBlankDocument - blank V_gamma document in the + % class-scoped wire shape. Mints a fresh did_uid for + % base.id and the current UTC timestamp for base.datestamp. + arguments + obj + className (1,:) char + end + doc = struct(); + schema = obj.getClass(className); + doc.x_classname = char(obj.extractField(schema, '_classname')); + doc.x_class_version = char(obj.extractField(schema, '_class_version')); + + ancestors = obj.superclasses(className); + sc = struct('x_classname', {}, 'x_class_version', {}); + for k = 1:numel(ancestors) + ancSchema = obj.getClass(ancestors{k}); + sc(end+1) = struct( ... + 'x_classname', char(obj.extractField(ancSchema, '_classname')), ... + 'x_class_version', char(obj.extractField(ancSchema, '_class_version'))); %#ok + end + doc.x_superclasses = sc; + doc.x_depends_on = struct('x_name', {}, 'value', {}); + + chain = obj.classChain(className); + for k = 1:numel(chain) + blockClass = chain{k}; + block = obj.buildBlockForClass(blockClass); + doc.(blockClass) = block; + end + end + + function validateDocument(obj, docOrStruct) + % validateDocument - raise did2:validation:* on a + % non-conforming document. Accepts a did2.document or a + % plain struct. + arguments + obj + docOrStruct + end + if isa(docOrStruct, 'did2.document') + s = docOrStruct.toStruct(); + elseif isstruct(docOrStruct) + s = docOrStruct; + else + error('did2:validation:badInput', ... + 'validateDocument expects a did2.document or a struct, got %s.', ... + class(docOrStruct)); + end + if ~isfield(s, 'x_classname') || isempty(s.x_classname) + error('did2:validation:missingClassName', ... + 'Document has no _classname; cannot validate.'); + end + className = char(s.x_classname); + chain = obj.classChain(className); + for k = 1:numel(chain) + blockClass = chain{k}; + if ~isfield(s, blockClass) + error('did2:validation:missingClassBlock', ... + 'Document is missing the "%s" property block.', blockClass); + end + block = s.(blockClass); + if ~isstruct(block) + error('did2:validation:badClassBlock', ... + 'Property block "%s" must be a struct, got %s.', ... + blockClass, class(block)); + end + own = obj.ownFields(blockClass); + for f = 1:numel(own) + fieldDef = own{f}; + fieldName = char(obj.extractField(fieldDef, '_name')); + obj.validateField(block, fieldDef, blockClass, fieldName); + end + end end end methods (Static) function obj = shared(varargin) % shared - return the process-wide cache singleton. - % - % shared() returns the existing singleton, creating it on - % first call using the default schema path. - % shared(schemaPath) sets the schema path on first call; if - % the singleton already exists, the argument is ignored - % (use setSchemaPath to rebuild). - % shared('-reset') drops the singleton. persistent instance if nargin == 1 && ischar(varargin{1}) && strcmp(varargin{1}, '-reset') instance = []; @@ -167,7 +271,7 @@ function validateDocument(~, ~) end function setSchemaPath(schemaPath) - % setSchemaPath - rebuild the singleton against a new schema path. + % setSchemaPath - rebuild the singleton at a new schema path. arguments schemaPath (1,:) char end @@ -176,21 +280,13 @@ function setSchemaPath(schemaPath) end function resetSingleton() - % resetSingleton - drop the cached singleton so the next - % .shared() call constructs a fresh instance. Intended for - % tests. + % resetSingleton - drop the cached singleton. did2.schema.cache.shared('-reset'); end end methods (Static, Access = private) function p = defaultSchemaPath() - % defaultSchemaPath - filesystem location of V_gamma schemas. - % - % The default looks for a sibling did-schema checkout next - % to the did-matlab repository. Override via the - % DID_SCHEMA_PATH environment variable or by calling - % setSchemaPath(). envOverride = getenv('DID_SCHEMA_PATH'); if ~isempty(envOverride) p = envOverride; @@ -199,6 +295,25 @@ function resetSingleton() toolboxDir = did.toolboxdir(); p = fullfile(toolboxDir, '..', '..', 'did-schema', 'schemas', 'V_gamma'); end + + function ts = currentUTCTimestamp() + dt = datetime('now', 'TimeZone', 'UTC'); + dt.Format = 'yyyy-MM-dd''T''HH:mm:ss.SSS''Z'''; + ts = char(string(dt)); + end + + function len = stringLength(value) + if isstring(value) + len = strlength(value); + if numel(len) > 1 + len = max(len); + end + elseif ischar(value) + len = numel(value); + else + len = 0; + end + end end methods (Access = private) @@ -210,9 +325,13 @@ function loadRegistry(obj) end function value = extractField(~, s, name) - % extractField - tolerate jsondecode's underscore-prefix - % renaming, which on some MATLAB releases mangles leading - % underscores into 'x_' or strips them. + % extractField - tolerate jsondecode's leading-underscore + % rewrites (`_` becomes `x_`), with backwards-compat + % probes for older quirks. + if ~isstruct(s) && ~isobject(s) + value = []; + return; + end candidates = {name, ['x' name], strrep(name, '_', '')}; for k = 1:numel(candidates) if isfield(s, candidates{k}) @@ -222,5 +341,217 @@ function loadRegistry(obj) end value = []; end + + function out = toCellArray(~, raw) + if iscell(raw) + out = raw(:)'; + elseif isstruct(raw) + out = arrayfun(@(i) raw(i), 1:numel(raw), 'UniformOutput', false); + else + out = {raw}; + end + end + + function elem = elementAt(obj, raw, idx) + cells = obj.toCellArray(raw); + elem = cells{idx}; + end + + function block = buildBlockForClass(obj, className) + % buildBlockForClass - one property block populated with + % _blank_value for every field the class declares + % directly. Base block also receives a fresh did_uid for + % `id` and the current UTC timestamp for `datestamp`. + block = struct(); + own = obj.ownFields(className); + for f = 1:numel(own) + fieldDef = own{f}; + fieldName = char(obj.extractField(fieldDef, '_name')); + blank = obj.extractField(fieldDef, '_blank_value'); + fieldType = char(obj.extractField(fieldDef, 'type')); + if strcmp(fieldType, 'structure') ... + && (isempty(blank) || (isstruct(blank) && isempty(fieldnames(blank)))) + block.(fieldName) = obj.buildBlankStructure(fieldDef); + else + block.(fieldName) = blank; + end + end + if strcmp(className, 'base') + if isfield(block, 'id') + block.id = did.ido.unique_id(); + end + if isfield(block, 'datestamp') + block.datestamp = did2.schema.cache.currentUTCTimestamp(); + end + end + end + + function s = buildBlankStructure(obj, fieldDef) + nested = obj.extractField(fieldDef, '_fields'); + s = struct(); + if isempty(nested) + return; + end + entries = obj.toCellArray(nested); + for k = 1:numel(entries) + subDef = entries{k}; + subName = char(obj.extractField(subDef, '_name')); + subBlank = obj.extractField(subDef, '_blank_value'); + subType = char(obj.extractField(subDef, 'type')); + if strcmp(subType, 'structure') ... + && (isempty(subBlank) || (isstruct(subBlank) && isempty(fieldnames(subBlank)))) + s.(subName) = obj.buildBlankStructure(subDef); + else + s.(subName) = subBlank; + end + end + end + + function validateField(obj, block, fieldDef, blockClass, fieldName) + % validateField - apply type, _mustBe* flags, and + % _constraints for one field against the property block. + % Skips absent fields unless the schema marks them + % _mustBeNonEmpty. + if ~isfield(block, fieldName) + if obj.extractField(fieldDef, '_mustBeNonEmpty') + error('did2:validation:missingField', ... + 'Required field "%s.%s" is missing.', ... + blockClass, fieldName); + end + return; + end + value = block.(fieldName); + fieldType = char(obj.extractField(fieldDef, 'type')); + qualifiedName = sprintf('%s.%s', blockClass, fieldName); + obj.validateTypeShape(value, fieldType, qualifiedName); + + mustBeNonEmpty = logical(obj.extractField(fieldDef, '_mustBeNonEmpty')); + mustBeScalar = logical(obj.extractField(fieldDef, '_mustBeScalar')); + mustNotHaveNaN = logical(obj.extractField(fieldDef, '_mustNotHaveNaN')); + if mustBeNonEmpty && obj.isEmptyValue(value) + error('did2:validation:emptyField', ... + 'Field "%s" is required to be non-empty.', qualifiedName); + end + if mustBeScalar && ~obj.isScalarValue(value, fieldType) + error('did2:validation:notScalar', ... + 'Field "%s" is required to be scalar.', qualifiedName); + end + if mustNotHaveNaN && isnumeric(value) && any(isnan(value(:))) + error('did2:validation:nanValue', ... + 'Field "%s" contains NaN.', qualifiedName); + end + constraints = obj.extractField(fieldDef, '_constraints'); + if isstruct(constraints) && ~isempty(fieldnames(constraints)) + obj.validateConstraints(value, constraints, fieldType, qualifiedName); + end + end + + function tf = isEmptyValue(~, value) + if isstring(value) + tf = all(strlength(value) == 0); + elseif ischar(value) + tf = isempty(value); + elseif isstruct(value) + tf = isempty(value) || isempty(fieldnames(value)); + else + tf = isempty(value); + end + end + + function tf = isScalarValue(~, value, fieldType) + switch fieldType + case {'char', 'string', 'did_uid', 'timestamp'} + tf = (ischar(value) && (isempty(value) || size(value,1) <= 1)) ... + || (isstring(value) && isscalar(value)); + otherwise + tf = isscalar(value); + end + end + + function validateTypeShape(~, value, fieldType, qualifiedName) + switch fieldType + case {'char', 'did_uid', 'timestamp'} + if ~(ischar(value) || (isstring(value) && isscalar(value))) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be char/string (type %s).', qualifiedName, fieldType); + end + case 'string' + if ~(ischar(value) || isstring(value)) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be string.', qualifiedName); + end + case 'boolean' + if ~(islogical(value) || (isnumeric(value) && all(value(:) == 0 | value(:) == 1))) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be boolean.', qualifiedName); + end + case 'integer' + if ~isnumeric(value) || any(mod(value(:), 1) ~= 0) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be integer.', qualifiedName); + end + case {'double', 'matrix'} + if ~isnumeric(value) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be numeric.', qualifiedName); + end + case 'structure' + if ~isstruct(value) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be a struct.', qualifiedName); + end + case {'duration','volume','mass','length','voltage','current','frequency','ontology_term'} + if ~isstruct(value) + error('did2:validation:typeMismatch', ... + 'Field "%s" must be a struct (named composite type %s).', ... + qualifiedName, fieldType); + end + otherwise + % Unknown type - tolerated; the meta-schema's enum gates + % new types, so an unknown means tooling drift. + end + end + + function validateConstraints(~, value, constraints, ~, qualifiedName) + cnames = fieldnames(constraints); + for k = 1:numel(cnames) + cname = cnames{k}; + cval = constraints.(cname); + switch cname + case 'maxLength' + len = did2.schema.cache.stringLength(value); + if len > cval + error('did2:validation:maxLength', ... + 'Field "%s" exceeds maxLength %d (got %d).', qualifiedName, cval, len); + end + case 'minLength' + len = did2.schema.cache.stringLength(value); + if len < cval + error('did2:validation:minLength', ... + 'Field "%s" below minLength %d (got %d).', qualifiedName, cval, len); + end + case 'minimum' + if isnumeric(value) && any(value(:) < cval) + error('did2:validation:minimum', ... + 'Field "%s" below minimum %g.', qualifiedName, cval); + end + case 'maximum' + if isnumeric(value) && any(value(:) > cval) + error('did2:validation:maximum', ... + 'Field "%s" above maximum %g.', qualifiedName, cval); + end + case 'enum' + choices = string(cval); + v = string(value); + if ~any(strcmp(v, choices)) + error('did2:validation:enum', ... + 'Field "%s" value "%s" not in enum.', qualifiedName, v); + end + otherwise + % Unrecognised constraint keys are tolerated; + % `pattern` and similar can be added later. + end + end + end end end From 7bd091242f916f0ce2e7afe2b1e9564dcac35796 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 20:52:09 -0400 Subject: [PATCH 16/32] =?UTF-8?q?v2:=20did2.document=20=E2=80=94=20V=5Fgam?= =?UTF-8?q?ma=20class-scoped=20shape=20and=20toJSON=20rewrite?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aligns did2.document with V_gamma's class-scoped property-block document layout (V_gamma_SPEC.md "JSON Format: Document Instances"). - className() / classVersion() now read from x_classname / x_class_version (MATLAB-legal renames of the leading-underscore JSON keys; mirrors what jsondecode produces). - toJSON() post-processes the jsonencode output with a regex that rewrites `"x_":` keys back to `"_":` so the wire form matches the spec. fromJSON relies on jsondecode's default behaviour to read it back. - Class metadata accessors no longer assume a `_class.name` nested path; they read flat top-level fields. - Header doc updated to describe the V_gamma layout, the x_ convention, and the round-trip path. dot-path get/set, iterate, fromJSON/fromStruct/blank, and the (className, valueStruct) constructor are unchanged in shape — they operate on whatever struct the document carries. --- src/did/+did2/document.m | 116 +++++++++++++++++---------------------- 1 file changed, 50 insertions(+), 66 deletions(-) diff --git a/src/did/+did2/document.m b/src/did/+did2/document.m index 0887571..599ac29 100644 --- a/src/did/+did2/document.m +++ b/src/did/+did2/document.m @@ -1,16 +1,20 @@ classdef document < handle - % did2.document V_gamma document object (DID v2 scaffold). + % did2.document V_gamma document object. % - % A did2.document holds a single DID document in its V_gamma JSON - % shape, validates it against the V_gamma schema set, and serialises - % it back to JSON. Unlike did.document, the internal representation - % is the flat V_gamma shape directly — no translation to/from the - % V_alpha base.* / document_class.* / nesting. + % Holds a single V_gamma document in the class-scoped wire shape + % (see V_gamma_SPEC.md "JSON Format: Document Instances"), validates + % it against the V_gamma schema set, and serialises it back to JSON. % - % This is the v2 development scaffold (PLAN.md §9, item 1). The - % outer API is intended to stabilise here; internal methods are - % filled in iteratively. Stubs throw 'did2:notImplemented' so that - % missing pieces surface loudly rather than silently no-op. + % In-memory representation. MATLAB struct field names cannot start + % with an underscore, so the four leading-underscore top-level + % keys (`_classname`, `_class_version`, `_superclasses`, + % `_depends_on`) are stored as `x_classname`, `x_class_version`, + % `x_superclasses`, `x_depends_on`, mirroring what `jsondecode` + % produces. Class-block keys (`base`, `daqsystem`, ...) are valid + % MATLAB identifiers and stay verbatim. `toJSON` rewrites + % `"x_":` back to `"_":` on the encoded output so the + % serialised form matches the spec; `fromJSON` relies on + % `jsondecode`'s default rename to read it back in. % % did2.document Properties: % documentProperties - struct mirroring the V_gamma JSON shape. @@ -20,12 +24,11 @@ % (className, valueStruct). % get - dot-path getter into documentProperties. % set - dot-path setter into documentProperties. - % iterate - element iterator over an array-of-structure path - % (used by the in-memory query evaluator for [*]). - % toJSON - serialise to a JSON string. + % iterate - element iterator over an array-of-structure path. + % toJSON - serialise to V_gamma JSON text. % toStruct - return the underlying struct. - % className - shorthand for get('_class.name'). - % classVersion - shorthand for get('_class.version'). + % className - shorthand for the document's `_classname`. + % classVersion - shorthand for the document's `_class_version`. % validate - validate this document against its schema. % % did2.document Static Methods: @@ -33,18 +36,13 @@ % fromStruct - construct from a struct. % blank - construct a blank instance of the named class. % - % See also: did2.schema.cache, did.document, docs/v2/PLAN.md. + % See also: did2.schema.cache, docs/v2/PLAN.md. properties - % documentProperties - struct mirroring the V_gamma JSON shape. - % Top-level keys are flat snake_case (e.g., id, session_id, - % name, datestamp) plus class-defined fields. System metadata - % carries a leading underscore (_class, _depends_on, _files). documentProperties (1,1) struct = struct() end properties (Access = private) - % schemaCacheHandle - lazily resolved did2.schema.cache instance. schemaCacheHandle = [] end @@ -81,10 +79,9 @@ function value = get(obj, fieldPath) % get - read documentProperties at a dot-path. % - % v = doc.get('sample_rate.hertz') returns the hertz field - % inside the sample_rate named composite. The [*] array - % iteration suffix is not handled here — use iterate() for - % that, since [*] returns a struct array rather than a scalar. + % v = doc.get('base.id') returns the id from the base + % property block. `[*]` array iteration is handled by + % `iterate(arrayPath)`, not by this method. arguments obj fieldPath (1,:) char @@ -94,9 +91,6 @@ function obj = set(obj, fieldPath, value) % set - write a value at a dot-path inside documentProperties. - % - % doc.set('app.app_name', 'ndi_app_spikeextractor') sets the - % nested field, creating intermediate structs as needed. arguments obj fieldPath (1,:) char @@ -108,11 +102,6 @@ function elements = iterate(obj, arrayPath) % iterate - return the element list at an array-of-structure path. - % - % els = doc.iterate('axes') returns the struct array stored - % at the 'axes' path. Used by the in-memory query evaluator - % to implement the V_gamma '[*]' existential semantics - % described in did_query_model.md. arguments obj arrayPath (1,:) char @@ -127,40 +116,44 @@ end function jsonText = toJSON(obj, opts) - % toJSON - serialise documentProperties to JSON text. + % toJSON - serialise documentProperties to V_gamma JSON text. + % Internal `x_` keys are rewritten to `_` on + % the encoded output to match the spec. arguments obj opts.PrettyPrint (1,1) logical = false end - jsonText = jsonencode(obj.documentProperties, ... + raw = jsonencode(obj.documentProperties, ... 'PrettyPrint', opts.PrettyPrint); + jsonText = did2.document.rewriteXUnderscoreKeys(raw); end function s = toStruct(obj) - % toStruct - return the underlying documentProperties struct. s = obj.documentProperties; end function name = className(obj) - % className - shorthand for get('_class.name'). - name = obj.get('_class.name'); + % className - the document's `_classname` value. + if isfield(obj.documentProperties, 'x_classname') + name = char(obj.documentProperties.x_classname); + else + error('did2:document:missingField', ... + 'Document has no _classname.'); + end end function v = classVersion(obj) - % classVersion - shorthand for get('_class.version'). - v = obj.get('_class.version'); + % classVersion - the document's `_class_version` value. + if isfield(obj.documentProperties, 'x_class_version') + v = char(obj.documentProperties.x_class_version); + else + error('did2:document:missingField', ... + 'Document has no _class_version.'); + end end function validate(obj, opts) % validate - check this document against its V_gamma schema. - % - % doc.validate() resolves the schema cache from the default - % path, looks up the document's class definition, and - % verifies required fields, type constraints, and the named - % composite layouts (ontology_term, duration, length, ...). - % - % doc.validate(SchemaCache=cache) uses the supplied cache - % instead of the shared singleton. arguments obj opts.SchemaCache = [] @@ -172,7 +165,6 @@ function validate(obj, opts) methods (Static) function obj = fromJSON(jsonText) - % fromJSON - construct a did2.document from a JSON string. arguments jsonText (1,:) char end @@ -180,7 +172,6 @@ function validate(obj, opts) end function obj = fromStruct(s) - % fromStruct - construct a did2.document from a struct. arguments s (1,1) struct end @@ -189,12 +180,6 @@ function validate(obj, opts) function obj = blank(className, opts) % blank - construct a blank V_gamma document of the named class. - % - % d = did2.document.blank('app') builds an instance of the - % 'app' class with every field set to its '_blank_value' as - % declared by the V_gamma schema. _class metadata is filled - % from the schema; id and datestamp are populated with a - % freshly generated did_uid and the current UTC timestamp. arguments className (1,:) char opts.SchemaCache = [] @@ -204,7 +189,6 @@ function validate(obj, opts) end function value = dotPathGet(s, fieldPath) - % dotPathGet - read a nested value out of struct s by dot-path. arguments s fieldPath (1,:) char @@ -228,7 +212,6 @@ function validate(obj, opts) end function s = dotPathSet(s, fieldPath, value) - % dotPathSet - write value into struct s at the given dot-path. arguments s (1,1) struct fieldPath (1,:) char @@ -264,9 +247,6 @@ function validate(obj, opts) end function s = mergeStruct(base, overlay) - % mergeStruct - shallow overlay of overlay onto base. - % Scalar struct fields in overlay overwrite base; nested - % structs recurse. Non-struct values overwrite. s = base; if ~isstruct(overlay) return; @@ -283,11 +263,6 @@ function validate(obj, opts) end function s = buildBlank(className, cacheOverride) - % buildBlank - assemble a blank V_gamma document by walking - % the schema cache for className and its superclasses, - % populating each field with its _blank_value, then filling - % _class metadata, id (freshly minted), session_id (blank), - % and datestamp (current UTC). if nargin < 2 cacheOverride = []; end @@ -298,6 +273,15 @@ function validate(obj, opts) end s = cache.buildBlankDocument(className); end + + function out = rewriteXUnderscoreKeys(jsonText) + % rewriteXUnderscoreKeys - convert `"x_":` keys to + % `"_":` on the encoded JSON text. The regex matches + % only JSON keys (colon-terminated, with optional + % whitespace) so values that happen to start with `x_` + % are unaffected. + out = regexprep(jsonText, '"x_([a-zA-Z][a-zA-Z0-9_]*)"(\s*):', '"_$1"$2:'); + end end methods (Access = private) From bf275a7b03b4f842b73f99cc1bd01486764cabd4 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 20:52:47 -0400 Subject: [PATCH 17/32] v2: add testSchemaCache covering V_gamma class-scoped shape Function-based unit tests for did2.schema.cache (plus end-to-end through did2.document) against the in-repo fixture set tests/+did2/fixtures/V_gamma/. Covers: - schema-path plumbing, getClass, missing-class error, CURIE registry presence - superclass chains: base (root), demoA (one parent), demoB (two-level) - classChain root-first ordering - ownFields counts (base=4, demoA=1) - fieldsFor declaring-class tagging across the chain - buildBlankDocument top-level metadata (x_classname, x_class_version, x_superclasses ancestor entries, empty x_depends_on) - buildBlankDocument class-scoped blocks: base, the concrete class, all chain blocks present; demoB has both demoA.value and demoB.value_b at distinct paths - buildBlankDocument base block: minted id (length 33), current-UTC datestamp starting with "20" and ending in "Z" - validateDocument: empty session_id throws emptyField, passes after filling; max-length boundary (300 fails, 256 passes); typeMismatch on numeric-where-char; missingClassName on structs without x_classname; missingClassBlock when a chain block is removed - End-to-end through did2.document: blank() / className() / classVersion() / get('base.id') / validate() round-trip - toJSON rewrite: serialised form contains "_classname", not "x_classname"; fromJSON re-parse preserves the class block. setupOnce points the cache at the fixture via setSchemaPath; teardownOnce resets the singleton. --- tests/+did2/testSchemaCache.m | 225 ++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 tests/+did2/testSchemaCache.m diff --git a/tests/+did2/testSchemaCache.m b/tests/+did2/testSchemaCache.m new file mode 100644 index 0000000..1f63231 --- /dev/null +++ b/tests/+did2/testSchemaCache.m @@ -0,0 +1,225 @@ +function tests = testSchemaCache +% testSchemaCache - exercises did2.schema.cache against the in-repo +% V_gamma fixtures at tests/+did2/fixtures/V_gamma/. Also covers +% did2.document.blank() and did2.document.validate() end-to-end. +% +% Documents in V_gamma use class-scoped property blocks (see +% V_gamma_SPEC.md "JSON Format: Document Instances"): top-level +% `_classname`, `_class_version`, `_superclasses`, `_depends_on`, plus +% one property block per class in the chain. MATLAB stores the four +% underscore-prefixed system keys as `x_` (mirroring jsondecode); +% class-block keys (`base`, `demoA`, ...) are plain MATLAB identifiers. + +tests = functiontests(localfunctions); +end + +function setupOnce(testCase) +fixtureDir = fullfile(fileparts(mfilename('fullpath')), 'fixtures', 'V_gamma'); +did2.schema.cache.setSchemaPath(fixtureDir); +testCase.TestData.fixtureDir = fixtureDir; +testCase.TestData.cache = did2.schema.cache.shared(); +end + +function teardownOnce(~) +did2.schema.cache.resetSingleton(); +end + +% ---- schema-cache plumbing ---- + +function testSchemaPathPointsAtFixtures(testCase) +verifyTrue(testCase, isfolder(testCase.TestData.fixtureDir)); +verifyEqual(testCase, testCase.TestData.cache.schemaPath, testCase.TestData.fixtureDir); +end + +function testGetClassLoadsBase(testCase) +s = testCase.TestData.cache.getClass('base'); +verifyTrue(testCase, isstruct(s)); +end + +function testGetClassMissingThrows(testCase) +verifyError(testCase, ... + @() testCase.TestData.cache.getClass('not_a_real_class'), ... + 'did2:schema:missingClass'); +end + +function testCurieRegistryLoaded(testCase) +verifyTrue(testCase, isstruct(testCase.TestData.cache.curieRegistry)); +verifyFalse(testCase, isempty(fieldnames(testCase.TestData.cache.curieRegistry))); +end + +% ---- superclass chains ---- + +function testSuperclassesBaseIsRoot(testCase) +verifyEmpty(testCase, testCase.TestData.cache.superclasses('base')); +end + +function testSuperclassesDemoAExtendsBase(testCase) +verifyEqual(testCase, testCase.TestData.cache.superclasses('demoA'), {'base'}); +end + +function testSuperclassesDemoBChain(testCase) +verifyEqual(testCase, testCase.TestData.cache.superclasses('demoB'), {'demoA', 'base'}); +end + +function testClassChainRootFirst(testCase) +verifyEqual(testCase, testCase.TestData.cache.classChain('demoB'), ... + {'base', 'demoA', 'demoB'}); +end + +% ---- field-list resolution ---- + +function testOwnFieldsBaseHasFour(testCase) +own = testCase.TestData.cache.ownFields('base'); +verifyEqual(testCase, numel(own), 4); +end + +function testOwnFieldsDemoAHasOne(testCase) +own = testCase.TestData.cache.ownFields('demoA'); +verifyEqual(testCase, numel(own), 1); +end + +function testFieldsForTagsDeclaringClass(testCase) +tagged = testCase.TestData.cache.fieldsFor('demoB'); +% base(4) + demoA(1) + demoB(1) = 6 entries +verifyEqual(testCase, numel(tagged), 6); +verifyEqual(testCase, tagged(1).declaringClass, 'base'); +verifyEqual(testCase, tagged(4).declaringClass, 'base'); +verifyEqual(testCase, tagged(5).declaringClass, 'demoA'); +verifyEqual(testCase, tagged(6).declaringClass, 'demoB'); +end + +% ---- buildBlankDocument: top-level metadata ---- + +function testBuildBlankDocumentTopLevelMetadata(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoB'); +verifyEqual(testCase, doc.x_classname, 'demoB'); +verifyEqual(testCase, doc.x_class_version, '1.0.0'); +verifyEqual(testCase, numel(doc.x_superclasses), 2); +verifyEqual(testCase, doc.x_superclasses(1).x_classname, 'demoA'); +verifyEqual(testCase, doc.x_superclasses(2).x_classname, 'base'); +end + +function testBuildBlankDocumentEmptyDependsOn(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'x_depends_on')); +verifyEmpty(testCase, doc.x_depends_on); +end + +% ---- buildBlankDocument: class-scoped blocks ---- + +function testBuildBlankDocumentHasBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'base')); +verifyTrue(testCase, isfield(doc.base, 'id')); +verifyTrue(testCase, isfield(doc.base, 'session_id')); +verifyTrue(testCase, isfield(doc.base, 'name')); +verifyTrue(testCase, isfield(doc.base, 'datestamp')); +end + +function testBuildBlankDocumentHasConcreteBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'demoA')); +verifyTrue(testCase, isfield(doc.demoA, 'value')); +verifyEqual(testCase, doc.demoA.value, ''); +end + +function testBuildBlankDocumentAllChainBlocksPresent(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoB'); +verifyTrue(testCase, isfield(doc, 'base')); +verifyTrue(testCase, isfield(doc, 'demoA')); +verifyTrue(testCase, isfield(doc, 'demoB')); +verifyTrue(testCase, isfield(doc.demoA, 'value')); +verifyTrue(testCase, isfield(doc.demoB, 'value_b')); +end + +function testBuildBlankDocumentMintsIdInBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyEqual(testCase, numel(doc.base.id), 33); % did_id format length +end + +function testBuildBlankDocumentSetsDatestampInBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyEqual(testCase, doc.base.datestamp(1:2), '20'); +verifyEqual(testCase, doc.base.datestamp(end), 'Z'); +end + +% ---- validateDocument ---- + +function testValidateBlankDocFailsOnEmptySessionId(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:emptyField'); +end + +function testValidatePassesAfterFillingSessionId(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +testCase.TestData.cache.validateDocument(doc); +end + +function testValidateCatchesMaxLength(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = repmat('a', 1, 300); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:maxLength'); +end + +function testValidateAcceptsValueAtMaxLength(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = repmat('a', 1, 256); +testCase.TestData.cache.validateDocument(doc); +end + +function testValidateCatchesTypeMismatch(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = 12345; +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:typeMismatch'); +end + +function testValidateMissingClassNameThrows(testCase) +doc = struct('base', struct('id', 'abc')); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:missingClassName'); +end + +function testValidateMissingClassBlockThrows(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc = rmfield(doc, 'base'); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:missingClassBlock'); +end + +% ---- end-to-end through did2.document ---- + +function testDocumentBlankConvenience(testCase) +doc = did2.document.blank('demoA'); +verifyEqual(testCase, doc.className(), 'demoA'); +verifyEqual(testCase, doc.classVersion(), '1.0.0'); +verifyEqual(testCase, numel(doc.get('base.id')), 33); +end + +function testDocumentValidateRoundTrip(testCase) +doc = did2.document.blank('demoA'); +doc.set('base.session_id', did.ido.unique_id()); +doc.set('demoA.value', 'hello'); +doc.validate(); +end + +function testDocumentToJSONRewritesUnderscoreKeys(testCase) +doc = did2.document.blank('demoA'); +text = doc.toJSON(); +verifyTrue(testCase, contains(text, '"_classname":"demoA"')); +verifyFalse(testCase, contains(text, '"x_classname"')); +doc2 = did2.document.fromJSON(text); +verifyEqual(testCase, doc2.className(), 'demoA'); +verifyTrue(testCase, isfield(doc2.toStruct(), 'base')); +end From cbc31c3879af46781e7c6b2d3861b917ffe02132 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 20:54:17 -0400 Subject: [PATCH 18/32] =?UTF-8?q?v2:=20PLAN.md=20=E2=80=94=20record=20V=5F?= =?UTF-8?q?gamma=20class-scoped=20shape=20+=20close=20step=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updates the living plan to the V_gamma "JSON Format: Document Instances" wire shape (upstream did-schema commit 137f583, which restored class-scoped property blocks) and logs the step-1 close. Changes: - Decision #8 in §1 is rewritten to describe class-scoped property blocks keyed by `_classname` verbatim (instead of the earlier "flat MATLAB struct" placeholder). Notes that MATLAB cannot have leading-underscore struct fields, hence the x_ internal convention and toJSON regex rewrite. - §4.1 "In-memory document shape" is replaced. Documents the V_gamma class-scoped layout, the universal top-level keys (`_classname`/`x_classname`, etc.), per-class property blocks including empty `{}` for zero-field classes, and the `(declaring_class, _name)` field-identity rule. Compares V_alpha → V_gamma at the document level for converter (§7) reference. - §12 progress log gains a 2026-05-12 entry covering: the SPEC update we drove upstream; the cache methods now implemented (classChain, ownFields, fieldsFor, buildBlankDocument, validateDocument); the document.m changes (className / classVersion read x_, toJSON post-processes the encoded text); the hermetic V_gamma fixture set under tests/+did2/fixtures/V_gamma/; and the new testSchemaCache.m coverage. Step 1 is recorded as complete; queryablePaths and detailed named-composite/`_depends_on`-value validation are noted as deferred to follow-up. --- docs/v2/PLAN.md | 138 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) diff --git a/docs/v2/PLAN.md b/docs/v2/PLAN.md index 32f27e6..3b079e6 100644 --- a/docs/v2/PLAN.md +++ b/docs/v2/PLAN.md @@ -20,6 +20,7 @@ users. | 5 | Validate on insert by default; expose an `unsafe_insert` escape hatch for bulk loads; offer a `revalidate_all` maintenance op. | Schema files are the source of truth for what "valid" means. | | 6 | Plan lives at `docs/v2/PLAN.md` on the v2 development branch. | This file. | | 7 | Provisional namespace: `+did2`. | Picked from §10 option A for the scaffold. Revisit before v2 reaches `main`. | +| 8 | Document instances use class-scoped property blocks (one block per class in the chain, keyed by `_classname` verbatim). | See §4.1. Matches V_gamma_SPEC.md "JSON Format: Document Instances" after the SPEC update that restored V_alpha-style class scoping. MATLAB struct field names can't begin with `_`, so the four top-level system keys are stored as `x_*` internally and rewritten to `_*` on serialise. | Open questions are in §10. @@ -155,6 +156,65 @@ Validation timing: explicit, deferred. The database layer calls bulk loads. A `revalidate_all(db)` maintenance op exists for the case where schemas change. +### 4.1 In-memory document shape + +A `did2.document`'s `documentProperties` is a MATLAB struct that mirrors the +V_gamma JSON shape *as specified in V_gamma_SPEC.md, "JSON Format: Document +Instances"*. V_gamma documents use **class-scoped property blocks**: there +is one top-level block per class in the inheritance chain, keyed by that +class's `_classname` verbatim. Fields live in the block of the class that +declared them. There is no `property_list_name` knob — the block key +*is* the class name. + +The MATLAB representation has one quirk: struct field names cannot begin +with `_`. The four underscore-prefixed top-level system keys (`_classname`, +`_class_version`, `_superclasses`, `_depends_on`) are stored as +`x_classname`, `x_class_version`, `x_superclasses`, `x_depends_on` — +mirroring exactly what `jsondecode` produces. Class-block keys (`base`, +`daqsystem`, ...) are valid MATLAB identifiers (V_gamma classnames match +`^[a-z][a-z0-9_]*$`) and stay verbatim. `did2.document.toJSON` rewrites +`"x_":` keys back to `"_":` on the encoded output so the wire +form matches the spec; `fromJSON` relies on `jsondecode`'s default rename +on parse. + +Top-level keys populated by `did2.schema.cache.buildBlankDocument`: + +| Key (JSON / MATLAB) | Type | Source | +|---|---|---| +| `_classname` / `x_classname` | char | concrete class's `_classname` | +| `_class_version` / `x_class_version` | char | concrete class's `_class_version` | +| `_superclasses` / `x_superclasses` | struct array | one elem per ancestor; each has `_classname` (`x_classname`) and `_class_version` (`x_class_version`) | +| `_depends_on` / `x_depends_on` | struct array | each entry has `_name` (`x_name`) and `value`; empty by default | +| `base` / `base` | struct | property block with the four base fields (`id`, `session_id`, `name`, `datestamp`); `id` auto-minted via `did.ido.unique_id`, `datestamp` set to current UTC ISO-8601 | +| `` / `` | struct | one property block per class in the chain, including the concrete class itself; each populated with `_blank_value` for the fields *that class* declares (empty `{}` if it declares none) | + +Field identity is `(declaring_class, _name)`. Same-named fields in +different classes of the chain are distinct paths (`base.id` vs. +`.id`), not an override. The SPEC's "no shadowing, by +construction" rule means validators and query engines work in +class-qualified paths. + +V_alpha → V_gamma at the document level: + +``` +V_alpha V_gamma +------- ------- +document_class.class_name _classname +document_class.class_version _class_version +document_class.superclasses _superclasses +document_class.property_list_name (gone; block key == _classname) +document_class.definition (gone; schema files own this) +document_class.validation (gone; schema files own this) +base.id, base.session_id, ... base.id, base.session_id, ... +. <_classname>. +depends_on (top-level list) _depends_on (renamed only) +``` + +The converter (§7) inverts the mapping when reading V_alpha documents: +strip `document_class`, write the four V_gamma top-level keys, and +rename each property block whose `property_list_name` differs from its +`class_name` so the block key equals the class name. + --- ## 5. Schema cache @@ -334,3 +394,81 @@ Next up: fill in `did2.schema.cache.fieldsFor`, `queryablePaths`, and `buildBlankDocument`; then `validateDocument` against the V_gamma meta-schema; then start the in-memory query evaluator (step 2). + +### 2026-05-12 — close step 1: class-scoped cache implementations + fixtures + +V_gamma_SPEC.md was amended (upstream did-schema commit `137f583`) to +restore class-scoped property blocks on document instances, replacing +the earlier "flatten on inheritance" wire shape. Decision #8 above is +revised accordingly, and §4.1 documents the resulting in-memory layout +(class-block top-level keys plus the four `_` keys stored as +`x_`). + +Filled in the schema cache to the level step 1 needs and wired +end-to-end tests through `did2.document.blank` / `did2.document.validate`. + +Implemented in `src/did/+did2/+schema/cache.m`: + +- `classChain(className)` — root-first list including the class itself + (e.g., `demoB -> {base, demoA, demoB}`). +- `ownFields(className)` — the `_fields` list the class declares + directly (no inheritance). +- `fieldsFor(className)` — merged inherited fields tagged with their + declaring class. Returns a struct array `(declaringClass, fieldDef)` + so callers know which property block each field belongs to. +- `buildBlankDocument(className)` — class-scoped V_gamma document: + top-level `x_classname` / `x_class_version` / `x_superclasses` + (struct array of `{x_classname, x_class_version}` ancestor entries + parent-first) / empty `x_depends_on`, plus one property block per + class in the chain. Base block has `id` auto-minted via + `did.ido.unique_id()` and `datestamp` set to current UTC + millisecond ISO-8601 with trailing `Z`. Empty blocks present for + classes that declare no fields, per the SPEC. +- `validateDocument(docOrStruct)` — accepts a `did2.document` or its + underlying struct, walks the class chain, and for each class + validates the fields *that class declares* against its property + block. Error messages use the qualified `.` form. + Type-shape runs before `_mustBe*` checks so a wrong-type value is + reported as `did2:validation:typeMismatch` rather than + `did2:validation:notScalar`. New error IDs added: + `did2:validation:missingClassBlock`, `did2:validation:badClassBlock`. +- `queryablePaths` stays a stub — it belongs to steps 3 and 4 and + will return class-qualified dot-paths (e.g., + `daqsystem.sample_rate.hertz`) once the storage layer lands. + +In `src/did/+did2/document.m`: + +- `className` / `classVersion` read from `x_classname` / `x_class_version`. +- `toJSON` post-processes the `jsonencode` output with a regex that + rewrites `"x_":` keys back to `"_":`. This is the only + place we serialise V_gamma JSON; `fromJSON` relies on + `jsondecode`'s default behaviour to read it back in. + +Added `tests/+did2/fixtures/V_gamma/`: + +| File | Origin | Why | +|---|---|---| +| `base.json` | upstream `did-schema` V_gamma | The root class. | +| `CURIE_lookups_meta.json` | upstream | Exercises registry load. | +| `demoA.json` | V_gamma translation of v1 `demoA.json` | `base` subclass with one queryable char field. | +| `demoB.json` | V_gamma translation of v1 `demoB.json` | Multi-level (`demoB → demoA → base`). | +| `demoC.json` | V_gamma translation of v1 `demoC.json` | Declares three `_depends_on` entries. | +| `demoFile.json` | V_gamma translation of v1 `demoFile.json` | Declares `_file` attachments. | +| `README.md` | new | Documents origins and refresh procedure. | + +New `tests/+did2/testSchemaCache.m` (function-based) points the cache +at the fixture via `setSchemaPath` in `setupOnce`, then covers: schema-path +plumbing, `getClass` (incl. missing-class error), CURIE registry presence, +superclass-chain depth, `classChain`, `ownFields`, `fieldsFor` declaring-class +tagging, `buildBlankDocument` (top-level metadata, all chain blocks present +as expected, minted id, current UTC datestamp), `validateDocument` +(empty-field, max-length boundary, type mismatch, missing class_name, +missing class block), and an end-to-end `toJSON` round-trip that asserts +the serialised form uses `_classname` and not `x_classname`. + +Step 1 is now complete to the level the rest of the plan needs. +`queryablePaths` is the only intentional stub left in the cache; +detailed per-named-composite validation and `_depends_on` value +checks are deferred to a focused follow-up. Next up: step 2 — the +in-memory query evaluator over the class-scoped paths, which will +also be the executable spec the SQL compiler is later tested against. From e1a2f322b0bdec6beb80c66b74977fa32ad1dcdd Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Mon, 11 May 2026 21:00:24 -0400 Subject: [PATCH 19/32] =?UTF-8?q?v2:=20did2.document=20=E2=80=94=20use=20i?= =?UTF-8?q?sscalar=20in=20assignNested=20per=20Code=20Analyzer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub Advanced Security / Code Analyzer flagged `numel(parts) == 1` in did2.document.assignNested as a length comparison that should be isscalar(parts) for clarity and performance. Cell array isscalar returns true iff the cell has exactly one element, so the substitution is equivalent. --- src/did/+did2/document.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/did/+did2/document.m b/src/did/+did2/document.m index 599ac29..6028367 100644 --- a/src/did/+did2/document.m +++ b/src/did/+did2/document.m @@ -234,7 +234,7 @@ function validate(obj, opts) function s = assignNested(s, parts, value) head = parts{1}; - if numel(parts) == 1 + if isscalar(parts) s.(head) = value; return; end From 4da98f8fe08565b4f4a0bd0410aef1289e871782 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 07:40:59 -0400 Subject: [PATCH 20/32] v2: update fixtures to V_gamma document_class header MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma_SPEC.md restored the V_alpha-style top-level `document_class` header with sub-keys `class_name`, `class_version`, `superclasses` (no leading underscore). Schemas and document instances now both carry that header; only `_depends_on` stays as a top-level underscore-prefixed key. Fixtures updated to match the new shape: - base.json: document_class header with empty superclasses - demoA.json: document_class.superclasses [{class_name: "base"}] - demoB.json: document_class.superclasses [{class_name: "demoA"}] - demoC.json: document_class header + three top-level _depends_on declarations - demoFile.json: document_class header + two top-level _file declarations Inside `_fields`, field definitions are unchanged (still _name, type, _blank_value, _default_value, _mustBe*, _queryable, _ontology, _documentation, _constraints). Inside `document_class.superclasses[i]` in a schema file, the keys are `class_name` + `_schema` (the schema-file path token). In a document instance the same array uses `class_name` + `class_version` instead — that distinction is enforced by the cache and the buildBlankDocument output. --- tests/+did2/fixtures/V_gamma/base.json | 8 +++++--- tests/+did2/fixtures/V_gamma/demoA.json | 18 ++++++++++-------- tests/+did2/fixtures/V_gamma/demoB.json | 18 ++++++++++-------- tests/+did2/fixtures/V_gamma/demoC.json | 18 ++++++++++-------- tests/+did2/fixtures/V_gamma/demoFile.json | 18 ++++++++++-------- 5 files changed, 45 insertions(+), 35 deletions(-) diff --git a/tests/+did2/fixtures/V_gamma/base.json b/tests/+did2/fixtures/V_gamma/base.json index 04c8deb..14f4e8b 100644 --- a/tests/+did2/fixtures/V_gamma/base.json +++ b/tests/+did2/fixtures/V_gamma/base.json @@ -1,8 +1,10 @@ { - "_classname": "base", - "_class_version": "1.0.0", + "document_class": { + "class_name": "base", + "class_version": "1.0.0", + "superclasses": [] + }, "_maturity_level": "work_in_progress", - "_superclasses": [], "_depends_on": [], "_file": [], "_fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoA.json b/tests/+did2/fixtures/V_gamma/demoA.json index 7a2304f..f59f64c 100644 --- a/tests/+did2/fixtures/V_gamma/demoA.json +++ b/tests/+did2/fixtures/V_gamma/demoA.json @@ -1,13 +1,15 @@ { - "_classname": "demoA", - "_class_version": "1.0.0", + "document_class": { + "class_name": "demoA", + "class_version": "1.0.0", + "superclasses": [ + { + "class_name": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ] + }, "_maturity_level": "work_in_progress", - "_superclasses": [ - { - "_classname": "base", - "_schema": "$DIDSCHEMAPATH/base.json" - } - ], "_depends_on": [], "_file": [], "_fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoB.json b/tests/+did2/fixtures/V_gamma/demoB.json index 29e00e7..4cc1819 100644 --- a/tests/+did2/fixtures/V_gamma/demoB.json +++ b/tests/+did2/fixtures/V_gamma/demoB.json @@ -1,13 +1,15 @@ { - "_classname": "demoB", - "_class_version": "1.0.0", + "document_class": { + "class_name": "demoB", + "class_version": "1.0.0", + "superclasses": [ + { + "class_name": "demoA", + "_schema": "$DIDSCHEMAPATH/demoA.json" + } + ] + }, "_maturity_level": "work_in_progress", - "_superclasses": [ - { - "_classname": "demoA", - "_schema": "$DIDSCHEMAPATH/demoA.json" - } - ], "_depends_on": [], "_file": [], "_fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoC.json b/tests/+did2/fixtures/V_gamma/demoC.json index 7007b10..1705561 100644 --- a/tests/+did2/fixtures/V_gamma/demoC.json +++ b/tests/+did2/fixtures/V_gamma/demoC.json @@ -1,13 +1,15 @@ { - "_classname": "demoC", - "_class_version": "1.0.0", + "document_class": { + "class_name": "demoC", + "class_version": "1.0.0", + "superclasses": [ + { + "class_name": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ] + }, "_maturity_level": "work_in_progress", - "_superclasses": [ - { - "_classname": "base", - "_schema": "$DIDSCHEMAPATH/base.json" - } - ], "_depends_on": [ { "_name": "item1", diff --git a/tests/+did2/fixtures/V_gamma/demoFile.json b/tests/+did2/fixtures/V_gamma/demoFile.json index 1f5fe4d..0932800 100644 --- a/tests/+did2/fixtures/V_gamma/demoFile.json +++ b/tests/+did2/fixtures/V_gamma/demoFile.json @@ -1,13 +1,15 @@ { - "_classname": "demoFile", - "_class_version": "1.0.0", + "document_class": { + "class_name": "demoFile", + "class_version": "1.0.0", + "superclasses": [ + { + "class_name": "base", + "_schema": "$DIDSCHEMAPATH/base.json" + } + ] + }, "_maturity_level": "work_in_progress", - "_superclasses": [ - { - "_classname": "base", - "_schema": "$DIDSCHEMAPATH/base.json" - } - ], "_depends_on": [], "_file": [ { From cbcc3dd724f318574c3e473f887d1e5940afefdd Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 07:42:10 -0400 Subject: [PATCH 21/32] =?UTF-8?q?v2:=20cache.m=20=E2=80=94=20track=20V=5Fg?= =?UTF-8?q?amma=20document=5Fclass=20header=20restoration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma_SPEC.md was amended (upstream did-schema commit 94091d8) to re-introduce the V_alpha-style `document_class` header in both schema files and document instances, with sub-keys `class_name`, `class_version`, `superclasses` (no underscore prefixes). `_depends_on` stays at the top level. cache.m changes: - superclasses(className) now walks `document_class.superclasses` (was top-level `_superclasses`). - buildBlankDocument emits the new shape: doc.document_class.class_name / class_version / superclasses doc.x_depends_on (empty) doc. for each class in the chain Each superclass entry is `{class_name, class_version}` (the document-instance form). The `_schema` path token used in schema files is dropped — document instances pin by version, not by path. - validateDocument reads `doc.document_class.class_name` and throws did2:validation:missingClassName if absent. Header comment updated to describe the new wire shape. The internal `x_depends_on` / `x_name` convention for the remaining underscore-prefixed keys is preserved; everything inside `document_class` is plain (no x_ rename needed since none of those keys start with an underscore). --- src/did/+did2/+schema/cache.m | 86 ++++++++++++++++++++++------------- 1 file changed, 54 insertions(+), 32 deletions(-) diff --git a/src/did/+did2/+schema/cache.m b/src/did/+did2/+schema/cache.m index cbe2126..e3ff72b 100644 --- a/src/did/+did2/+schema/cache.m +++ b/src/did/+did2/+schema/cache.m @@ -7,24 +7,24 @@ % docs/v2/PLAN.md §5. % % Document shape (V_gamma "JSON Format: Document Instances"): - % _classname string concrete class - % _class_version string semver of the concrete class - % _superclasses array [{_classname, _class_version}] - % _depends_on array [{_name, value}] - % object one property block per class in - % the chain, keyed by _classname. - % Contains the field values that - % class declared (empty {} if the - % class declares no fields). + % document_class + % .class_name string concrete class + % .class_version string semver of the concrete class + % .superclasses array [{class_name, class_version}] + % _depends_on array [{_name, value}] + % object one property block per class + % in the chain. Contains the + % field values that class + % declared (empty {} if it + % declares no fields). % - % MATLAB representation: leading-underscore JSON keys can't be - % MATLAB struct field names, so we store them with the same `x_` - % prefix MATLAB's `jsondecode` produces — `x_classname`, - % `x_class_version`, `x_superclasses`, `x_depends_on`. The - % class-block keys (`base`, `demoA`, ...) are plain snake_case - % identifiers and stay as written. did2.document.toJSON rewrites - % `"x_":` back to `"_":` on serialisation; jsondecode - % reverses that on parse. + % MATLAB representation: the `document_class` header and its + % sub-keys (`class_name`, `class_version`, `superclasses`) are + % already valid MATLAB struct field names and stay as written. + % Only `_depends_on` (top-level) and `_name` (inside its entries) + % keep MATLAB's `x_` rename — stored as `x_depends_on` and + % `x_name`. did2.document.toJSON rewrites `"x_":` back to + % `"_":` on serialisation. % % did2.schema.cache Properties: % schemaPath - filesystem path to a V_gamma schema dir. @@ -89,7 +89,8 @@ function names = superclasses(obj, className) % superclasses - ancestor chain (parent first, root last). - % For 'demoA' -> {'base'}. For 'base' -> {}. + % Walks `document_class.superclasses[i].class_name` up the + % chain. For 'demoA' -> {'base'}. For 'base' -> {}. arguments obj className (1,:) char @@ -104,12 +105,16 @@ end visited(current) = true; s = obj.getClass(current); - parents = obj.extractField(s, '_superclasses'); + dc = obj.extractField(s, 'document_class'); + if isempty(dc) + break; + end + parents = obj.extractField(dc, 'superclasses'); if isempty(parents) break; end parent = obj.elementAt(parents, 1); - parentName = obj.extractField(parent, '_classname'); + parentName = obj.extractField(parent, 'class_name'); names{end+1} = char(parentName); %#ok current = char(parentName); end @@ -182,21 +187,31 @@ className (1,:) char end doc = struct(); + % document_class header. schema = obj.getClass(className); - doc.x_classname = char(obj.extractField(schema, '_classname')); - doc.x_class_version = char(obj.extractField(schema, '_class_version')); + schemaDC = obj.extractField(schema, 'document_class'); + classNameVal = char(obj.extractField(schemaDC, 'class_name')); + classVersionVal = char(obj.extractField(schemaDC, 'class_version')); ancestors = obj.superclasses(className); - sc = struct('x_classname', {}, 'x_class_version', {}); + sc = struct('class_name', {}, 'class_version', {}); for k = 1:numel(ancestors) ancSchema = obj.getClass(ancestors{k}); + ancDC = obj.extractField(ancSchema, 'document_class'); sc(end+1) = struct( ... - 'x_classname', char(obj.extractField(ancSchema, '_classname')), ... - 'x_class_version', char(obj.extractField(ancSchema, '_class_version'))); %#ok - end - doc.x_superclasses = sc; + 'class_name', char(obj.extractField(ancDC, 'class_name')), ... + 'class_version', char(obj.extractField(ancDC, 'class_version'))); %#ok + end + dc = struct( ... + 'class_name', classNameVal, ... + 'class_version', classVersionVal, ... + 'superclasses', sc); + doc.document_class = dc; + + % Top-level _depends_on (empty struct array of {_name, value}). doc.x_depends_on = struct('x_name', {}, 'value', {}); + % One property block per class in the chain. chain = obj.classChain(className); for k = 1:numel(chain) blockClass = chain{k}; @@ -222,11 +237,16 @@ function validateDocument(obj, docOrStruct) 'validateDocument expects a did2.document or a struct, got %s.', ... class(docOrStruct)); end - if ~isfield(s, 'x_classname') || isempty(s.x_classname) + if ~isfield(s, 'document_class') || ~isstruct(s.document_class) + error('did2:validation:missingClassName', ... + 'Document has no document_class header; cannot validate.'); + end + dc = s.document_class; + if ~isfield(dc, 'class_name') || isempty(dc.class_name) error('did2:validation:missingClassName', ... - 'Document has no _classname; cannot validate.'); + 'Document has no document_class.class_name; cannot validate.'); end - className = char(s.x_classname); + className = char(dc.class_name); chain = obj.classChain(className); for k = 1:numel(chain) blockClass = chain{k}; @@ -326,8 +346,10 @@ function loadRegistry(obj) function value = extractField(~, s, name) % extractField - tolerate jsondecode's leading-underscore - % rewrites (`_` becomes `x_`), with backwards-compat - % probes for older quirks. + % rewrites (`_` becomes `x_`). Used for both schema + % keys with leading underscores (e.g. `_fields`, + % `_maturity_level`) and plain non-prefixed keys (e.g. + % `document_class`, `class_name`). if ~isstruct(s) && ~isobject(s) value = []; return; From 397c27b5cf7d04b4ba4a19332f2e72e9501810a8 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 07:42:53 -0400 Subject: [PATCH 22/32] =?UTF-8?q?v2:=20did2.document=20=E2=80=94=20read=20?= =?UTF-8?q?class=20metadata=20from=20document=5Fclass=20header?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma re-introduced the V_alpha-style `document_class` header in both schema files and document instances. did2.document is updated to match: - className() now reads `documentProperties.document_class.class_name` (was `x_classname`). - classVersion() now reads `documentProperties.document_class.class_version` (was `x_class_version`). - Header doc explains the new wire shape and notes that `document_class` + its sub-keys (`class_name`, `class_version`, `superclasses`) and the class-block keys are plain MATLAB identifiers, while only `_depends_on` (top-level) and `_name` (inside its entries) need the `x_` rename. The `toJSON` regex rewrite is unchanged — it correctly rewrites `"x_depends_on":` and `"x_name":` on encode and leaves `document_class` / `class_name` / `class_version` / `superclasses` verbatim because they never carried the `x_` prefix. --- src/did/+did2/document.m | 49 ++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/src/did/+did2/document.m b/src/did/+did2/document.m index 6028367..6c7671a 100644 --- a/src/did/+did2/document.m +++ b/src/did/+did2/document.m @@ -5,15 +5,17 @@ % (see V_gamma_SPEC.md "JSON Format: Document Instances"), validates % it against the V_gamma schema set, and serialises it back to JSON. % - % In-memory representation. MATLAB struct field names cannot start - % with an underscore, so the four leading-underscore top-level - % keys (`_classname`, `_class_version`, `_superclasses`, - % `_depends_on`) are stored as `x_classname`, `x_class_version`, - % `x_superclasses`, `x_depends_on`, mirroring what `jsondecode` - % produces. Class-block keys (`base`, `daqsystem`, ...) are valid - % MATLAB identifiers and stay verbatim. `toJSON` rewrites - % `"x_":` back to `"_":` on the encoded output so the - % serialised form matches the spec; `fromJSON` relies on + % In-memory representation. The V_gamma document shape carries a + % top-level `document_class` header (with sub-keys `class_name`, + % `class_version`, `superclasses`), plus a top-level `_depends_on` + % array, plus one property block per class in the chain keyed by + % class name. `document_class` and its sub-keys, and the class + % block keys, are all valid MATLAB identifiers and stay verbatim. + % Only `_depends_on` (top-level) and `_name` (inside its entries) + % keep MATLAB's `x_` rename — stored as `x_depends_on` and + % `x_name`, matching what `jsondecode` produces. `toJSON` + % rewrites `"x_":` back to `"_":` on the encoded + % output so the wire form matches the spec; `fromJSON` relies on % `jsondecode`'s default rename to read it back in. % % did2.document Properties: @@ -27,8 +29,8 @@ % iterate - element iterator over an array-of-structure path. % toJSON - serialise to V_gamma JSON text. % toStruct - return the underlying struct. - % className - shorthand for the document's `_classname`. - % classVersion - shorthand for the document's `_class_version`. + % className - shorthand for document_class.class_name. + % classVersion - shorthand for document_class.class_version. % validate - validate this document against its schema. % % did2.document Static Methods: @@ -118,7 +120,10 @@ function jsonText = toJSON(obj, opts) % toJSON - serialise documentProperties to V_gamma JSON text. % Internal `x_` keys are rewritten to `_` on - % the encoded output to match the spec. + % the encoded output to match the spec. Currently the + % only two such keys at the document-instance level are + % `x_depends_on` (top-level) and `x_name` (inside each + % `_depends_on` entry); everything else is already plain. arguments obj opts.PrettyPrint (1,1) logical = false @@ -133,22 +138,26 @@ end function name = className(obj) - % className - the document's `_classname` value. - if isfield(obj.documentProperties, 'x_classname') - name = char(obj.documentProperties.x_classname); + % className - the document's `document_class.class_name`. + if isfield(obj.documentProperties, 'document_class') ... + && isstruct(obj.documentProperties.document_class) ... + && isfield(obj.documentProperties.document_class, 'class_name') + name = char(obj.documentProperties.document_class.class_name); else error('did2:document:missingField', ... - 'Document has no _classname.'); + 'Document has no document_class.class_name.'); end end function v = classVersion(obj) - % classVersion - the document's `_class_version` value. - if isfield(obj.documentProperties, 'x_class_version') - v = char(obj.documentProperties.x_class_version); + % classVersion - the document's `document_class.class_version`. + if isfield(obj.documentProperties, 'document_class') ... + && isstruct(obj.documentProperties.document_class) ... + && isfield(obj.documentProperties.document_class, 'class_version') + v = char(obj.documentProperties.document_class.class_version); else error('did2:document:missingField', ... - 'Document has no _class_version.'); + 'Document has no document_class.class_version.'); end end From 0e409d2f5018d5c131742d7ba9d8865ccc558330 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 08:52:26 -0400 Subject: [PATCH 23/32] =?UTF-8?q?v2:=20fixtures=20=E2=80=94=20drop=20all?= =?UTF-8?q?=20underscore=20prefixes=20per=20V=5Fgamma=20SPEC=20update?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma_SPEC.md was further updated (upstream did-schema commit 77c6363) to drop every leading underscore on NDI-extension keys, in both schema files and document instances. Schema files now use: document_class { class_name, class_version, superclasses } maturity_level depends_on file directory fields[i]: name, type, blank_value, default_value, mustBeNonEmpty, mustBeScalar, mustNotHaveNaN, queryable, ontology { node, name }, documentation, constraints depends_on[i]: name, mustBeNonEmpty, documentation, must_refer_to_document_class, multiple? file[i] / directory[i]: name, documentation superclasses[i] (schema file): class_name, schema superclasses[i] (document): class_name, class_version All NDI-reserved key names live in ndi_reserved_keys.json upstream; schema authors must not reuse them. Fixtures (base, demoA, demoB, demoC, demoFile) rewritten to the plain-key shape. Inheritance, fields, dependency declarations, file records all preserved structurally — only the key names changed. --- tests/+did2/fixtures/V_gamma/base.json | 104 ++++++++++----------- tests/+did2/fixtures/V_gamma/demoA.json | 30 +++--- tests/+did2/fixtures/V_gamma/demoB.json | 30 +++--- tests/+did2/fixtures/V_gamma/demoC.json | 54 +++++------ tests/+did2/fixtures/V_gamma/demoFile.json | 38 ++++---- 5 files changed, 128 insertions(+), 128 deletions(-) diff --git a/tests/+did2/fixtures/V_gamma/base.json b/tests/+did2/fixtures/V_gamma/base.json index 14f4e8b..3763d98 100644 --- a/tests/+did2/fixtures/V_gamma/base.json +++ b/tests/+did2/fixtures/V_gamma/base.json @@ -4,75 +4,75 @@ "class_version": "1.0.0", "superclasses": [] }, - "_maturity_level": "work_in_progress", - "_depends_on": [], - "_file": [], - "_fields": [ + "maturity_level": "work_in_progress", + "depends_on": [], + "file": [], + "fields": [ { - "_name": "id", + "name": "id", "type": "did_uid", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": true, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": { - "_node": "iao:0000578", - "_name": "centrally registered identifier" + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": true, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": { + "node": "iao:0000578", + "name": "centrally registered identifier" }, - "_documentation": "Unique identifier for this document instance.", - "_constraints": {} + "documentation": "Unique identifier for this document instance.", + "constraints": {} }, { - "_name": "session_id", + "name": "session_id", "type": "did_uid", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": true, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": { - "_node": "ncit:C169028", - "_name": "Study Unique Identifier" + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": true, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": { + "node": "ncit:C169028", + "name": "Study Unique Identifier" }, - "_documentation": "Unique identifier of the session this document belongs to.", - "_constraints": {} + "documentation": "Unique identifier of the session this document belongs to.", + "constraints": {} }, { - "_name": "name", + "name": "name", "type": "char", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": false, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": { - "_node": "schema:name", - "_name": "name" + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": false, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": { + "node": "schema:name", + "name": "name" }, - "_documentation": "Human-readable name for this document.", - "_constraints": { + "documentation": "Human-readable name for this document.", + "constraints": { "maxLength": 256 } }, { - "_name": "datestamp", + "name": "datestamp", "type": "timestamp", - "_blank_value": "", - "_default_value": "2018-12-05T18:36:47.241Z", - "_mustBeNonEmpty": true, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": { - "_node": "schema:dateCreated", - "_name": "dateCreated" + "blank_value": "", + "default_value": "2018-12-05T18:36:47.241Z", + "mustBeNonEmpty": true, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": { + "node": "schema:dateCreated", + "name": "dateCreated" }, - "_documentation": "UTC timestamp of document creation in ISO 8601 format.", - "_constraints": {} + "documentation": "UTC timestamp of document creation in ISO 8601 format.", + "constraints": {} } ] } diff --git a/tests/+did2/fixtures/V_gamma/demoA.json b/tests/+did2/fixtures/V_gamma/demoA.json index f59f64c..e776494 100644 --- a/tests/+did2/fixtures/V_gamma/demoA.json +++ b/tests/+did2/fixtures/V_gamma/demoA.json @@ -5,26 +5,26 @@ "superclasses": [ { "class_name": "base", - "_schema": "$DIDSCHEMAPATH/base.json" + "schema": "$DIDSCHEMAPATH/base.json" } ] }, - "_maturity_level": "work_in_progress", - "_depends_on": [], - "_file": [], - "_fields": [ + "maturity_level": "work_in_progress", + "depends_on": [], + "file": [], + "fields": [ { - "_name": "value", + "name": "value", "type": "char", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": false, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": null, - "_documentation": "A demo scalar value used by the +did2 self-tests.", - "_constraints": { + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": false, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": null, + "documentation": "A demo scalar value used by the +did2 self-tests.", + "constraints": { "maxLength": 256 } } diff --git a/tests/+did2/fixtures/V_gamma/demoB.json b/tests/+did2/fixtures/V_gamma/demoB.json index 4cc1819..cce4359 100644 --- a/tests/+did2/fixtures/V_gamma/demoB.json +++ b/tests/+did2/fixtures/V_gamma/demoB.json @@ -5,26 +5,26 @@ "superclasses": [ { "class_name": "demoA", - "_schema": "$DIDSCHEMAPATH/demoA.json" + "schema": "$DIDSCHEMAPATH/demoA.json" } ] }, - "_maturity_level": "work_in_progress", - "_depends_on": [], - "_file": [], - "_fields": [ + "maturity_level": "work_in_progress", + "depends_on": [], + "file": [], + "fields": [ { - "_name": "value_b", + "name": "value_b", "type": "char", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": false, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": null, - "_documentation": "A second demo scalar value, declared by demoB on top of demoA.", - "_constraints": { + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": false, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": null, + "documentation": "A second demo scalar value, declared by demoB on top of demoA.", + "constraints": { "maxLength": 256 } } diff --git a/tests/+did2/fixtures/V_gamma/demoC.json b/tests/+did2/fixtures/V_gamma/demoC.json index 1705561..5f14438 100644 --- a/tests/+did2/fixtures/V_gamma/demoC.json +++ b/tests/+did2/fixtures/V_gamma/demoC.json @@ -5,45 +5,45 @@ "superclasses": [ { "class_name": "base", - "_schema": "$DIDSCHEMAPATH/base.json" + "schema": "$DIDSCHEMAPATH/base.json" } ] }, - "_maturity_level": "work_in_progress", - "_depends_on": [ + "maturity_level": "work_in_progress", + "depends_on": [ { - "_name": "item1", - "_mustBeNonEmpty": true, - "_documentation": "The first dependency item, by id.", - "_must_refer_to_document_class": "" + "name": "item1", + "mustBeNonEmpty": true, + "documentation": "The first dependency item, by id.", + "must_refer_to_document_class": "" }, { - "_name": "item2", - "_mustBeNonEmpty": true, - "_documentation": "The second dependency item, by id.", - "_must_refer_to_document_class": "" + "name": "item2", + "mustBeNonEmpty": true, + "documentation": "The second dependency item, by id.", + "must_refer_to_document_class": "" }, { - "_name": "item3", - "_mustBeNonEmpty": false, - "_documentation": "The third dependency item; optional.", - "_must_refer_to_document_class": "" + "name": "item3", + "mustBeNonEmpty": false, + "documentation": "The third dependency item; optional.", + "must_refer_to_document_class": "" } ], - "_file": [], - "_fields": [ + "file": [], + "fields": [ { - "_name": "value", + "name": "value", "type": "char", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": false, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": null, - "_documentation": "A demo scalar value carried alongside the dependencies.", - "_constraints": { + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": false, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": null, + "documentation": "A demo scalar value carried alongside the dependencies.", + "constraints": { "maxLength": 256 } } diff --git a/tests/+did2/fixtures/V_gamma/demoFile.json b/tests/+did2/fixtures/V_gamma/demoFile.json index 0932800..e3574bc 100644 --- a/tests/+did2/fixtures/V_gamma/demoFile.json +++ b/tests/+did2/fixtures/V_gamma/demoFile.json @@ -5,35 +5,35 @@ "superclasses": [ { "class_name": "base", - "_schema": "$DIDSCHEMAPATH/base.json" + "schema": "$DIDSCHEMAPATH/base.json" } ] }, - "_maturity_level": "work_in_progress", - "_depends_on": [], - "_file": [ + "maturity_level": "work_in_progress", + "depends_on": [], + "file": [ { - "_name": "filename1.ext", - "_documentation": "First demo binary file attached to the document." + "name": "filename1.ext", + "documentation": "First demo binary file attached to the document." }, { - "_name": "filename2.ext2", - "_documentation": "Second demo binary file attached to the document." + "name": "filename2.ext2", + "documentation": "Second demo binary file attached to the document." } ], - "_fields": [ + "fields": [ { - "_name": "value", + "name": "value", "type": "char", - "_blank_value": "", - "_default_value": "", - "_mustBeNonEmpty": false, - "_mustBeScalar": true, - "_mustNotHaveNaN": false, - "_queryable": true, - "_ontology": null, - "_documentation": "A demo scalar value carried alongside the file attachments.", - "_constraints": { + "blank_value": "", + "default_value": "", + "mustBeNonEmpty": false, + "mustBeScalar": true, + "mustNotHaveNaN": false, + "queryable": true, + "ontology": null, + "documentation": "A demo scalar value carried alongside the file attachments.", + "constraints": { "maxLength": 256 } } From ec466b938058714ae94f0339a95af063ee98f359 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 08:53:35 -0400 Subject: [PATCH 24/32] =?UTF-8?q?v2:=20cache.m=20=E2=80=94=20direct=20fiel?= =?UTF-8?q?d=20access=20for=20plain-key=20V=5Fgamma=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma now uses plain keys everywhere (no leading underscores on NDI-extension keys per upstream did-schema commit 77c6363). cache.m follows suit: - Drops the `extractField` helper that probed for `_` / `x_` jsondecode-rename variants. Every key now resolves via direct `isfield`/`s.X` access. - superclasses() reads `s.document_class.superclasses[i].class_name` (no x_ rename anywhere). - ownFields() reads `s.fields` (was `_fields` / `x_fields`). - buildBlankDocument emits: doc.document_class.{class_name, class_version, superclasses} doc.depends_on (empty struct array of {name, value}) doc. for each class in the chain No more `x_depends_on` or `x_name` MATLAB-side. - buildBlockForClass / buildBlankStructure read field-definition keys directly (name, type, blank_value, fields). - validateField reads mustBeNonEmpty / mustBeScalar / mustNotHaveNaN / constraints / type / name directly. Same error IDs and validation semantics as before. The in-memory shape now matches the JSON shape one-to-one — `jsondecode(jsonencode(s))` is the identity for any well-formed V_gamma document. --- src/did/+did2/+schema/cache.m | 129 +++++++++++++--------------------- 1 file changed, 49 insertions(+), 80 deletions(-) diff --git a/src/did/+did2/+schema/cache.m b/src/did/+did2/+schema/cache.m index e3ff72b..2d9c006 100644 --- a/src/did/+did2/+schema/cache.m +++ b/src/did/+did2/+schema/cache.m @@ -11,20 +11,19 @@ % .class_name string concrete class % .class_version string semver of the concrete class % .superclasses array [{class_name, class_version}] - % _depends_on array [{_name, value}] + % depends_on array [{name, value}] % object one property block per class % in the chain. Contains the % field values that class % declared (empty {} if it % declares no fields). % - % MATLAB representation: the `document_class` header and its - % sub-keys (`class_name`, `class_version`, `superclasses`) are - % already valid MATLAB struct field names and stay as written. - % Only `_depends_on` (top-level) and `_name` (inside its entries) - % keep MATLAB's `x_` rename — stored as `x_depends_on` and - % `x_name`. did2.document.toJSON rewrites `"x_":` back to - % `"_":` on serialisation. + % MATLAB representation: every key in the V_gamma wire shape is a + % valid MATLAB struct field name (no leading underscores anywhere + % after the V_gamma SPEC's "drop underscore prefixes" update), so + % the in-memory representation is the JSON shape verbatim. + % `jsondecode` returns a struct with the same field names, and + % `jsonencode` writes them back without any rename pass. % % did2.schema.cache Properties: % schemaPath - filesystem path to a V_gamma schema dir. @@ -40,7 +39,7 @@ % getClass - resolved class definition for a name. % superclasses - ancestor chain (parent first, root last). % classChain - root-first list including the class itself. - % ownFields - the _fields list a class declares directly. + % ownFields - the `fields` list a class declares directly. % fieldsFor - merged inherited fields tagged with the % declaring class (struct array). % queryablePaths - scalar and array-iteration paths (stub). @@ -105,18 +104,19 @@ end visited(current) = true; s = obj.getClass(current); - dc = obj.extractField(s, 'document_class'); - if isempty(dc) + if ~isstruct(s) || ~isfield(s, 'document_class') ... + || ~isstruct(s.document_class) ... + || ~isfield(s.document_class, 'superclasses') ... + || isempty(s.document_class.superclasses) break; end - parents = obj.extractField(dc, 'superclasses'); - if isempty(parents) + parent = obj.elementAt(s.document_class.superclasses, 1); + if ~isstruct(parent) || ~isfield(parent, 'class_name') break; end - parent = obj.elementAt(parents, 1); - parentName = obj.extractField(parent, 'class_name'); - names{end+1} = char(parentName); %#ok - current = char(parentName); + parentName = char(parent.class_name); + names{end+1} = parentName; %#ok + current = parentName; end end @@ -138,19 +138,18 @@ className (1,:) char end s = obj.getClass(className); - raw = obj.extractField(s, '_fields'); - if isempty(raw) + if ~isstruct(s) || ~isfield(s, 'fields') || isempty(s.fields) fields = {}; - else - fields = obj.toCellArray(raw); + return; end + fields = obj.toCellArray(s.fields); end function tagged = fieldsFor(obj, className) % fieldsFor - merged inherited fields tagged with the % declaring class. Returns a struct array with fields % `declaringClass` (char) and `fieldDef` (the schema's - % _fields entry). + % `fields` entry). arguments obj className (1,:) char @@ -187,36 +186,28 @@ className (1,:) char end doc = struct(); - % document_class header. schema = obj.getClass(className); - schemaDC = obj.extractField(schema, 'document_class'); - classNameVal = char(obj.extractField(schemaDC, 'class_name')); - classVersionVal = char(obj.extractField(schemaDC, 'class_version')); + schemaDC = schema.document_class; ancestors = obj.superclasses(className); sc = struct('class_name', {}, 'class_version', {}); for k = 1:numel(ancestors) - ancSchema = obj.getClass(ancestors{k}); - ancDC = obj.extractField(ancSchema, 'document_class'); + ancDC = obj.getClass(ancestors{k}).document_class; sc(end+1) = struct( ... - 'class_name', char(obj.extractField(ancDC, 'class_name')), ... - 'class_version', char(obj.extractField(ancDC, 'class_version'))); %#ok + 'class_name', char(ancDC.class_name), ... + 'class_version', char(ancDC.class_version)); %#ok end - dc = struct( ... - 'class_name', classNameVal, ... - 'class_version', classVersionVal, ... + doc.document_class = struct( ... + 'class_name', char(schemaDC.class_name), ... + 'class_version', char(schemaDC.class_version), ... 'superclasses', sc); - doc.document_class = dc; - % Top-level _depends_on (empty struct array of {_name, value}). - doc.x_depends_on = struct('x_name', {}, 'value', {}); + doc.depends_on = struct('name', {}, 'value', {}); - % One property block per class in the chain. chain = obj.classChain(className); for k = 1:numel(chain) blockClass = chain{k}; - block = obj.buildBlockForClass(blockClass); - doc.(blockClass) = block; + doc.(blockClass) = obj.buildBlockForClass(blockClass); end end @@ -263,8 +254,7 @@ function validateDocument(obj, docOrStruct) own = obj.ownFields(blockClass); for f = 1:numel(own) fieldDef = own{f}; - fieldName = char(obj.extractField(fieldDef, '_name')); - obj.validateField(block, fieldDef, blockClass, fieldName); + obj.validateField(block, fieldDef, blockClass, char(fieldDef.name)); end end end @@ -344,26 +334,6 @@ function loadRegistry(obj) end end - function value = extractField(~, s, name) - % extractField - tolerate jsondecode's leading-underscore - % rewrites (`_` becomes `x_`). Used for both schema - % keys with leading underscores (e.g. `_fields`, - % `_maturity_level`) and plain non-prefixed keys (e.g. - % `document_class`, `class_name`). - if ~isstruct(s) && ~isobject(s) - value = []; - return; - end - candidates = {name, ['x' name], strrep(name, '_', '')}; - for k = 1:numel(candidates) - if isfield(s, candidates{k}) - value = s.(candidates{k}); - return; - end - end - value = []; - end - function out = toCellArray(~, raw) if iscell(raw) out = raw(:)'; @@ -381,16 +351,16 @@ function loadRegistry(obj) function block = buildBlockForClass(obj, className) % buildBlockForClass - one property block populated with - % _blank_value for every field the class declares + % `blank_value` for every field the class declares % directly. Base block also receives a fresh did_uid for % `id` and the current UTC timestamp for `datestamp`. block = struct(); own = obj.ownFields(className); for f = 1:numel(own) fieldDef = own{f}; - fieldName = char(obj.extractField(fieldDef, '_name')); - blank = obj.extractField(fieldDef, '_blank_value'); - fieldType = char(obj.extractField(fieldDef, 'type')); + fieldName = char(fieldDef.name); + blank = fieldDef.blank_value; + fieldType = char(fieldDef.type); if strcmp(fieldType, 'structure') ... && (isempty(blank) || (isstruct(blank) && isempty(fieldnames(blank)))) block.(fieldName) = obj.buildBlankStructure(fieldDef); @@ -409,17 +379,16 @@ function loadRegistry(obj) end function s = buildBlankStructure(obj, fieldDef) - nested = obj.extractField(fieldDef, '_fields'); s = struct(); - if isempty(nested) + if ~isfield(fieldDef, 'fields') || isempty(fieldDef.fields) return; end - entries = obj.toCellArray(nested); + entries = obj.toCellArray(fieldDef.fields); for k = 1:numel(entries) subDef = entries{k}; - subName = char(obj.extractField(subDef, '_name')); - subBlank = obj.extractField(subDef, '_blank_value'); - subType = char(obj.extractField(subDef, 'type')); + subName = char(subDef.name); + subBlank = subDef.blank_value; + subType = char(subDef.type); if strcmp(subType, 'structure') ... && (isempty(subBlank) || (isstruct(subBlank) && isempty(fieldnames(subBlank)))) s.(subName) = obj.buildBlankStructure(subDef); @@ -430,12 +399,13 @@ function loadRegistry(obj) end function validateField(obj, block, fieldDef, blockClass, fieldName) - % validateField - apply type, _mustBe* flags, and - % _constraints for one field against the property block. + % validateField - apply type, mustBe* flags, and + % constraints for one field against the property block. % Skips absent fields unless the schema marks them - % _mustBeNonEmpty. + % mustBeNonEmpty. + mustBeNonEmpty = logical(fieldDef.mustBeNonEmpty); if ~isfield(block, fieldName) - if obj.extractField(fieldDef, '_mustBeNonEmpty') + if mustBeNonEmpty error('did2:validation:missingField', ... 'Required field "%s.%s" is missing.', ... blockClass, fieldName); @@ -443,13 +413,12 @@ function validateField(obj, block, fieldDef, blockClass, fieldName) return; end value = block.(fieldName); - fieldType = char(obj.extractField(fieldDef, 'type')); + fieldType = char(fieldDef.type); qualifiedName = sprintf('%s.%s', blockClass, fieldName); obj.validateTypeShape(value, fieldType, qualifiedName); - mustBeNonEmpty = logical(obj.extractField(fieldDef, '_mustBeNonEmpty')); - mustBeScalar = logical(obj.extractField(fieldDef, '_mustBeScalar')); - mustNotHaveNaN = logical(obj.extractField(fieldDef, '_mustNotHaveNaN')); + mustBeScalar = logical(fieldDef.mustBeScalar); + mustNotHaveNaN = logical(fieldDef.mustNotHaveNaN); if mustBeNonEmpty && obj.isEmptyValue(value) error('did2:validation:emptyField', ... 'Field "%s" is required to be non-empty.', qualifiedName); @@ -462,7 +431,7 @@ function validateField(obj, block, fieldDef, blockClass, fieldName) error('did2:validation:nanValue', ... 'Field "%s" contains NaN.', qualifiedName); end - constraints = obj.extractField(fieldDef, '_constraints'); + constraints = fieldDef.constraints; if isstruct(constraints) && ~isempty(fieldnames(constraints)) obj.validateConstraints(value, constraints, fieldType, qualifiedName); end From af0092978daf71914b66e263cc3f9be67323a273 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 08:54:10 -0400 Subject: [PATCH 25/32] =?UTF-8?q?v2:=20did2.document=20=E2=80=94=20drop=20?= =?UTF-8?q?toJSON=20regex=20rewrite=20for=20plain-key=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma's "drop underscore prefixes" pass removed every leading underscore on NDI-extension keys, so the in-memory MATLAB struct now matches the JSON shape one-to-one. did2.document follows suit: - toJSON is now a bare jsonencode call. The rewriteXUnderscoreKeys regex helper is removed — there are no "x_" keys left to translate. - Header doc updated to describe the simpler representation (jsonencode/jsondecode round-trip without any rewrite pass). - className() / classVersion() continue to read documentProperties.document_class.class_name / documentProperties.document_class.class_version unchanged. dot-path get/set, iterate, fromJSON/fromStruct/blank, and the (className, valueStruct) constructor are unchanged. --- src/did/+did2/document.m | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-) diff --git a/src/did/+did2/document.m b/src/did/+did2/document.m index 6c7671a..6727be1 100644 --- a/src/did/+did2/document.m +++ b/src/did/+did2/document.m @@ -7,16 +7,13 @@ % % In-memory representation. The V_gamma document shape carries a % top-level `document_class` header (with sub-keys `class_name`, - % `class_version`, `superclasses`), plus a top-level `_depends_on` + % `class_version`, `superclasses`), plus a top-level `depends_on` % array, plus one property block per class in the chain keyed by - % class name. `document_class` and its sub-keys, and the class - % block keys, are all valid MATLAB identifiers and stay verbatim. - % Only `_depends_on` (top-level) and `_name` (inside its entries) - % keep MATLAB's `x_` rename — stored as `x_depends_on` and - % `x_name`, matching what `jsondecode` produces. `toJSON` - % rewrites `"x_":` back to `"_":` on the encoded - % output so the wire form matches the spec; `fromJSON` relies on - % `jsondecode`'s default rename to read it back in. + % class name. After V_gamma's "drop underscore prefixes" pass, + % every key in the wire shape is a valid MATLAB struct field name, + % so the in-memory representation is the JSON shape verbatim. + % `jsonencode`/`jsondecode` round-trip without any name-rewrite + % pass. % % did2.document Properties: % documentProperties - struct mirroring the V_gamma JSON shape. @@ -119,18 +116,12 @@ function jsonText = toJSON(obj, opts) % toJSON - serialise documentProperties to V_gamma JSON text. - % Internal `x_` keys are rewritten to `_` on - % the encoded output to match the spec. Currently the - % only two such keys at the document-instance level are - % `x_depends_on` (top-level) and `x_name` (inside each - % `_depends_on` entry); everything else is already plain. arguments obj opts.PrettyPrint (1,1) logical = false end - raw = jsonencode(obj.documentProperties, ... + jsonText = jsonencode(obj.documentProperties, ... 'PrettyPrint', opts.PrettyPrint); - jsonText = did2.document.rewriteXUnderscoreKeys(raw); end function s = toStruct(obj) @@ -282,15 +273,6 @@ function validate(obj, opts) end s = cache.buildBlankDocument(className); end - - function out = rewriteXUnderscoreKeys(jsonText) - % rewriteXUnderscoreKeys - convert `"x_":` keys to - % `"_":` on the encoded JSON text. The regex matches - % only JSON keys (colon-terminated, with optional - % whitespace) so values that happen to start with `x_` - % are unaffected. - out = regexprep(jsonText, '"x_([a-zA-Z][a-zA-Z0-9_]*)"(\s*):', '"_$1"$2:'); - end end methods (Access = private) From a160bcf4615e635a8435eacbb3a19e39487bd523 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 08:54:46 -0400 Subject: [PATCH 26/32] =?UTF-8?q?v2:=20testSchemaCache.m=20=E2=80=94=20ass?= =?UTF-8?q?ertions=20on=20plain-key=20V=5Fgamma=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma dropped all leading underscores on NDI-extension keys, so the in-memory document is the JSON shape verbatim. Tests updated: - Assertions on the document_class header read doc.document_class.class_name / class_version / superclasses(i).class_name (was doc.x_classname / x_class_version / x_superclasses(i).x_classname). - depends_on assertion: isfield(doc, 'depends_on') (was 'x_depends_on'). - Validation tests unchanged — same error IDs and semantics. - toJSON test now confirms the wire form uses "document_class" and "class_name":"demoA" rather than the previous "_classname":"demoA" / "x_classname" pattern. Also re-parses and verifies the class blocks survive a JSON round-trip. 22 tests; same coverage as before, just adapted to the new shape. --- tests/+did2/testSchemaCache.m | 41 ++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/tests/+did2/testSchemaCache.m b/tests/+did2/testSchemaCache.m index 1f63231..8a86d17 100644 --- a/tests/+did2/testSchemaCache.m +++ b/tests/+did2/testSchemaCache.m @@ -3,12 +3,11 @@ % V_gamma fixtures at tests/+did2/fixtures/V_gamma/. Also covers % did2.document.blank() and did2.document.validate() end-to-end. % -% Documents in V_gamma use class-scoped property blocks (see -% V_gamma_SPEC.md "JSON Format: Document Instances"): top-level -% `_classname`, `_class_version`, `_superclasses`, `_depends_on`, plus -% one property block per class in the chain. MATLAB stores the four -% underscore-prefixed system keys as `x_` (mirroring jsondecode); -% class-block keys (`base`, `demoA`, ...) are plain MATLAB identifiers. +% Documents in V_gamma use a top-level `document_class` header plus +% class-scoped property blocks (see V_gamma_SPEC.md "JSON Format: +% Document Instances"). After V_gamma's "drop underscore prefixes" +% pass, every key in the wire shape is a valid MATLAB struct field +% name, so `jsonencode`/`jsondecode` round-trip without any rewrite. tests = functiontests(localfunctions); end @@ -34,6 +33,7 @@ function testSchemaPathPointsAtFixtures(testCase) function testGetClassLoadsBase(testCase) s = testCase.TestData.cache.getClass('base'); verifyTrue(testCase, isstruct(s)); +verifyTrue(testCase, isfield(s, 'document_class')); end function testGetClassMissingThrows(testCase) @@ -88,21 +88,22 @@ function testFieldsForTagsDeclaringClass(testCase) verifyEqual(testCase, tagged(6).declaringClass, 'demoB'); end -% ---- buildBlankDocument: top-level metadata ---- +% ---- buildBlankDocument: document_class header ---- -function testBuildBlankDocumentTopLevelMetadata(testCase) +function testBuildBlankDocumentHeader(testCase) doc = testCase.TestData.cache.buildBlankDocument('demoB'); -verifyEqual(testCase, doc.x_classname, 'demoB'); -verifyEqual(testCase, doc.x_class_version, '1.0.0'); -verifyEqual(testCase, numel(doc.x_superclasses), 2); -verifyEqual(testCase, doc.x_superclasses(1).x_classname, 'demoA'); -verifyEqual(testCase, doc.x_superclasses(2).x_classname, 'base'); +verifyTrue(testCase, isfield(doc, 'document_class')); +verifyEqual(testCase, doc.document_class.class_name, 'demoB'); +verifyEqual(testCase, doc.document_class.class_version, '1.0.0'); +verifyEqual(testCase, numel(doc.document_class.superclasses), 2); +verifyEqual(testCase, doc.document_class.superclasses(1).class_name, 'demoA'); +verifyEqual(testCase, doc.document_class.superclasses(2).class_name, 'base'); end function testBuildBlankDocumentEmptyDependsOn(testCase) doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyTrue(testCase, isfield(doc, 'x_depends_on')); -verifyEmpty(testCase, doc.x_depends_on); +verifyTrue(testCase, isfield(doc, 'depends_on')); +verifyEmpty(testCase, doc.depends_on); end % ---- buildBlankDocument: class-scoped blocks ---- @@ -214,12 +215,16 @@ function testDocumentValidateRoundTrip(testCase) doc.validate(); end -function testDocumentToJSONRewritesUnderscoreKeys(testCase) +function testDocumentToJSONRoundTrip(testCase) +% V_gamma has no leading-underscore keys, so jsonencode/jsondecode is +% identity for any well-formed document. Confirm the wire shape uses +% the V_gamma key names and re-parses to an equivalent document. doc = did2.document.blank('demoA'); text = doc.toJSON(); -verifyTrue(testCase, contains(text, '"_classname":"demoA"')); -verifyFalse(testCase, contains(text, '"x_classname"')); +verifyTrue(testCase, contains(text, '"document_class"')); +verifyTrue(testCase, contains(text, '"class_name":"demoA"')); doc2 = did2.document.fromJSON(text); verifyEqual(testCase, doc2.className(), 'demoA'); verifyTrue(testCase, isfield(doc2.toStruct(), 'base')); +verifyTrue(testCase, isfield(doc2.toStruct(), 'demoA')); end From d5d63ed90beae0bc52bcf502d3acd20e4d6f2bdb Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 08:56:14 -0400 Subject: [PATCH 27/32] =?UTF-8?q?v2:=20PLAN.md=20=E2=80=94=20simplify=20?= =?UTF-8?q?=C2=A74.1=20+=20log=20the=20plain-key=20V=5Fgamma=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V_gamma went through two back-to-back SPEC revisions on 2026-05-12: restore class-scoped property blocks (did-schema 137f583), then drop the leading-underscore convention on every NDI-extension key (did-schema 77c6363). Net result: the in-memory MATLAB struct is the JSON shape verbatim — no x_ aliasing or toJSON rewrite pass. Changes: - Decision #8 in §1 updated to describe the new layout (document_class header + class-scoped blocks + plain keys everywhere) and to record the two-step SPEC history. - §4.1 "In-memory document shape" rewritten. The whole section is now a few short paragraphs plus a four-row table — the MATLAB representation is plain, no encoding/aliasing remarks needed. The V_alpha → V_gamma comparison table is updated to show that the converter (§7) is now a thin per-document data migration (V_alpha's class-scoped layout maps to V_gamma almost verbatim once `property_list_name` and `definition`/`validation` sub-keys are dropped). - §12 progress log gains a 2026-05-12 entry covering both SPEC revisions and the resulting +did2 changes (cache, document, fixtures, tests). - Stray `_queryable` / `_ontology` references in §§3.2, 5, 7 were corrected to the plain names. --- docs/v2/PLAN.md | 192 ++++++++++++++++++++++-------------------------- 1 file changed, 86 insertions(+), 106 deletions(-) diff --git a/docs/v2/PLAN.md b/docs/v2/PLAN.md index 3b079e6..3d1c727 100644 --- a/docs/v2/PLAN.md +++ b/docs/v2/PLAN.md @@ -20,7 +20,7 @@ users. | 5 | Validate on insert by default; expose an `unsafe_insert` escape hatch for bulk loads; offer a `revalidate_all` maintenance op. | Schema files are the source of truth for what "valid" means. | | 6 | Plan lives at `docs/v2/PLAN.md` on the v2 development branch. | This file. | | 7 | Provisional namespace: `+did2`. | Picked from §10 option A for the scaffold. Revisit before v2 reaches `main`. | -| 8 | Document instances use class-scoped property blocks (one block per class in the chain, keyed by `_classname` verbatim). | See §4.1. Matches V_gamma_SPEC.md "JSON Format: Document Instances" after the SPEC update that restored V_alpha-style class scoping. MATLAB struct field names can't begin with `_`, so the four top-level system keys are stored as `x_*` internally and rewritten to `_*` on serialise. | +| 8 | Document instances use a top-level `document_class` header plus class-scoped property blocks (one block per class in the chain, keyed by `class_name` verbatim). | See §4.1. Matches V_gamma_SPEC.md "JSON Format: Document Instances" after the SPEC's two-step revision: (i) restore class-scoped blocks; (ii) drop the underscore prefix on all NDI-extension keys. Every key in the wire shape is a plain MATLAB identifier, so the in-memory MATLAB struct is the JSON shape verbatim. | Open questions are in §10. @@ -89,7 +89,7 @@ CREATE INDEX depends_on_name_value ON depends_on(name, value); Test 4 in the JSON1 probe confirmed that `STORED GENERATED ALWAYS AS (json_extract(body, '$.foo.bar'))` works with `mksqlite`. So for each scalar -`_queryable: true` path declared by the V_gamma schemas, we add a stored +`queryable: true` path declared by the V_gamma schemas, we add a stored generated column directly on `documents` plus an index on it. The set of paths is computed at database open by walking the loaded schemas: @@ -160,60 +160,46 @@ schemas change. A `did2.document`'s `documentProperties` is a MATLAB struct that mirrors the V_gamma JSON shape *as specified in V_gamma_SPEC.md, "JSON Format: Document -Instances"*. V_gamma documents use **class-scoped property blocks**: there -is one top-level block per class in the inheritance chain, keyed by that -class's `_classname` verbatim. Fields live in the block of the class that -declared them. There is no `property_list_name` knob — the block key -*is* the class name. - -The MATLAB representation has one quirk: struct field names cannot begin -with `_`. The four underscore-prefixed top-level system keys (`_classname`, -`_class_version`, `_superclasses`, `_depends_on`) are stored as -`x_classname`, `x_class_version`, `x_superclasses`, `x_depends_on` — -mirroring exactly what `jsondecode` produces. Class-block keys (`base`, -`daqsystem`, ...) are valid MATLAB identifiers (V_gamma classnames match -`^[a-z][a-z0-9_]*$`) and stay verbatim. `did2.document.toJSON` rewrites -`"x_":` keys back to `"_":` on the encoded output so the wire -form matches the spec; `fromJSON` relies on `jsondecode`'s default rename -on parse. +Instances"*, exactly. After V_gamma's "drop underscore prefixes" pass, +every key in the wire shape is a plain identifier with no leading +underscore, so the MATLAB struct field names match the JSON keys +one-to-one. `jsonencode` / `jsondecode` round-trip without any rewrite. Top-level keys populated by `did2.schema.cache.buildBlankDocument`: -| Key (JSON / MATLAB) | Type | Source | -|---|---|---| -| `_classname` / `x_classname` | char | concrete class's `_classname` | -| `_class_version` / `x_class_version` | char | concrete class's `_class_version` | -| `_superclasses` / `x_superclasses` | struct array | one elem per ancestor; each has `_classname` (`x_classname`) and `_class_version` (`x_class_version`) | -| `_depends_on` / `x_depends_on` | struct array | each entry has `_name` (`x_name`) and `value`; empty by default | -| `base` / `base` | struct | property block with the four base fields (`id`, `session_id`, `name`, `datestamp`); `id` auto-minted via `did.ido.unique_id`, `datestamp` set to current UTC ISO-8601 | -| `` / `` | struct | one property block per class in the chain, including the concrete class itself; each populated with `_blank_value` for the fields *that class* declares (empty `{}` if it declares none) | - -Field identity is `(declaring_class, _name)`. Same-named fields in +| Key | Type | Contents | +|------------------|--------------|----------| +| `document_class` | struct | `class_name` (concrete class), `class_version` (semver), `superclasses` (struct array; each entry has `class_name` + `class_version` — the document-instance form). | +| `depends_on` | struct array | Each entry: `name` (role) and `value` (the referenced document's id). Empty by default. | +| `base` | struct | Property block with the four base fields (`id`, `session_id`, `name`, `datestamp`). `id` auto-minted via `did.ido.unique_id`, `datestamp` set to current UTC millisecond ISO-8601 with trailing `Z`. | +| `` | struct | One property block per class in the chain (root through concrete class). Each populated with `blank_value` for the fields *that class* declares. Empty `{}` if it declares none. | + +Field identity is `(declaring_class, name)`. Same-named fields in different classes of the chain are distinct paths (`base.id` vs. -`.id`), not an override. The SPEC's "no shadowing, by -construction" rule means validators and query engines work in -class-qualified paths. +`.id`), not an override. V_alpha → V_gamma at the document level: ``` V_alpha V_gamma ------- ------- -document_class.class_name _classname -document_class.class_version _class_version -document_class.superclasses _superclasses -document_class.property_list_name (gone; block key == _classname) +document_class.class_name document_class.class_name +document_class.class_version document_class.class_version +document_class.superclasses document_class.superclasses +document_class.property_list_name (gone; block key == class_name) document_class.definition (gone; schema files own this) document_class.validation (gone; schema files own this) base.id, base.session_id, ... base.id, base.session_id, ... -. <_classname>. -depends_on (top-level list) _depends_on (renamed only) +. . +depends_on depends_on ``` -The converter (§7) inverts the mapping when reading V_alpha documents: -strip `document_class`, write the four V_gamma top-level keys, and -rename each property block whose `property_list_name` differs from its -`class_name` so the block key equals the class name. +The converter (§7) is now a thin per-document data migration: strip the +extra `document_class` sub-keys (`property_list_name`, `definition`, +`validation`); rename each property block whose `property_list_name` +differs from its `class_name` so the block key equals the class name; +done. NDI-matlab consumers that already speak the V_alpha class-scoped +layout need no source-code rewrites for the wire shape itself. --- @@ -223,7 +209,7 @@ A `+did2/+schema/cache.m` (or similar) loads all V_gamma schema files once, resolves superclass chains, and pre-computes: - For each classname: the full inherited field list. -- The subset of fields with `_queryable: true`, split into scalar paths and +- The subset of fields with `queryable: true`, split into scalar paths and array-iteration paths. - The named composite type expansions (`duration` → `.seconds`, `.approximate`, `.source_unit`, `.source_value`). @@ -271,7 +257,7 @@ A `+did2/+convert/v1_to_v2.m` tool: the table lives next to the v2 schema package). 3. Renames top-level keys (`base.id` → `id`, etc.), rewrites collapsed fields on classes that bumped to `2.0.0` (`probe_location`, `treatment`, - `ontology_image`, `ontology_label`), and reshapes `_ontology` annotations + `ontology_image`, `ontology_label`), and reshapes `ontology` annotations to the V_gamma two-key form. 4. Validates against V_gamma. Successful docs insert into the new DB; failures land in a `quarantine` table with the original body and a reason string. @@ -395,80 +381,74 @@ Next up: fill in `did2.schema.cache.fieldsFor`, against the V_gamma meta-schema; then start the in-memory query evaluator (step 2). -### 2026-05-12 — close step 1: class-scoped cache implementations + fixtures - -V_gamma_SPEC.md was amended (upstream did-schema commit `137f583`) to -restore class-scoped property blocks on document instances, replacing -the earlier "flatten on inheritance" wire shape. Decision #8 above is -revised accordingly, and §4.1 documents the resulting in-memory layout -(class-block top-level keys plus the four `_` keys stored as -`x_`). - -Filled in the schema cache to the level step 1 needs and wired -end-to-end tests through `did2.document.blank` / `did2.document.validate`. +### 2026-05-12 — class-scoped property blocks, then drop underscores + +Two upstream did-schema SPEC revisions landed back-to-back and both +required reworking the +did2 in-memory shape: + +1. **Class-scoped property blocks restored** (did-schema commit + `137f583`). V_gamma was amended to organise document instances + into per-class property blocks keyed by class name (one per class + in the chain), instead of the earlier flat namespace. Also moved + `class_name`/`class_version`/`superclasses` under a top-level + `document_class` header. +2. **Drop underscore prefixes** (did-schema commit `77c6363`). The + `_` convention for NDI-extension keys was replaced by plain + keys (`maturity_level`, `depends_on`, `file`, `fields`, + `mustBeNonEmpty`, `blank_value`, `ontology`, etc.). The + authoritative reserved-name list moved to upstream + `ndi_reserved_keys.json`. + +Combined, every key in a V_gamma wire shape is now a plain MATLAB +identifier, so the in-memory MATLAB struct is the JSON shape verbatim +— no `x_` aliasing, no `jsonencode`-time rewrite pass, no +`extractField` underscore-probe helper. Round-tripping a V_gamma +document is `jsondecode` then `jsonencode`. Implemented in `src/did/+did2/+schema/cache.m`: - `classChain(className)` — root-first list including the class itself (e.g., `demoB -> {base, demoA, demoB}`). -- `ownFields(className)` — the `_fields` list the class declares - directly (no inheritance). +- `ownFields(className)` — the `fields` list the class declares + directly (no inheritance), via direct `s.fields` access. - `fieldsFor(className)` — merged inherited fields tagged with their - declaring class. Returns a struct array `(declaringClass, fieldDef)` - so callers know which property block each field belongs to. + declaring class. Returns a struct array + `{declaringClass, fieldDef}`. +- `superclasses(className)` — walks + `s.document_class.superclasses[i].class_name` up the chain. - `buildBlankDocument(className)` — class-scoped V_gamma document: - top-level `x_classname` / `x_class_version` / `x_superclasses` - (struct array of `{x_classname, x_class_version}` ancestor entries - parent-first) / empty `x_depends_on`, plus one property block per - class in the chain. Base block has `id` auto-minted via - `did.ido.unique_id()` and `datestamp` set to current UTC - millisecond ISO-8601 with trailing `Z`. Empty blocks present for - classes that declare no fields, per the SPEC. + `doc.document_class.{class_name, class_version, superclasses}` + `doc.depends_on` — empty struct array of `{name, value}` + `doc.` for each class in the chain + Base block has `id` auto-minted via `did.ido.unique_id()` and + `datestamp` set to current UTC ISO-8601. - `validateDocument(docOrStruct)` — accepts a `did2.document` or its - underlying struct, walks the class chain, and for each class - validates the fields *that class declares* against its property - block. Error messages use the qualified `.` form. - Type-shape runs before `_mustBe*` checks so a wrong-type value is - reported as `did2:validation:typeMismatch` rather than - `did2:validation:notScalar`. New error IDs added: - `did2:validation:missingClassBlock`, `did2:validation:badClassBlock`. -- `queryablePaths` stays a stub — it belongs to steps 3 and 4 and - will return class-qualified dot-paths (e.g., - `daqsystem.sample_rate.hertz`) once the storage layer lands. + underlying struct, walks the class chain, and validates each + class's `fields` against its property block. Error messages use + the qualified `.` form; new error IDs + `did2:validation:missingClassBlock` and `:badClassBlock`. +- `queryablePaths` stays a stub (belongs to steps 3 and 4). In `src/did/+did2/document.m`: -- `className` / `classVersion` read from `x_classname` / `x_class_version`. -- `toJSON` post-processes the `jsonencode` output with a regex that - rewrites `"x_":` keys back to `"_":`. This is the only - place we serialise V_gamma JSON; `fromJSON` relies on - `jsondecode`'s default behaviour to read it back in. +- `className` / `classVersion` read + `documentProperties.document_class.class_name` / + `documentProperties.document_class.class_version`. +- `toJSON` is a bare `jsonencode` (no rewrite pass). The previous + `rewriteXUnderscoreKeys` helper is removed. -Added `tests/+did2/fixtures/V_gamma/`: +Fixtures at `tests/+did2/fixtures/V_gamma/` (`base.json`, +`demoA.json`, `demoB.json`, `demoC.json`, `demoFile.json`, +`CURIE_lookups_meta.json`, `README.md`) rewritten to the +plain-key V_gamma shape. -| File | Origin | Why | -|---|---|---| -| `base.json` | upstream `did-schema` V_gamma | The root class. | -| `CURIE_lookups_meta.json` | upstream | Exercises registry load. | -| `demoA.json` | V_gamma translation of v1 `demoA.json` | `base` subclass with one queryable char field. | -| `demoB.json` | V_gamma translation of v1 `demoB.json` | Multi-level (`demoB → demoA → base`). | -| `demoC.json` | V_gamma translation of v1 `demoC.json` | Declares three `_depends_on` entries. | -| `demoFile.json` | V_gamma translation of v1 `demoFile.json` | Declares `_file` attachments. | -| `README.md` | new | Documents origins and refresh procedure. | - -New `tests/+did2/testSchemaCache.m` (function-based) points the cache -at the fixture via `setSchemaPath` in `setupOnce`, then covers: schema-path -plumbing, `getClass` (incl. missing-class error), CURIE registry presence, -superclass-chain depth, `classChain`, `ownFields`, `fieldsFor` declaring-class -tagging, `buildBlankDocument` (top-level metadata, all chain blocks present -as expected, minted id, current UTC datestamp), `validateDocument` -(empty-field, max-length boundary, type mismatch, missing class_name, -missing class block), and an end-to-end `toJSON` round-trip that asserts -the serialised form uses `_classname` and not `x_classname`. - -Step 1 is now complete to the level the rest of the plan needs. +`tests/+did2/testSchemaCache.m` updated: 22 tests assert on the +plain-key shape (`doc.document_class.class_name`, `doc.depends_on`, +etc.) and check that a V_gamma document round-trips through +`toJSON`/`fromJSON` unchanged. + +Step 1 is complete to the level the rest of the plan needs. `queryablePaths` is the only intentional stub left in the cache; -detailed per-named-composite validation and `_depends_on` value -checks are deferred to a focused follow-up. Next up: step 2 — the -in-memory query evaluator over the class-scoped paths, which will -also be the executable spec the SQL compiler is later tested against. +detailed per-named-composite validation and dependency-value checks +are deferred to focused follow-ups. Next up: step 2 — the in-memory +query evaluator over the class-qualified dot-paths. From 56fa3b74b9c47b99fced770bbe34acd35cfeb2ad Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 09:05:06 -0400 Subject: [PATCH 28/32] =?UTF-8?q?v2:=20fixtures=20=E2=80=94=20move=20matur?= =?UTF-8?q?ity=5Flevel=20inside=20document=5Fclass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream did-schema (commit b54c940) moved `maturity_level` from a top-level key into the `document_class` header alongside `class_name`, `class_version`, `superclasses`. Fixtures updated to match: all five fixture schemas (base, demoA, demoB, demoC, demoFile) now carry `maturity_level: "work_in_progress"` as the fourth sub-key of `document_class`. No +did2 code reads `maturity_level` today, so cache.m / document.m are unchanged. Tests are unchanged (no assertions on `maturity_level`). --- tests/+did2/fixtures/V_gamma/base.json | 4 ++-- tests/+did2/fixtures/V_gamma/demoA.json | 4 ++-- tests/+did2/fixtures/V_gamma/demoB.json | 4 ++-- tests/+did2/fixtures/V_gamma/demoC.json | 4 ++-- tests/+did2/fixtures/V_gamma/demoFile.json | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/tests/+did2/fixtures/V_gamma/base.json b/tests/+did2/fixtures/V_gamma/base.json index 3763d98..f8e73f0 100644 --- a/tests/+did2/fixtures/V_gamma/base.json +++ b/tests/+did2/fixtures/V_gamma/base.json @@ -2,9 +2,9 @@ "document_class": { "class_name": "base", "class_version": "1.0.0", - "superclasses": [] + "superclasses": [], + "maturity_level": "work_in_progress" }, - "maturity_level": "work_in_progress", "depends_on": [], "file": [], "fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoA.json b/tests/+did2/fixtures/V_gamma/demoA.json index e776494..7233b69 100644 --- a/tests/+did2/fixtures/V_gamma/demoA.json +++ b/tests/+did2/fixtures/V_gamma/demoA.json @@ -7,9 +7,9 @@ "class_name": "base", "schema": "$DIDSCHEMAPATH/base.json" } - ] + ], + "maturity_level": "work_in_progress" }, - "maturity_level": "work_in_progress", "depends_on": [], "file": [], "fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoB.json b/tests/+did2/fixtures/V_gamma/demoB.json index cce4359..7e6c91e 100644 --- a/tests/+did2/fixtures/V_gamma/demoB.json +++ b/tests/+did2/fixtures/V_gamma/demoB.json @@ -7,9 +7,9 @@ "class_name": "demoA", "schema": "$DIDSCHEMAPATH/demoA.json" } - ] + ], + "maturity_level": "work_in_progress" }, - "maturity_level": "work_in_progress", "depends_on": [], "file": [], "fields": [ diff --git a/tests/+did2/fixtures/V_gamma/demoC.json b/tests/+did2/fixtures/V_gamma/demoC.json index 5f14438..fed0eab 100644 --- a/tests/+did2/fixtures/V_gamma/demoC.json +++ b/tests/+did2/fixtures/V_gamma/demoC.json @@ -7,9 +7,9 @@ "class_name": "base", "schema": "$DIDSCHEMAPATH/base.json" } - ] + ], + "maturity_level": "work_in_progress" }, - "maturity_level": "work_in_progress", "depends_on": [ { "name": "item1", diff --git a/tests/+did2/fixtures/V_gamma/demoFile.json b/tests/+did2/fixtures/V_gamma/demoFile.json index e3574bc..cea0da6 100644 --- a/tests/+did2/fixtures/V_gamma/demoFile.json +++ b/tests/+did2/fixtures/V_gamma/demoFile.json @@ -7,9 +7,9 @@ "class_name": "base", "schema": "$DIDSCHEMAPATH/base.json" } - ] + ], + "maturity_level": "work_in_progress" }, - "maturity_level": "work_in_progress", "depends_on": [], "file": [ { From 40ab0d9f8ce56f12ea6006d2b983b188fa26b2ea Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 09:51:22 -0400 Subject: [PATCH 29/32] v2: relocate did2 tests under +unittest, reserve +test and +symmetry slots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirrors the legacy +did test layout exactly: tests/+did/+test/+fixture/ → tests/+did2/+test/+fixture/ (placeholder) tests/+did/+test/+helper/ → tests/+did2/+test/+helper/ (placeholder) tests/+did/+unittest/+abstract/ → tests/+did2/+unittest/+abstract/ (placeholder) tests/+did/+unittest/*.m → tests/+did2/+unittest/*.m (real tests) tests_symmetry/+did/+symmetry/+makeArtifacts/ (placeholder) tests_symmetry/+did/+symmetry/+readArtifacts/ (placeholder) Adds: tests/+did2/+unittest/testDocumentScaffold.m Function-based tests for did2.document's surface API. Same content as the previous tests/+did2/testDocumentScaffold.m; docstring updated to advertise the new runtests path: runtests('did2.unittest.testDocumentScaffold') tests/+did2/+unittest/testSchemaCache.m Function-based cache + end-to-end tests (22 tests). `setupOnce` recomputes the fixture path: two fileparts up from mfilename's directory now (was one), since this file moved down one package level. Empty placeholders (.gitkeep with a one-line comment explaining the legacy slot they reserve) under: tests/+did2/+unittest/+abstract/ tests/+did2/+test/+fixture/ tests/+did2/+test/+helper/ tests_symmetry/+did2/+symmetry/+makeArtifacts/ tests_symmetry/+did2/+symmetry/+readArtifacts/ The fixture data directory tests/+did2/fixtures/V_gamma/ stays put — it's a non-package sibling holding static JSON, not a MATLAB package. The old test-file locations (tests/+did2/testDocumentScaffold.m and tests/+did2/testSchemaCache.m) are removed in follow-up commits. CI discovery via TestSuite.fromPackage will land in a separate test-code.yml update so the runner finds the relocated tests under did2.unittest. --- tests/+did2/+test/+fixture/.gitkeep | 4 + tests/+did2/+test/+helper/.gitkeep | 2 + tests/+did2/+unittest/+abstract/.gitkeep | 2 + tests/+did2/+unittest/testDocumentScaffold.m | 65 +++++ tests/+did2/+unittest/testSchemaCache.m | 237 ++++++++++++++++++ .../+did2/+symmetry/+makeArtifacts/.gitkeep | 3 + .../+did2/+symmetry/+readArtifacts/.gitkeep | 3 + 7 files changed, 316 insertions(+) create mode 100644 tests/+did2/+test/+fixture/.gitkeep create mode 100644 tests/+did2/+test/+helper/.gitkeep create mode 100644 tests/+did2/+unittest/+abstract/.gitkeep create mode 100644 tests/+did2/+unittest/testDocumentScaffold.m create mode 100644 tests/+did2/+unittest/testSchemaCache.m create mode 100644 tests_symmetry/+did2/+symmetry/+makeArtifacts/.gitkeep create mode 100644 tests_symmetry/+did2/+symmetry/+readArtifacts/.gitkeep diff --git a/tests/+did2/+test/+fixture/.gitkeep b/tests/+did2/+test/+fixture/.gitkeep new file mode 100644 index 0000000..9a47dfc --- /dev/null +++ b/tests/+did2/+test/+fixture/.gitkeep @@ -0,0 +1,4 @@ +% Placeholder. did2 test fixture helpers go here, matching the +% legacy tests/+did/+test/+fixture/ slot. Static V_gamma JSON +% fixtures live as a sibling at tests/+did2/fixtures/V_gamma/ — +% this MATLAB package is for fixture-helper code, not data files. diff --git a/tests/+did2/+test/+helper/.gitkeep b/tests/+did2/+test/+helper/.gitkeep new file mode 100644 index 0000000..de4d847 --- /dev/null +++ b/tests/+did2/+test/+helper/.gitkeep @@ -0,0 +1,2 @@ +% Placeholder. did2 test helpers go here, matching the legacy +% tests/+did/+test/+helper/ slot. diff --git a/tests/+did2/+unittest/+abstract/.gitkeep b/tests/+did2/+unittest/+abstract/.gitkeep new file mode 100644 index 0000000..0c4787a --- /dev/null +++ b/tests/+did2/+unittest/+abstract/.gitkeep @@ -0,0 +1,2 @@ +% Placeholder. Abstract base test classes for did2.unittest go here, +% matching the legacy tests/+did/+unittest/+abstract/ slot. diff --git a/tests/+did2/+unittest/testDocumentScaffold.m b/tests/+did2/+unittest/testDocumentScaffold.m new file mode 100644 index 0000000..b5ab5cc --- /dev/null +++ b/tests/+did2/+unittest/testDocumentScaffold.m @@ -0,0 +1,65 @@ +function tests = testDocumentScaffold +% testDocumentScaffold - smoke tests for the did2.document scaffold. +% +% Run with: +% results = runtests('did2.unittest.testDocumentScaffold'); +% +% These tests exercise the surface API of did2.document that does not +% depend on the schema cache being fully implemented yet: +% construction from a struct or JSON, dot-path get/set, iterate(), +% and toJSON()/toStruct() round-trips. Tests that depend on the +% schema cache live in did2.unittest.testSchemaCache. + +tests = functiontests(localfunctions); +end + +function testConstructFromStruct(testCase) +s = struct('id', 'abc', 'session_id', 'sess', 'name', 'unit-test'); +doc = did2.document(s); +verifyEqual(testCase, doc.get('id'), 'abc'); +verifyEqual(testCase, doc.get('name'), 'unit-test'); +end + +function testConstructFromJSON(testCase) +jsonText = '{"id":"abc","sample_rate":{"hertz":30000,"approximate":false}}'; +doc = did2.document(jsonText); +verifyEqual(testCase, doc.get('id'), 'abc'); +verifyEqual(testCase, doc.get('sample_rate.hertz'), 30000); +verifyFalse(testCase, doc.get('sample_rate.approximate')); +end + +function testSetCreatesNestedPath(testCase) +doc = did2.document(); +doc.set('app.app_name', 'ndi_app_spikeextractor'); +verifyEqual(testCase, doc.get('app.app_name'), 'ndi_app_spikeextractor'); +end + +function testToJSONRoundTrip(testCase) +s = struct('id', 'abc', 'datestamp', '2026-05-11T00:00:00.000Z'); +doc = did2.document(s); +jsonText = doc.toJSON(); +doc2 = did2.document(jsonText); +verifyEqual(testCase, doc2.toStruct(), s); +end + +function testIterateReturnsStructArray(testCase) +s = struct('axes', struct('name', {'x','y','z'}, 'unit', {'um','um','um'})); +doc = did2.document(s); +elements = doc.iterate('axes'); +verifyEqual(testCase, numel(elements), 3); +verifyEqual(testCase, elements(2).name, 'y'); +end + +function testGetMissingFieldErrors(testCase) +doc = did2.document(struct('id', 'abc')); +verifyError(testCase, @() doc.get('nope.missing'), 'did2:document:missingField'); +end + +function testGetRejectsArrayPath(testCase) +doc = did2.document(struct('axes', struct('name', {'x'}))); +verifyError(testCase, @() doc.get('axes[*].name'), 'did2:document:arrayPathHere'); +end + +function testBadConstructorInput(testCase) +verifyError(testCase, @() did2.document(42), 'did2:document:badInput'); +end diff --git a/tests/+did2/+unittest/testSchemaCache.m b/tests/+did2/+unittest/testSchemaCache.m new file mode 100644 index 0000000..c9b0177 --- /dev/null +++ b/tests/+did2/+unittest/testSchemaCache.m @@ -0,0 +1,237 @@ +function tests = testSchemaCache +% testSchemaCache - exercises did2.schema.cache against the in-repo +% V_gamma fixtures at tests/+did2/fixtures/V_gamma/. Also covers +% did2.document.blank() and did2.document.validate() end-to-end. +% +% Run with: +% results = runtests('did2.unittest.testSchemaCache'); +% +% Documents in V_gamma use a top-level `document_class` header plus +% class-scoped property blocks (see V_gamma_SPEC.md "JSON Format: +% Document Instances"). After V_gamma's "drop underscore prefixes" +% pass, every key in the wire shape is a valid MATLAB struct field +% name, so `jsonencode`/`jsondecode` round-trip without any rewrite. + +tests = functiontests(localfunctions); +end + +function setupOnce(testCase) +% Fixtures live at tests/+did2/fixtures/V_gamma/. This file is at +% tests/+did2/+unittest/testSchemaCache.m, so the fixture directory is +% two `fileparts` levels above mfilename's directory. +thisDir = fileparts(mfilename('fullpath')); +fixtureDir = fullfile(fileparts(thisDir), 'fixtures', 'V_gamma'); +did2.schema.cache.setSchemaPath(fixtureDir); +testCase.TestData.fixtureDir = fixtureDir; +testCase.TestData.cache = did2.schema.cache.shared(); +end + +function teardownOnce(~) +did2.schema.cache.resetSingleton(); +end + +% ---- schema-cache plumbing ---- + +function testSchemaPathPointsAtFixtures(testCase) +verifyTrue(testCase, isfolder(testCase.TestData.fixtureDir)); +verifyEqual(testCase, testCase.TestData.cache.schemaPath, testCase.TestData.fixtureDir); +end + +function testGetClassLoadsBase(testCase) +s = testCase.TestData.cache.getClass('base'); +verifyTrue(testCase, isstruct(s)); +verifyTrue(testCase, isfield(s, 'document_class')); +end + +function testGetClassMissingThrows(testCase) +verifyError(testCase, ... + @() testCase.TestData.cache.getClass('not_a_real_class'), ... + 'did2:schema:missingClass'); +end + +function testCurieRegistryLoaded(testCase) +verifyTrue(testCase, isstruct(testCase.TestData.cache.curieRegistry)); +verifyFalse(testCase, isempty(fieldnames(testCase.TestData.cache.curieRegistry))); +end + +% ---- superclass chains ---- + +function testSuperclassesBaseIsRoot(testCase) +verifyEmpty(testCase, testCase.TestData.cache.superclasses('base')); +end + +function testSuperclassesDemoAExtendsBase(testCase) +verifyEqual(testCase, testCase.TestData.cache.superclasses('demoA'), {'base'}); +end + +function testSuperclassesDemoBChain(testCase) +verifyEqual(testCase, testCase.TestData.cache.superclasses('demoB'), {'demoA', 'base'}); +end + +function testClassChainRootFirst(testCase) +verifyEqual(testCase, testCase.TestData.cache.classChain('demoB'), ... + {'base', 'demoA', 'demoB'}); +end + +% ---- field-list resolution ---- + +function testOwnFieldsBaseHasFour(testCase) +own = testCase.TestData.cache.ownFields('base'); +verifyEqual(testCase, numel(own), 4); +end + +function testOwnFieldsDemoAHasOne(testCase) +own = testCase.TestData.cache.ownFields('demoA'); +verifyEqual(testCase, numel(own), 1); +end + +function testFieldsForTagsDeclaringClass(testCase) +tagged = testCase.TestData.cache.fieldsFor('demoB'); +% base(4) + demoA(1) + demoB(1) = 6 entries +verifyEqual(testCase, numel(tagged), 6); +verifyEqual(testCase, tagged(1).declaringClass, 'base'); +verifyEqual(testCase, tagged(4).declaringClass, 'base'); +verifyEqual(testCase, tagged(5).declaringClass, 'demoA'); +verifyEqual(testCase, tagged(6).declaringClass, 'demoB'); +end + +% ---- buildBlankDocument: document_class header ---- + +function testBuildBlankDocumentHeader(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoB'); +verifyTrue(testCase, isfield(doc, 'document_class')); +verifyEqual(testCase, doc.document_class.class_name, 'demoB'); +verifyEqual(testCase, doc.document_class.class_version, '1.0.0'); +verifyEqual(testCase, numel(doc.document_class.superclasses), 2); +verifyEqual(testCase, doc.document_class.superclasses(1).class_name, 'demoA'); +verifyEqual(testCase, doc.document_class.superclasses(2).class_name, 'base'); +end + +function testBuildBlankDocumentEmptyDependsOn(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'depends_on')); +verifyEmpty(testCase, doc.depends_on); +end + +% ---- buildBlankDocument: class-scoped blocks ---- + +function testBuildBlankDocumentHasBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'base')); +verifyTrue(testCase, isfield(doc.base, 'id')); +verifyTrue(testCase, isfield(doc.base, 'session_id')); +verifyTrue(testCase, isfield(doc.base, 'name')); +verifyTrue(testCase, isfield(doc.base, 'datestamp')); +end + +function testBuildBlankDocumentHasConcreteBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyTrue(testCase, isfield(doc, 'demoA')); +verifyTrue(testCase, isfield(doc.demoA, 'value')); +verifyEqual(testCase, doc.demoA.value, ''); +end + +function testBuildBlankDocumentAllChainBlocksPresent(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoB'); +verifyTrue(testCase, isfield(doc, 'base')); +verifyTrue(testCase, isfield(doc, 'demoA')); +verifyTrue(testCase, isfield(doc, 'demoB')); +verifyTrue(testCase, isfield(doc.demoA, 'value')); +verifyTrue(testCase, isfield(doc.demoB, 'value_b')); +end + +function testBuildBlankDocumentMintsIdInBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyEqual(testCase, numel(doc.base.id), 33); % did_id format length +end + +function testBuildBlankDocumentSetsDatestampInBaseBlock(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyEqual(testCase, doc.base.datestamp(1:2), '20'); +verifyEqual(testCase, doc.base.datestamp(end), 'Z'); +end + +% ---- validateDocument ---- + +function testValidateBlankDocFailsOnEmptySessionId(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:emptyField'); +end + +function testValidatePassesAfterFillingSessionId(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +testCase.TestData.cache.validateDocument(doc); +end + +function testValidateCatchesMaxLength(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = repmat('a', 1, 300); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:maxLength'); +end + +function testValidateAcceptsValueAtMaxLength(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = repmat('a', 1, 256); +testCase.TestData.cache.validateDocument(doc); +end + +function testValidateCatchesTypeMismatch(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc.base.session_id = did.ido.unique_id(); +doc.demoA.value = 12345; +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:typeMismatch'); +end + +function testValidateMissingClassNameThrows(testCase) +doc = struct('base', struct('id', 'abc')); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:missingClassName'); +end + +function testValidateMissingClassBlockThrows(testCase) +doc = testCase.TestData.cache.buildBlankDocument('demoA'); +doc = rmfield(doc, 'base'); +verifyError(testCase, ... + @() testCase.TestData.cache.validateDocument(doc), ... + 'did2:validation:missingClassBlock'); +end + +% ---- end-to-end through did2.document ---- + +function testDocumentBlankConvenience(testCase) +doc = did2.document.blank('demoA'); +verifyEqual(testCase, doc.className(), 'demoA'); +verifyEqual(testCase, doc.classVersion(), '1.0.0'); +verifyEqual(testCase, numel(doc.get('base.id')), 33); +end + +function testDocumentValidateRoundTrip(testCase) +doc = did2.document.blank('demoA'); +doc.set('base.session_id', did.ido.unique_id()); +doc.set('demoA.value', 'hello'); +doc.validate(); +end + +function testDocumentToJSONRoundTrip(testCase) +% V_gamma has no leading-underscore keys, so jsonencode/jsondecode is +% identity for any well-formed document. Confirm the wire shape uses +% the V_gamma key names and re-parses to an equivalent document. +doc = did2.document.blank('demoA'); +text = doc.toJSON(); +verifyTrue(testCase, contains(text, '"document_class"')); +verifyTrue(testCase, contains(text, '"class_name":"demoA"')); +doc2 = did2.document.fromJSON(text); +verifyEqual(testCase, doc2.className(), 'demoA'); +verifyTrue(testCase, isfield(doc2.toStruct(), 'base')); +verifyTrue(testCase, isfield(doc2.toStruct(), 'demoA')); +end diff --git a/tests_symmetry/+did2/+symmetry/+makeArtifacts/.gitkeep b/tests_symmetry/+did2/+symmetry/+makeArtifacts/.gitkeep new file mode 100644 index 0000000..6056c39 --- /dev/null +++ b/tests_symmetry/+did2/+symmetry/+makeArtifacts/.gitkeep @@ -0,0 +1,3 @@ +% Placeholder. did2 symmetry-test makeArtifacts package goes here +% in step 2/3, matching the legacy +% tests_symmetry/+did/+symmetry/+makeArtifacts/ slot. diff --git a/tests_symmetry/+did2/+symmetry/+readArtifacts/.gitkeep b/tests_symmetry/+did2/+symmetry/+readArtifacts/.gitkeep new file mode 100644 index 0000000..9bd0b3f --- /dev/null +++ b/tests_symmetry/+did2/+symmetry/+readArtifacts/.gitkeep @@ -0,0 +1,3 @@ +% Placeholder. did2 symmetry-test readArtifacts package goes here +% in step 2/3, matching the legacy +% tests_symmetry/+did/+symmetry/+readArtifacts/ slot. From 04226f54623f06ea13378174b37a4eef6bd31758 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 09:51:38 -0400 Subject: [PATCH 30/32] v2: remove old top-level tests/+did2/testDocumentScaffold.m Relocated in the previous commit to tests/+did2/+unittest/testDocumentScaffold.m to mirror the legacy +did/+unittest/ layout exactly. Invoke via runtests('did2.unittest.testDocumentScaffold') going forward. --- tests/+did2/testDocumentScaffold.m | 66 ------------------------------ 1 file changed, 66 deletions(-) delete mode 100644 tests/+did2/testDocumentScaffold.m diff --git a/tests/+did2/testDocumentScaffold.m b/tests/+did2/testDocumentScaffold.m deleted file mode 100644 index d85375e..0000000 --- a/tests/+did2/testDocumentScaffold.m +++ /dev/null @@ -1,66 +0,0 @@ -function tests = testDocumentScaffold -% testDocumentScaffold - smoke tests for the did2.document scaffold. -% -% Run with: -% results = runtests('testDocumentScaffold'); -% -% These tests exercise the surface API of did2.document that does not -% depend on the schema cache being fully implemented yet: -% construction from a struct or JSON, dot-path get/set, iterate(), -% and toJSON()/toStruct() round-trips. Tests that depend on the -% schema cache are deferred until did2.schema.cache implements -% buildBlankDocument and validateDocument. - -tests = functiontests(localfunctions); -end - -function testConstructFromStruct(testCase) -s = struct('id', 'abc', 'session_id', 'sess', 'name', 'unit-test'); -doc = did2.document(s); -verifyEqual(testCase, doc.get('id'), 'abc'); -verifyEqual(testCase, doc.get('name'), 'unit-test'); -end - -function testConstructFromJSON(testCase) -jsonText = '{"id":"abc","sample_rate":{"hertz":30000,"approximate":false}}'; -doc = did2.document(jsonText); -verifyEqual(testCase, doc.get('id'), 'abc'); -verifyEqual(testCase, doc.get('sample_rate.hertz'), 30000); -verifyFalse(testCase, doc.get('sample_rate.approximate')); -end - -function testSetCreatesNestedPath(testCase) -doc = did2.document(); -doc.set('app.app_name', 'ndi_app_spikeextractor'); -verifyEqual(testCase, doc.get('app.app_name'), 'ndi_app_spikeextractor'); -end - -function testToJSONRoundTrip(testCase) -s = struct('id', 'abc', 'datestamp', '2026-05-11T00:00:00.000Z'); -doc = did2.document(s); -jsonText = doc.toJSON(); -doc2 = did2.document(jsonText); -verifyEqual(testCase, doc2.toStruct(), s); -end - -function testIterateReturnsStructArray(testCase) -s = struct('axes', struct('name', {'x','y','z'}, 'unit', {'um','um','um'})); -doc = did2.document(s); -elements = doc.iterate('axes'); -verifyEqual(testCase, numel(elements), 3); -verifyEqual(testCase, elements(2).name, 'y'); -end - -function testGetMissingFieldErrors(testCase) -doc = did2.document(struct('id', 'abc')); -verifyError(testCase, @() doc.get('nope.missing'), 'did2:document:missingField'); -end - -function testGetRejectsArrayPath(testCase) -doc = did2.document(struct('axes', struct('name', {'x'}))); -verifyError(testCase, @() doc.get('axes[*].name'), 'did2:document:arrayPathHere'); -end - -function testBadConstructorInput(testCase) -verifyError(testCase, @() did2.document(42), 'did2:document:badInput'); -end From 4d6839acef24f495fdef85e3ca7ef85e66169986 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 09:51:45 -0400 Subject: [PATCH 31/32] v2: remove old top-level tests/+did2/testSchemaCache.m Relocated in the prior batch commit to tests/+did2/+unittest/testSchemaCache.m, with the fixture-path helper updated to walk one extra fileparts level so it still resolves to tests/+did2/fixtures/V_gamma/. Invoke via runtests('did2.unittest.testSchemaCache'). --- tests/+did2/testSchemaCache.m | 230 ---------------------------------- 1 file changed, 230 deletions(-) delete mode 100644 tests/+did2/testSchemaCache.m diff --git a/tests/+did2/testSchemaCache.m b/tests/+did2/testSchemaCache.m deleted file mode 100644 index 8a86d17..0000000 --- a/tests/+did2/testSchemaCache.m +++ /dev/null @@ -1,230 +0,0 @@ -function tests = testSchemaCache -% testSchemaCache - exercises did2.schema.cache against the in-repo -% V_gamma fixtures at tests/+did2/fixtures/V_gamma/. Also covers -% did2.document.blank() and did2.document.validate() end-to-end. -% -% Documents in V_gamma use a top-level `document_class` header plus -% class-scoped property blocks (see V_gamma_SPEC.md "JSON Format: -% Document Instances"). After V_gamma's "drop underscore prefixes" -% pass, every key in the wire shape is a valid MATLAB struct field -% name, so `jsonencode`/`jsondecode` round-trip without any rewrite. - -tests = functiontests(localfunctions); -end - -function setupOnce(testCase) -fixtureDir = fullfile(fileparts(mfilename('fullpath')), 'fixtures', 'V_gamma'); -did2.schema.cache.setSchemaPath(fixtureDir); -testCase.TestData.fixtureDir = fixtureDir; -testCase.TestData.cache = did2.schema.cache.shared(); -end - -function teardownOnce(~) -did2.schema.cache.resetSingleton(); -end - -% ---- schema-cache plumbing ---- - -function testSchemaPathPointsAtFixtures(testCase) -verifyTrue(testCase, isfolder(testCase.TestData.fixtureDir)); -verifyEqual(testCase, testCase.TestData.cache.schemaPath, testCase.TestData.fixtureDir); -end - -function testGetClassLoadsBase(testCase) -s = testCase.TestData.cache.getClass('base'); -verifyTrue(testCase, isstruct(s)); -verifyTrue(testCase, isfield(s, 'document_class')); -end - -function testGetClassMissingThrows(testCase) -verifyError(testCase, ... - @() testCase.TestData.cache.getClass('not_a_real_class'), ... - 'did2:schema:missingClass'); -end - -function testCurieRegistryLoaded(testCase) -verifyTrue(testCase, isstruct(testCase.TestData.cache.curieRegistry)); -verifyFalse(testCase, isempty(fieldnames(testCase.TestData.cache.curieRegistry))); -end - -% ---- superclass chains ---- - -function testSuperclassesBaseIsRoot(testCase) -verifyEmpty(testCase, testCase.TestData.cache.superclasses('base')); -end - -function testSuperclassesDemoAExtendsBase(testCase) -verifyEqual(testCase, testCase.TestData.cache.superclasses('demoA'), {'base'}); -end - -function testSuperclassesDemoBChain(testCase) -verifyEqual(testCase, testCase.TestData.cache.superclasses('demoB'), {'demoA', 'base'}); -end - -function testClassChainRootFirst(testCase) -verifyEqual(testCase, testCase.TestData.cache.classChain('demoB'), ... - {'base', 'demoA', 'demoB'}); -end - -% ---- field-list resolution ---- - -function testOwnFieldsBaseHasFour(testCase) -own = testCase.TestData.cache.ownFields('base'); -verifyEqual(testCase, numel(own), 4); -end - -function testOwnFieldsDemoAHasOne(testCase) -own = testCase.TestData.cache.ownFields('demoA'); -verifyEqual(testCase, numel(own), 1); -end - -function testFieldsForTagsDeclaringClass(testCase) -tagged = testCase.TestData.cache.fieldsFor('demoB'); -% base(4) + demoA(1) + demoB(1) = 6 entries -verifyEqual(testCase, numel(tagged), 6); -verifyEqual(testCase, tagged(1).declaringClass, 'base'); -verifyEqual(testCase, tagged(4).declaringClass, 'base'); -verifyEqual(testCase, tagged(5).declaringClass, 'demoA'); -verifyEqual(testCase, tagged(6).declaringClass, 'demoB'); -end - -% ---- buildBlankDocument: document_class header ---- - -function testBuildBlankDocumentHeader(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoB'); -verifyTrue(testCase, isfield(doc, 'document_class')); -verifyEqual(testCase, doc.document_class.class_name, 'demoB'); -verifyEqual(testCase, doc.document_class.class_version, '1.0.0'); -verifyEqual(testCase, numel(doc.document_class.superclasses), 2); -verifyEqual(testCase, doc.document_class.superclasses(1).class_name, 'demoA'); -verifyEqual(testCase, doc.document_class.superclasses(2).class_name, 'base'); -end - -function testBuildBlankDocumentEmptyDependsOn(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyTrue(testCase, isfield(doc, 'depends_on')); -verifyEmpty(testCase, doc.depends_on); -end - -% ---- buildBlankDocument: class-scoped blocks ---- - -function testBuildBlankDocumentHasBaseBlock(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyTrue(testCase, isfield(doc, 'base')); -verifyTrue(testCase, isfield(doc.base, 'id')); -verifyTrue(testCase, isfield(doc.base, 'session_id')); -verifyTrue(testCase, isfield(doc.base, 'name')); -verifyTrue(testCase, isfield(doc.base, 'datestamp')); -end - -function testBuildBlankDocumentHasConcreteBlock(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyTrue(testCase, isfield(doc, 'demoA')); -verifyTrue(testCase, isfield(doc.demoA, 'value')); -verifyEqual(testCase, doc.demoA.value, ''); -end - -function testBuildBlankDocumentAllChainBlocksPresent(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoB'); -verifyTrue(testCase, isfield(doc, 'base')); -verifyTrue(testCase, isfield(doc, 'demoA')); -verifyTrue(testCase, isfield(doc, 'demoB')); -verifyTrue(testCase, isfield(doc.demoA, 'value')); -verifyTrue(testCase, isfield(doc.demoB, 'value_b')); -end - -function testBuildBlankDocumentMintsIdInBaseBlock(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyEqual(testCase, numel(doc.base.id), 33); % did_id format length -end - -function testBuildBlankDocumentSetsDatestampInBaseBlock(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyEqual(testCase, doc.base.datestamp(1:2), '20'); -verifyEqual(testCase, doc.base.datestamp(end), 'Z'); -end - -% ---- validateDocument ---- - -function testValidateBlankDocFailsOnEmptySessionId(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -verifyError(testCase, ... - @() testCase.TestData.cache.validateDocument(doc), ... - 'did2:validation:emptyField'); -end - -function testValidatePassesAfterFillingSessionId(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -doc.base.session_id = did.ido.unique_id(); -testCase.TestData.cache.validateDocument(doc); -end - -function testValidateCatchesMaxLength(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -doc.base.session_id = did.ido.unique_id(); -doc.demoA.value = repmat('a', 1, 300); -verifyError(testCase, ... - @() testCase.TestData.cache.validateDocument(doc), ... - 'did2:validation:maxLength'); -end - -function testValidateAcceptsValueAtMaxLength(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -doc.base.session_id = did.ido.unique_id(); -doc.demoA.value = repmat('a', 1, 256); -testCase.TestData.cache.validateDocument(doc); -end - -function testValidateCatchesTypeMismatch(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -doc.base.session_id = did.ido.unique_id(); -doc.demoA.value = 12345; -verifyError(testCase, ... - @() testCase.TestData.cache.validateDocument(doc), ... - 'did2:validation:typeMismatch'); -end - -function testValidateMissingClassNameThrows(testCase) -doc = struct('base', struct('id', 'abc')); -verifyError(testCase, ... - @() testCase.TestData.cache.validateDocument(doc), ... - 'did2:validation:missingClassName'); -end - -function testValidateMissingClassBlockThrows(testCase) -doc = testCase.TestData.cache.buildBlankDocument('demoA'); -doc = rmfield(doc, 'base'); -verifyError(testCase, ... - @() testCase.TestData.cache.validateDocument(doc), ... - 'did2:validation:missingClassBlock'); -end - -% ---- end-to-end through did2.document ---- - -function testDocumentBlankConvenience(testCase) -doc = did2.document.blank('demoA'); -verifyEqual(testCase, doc.className(), 'demoA'); -verifyEqual(testCase, doc.classVersion(), '1.0.0'); -verifyEqual(testCase, numel(doc.get('base.id')), 33); -end - -function testDocumentValidateRoundTrip(testCase) -doc = did2.document.blank('demoA'); -doc.set('base.session_id', did.ido.unique_id()); -doc.set('demoA.value', 'hello'); -doc.validate(); -end - -function testDocumentToJSONRoundTrip(testCase) -% V_gamma has no leading-underscore keys, so jsonencode/jsondecode is -% identity for any well-formed document. Confirm the wire shape uses -% the V_gamma key names and re-parses to an equivalent document. -doc = did2.document.blank('demoA'); -text = doc.toJSON(); -verifyTrue(testCase, contains(text, '"document_class"')); -verifyTrue(testCase, contains(text, '"class_name":"demoA"')); -doc2 = did2.document.fromJSON(text); -verifyEqual(testCase, doc2.className(), 'demoA'); -verifyTrue(testCase, isfield(doc2.toStruct(), 'base')); -verifyTrue(testCase, isfield(doc2.toStruct(), 'demoA')); -end From 6e52dfd20d251ec76106aeb0646151851be65457 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Tue, 12 May 2026 09:52:07 -0400 Subject: [PATCH 32/32] ci(test-code): discover did2 tests via TestSuite.fromPackage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit did2 self-tests moved into the tests/+did2/+unittest/ MATLAB package to mirror the legacy +did/+unittest/ layout. TestSuite.fromFolder does not recurse into +package directories, so switch to TestSuite.fromPackage("did2.unittest", "IncludingSubpackages", true) — same pattern test-symmetry.yml already uses for did.symmetry.*. The IncludingSubpackages flag also picks up future +abstract base classes when they land under tests/+did2/+unittest/+abstract/. --- .github/workflows/test-code.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-code.yml b/.github/workflows/test-code.yml index 95506e3..30fb99f 100644 --- a/.github/workflows/test-code.yml +++ b/.github/workflows/test-code.yml @@ -69,7 +69,12 @@ jobs: import matlab.unittest.TestRunner; import matlab.unittest.TestSuite; runner = TestRunner.withTextOutput; - suite = TestSuite.fromFolder("tests", "IncludingSubfolders", true); + % did2 self-tests live under tests/+did2/+unittest/ now (the + % legacy +did/+unittest/ layout). TestSuite.fromPackage with + % IncludingSubpackages=true picks up the function-based test + % files and any future +abstract base classes. Mirrors the + % discovery pattern in test-symmetry.yml. + suite = TestSuite.fromPackage("did2.unittest", "IncludingSubpackages", true); results = runner.run(suite); disp(table(results)); nFailed = sum([results.Failed]);