Summary
As a data engineer loading PDS4 labels into the registry, I want Harvest to correctly extract and index XML attributes from PDS4 elements (e.g., xsi:nil="true", nilReason, unit, and other PDS4-defined attributes), so that the registry faithfully captures all metadata encoded in XML attributes rather than silently dropping them.
Motivation
The current AutogenExtractor walks the XML DOM and for each leaf element calls getTextContent() to extract the field value. XML attributes are never read. This means any metadata encoded as an XML attribute — not just element text content — is silently dropped during harvest. Examples include:
xsi:nil="true" and nilReason on nillable elements (e.g., <pds:stop_date_time xsi:nil="true" nilReason="unknown"/>) — indexed as an empty string "", indistinguishable from a genuinely blank value.unit on measurement elements (e.g., <pds:length unit="km">1200</pds:length>) — the unit is lost.- Any other PDS4 discipline dictionary or mission dictionary attributes on elements.
Downstream consumers (API, search clients) have no way to access or search this information.
This is directly relevant to the PDS4 CCB discussion in NASA-PDS/PDS4-CCB#95 regarding how nil attributes and empty classes should be handled, but the broader issue of XML attribute indexing applies beyond nil alone.
Proposed Solution
When AutogenExtractor (or equivalent XML extraction logic) processes a leaf element:
- In addition to extracting element text content, read any XML attributes present on the element.
- Index each attribute as a companion field, e.g.,
<fieldName>/<attrName> or <fieldName>__<attrName> (naming convention TBD during design).
- For
xsi:nil="true", also store a clear sentinel or boolean so consumers can distinguish nil-by-declaration from empty/absent.
The exact field naming convention and OpenSearch representation should be determined during implementation in coordination with the registry-api team to ensure consistent searchability.
Additional Context
🦄 Related requirements
NASA-PDS/registry-api#789
For Internal Dev Team To Complete
⚙️ Engineering Details
To be completed by engineering team
🎉 Integration & Test
To be completed by engineering team
Summary
As a data engineer loading PDS4 labels into the registry, I want Harvest to correctly extract and index XML attributes from PDS4 elements (e.g.,
xsi:nil="true",nilReason,unit, and other PDS4-defined attributes), so that the registry faithfully captures all metadata encoded in XML attributes rather than silently dropping them.Motivation
The current
AutogenExtractorwalks the XML DOM and for each leaf element callsgetTextContent()to extract the field value. XML attributes are never read. This means any metadata encoded as an XML attribute — not just element text content — is silently dropped during harvest. Examples include:xsi:nil="true"andnilReasonon nillable elements (e.g.,<pds:stop_date_time xsi:nil="true" nilReason="unknown"/>) — indexed as an empty string"", indistinguishable from a genuinely blank value.uniton measurement elements (e.g.,<pds:length unit="km">1200</pds:length>) — the unit is lost.Downstream consumers (API, search clients) have no way to access or search this information.
This is directly relevant to the PDS4 CCB discussion in NASA-PDS/PDS4-CCB#95 regarding how nil attributes and empty classes should be handled, but the broader issue of XML attribute indexing applies beyond nil alone.
Proposed Solution
When
AutogenExtractor(or equivalent XML extraction logic) processes a leaf element:<fieldName>/<attrName>or<fieldName>__<attrName>(naming convention TBD during design).xsi:nil="true", also store a clear sentinel or boolean so consumers can distinguish nil-by-declaration from empty/absent.The exact field naming convention and OpenSearch representation should be determined during implementation in coordination with the registry-api team to ensure consistent searchability.
Additional Context
AutogenExtractor.processLeafNode()inregistry-common🦄 Related requirements
NASA-PDS/registry-api#789
For Internal Dev Team To Complete
⚙️ Engineering Details
To be completed by engineering team
🎉 Integration & Test
To be completed by engineering team