When a TEI manuscript file's nested item/part identifier ( or ) is edited, the old identifier's previously-minted canonical Work is never suppressed. Because TeiTransformer rebuilds a manuscript's internalWorkStubs from scratch on every reprocess, the old id simply stops appearing in new output — but nothing tells the id_minter or the search index that it's gone. The old "child" record lingers forever alongside the new one, so a manuscript that should show 1 child/part shows 2 (or more, if it happens repeatedly).
This is a distinct issue from the top-level path/id rename bug — that one involved tei_updater/tei_id_extractor's path‑tracking. This one has nothing to do with that machinery at all: the top-level file path and top-level xml:id never change, so PathIdTable/tei_updater never see anything happen. The gap is entirely in the transformer + id_minter layer, which has no deletion path whatsoever for internal-work identifiers — unlike the top-level path/id case, this isn't even a race condition or an edge case being missed; there is no delete mechanism here at all, by design.
Example wonky record
Spanish/MS_Amer_8.xml (top-level xml:id="MS_Amer_8", unchanged throughout):
|
Commit 80ca5eb9 (before) |
Current |
top-level <TEI xml:id=...> |
MS_Amer_8 |
MS_Amer_8 (unchanged) |
<msItem xml:id=...> |
Amer_8_1 |
MS_Amer_8_1 (renamed) |
The item was renamed from Amer_8_1 → MS_Amer_8_1 on 2026-07-01. Amer_8_1's canonical Work (minted at some point while that content was live) should have been suppressed once the rename landed and the transformer re-ran — instead it's still sitting in the catalogue as a phantom extra child, alongside the correctly-current MS_Amer_8_1.
Check steps (to confirm and to find other affected records)
- Confirm the orphan exists — query the id_minter's ground-truth table for the old identifier:
SELECT CanonicalId FROM identifiers.identifiers
WHERE SourceSystem = 'tei-manuscript-id' AND SourceId = 'Amer_8_1' AND OntologyType = 'Work';
- A returned CanonicalId confirms the orphan. Cross-check it's Visible (not already Deleted) via GET works-identified-/_doc/.
- To find other affected manuscripts at scale: for each TEI file in the source repo, diff the set of msItem/msPart xml:ids in the current file against every historical version's set of ids (via the GitHub commits API, repos/wellcomecollection/wellcome-collection-tei/commits?path=, walking each revision and extracting ids the same way TeiOps.getIdFrom does — root plus every msItem/msPart xml:id under msDesc). Any id that appears in an old revision but not in the current one is a candidate orphan — confirm each via the identifiers table query above.
- Reference implementation for id extraction to reuse in a scan script: TeiNestedData.nestedTeiDataFromItems (pipeline/transformer/transformer_tei/src/main/scala/weco/pipeline/transformer/tei/transformers/TeiNestedData.scala:97‑141) for single-part manuscripts (msItem), and TeiNestedData.nestedTeiDataFromParts (same file, lines 54‑92) for multi-part manuscripts (msPart).
Root cause (code references)
- TeiTransformer.handleTeiChange (pipeline/transformer/transformer_tei/src/main/scala/weco/pipeline/transformer/tei/TeiTransformer.scala:44‑55) fully re-parses the XML on every change and builds a fresh Work.Visible[Source].
- TeiData.toWork's iterateNestedData (TeiData.scala:49‑65) rebuilds internalWorkStubs entirely from the current parse — nothing is diffed against the previous version's internal work ids.
- The id_minter (catalogue_graph/src/id_minter/) is purely additive — it mints canonical IDs for any SourceIdentifier it sees referenced (including nested internalWorkStubs[].sourceIdentifier), but has no concept of "this identifier used to be referenced by this parent and no longer is, so suppress it."
- Contrast with the top-level file case, where at least an attempted (if fragile) mechanism exists (PathIdManager's delete-on-path-change logic, tei_adapter/tei_id_extractor/.../PathIdManager.scala) — for internal works, no equivalent was ever built.
Suggested fix direction
Some form of "internal work reconciliation" is needed at transform time: when reprocessing a TEI file, diff the newly-parsed set of internal-work SourceIdentifiers against whichever ones were part of the previous version of that Work (available from the previous Work.Visible[Source] in the pipeline storage, or a small persisted "known internal ids per top-level id" table analogous to PathIdTable), and emit Work.Deleted for any that have disappeared. Given the id_minter and transformer are being actively ported to Python (catalogue_graph/src/id_minter/, catalogue_graph/src/adapters/transformers/), this might be a natural point to build the reconciliation logic properly rather than porting the current gap forward.
Impact
Likely affects every manuscript that has ever had an item/part xml:id relabelled (cataloguing corrections, encoding standardisation passes like the "Made edits to file names and xml IDs" / "Edited xml:ids" commits we found are common in this repo's history) — potentially a non-trivial number of orphaned "phantom child" records across the TEI collection, not just this one manuscript.
When a TEI manuscript file's nested item/part identifier ( or ) is edited, the old identifier's previously-minted canonical Work is never suppressed. Because TeiTransformer rebuilds a manuscript's internalWorkStubs from scratch on every reprocess, the old id simply stops appearing in new output — but nothing tells the id_minter or the search index that it's gone. The old "child" record lingers forever alongside the new one, so a manuscript that should show 1 child/part shows 2 (or more, if it happens repeatedly).
This is a distinct issue from the top-level path/id rename bug — that one involved tei_updater/tei_id_extractor's path‑tracking. This one has nothing to do with that machinery at all: the top-level file path and top-level xml:id never change, so PathIdTable/tei_updater never see anything happen. The gap is entirely in the transformer + id_minter layer, which has no deletion path whatsoever for internal-work identifiers — unlike the top-level path/id case, this isn't even a race condition or an edge case being missed; there is no delete mechanism here at all, by design.
Example wonky record
Spanish/MS_Amer_8.xml (top-level xml:id="MS_Amer_8", unchanged throughout):
80ca5eb9(before)<TEI xml:id=...>MS_Amer_8MS_Amer_8(unchanged)<msItem xml:id=...>Amer_8_1MS_Amer_8_1(renamed)The item was renamed from Amer_8_1 → MS_Amer_8_1 on 2026-07-01. Amer_8_1's canonical Work (minted at some point while that content was live) should have been suppressed once the rename landed and the transformer re-ran — instead it's still sitting in the catalogue as a phantom extra child, alongside the correctly-current MS_Amer_8_1.
Check steps (to confirm and to find other affected records)
SELECT CanonicalId FROM identifiers.identifiers
WHERE SourceSystem = 'tei-manuscript-id' AND SourceId = 'Amer_8_1' AND OntologyType = 'Work';
Root cause (code references)
Suggested fix direction
Some form of "internal work reconciliation" is needed at transform time: when reprocessing a TEI file, diff the newly-parsed set of internal-work SourceIdentifiers against whichever ones were part of the previous version of that Work (available from the previous Work.Visible[Source] in the pipeline storage, or a small persisted "known internal ids per top-level id" table analogous to PathIdTable), and emit Work.Deleted for any that have disappeared. Given the id_minter and transformer are being actively ported to Python (catalogue_graph/src/id_minter/, catalogue_graph/src/adapters/transformers/), this might be a natural point to build the reconciliation logic properly rather than porting the current gap forward.
Impact
Likely affects every manuscript that has ever had an item/part xml:id relabelled (cataloguing corrections, encoding standardisation passes like the "Made edits to file names and xml IDs" / "Edited xml:ids" commits we found are common in this repo's history) — potentially a non-trivial number of orphaned "phantom child" records across the TEI collection, not just this one manuscript.