RFC 089: Identifiers API

[kenoir]

Preview

View the rendered RFC on this branch:

• RFC 089 README
• Rendered API contract (openapi.md) · openapi.yaml

---

What does this change?

The proposal is a small, read-only Identifiers API. Wellcome Collection gives every catalogue thing (a work, an image, an item) a stable public "canonical" id, and keeps a registry recording which underlying source ids that canonical id was built from. This API does one job: given a canonical id it returns the source id(s) behind it, and given a source id it returns the canonical id (optionally with its siblings). It only ever reads that registry; it never creates or changes ids.

It exists because of the Sierra/CALM to FOLIO/Axiell migration. As records move between systems a single canonical id accumulates several source ids over time (an original plus inherited "predecessor" aliases), and a couple of internal services sit right at the edges where that translation has to happen: the IIIF viewer needs to turn old b-numbers and CALM refs into the canonical id it presents under, and requesting needs to turn a canonical item id into the FOLIO UUID a hold is placed on, and back again. The guiding principle is that everything public speaks canonical and source ids only appear at those two edges (ingest and the FOLIO boundary). Rather than have each consumer re-derive the mapping or query the catalogue by source id, this API is the single shared place that translation lives. Because the main running cost is database queries, it is also the natural place to cache aggressively (at the edge, to keep requests off the database) and to attribute that database cost to the consumers driving it.

How it relates to the other RFCs:

• It reads the ID Registry defined in RFC 083 (stable identifiers). RFC 083 owns the data and all the writes; this API is just a read-only window onto it.
• It serves the IIIF/DDS lookup that RFC 085 (IIIF identities, open PR RFC 085: IIIF identities #143) describes, where the canonical Work id becomes the IIIF URI and older identifier forms redirect to it.
• It is the concrete "service" answer to the open question in RFC 088 (the Sierra to FOLIO identity/requesting migration, open PR RFC 088: Migrating identity, requesting and items APIs from Sierra to FOLIO #153) about how requesting should translate item ids. RFC 088 left that access mechanism open (a direct database read, a sync, or a service); this RFC proposes the service and is for RFC 088 to ratify.

The RFC is written to stand on its own and carries the API contract alongside it, so it can be reviewed without access to the closed discovery/prototype repository where the working prototype lives.

Files added:

• rfcs/089-identifiers-api/README.md: the RFC document.
• rfcs/README.md: refreshed RFC listing table (RFC 089 row added).
• rfcs/089-identifiers-api/openapi.yaml: the OpenAPI 3.0 spec for the two lookup operations (the source of truth).
• rfcs/089-identifiers-api/openapi.md: generated human-readable rendering of the spec, browsable on GitHub without a Swagger/Redoc renderer.
• render_docs.py, pyproject.toml, .python-version, .gitignore, uv.lock: a self-contained uv project that validates openapi.yaml and regenerates openapi.md.

How to test

• Read rfcs/089-identifiers-api/README.md and review the contract, architecture, caching strategy and open questions.
• Confirm the RFC passes repo validation: .scripts/validate_rfc.py.
• Confirm the listing table is in sync: .scripts/create_table_summary.py --check-readme.
• (Optional) Regenerate the rendered contract and confirm no diff: from the RFC directory, uv run python render_docs.py: this validates openapi.yaml against the OpenAPI spec validator and rewrites openapi.md.

How can we measure success?

No measurable runtime success criteria; this is a documentation RFC. Success is the RFC being reviewed and providing a clear, self-contained contract and architecture that the team can align on, and a decision on whether this service is the access mechanism for identifier translation in RFC 088.

Have we considered potential risks?

• Documentation only; no production code or infrastructure is changed by this PR, so there is no runtime or deployment risk.
• The design risks themselves (the caching strategy and the database cost it controls, the unmet FOLIO-item ingestion dependency, item canonical-id stability) are enumerated in the RFC's Open questions section and are intended to be the subject of review.
• The OpenAPI spec is validated and the rendered Markdown is generated from it, reducing the chance of the contract and its human-readable rendering drifting apart.

[tomcrane]

I can see the secondary index argument against supporting this, but for a consumer, it's not as clean as it could be. Consider an incoming IIIF manifest request to the DDS (iiif-builder):

iiif.wellcomelibrary.org/presentation/{identifier}

What does the DDS do with this? We have no other context at this point, we don't know what source system {identifier} corresponds to. This is the first point of contact.

The DDS current behaviour has lots of knowledge of what identifiers look like and how they work. First, we see if it's a b number, or a permitted variation on a b number. If the identifier is a permitted b number variant form, we normalise it to the standard form:

// 7-digit form:
https://iiif.wellcomecollection.org/presentation/b1803597
// calculate missing checksum (8) and redirect to https://iiif.wellcomecollection.org/presentation/b18035978

This was needed pre wc.org, when there were generated links from the Sierra catalogue to wellcomelibrary.org viewer pages using the pure numerical id 1803597. Now the Sierra catalogue is kept hidden from the public, so 7-digit links are unlikely, but the code still supports them.

We then see if we have the manifest stored under this key in S3. If so, great, we return it.

If it's not in S3 at the normalised b number form, or if it wasn't even a b number, then the DDS queries its Manifestations table, populated when the item was processed: information about its identities that DDS hangs on to but is not authoritative for. This allows us to handle things like:

https://iiif.wellcomecollection.org/presentation/zjytxny8
// redirect to https://iiif.wellcomecollection.org/presentation/b18035978

https://iiif.wellcomecollection.org/presentation/dpv7wsth
// redirect to https://iiif.wellcomecollection.org/presentation/b19995271

https://iiif.wellcomecollection.org/presentation/SB/1/1/298
// redirect to https://iiif.wellcomecollection.org/presentation/b19995271
// Note these last two are the same redirect, both work ID and Calm Ref redirect to the b number it was digitised under

In the future, the workID form becomes the canonical one, and we will redirect all other forms to that.

If we have a bare-value lookup - "tell us what you know about 'b19995271'" - then the DDS can defer to an authority for the identity. If not, we'll still have to do some tricks locally.

When we see b19995271 or b1999527 for {identity}, we know it's a b number and not a work id, because we can run it through the code linked above. But DDS has to retain that b-number-aware code to do so. And then, it can specifically ask for

/v1/identifiers/by-source/sierra-system-number/b19995271?type=Work

or if we see SB/1/1/298 for {identity}, the presence of / characters may trigger us to ask for:

/v1/identifiers/by-source/calm-ref-number/SB/1/1/298?type=Work
// identity service needs to handle the unbounded trailing {**path} elements!

... and if the worst came to the worst, we could ask for each source system in turn to see if we have a match. The more identifier format knowledge we retain in the DDS, the more likely we ask for the right source system on the first attempt.

Consider the string b3288596

Is this a work ID or is it a b number? It's actually valid under both schemes. It matches the 8-character work ID reduced alphanumeric set, AND expands to a valid b+8-digits b number:

iiif.wellcomecollection.org/presentation/b3288596

We don't know if we need to redirect or or not, until we've made a call to the identity service to resolve it.

How many work IDs are there that are a b followed by 7 digits? Probably not many. And it's even more unlikely that any of them are real Sierra b numbers. And if there are any of those, it's astronomically unlikely to be the same work!

The question is, how much knowledge of identifier forms do we want DDS to retain? It could defer everything to the identity service (with a short term memory cache), always resolving {identity} using the service and never deducing anything about it. If on receipt of the query results, {identity} turns out not to have been the canonical form, then redirect to the canonical form. This reduces the knowledge the DDS has, AND works for any future identifier scheme, but at the expense of that secondary index!

[kenoir]

The question is, how much knowledge of identifier forms do we want DDS to retain?

Some, and I think the floor is set by the data rather than by this API. The "defer everything
to this service" option doesn't quite work at this level, even if we paid for the secondary
index, for two reasons.

First, normalisation stays with the DDS either way. The registry only holds the forms the
minter minted from: b18035978 is a row, 1803597 and b1803597 are not. So the
checksum/variant handling in WellcomeLibraryIdentifiers.cs has to run before any lookup,
bare or qualified. A bare-value endpoint would remove the need to guess the source system,
not the need to normalise.

Second, a bare-value lookup can enumerate the ambiguity but it can't resolve it, and the
ambiguous case is routine rather than rare. The registry is pre-merge: as @agnesgaroux pointed out
above, every digitised b-number sits under both sierra-system-number and mets, resolving
to two different canonical ids (the public work and the invisible METS work), and both rows
are type=Work so the type filter doesn't split them. Which of those canonical ids is the
public one is decided by the merger, and the registry doesn't know that, so this API has no
basis to rank the candidates. "Tell me what you know about b19995271" comes back as two
labelled matches and the DDS still holds a precedence rule (prefer sierra-system-number
over mets). We'd have swapped format-parsing for a selection policy, plus the index. And
the b3288596 collision straddles the canonical and source id spaces, so a bare source
lookup doesn't settle it either; the same policy decides.