feat(ref): PMID is a parked variant, so PubMed-identified entries are silently skipped — and life sciences is the market where that matters

[sotashimozono]

Filed after checking #474 — this is the one source-layer gap that survey does not cover, because it is an identifier gap rather than a source gap.

What is missing

Ref::Pmid does not exist. It is parked, deliberately and in writing:

• crates/doiget-core/src/refs.rs:17 — "Identifier-pick priority per ADR-0030 D3: doi > arxiv > pmid"
• crates/doiget-core/src/refs.rs:18 — "PMID adapter parking until the Ref::Pmid variant lands"
• crates/doiget-core/src/refs.rs:233 — "(PMID parking — Ref::Pmid not yet defined …)"
• crates/doiget-core/src/canonical.rs:22 — future variants "(Pmid, Handle, …)" are anticipated by the canonical-class contract but not implemented

ADR-0030 D3 already decided PMID's priority. Nothing implemented its existence.

The consequence is a silent skip

crates/doiget-core/src/refs.rs:406: an entry whose only identifier is a PMID — eprinttype = {pubmed} — "is skipped rather than" parsed. A bibliography of biomedical references therefore yields fewer entries than it contains, and nothing says which ones went missing or why.

That is the same failure class as #476, #441, #458 and the arXiv claim in #493: the code is correct about what it does, and the user is never told what it did not do. A .bib exported from PubMed is the single most common bibliography shape in the life sciences, and against doiget it silently under-resolves.

Why this is worth doing before the rest of #474

#474 argues, correctly for its time, that adding sources to a chain with a known skip bug (#458) and no end-to-end harness (#462) multiplies the #442 defect class. That argument still holds for sources. It does not apply here, for two reasons:

1. This is a Ref variant plus a resolver, not a new leg on the Tier-2/Tier-3 chain. It does not widen the surface bug(tdm): the Tier-3 chain is skipped whenever Crossref answers, so a TDM source can never close the fetch gap it was added for #458 and test: an end-to-end use-case suite driven through the real entry points, because every "unreachable source" bug passed its unit tests #462 are about.
2. Europe PMC is already integrated (feat(source): Europe PMC — OA full text that Unpaywall does not index #415) and already resolves much of the corpus. What is missing is the identifier, so a large part of the work is routing PMIDs to a source that exists.

Proposed shape

• Ref::Pmid(...) and Ref::Pmcid(...) as canonical classes, with the "pmid" / "pmcid" class strings reserved per the canonical.rs:22 contract and a safekey story that matches ADR-0007.
• NCBI E-utilities (eutils.ncbi.nlm.nih.gov) as the PMID → DOI / metadata resolver. Free, no key required; an optional free key raises the documented rate from 3 to 10 requests/second. None of that is verified against NCBI's own documentation — per survey(source): trustworthy sources not yet covered — preprint servers, physics indexes, regional platforms #474's opening rule and the lesson of test(tdm-ieee): the unexpected-shape fixture is a JSON guess; IEEE actually answers text/xml, and the 403 never reaches the JSON path at all #460, check the endpoint, auth shape, response shape, rate limits and terms against the primary source before writing code. NCBI's usage policy also asks for a tool name and contact email on every request, which maps onto the existing polite-pool plumbing.
• Once a PMID resolves to a DOI, the existing chain takes over unchanged; PMC full text routes through Europe PMC, which is already there.
• The eprinttype = {pubmed} path at refs.rs:406 stops skipping, and whatever it still cannot resolve says so.

Not in scope here

The rest of #474's candidate list stays in #474. Two of its entries are additionally claimed by #461 (BASE, Zenodo) and one by #463 (ACM), and this issue deliberately does not touch any of them.

Sources named in paper-search-mcp but absent from both #474 and here — SSRN, IACR ePrint, CiteSeerX, chemRxiv beyond #474's mention — are lower value per integration and belong as additions to the #474 table, not as separate issues.

Refs #474, #415, #462, ADR-0030, ADR-0007.

[sotashimozono]

Status check while working through the open issues.

Confirmed: this is not blocked by #474's sequencing

This issue's argument holds and I checked it against the code. Ref::Pmid does not exist, refs.rs:406 skips an entry whose only identifier is a PMID, and the skip is silent — a .bib exported from PubMed yields fewer entries than it contains with nothing saying which went missing. That is the #476 / #441 / #458 class again, on the identifier layer.

It is a Ref variant plus a resolver, not a new leg on the Tier-2/Tier-3 chain, so it does not widen the surface #462 is about. I have said so on #474 as well.

Not implemented here, and the reason is the one this issue states

None of that is verified against NCBI's own documentation — per #474's opening rule and the lesson of #460, check the endpoint, auth shape, response shape, rate limits and terms against the primary source before writing code.

That is now a repository rule, not just good practice: ADR-0046 says a claim about a vendor is normative and the URL where the vendor says it is a pointer. Implementing E-utilities from general knowledge would put an unverified contract into SOURCES.md, which is what #460 cost and what ADR-0042 had to carve an explicit exception for.

Verifying it means reading NCBI's E-utilities documentation and terms directly, including the tool-name/contact-email requirement and the 3-vs-10 req/s figures this issue quotes without a source. That is a session of primary-source reading, not a code change, and it should be the first commit on this issue rather than a footnote to one.

What is already settled

• ADR-0030 D3 fixed the identifier-pick priority (doi > arxiv > pmid), so the ordering question does not need reopening.
• canonical.rs:22 already reserves the class-string contract for future variants, so "pmid" / "pmcid" land in a shape ADR-0007 anticipated.
• Europe PMC is integrated (feat(source): Europe PMC — OA full text that Unpaywall does not index #415) and — since bug(source): europe-pmc refuses on isOpenAccess=N before consulting fullTextUrlList — discarding a Free PDF at an already-allowlisted host #503 — accepts a record outside the bulk OA subset that still advertises a Free PDF, which materially improves what a resolved PMID can actually retrieve.

Keeping open. The first step is documentation, not code.

[sotashimozono]

Reporting half is #576. Step 2 is blocked on something this issue did not anticipate, and I checked it before writing any of the Ref::Pmid work.

NCBI is fine — verified against the primary source

The issue flagged its own NCBI claims as unverified. They hold. Against NBK25497 and a live round-trip:

base	https://eutils.ncbi.nlm.nih.gov/entrez/eutils/
rate	3 req/sec without a key, 10 with one
key	optional
tool / email	encouraged; mandatory only to lift an IP block
large jobs	weekends, or 21:00–05:00 ET on weekdays

esearch.fcgi?db=pubmed&term=10.1176/ajp.155.7.895[AID]   →  9659853
esummary.fcgi?db=pubmed&id=9659853                       →  articleids[] doi = 10.1176/ajp.155.7.895
                                                             + title, pubdate, fulljournalname, authors

So the resolver works, in both directions, and it happens to be the Coryell 1998 paper from #536.

One thing to carry forward: doiget's global cap is 5/sec and NCBI's keyless limit is 3. The cap alone would exceed NCBI's published rate, so this needs a SOURCE_RATE_OVERRIDES entry the way arXiv does (#493). That is a legal-safeguard matter, not a politeness one.

The blocker: safekey is a contract with another repository

docs/SAFEKEY.md opens with:

Status: NORMATIVE (shared spec). Binding for both doiget and BiblioFetch.jl. Any change requires a coordinated ADR and an update to the reference test vectors at tests/fixtures/safekey/vectors.json.

and its reference Julia implementation discriminates ref classes with a binary test:

raw = startswith(ref, "10.") ? "doi_$ref" : "arxiv_$ref"

A PMID is neither. 9659853 would be keyed arxiv_9659853 by BiblioFetch.jl and pmid_9659853 by doiget — a store one tool writes and the other misreads, silently, which is worse than the skip this issue reports.

safekey_matches_reference_vectors also pins the fixture at exactly 100 entries, with the comment that the count guard "ensures the set cannot silently grow or shrink without a coordinated ADR bump".

So step 2 needs, in order:

1. an ADR adding the pmid (and pmcid) canonical classes — canonical.rs:22 already reserves the names, and ADR-0030 D3 already decided the priority, so what is missing is the existence decision plus the safekey spec;
2. docs/SAFEKEY.md §3 amended, including the Julia reference implementation, since a two-way branch cannot express three classes;
3. vectors.json extended, with the 100-count guard updated in the same commit;
4. the matching change in BiblioFetch.jl — a different repository, and the reason I have not landed doiget's side alone.

Until 4 lands, shipping Ref::Pmid in doiget means the two tools disagree about where a paper is stored. That is not a thing to do quietly, so I have not done it.

What #576 does instead

The half of the reported consequence that needs no new class: "nothing says which ones went missing or why". An entry carrying a PMID no longer reports "entry has no DOI / arXiv id" — a sentence that puts the deficiency on the user's bibliography when it is on doiget — and it reports NOT_IMPLEMENTED rather than INVALID_REF, so the advice is "wait for a release" rather than "correct your input".

[sotashimozono]

Reporting half shipped in 0.8.13. Ref::Pmid itself is still parked — grep -n 'Pmid' crates/doiget-core/src/lib.rs returns nothing.

A PubMed-identified entry is no longer told its bibliography is broken. ParseError::UnsupportedIdentifier carries the kind and value, and four surfaces report it as NOT_IMPLEMENTED (terminal, "wait for a release") rather than INVALID_REF ("correct your input"):

• doiget verify
• doiget_batch_from_bibliography (MCP)
• doiget batch — this one needed a second pass; see below
• CSL-JSON as well as BibTeX — also a second pass

Two gaps found by reviewing the first attempt:

doiget batch still said INVALID_REF. The parser's verdict was flattened into a placeholder string (<unsupported-PMID:...>) and handed back to Ref::parse, which has exactly one thing to say. So the claim this issue exists to prevent survived on that surface, and the message described the placeholder rather than the entry. The verdict travels as a value now (BatchEntry::Rejected).

CSL-JSON had no check at all. Only BibTeX did — and CSL-JSON is what Zotero exports, which is where most of these records come from. Both the top-level PMID/PMCID fields and Zotero's note: "PMID: …" form are recognised now.

Why the variant is still parked

Unchanged from the analysis in the issue: docs/SAFEKEY.md is a shared spec with BiblioFetch.jl, whose reference implementation discriminates classes as startswith("10.") ? doi : arxiv. A PMID is neither, so doiget and BiblioFetch.jl would key the same paper differently. That needs a coordinated ADR and a change in the other repository — not something to slip into a release.

Leaving open for the variant.