test: an end-to-end use-case suite driven through the real entry points, because every "unreachable source" bug passed its unit tests

[sotashimozono]

Four bugs this cycle share one shape: a source was implemented, gated, allowlisted and unit-tested,
and did not work.

	found by	passed its unit tests
#413	five Tier-2 sources never called	yes
#442	three Tier-3 sources never called	yes
#454	Tier-3 allowlists never registered in the client	yes
#458	Tier-3 chain skipped whenever Crossref answers	yes

In every case the unit test drove the Source impl directly, or asserted that a builder returned
the right value, and the production entry point was never in the picture. #454's guard even had a
doc comment describing the failure it could not catch.

Each fix added a targeted regression test, which is right, but they were written after a human
drove the real thing and noticed. Nothing drives the real thing on a schedule.

What is missing

A suite that starts where a user starts — doiget fetch, doiget_fetch_paper over MCP stdio —
and asserts on the observable outcome, for a small fixed set of DOIs chosen because each one
exercises a different route.

Not a replacement for the unit tests. The thing they cannot do is notice that a correct component
is never reached.

Sketch

DOI	route it exercises	expected
10.1109/TSP.2018.2812747	hybrid OA, repository copy, host not curated	blocked, denial_context.attempted names the repository host
same, trust_academic_repos = true	the user-extension trust flag	fetched, non-zero bytes
10.1090/s0025-5718-04-01692-8	bronze OA, publisher rate-limits (#445)	blocked, and the trace names every enabled source
an arXiv-preprinted DOI	the #325 preprint fallback	fetched via arXiv, PreprintFallback
a gold-OA DOAJ DOI	the ADR-0037 apex redirect	fetched
a DOI Crossref cannot resolve	the Tier-2 optional chain	the trace distinguishes "asked, nothing" from "never asked"

Each row asserts which route produced the outcome, not merely that one occurred — that is the
distinction every bug above slipped through.

Constraints, and the awkward part

These need the network, so they cannot sit in the ordinary test job: the network-purity job
exists precisely to keep the default suite hermetic. Options:

• a separate scheduled workflow (nightly), continue-on-error: false but off the PR path, so an
  upstream change surfaces as a failing cron rather than a blocked PR;
• #[ignore] by default, run explicitly;
• record the responses as fixtures and replay them, with a much rarer live job that re-records.

The third is the most stable and the least likely to notice that the real world changed, which
is the whole point. Probably the first, accepting the flakiness, with the DOI list kept small and
the failures triaged rather than muted.

Politeness matters: a fixed handful of DOIs, a contact address set, and the existing rate limiter.

Why now

#430 shipped a source justified by a corpus nobody had fetched end-to-end. Driving it revealed
#458, #459 and #460 in one session — none of which any unit test could have found. That is a
strong argument that this suite pays for itself.

Refs #413, #442, #454, #458, #459, #460, #445.

[sotashimozono]

Status, after working through the batch. Not implemented — and the case for it got stronger today, twice.

Two more entries for the table

	found by	passed its unit tests
#413	five Tier-2 sources never called	yes
#442	three Tier-3 sources never called	yes
#454	Tier-3 allowlists never registered in the client	yes
#458	Tier-3 chain skipped whenever Crossref answers	yes
#516	Tier-2 allowlist gated on citation while the sources are gated on metadata — every Tier-2 source dies at UnknownSource in a metadata build	yes
#503	europe-pmc refused a Free PDF one line before the code written to find it	yes — the fixture could not even distinguish the two refusal reasons

Both are the same shape, and both were found by a human reading the code rather than by anything driving it.

Worth adding: #516's fix also revealed that doiget-cli's citation feature did not imply its own metadata, so capabilities under-reported the compiled feature set. A suite that drove doiget capabilities across the feature matrix would have shown that as a difference in output.

Design notes from this pass

The DOI table needs one more row. ADR-0052 settled that a closed DOI is unreachable on the open route because Crossref's link[] is programme-scoped. That makes "closed DOI, everything enabled, still exits 0" a specified outcome rather than a bug — which is exactly the kind of thing that should be pinned, because the next person to look at it will assume it is a bug and re-open the question.

Option 1 is right, for the reason this issue gives. A scheduled workflow, continue-on-error: false but off the PR path. The fixture-replay option "is the most stable and the least likely to notice that the real world changed, which is the whole point". tests/fixtures/real_world/ already exists for response shapes and should stay that — it is the replay corpus, not the reachability suite.

One thing the fixture corpus just proved it is good for. Seven of its expected.toml files asserted that oa_url held a Similarity Check / TDM link. That was a real behaviour captured from real responses, and it is what made the #517 measurement decisive rather than anecdotal. Whatever the live suite records, it should record it in a form that can be read back like that.

Politeness constraints stand: a fixed handful of DOIs, DOIGET_CONTACT_EMAIL set, and the existing rate limiter — and, since #504, the contact address can now come from config.toml too, which is one fewer secret to thread through a workflow.

This is the stated prerequisite for #474, #463 and (less strictly) #500. It should be next.

[sotashimozono]

#577 now closes three of the four gaps and reproduces the fourth as a live defect.

Coverage: 1 of 5 → 4 of 5

Measured before writing anything, exactly one route (blocked) was asserted anywhere. Now fetched, no_oa_url and preprint_fallback are too.

preprint_fallback is the one worth naming. The existing blocked-leg test asserted suggested_arxiv_id — the suggestion. The #325 fallback actually running is a different route, one field away in the same envelope, and the only thing separating them in the harness was an unset DOIGET_ARXIV_BASE: the fallback fetches through the arXiv source, not oa-publisher, so it left the machine, failed, and the leg stayed blocked. Precisely the "a correct component is never reached" shape this issue is about, hiding inside a passing test.

The fifth found a defect

Two things, both worth recording.

Only tdm-aps can produce tdm_fetched. It is the sole Tier-3 source that implements fetch_content; Elsevier, Springer and IEEE inherit the default Ok(None) and are metadata-only. Three of the four Tier-3 sources cannot reach the route the tier exists for.

And driving it over MCP fails. With the grant set and the DOI prefix matched, the chain does fire — the attempts row shows tdm-aps consulted — and then:

"detail": "network error: no allowlist registered for source tdm-aps"

That is HttpError::UnknownSource: the source key is not in the client's map at all. But tier_3_allowlists() is purely #[cfg]-gated, and doiget-mcp does allowlists.extend(tier_3_allowlists()) with a comment citing #454. So the two disagree somewhere between construction and use.

This is #454's exact shape — "Tier-3 allowlists never registered in the client" — reachable again, and it is the class of defect this suite exists to surface. I have not chased the root cause; the test is #[ignore]d with the error in its doc comment so it is a reproduction someone can run rather than a gap someone has to rediscover:

cargo test -p doiget-mcp --features tdm-aps --test fetch_paper_e2e -- --ignored

Worth deciding whether that wants its own issue or belongs on #454.

Still open here

The live suite against real DOIs, to notice the world changing. Deliberately not built: every bug in this issue's table is a wiring defect, observable hermetically at the real entry point — so it belongs where it runs on every PR, not nightly. Upstream drift is a separate concern and still worth a small scheduled job.

[sotashimozono]

0.8.13 shipped route coverage, and in doing so corrected a wrong diagnosis this issue carried — worth recording, because the wrong version was about to gate a release.

What the reproduction was said to show

fetch_paper_doi_served_by_the_publisher_reports_the_tdm_fetched_route was written, failed, and was left #[ignore]d with this reading:

tier_3_allowlists() is purely #[cfg]-gated and the MCP server does extend its allowlists with it, so the two disagree somewhere between construction and use. That is #454's shape — "Tier-3 allowlists never registered in the client" — reachable again.

#580's description then raised it as "worth deciding before the cut whether 0.8.13 ships with it."

What was actually true

Both client builders have two branches. The production branch — taken only when no DOIGET_*_BASE is set — does call tier_3_allowlists(), and always did. The test-override branch — taken whenever any DOIGET_*_BASE is set, which every wiremock test does — built its allowlist from a fixed table of Tier-1/2 keys with no Tier-3 entry at all.

So no e2e on either surface could reach the route. Nothing had regressed; the harness had no way to register a Tier-3 source. Registering those keys makes the reproduction pass, so it is no longer #[ignore]d, it runs in the test (tdm features) job, and route_coverage_e2e has no remaining Gap.

That mechanism is #587, filed separately — correctly, because it is not #454's shape.

What survives, and is still true

fetch_content is implemented by tdm-aps alone. Elsevier, Springer and IEEE inherit the default Ok(None) and are metadata-only, so three of the four Tier-3 sources cannot reach the route the tier exists for. Read the new assertion as APS coverage, not Tier-3 coverage.

Why this is worth writing down

The claim was accurate about the code — the test did fail with exactly that string — and wrong about the world. That is the defect class 0.8.13 is otherwise entirely about, and it had reached the release-blocking decision before anyone checked the second branch of the function.

Leaving this open: the issue asks for an end-to-end suite through the real entry points, and one route now has one. The elsevier/springer/ieee override rows added alongside tdm-aps are still unexercised — only the APS one is proven.