The rate cap bounds throughput but not persistence, so a retrying agent can bulk-scrape without ever exceeding 5/sec

[sotashimozono]

docs/LEGAL.md §6a safeguard 5 claims the hard-coded rate limit

prevents bulk-scraping patterns and cannot be overridden by configuration

It bounds 5 requests per second. It does not bound how many times the same request is made.

An agent that retries 500 failed DOIs ten times each emits 5,000 requests while never once exceeding 5/sec. That is a bulk pattern, produced entirely within the safeguard, by a caller behaving exactly as LLM callers behave when a failure looks retryable. The sibling issue on disposition (#506) covers why so many of them look retryable; this issue is about the fact that even a correct disposition is only advice, and advice is not a safeguard.

This is the same class as #493: a NORMATIVE claim that is broader than the mechanism behind it.

doiget is the only tool that can close this

Every fetch is already recorded. docs/PROVENANCE_LOG.md / ADR-0006 define a fail-closed, hash-chained JSON Lines log, and a fetch that cannot be logged does not proceed. So the data needed to answer "has this exact request already been made, and what happened?" is on disk before the network is touched.

That is worth stating plainly because it is not a portable idea: a fetcher that does not record its attempts cannot implement this at all. It turns safeguard 5 from "bounds rate" into "bounds rate and persistence", which is a materially stronger claim and one that can be demonstrated rather than asserted.

Proposed shape

Before the network leg, consult the provenance log for prior attempts on the same (ref, source) within a short window. Then:

• prior outcome was terminal → do not re-fetch. Return the prior outcome, marked as replayed, with the original error and its denial_context.
• prior outcome was needs_config → do not re-fetch, and return the remediation again. Nothing has changed unless the config changed, and whether it changed is checkable.
• prior outcome was retry_after → enforce the wait rather than refuse. If the caller returns early, sleep the remainder or return RATE_LIMITED with the true remaining time.

Depends on #506: "the prior outcome was terminal" has to be a fact the code carries before it can be acted on.

Design constraints worth deciding up front

• Staleness. A negative result is not permanent in the real world — a paper can become OA, an embargo can lift, a config can change. The window must be short (minutes, not days), and there must be an explicit way to force a real fetch. The point is to stop a loop inside one agent session, not to cache a "no" forever.
• Config changes must invalidate. A needs_config replay is wrong the moment the user actually applies the remediation. Keying the decision on the capability profile / config fingerprint alongside the ref would handle it.
• It must be visible, not silent. A replayed answer that looks identical to a fresh one is a new silent failure, which is the defect class this repo keeps finding (bug(cli): config path and config show print nothing and exit 0 when not on a TTY — the #219/#220 deadlock, still open for config #476, [store] root in config.toml is ignored (env and flag work) — while [network] from the same file is honoured, and config init/doctor both recommend the setting that does nothing #441, bug(tdm): the Tier-3 chain is skipped whenever Crossref answers, so a TDM source can never close the fetch gap it was added for #458, text: silent empty output (exit 0, 0 bytes) when ar5iv text is unavailable — callers misdiagnose as a wrong DOI #302). The envelope should say the answer was replayed and why, and the CLI should say so too.
• Do not make it configurable off. Safeguard 5's stated property is that it cannot be overridden by configuration. Anything added under that heading inherits the constraint, or the heading has to change.

Notes

• docs/LEGAL.md is Status: NORMATIVE and safeguard 5's wording would be amended by this, so ADR-0014 requires an ADR. LEGAL.md under-declares the default binary's network surface and does not know tdm-ieee exists #494 is already correcting other parts of §6a — this should land after it rather than conflict with it.
• Worth a line in whatever posture/self-report surface ends up existing: "repeat suppression: on" is the kind of statement a security review can act on.

Refs #506, #493, #494, ADR-0006, ADR-0014.

[sotashimozono]

Status check while working through the open issues.

Unchanged and correctly ordered: this depends on #506, and the dependency is real rather than tidiness. Every branch of the proposed design keys off a disposition — terminal → replay, needs_config → replay the remediation, retry_after → enforce the wait — and none of those is a fact the code carries today. #506 is what makes them facts.

Two notes from reading around it while working on the other issues:

The provenance-log argument in this issue is sound and worth restating. ADR-0006 makes the log fail-closed and hash-chained, so the data needed to answer "has this exact request already been made, and what happened?" is on disk before the network is touched. That is not a portable idea — a fetcher that does not record its attempts cannot implement this — and it is what turns safeguard 5 from "bounds rate" into "bounds rate and persistence".

The LEGAL.md §6a wording will need care. Safeguard 5 says the rate limit "cannot be overridden by configuration". Anything added under that heading inherits the constraint, so the staleness window and the force-a-real-fetch escape hatch this issue asks for have to be designed as not configuration in that sense, or §6a's wording changes. ADR-0014 requires an ADR either way, and #494 / ADR-0047 just went through the exercise of making every §6a clause name enforcement that exists — the new one should arrive already meeting that bar rather than being corrected later.

One thing that has moved: #517 (ADR-0052) widens what a closed-DOI fetch attempts, so the class of run this issue is about — a retrying agent hammering the same refused ref — now produces a named CAPABILITY_DENIED rather than a silent exit 0. That makes the disposition more obviously needs_config than it was, which helps #506 and therefore this.

[sotashimozono]

Spiked this and stopped before shipping, because the mechanism as proposed cannot act on the data the provenance log currently carries. Two findings, one good and one blocking.

1. Session scoping removes the config-fingerprint problem

The issue lists "config changes must invalidate" as a design constraint, and suggests keying on a capability-profile fingerprint alongside the ref.

That is not needed. A CapabilityProfile is built once from the environment at process start and never mutated, so within a single session_id the configuration provably has not changed. Scoping suppression to the session satisfies the constraint by construction, rather than by a fingerprint that could drift out of step with the thing it fingerprints.

It also matches the target exactly — "an agent that retries 500 failed DOIs ten times each" is one session — and matches the issue's own framing that the point is to stop a loop inside one agent session, not to cache a "no" forever. session_id is already on every row.

Worth keeping whoever implements this.

2. The blocker: the log records the leg's outcome, not the call's

The proposal is to consult prior attempts and branch on their disposition (now available as ErrorCode::disposition, ADR-0055). Measured against what actually gets written:

	writes a per-ref Fetch/Err row?	error_code	disposition
OA leg refused pre-fetch (allowlist)	yes	NETWORK_ERROR	retry_after
OA leg failed post-fetch	yes	NETWORK_ERROR	retry_after
metadata-only, no OA URL anywhere	no row	—	—
NOT_FOUND	no row	—	—
CAPABILITY_DENIED	no row	—	—

Those are the only two LogResult::Err appends in orchestrator.rs that carry a ref, and both hard-code ErrorCode::NetworkError.

So a guard that suppresses terminal / needs_config repeats — which is the correct rule, since retry_after must be let through or correct backoff breaks — suppresses nothing. I built it and confirmed it: full suite green precisely because the guard never fires.

Spike is on feat/507-repeat-suppression (5383a6d, unpushed, not for merge).

Suggested sequencing

Step 1 — make the log record the call's outcome. Today a metadata-only "no OA anywhere", a NOT_FOUND and a CAPABILITY_DENIED all leave no per-ref failure row, and the rows that do exist are all NETWORK_ERROR regardless of the fine reason. That is a gap in the audit trail on its own terms, independent of this issue: the provenance log cannot currently answer "what did this session tell the caller about this ref?"

Step 2 — then suppress. Once the row carries the outcome and its real code, the rule in this issue works as written, session-scoped per finding 1.

Note also that ERRORS.md §6.1 says a redirect-denied leg is CAPABILITY_DENIED and not NETWORK_ERROR, while both log appends write NETWORK_ERROR. I have not chased whether that is a real inconsistency or a deliberate split between the logged code and the surfaced one — but whoever does step 1 will have to decide it, so flagging it here.

One more thing to decide in step 2

Refusing a repeat turns a call that previously returned ok: true with a blocked PDF leg into ok: false. That may well be right — the issue asks for exactly that for needs_config — but it is a wire-visible shape change for the same input and deserves to be an explicit decision in the ADR rather than a side effect. Replaying the outcome instead, as the issue's first bullet suggests, needs data the log does not carry either (path, size_bytes, digest, title), so it is not a cheaper alternative.

[sotashimozono]

Step 1 is #572.

Correcting the flag I raised in my previous comment. I wrote that ERRORS.md §6.1 says a redirect-denied leg is CAPABILITY_DENIED while both log appends write NETWORK_ERROR, and that someone would have to decide whether that is a real inconsistency. It is not. §6.1 already settles it explicitly:

The provenance row keeps error_code = NETWORK_ERROR (it records the transport mechanism); the user-facing code/exit is CAPABILITY_DENIED / 3.

So the two rows written for one blocked fetch legitimately differ and each is true about its own layer. Nothing to decide. That layering is now stated in docs/PROVENANCE_LOG.md §3 next to the schema, rather than only in a §6.1 aside about the CLI — which is where I should have found it.

What step 1 does. The session_end bookend records the code the caller was given, so the log can answer "what did this session tell the caller about this ref?". For doiget fetch that is deliberately not always the Result's: a blocked PDF leg is Ok with a failed leg, and it is the outcome an agent is most likely to retry, so the leg's code is recorded rather than null. A batch bookend still records null — it spans many refs and has no single code.

docs/PROVENANCE_LOG.md §3's field table also never had an error_code row at all; it does now.

Still open for step 2, unchanged from the previous comment: session scoping removes the config-fingerprint problem, and two things need deciding in the ADR rather than assumed —

1. refusing a repeat turns an ok: true with a blocked leg into ok: false, which may well be right for needs_config but is a wire-visible shape change for the same input;
2. replaying the outcome instead, as this issue's first bullet suggests, needs path / size_bytes / digest / title, which the log still does not carry — so it is not the cheaper alternative it looks like.

[sotashimozono]

Not fixed. Step 1 only. Recording this because the 0.8.13 CHANGELOG mentions #507 and a grep for the number would otherwise read as "shipped".

What landed: the SessionEnd bookend now records what a call failed with. Every row used to carry error_code: null, including rows carrying a ref, so the log could not answer "what did this session tell the caller about this ref?" — only "something went wrong".

What did not land: the repeat suppression this issue asks for. grep -rn 'suppress' crates/ returns nothing in production code. It was built, measured, and deliberately not shipped — the guard fires on nothing. The only per-ref failure rows carry NETWORK_ERROR, whose disposition is correctly retry_after and which must be let through; the terminal outcomes wrote no row at all. A guard written against the old log would have suppressed zero things.

Two corrections found by reviewing that work, both now fixed, both of which would have made the suppression wrong even once step 2 is written:

• The MCP surface logged result: "ok" beside a non-null error_code for a blocked PDF leg — a self-contradictory row for exactly the outcome suppression has to be able to see. The CLI's clean-success rule had been ported in the error_code half only.
• Then the two batch tools were found still calling .all(|r| r.outcome.is_ok()) a success, so a batch where every entry was refused wrote result: ok. All four call sites now share FetchPaperOutcome::is_clean_success.

So the foundation is sound now in a way it was not when step 1 first landed. Step 2 remains unwritten.