[DBIP] The link check cannot validate the x and linkedin columns: 60% of its errors are X and LinkedIn refusing CI, not broken links

[EazyHood]

Proposal type: Other (CI / validation configuration)

Affected scope (files/folders/chains): .github/workflows/link-check-analysis.yaml, .lycheeignore; affects the x and linkedin columns of references/providers/providers.csv

Motivation / problem statement

Correct values in the x and linkedin columns cannot reliably pass this repository's link check, and there is no value a contributor can write that fixes it. X and LinkedIn both throttle or deny automated requests from GitHub-hosted runners. The URL is fine; the caller is the problem.

This is measurable in this repository's own CI. The most recent link-check run on PR #2977 (run 33375659246 era, report posted 2026-08-31T15:04Z):

Links checked	2663
Successful	2621
Errors	42
— of those, on x.com or linkedin.com	25 (60%)
— on real websites and docs	17

Six of the reported failures:

[ERROR] <https://x.com/stripe>
[ERROR] <https://x.com/SuiNetwork>
[ERROR] <https://x.com/ton_blockchain>
[ERROR] <https://x.com/FoxWallet>
[ERROR] <https://x.com/gemwallet>
[999]   <https://www.linkedin.com/company/artemis-xyz>

x.com/stripe and x.com/SuiNetwork are not broken links. All six return 200 when requested from an ordinary residential connection, which I checked before filing this. Sixty percent of the error budget of the link check is currently spent on two hosts refusing to talk to CI.

The LinkedIn case has a precise, fixable cause. The workflow accepts:

--accept 200,202,204,400,401,403,405,429

LinkedIn answers bots with HTTP 999, a non-standard code of its own invention. Note that 403 is already accepted — a site that blocks bots with a standard Forbidden passes. LinkedIn fails only because it picked a code outside the standard range. X is worse: it varies between a connection-level ERROR, a 404, and a 500 for the same live account depending on throttling.

Why this costs the program

The failure is intermittent, which is the worst property a gate can have: the same unchanged data produces a clean run and a failing run, so a contributor cannot distinguish a real dead link from an artefact. And because the check reads whole rows, this reaches any PR that edits a row that happens to carry an x or linkedin value — x is populated in 479 of 720 provider rows (66%) and linkedin in 187 (26%).

The effect is visible: PRs #2977, #2978 and #2979 have been open since 2026-08-16 — fifteen days — cycling through REQUEST_CHANGES reviews whose findings are of the form "the current-cycle link-check reports https://x.com/ton_blockchain as an error". That account is live. Each cycle a different random subset fails, so the requested clean run does not arrive by fixing anything.

It also creates the wrong incentive: the cheapest way for a contributor to keep the error count at zero is to stop filling x and linkedin at all. That is what I did — I removed 40 verified linkedin cells from #3323, #3324, #3325 and #3326 rather than defend them one at a time, and I am now facing the same question for 75 verified x cells.

Detailed proposal

• Change type: other (CI configuration). No schema change, no column added, removed or modified.

Option A — exclude the two hosts. Add to .lycheeignore, which today contains only rpc*:

x.com
twitter.com
linkedin.com

Option B — accept what they return. Add 999 to the accept list, and set --max-retries with a delay for these hosts. This helps LinkedIn but not X, whose failure is at the connection level and is not a status code at all.

Option A is the one I would suggest, with one caveat stated plainly: it means a wrong-but-well-formed handle stops being caught. That is a real loss. It is smaller than it looks, because a 200 from these hosts never proved the handle belonged to the right provider either — X serves a page for any syntactically valid handle, and the check was only ever catching hard 404s. If you want that signal back, it belongs in a separate job that runs on a schedule and is allowed to fail without blocking a PR, rather than in the gate that decides whether a contribution is merged.

Keeping both columns is worth it. It is only the checking of them that is broken.

Reproduction

The accept list:

grep -n "accept" .github/workflows/link-check-analysis.yaml

Current output: --accept 200,202,204,400,401,403,405,429

The exposure:

import csv, io
rows = list(csv.reader(io.open('references/providers/providers.csv', encoding='utf-8-sig')))
h = [c.strip() for c in rows[0]]
for col in ('x', 'linkedin'):
    i = h.index(col)
    n = sum(1 for r in rows[1:] if i < len(r) and r[i].strip())
    print(col, n, 'of', len(rows) - 1)

Current output:

x 479 of 720
linkedin 187 of 720

The failures themselves need no new tooling to confirm — they are in this repository's own link-check comments on #2977, #2978 and #2979.

Rewards address: 0x06f44f4839fd5df4f4670036d028b29dec939363