Repository navigation
Stop the link checker deleting live links, and merge duplicates instead of dropping them - #13
Merged
Merged
Conversation
…ad of dropping them The library is 645 records whose value is their links, and two things in the pipeline were quietly wrong about them. THE LINK CHECKER COULD NOT TELL A 403 FROM A 404 verify_urls.py had one bucket for "not accessible" and offered --remove-broken against it. Its report of 2026-02-28, seven months stale, put 162 of 516 URLs in that bucket -- and 102 of the 162 were HTTP 403: 35 from documents1.worldbank.org, 26 from thelancet.com, the rest from ResearchGate, Science, SAGE and ScienceDirect, publishers that refuse robots as policy. Every one opens in a browser. Running the flag would have deleted a fifth of the library, most of it live, and the diff would have read as maintenance. It also sent User-Agent: DevDiscourses-URLChecker/1.0, close to the perfect string for getting blocked, so the report described the checker rather than the web. Rewritten with five states -- ok, paywalled, blocked, unknown, broken -- a browser user-agent, GET after any non-2xx HEAD rather than only after 403/405, and one serial retry for unknowns. Only `broken` is ever removable. Re-run: 644 unique URLs, 25 genuinely broken, not 162. Seventeen traced to their current home and replaced, each probed before it went in (FAO's CSA sourcebook, the AEA landing pages for three papers whose author's .edu directory went away, Princeton for Running Randomized Evaluations, the ILO's reorganised publication paths, and others). Eight remain and are now visible to a reader rather than silent. One of the eight was never right: the WHO mental-health entry pointed at who.int/publications/i/item/9789241511810, and that ISBN is a standard operating procedure for measuring nicotine in cigarette smoke. No amount of re-checking finds a cataloguing error. --fail-on-new-broken compares against the committed report and is what link-check.yml runs daily. DEDUPLICATION WAS DISCARDING CURATORIAL DECISIONS build.py kept the first entry under a normalised title and dropped the rest silently: 24 of 669, and all 21 affected titles spanned more than one topic. What it discarded was a topic assignment somebody made on purpose. Conditional Cash Transfers is filed under both Public Policy & Governance and Social Protection; afterwards it existed only under the first, because that file sorts earlier, and a reader filtering to Social Protection could not find it. stats.json agreed with itself throughout. Eleven of the 21 also carried different URLs under the same title, usually a working paper beside the published version. merge_duplicates() keeps one record with `topics` for every topic and `alt_urls` for the other links; `topic` stays as the first so scalar readers keep working. Research Methods 46 -> 53, Public Policy & Governance 50 -> 55, Livelihoods 49 -> 54. A BADGE THAT COULD NEVER APPEAR resource.html had a "Verified" badge driven by r.verified, and enrich_data.py set that field to False on every resource on every build. Unreachable code, and the word claimed an editorial judgement the pipeline never made. Replaced by link health, which says what the checker actually knows: a publisher refusing a robot reads "not confirmed from here", a document that is gone gets DOI, Scholar, web and Wayback searches. COLOUR AND LAYOUT --color-text-muted was 3.28:1 on the page background across twelve rules. --color-book and --color-grey were similar. .access-badge was a second copy of the badge system written as literals with no dark value, so in dark mode an Open Access chip painted #059669 on #ecfdf5 at about 1.7:1. check_contrast.py reads both stylesheets in load order and measures every ink against every surface in both themes. It gates pull requests: no browser, no network, under a second. Fault-injected. What it could not see, and axe over the built pages could: `color: white` on `background: var(--color-accent)` in seven rules. The accent is a fill there, not a surface, and white is a literal -- 7.51:1 in light, 2.22:1 in dark where the kit aliases the accent to #6cb2ff. Every filled pill, the Read at Source button, the active citation tab and the saved toggle. It is --color-on-accent now and FILLS measures that direction too. Also from the browser: .header-nav was a flex row with no wrap, 545px inside a 390px viewport, so the whole page scrolled sideways by 179px on a phone and looked perfect on a desktop. Four filter groups were <label>Topic</label> followed by a <div> of buttons, which associates nothing, so a screen reader met unnamed groups announcing "All", "All", "Any Year". Icon-only buttons had only `title`. Prose links had no underline, so they were distinguished from their paragraph by colour alone. After: 0 serious or critical axe nodes across both pages, both themes, both viewports, and no horizontal overflow at 390px. CACHE_VERSION dd-v3 -> dd-v4. The shell is cache-first, so without the bump the new markup would have paired with the old script on every device that had visited before and the feature would have been invisible. CI splits: ci.yml keeps build, drift, tests and adds the contrast check. link-check.yml is new and daily -- link rot needs no commit, and a long third-party crawl cancelled by its own timeout cancels the whole run. The lychee step also crawled only **/*.html and **/*.md, so the 644 links the library exists to hold were outside it entirely. Tests 24 -> 31. CLAUDE.md written; README corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL
Two small things the first CI dry-run exposed. `--recheck` carries forward the last run's verdict for any URL it did not re-check, and it was writing a `"stale": true` key into each of those records. That is a working flag, not data: its only job is to decide whether link_checked gets refreshed. Committed, it added churn to every --recheck diff and told a reader nothing. It is `_stale` now and underscore keys are stripped before the report is serialised. `--no-write` meant "leave the topic files alone" but still rewrote the report, so the scheduled gate modified a committed file on every run. The flag now means what it says: nothing on disk is touched. That is what link-check.yml passes, since a gate has no business rewriting the baseline it is checking against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL
The shared dyslexia-font.js sets `opacity: var(--dys-idle-opacity, .72)` on #dys-font-btn, compositing whatever ink the surrounding bar uses down by 28%. Measured on pinpointventures, where that ink is #4b5563 at 6.41:1, the idle button came out #7d858f: 3.73:1, under AA. The comment above that rule argues, correctly, that leaving the site's own ink alone is the one version that cannot go wrong, because it is the ink the site already chose for that bar. Then it takes 28% off it. A control that exists so people who find text hard to read can read it should not be the hardest thing on the page to read. The default is 1 now. The idle/on distinction already rode on the border and the accent wash, with OpenDyslexic in the label and aria-pressed for anyone who cannot see either. The file is copied into 18 repositories. Seventeen were byte-identical and one, JanVayu's, was ahead: a 44px minimum touch target (WCAG 2.5.5) and a placement option the rest never got. Both are merged and all 18 now hold the same file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL
`make contrast` reads the two stylesheets, so a colour written at runtime is outside it. #8a8a8a — the exact value this repository replaced at 3.28:1 on the page background — was still being emitted by three lines of JavaScript: the "no resource specified" message and the Unknown access badge in js/resource.js, and the "no strong connections found" text drawn into the SVG in js/litmap.js. Swept both pages with axe in each colour scheme at 390x844: 1 node before, 0 after. 31 tests and the contrast check pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A 403 is not a 404, and conflating them nearly cost a fifth of the library.
verify_urls.pyhad one bucket for "not accessible" and offered--remove-brokenagainst it. Its report of 2026-02-28 put 162 of 516 URLs in that bucket, and 102 of the 162 were HTTP 403 — 35 from documents1.worldbank.org, 26 from thelancet.com, the rest from ResearchGate, Science, SAGE and ScienceDirect. Every one opens in a browser. Running the flag would have deleted a fifth of the library, most of it live, and the diff would have read as routine maintenance. It also sentUser-Agent: DevDiscourses-URLChecker/1.0, close to the perfect string for getting blocked, so the report described the checker rather than the web.URLs now sort into
ok,paywalled,blocked,unknownandbroken, and onlybrokenis ever removable. Re-run honestly: 644 unique URLs, 25 genuinely broken. Seventeen were traced to their current home and replaced, each probed before it went in; eight remain and are flagged on their own resource pages rather than hidden.Duplicates are merged, not dropped.
build.pykept the first entry under a normalised title and discarded the rest silently: 24 of 669 entries, and all 21 affected titles spanned more than one topic. What it discarded was a curator's topic assignment — a reader filtering to Social Protection could not find a paper filed under it, andstats.jsonagreed with itself throughout. Eleven also carried different URLs under one title, usually a working paper beside the published version. Restoring the assignments moved Research Methods 46→53, Public Policy & Governance 50→55, Livelihoods 49→54.A badge that could never appear.
resource.htmldrove a "Verified" badge offr.verified, which the pipeline set toFalseon every resource on every build. The branch was unreachable and the word promised an editorial judgement nothing made. Gone.Colour.
--color-text-mutedat 3.28:1 across twelve rules;.access-badgewas a second copy of the badge system written as literals with no dark value, painting#059669on#ecfdf5over a near-black card at about 1.7:1. Three#8a8a8aliterals in JavaScript survived the token pass, becausemake contrastreads CSS — grep the JavaScript too when you move a colour.Type of change
Checklist
make build,make test(31 checks) andmake contrastpassservice-worker.jsCACHE_VERSIONbumped — the shell is cache-first and this change adds markup only the new JS fills🤖 Generated with Claude Code
https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL
Generated by Claude Code