From 557f454f31cb84719de1401ace87a3c7855d030e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 23 Sep 2026 05:46:18 +0000 Subject: [PATCH 1/4] Stop the link checker deleting live links, and merge duplicates instead of dropping them The library is 645 records whose value is their links, and two things in the pipeline were quietly wrong about them. THE LINK CHECKER COULD NOT TELL A 403 FROM A 404 verify_urls.py had one bucket for "not accessible" and offered --remove-broken against it. Its report of 2026-02-28, seven months stale, put 162 of 516 URLs in that bucket -- and 102 of the 162 were HTTP 403: 35 from documents1.worldbank.org, 26 from thelancet.com, the rest from ResearchGate, Science, SAGE and ScienceDirect, publishers that refuse robots as policy. Every one opens in a browser. Running the flag would have deleted a fifth of the library, most of it live, and the diff would have read as maintenance. It also sent User-Agent: DevDiscourses-URLChecker/1.0, close to the perfect string for getting blocked, so the report described the checker rather than the web. Rewritten with five states -- ok, paywalled, blocked, unknown, broken -- a browser user-agent, GET after any non-2xx HEAD rather than only after 403/405, and one serial retry for unknowns. Only `broken` is ever removable. Re-run: 644 unique URLs, 25 genuinely broken, not 162. Seventeen traced to their current home and replaced, each probed before it went in (FAO's CSA sourcebook, the AEA landing pages for three papers whose author's .edu directory went away, Princeton for Running Randomized Evaluations, the ILO's reorganised publication paths, and others). Eight remain and are now visible to a reader rather than silent. One of the eight was never right: the WHO mental-health entry pointed at who.int/publications/i/item/9789241511810, and that ISBN is a standard operating procedure for measuring nicotine in cigarette smoke. No amount of re-checking finds a cataloguing error. --fail-on-new-broken compares against the committed report and is what link-check.yml runs daily. DEDUPLICATION WAS DISCARDING CURATORIAL DECISIONS build.py kept the first entry under a normalised title and dropped the rest silently: 24 of 669, and all 21 affected titles spanned more than one topic. What it discarded was a topic assignment somebody made on purpose. Conditional Cash Transfers is filed under both Public Policy & Governance and Social Protection; afterwards it existed only under the first, because that file sorts earlier, and a reader filtering to Social Protection could not find it. stats.json agreed with itself throughout. Eleven of the 21 also carried different URLs under the same title, usually a working paper beside the published version. merge_duplicates() keeps one record with `topics` for every topic and `alt_urls` for the other links; `topic` stays as the first so scalar readers keep working. Research Methods 46 -> 53, Public Policy & Governance 50 -> 55, Livelihoods 49 -> 54. A BADGE THAT COULD NEVER APPEAR resource.html had a "Verified" badge driven by r.verified, and enrich_data.py set that field to False on every resource on every build. Unreachable code, and the word claimed an editorial judgement the pipeline never made. Replaced by link health, which says what the checker actually knows: a publisher refusing a robot reads "not confirmed from here", a document that is gone gets DOI, Scholar, web and Wayback searches. COLOUR AND LAYOUT --color-text-muted was 3.28:1 on the page background across twelve rules. --color-book and --color-grey were similar. .access-badge was a second copy of the badge system written as literals with no dark value, so in dark mode an Open Access chip painted #059669 on #ecfdf5 at about 1.7:1. check_contrast.py reads both stylesheets in load order and measures every ink against every surface in both themes. It gates pull requests: no browser, no network, under a second. Fault-injected. What it could not see, and axe over the built pages could: `color: white` on `background: var(--color-accent)` in seven rules. The accent is a fill there, not a surface, and white is a literal -- 7.51:1 in light, 2.22:1 in dark where the kit aliases the accent to #6cb2ff. Every filled pill, the Read at Source button, the active citation tab and the saved toggle. It is --color-on-accent now and FILLS measures that direction too. Also from the browser: .header-nav was a flex row with no wrap, 545px inside a 390px viewport, so the whole page scrolled sideways by 179px on a phone and looked perfect on a desktop. Four filter groups were followed by a
of buttons, which associates nothing, so a screen reader met unnamed groups announcing "All", "All", "Any Year". Icon-only buttons had only `title`. Prose links had no underline, so they were distinguished from their paragraph by colour alone. After: 0 serious or critical axe nodes across both pages, both themes, both viewports, and no horizontal overflow at 390px. CACHE_VERSION dd-v3 -> dd-v4. The shell is cache-first, so without the bump the new markup would have paired with the old script on every device that had visited before and the feature would have been invisible. CI splits: ci.yml keeps build, drift, tests and adds the contrast check. link-check.yml is new and daily -- link rot needs no commit, and a long third-party crawl cancelled by its own timeout cancels the whole run. The lychee step also crawled only **/*.html and **/*.md, so the 644 links the library exists to hold were outside it entirely. Tests 24 -> 31. CLAUDE.md written; README corrected. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL --- .github/workflows/ci.yml | 16 +- .github/workflows/link-check.yml | 66 + CLAUDE.md | 158 + Makefile | 10 +- README.md | 61 +- build.py | 80 +- check_contrast.py | 263 + css/style.css | 181 +- data/resources.json | 4070 ++++++++++-- data/search-index.json | 2 +- data/stats.json | 24 +- .../agriculture_food_systems_resources.json | 98 +- .../topics/climate_environment_resources.json | 138 +- .../conflict_humanitarian_resources.json | 72 +- .../data_technology_for_development.json | 138 +- .../development_economics_resources.json | 156 +- .../disability_inclusion_resources.json | 78 +- data/topics/education_resources.json | 60 +- .../energy_just_transition_resources.json | 40 +- .../topics/financial_inclusion_resources.json | 68 +- .../gender_social_inclusion_resources.json | 141 +- data/topics/livelihoods_resources.json | 171 +- data/topics/meal_resources.json | 136 +- .../mental_health_wellbeing_resources.json | 36 +- .../migration_urbanization_resources.json | 93 +- data/topics/public_health_resources.json | 111 +- .../public_policy_governance_resources.json | 170 +- data/topics/research_methods_resources.json | 162 +- data/topics/social_protection_resources.json | 92 +- data/topics/wash_resources.json | 93 +- enrich_data.py | 11 +- feed.json | 2 +- generate_assets.py | 13 +- index.html | 22 +- js/app.js | 49 +- js/resource.js | 102 +- resource.html | 17 +- service-worker.js | 10 +- tests/test_data.py | 107 +- url_verification_report.json | 5883 ++++++++++------- verify_urls.py | 520 +- 41 files changed, 9479 insertions(+), 4241 deletions(-) create mode 100644 .github/workflows/link-check.yml create mode 100644 CLAUDE.md create mode 100644 check_contrast.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4ef13f7..5c42c30 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,14 +33,8 @@ jobs: - name: Run tests run: python -m pytest -q - check-links: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - - - name: Link Checker - uses: lycheeverse/lychee-action@v2 - with: - args: --accept 200,204,301,302,403,429 --exclude-mail --exclude "localhost|127.0.0.1|example.com|twitter.com|x.com|linkedin.com|facebook.com|instagram.com|t.me|wa.me|play.google.com|apps.apple.com" --timeout 30 --max-retries 2 "**/*.html" "**/*.md" - fail: false - + # Reads the two token blocks out of css/style.css and measures every ink + # against every surface it can land on. No browser and no network, so it + # belongs in front of a pull request rather than on a schedule. + - name: Colour contrast + run: python check_contrast.py diff --git a/.github/workflows/link-check.yml b/.github/workflows/link-check.yml new file mode 100644 index 0000000..8139ea2 --- /dev/null +++ b/.github/workflows/link-check.yml @@ -0,0 +1,66 @@ +# Link health, on a clock rather than on a pull request. +# +# Two reasons it is not in ci.yml. +# +# A link rots with no commit behind it, so a gate that only runs on a pull +# request cannot see the failure it exists for. And a job that talks to six +# hundred third-party hosts is long: a job cancelled by its own timeout-minutes +# cancels the whole workflow run, taking the build and the tests down with it +# and reporting the run as cancelled rather than failed. +# +# The lychee step used to live in ci.yml on pull requests with `fail: false`, +# so a contributor got a result they could neither act on nor be blocked by. +# It also crawled `**/*.html` and `**/*.md`, which is the site chrome: the 644 +# links this library exists to hold are in data/topics/*.json and were outside +# the crawl entirely. + +name: Link health + +on: + schedule: + - cron: "17 5 * * *" + workflow_dispatch: + +permissions: + contents: read + +jobs: + resource-urls: + name: The library's own 644 links + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v6 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + # --fail-on-new-broken compares against the committed report, so a link + # that was already gone does not keep failing the run while one that + # broke last night does. --no-write because a scheduled run should not + # try to commit; a maintainer runs `make verify` and commits the result. + - name: Check every resource URL + run: python verify_urls.py --no-write --fail-on-new-broken --workers 8 + + site-chrome: + name: Links in the pages themselves + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v6 + + - name: Link Checker + uses: lycheeverse/lychee-action@v2 + with: + # 403 and 429 are accepted here because this crawl has no way to tell + # a publisher refusing a robot from a document that is gone. The job + # above does have that distinction, which is why it is the one that + # covers the resources. + args: >- + --accept 200,204,301,302,403,429 + --exclude-mail + --exclude "localhost|127.0.0.1|example.com|twitter.com|x.com|linkedin.com|facebook.com|instagram.com|t.me|wa.me|play.google.com|apps.apple.com" + --timeout 30 --max-retries 2 + "**/*.html" "**/*.md" + fail: true diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..133404c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,158 @@ +# development-discourses + +A curated open-access library: 645 research papers, books and grey literature +for development practitioners in South Asia, across 19 topics. Static HTML, no +build step for the part that runs, a standard-library Python pipeline for the +part that generates. Deployed to GitHub Pages under `/development-discourses/`. + +## Commands + +```bash +make build # enrich topic files, merge, regenerate every derived asset +make test # 31 checks +make contrast # every ink token against every surface, both themes +make verify # check every resource URL, write the report and link_status +make recheck # re-check only what was not ok last time (much faster) +make serve # http://localhost:8000 +``` + +There are no dependencies. `pytest` is the only dev one. The front end is +vanilla HTML, CSS and JavaScript; the pipeline is the Python standard library. +Keep it that way. + +## One source, five derived files + +`data/topics/*.json` is the only thing anyone edits, one file per topic. +`build.py` merges them into `data/resources.json`, and `generate_assets.py` +derives `data/search-index.json`, `data/stats.json`, `sitemap.xml`, `feed.json` +and `opensearch.xml`. CI fails if any of those has drifted from the source, so +**run `make build` and commit the result** with any data change. + +## A 403 is not a 404, and conflating them nearly cost a fifth of the library + +`verify_urls.py` sorts every URL into `ok`, `paywalled`, `blocked`, `unknown` +or `broken`. That distinction is the whole point of the script. + +The version before 2026-09-23 had one bucket for "not accessible" and offered +`--remove-broken` against it. Its report of 2026-02-28 put **162 of 516** URLs +in that bucket, and **102 of the 162 were HTTP 403**: 35 from +documents1.worldbank.org, 26 from thelancet.com, the rest from ResearchGate, +Science, SAGE and ScienceDirect. Every one opens in a browser. Running the flag +would have deleted a fifth of the library, most of it live, and the diff would +have read as routine maintenance. + +It also sent `User-Agent: DevDiscourses-URLChecker/1.0`, which is close to the +perfect string for getting blocked, so the report described the checker rather +than the web. + +Re-run honestly: 644 unique URLs, 25 genuinely broken. Seventeen were traced to +their current home and replaced, each probed before it went in; eight remain +and are flagged on their own resource pages rather than hidden. + +- **Only `broken` is ever removable.** `blocked` and `unknown` are not, at any + flag, and `--remove-broken` prints every entry before touching anything. +- **`--fail-on-new-broken` is the gate**, run daily by `link-check.yml`. It + compares against the committed report, so a link that was already gone does + not keep failing the run while one that broke last night does. +- **Re-checking cannot find a cataloguing error.** The entry for WHO's economic + case for investing in mental health pointed at + `who.int/publications/i/item/9789241511810`; that ISBN is a standard + operating procedure for measuring nicotine in cigarette smoke. The URL was + wrong the day it was written. + +## Duplicates are merged, not dropped + +`build.py` used to keep the first entry under a normalised title and discard +the rest silently: 24 of 669 entries, and **all 21 affected titles spanned more +than one topic**. What it discarded was a curator's topic assignment. +"Conditional Cash Transfers: Reducing Present and Future Poverty" is filed +under both Public Policy & Governance and Social Protection; after +deduplication it existed only under the first, because that file sorts earlier, +and a reader filtering to Social Protection could not find it. The count in +`stats.json` agreed with itself throughout. + +Eleven of the 21 also carried **different URLs** under the same title, usually +a working paper beside the published version. Dropping one lost a real link. + +So `merge_duplicates()` keeps one record with `topics` carrying every topic and +`alt_urls` carrying the other links. `topic` stays as the first topic, so +anything reading a scalar keeps working. Restoring those assignments moved +Research Methods from 46 to 53, Public Policy & Governance from 50 to 55 and +Livelihoods from 49 to 54. + +**If you add a consumer of topic, read `topics`.** `topicsOf()` in `js/app.js` +and `js/resource.js` is the accessor; `build_stats` counts across the list. + +## The badge that could never appear + +`resource.html` carried a "Verified" badge driven by `r.verified`, and +`enrich_data.py` set that field to `False` on every resource on every build. +The branch was unreachable, and the word promised an editorial judgement the +pipeline never made. It is gone. What the pipeline actually knows is +`link_status` and `link_checked`, written by `verify_urls.py` and deliberately +not touched by `enrich_data.py`. + +## Colour + +`make contrast` reads the token blocks out of **both** stylesheets, in the +order the pages load them, and measures every ink against every surface in both +themes. It gates pull requests because it needs no browser and no network. + +Two things it caught and two it could not. + +- `--color-text-muted` was `#8a8a8a`, **3.28:1** on the page background, across + twelve rules: every card byline, count and helper line. `--color-book` and + `--color-grey` were similar. +- `.resource-type-badge` used tokens and `.access-badge` was a **second copy of + the same system written as literals**, and neither had a dark value. In dark + mode an Open Access badge painted `#059669` on `#ecfdf5`, a pale mint chip on + a near-black card, at about 1.7:1. The fills are `--badge-*-bg` tokens now, + which is what makes them checkable at all. +- **What it could not see**: `color: white` on `background: var(--color-accent)` + in seven rules. The accent is a *fill* there, not a surface, and `white` is a + literal. Light theme 7.51:1, dark theme **2.22:1**, because the kit re-aliases + the accent to `#6cb2ff`. Every filled pill, the Read at Source button, the + active citation tab and the saved toggle were failing in dark mode as + shipped. It is `--color-on-accent` now, and `FILLS` in the script measures + that direction too. +- **The header scrolled sideways on a phone.** `.header-nav` was a flex row + with no wrap: 545px inside a 390px viewport, 179px of horizontal scroll. It + looks perfect on a desktop, which is why it survived. + +The last two were found by running axe over the built pages in both themes at +both viewports. A token check and a browser audit are not substitutes for each +other; run both when you touch colour. + +## Watch out for + +- **The service worker shell is cache-first.** Change any of `index.html`, + `resource.html`, `css/*` or `js/*` and bump `CACHE_VERSION` in + `service-worker.js`, or a returning visitor keeps the old files. The + 2026-09-23 change added markup that only the new JS fills and only the new + CSS styles; without the bump the feature would have been invisible to + everyone who had visited before. +- **A `