Stop the link checker deleting live links, and merge duplicates instead of dropping them

.github/workflows/ci.yml

@@ -33,14 +33,8 @@ jobs:
       - name: Run tests
         run: python -m pytest -q
 
-  check-links:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-
-      - name: Link Checker
-        uses: lycheeverse/lychee-action@v2
-        with:
-          args: --accept 200,204,301,302,403,429 --exclude-mail --exclude "localhost|127.0.0.1|example.com|twitter.com|x.com|linkedin.com|facebook.com|instagram.com|t.me|wa.me|play.google.com|apps.apple.com" --timeout 30 --max-retries 2 "**/*.html" "**/*.md"
-          fail: false
-
+      # Reads the two token blocks out of css/style.css and measures every ink
+      # against every surface it can land on. No browser and no network, so it
+      # belongs in front of a pull request rather than on a schedule.
+      - name: Colour contrast
+        run: python check_contrast.py

.github/workflows/link-check.yml

@@ -0,0 +1,66 @@
+# Link health, on a clock rather than on a pull request.
+#
+# Two reasons it is not in ci.yml.
+#
+# A link rots with no commit behind it, so a gate that only runs on a pull
+# request cannot see the failure it exists for. And a job that talks to six
+# hundred third-party hosts is long: a job cancelled by its own timeout-minutes
+# cancels the whole workflow run, taking the build and the tests down with it
+# and reporting the run as cancelled rather than failed.
+#
+# The lychee step used to live in ci.yml on pull requests with `fail: false`,
+# so a contributor got a result they could neither act on nor be blocked by.
+# It also crawled `**/*.html` and `**/*.md`, which is the site chrome: the 644
+# links this library exists to hold are in data/topics/*.json and were outside
+# the crawl entirely.
+
+name: Link health
+
+on:
+  schedule:
+    - cron: "17 5 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  resource-urls:
+    name: The library's own 644 links
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      # --fail-on-new-broken compares against the committed report, so a link
+      # that was already gone does not keep failing the run while one that
+      # broke last night does. --no-write because a scheduled run should not
+      # try to commit; a maintainer runs `make verify` and commits the result.
+      - name: Check every resource URL
+        run: python verify_urls.py --no-write --fail-on-new-broken --workers 8
+
+  site-chrome:
+    name: Links in the pages themselves
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Link Checker
+        uses: lycheeverse/lychee-action@v2
+        with:
+          # 403 and 429 are accepted here because this crawl has no way to tell
+          # a publisher refusing a robot from a document that is gone. The job
+          # above does have that distinction, which is why it is the one that
+          # covers the resources.
+          args: >-
+            --accept 200,204,301,302,403,429
+            --exclude-mail
+            --exclude "localhost|127.0.0.1|example.com|twitter.com|x.com|linkedin.com|facebook.com|instagram.com|t.me|wa.me|play.google.com|apps.apple.com"
+            --timeout 30 --max-retries 2
+            "**/*.html" "**/*.md"
+          fail: true

CLAUDE.md

@@ -0,0 +1,164 @@
+# development-discourses
+
+A curated open-access library: 645 research papers, books and grey literature
+for development practitioners in South Asia, across 19 topics. Static HTML, no
+build step for the part that runs, a standard-library Python pipeline for the
+part that generates. Deployed to GitHub Pages under `/development-discourses/`.
+
+## Commands
+
+```bash
+make build       # enrich topic files, merge, regenerate every derived asset
+make test        # 31 checks
+make contrast    # every ink token against every surface, both themes
+make verify      # check every resource URL, write the report and link_status
+make recheck     # re-check only what was not ok last time (much faster)
+make serve       # http://localhost:8000
+```
+
+There are no dependencies. `pytest` is the only dev one. The front end is
+vanilla HTML, CSS and JavaScript; the pipeline is the Python standard library.
+Keep it that way.
+
+## One source, five derived files
+
+`data/topics/*.json` is the only thing anyone edits, one file per topic.
+`build.py` merges them into `data/resources.json`, and `generate_assets.py`
+derives `data/search-index.json`, `data/stats.json`, `sitemap.xml`, `feed.json`
+and `opensearch.xml`. CI fails if any of those has drifted from the source, so
+**run `make build` and commit the result** with any data change.
+
+## A 403 is not a 404, and conflating them nearly cost a fifth of the library
+
+`verify_urls.py` sorts every URL into `ok`, `paywalled`, `blocked`, `unknown`
+or `broken`. That distinction is the whole point of the script.
+
+The version before 2026-09-23 had one bucket for "not accessible" and offered
+`--remove-broken` against it. Its report of 2026-02-28 put **162 of 516** URLs
+in that bucket, and **102 of the 162 were HTTP 403**: 35 from
+documents1.worldbank.org, 26 from thelancet.com, the rest from ResearchGate,
+Science, SAGE and ScienceDirect. Every one opens in a browser. Running the flag
+would have deleted a fifth of the library, most of it live, and the diff would
+have read as routine maintenance.
+
+It also sent `User-Agent: DevDiscourses-URLChecker/1.0`, which is close to the
+perfect string for getting blocked, so the report described the checker rather
+than the web.
+
+Re-run honestly: 644 unique URLs, 25 genuinely broken. Seventeen were traced to
+their current home and replaced, each probed before it went in; eight remain
+and are flagged on their own resource pages rather than hidden.
+
+- **Only `broken` is ever removable.** `blocked` and `unknown` are not, at any
+  flag, and `--remove-broken` prints every entry before touching anything.
+- **`--fail-on-new-broken` is the gate**, run daily by `link-check.yml`. It
+  compares against the committed report, so a link that was already gone does
+  not keep failing the run while one that broke last night does.
+- **Re-checking cannot find a cataloguing error.** The entry for WHO's economic
+  case for investing in mental health pointed at
+  `who.int/publications/i/item/9789241511810`; that ISBN is a standard
+  operating procedure for measuring nicotine in cigarette smoke. The URL was
+  wrong the day it was written.
+
+## Duplicates are merged, not dropped
+
+`build.py` used to keep the first entry under a normalised title and discard
+the rest silently: 24 of 669 entries, and **all 21 affected titles spanned more
+than one topic**. What it discarded was a curator's topic assignment.
+"Conditional Cash Transfers: Reducing Present and Future Poverty" is filed
+under both Public Policy & Governance and Social Protection; after
+deduplication it existed only under the first, because that file sorts earlier,
+and a reader filtering to Social Protection could not find it. The count in
+`stats.json` agreed with itself throughout.
+
+Eleven of the 21 also carried **different URLs** under the same title, usually
+a working paper beside the published version. Dropping one lost a real link.
+
+So `merge_duplicates()` keeps one record with `topics` carrying every topic and
+`alt_urls` carrying the other links. `topic` stays as the first topic, so
+anything reading a scalar keeps working. Restoring those assignments moved
+Research Methods from 46 to 53, Public Policy & Governance from 50 to 55 and
+Livelihoods from 49 to 54.
+
+**If you add a consumer of topic, read `topics`.** `topicsOf()` in `js/app.js`
+and `js/resource.js` is the accessor; `build_stats` counts across the list.
+
+## The badge that could never appear
+
+`resource.html` carried a "Verified" badge driven by `r.verified`, and
+`enrich_data.py` set that field to `False` on every resource on every build.
+The branch was unreachable, and the word promised an editorial judgement the
+pipeline never made. It is gone. What the pipeline actually knows is
+`link_status` and `link_checked`, written by `verify_urls.py` and deliberately
+not touched by `enrich_data.py`.
+
+## Colour
+
+`make contrast` reads the token blocks out of **both** stylesheets, in the
+order the pages load them, and measures every ink against every surface in both
+themes. It gates pull requests because it needs no browser and no network.
+
+Two things it caught and two it could not.
+
+- `--color-text-muted` was `#8a8a8a`, **3.28:1** on the page background, across
+  twelve rules: every card byline, count and helper line. `--color-book` and
+  `--color-grey` were similar.
+- `.resource-type-badge` used tokens and `.access-badge` was a **second copy of
+  the same system written as literals**, and neither had a dark value. In dark
+  mode an Open Access badge painted `#059669` on `#ecfdf5`, a pale mint chip on
+  a near-black card, at about 1.7:1. The fills are `--badge-*-bg` tokens now,
+  which is what makes them checkable at all.
+- **Three `#8a8a8a` literals in JavaScript survived the token pass**, because
+  `make contrast` reads CSS. Two in `js/resource.js` (the "no resource
+  specified" line and the Unknown access badge) and one in `js/litmap.js` (the
+  "no strong connections" message drawn into the SVG). That is the same colour
+  the CSS pass replaced at 3.28:1, still being written at runtime. Grep the
+  JavaScript too when you move a colour.
+- **What it could not see**: `color: white` on `background: var(--color-accent)`
+  in seven rules. The accent is a *fill* there, not a surface, and `white` is a
+  literal. Light theme 7.51:1, dark theme **2.22:1**, because the kit re-aliases
+  the accent to `#6cb2ff`. Every filled pill, the Read at Source button, the
+  active citation tab and the saved toggle were failing in dark mode as
+  shipped. It is `--color-on-accent` now, and `FILLS` in the script measures
+  that direction too.
+- **The header scrolled sideways on a phone.** `.header-nav` was a flex row
+  with no wrap: 545px inside a 390px viewport, 179px of horizontal scroll. It
+  looks perfect on a desktop, which is why it survived.
+
+The last two were found by running axe over the built pages in both themes at
+both viewports. A token check and a browser audit are not substitutes for each
+other; run both when you touch colour.
+
+## Watch out for
+
+- **The service worker shell is cache-first.** Change any of `index.html`,
+  `resource.html`, `css/*` or `js/*` and bump `CACHE_VERSION` in
+  `service-worker.js`, or a returning visitor keeps the old files. The
+  2026-09-23 change added markup that only the new JS fills and only the new
+  CSS styles; without the bump the feature would have been invisible to
+  everyone who had visited before.
+- **A `<label>` cannot name a `<div>`.** Four filter groups were
+  `<label>Topic</label><div class="filter-pills">`, which associates nothing, so
+  a screen reader met four unnamed groups of buttons announcing "All", "All",
+  "Any Year". They are `role="group"` with `aria-labelledby` now, and the
+  selected pill carries `aria-pressed` rather than only a CSS class.
+- **`a { text-decoration: none }` is wrong inside a paragraph.** A link
+  distinguished from its surrounding text by colour alone fails WCAG 1.4.1.
+  Underlined in `.about-content p`, the footer tagline and the link notes.
+- **`title` is not an accessible name.** It shows on no touch device and is
+  announced unreliably. Icon-only buttons carry `aria-label`.
+- **Link rot needs no commit**, which is why `link-check.yml` is on a daily
+  schedule and not on pull requests. It also runs in its own workflow: a job
+  cancelled by its own `timeout-minutes` cancels the whole run, so a long
+  third-party crawl sharing a run with the build would take the build down
+  with it.
+- **The lychee step used to crawl the wrong thing.** It globbed `**/*.html` and
+  `**/*.md`, which is the site chrome. The 644 links this library exists to
+  hold are in `data/topics/*.json` and were outside the crawl entirely.
+
+## Testing
+
+`.github/workflows/ci.yml` runs the build, the drift check, 31 tests and the
+contrast check on every push and pull request. `.github/workflows/link-check.yml`
+runs the resource-URL check and the page crawl daily at 05:17 UTC, plus on
+demand.

Makefile

@@ -6,7 +6,7 @@ PORT   ?= 8000
 
 .DEFAULT_GOAL := help
 
-.PHONY: help build enrich assets stats validate verify serve test clean
+.PHONY: help build enrich assets stats validate verify recheck contrast serve test clean
 
 help: ## Show this help
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
@@ -28,9 +28,15 @@ stats: ## Print library statistics
 validate: ## Validate topic files without writing
 	$(PYTHON) build.py --validate --dry-run
 
-verify: ## Check that all resource URLs are reachable
+verify: ## Check every resource URL and record what came back
 	$(PYTHON) verify_urls.py
 
+recheck: ## Re-check only the URLs that were not ok last time
+	$(PYTHON) verify_urls.py --recheck
+
+contrast: ## Measure every ink token against every surface, both themes
+	$(PYTHON) check_contrast.py
+
 serve: ## Serve the site locally at http://localhost:$(PORT)
 	$(PYTHON) -m http.server $(PORT)
 

README.md

@@ -6,7 +6,7 @@
 [![GitHub Last Commit](https://img.shields.io/github/last-commit/Varnasr/development-discourses)](https://github.com/Varnasr/development-discourses/commits/main)
 [![Part of ImpactMojo](https://img.shields.io/badge/Part%20of-ImpactMojo-orange)](https://www.impactmojo.in)
 
-**A curated open-access library of 640+ research papers, books, and grey literature for development practitioners in South Asia.**
+**A curated open-access library of 645 research papers, books, and grey literature for development practitioners in South Asia.**
 
 Searchable, filterable, and fully free — part of the [ImpactMojo](https://www.impactmojo.in) learning platform.
 
@@ -49,13 +49,15 @@ Plus Agriculture & Food Systems, Conflict & Humanitarian, Disability & Inclusion
 | **Shareable state** | Every search and filter combination is encoded in the URL, so any view can be bookmarked or shared |
 | **Export** | Download the current filtered list as **BibTeX**, **CSV**, or **JSON** |
 | **Connection Map** | An interactive, force-directed graph of related resources on every detail page (pure SVG, no dependencies) |
+| **Alternate links** | Where a record carries a second URL, usually a working paper beside the published version, the detail page lists it |
 | **Citations** | One-click APA, BibTeX, and Chicago citations per resource |
 | **Recently viewed** | Your last-viewed resources are remembered locally and surfaced on detail pages |
 | **Surprise me** | Jump to a random resource from the current selection |
 | **Light / dark / system theme** | Respects your OS preference and remembers your choice |
 | **Installable & offline** | A Progressive Web App with a service worker — install it and browse offline |
 | **Open access first** | Every resource is tagged *Open Access*, *Free to Read*, or *Check Access* |
-| **Verified URLs** | All links validated by `verify_urls.py` — broken links flagged automatically |
+| **Link health** | Every URL is checked by `verify_urls.py`, and a resource page says what came back. A publisher that refuses robots is reported as *not confirmed from here*, not as broken; a document that is actually gone gets DOI, Scholar, web and Wayback links to find it again |
+| **More than one topic** | A resource curated into two topics appears under both, with every topic on the card and both filters counting it |
 | **Zero runtime dependencies** | Vanilla HTML/CSS/JS on the front end; standard-library Python for the build |
 
 ---
@@ -122,15 +124,56 @@ Run `make help` to see all available tasks (`build`, `enrich`, `assets`, `stats`
 ## URL Verification
 
 ```bash
-# Check all resource URLs
-python verify_urls.py
-
-# Results saved to url_verification_report.json
+make verify      # check every resource URL, write the report and link_status
+make recheck     # re-check only what was not ok last time (much faster)
 ```
 
-Broken links are flagged for manual review. Run periodically to maintain library quality.
+The checker sorts every URL into one of five states, and the distinction
+between the middle two is the point of it:
+
+| state | meaning |
+|---|---|
+| `ok` | 2xx after redirects. The document is there. |
+| `paywalled` | A publisher landing page. The text may be open access behind it. |
+| `blocked` | 401 / 403 / 406 / 429. The server refuses a robot. A reader is fine. |
+| `unknown` | Timeout or 5xx. Server trouble, usually transient. Retried once. |
+| `broken` | 404 / 410, DNS failure, TLS failure. The document is gone. |
+
+**Why that matters here.** The previous checker had one bucket for "not
+accessible" and offered `--remove-broken` against it. Its report of 2026-02-28
+put 162 of 516 URLs in that bucket, and **102 of the 162 were HTTP 403** — 35
+from documents1.worldbank.org, 26 from thelancet.com, the rest from
+ResearchGate, Science, SAGE and other publishers that refuse robots by policy.
+Every one opens in a browser. Running `--remove-broken` would have deleted a
+fifth of the library, most of it live, and the diff would have looked like
+maintenance.
+
+Re-run with the current checker on 2026-09-23: 644 unique URLs, **25 genuinely
+broken**. Seventeen were traced to their current home and replaced, each probed
+with a browser user-agent before it went in, leaving 551 `ok`, 77 `blocked`,
+1 `paywalled`, 7 `unknown` and 8 broken.
+
+One of the eight was never right. The entry for WHO's economic case for
+investing in mental health pointed at
+`who.int/publications/i/item/9789241511810`, and that ISBN belongs to a
+standard operating procedure for measuring nicotine in cigarette smoke. That is
+a cataloguing error rather than rot, and no amount of re-checking would have
+found it. The other seven are documents whose publisher reorganised and whose
+new home could not be established with confidence; rather than substitute a
+plausible-looking replacement, the record keeps its URL and the resource page
+says the link was unreachable and offers DOI, Scholar, web and Wayback
+searches.
+
+Only `broken` is ever removable, and `--remove-broken` prints every entry
+before it touches anything. `blocked` and `unknown` are never removable at any
+flag.
 
----
+```bash
+python3 verify_urls.py --fail-on-new-broken   # the gate link-check.yml runs
+```
+
+That compares against the committed report, so a link that was already gone
+does not keep failing the run while one that broke last night does.
 
 ## Local Development
 
@@ -157,7 +200,7 @@ python3 -m http.server 8000
 | Data | JSON (one file per topic) | Resource metadata, merged and deduplicated at build time |
 | Build | Python stdlib (`enrich_data.py`, `build.py`, `generate_assets.py`) | Enrich, merge, and generate derived assets |
 | Testing | pytest | Data-integrity and asset-generation checks |
-| CI | GitHub Actions | Build validation, drift check, tests, and link checking |
+| CI | GitHub Actions | `ci.yml` on every push and pull request: build, drift check, tests, colour contrast. `link-check.yml` daily: the 644 resource URLs and the pages' own links |
 | Verification | Python (`verify_urls.py`) | URL health checks |
 
 No runtime dependencies: the front end ships as static files, and the entire