Expand library to 645 resources; add faceted filters, PWA, and build tooling

.github/workflows/ci.yml

@@ -7,6 +7,32 @@ on:
     branches: [main]
 
 jobs:
+  build-and-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dev dependencies
+        run: pip install pytest
+
+      - name: Build data pipeline
+        run: |
+          python enrich_data.py
+          python build.py --validate --stats
+
+      - name: Verify build output is committed (no drift)
+        run: |
+          git diff --exit-code data/resources.json data/stats.json data/search-index.json sitemap.xml feed.json opensearch.xml \
+            || (echo "::error::Generated files are stale. Run 'make build' and commit the result." && exit 1)
+
+      - name: Run tests
+        run: python -m pytest -q
+
   check-links:
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -16,3 +16,10 @@ node_modules/
 build/
 dist/
 coverage/
+
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.venv/
+venv/

Makefile

@@ -0,0 +1,41 @@
+# Development Discourses - build & maintenance tasks
+# Pure standard-library Python; no third-party runtime dependencies.
+
+PYTHON ?= python3
+PORT   ?= 8000
+
+.DEFAULT_GOAL := help
+
+.PHONY: help build enrich assets stats validate verify serve test clean
+
+help: ## Show this help
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
+		awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-10s\033[0m %s\n", $$1, $$2}'
+
+build: ## Enrich topic files, merge into resources.json, generate assets
+	$(PYTHON) enrich_data.py
+	$(PYTHON) build.py --stats
+
+enrich: ## Add ids, access types, tags to topic files
+	$(PYTHON) enrich_data.py
+
+assets: ## Regenerate sitemap.xml, feed.json, opensearch.xml, stats.json
+	$(PYTHON) generate_assets.py
+
+stats: ## Print library statistics
+	$(PYTHON) build.py --stats --dry-run
+
+validate: ## Validate topic files without writing
+	$(PYTHON) build.py --validate --dry-run
+
+verify: ## Check that all resource URLs are reachable
+	$(PYTHON) verify_urls.py
+
+serve: ## Serve the site locally at http://localhost:$(PORT)
+	$(PYTHON) -m http.server $(PORT)
+
+test: ## Run the test suite
+	$(PYTHON) -m pytest -q
+
+clean: ## Remove generated report artifacts
+	rm -f url_verification_report.json

README.md

@@ -6,7 +6,7 @@
 [![GitHub Last Commit](https://img.shields.io/github/last-commit/Varnasr/development-discourses)](https://github.com/Varnasr/development-discourses/commits/main)
 [![Part of ImpactMojo](https://img.shields.io/badge/Part%20of-ImpactMojo-orange)](https://www.impactmojo.in)
 
-**A curated open-access library of 500+ research papers, books, and grey literature for development practitioners in South Asia.**
+**A curated open-access library of 640+ research papers, books, and grey literature for development practitioners in South Asia.**
 
 Searchable, filterable, and fully free — part of the [ImpactMojo](https://www.impactmojo.in) learning platform.
 
@@ -30,39 +30,66 @@ Every resource is open-access, verified, and contextualised for South Asian deve
 | **Climate & Environment** | Climate resilience, adaptation finance, just transition, environmental justice |
 | **Public Health** | Universal health coverage, nutrition, community health workers, WASH |
 | **Public Policy & Governance** | Decentralisation, digital governance, RTI, social accountability |
-| **Data & Technology** | Data governance, open data, AI ethics, digital development |
+| **Data & Technology** | Data governance, open data, AI ethics, digital public infrastructure |
 | **Livelihoods** | Self-help groups, skills development, migration, rural livelihoods |
 | **Research Methods** | RCTs, mixed methods, participatory action research, ethnography |
+| **Energy & Just Transition** | Energy access, renewables, clean cooking, just transition of work |
+| **Mental Health & Wellbeing** | Global mental health, psychosocial support, the treatment gap |
+
+Plus Agriculture & Food Systems, Conflict & Humanitarian, Disability & Inclusion, Education, Financial Inclusion, Migration & Urbanization, Social Protection, and WASH — **19 topics** in all.
 
 ---
 
 ## Features
 
 | Feature | Description |
 |---------|-------------|
-| **Search** | Full-text search across titles, authors, abstracts, and tags |
-| **Filter** | Filter by theme, resource type, region, publication year |
-| **Resource Types** | Academic papers, books, reports, toolkits, grey literature, policy briefs |
-| **Open Access Only** | Every resource is freely and legally accessible |
+| **Search** | Full-text search across titles, authors, abstracts, and tags, with match highlighting and a relevance-ranked "Best Match" sort |
+| **Faceted filters** | Filter by topic, resource type, **access level**, and **publication decade** — with live counts and one-click removable filter chips |
+| **Shareable state** | Every search and filter combination is encoded in the URL, so any view can be bookmarked or shared |
+| **Export** | Download the current filtered list as **BibTeX**, **CSV**, or **JSON** |
+| **Connection Map** | An interactive, force-directed graph of related resources on every detail page (pure SVG, no dependencies) |
+| **Citations** | One-click APA, BibTeX, and Chicago citations per resource |
+| **Recently viewed** | Your last-viewed resources are remembered locally and surfaced on detail pages |
+| **Surprise me** | Jump to a random resource from the current selection |
+| **Light / dark / system theme** | Respects your OS preference and remembers your choice |
+| **Installable & offline** | A Progressive Web App with a service worker — install it and browse offline |
+| **Open access first** | Every resource is tagged *Open Access*, *Free to Read*, or *Check Access* |
 | **Verified URLs** | All links validated by `verify_urls.py` — broken links flagged automatically |
-| **Zero Dependencies** | Vanilla HTML/CSS/JS — no build step required |
+| **Zero runtime dependencies** | Vanilla HTML/CSS/JS on the front end; standard-library Python for the build |
 
 ---
 
 ## Project Structure
 
 ```
 development-discourses/
-├── index.html              # Main library interface (search + filter)
-├── resource.html           # Individual resource view
-├── data/                   # Resource database (JSON/CSV)
-├── css/                    # Stylesheets
-├── js/                     # Search and filter logic
-├── build.py                # Builds the static site from data files
+├── index.html              # Main library interface (search + faceted filters)
+├── resource.html           # Individual resource view + connection map
+├── manifest.webmanifest    # PWA manifest
+├── service-worker.js       # Offline caching (app shell + data)
+├── icon.svg                # App / favicon icon
+├── sitemap.xml             # Generated — search-engine sitemap
+├── feed.json               # Generated — JSON Feed of recent resources
+├── opensearch.xml          # Generated — browser search integration
+├── data/
+│   ├── topics/*.json       # Source of truth: one file per topic
+│   ├── resources.json      # Generated — merged, deduplicated library
+│   ├── search-index.json   # Generated — trimmed index
+│   └── stats.json          # Generated — aggregate counts
+├── css/style.css           # Stylesheet (light + dark themes)
+├── js/
+│   ├── app.js              # Library: search, filters, export, URL state
+│   ├── resource.js         # Detail page: citations, SEO, recently viewed
+│   └── litmap.js           # Force-directed connection graph (SVG)
+├── build.py                # Merge topic files → resources.json + assets
+├── enrich_data.py          # Add ids, access types, DOIs, tags
+├── generate_assets.py      # Build sitemap, feed, opensearch, stats
 ├── add_resource.py         # CLI tool to add new resources
-├── enrich_data.py          # Enriches resource metadata (abstracts, tags)
-├── verify_urls.py          # Checks all URLs are live and accessible
-├── url_verification_report.json  # Latest URL health check
+├── verify_urls.py          # Check all URLs are live and accessible
+├── tests/                  # pytest data-integrity suite
+├── Makefile                # build / test / serve / verify tasks
+├── pyproject.toml          # Project + dev-tooling config
 ├── LICENSE
 └── README.md
 ```
@@ -82,7 +109,13 @@ python add_resource.py --title "Resource Title" \
   --year 2024
 ```
 
-Or edit the data files directly and run `python build.py` to regenerate the static site.
+Or edit the topic files in `data/topics/` directly, then regenerate everything with one command:
+
+```bash
+make build      # enrich topic files → resources.json → sitemap/feed/stats/index
+```
+
+Run `make help` to see all available tasks (`build`, `enrich`, `assets`, `stats`, `validate`, `verify`, `serve`, `test`, `clean`).
 
 ---
 
@@ -118,11 +151,32 @@ python3 -m http.server 8000
 
 | Layer | Technology | Purpose |
 |-------|-----------|---------|
-| Frontend | Vanilla HTML / CSS / JavaScript | Zero-dependency search and browse interface |
-| Data | JSON / CSV | Resource metadata and content |
-| Build | Python (`build.py`) | Generates static HTML from data files |
+| Frontend | Vanilla HTML / CSS / JavaScript | Zero-dependency search, faceted filtering, export, and theming |
+| Offline | Service Worker + Web App Manifest | Installable PWA with offline caching |
+| Discovery | JSON-LD structured data, Open Graph, sitemap, JSON Feed, OpenSearch | SEO and machine-readable metadata |
+| Data | JSON (one file per topic) | Resource metadata, merged and deduplicated at build time |
+| Build | Python stdlib (`enrich_data.py`, `build.py`, `generate_assets.py`) | Enrich, merge, and generate derived assets |
+| Testing | pytest | Data-integrity and asset-generation checks |
+| CI | GitHub Actions | Build validation, drift check, tests, and link checking |
 | Verification | Python (`verify_urls.py`) | URL health checks |
 
+No runtime dependencies: the front end ships as static files, and the entire
+Python pipeline uses only the standard library. `pytest` is the sole (dev-only)
+dependency.
+
+---
+
+## Testing
+
+```bash
+pip install pytest      # dev dependency only
+make test               # or: python3 -m pytest -q
+```
+
+The suite validates that every topic entry has the required fields, that
+resource IDs and titles are unique, that access types are valid, and that the
+asset generators produce well-formed output.
+
 ---
 
 ## Part of the ImpactMojo Ecosystem

build.py

@@ -168,6 +168,29 @@ def main():
 
     print(f"Wrote {len(resources)} resources to {OUTPUT_FILE}")
 
+    if "--no-assets" not in args:
+        generate_assets(resources)
+
+
+def generate_assets(resources):
+    """Generate derived assets (sitemap, feed, stats...) after a successful build."""
+    try:
+        import generate_assets as ga
+    except Exception as e:  # pragma: no cover - optional step
+        print(f"Skipping asset generation ({e}).")
+        return
+
+    base_url = ga.resolve_base_url(None)
+    print(f"Generating site assets (base URL: {base_url})...")
+    ga.write(os.path.join(SCRIPT_DIR, "sitemap.xml"), ga.build_sitemap(resources, base_url), False)
+    ga.write(os.path.join(SCRIPT_DIR, "feed.json"),
+             json.dumps(ga.build_feed(resources, base_url), indent=2, ensure_ascii=False) + "\n", False)
+    ga.write(os.path.join(SCRIPT_DIR, "opensearch.xml"), ga.build_opensearch(base_url), False)
+    ga.write(os.path.join(SCRIPT_DIR, "data", "stats.json"),
+             json.dumps(ga.build_stats(resources), indent=2, ensure_ascii=False) + "\n", False)
+    ga.write(os.path.join(SCRIPT_DIR, "data", "search-index.json"),
+             json.dumps(ga.build_search_index(resources), ensure_ascii=False) + "\n", False)
+
 
 if __name__ == "__main__":
     main()