Auto-updating tracker of Indian development policy across 22 sectors — fetching from ~350 official sources concurrently every 6 hours via GitHub Actions, with a news filter and a policy-forward homepage.
A complement to impactmojo.in and the OpenStacks for Change ecosystem.
Live at varnasr.github.io/PolicyDhara
PolicyDhara (पॉलिसीधारा — "Policy Stream") provides a living, searchable archive of Indian development policy. It aggregates policy documents, schemes, legislation, budget allocations, and analysis from official government portals and research institutions — automatically refreshed every 6 hours.
It is designed for researchers, journalists, advocates, and practitioners who need to track policy developments across India's complex governance landscape without manually monitoring dozens of government portals.
- Central and state government policies, schemes, and legislation
- Union Budget allocations and fiscal policy statements
- Parliament bills, session data, and legislative productivity
- Notifications, gazette circulars, and executive orders
- Think tank analysis and international development body reports
| Cluster | Sectors |
|---|---|
| Social | Education, Health, Gender, Social Protection, Nutrition |
| Economy | Finance, Agriculture, Labour, Infrastructure, Digital |
| Environment | Climate, Water, Energy, Environment |
| Governance | Democracy, Legal, Defence, Urban, Rural |
| International | Trade, Foreign Policy, Development Cooperation |
The pipeline fetches from ~350 sources concurrently. A few high-value ones:
| Source | Type |
|---|---|
| PIB (Press Information Bureau) | Government press releases |
| PRS Legislative Research | Bills, acts, and parliament analysis |
| India Code | Legislation database |
| eGazette of India | Official gazette notifications |
| NITI Aayog | Policy reports and flagship scheme updates |
| RBI | Monetary policy, financial sector regulation |
| Supreme Court of India | Landmark judgements |
| ORF | Strategic policy analysis |
| CPR India | Governance and accountability research |
| ICRIER | Trade, investment, and economic policy |
| World Bank India | Development policy assessments |
| UNDP India | SDG and human development updates |
┌──────────────────────────────────────────────────────┐
│ GitHub Actions (every 6 hours) │
│ │
│ update-policies.yml │
│ ├── Fetch ~350 sources concurrently (RSS/API/scrape) │
│ ├── Filter news noise + cap media at 40% │
│ ├── Classify by sector (22 categories) │
│ ├── Deduplicate and detect amendments │
│ ├── Commit updated data to repository │
│ └── Trigger GitHub Pages rebuild │
└──────────────────────────────────────────────────────┘
│
GitHub Pages (static hosting)
│
┌──────────────────────────────────────────────────────┐
│ varnasr.github.io/PolicyDhara │
│ ├── Searchable policy database │
│ ├── Sector filters (22 categories) │
│ ├── Source attribution for every entry │
│ └── Email digest for subscribers │
└──────────────────────────────────────────────────────┘
| Task | Frequency | Mechanism |
|---|---|---|
| Policy data refresh | Every 6 hours | update-policies.yml (GitHub Actions) |
| Site deploy | On data commit | deploy.yml (GitHub Pages) |
| PyPI package publish | On release | publish-pypi.yml |
| Newsletter (optional) | On data refresh | scripts/send_newsletter.py (Buttondown) |
| Telegram alerts (optional) | On data refresh | scripts/push_telegram.py |
Both run as best-effort steps in update-policies.yml and skip silently if the relevant secrets are not set.
Buttondown newsletter — set BUTTONDOWN_API_KEY as a GitHub Secret.
Telegram alerts for high-priority new policies — set:
TELEGRAM_BOT_TOKEN— from @BotFather (/newbot)TELEGRAM_CHAT_ID— channel@username(e.g.@policydhara) or numeric chat id
Add the bot as an administrator on the channel with "Post Messages" permission. Test locally with python3 scripts/push_telegram.py --dry-run.
git clone https://github.com/Varnasr/PolicyDhara.git
cd PolicyDhara
# Python pipeline
pip install -r scripts/requirements.txt
python3 scripts/fetch_all.py # fetch + filter + classify
python3 scripts/check_source_health.py # bucket every source (WORKS/REDUNDANT/SHELL/DEAD)
# Astro site
npm install
npm run dev # local dev server
npm run build # production build (Astro + Pagefind)
# Run the test suite
python3 -m pytest tests/| Layer | Technology | Purpose |
|---|---|---|
| Data Pipeline | Python | Concurrent fetching, news filter, classification, dedup |
| Automation | GitHub Actions | Scheduled data refresh (6-hourly) and deploy |
| Frontend | Astro 7 | Static searchable site (+ Pagefind search) |
| Hosting | GitHub Pages | Auto-deploy from repository |
| Package | PyPI (policydhara) |
Python library for the policy data pipeline |
- ImpactMojo — Main platform (PolicyDhara is embedded)
- PolicyStack — OpenStacks policy data layer
- INDIA-DATA — Bihar electoral data analysis
- someperspective — India's political economy analysis
MIT License — see LICENSE for details.
- Platform: impactmojo.in
- GitHub: github.com/Varnasr