Procurement-Crawler

A modular Playwright crawler system for procurement notices, bid data, and project information from government and international procurement platforms. Crawlers are registry-based, configuration-driven, and export to JSON, CSV, and Excel.

Crawler	Source	What it collects
ungm	United Nations Global Marketplace	Procurement notices, ASR highlights, homepage statistics
fredericksburg	City of Fredericksburg, VA	Active/awarded bids and award notices
worldbank	World Bank e-RFX	Procurement notices with detail pages
iadb	Inter-American Development Bank	Project listings (paginated search table)
iadb_procurement	IADB procurement portal	Procurement notices via the public IADB API, or saved PowerBI exports in import mode
capture	Any URL (configurable)	Raw HTML + full-page screenshot per URL, with a JSON manifest
disa_stigs	DISA STIG document library (public.cyber.mil)	STIG document title, link, size, updated date (paginated table)
stigviewer	stigviewer.com/stigs	STIG title, version, updated date, link

Install

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install .
python -m playwright install chromium

Optional extras:

pip install ".[google]"   # Google Sheets sync (modules/google)
pip install ".[dev]"      # pytest + ruff

Run

procurement-crawler --list-crawlers
procurement-crawler --crawler ungm --max-scroll 10
procurement-crawler --crawler fredericksburg --max-details 50 --max-workers 8
procurement-crawler --crawler iadb --max-scroll 5 --formats json csv

Without installing, from the project root:

python main.py --crawler worldbank --max-scroll 5

Output goes to timestamped files under output/<crawler>/. Logs go to logs/crawler.log (created automatically).

Capture pages, or scrape the STIG listings (folded in from ss-playwright; see CHANGELOG.md for what was ported and what was dropped):

procurement-crawler --crawler capture --urls https://example.com https://example.org
procurement-crawler --crawler disa_stigs
procurement-crawler --crawler stigviewer

capture URLs, device emulation, storage state, and cookies live in config/crawlers/capture.json; --urls overrides the configured list. For interactive selector debugging, set pause: true in that file and run with PWDEBUG=1.

Import a saved IADB procurement export instead of crawling:

python scripts/import_iadb_procurement.py --input-file export.csv
python scripts/process_iadb_auto.py   # scans the dirs in .env.example

Replay a captured UNGM Search API sample instead of crawling (manual only, hits the live site; use --delay 2 for the original 0.5 req/s pace):

python scripts/replay_ungm_search.py samples-dir --delay 2

samples-dir holds an api-samples.json captured from a browser run: the Public/Notice/Search POST template is re-issued page by page with no browser, and the notice URLs are written to notice-urls.json. See crawlers/ungm_search.py for the format and the fixes over the original ss-ungm helper.

Configuration

One default file, config/default.json, plus one JSON file per crawler in config/crawlers/. Values may reference environment variables with ${VAR} or $VAR syntax; unset variables become empty strings with a warning. The registry and config file resolve relative to the project root, so the installed procurement-crawler command works from any directory.

Sensitive settings live in the environment or a .env file, never in the repo:

cp .env.example .env   # fill in real values, then: set -a; source .env; set +a

• PROCUREMENT_CRAWLER_EMAIL_USERNAME / PROCUREMENT_CRAWLER_EMAIL_APP_PASSWORD / PROCUREMENT_CRAWLER_EMAIL_RECIPIENTS: SMTP completion and error notifications (Gmail app password, not your account password).
• PROCUREMENT_CRAWLER_GOOGLE_CREDENTIALS_PATH: service-account JSON for Google Sheets sync (disabled unless configured).

See QUICK_START.md and docs/SERVICES_SETUP.md for the full setup. Run bash scripts/setup_services.sh for a setup check that writes no credentials.

Architecture

• crawlers/: BaseCrawler plus one crawler per source, discovered by config/crawler_registry.py from config/crawlers/*.json. crawlers/ungm_search.py replays captured UNGM Search API samples with no browser; crawlers/iadb_importer.py imports saved IADB PowerBI exports.
• data_processors/: cleaning and validation of extracted records.
• exporters/: JSON/CSV/Excel export, with per-source formatting.
• utils/: text, URL, and browser helpers (including randomised delays and a rotating modern user-agent list).
• modules/notifications/: SMTP notification channels and Jinja email templates.
• modules/google/: optional Google Sheets sync (lazy imports; needs the google extra).

Verification status (2026-10-28)

Checked, not assumed. The IADB project-search listing still renders the same 8-column table (project number, operation, country, sector, title, total cost, status, approval date), 10 rows per page, and ?page=1 returns the next 10 rows in date order. The IADB row parser is covered by fixture tests against saved listing pages (see tests/fixtures/). The projects-json endpoint returns aggregate totals only, so it is not used for row extraction.

Not live-verified in this build: UNGM, World Bank, and Fredericksburg selectors (saved-page parsing only), the UNGM Search API pagination (fixture tests with fake responses only; a hand-run replay against a real captured sample is the gate), the DISA STIG and stigviewer.com selectors (fake-page tests only; a hand-run crawl is the gate), the IADB APIM procurement-notice API, SMTP delivery, and Google Sheets sync. A full live crawl is the remaining gate before relying on any one crawler's output.

Known limitations, kept honest in the code:

• IADB Procurement PowerBI authentication is not implemented. The API mode and the CSV/XLSX import mode are the working paths; the browser-mode _extract_* methods return empty lists and log that they are unimplemented. Expected Processes extraction is unavailable.
• Parallel IADB page fetches are deduplicated by project number (page-order, first copy wins), because the listing is date-ordered and can shift between requests.
• Respect each site's terms of service and robots rules before crawling. The crawlers use randomised delays and a single browser user agent per run; they do not try to evade access controls.

Development

pip install ".[dev]"
python -m pytest -q
ruff check .

Tests cover config/env substitution, registry discovery from any working directory, the IADB row mapper and dedup on saved fixtures, the IADB importer, the UNGM Search replay on a fictional fixture with fake responses, the ss-playwright fold-in (capture filename/manifest handling, DISA pagination and row cleaning, stigviewer row normalisation, STIG exporters) on fake pages, exporters, helpers, and email template rendering. No test launches a browser or touches the network.

Changelog

See CHANGELOG.md. v2.0.0 is a squashed, single-commit export of the modernised tree; the original private history is intentionally not included. v2.1.0 adds the ss-ungm Search replay and v2.2.0 adds the ss-playwright fold-in, each as one commit on top of it.

License

MIT. See LICENSE.