Skip to content

PR-O: make refresh — update EXISTING models' metadata from HF (crawl stays trending) - #37

Merged
andyholst merged 1 commit into
mainfrom
feat/refresh-crawler
Aug 10, 2026
Merged

andyholst merged 1 commit into
mainfrom
feat/refresh-crawler

Conversation

@andyholst

Copy link
Copy Markdown
Owner

What

make refresh — a crawler that updates the EXISTING models (user: "integrate the make command with a new crawler that crawls the models that already exist to update their data — metadata, downloads etc; then a separate make command that just crawls trending HF models").

  • make crawl — unchanged: crawls trending HF models into models/
  • make refresh (NEW, nerdctl + host network + mounts) — updates each existing model file in models/: re-fetches the HF detail (full expand set) and refreshes downloads, trending_score, name/author, license, commercial_ok, context_window, model_type, languages, knowledge_cutoff, benchmarks, servers, profile (rebuilt from the live README/config), last_updated — while preserving quants + hardware flags (file sizes are stable)
  • CLI: python -m whatllm.crawl_models --refresh [--out DIR] [--in DIR] (--in defaults to --out; index.json/meta.json artifacts skipped)
  • Tests: 5 hermetic refresh tests (mocked HF: volatile fields update, quants/hardware preserved, profile rebuilt, schema-valid emit + artifacts, missing-dir raise, artifacts skipped, parser flags) + make -n refresh dry-run test. 130 pytest · 22/22 jsdom · 7/7 Playwright · 10× openspec valid

Closes #36

#36)

User: 'integrate the make command with a new crawler that crawls all models
that already exist to update its data (metadata, downloads etc); then a
separate make command that just crawls trending HF models.'

- crawl_models.py: --refresh + --in flags; Crawler.refresh() re-fetches each
  existing model's HF detail (full expand set) and refreshes downloads,
  trending_score, name/author, license, commercial_ok, context_window,
  model_type, languages, knowledge_cutoff, benchmarks, servers, profile
  (rebuilt from live README/config), last_updated. Quants + hardware flags
  preserved (file sizes stable). index.json/meta.json skipped.
- Makefile: 'refresh' target (nerdctl, host network, models+data mounts) —
  'make crawl' stays the trending crawler; help + .PHONY updated
- Tests: tests/test_crawler_refresh.py (5 hermetic tests: volatile fields
  update, quants/hardware preserved, profile rebuilt, schema-valid emit +
  artifacts, missing-dir raise, artifacts skipped, parser flags) +
  test_makefile refresh dry-run. pytest 130, jsdom 22/22, Playwright 7/7.
Closes #36
@andyholst
andyholst merged commit 69bdf0c into main Aug 10, 2026
1 check passed
@andyholst
andyholst deleted the feat/refresh-crawler branch August 10, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

make refresh: crawler that updates EXISTING models' metadata (downloads/trending/etc) + keep make crawl = trending

2 participants