Skip to content

Add hourly Hacker News ingestion - #52

Open
Sumedhbhat wants to merge 1 commit into
masterfrom
feat/hacker-news-hourly-ingestion
Open

Add hourly Hacker News ingestion#52
Sumedhbhat wants to merge 1 commit into
masterfrom
feat/hacker-news-hourly-ingestion

Conversation

@Sumedhbhat

Copy link
Copy Markdown
Owner

Summary

  • add an hourly Cloudflare Worker ingestion job for Hacker News top, new, and best feeds
  • persist canonical stories, temporal story versions, hourly feed-rank observations, and auditable ingestion runs in D1
  • share Hacker News normalization between the live UI and server ingestion path
  • dispatch Hacker News and Hugging Face cron schedules independently
  • start with 15 stories per feed so the job remains within the Workers Free external-subrequest limit

Reliability

  • stable cross-feed deduplication and one-based ranks
  • bounded story-fetch concurrency with transient retries
  • idempotent same-hour reruns
  • preservation of dead and deleted story transitions
  • structured success and failure audit records

Validation

  • bun run check
  • bun run test: 42 files and 267 tests passed
  • bun run db:migrate:local: all five migrations applied successfully to a fresh local D1 database
  • bun run build
  • local scheduled-handler smoke test: 45 selected, 44 persisted, 1 skipped, 44 versions, and 44 observations

Deployment note

Apply the D1 migration remotely before deploying the Worker version that enables the hourly cron. The feed limit can be raised from 15 to 100 after confirming that the deployed account uses Workers Paid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant