Context
The scrapers appeared stuck for several days with no obvious errors in the logs. Right now the service reports progress counts, but it does not explicitly detect or surface the "alive but not making progress" state.
Why this looks plausible
lib/processing-stats.ts logs processed counts, elapsed time, and throughput.
lib/prometheus.ts exposes counters and request histograms, but nothing tracks last-success time, oldest in-flight task, or stalled state.
- If progress stops while the process is still running, we do not emit a clear signal that the scraper is wedged.
Proposed fix
- Track
lastProgressAt / lastSuccessAt per service.
- Track in-flight task count and oldest in-flight age.
- Emit a dedicated warning/error when no progress has occurred for a configurable interval while work is still pending.
- Add Prometheus gauges for stalled state and last successful scrape timestamp.
- Optionally make prolonged stall detection fatal so the service self-recovers instead of waiting for a manual restart.
Acceptance criteria
- Operators can distinguish "empty queue" from "stuck queue" quickly.
- Prometheus can alert on stale
lastSuccessAt or a stalled=1 gauge.
- Logs clearly identify the first moment a service stops making progress.
Relevant code
lib/processing-stats.ts
lib/prometheus.ts
services/metadata/run.ts
Context
The scrapers appeared stuck for several days with no obvious errors in the logs. Right now the service reports progress counts, but it does not explicitly detect or surface the "alive but not making progress" state.
Why this looks plausible
lib/processing-stats.tslogs processed counts, elapsed time, and throughput.lib/prometheus.tsexposes counters and request histograms, but nothing tracks last-success time, oldest in-flight task, or stalled state.Proposed fix
lastProgressAt/lastSuccessAtper service.Acceptance criteria
lastSuccessAtor astalled=1gauge.Relevant code
lib/processing-stats.tslib/prometheus.tsservices/metadata/run.ts