Skip to content

Add stalled-scraper detection and last-success observability #269

Description

@DenisCarriere

Context

The scrapers appeared stuck for several days with no obvious errors in the logs. Right now the service reports progress counts, but it does not explicitly detect or surface the "alive but not making progress" state.

Why this looks plausible

  • lib/processing-stats.ts logs processed counts, elapsed time, and throughput.
  • lib/prometheus.ts exposes counters and request histograms, but nothing tracks last-success time, oldest in-flight task, or stalled state.
  • If progress stops while the process is still running, we do not emit a clear signal that the scraper is wedged.

Proposed fix

  • Track lastProgressAt / lastSuccessAt per service.
  • Track in-flight task count and oldest in-flight age.
  • Emit a dedicated warning/error when no progress has occurred for a configurable interval while work is still pending.
  • Add Prometheus gauges for stalled state and last successful scrape timestamp.
  • Optionally make prolonged stall detection fatal so the service self-recovers instead of waiting for a manual restart.

Acceptance criteria

  • Operators can distinguish "empty queue" from "stuck queue" quickly.
  • Prometheus can alert on stale lastSuccessAt or a stalled=1 gauge.
  • Logs clearly identify the first moment a service stops making progress.

Relevant code

  • lib/processing-stats.ts
  • lib/prometheus.ts
  • services/metadata/run.ts

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions