fix: detect and log silent worker stalls (pending>0 while running=0)

[telleroutlook]

Problem

Multiple times today the system reached a state where:

• SELECT * FROM jobs WHERE state="running" returned 0 rows
• SELECT * FROM jobs WHERE state="pending" returned 7+ rows
• systemctl is-active claude-worker@1 returned active
• No log activity for minutes

This is a silent stall. Worker is technically running but not picking up work. Causes seen:

1. All pending jobs skipped by claim-time gates (provider paused, rate limit deferral, self-update pending)
2. All pending jobs were on repos not in get_executable_repos() (repos.yml corruption)
3. Migration preflight failing → worker crash-restart loop → looks "active" between restarts
4. Priority ordering picked a job whose worktree was corrupt → immediate failure → no next job attempted

Desired detection & recovery

1. Idle heartbeat with pending job count

• Worker logs INFO idle: 0 running, 7 pending, waiting Ns for next scan every N seconds when idle
• If pending > 0 and no job claimed for > 5 min → log WARNING with skip reasons per pending job

2. Skip reason logging (structured)

When claim_job iterates candidates and skips them, log why:

• skip job_id=130: provider zai-glm state=rate_limited
• skip job_id=147: repo not in executable list (repos.yml)
• skip job_id=48: hit retry_count cap

Currently these gates are silent. First evidence a job was skipped is when the queue stays full while worker idles.

3. Watchdog alarm: pending > 0 && running == 0 for > 10min

Add a bin/watchdog.sh check that files an issue (or restarts worker) if this condition holds.

4. Status HTTP endpoint field

The existing /status endpoint (claude-status.service) should expose:

• running_jobs: N
• pending_jobs: N
• last_claim_attempt: timestamp
• last_skip_reasons: [{job_id, reason}]

Priority

High — this is the class of silent failure that requires human SSH to diagnose. Every occurrence violates the autonomy principle.