fix: alert on every failure streak of a watch, not just the first

[asasemahmed]

What changed

A failing watch sends two alerts: "retry scheduled" on the first failure, and "paused" after five failures. The notification ID is hash("watch-error:<task>:<retry|paused>"), stored with insertIfAbsent, and notifications are only marked read, never deleted.

That key never changes for the life of the watch. So after the watch is resumed, or recovers, and later fails again, both alerts dedupe against the first streak's. The owner is never told that the watch paused a second time.

Now the task state keeps a failureStreak counter. It increases when failures goes from 0 to 1 (after a success or a resume, both of which already reset failures), and it is part of the alert key: watch-error:<task>:<streak>:<retry|paused>. Each failure streak alerts once for the retry and once for the pause. Replaying the same outcome still dedupes, because the key comes from the saved state.

Verification

• A new test in tests/workflows.test.ts fails a watch until it pauses, resumes it with controlMonitor(resume), and fails it until it pauses again. It expects 4 distinct alerts, and 4 after one more replayed tick.
  • main: 2 alerts.
  • This branch: 4.
• The existing backoff/pause test still passes.
• pnpm typecheck and pnpm build:server pass. Biome reports no issues for the changed files.
• pnpm test: 171 passed, 1 failed. The failure is Docker subprocess uses literal argv…, which fails on Windows with or without this change.

Integration limits

Tasks saved before this change have no failureStreak, so their next failure alert uses the new key format and may notify once more than before.

[davidmckayv]

Approved to merge, but it now conflicts with main in apps/server/src/engine/service.ts after #41 landed. Rebase onto main and it will be merged once CI is green.

[davidmckayv]

The rebase preserves the accepted failure-streak fix and its recovery regressions. All seven CI checks pass on the reviewed head.