Skip to content

[neat_periodic_task] Add heartbeat locks to NeatPeriodicTaskScheduler - #408

Open
sigurdm wants to merge 3 commits into
google:masterfrom
sigurdm:heartbeat-locks
Open

[neat_periodic_task] Add heartbeat locks to NeatPeriodicTaskScheduler#408
sigurdm wants to merge 3 commits into
google:masterfrom
sigurdm:heartbeat-locks

Conversation

@sigurdm

@sigurdm sigurdm commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds heartbeat support to NeatPeriodicTaskScheduler in package:neat_periodic_task.

Motivation

Currently, when a periodic task is claimed by a worker instance, the status file is set to state: 'running' with started: now. If that worker is terminated abruptly (e.g. SIGKILL, container replacement, App Engine / GCE VM shutdown, OOM):

  • The status file remains in state: 'running'.
  • Other workers observe state: 'running' and assume the task is still executing. They will back off and not retry until _timeout has elapsed.
  • For tasks configured with long timeouts (e.g. 12 hours) to accommodate worst-case execution times, an early crash prevents any other worker from reclaiming the task for up to 12 hours, potentially triggering monitoring alerts.

Changes

  1. NeatTaskStatus:
    • Added optional DateTime? heartbeat property.
    • Fully backward-compatible JSON serialization and deserialization.
  2. NeatPeriodicTaskScheduler:
    • Added optional heartbeatInterval and heartbeatTimeout constructor parameters.
    • Defaults: If heartbeatInterval is provided, heartbeatTimeout defaults to 3 * heartbeatInterval. If only heartbeatTimeout is provided, heartbeatInterval defaults to heartbeatTimeout ~/ 3.
    • While _task() is executing, a periodic timer updates the heartbeat timestamp in _statusProvider.
    • In _iteration(), if status.state == 'running', other schedulers check whether the heartbeat has expired (now.difference(status.heartbeat) >= _heartbeatTimeout). If expired, the worker is presumed dead and the task is reclaimed immediately without waiting for the full _timeout.
    • trigger() also reclaims the task if the heartbeat is expired.
  3. Tests:
    • Added tests verifying heartbeat updates during execution, reclamation of abandoned tasks after heartbeat expiration without waiting for _timeout, and trigger reclamation.

Enhance NeatPeriodicTaskScheduler to periodically refresh a heartbeat
timestamp in the status file while running.

If an instance crashes, encounters an OOM, or is killed abruptly while
running a periodic task, other schedulers can detect the stale heartbeat
after `heartbeatTimeout` and reclaim the task without having to wait for
the full (often long, e.g. 12-hour) task `timeout`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant