fix: fence tab recovery on visible heartbeat proof - #61
Conversation
2c16eaf to
e5a5ebb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c16eaf5f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| isinstance(heartbeat.get("recorded_ts"), (int, float)) | ||
| and heartbeat["recorded_ts"] > pending_since | ||
| ): | ||
| return { |
There was a problem hiding this comment.
Let rejected heartbeat proofs reach bounded recovery
When a newer post-poke heartbeat has headless ownership, lacks authority, or reports the wrong epoch, this return leaves the pending tab attempt unchanged. The same heartbeat remains newer than pending_since on every later watchdog tick, so execution never reaches heartbeat staleness, MAX_TAB_POKES, lease expiry, or the headless fallback; one rejected proof can therefore wedge recovery permanently. Expire or clear the pending attempt after a bounded wait so normal retry and escalation can proceed.
Useful? React with 👍 / 👎.
| expected_epoch=( | ||
| int(pending_epoch) | ||
| if isinstance(pending_epoch, int) and not isinstance(pending_epoch, bool) | ||
| else None |
There was a problem hiding this comment.
Migrate pending tab attempts that predate the epoch field
If this version is deployed while watchdog-state.json contains a pending tab poke created by the previous version, that persisted record has pending_since and pending_transport but no pending_epoch. This converts the expected epoch to None, causing even a valid visible authoritative heartbeat to be rejected; combined with the pending-proof return, the watchdog remains stuck until state is manually reset. Treat legacy pending attempts as expired or recover their epoch during migration.
Useful? React with 👍 / 👎.
Summary
Testing