Skip to content

fix: fence tab recovery on visible heartbeat proof - #61

Merged
CondorCommodore merged 1 commit into
mainfrom
agent/cos-next-slice
Aug 4, 2026
Merged

fix: fence tab recovery on visible heartbeat proof#61
CondorCommodore merged 1 commit into
mainfrom
agent/cos-next-slice

Conversation

@CondorCommodore

Copy link
Copy Markdown
Owner

Summary

  • require pending tab recovery to observe a visible authoritative heartbeat on the expected epoch before clearing as recovered
  • fail closed on mismatched ownership or epoch instead of treating any newer heartbeat as proof of recovery
  • extend watchdog tests for mismatched headless/visible heartbeat recovery cases

Testing

  • ./.venv/bin/ruff check scripts/cos_bootstrap_watchdog.py tests/test_cos_bootstrap_watchdog.py
  • python3 -m pytest -q tests/test_cos_bootstrap_watchdog.py tests/test_cos_bootstrap_supervisor.py tests/test_cos_current_actions.py

@CondorCommodore
CondorCommodore merged commit f9c1ba9 into main Aug 4, 2026
5 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c16eaf5f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +780 to +783
isinstance(heartbeat.get("recorded_ts"), (int, float))
and heartbeat["recorded_ts"] > pending_since
):
return {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Let rejected heartbeat proofs reach bounded recovery

When a newer post-poke heartbeat has headless ownership, lacks authority, or reports the wrong epoch, this return leaves the pending tab attempt unchanged. The same heartbeat remains newer than pending_since on every later watchdog tick, so execution never reaches heartbeat staleness, MAX_TAB_POKES, lease expiry, or the headless fallback; one rejected proof can therefore wedge recovery permanently. Expire or clear the pending attempt after a bounded wait so normal retry and escalation can proceed.

Useful? React with 👍 / 👎.

Comment on lines +728 to +731
expected_epoch=(
int(pending_epoch)
if isinstance(pending_epoch, int) and not isinstance(pending_epoch, bool)
else None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Migrate pending tab attempts that predate the epoch field

If this version is deployed while watchdog-state.json contains a pending tab poke created by the previous version, that persisted record has pending_since and pending_transport but no pending_epoch. This converts the expected epoch to None, causing even a valid visible authoritative heartbeat to be rejected; combined with the pending-proof return, the watchdog remains stuck until state is manually reset. Treat legacy pending attempts as expired or recover their epoch during migration.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant