Skip to content

fix(heartbeat): bind every runner to a live-verified Slack destination - #749

Merged
JSap0914 merged 3 commits into
devfrom
codex/heartbeat-live-binding
Sep 11, 2026
Merged

fix(heartbeat): bind every runner to a live-verified Slack destination#749
JSap0914 merged 3 commits into
devfrom
codex/heartbeat-live-binding

Conversation

@JSap0914

Copy link
Copy Markdown
Collaborator

Depends on the run-identity layer below.

Problem: a heartbeat destination could be syntactically complete while pairing a Slack channel with a stale or unrelated parent ts. The runner spent model/script work before discovering it had nowhere safe to send, live failures were visible only in logs, and Auto runtimes could call Slack outside the scheduled destination.

This layer hard-binds every runner:

  • before main/employee/script work, threaded Slack jobs call conversations.replies(channel, ts, limit=1); the first row must be the configured parent;
  • stale/mismatch/permission/auth/rate-limit/transport failures hold the tick before work and send, with no retry and no positive cache;
  • live hold reasons remain visible through GET /api/heartbeat/UI until recovery and invalidate when the destination changes;
  • every Slack runner gets a 25-minute enforceDestination grant (collector ceiling: 20 minutes);
  • activation happens before runtime selection: print/employee children inherit it, Cursor/Grok use request-lifetime acquisition, Codex App/Claude/Pi use fresh grant-bearing acquisition, Pi launches from captured env, and script gets explicit child env;
  • omitted agent target pins to the job; mismatch is slack_destination_mismatch;
  • unrelated Auto/interactive sends are never process-globally locked;
  • mention-watch remains separate because its destination is the hit thread it just read.

Operational migration already applied on the 3457 home: the five channel-only jobs now explicitly use scope:"channel_root"; the three threaded jobs were read-only verified live with exact parent ts.

Validation: heartbeat/API/security suite 158/158; final runtime/Pi/Grok batch 114/114; audit-fix batch 284/284; npm run gate:all 23/23; private boundary pass. Independent grok-4.6 review: PASS, no merge blocker.

Closes #744.
Closes #745.

suji lee added 3 commits September 12, 2026 00:37
Before any main, employee, or script work, a threaded Slack job now proves that
conversations.replies returns the configured parent in the configured channel.
Missing, stale, mismatched, rate-limited, unauthenticated, and transport-failed
lookups hold the tick without retry or cached success. Live hold reasons remain
visible through GET/UI until recovery and invalidate when the destination
changes.

Every Slack runner reserves enforceDestination authority for its lifetime.
Print children present the grant and are pinned to it. Native/pool, employee,
and script paths cannot receive a fresh per-turn header, so their headerless
Slack sends are refused during the reservation and the server owns final
delivery. Interactive Auto grants keep their existing instance-wide contract
outside that scheduled window.

Planner retries receive fresh grants; channel-root and non-Slack jobs require no
Slack read; mention-watch keeps the hit thread it just discovered.

Refs #742 #745
A process-global safeguard fixed native heartbeat escape by blocking unrelated
Slack sends for the full runner lifetime, which could be ten minutes. It also
expired at 15 minutes while a native collector may legitimately wait 20.

Destination enforcement is now per process. Grant activation happens before
any runtime branch. Print and employee children inherit the header; Cursor and
Grok use a request-lifetime native process; Codex App, Claude, and Pi get fresh
grant-bearing acquisitions; Pi now launches from its captured env; script
children receive the same header explicitly. The enforced grant lives 25
minutes, beyond every runner ceiling, then follows normal revoke cleanup.

Unrelated Auto/interactive Slack sends remain available while scheduled work
runs. Actual native Grok verifies the grant reached its environment, and source
contract tests cover all runtime branches plus the TTL relation.

Refs #745
Replace the process-global Slack lock with per-runtime grant injection. Grant
activation now precedes every spawn branch. Cursor/Grok run request-lifetime,
Codex App/Claude/Pi acquire fresh grant-bearing processes, employee metadata
carries the same request and target, and scripts receive the grant in their
child environment. Employee Pi now passes the captured spawn env to spawnPiRpc
as well as the main pool path.

Enforced grants live 25 minutes, longer than the 20-minute collector ceiling,
without blocking unrelated Auto or interactive Slack sends.

Refs #745
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a5951f6c-845c-4507-94ae-1266930946f8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Base automatically changed from codex/slack-run-isolation to dev September 11, 2026 16:27
@JSap0914
JSap0914 changed the base branch from codex/slack-run-isolation to dev September 11, 2026 16:27
@JSap0914
JSap0914 merged commit 83079cc into dev Sep 11, 2026
16 checks passed
@JSap0914
JSap0914 deleted the codex/heartbeat-live-binding branch September 11, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant