Skip to content

Replace PauseState flag with PAUSE_REQUESTED state machine state#10265

Open
dandavison wants to merge 8 commits into
feature/activity-operator-cmdsfrom
feature/activity-operator-cmds-pause-requested
Open

Replace PauseState flag with PAUSE_REQUESTED state machine state#10265
dandavison wants to merge 8 commits into
feature/activity-operator-cmdsfrom
feature/activity-operator-cmds-pause-requested

Conversation

@dandavison
Copy link
Copy Markdown
Contributor

@dandavison dandavison commented May 14, 2026

This is a proposed change to #10106.

See https://github.com/temporalio/temporal/pull/10106/changes#r3238854955


Note

High Risk
Changes core activity execution, pause/unpause, retry, timeout, and task-token semantics; incorrect transitions could strand activities or break worker heartbeats.

Overview
Replaces the hybrid pause model (STARTED + PauseState flag, blocked dispatch on SCHEDULED) with an explicit ACTIVITY_EXECUTION_STATUS_PAUSE_REQUESTED state in the CHASM activity state machine.

Pause / unpause: SCHEDULED still goes to PAUSED; STARTED now transitions to PAUSE_REQUESTED via TransitionPauseRequested (records pause_state, no attempt stamp bump so worker tokens and timeout tasks stay valid). Unpause uses PAUSED → SCHEDULED or PAUSE_REQUESTED → STARTED (TransitionUnpausedToStarted). isPaused() keys off status, not PauseState != nil.

Retries: When the worker yields with retries left while in PAUSE_REQUESTED, tryReschedule uses TransitionAttemptFailedToPaused instead of rescheduling to SCHEDULED—activity lands in PAUSED with attempt incremented.

API / tasks: Heartbeats set ActivityPaused from PAUSE_REQUESTED status; run state mapping is status-only (no hybrid SCHEDULED+flag logic). Dispatch and schedule-to-start validators no longer block on PauseState; start-to-close and heartbeat timeout validators treat PAUSE_REQUESTED like STARTED/CANCEL_REQUESTED. Options updates re-emit attempt timeouts for PAUSE_REQUESTED.

Reset / cancel: Reset handles PAUSE_REQUESTED (optional unpause when not keep_paused); simplified PAUSED reset path. Cancel+unpause test expectations updated (pause not surfaced on heartbeat when cancel wins).

Adds integration tests for start-to-close / heartbeat timeout and options update while PAUSE_REQUESTED.

Reviewed by Cursor Bugbot for commit b55a3b3. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 3b0b77a. Configure here.

Comment thread chasm/lib/activity/activity_tasks.go
Comment thread chasm/lib/activity/statemachine.go
@dandavison dandavison force-pushed the feature/activity-operator-cmds-pause-requested branch 2 times, most recently from e28990c to 3d8f7cf Compare May 14, 2026 12:41
@dandavison dandavison force-pushed the feature/activity-operator-cmds-pause-requested branch from 055b7ed to 7efdd4a Compare May 26, 2026 16:26
Adds three sub-tests to TestPauseActivityExecution that exercise the
PAUSE_REQUESTED + worker-does-not-yield interaction. Each was missing
before; verified by reverting the corresponding fix and observing the
specific test fail:

- StartToCloseTimeoutWhilePauseRequested:
  Pause a STARTED activity; the worker stops responding; the
  StartToCloseTimeoutTask must still fire (validator must accept
  PAUSE_REQUESTED) and the retry must land in PAUSED.

- HeartbeatTimeoutWhilePauseRequested:
  Same pattern for HeartbeatTimeoutTask.

- UpdateOptionsPreservesTimeoutsWhilePauseRequested:
  Pause a STARTED activity, call UpdateActivityExecutionOptions with a
  shorter StartToCloseTimeout; the handler must re-emit a fresh timeout
  task for PAUSE_REQUESTED activities, not only STARTED/CANCEL_REQUESTED.
  Otherwise the stamp bump silently strips timeout enforcement from the
  running worker.
@dandavison dandavison force-pushed the feature/activity-operator-cmds-pause-requested branch from 7efdd4a to d1b56e3 Compare May 26, 2026 16:28
Comment on lines +50 to +54
// An operator pause was received while the activity was STARTED. The worker is still executing
// under its existing task token. The worker will be notified via ActivityPaused=true on its next
// heartbeat response. When the worker yields (failure with retries remaining, or timeout with
// retries remaining), the activity will transition to PAUSED.
ACTIVITY_EXECUTION_STATUS_PAUSE_REQUESTED = 10;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't entirely true, when the worker yields because an activity reaches a terminal state (failed,completed) it won't transition to pause requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants