[Bug] Flaky panic triggered by test_workflow_cancel_activity in sdk-python

[brianstrauch]

Two dbg_panic! sites fire from sdk-core during a sdk-python test that rapidly cancels local activities under a 1s workflow task timeout.

• thread 'workflow-processing' panicked at crates/sdk-core/src/worker/workflow/managed_run.rs:401:17: Attempted to complete activation for run 019db732-4d78-7616-9a36-20b44eff6c09 without associated workflow task
• thread 'tokio-runtime-worker' panicked at crates/sdk-core/src/worker/workflow/mod.rs:575:13: Send half of activation complete response channel went missing
• Surfaces to Python as pyo3_async_runtimes.RustPanic: rust future panicked: unknown error from complete_workflow_activation, and subsequent QueryWorkflow RPCs return WorkflowNotReadyFailure.

Reproduction

sdk-python (debug build):

uv run pytest tests/worker/test_workflow.py::test_workflow_cancel_activity -k True -x -s

Timing-sensitive; doesn't always reproduce.

CI run: https://github.com/temporalio/sdk-python/actions/runs/24804169223

[brianstrauch]

Observed again: https://github.com/temporalio/sdk-python/actions/runs/25233656611/job/74001362489?pr=1498

[tconley1428]

Some AI Investigation. Most importantly,
• I’m making a temporary one-line local test change to shrink the workflow task timeout, only to reproduce the core panic with RUST_BACKTRACE=1. I’ll restore the file immediately after the run.

• Edited tests/worker/test_workflow.py (+1 -1)
    962      # so the task can timeout
    963 -    task_timeout = timedelta(seconds=1)
    963 +    task_timeout = timedelta(milliseconds=1)
    964      assert_timeout = timedelta(seconds=10)

This change makes the pattern reproducible.

[brianstrauch]

Also did some reading and investigation. Sounds like the panic is needed for antithesis, and the fix is more likely to happen on the Python side with a pretty substantial rewrite of the workflow test that can hopefully make use of Temporal primitives instead of flaky timeouts.

[Sushisource]

@brianstrauch I also think there is some chance #1239 will just fix this incidentally, so, worth redoing the repro after that's merged before investing more time.