Skip to content

Local worker release reports failure or hangs when cloud identity cleanup is unconfirmed #1821

Description

@kjgbot

Problem

DELETE /api/spawned/:name successfully stopped and removed local workers, but the request then waited on cloud fleet/identity cleanup. Under a saturated Relaycast workspace it timed out or returned HTTP 500:

owned identity cleanup unconfirmed ... retry retained:
fleet deregistration connection closed before acknowledgement

The worker was already absent from GET /api/spawned, so the response makes a successful local release look like a total failure. Some requests hung beyond 30 seconds.

Expected

Local process release and remote identity cleanup should have explicit, independently observable outcomes. Once the worker is stopped, the API should return bounded structured success such as:

{"released":true,"identityCleanup":"pending"}

Remote cleanup can remain journaled and retryable without blocking or converting the local release into an ambiguous 500.

Add regression coverage for a disconnected or saturated control plane and idempotent retries.

Impact

During a workspace-backlog incident, operators cannot reliably drain the workers generating pressure because release itself waits on the impaired cloud path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    workflow-readyReady for workflow implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions