Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Packages without a separate changelog are covered by the cross-package notes below.

## [Unreleased]
## [Unreleased - Patch]

### Fixed

- Releasing an agent hosted by a relay broker completes again. It was failing with `agent_release_generation_conflict` because the broker deregisters the agent just before reporting the release result.

## [8.13.0] - 2026-09-25

Expand Down
6 changes: 5 additions & 1 deletion packages/engine/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,11 @@ See the [root changelog](../../CHANGELOG.md) for cross-package release highlight
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [Unreleased - Patch]

### Fixed

- A node-completed release is accepted when that node already deregistered the agent (a relay broker sends `agent.deregister` before `action.result`), instead of failing with `agent_release_generation_conflict`. A release whose agent has since been bound to a different node is still refused.

## [8.13.0] - 2026-09-25

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { eq } from 'drizzle-orm';
import { makeNodeStack, createWorkspace, FakeSocket, type TestStack } from './harness.js';
import { actionInvocations, agentNodeBindings, agents } from '../../db/schema.js';

type Json = Record<string, unknown>;

/**
* A relay broker completes a release by stopping the worker, queueing
* `agent.deregister` and then sending `action.result`, in that order on its
* one control channel. The deregister deactivates the agent's binding on the
* node before the result arrives, so the completion must accept a binding the
* same release already removed, while still refusing an agent that has since
* been bound to a different node.
*/
describe('node release completed after the node deregistered the agent', () => {
let stack: TestStack;
beforeEach(() => { stack = makeNodeStack({ ttlMs: 60_000 }); });
afterEach(async () => {
await stack.close();
});

async function enrollBroker(ws: { workspaceKey: string; workspaceId: string }, nodeId: string, name: string) {
const enroll = await stack.app.request('/v1/nodes', {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${ws.workspaceKey}` },
body: JSON.stringify({ node_id: nodeId, name, capabilities: ['spawn:claude'], max_agents: 4, version: 'test-node' }),
});
expect(enroll.status).toBe(201);
const sock = new FakeSocket();
const handle = stack.runtime.realtime.attachNodeSocket(ws.workspaceId, nodeId, sock);
await handle.handleMessage(JSON.stringify({
v: 1, type: 'node.register', name, node_id: nodeId,
capabilities: [{ name: 'spawn:claude', kind: 'capacity' }, { name: 'release', kind: 'capacity' }],
max_agents: 4, tags: [], version: 'test-node', resume_cursor: null,
}));
await handle.handleMessage(JSON.stringify({
v: 1, type: 'node.heartbeat', load: 0, active_agents: 0, handlers_live: true,
}));
return { sock, handle };
}

async function registerOnNode(node: Awaited<ReturnType<typeof enrollBroker>>, name: string) {
await node.handle.handleMessage(JSON.stringify({
v: 1, type: 'agent.register', name, resumable: true, session_ref: `sess-${name}`,
}));
const reply = node.sock.ofType('reply').at(-1) as { ok: boolean; data: { agent_id: string } };
expect(reply?.ok).toBe(true);
return reply.data.agent_id;
}

async function releaseDispatched(workspaceKey: string, name: string) {
const res = await stack.app.request('/v1/agents/release', {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${workspaceKey}` },
body: JSON.stringify({ name }),
});
expect(res.status).toBe(201);
const { data } = (await res.json()) as { data: { status: string; invocation_id: string } };
expect(data.status).toBe('dispatched');
return data.invocation_id;
}

async function invocation(id: string) {
const [row] = await stack.runtime.deps.db
.select({ status: actionInvocations.status, error: actionInvocations.error })
.from(actionInvocations)
.where(eq(actionInvocations.id, id));
return row;
}

it('completes a plain release whose node deregistered the agent before reporting the result', async () => {
const ws = await createWorkspace(stack.app, 'release-after-deregister');
const node = await enrollBroker(ws, 'node_a', 'node-a');
const agentId = await registerOnNode(node, 'worker');

const invocationId = await releaseDispatched(ws.workspaceKey, 'worker');
expect(node.sock.ofType('action.invoke').map((frame) => (frame as Json).action)).toContain('release');

// The broker's order: deregister first, then the release result.
await node.handle.handleMessage(JSON.stringify({ v: 1, type: 'agent.deregister', agent_id: agentId, name: 'worker' }));
await node.handle.handleMessage(JSON.stringify({
v: 1, type: 'action.result', invocation_id: invocationId, output: { released: true },
}));

expect(await invocation(invocationId)).toEqual({ status: 'completed', error: null });
const [agent] = await stack.runtime.deps.db
.select({ status: agents.status, locationNodeId: agents.locationNodeId })
.from(agents)
.where(eq(agents.id, agentId));
expect(agent).toEqual({ status: 'offline', locationNodeId: null });
});

it('still refuses the release once the agent is bound to a different node', async () => {
const ws = await createWorkspace(stack.app, 'release-after-move');
const nodeA = await enrollBroker(ws, 'node_a', 'node-a');
const agentId = await registerOnNode(nodeA, 'worker');

const invocationId = await releaseDispatched(ws.workspaceKey, 'worker');
await nodeA.handle.handleMessage(JSON.stringify({ v: 1, type: 'agent.deregister', agent_id: agentId, name: 'worker' }));
// Before node A reports, the identity is bound to node B: the state a
// concurrent move leaves behind, written directly.
await enrollBroker(ws, 'node_b', 'node-b');
const db = stack.runtime.deps.db;
await db.insert(agentNodeBindings).values({
id: 'bind_moved_to_b', workspaceId: ws.workspaceId, agentId, nodeId: 'node_b', status: 'active', priority: 0,
}).onConflictDoUpdate({
target: [agentNodeBindings.agentId, agentNodeBindings.nodeId],
set: { status: 'active' },
});
await db.update(agents).set({ locationType: 'via_node', locationNodeId: 'node_b', status: 'active' }).where(eq(agents.id, agentId));

await nodeA.handle.handleMessage(JSON.stringify({
v: 1, type: 'action.result', invocation_id: invocationId, output: { released: true },
}));

expect(await invocation(invocationId)).toEqual({ status: 'failed', error: 'agent_release_generation_conflict' });
const [agent] = await stack.runtime.deps.db
.select({ status: agents.status, locationNodeId: agents.locationNodeId })
.from(agents)
.where(eq(agents.id, agentId));
expect(agent.locationNodeId).toBe('node_b');
expect(agent.status).not.toBe('offline');
});
});
23 changes: 22 additions & 1 deletion packages/engine/src/engine/action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2334,7 +2334,28 @@ async function completeReleaseNodeInvocation(
AND ${actionInvocations.dispatchedProvider} = ${providerName}
AND ${actionInvocations.status} = 'completed'
)`;
const releaseCanApply = sql`(${generationStillCurrent}) AND (${activeBindingStillCurrent})`;
// A relay broker queues `agent.deregister` ahead of the release result on
// its one control channel, so the binding this release targets is usually
// already inactive when the result lands. Accept that, as long as the agent
// was bound to this node and has not since been bound to any other node
// than its own implicit direct node (where deregistration re-homes it).
Comment on lines +2337 to +2341

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire the fallback direct binding after accepting the release

For the ordinary non-deleting release covered by the new test, agent.deregister creates an active implicit-direct binding and sets that direct node's activeAgents to 1. This predicate now accepts that state, but the completion batch only deactivates/decrements the dispatched broker node before setting the agent's location to null, leaving the fallback binding active and its node capacity occupied. Node-agent listings and later binding selection consequently continue to report the released agent as hosted; the accepted fallback binding and its capacity need to be retired as part of completion.

Useful? React with 👍 / 👎.

const boundOnlyHere = sql`(
EXISTS (
SELECT 1 FROM ${agentNodeBindings}
WHERE ${agentNodeBindings.workspaceId} = ${workspaceId}
AND ${agentNodeBindings.agentId} = ${agent.id}
AND ${agentNodeBindings.nodeId} = ${nodeId}
)
AND NOT EXISTS (
SELECT 1 FROM ${agentNodeBindings}
WHERE ${agentNodeBindings.workspaceId} = ${workspaceId}
AND ${agentNodeBindings.agentId} = ${agent.id}
AND ${agentNodeBindings.status} = 'active'
AND ${agentNodeBindings.nodeId} <> ${nodeId}
AND ${agentNodeBindings.nodeId} <> ${`node_direct_${agent.id}`}
Comment on lines +2353 to +2355

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Plain release strands direct-node capacity

After broker deregistration, boundOnlyHere accepts the new active direct binding for a plain release. Completion clears the agent's location but leaves that binding active and its capacity slot occupied.

Learn more

Broker deregistration re-homes the agent using ensureDirectNodeForAgent, creating an active direct binding and setting the direct node's activeAgents to one. The node-completion writes only decrement and deactivate bindings on the reporting broker, then set the agent offline with no location. As a result, the accepted plain release retains a binding and a charged direct-node slot. completeLocally releases all active bindings and their slots.

Example: Node A deregisters worker, creating an active node_direct_workerId binding. A then returns a successful plain release; worker is offline without a location, but the direct binding stays active and its node still has activeAgents = 1.

Recommended fix: In completeReleaseNodeInvocation, conditionally deactivate and decrement the implicit direct binding and node as part of the same atomic completion, provided the invocation won and no newer direct session owns the binding. Cover both plain and delete_agent releases and verify capacity and bindings in the deregister-before-result test.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +2353 to +2355

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Broker result releases a new direct session

If an agent reconnects directly after broker deregistration, boundOnlyHere still accepts the broker's plain release result. The old result takes the new direct session offline and clears its location.

Learn more

deregisterAgentViaNode re-homes the agent to its implicit direct node, which has an active binding even while its node is offline. The direct connection can subsequently become live before the original broker reports its release result. This predicate treats that active binding exactly like the offline fallback. A plain release lacks an expected_token_hash; completion therefore changes the active direct agent to offline and nulls its location.

Example: A dispatches a plain release for worker. A deregisters worker; worker connects through node_direct_<id>. A's delayed result then completes and sets worker offline, despite its new direct connection.

Recommended fix: Distinguish a direct fallback left offline by deregistration from a subsequent active direct connection. Fence the completion against a newer direct session using an atomic agent/binding state or generation check; do not allow the reporting broker to release the newer session.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +2353 to +2355

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Distinguish a live direct rebind from the fallback binding

If the agent opens its direct-node connection after agent.deregister but before the broker's action.result, the direct binding is genuinely live, yet this exception ignores it and accepts the stale result. The completion then overwrites the newly hosted agent's location/status (and with delete_agent, deletes its live direct node), even though the same code rejects a rebind to every other node. Check whether the implicit node is merely the offline fallback created by deregistration rather than exempting every direct binding.

Useful? React with 👍 / 👎.

)
)`;
const releaseCanApply = sql`(${generationStillCurrent}) AND (${boundOnlyHere})`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid emitting agent.exited twice for one broker release

In the broker ordering this change is designed to accept, deregisterAgentViaNode has already called emitAgentExitedEffects(... reason: 'deregistered'); after this predicate permits completion, completeReleaseNodeInvocation calls it again with reason released. The workspace event log, observer stream, and webhook outbox therefore receive two distinct agent.exited events for the same exit (the reason is part of the delivery key, so it is not deduplicated), which can trigger downstream cleanup twice.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release leaves implicit binding active

Low Severity

Accepting a release after agent.deregister leaves the implicit-direct binding that deregister just activated as active. The agent is marked offline with no location, but host discovery still treats that binding as a live placement.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 140b1b5. Configure here.

const releasedName = releasedAgentName(agent.name, agent.id);
const releasedTokenHash = await sha256Hex(`released:${agent.id}:${randomHex(16)}`);
let successIndex = -1;
Expand Down
Loading