POST /v1/nodes returns 500 for node 'factory-primary' — factory container cannot boot, P0 dispatch is down

[khaliqgant]

Impact

The factory-cloud container cannot complete bootstrap. It dies enrolling its node and never starts dispatching:

phase: failed
factoryProcess: not-running
duringPhase: enrolling-node
message: Cloud node enrollment failed after 5 attempt(s): 500 internal_error Internal server error
  at enrollNode (container/node-enrollment.mjs:286)
  at async bootstrap (container/entrypoint.mjs:1369)

• Node name: factory-primary
• Broker instance: agentworkforce-factory-6b720b1d
• Call: POST https://cast.agentrelay.com/v1/nodes
• Retries: 5, all 500

This is the current blocker on the factory P0. Everything downstream of it is fixed and deployed; nothing dispatches because the container never reaches running.

The service itself is healthy — this is specific to that enrollment

Three credential-free controls, run at 14:05Z, all behaving correctly:

POST /v1/nodes            Bearer rk_live_<bogus>   -> 401 {"code":"unauthorized"}   (not 500)
POST /v1/nodes-bogus-route-control                 -> 404 {"code":"not_found"}
GET  /health                                       -> 200 {"ok":true}

So routing, auth, and liveness are fine. A generic bad request gets a clean 401, not a 500. The 500 is reached only on the real enrollment for this node/workspace, which points at server-side state for that row rather than at the endpoint being down.

Timeline — it worked, then stopped

• 13:26:26Z container boots after a deploy, enrolls successfully, phase: running, readinessReconcile: healthy, completes a full discovery sweep (19 candidates)
• 14:03Z phase: failed, factoryProcess: not-running, enrollment 500 x5

So factory-primary enrolled cleanly once and then could not re-enroll after the container recycled.

Hypothesis — flagged as such, not asserted

POST /v1/nodes is documented as "enroll or rotate" and is expected to return 201 with the same data.id and a fresh token on a repeat call for the same name. The failure appears on exactly that repeat path.

That sits close to two known open issues:

• No DELETE /v1/nodes/:name — node rows are permanent, and one production roster is already at 1225 #353 — no DELETE /v1/nodes/:name, node rows are permanent, one production roster already at 1225
• An orphaned agent row makes its name permanently unreclaimable — the agent runs but can never receive #359 — an orphaned agent row makes its name permanently unreclaimable

What would discriminate: the server-side error/stack behind that 500 for factory-primary, and whether the existing row is in a state the rotate path cannot handle. I can only see the 500 from outside.

Disclosure, since it is relevant to #353: while measuring the sandbox-ensure path today I provisioned roughly a dozen JIT fleet nodes into workspace 50587328-441d-4acb-b8f3-dbe1b3c5de99 (rw_7ccfea89), named fleet-ensure-*. Given #353 says rows are permanent and unbounded, I cannot rule out that I contributed to roster pressure, and I would rather say so than have someone discover it. I could not verify their state — my CLI token 403s on the roster read.

Ask

Someone with server-side visibility needs the actual 500. Until factory-primary can enroll, the factory container cannot boot and no dispatch happens regardless of what is fixed elsewhere.

Filed without a readiness label.

Reported by cloud-ensure-fix-r1.