Cancelling a running request tells the client it was cancelled, but the model actor keeps executing and holds its replica until the block finishes on its own. On a shared deployment, killing a runaway job does not free the GPU.
Observed
Submitted a ~30s block, then ran ndif kill <id> about 2s after it reached RUNNING. The client correctly got ERROR: Request cancelled by operator. The full log trail for that request id:
api ndif.request stage=received Your job has been received...
api ndif.queue.processor stage=queued request enqueued
api ndif.request stage=dispatched Your job has been sent to the model deployment.
model ndif.request stage=running Your job has started running.
api ndif.request stage=error Request cancelled by operator.
api ndif.queue.replica stage=running err=cancelled request errored: cancelled mid-dispatch
model ndif.request stage=completed Your job has been completed. <-- ran to completion
The cancel lands on the queue side (queue/replica.py, "cancelled mid-dispatch"). BaseModelDeployment.cancel sets a kill switch and run() has a cancellation branch that reports status="cancelled" (modeling/base.py:422), but nothing reached it here — the actor completed normally roughly 28s after the client had been told otherwise.
Why it matters
- The replica stays busy for the full duration, so cancellation does not reclaim capacity. With a small
NDIF_AUTOSCALING_MAX_REPLICAS, a few uncancellable long jobs still take the model out.
- The client and the server disagree about what happened to the request. The client is told ERROR; the actor records COMPLETED.
- Related: the actor-level cancel path appears to be unreachable in practice — see the companion issue on
execution_time, where cancelled never appears as a status.
Expected
kill should interrupt the execution on the actor, not only fail the dispatch — or, if interrupting mid-block is not safe, the limitation should be documented so operators know cancellation does not reclaim the replica.
Reproduced on the Test deployment, ndif 0.8 @ be38d78.
Cancelling a running request tells the client it was cancelled, but the model actor keeps executing and holds its replica until the block finishes on its own. On a shared deployment, killing a runaway job does not free the GPU.
Observed
Submitted a ~30s block, then ran
ndif kill <id>about 2s after it reached RUNNING. The client correctly gotERROR: Request cancelled by operator.The full log trail for that request id:The cancel lands on the queue side (
queue/replica.py, "cancelled mid-dispatch").BaseModelDeployment.cancelsets a kill switch andrun()has a cancellation branch that reportsstatus="cancelled"(modeling/base.py:422), but nothing reached it here — the actor completed normally roughly 28s after the client had been told otherwise.Why it matters
NDIF_AUTOSCALING_MAX_REPLICAS, a few uncancellable long jobs still take the model out.execution_time, wherecancellednever appears as a status.Expected
killshould interrupt the execution on the actor, not only fail the dispatch — or, if interrupting mid-block is not safe, the limitation should be documented so operators know cancellation does not reclaim the replica.Reproduced on the Test deployment, ndif 0.8 @ be38d78.