Skip to content

server: remove two host side stalls in the decode loop at high concurrency - #200

Draft
danielhanchen wants to merge 2 commits into
masterfrom
perf/server-result-queue
Draft

server: remove two host side stalls in the decode loop at high concurrency#200
danielhanchen wants to merge 2 commits into
masterfrom
perf/server-result-queue

server: only keep per token probabilities when the request asked for …

755fb62
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs