Add --data_parallel option to benchmark for concurrent example processing - #210
Open
RobotSail wants to merge 1 commit into
Open
Add --data_parallel option to benchmark for concurrent example processing#210RobotSail wants to merge 1 commit into
RobotSail wants to merge 1 commit into
Conversation
…sing The benchmark previously processed dataset examples sequentially, with each call to scaling_alg.infer() creating a new asyncio event loop via asyncio.run(). For 500 examples x 4 budgets, this meant 2000 sequential inference calls with 2000 event loop creations - severely underutilizing GPU capacity on small models. Refactored the benchmark loop to use a single async event loop with a semaphore- bounded concurrency model. With --data_parallel N, N examples now run concurrently via asyncio.gather, sharing a single event loop and directly calling ainfer() instead of the sync wrapper. This keeps the vLLM server's continuous batching pipeline full, which is especially impactful for fast 7-8B parameter models where per-call overhead dominates. Default is 1 (sequential) for backward compatibility. Recommended: 8-32 for local vLLM with small models. Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark previously processed dataset examples sequentially, with each
call to scaling_alg.infer() creating a new asyncio event loop via asyncio.run().
For 500 examples x 4 budgets, this meant 2000 sequential inference calls with
2000 event loop creations - severely underutilizing GPU capacity on small models.
Refactored the benchmark loop to use a single async event loop with a semaphore-
bounded concurrency model. With --data_parallel N, N examples now run
concurrently via asyncio.gather, sharing a single event loop and directly
calling ainfer() instead of the sync wrapper. This keeps the vLLM server's
continuous batching pipeline full, which is especially impactful for fast
7-8B parameter models where per-call overhead dominates.
Default is 1 (sequential) for backward compatibility. Recommended: 8-32 for
local vLLM with small models.
Signed-off-by: Claude noreply@anthropic.com