feat(PR-A3c): batched decode scheduler — 8.45× served multi-tenant throughput + detailed test report - #130
Merged
Conversation
…ode into one batched forward (served-path throughput) Fixed-cohort batched decoder over PerSessionVerifierRegistry adapters: stacks the per-session restored caches along batch dim, one verifier forward per step, drops finished rows. Served-path realization of the §3.5 parallel throughput. + bench (batched vs serialized, recall-checked) + guard unit tests. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…t (recall 1.0) + detailed PR-A3c multi-tenant test report Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The throughput step for the served multi-tenant path. §3.6 made serving correct multi-tenant (per-session binding) but RPC-serialized.
BatchedDecodeSchedulerfuses a cohort of per-session adapters' decode steps into one batched verifier forward (stacks their restored caches along the batch dim, drops finished rows) — the served-path realisation of the §3.5 parallel decode.Result (H200, 8 sessions, NIAH ctx≈1238) ✅
The served path goes from correct-but-serialized to 8.45× aggregate throughput at 8 sessions, recall preserved — matching the engine-level near-linear scaling, now driven through the served per-session adapters.
Detailed test report
Adds
docs/reports/pr-a3c-multitenant-serving-test-report.md— the full PR-A3c multi-tenant record (environment, methodology, all numbers, evidence index) across §3.4 memory A/B (~4.2× agents), §3.5 parallel throughput (8.04×), §3.6 served isolation (recall 1.0), §3.7 batched scheduler (8.45×).Scope / follow-ups
GenerateRPC coroutines feed one batch loop), are the remaining productization. Batched fused spec-decode (DFlash batch-1) is separate.Files
inference_engine/session/batch_scheduler.py(new) +tests/.../test_batch_scheduler.py(guard tests)scripts/research/k3_served_batched_scheduler_bench.py(new bench)docs/adr/0014-…md§3.7 + Appendix A;docs/reports/pr-a3c-multitenant-serving-test-report.md(new)results/research/k3_served_batched_scheduler_gpu.jsonTesting
pytest tests/inference_engine/session/test_batch_scheduler.py(4 passed — guards)