Request Metadata on COMPLETED/ERROR - #277
Open
AdamBelfki3 wants to merge 3 commits into
Open
AdamBelfki3 wants to merge 3 commits into
AdamBelfki3 wants to merge 3 commits into
Conversation
run() already measures both -- gpu_baselines/gpu_peaks either side of the block, and exec_ms -- but only to feed GPUMemMetric, so the numbers were visible to an operator on a dashboard and to nobody else. request_meta shapes the same measurements into nnsight's ResponseModel.meta_data and they ride out on the COMPLETED response. Memory is reported above the resident weights (peak minus baseline), not the card's total: the weights are the actor's, not the request's. The percentage is against the headroom the request actually had -- this actor's assignment on the card less what the weights already hold -- so 100% means it filled what was left for it. GPU keys are strings. A response reaches the client as JSON or as torch.save bytes when the result rides along, and only JSON stringifies dict keys; emitting strings means the client isn't parsing a different shape depending on how big its result was. Best-effort throughout -- no CUDA, an unknown assignment, or a block that freed memory each yield zeros rather than raising. A report is not worth failing a job that already succeeded. Tests: test_utils.py covers the arithmetic with no GPU or server. The live suite gains coverage of the report end to end, plus the first tests here to drive the async backend at all -- await and async for, both against a real job.
…short
A request that dies is the one that most needs to explain itself, and it was
the one saying nothing: meta_data rode only on COMPLETED. It now rides on the
three terminal failures too -- cancelled, timed out, and raised -- because the
run is over either way and a timeout that peaked at 99% of its headroom
explains itself.
On an out-of-memory failure the report also carries extra_memory_needed:
{gpu_id: bytes} the block reached past its allowance, per card that ran out.
This is the number an OOM cannot otherwise give. The refused allocation never
lands, so it never reaches the peak counters -- a real OOM measured here
reported 1024 bytes used and 0.0% of headroom, which reads as a contradiction
next to "out of memory". Nor does the size of the refused allocation answer it
alone: asking for 2 GB with 1.9 GB free and asking for it with nothing free are
the same number and completely different problems.
Only the requested size is read, out of the allocator's own refusal message.
The budget is this actor's and the reserved figure is torch's, so both are
exact; against a live 8 GiB refusal this agrees with torch's OOM observer to
the byte. The message is used in preference to that observer because the
observer is out-of-band state that fires on refusals torch then recovers from,
so it needs clearing per request and can still hand a later failure someone
else's number. A message belongs to the exception being reported and cannot
desynchronize. The cost is a dependency on torch's phrasing, which one test
pins verbatim so an upgrade fails loudly rather than silently returning None.
Reserved, not allocated, is the basis: the per-process cap is enforced against
what the allocator reserved from the driver, not what tensors hold.
request_meta builds the whole report, exception included, so the actor has one
call for every terminal response and the key is simply absent when there was no
OOM.
request_meta returned Dict[str, Any], so the report's shape was agreed by convention between two repos and checked by neither. It now constructs nnsight's MetaData, which that side declares, so drift becomes a type error here instead of a stale sentence in docs/reference/schemas.md. That also moves two mistakes forward to where they can be fixed. Integer GPU keys are rejected outright rather than silently stringified by one encoding and not the other -- the bug this code previously avoided by hand. And a key that does not exist on the model fails in the actor, with a traceback, rather than arriving at a user as a payload they cannot find documented. The out-of-memory field is renamed extra_memory_needed -> alloc_shortfall_by_gpu so it matches the other per-device maps beside it, and the helper that computes it is alloc_shortfall. The docs now lead with what it is not: the part of the refused allocation that would not fit, rather than the size of the refused allocation, which is the one thing a reader could reasonably get wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reports a job's runtime and GPU footprint back to the client on the response
that ends it, and tells an out-of-memory failure how much memory it was short.
run()already measures both —gpu_baselines/gpu_peakseither side of theblock, and
exec_ms— but only feeds them toGPUMemMetric, so the numbers werevisible to an operator on a dashboard and to nobody else.
What's here
request_metabuilds nnsight'sMetaDataand it rides out onCOMPLETED.The run is over either way, and a timeout that peaked at 99% of its headroom
explains itself.
alloc_shortfall_by_gpu:{gpu_id: bytes}of the refused allocation that would not fit, per card.Testing
30 unit tests in
tests/test_utils.py(no GPU, no Ray, no server) plus end-to-endcoverage in
tests/test_nnsight_remote.py, including a forced OOM against alocal stack on a real GPU and the first tests here to drive the async backend at
all. Full suite: 134 passed, 15 skipped.