feat(agent): completion webhook + always-on heartbeat tag (#497) - #498
Merged
Conversation
Closes the caller-facing gap where waiting for a launched instance's workload meant polling an artifact against a pre-guessed wall-clock deadline (the flaw behind a real calque incident). --completion-webhook-url mirrors the existing --spot-webhook-url (#228) fire-once POST pattern, fired from checkCompletion before the grace-period sleep. Independently, spawn:last-heartbeat is now stamped on every monitor tick (throttled to 1/min) unconditionally, giving a poller a liveness signal that needs no guessed timeout at all.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes spawn#497's two independent asks:
--completion-webhook-url— spored now POSTs a fire-once, best-effort notice when the completion sentinel fires, mirroring the existing--spot-webhook-url(Optional fire-and-forget spot-interruption webhook (the named agent.go:1066 enhancement) #228) pattern exactly. Fired fromcheckCompletionbefore the grace-period sleep/action, so a caller learns of completion as early as spored itself does. Shares--webhook-correlation/--webhook-timeoutwith the spot webhook. This lets a consumer (e.g. calque) register its own webhook/queue target instead of reinventing an S3-artifact-polling loop unaware of spored's own on-instance sentinel.spawn:last-heartbeatEC2 tag — stamped with the current time on every monitor tick (throttled to once/minute, matching the production monitor interval), unconditionally, no opt-in flag needed. Gives any poller (calque,spawn task run --wait, or a third party) a way to tell "still alive and ticking" from "hung" or "gone" — closing the gap where a caller had no choice but to pre-commit to one wall-clock number and treat exceeding it as failure.Both changes are additive: no existing behavior, flag, or tag changes shape when unconfigured (both new capabilities are strictly off by default except the heartbeat, which is unconditional but harmless to callers who never read it).
Real-world motivation (from the issue): a real vLLM run under calque was still legitimately executing at 40 minutes with no way to distinguish "almost done" from "stuck" other than the caller's own guessed deadline.
Test plan
go build ./...go vet ./...make check(fmt/vet/lint/short tests) — greencheckCompletion, heartbeat-tag throttle guard,buildTagsopt-in tag round-tripmake gen-docs—docs-gen/launch.mdregenerated and committed