Skip to content

fix(remote): renew the lease while an artifact upload runs (#2946) - #2990

Open
thymikee wants to merge 9 commits into
mainfrom
fix/2946-upload-lease-heartbeat
Open

thymikee wants to merge 9 commits into
mainfrom
fix/2946-upload-lease-heartbeat

Conversation

@thymikee

@thymikee thymikee commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

A remote install uploaded its artifact before the install request was admitted, so nothing renewed the lease while bytes moved: a 449 MB APK at 1m47s against a 60s TTL expired the lease paying for the device (#2946). sendToDaemon now beats that lease during the upload.

  • heartbeatLease with no ttlMs renewed to the registry default, shortening any lease allocated above it. It now renews for the window the lease carries. Admission did the same with a proxy default of its own, unreachable from the client; it is retired.
  • The first beat fires immediately, each successor a third of the window the last beat renewed, armed from that beat's completion: no fixed cadence outlives a short lease, and a slow beat delays its successor instead of thinning the schedule.
  • A beat finding the lease gone, or this request unable to renew it, ends the phase and aborts the upload. Preflight, finalize, the PUT, and the legacy stream honour that signal; an aborted upload rejects with the signal's own reason, not a wrapped transport error a retry policy could misread.

Local and lease-less requests get no timer. Closes #2946

Gross diff ~1,600 is over the 1,000-line budget: the admission and cancellation halves cannot split without shipping the shortening bug.

Validation

Commit 410711c2a: pnpm check:affected --run, format, lint, typecheck, fallow, and wire-compat passed; new assertions are mutation-verified. CI green. No live provider run: needs a remote daemon and a >60s upload. agent-browser-lifecycle.test.ts fails pre-existing at bce6f526d.

…ready carries

`heartbeatLease` resolved an absent `ttlMs` through the registry's default resolver, so a caller that
heartbeats without repeating its allocation TTL silently shortened the lease to the daemon default.
Every renewal that is not asked to change the window now renews for the window the lease is living
on, which is the rule `refreshProtectedLease` already applied to protected work; that arithmetic moves
to `leaseOwnTtlMs` beside the other lease-scope rules so both callers read one definition.

The caller that heartbeats without a TTL is not asking for the default, it is asking for the same
lease to keep going. An admitted request does exactly that, and this is half of why a long upload
expired the lease paying for its own device (#2946).
…ler's side

A remote install uploaded the artifact from the caller before the install request was admitted, so
nothing renewed the lease while the bytes moved: the daemon protects a lease while admitted work runs
on it (#2509, ADR 0007), and an upload is the mirror image of that. A 449 MB APK that took 1m47s to
upload against the one-minute default TTL expired the lease paying for the device it was uploading to,
and the install then failed `Lease is not active` (#2946).

`sendToDaemon` now brackets the upload phase with beats over the same transport the command uses. A
beat names the command's lease scope, its own request id, and nothing else: the scope is what the
daemon needs, and reusing the install request would send the upload's own payload once per beat. Each
beat gets a fresh id because a beat that times out is canceled under its own.

Only a remote daemon uploads and only a remote daemon holds a billed device, so a local command and a
command that names no lease get no timer at all, and the interval starts rather than fires
immediately, so an install that beats never sends no extra request. A beat that finds the lease gone
ends the upload with that lease error rather than finishing bytes to a device nobody owns; a beat that
fails for any other reason is reported through diagnostics and survived, since a later beat covers one
lost request.

ADR 0007 gains the rule this closes: what protects work that happens before admission.
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.85 MB 4.85 MB +2.2 kB
Package (unpacked) 4.85 MB 4.85 MB +2.2 kB
Package (download) 1.45 MB 1.45 MB +714 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.7 ms 28.6 ms -0.2 ms
CLI --help 82.7 ms 82.4 ms -0.4 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread src/daemon-client/daemon-client.ts Outdated
Comment thread src/daemon/lease-registry.ts
Comment thread src/daemon-client/daemon-client.ts Outdated
Comment thread src/daemon-client/daemon-client.ts Outdated
Comment thread src/daemon-client/__tests__/daemon-client-upload-lease.test.ts
Comment thread src/daemon-client/daemon-client.ts Outdated
Admission named its own proxy default on every request, so a client that
rented a device for longer than that default lost the window on the next
command — the lease-side twin of the heartbeat bug in #2946, which the
client fix could not reach because the shortening happened at admission.
The window a lease carries is the one its client named when it allocated;
only a request naming its own window changes it now, which retires
DEFAULT_PROXY_LEASE_TTL_MS with no producer left.
The upload beat waited a fixed twenty seconds for its first renewal, so a
lease admitted with a shorter window lapsed while a perfectly valid upload
ran — the #2946 symptom one seam earlier. A beat answers with the window it
just renewed, so the phase now beats immediately and then a third of that
window after each beat lands: a beat slower than the cadence delays its
successor instead of replacing the schedule or silencing every beat behind
it, and a floor keeps a pathological window from becoming a request loop.

A beat refused for a missing or mismatched owner scope says this request can
never renew the lease, so retrying it only spent the upload against a lease
that had stopped renewing. Those reasons now end the phase beside the
lost-lease ones, and the phase hands its task an abort signal the upload
chain honours: preflight, finalize, the direct PUT, and the legacy stream all
run under the caller's cancellation combined with their own timeout, and a
canceled upload stops asking the daemon for a fresh ticket. node:http's own
request signal is what stops bytes already piped at a device nobody holds.
Every moved declaration is a client-side optional parameter — the request
bytes an older daemon parses are unchanged, and an aborted request is the
client disconnect it already handles — so each gets its own digest-keyed
compatibleChanges entry rather than a protocol bump. AbortSignal joins
TS_GLOBALS: a lib global with no declaration site to digest, like URL.
…inal

The beat's scope and window are fixed where it is built, so a daemon that
refuses them — a ttl outside [minLeaseTtlMs, maxLeaseTtlMs], an unusable
lease id — refuses every successor identically. Such a refusal carries no
reason to key on, so the code is the signal: end the phase the way a lost
lease does instead of spending the upload against a lease that stopped
renewing on the first beat.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 20 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/daemon/__tests__/request-admission.test.ts
Comment thread src/__tests__/upload-client-cancellation.test.ts
…on test

The byte counts are what prove the stream stopped early; waiting for the
server to see the socket close added a second promise that only loopback
timing could settle, and under a loaded coverage lane it never did inside
the test timeout. A smaller payload keeps the same pause-and-abort scenario
off the CPU.
The contract on the upload signal is that an aborted upload rejects with the
signal's own reason. The preflight leg honoured it and the stream leg did
not: node:http reports an aborted request as a transport error, and the
shared handler wrapped that into COMMAND_FAILED. That is not just the wrong
message — a wrapped cancellation is indistinguishable from a broken
transport, so the direct-upload retry policy is one refactor away from
re-preflighting for a fresh ticket after the caller asked for none. Finalize
wrapped the same way.

Both tests now pin the rejection instead of accepting anything, and the
admission test pins heartbeatAt so the window assertion cannot pass on a
lease that was never renewed.
finalizeDirectUpload and streamFileToHttpRequestAttempt changed shape; both
edits are client-side only — the bytes a protocol-2 daemon parses for an
uncancelled request are unchanged, and a canceled request is the disconnect
it already handles.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote install: lease is not renewed during artifact upload

1 participant