Round transition leaves the GPU idle for 4-87s (avg ~37s) between `new round` and the miner actually starting work

[ntllinh2511]

Summary

After every round change, there is a silent gap -- zero log output from either the coordinator or the miner -- between the new round generation=N ... log line and the miner's self-feeding session launching with N/N nonces active line. During this gap the GPU is not computing (confirmed via nvidia-smi, 0% activity attributable to the miner process). Measured across 14 consecutive round transitions on a single node, the gap averaged 37.5s with a range of 4.1s to 86.7s -- no round transitioned in anywhere close to the same amount of time twice, which suggests the delay isn't a fixed/expected setup cost but something variable and likely fixable.

At a typical round length of ~8-9 minutes on this node's topology, ~37.5s of dead time per round is roughly 7% of total round time spent not mining -- before the new Aglais points program (which rewards per-qblock participation), this was just lost throughput; now it's also lost participation points.

Environment

• quip-coordinator version: 0.3.2, protocol 1
• Miner: quip-cuda-sa
• GPU: RTX 5090, driver supports CUDA 12.9
• Host: Docker Compose deployment (nodes.quip.network repo), Ubuntu-based container, colocated validator (ws://quip-validator:9944, no network hop)
• Topology: nodes=4577 edges=41514, hash 0xcbec1eb4e9dcc7ae343b386902d4dbba19b694a66cebf95bc06bd9260c3270e7
• Nonces: 170/170 self-feeding session, [gpu] utilization = 100

Data -- 14 consecutive round transitions, same node, same 2h window

Gap measured as: timestamp of self-feeding session launching minus timestamp of the new round generation=N line that precedes it (both timestamps taken from docker logs, UTC).

generation	gap (s)
64	23.3
65	18.3
66	4.1
67	26.4
68	34.5
69	63.9
70	23.2
71	48.9
72	36.9
73	55.0
74	47.0
75	8.1
76	49.0
77	86.7

n=14, min=4.1s, max=86.7s, avg=37.5s

Example -- generation=73 in full, unfiltered

2026-09-11T13:35:56.315202Z  INFO quip_coordinator::chain::real: proof included and dispatched successfully block=0x63f6c2405d7ca506abc838bc82bbd6ecb947b5f6e798985e82a6c21d83c09ac0 number=134685
2026-09-11T13:35:58.331286Z  INFO quip_coordinator::runtime: miner throughput miner=cuda-0 jobs_completed=223 jobs_completed_total=93098 staged=256 window=256
2026-09-11T13:35:58.331300Z  INFO quip_coordinator::runtime: coordinator alive block=134685 generation=72 results_validated=93098 best_energy=-14540
2026-09-11T13:36:00.355568Z  INFO quip_solver_core::session: [quip-miner-cuda] progress: 93100 jobs | 2.5 jobs/s | reads=217 sweeps=1739 | best=-14568 | requires energy<=-14541, solutions>=1
2026-09-11T13:36:06.626478Z  INFO quip_coordinator::runtime: new round generation=73 qblock_id=1249 block=134686 cancelled_jobs=253 miners_told=1 topology=0xcbec1eb4e9dcc7ae343b386902d4dbba19b694a66cebf95bc06bd9260c3270e7 nodes=4577 edges=41514 max_energy=-14524 min_solutions=1 min_diversity_milli=0 allowed_h_milli=[0] allowed_j_milli=[-1000, 1000] allowed_spin_milli=[-1000, 1000]

  <-- nothing logged for 55 seconds -->

2026-09-11T13:37:01.667876Z  INFO quip_coordinator::runtime: miner throughput miner=cuda-0 jobs_completed=4 jobs_completed_total=93102 staged=256 window=256
2026-09-11T13:37:01.667893Z  INFO quip_coordinator::runtime: coordinator alive block=134696 generation=73 results_validated=93102 best_energy=none
quip-miner-cuda: self-feeding session launching with 170/170 nonces active
2026-09-11T13:37:59.802953Z  INFO quip_solver_core::session: [quip-miner-cuda] attempt bdf360b8812ce8d2..: energy -14402, valid 0/210 | 1m 48s wall, 49.3s device

The first attempt after the relaunch reports 1m 48s wall, 49.3s device -- the ~59s gap between wall and device time on this specific attempt lines up almost exactly with the ~55s silent gap above, confirming the delay is before device compute starts, not an artifact of how "wall" is measured.

Expected behavior

Either:

• The gap between round declaration and the miner resuming work is small and roughly constant (some fixed, unavoidable setup cost is fine -- e.g. re-staging topology data), or
• If the delay is inherent to some network/chain-side step (fetching mempool/snapshot data, waiting on chain finality, etc.), it would help to have some log line during that window so an operator watching logs (or a watchdog script) can tell "still working" apart from "actually stuck" — right now a 4s gap and an 87s gap look identical from the outside until they resolve.

Actual behavior

Gap is large, highly variable (4-87s, ~9x spread), and completely silent -- nothing logged by either the coordinator or the miner subprocess during it.

Possible impact

• ~7% of round time not spent mining (rough estimate from this sample: 37.5s avg gap / ~510s avg round length)
• With the new Aglais points program rewarding per-qblock participation, this dead time now has a direct points cost in addition to the throughput cost
• Indirectly contributed to a false-positive stall detection in our own watchdog tooling in the past: two short rounds landing back-to-back (each already carrying this gap) can add up to several minutes with zero completed attempts, which external monitoring can mistake for a hung miner

Would help to know

• What happens on the coordinator side during this window (topology re-encoding? job batch preparation for the 256-wide staging window? IPC round-trip to the miner subprocess?) -- happy to pull more logs at --log-level debug if that would help narrow it down.