Skip to content

fix(pipeline): raise turnHardCap 600s -> 1800s - #9

Merged
sixvolts merged 1 commit into
mainfrom
ci/raise-turn-cap
Sep 2, 2026
Merged

fix(pipeline): raise turnHardCap 600s -> 1800s#9
sixvolts merged 1 commit into
mainfrom
ci/raise-turn-cap

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Canyon lost a turn to the hard cap at exactly 10m0.002s: six iterations deep
(read_page x4, update_page x2 over 10k-char pages), killed on iteration 5 of 10.

600s was chosen when chat ran on mainframe. Since that box was retired furnace
carries chat AND research on the same six slots, decoding at roughly 33-43
tok/s, so one iteration writing ~2.3k tokens costs 55-67s and six iterations
simply does not fit. Two wikiknowledge extracts in the same turn also failed
with context deadline exceeded, which is the backend saying it is saturated.

The failure mode is worse than a timeout. Tool side effects commit as each tool
runs, but the transcript is only persisted at end of turn - so both page updates
landed, with revisions at 01:44:32 and 01:49:25, while the user saw an empty
reply and reasonably concluded nothing had happened. Work done, no record of it.

This is a stopgap and the comment says so. A fixed ceiling cannot tell a turn
doing steady useful work from one wedged on a hung backend; the right shape is
an idle watchdog that resets on progress (each completed iteration or tool
dispatch), killing a stuck turn in ~2min while letting a productive one run as
long as it keeps earning it. Until that exists, err long: a killed turn loses
the transcript but keeps the side effects, which is the confusing outcome.

Not addressed here: persisting partial turn state on cap-cancellation, which
would make even a killed turn leave a record.


Opened automatically after the tiers check passed on 7d18b4cf.

The check is attached to this branch head, so this is mergeable now.

Canyon lost a turn to the hard cap at exactly 10m0.002s: six iterations deep
(read_page x4, update_page x2 over 10k-char pages), killed on iteration 5 of 10.

600s was chosen when chat ran on mainframe. Since that box was retired furnace
carries chat AND research on the same six slots, decoding at roughly 33-43
tok/s, so one iteration writing ~2.3k tokens costs 55-67s and six iterations
simply does not fit. Two wikiknowledge extracts in the same turn also failed
with context deadline exceeded, which is the backend saying it is saturated.

The failure mode is worse than a timeout. Tool side effects commit as each tool
runs, but the transcript is only persisted at end of turn - so both page updates
landed, with revisions at 01:44:32 and 01:49:25, while the user saw an empty
reply and reasonably concluded nothing had happened. Work done, no record of it.

This is a stopgap and the comment says so. A fixed ceiling cannot tell a turn
doing steady useful work from one wedged on a hung backend; the right shape is
an idle watchdog that resets on progress (each completed iteration or tool
dispatch), killing a stuck turn in ~2min while letting a productive one run as
long as it keeps earning it. Until that exists, err long: a killed turn loses
the transcript but keeps the side effects, which is the confusing outcome.

Not addressed here: persisting partial turn state on cap-cancellation, which
would make even a killed turn leave a record.
@sixvolts
sixvolts merged commit a46e332 into main Sep 2, 2026
1 check passed
@sixvolts
sixvolts deleted the ci/raise-turn-cap branch September 2, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant