shim: make container teardown timeouts configurable - #2855
Open
Jonas Heinle (Kataglyphis) wants to merge 1 commit into
Open
shim: make container teardown timeouts configurable#2855Jonas Heinle (Kataglyphis) wants to merge 1 commit into
Jonas Heinle (Kataglyphis) wants to merge 1 commit into
Conversation
Jonas Heinle (Kataglyphis)
force-pushed
the
feature/configurable-teardown-timeout
branch
from
August 6, 2026 10:50
1cd2944 to
6830a70
Compare
Jonas Heinle (Kataglyphis)
added a commit
to Kataglyphis/Kataglyphis-ContainerHub
that referenced
this pull request
Aug 6, 2026
…ement Filed microsoft/hcsshim#2855 as a draft from Kataglyphis/hcsshim:feature/configurable-teardown-timeout. Corrects a wrong claim in the README: hcsshim enforces a DCO check in addition to the Microsoft CLA. The repo README advertises only the CLA, so the PR went up with a failing DCO gate and needed an amend with -s. Both the requirement and the fix are now written down, and the in-tree patch carries the Signed-off-by trailer. Still open: the issue from ISSUE.md is not filed, the comment on Windows-Containers#547 is not posted, and the runtime measurement still comes from the constants-in-place build rather than the submitted env-var build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree |
Jonas Heinle (Kataglyphis)
marked this pull request as ready for review
August 7, 2026 08:25
Jonas Heinle (Kataglyphis)
added a commit
to Kataglyphis/Kataglyphis-ContainerHub
that referenced
this pull request
Aug 7, 2026
…ry room Both drivers now gate on the two conditions that cost hours when discovered late. Both were manual checklist items in docs/windows-host-setup.md D3, and both bit within the last 24 hours. Assert-DiskHeadroom (both lanes): refuses below 40 GB free. Under ~25 GB hcsshim stops failing honestly -- on 2026-08-06 a full chain ran 2.5 h and died with "ninja is not recognized" at 4.8 GB free, and the poisoned snapshots it left (0xb7, 0x20) cost two more runs to sidestep. -MinFreeGb tunes the floor, -SkipHostChecks overrides deliberately. Assert-ShimPatch (BuildKit lane only -- the classic lane's run+commit path uses Hyper-V isolation, where the teardown-timeout defect does not apply): the patched runhcs shim is a LOCAL patch pending microsoft/hcsshim#2855, and any Stevedore/containerd update silently restores the stock binary. The symptom is ExportLayer 0x3 at the first heavy media finalize, hours in, after the compile is already paid for. Detection is by size because the shim logs its effective timeout at Debug, which never reaches containerd's log -- so a KNOWN-STOCK size fails hard while an unrecognised size only warns, since that list will rot as hcsshim moves and refusing to build on a newer patched shim would be worse than the risk. Invoke-BkStage takes -MaxAttempts (default 3, unchanged) and the media merge stage passes 5. It fans in three branch images, does far more mount work than any other stage, and is the only one measured burning its whole budget: two `failed to mount {windows-layer}` failures on 2026-08-06, green only on the third and last attempt. Retries are cheap -- completed RUN vertices stay cached, only the failed finalize re-runs. Verified: lint 104 files 0/0; Invoke-Tests.ps1 287/287; both gates exercised on this host through their happy, throw and -Force paths; both drivers bind the new parameters. NOT changed, deliberately: the VS bootstrapper fallback. setup-vs.ps1 already tries the major-pinned aka.ms channel then falls back to aka.ms/vs/stable with a warning and a wider retry budget, which is exactly what happened on 2026-08-06 -- the design handled it. Worth knowing though that the major-18 alias fails consistently, so the VS major pin is currently decorative and every base build gets `stable`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
|
Can i provide further information to this PR? |
Jonas Heinle (Kataglyphis)
force-pushed
the
feature/configurable-teardown-timeout
branch
from
August 27, 2026 23:09
701fbc4 to
e2b180f
Compare
The shim hardcodes 30s for container shutdown, for terminate, for the DeleteExec resource cleanup and for the delete command's wait. Teardown of a process isolated container is host-side work: detaching the layer filter stack and flushing the registry hives into the scratch. It scales with how much the container wrote. We measured 117s for one OpenCV build container on Windows 11 26200 with ltsc2025 base images. When 30s expires mid-flush the container is terminated while its scratch is still being written, and every later finalize of that snapshot fails with hcsshim::ExportLayer 0x3. The damage survives fresh snapshots and host reboots. All silo processes do exit, so no in-container mitigation helps: overriding WaitToKillServiceTimeout and a full pre-exit teardown of non-essential services both lost the notification the same way. Make the limits configurable via the environment the shim inherits from containerd, named after CONTAINERD_SHIM_RUNHCS_V1_WAIT_DEBUGGER: CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT hcsTask.close, delete CONTAINERD_SHIM_RUNHCS_V1_TASK_CLOSE_TIMEOUT hcsTask.DeleteExec DeleteExec waits on the channel close closes, so the second is derived from the first when it is not set. Defaults stay 30s. Also log how long a successful shutdown took, which is the number needed to size the timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonas Heinle <jonasheinle@googlemail.com>
Jonas Heinle (Kataglyphis)
force-pushed
the
feature/configurable-teardown-timeout
branch
from
August 27, 2026 23:19
e2b180f to
1925142
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Heavy container builds need more than 30s to tear down. This PR makes the limit configurable.
Problem
Tearing down a process isolated container is host-side work: the layer filter stack is detached and the container's registry hives are flushed back into its scratch. The cost scales with how much the container touched the filesystem. Most workloads finish in well under a second, source builds do not.
Measured on my host: 117s for a single OpenCV build container.
HcsShutDownComputeSystemreturned in milliseconds, the completion notification arrived 117s later.The shim gives up at 30s and terminates the container while its scratch is still being written. The snapshot is then permanently damaged - every later finalize fails with
and it survives fresh snapshots and host reboots, because the unflushed hive deltas live inside
sandbox.vhdx.All silo processes do exit, so this is not a container-side problem. I tried overriding
WaitToKillServiceTimeoutand a full pre-exit teardown of non-essential services - both lost the notification identically.Change
Two environment variables, which the shim inherits from containerd, named after the existing
CONTAINERD_SHIM_RUNHCS_V1_WAIT_DEBUGGER:CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUThcsTask.close, and thedeletecommandCONTAINERD_SHIM_RUNHCS_V1_TASK_CLOSE_TIMEOUThcsTask.DeleteExecDeleteExecwaits on the channelclose()closes, so raising only the first would not help. When the second is unset and the first was raised, it is derived to coverclose()'s worst case. Defaults stay 30s, so behaviour is unchanged unless a host opts in.The PR also logs how long a successful shutdown took - the number you need to size the timeout, and currently not observable.
Left alone on purpose: the
SIGKILLtimer (guards the hosting UVM),cmd/runhcs(separate binary, already allows 5 minutes) andcmd/containerd-shim-lcow-v2(different shim, same pattern though).Verification
Go 1.26.5,
windows/amd64: build,gofmt,go vet,golangci-lintwith the CI-pinned v2.11 andgo test ./cmd/containerd-shim-runhcs-v1/all clean.Happy to adapt anything to your guidelines.