feat(skillfs): supervise sidecar mounts - #2701
Draft
kongche-jbw wants to merge 1 commit into
Draft
Conversation
- monitor real FUSE I/O and remount failed sidecar sessions - reuse preflight cleanup with bounded retries and signal forwarding - cover recovery and shutdown without requiring /dev/fuse Signed-off-by: kongche-jbw <kongche.jbw@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On ACS, an unrelated container restart can leave the propagated SkillFS FUSE
session returning I/O errors while the SkillFS worker remains alive. The
existing kubelet liveness probe eventually restarts the sidecar, but it cannot
repair the mount in place and may leave a longer availability gap.
What changed
Add a PID 1 sidecar supervisor that reuses the existing real-I/O mount probe
and exact residual-FUSE cleanup path. It restarts unhealthy foreground mount
workers with consecutive-failure detection, a stable-health reset window,
bounded exponential backoff, and signal-safe shutdown. Keep kubelet probes as
the outer recovery boundary when in-container recovery is exhausted.
Related issue
no-issue: ACS sidecar recovery validation
User / Agent impact
The Kubernetes sidecar can now recover a failed SkillFS FUSE session without
waiting for the whole container to restart. Default in-container detection uses
a 2-second interval and two consecutive failures; after five consecutive failed
recovery cycles, the supervisor exits so kubelet liveness can restart it.
Risk and compatibility
The change introduces sidecar supervisor environment variables and moves PID 1
from the SkillFS worker to the supervisor. Cleanup remains restricted to a FUSE
filesystem at the exact configured mountpoint, and foreground and managed CLI
semantics are unchanged. The main residual risk is behavior under real ACS
mount propagation, which requires cluster validation.
Validation
cargo +1.86.0 fmt --all -- --checkcargo +1.86.0 clippy --workspace --all-targets -- -D warningscargo +1.86.0 test --workspacescripts/test.shDocker Hub before Dockerfile execution
Documentation and rollback
Container defaults and the reference Kubernetes manifest document the new
supervisor controls inline. Roll back by deploying the previous SkillFS image
and manifest, or by reverting commit
229321454; no persisted state migrationis involved.