You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Kubescape node agent keeps encrypted volumes from detaching after their pod is gone #6327
The Kubescape node agent can keep an encrypted Longhorn volume's device open after its pod is gone, so the volume cannot detach. This blocked an Email recovery copy's removal (#6041). It could also stop the encrypted mail store (#5387) from moving to another node.
What happens
The node agent bind-mounts the host's root, /var and the kubelet directory, so it sees every pod volume mounted on its node.
An unmount that reaches a busy copy fails silently. If the agent has a file open under a pod's volume when that volume is unmounted, its copy of the mount survives.
For an encrypted volume, the CSI driver then cannot close the LUKS device: cryptsetup luksClose … Device … is still in use. NodeUnstageVolume keeps failing, and the volume never detaches.
Observed on 2026-11-15: copy 42818236850's reader pod was deleted at 05:55 UTC. The publish and staging paths were unmounted, but the device stayed in use for at least 45 minutes. The only pod that had watched the reader up to that point was the node agent, which processed the container at 05:34 and its removal at 05:55.
Why it matters
The mail store's volume is encrypted. If the mail pod has to move to another node (a drain, an eviction or a node failure), the old node may be unable to close the device while the agent holds the mount. The volume is ReadWriteOnce, so the new pod could not attach it, and mail would be down until the agent pod is restarted by hand.
Options
Keep the node agent from opening files under pod volume mounts, or close them promptly, with an upstream issue if this is how it scans.
Narrow the agent's host mounts, so pod volume paths are outside its view.
At minimum, document the recovery: restarting the agent pod on the volume's node releases the device.
Acceptance
An encrypted Longhorn volume detaches within a normal interval after its pod is deleted, while the node agent runs on that node, verified on a disposable volume.
The cause in the node agent is confirmed, with upstream evidence or an upstream issue.
This has now hit the mail store (2026-11-15).#6333's deploy restarted private-mail-0 at 08:38 UTC, and the new pod was scheduled on another worker. The encrypted mail volume cannot detach from its old worker, because cryptsetup luksClose reports the device still in use. The new pod is Pending, and Email is down.
The likely holder, again an inference, is that worker's Kubescape node-agent pod. It started at 05:32, after the volume attached there at 03:44, and has been processing the mail pod. Releasing it means replacing that one DaemonSet pod, which has been asked of the maintainer as the fastest safe step. This issue needs a lasting fix before the mail store moves nodes again.
Likely cause in the pinned source (verified statically)
The deployed node-agent is built from kubescape/node-agent@4956ea26aabadb9daff42c4717842dd7bda26aa3, which uses inspektor-gadget@c91c23581ebf15ed6fb1faad6dc2700ceec783ca. In that inspektor-gadget commit:
pkg/container-collection/options.go, WithTracerCollection: the enricher opens /proc/<pid>/ns/mnt and /proc/<pid>/ns/net for every container it sees, and stores the descriptors as mntNsFd and netNsFd. An open namespace descriptor keeps that namespace alive.
pkg/container-collection/container-collection.go, AddContainer: every enricher runs first, then containers.LoadOrStore. When the container ID is already stored (loaded), it returns without container.close(). The rejected candidate is never registered or cached, so the cache's own closer never releases its descriptors either. Only an enricher that rejects a container closes it.
The same enricher, re-run on a container whose descriptor is already set, logs a warning and overwrites it, so the earlier descriptor leaks too.
A leaked mount-namespace descriptor keeps the container's mount namespace alive after the container exits. That namespace still has the pod's volume mounted, so the encrypted device stays open. That would explain all three observations of 2026-11-15:
luksClose reports the device in use after the host has unmounted it.
So the deployed agent compiles kubescape/inspektor-gadget@06b0d12baca0 (2026-08-26). The source analysis above, and the native descriptor proof, were against c91c2358. In the fork:
The distinct-duplicate leak is already fixed.AddContainer calls container.close() on the discarded candidate when LoadOrStore finds the ID already stored.
A same-pointer re-add is still wrong in two ways. The WithTracerCollection enricher still warns mntns reference already set and overwrites the open descriptor, which leaks the original. The duplicate branch then calls close() on that same object, which closes the registered container's own namespace descriptors.
Live evidence, read-only. The node-agent pod on the mail store's old worker logged no reference already set warning in its last six hours. It logged only its normal container callbacks for the mail pod, at 05:32 and 08:38. So the same-pointer path is not evidenced for this hold.
The dependency defect proven natively is therefore not the deployed behaviour, and the process holding the mail device is still unidentified. Replacing the agent pod releases the device, which still points at the agent process. Other code in it that opens files through a container's process root would produce the same symptom: an open file keeps a lazily unmounted filesystem, and its encrypted device, busy.
Next: a read-only node diagnostic that lists which process holds the volume's filesystem or device, and the agent's open namespace and file descriptors, before any code fix is chosen. The fork's same-pointer defect is real and worth fixing too, but it is not established as this incident's cause.
Mail restored, 2026-11-15. Email was down from 08:38 to 11:02 UTC. The mail store's encrypted volume could not detach from its old worker.
The reviewed recovery (#6335, run 43706667776) observed:
the volume pvc-6a149990-…, attached on the old worker through one attachment;
the CSI plugin's stuck luksClose in its log;
one node-agent pod there, node-agent-xhzjm, at the DaemonSet's current revision.
It deleted only that pod, with a UID precondition, and its replacement became Ready. The original attachment then went the ordinary way: COMPLETE … detached=ORDINARY.
Read-only afterwards:
private-mail-0 is Running and Ready on the new worker, with no restarts.
The copy step found the store already on the encrypted volume. Setup verified the configuration.
The volume is encrypted, attached and healthy, with three running replicas.
The web route answers 200.
That is the third time replacing a node-agent pod released an encrypted device today. Which process held it is still unproven at process level. #6338 fixes the inspektor-gadget fork's same-object descriptor defect, a real defect but not shown to be this cause. The secure-defaults adoption (#6333) moves the mail pod again, so it waits for #6338 to deploy. If it recurs, #6335 is the reviewed release path.
The patched node-agent runs on every node.#6342 merged at 14:41:52 UTC as 5101bf589. Its merge-group CI run passed, and the rollout happened during that run.
Read-only readback at 14:42 UTC:
Node
Pod
Started (UTC)
Restarts
Ready
Running image
autoscale-longhorn-v1-593d9e9b9f04c557
node-agent-gzzvs
14:19:44
0
True
sha256:b2965418…
prod-worker-1
node-agent-fx5t4
14:22:13
0
True
sha256:b2965418…
prod-worker-3
node-agent-sc4pj
14:24:41
0
True
sha256:b2965418…
autoscale-longhorn-v1-48cc0248100315b5
node-agent-krrtv
14:27:11
0
True
sha256:b2965418…
autoscale-longhorn-v2-13a45a5e556b2239
node-agent-jzjmm
14:29:37
0
True
sha256:b2965418…
prod-worker-2
node-agent-pgd2m
14:32:06
0
True
sha256:b2965418…
DaemonSet: generation 41 is observed. Desired, current, updated, ready and available are all 6; none is misscheduled. Every pod carries the same revision hash, 79ff96c66c.
Image: the image each container actually runs, imageID, is the published index sha256:b296541856c36514a3faeb58e43f4a26460b869c12953e6ddb95d2086db6bd59.
Flux: the kubescape HelmRelease and the infrastructure-controllers, infrastructure and apps Kustomizations are Ready.
Mail, unaffected:private-mail-0 has been Running and Ready on prod-worker-1 with 0 restarts. Its encrypted volume is attached and healthy with 3 of 3 replicas running.
Still open here: the fix stops the namespace-descriptor leak that we inferred held the device. It does not yet prove that a detach now succeeds. The first live test is the next time the mail pod is replaced, when #6333 deploys. If luksClose reports the device still in use again, the reviewed release-encrypted-volume-holder.yaml recovery stays available. This issue stays open until a replacement detaches cleanly without it.
Next mail stop:#6333 entered the merge queue at 14:45 UTC, second behind #6341. Its merge-group deploy will replace private-mail-0, expected between about 15:10 and 15:40 UTC. Please hold any storage cutover until I post the result here.
16:03:54 onwards: the Longhorn CSI plugin on prod-worker-1 could not unstage the volume. cryptsetup luksClose pvc-6a149990-5a81-496d-bf0a-54f347e9cd9d failed with Device … is still in use (exit 5), 13 times. The volume stayed attached to prod-worker-1 through its one VolumeAttachment, and no pod on that node used the claim.
16:13:02: I dispatched Release Encrypted Volume Holder, run 43734215278. It recorded holder node-agent-fx5t4, which was running the patched image sha256:b2965418… at the DaemonSet's current revision. It deleted that one pod with a UID precondition and waited for its Ready replacement, node-agent-f2g6t. It then saw the original attachment go away through the ordinary detach: ENCRYPTED_VOLUME_HOLDER_RELEASE=COMPLETE … detached=ORDINARY.
16:14:43:private-mail-0 was Ready on prod-worker-2 with the new bootstrap image sha256:7626ff9d…, with 0 restarts. The volume is attached there and healthy, with 3 of 3 replicas running.
Mail was down for about 11 minutes.
What this shows. The namespace-descriptor patch fixed a real descriptor leak in the gadget fork. It did not stop this hold. Replacing the node-agent pod on the volume's node has now released the device every time it was tried, including against the patched build. So the node-agent is still the likely holder, but it holds the device some other way than the leaked namespace descriptors. This is still an inference: no process on the node has been shown holding the device.
Next on this issue:
A read-only node diagnostic, for the next time the hold occurs. Before any recovery, it would record which process holds the device: open descriptors and mount-namespace membership on the node, plus the device-mapper open count. That would name the process instead of inferring it.
Until then, the reviewed workflow is the recovery for any encrypted volume that has to move between nodes.
This issue stays open.
For storage work: the mail stop is over. Any planned move of an encrypted volume between nodes can hit this hold again, and the workflow above is its recovery.
luksClose refused 4 times in 4 seconds, then closed on its own at 18:23:14. No pod was replaced.
The 4-second case matches the gadget fork's normal removal path. It polls /proc every second for the container's process, then closes the container's namespace descriptors after a 2-second cache delay, checked every 5 seconds. So a hold that lasts means that container's descriptors were never closed.
2. The patched node-agent still holds a namespace with nothing left in it. Using Talos reader access, I compared each node-agent's open namespace descriptors with the namespaces of every live process on its node:
Node
Mount namespaces held
With no live process
Network namespaces held
With no live process
autoscale-longhorn-v1-593d…
47
0
41
0
autoscale-longhorn-v2-13a4…
46
0
37
0
autoscale-longhorn-v1-48cc…
60
0
54
0
prod-worker-1
16
0
11
0
prod-worker-2
42
1
29
1
prod-worker-3
54
0
43
0
On prod-worker-2, the dead mount namespace is held at descriptor 172. That is inside the block of descriptors the node-agent opened in one go at startup (54–194); containers added later have higher numbers. The node-agent's own add and remove log does not account for it. Every container it logged as added and that has since exited was also logged as removed. So something other than the plain add/remove pairing opened, or kept, that descriptor.
What this means.
The holder is still the node-agent; replacing it has cleared every lasting hold.
The mechanism looks like a namespace descriptor that outlives its container. That fits a container the node-agent picked up when it started.
The source of that descriptor is not identified yet, so there is no root fix yet.
Next (one reviewed change): the release workflow will record, read-only, at the moment of the next hold and before it replaces anything:
the holder's dead namespace descriptors;
whether the holder's own mount table contains the volume's device;
the same two facts again after the replacement.
It will print only counts and process names, never paths, so no mailbox data reaches the run log. That turns the inference into proof and shows where to fix it.
Until then, the standing remedy is the reviewed release-encrypted-volume-holder.yaml dispatch for the stuck volume. It refuses unless the CSI plugin has just logged the refused luksClose, and it reports whether the ordinary detach followed. Moving an encrypted volume off a node whose node-agent started after the volume's pod should be expected to need it.
Activity
devantler commentedon Nov 15, 2026
This has now hit the mail store (2026-11-15). #6333's deploy restarted
private-mail-0at 08:38 UTC, and the new pod was scheduled on another worker. The encrypted mail volume cannot detach from its old worker, becausecryptsetup luksClosereports the device still in use. The new pod is Pending, and Email is down.The likely holder, again an inference, is that worker's Kubescape node-agent pod. It started at 05:32, after the volume attached there at 03:44, and has been processing the mail pod. Releasing it means replacing that one DaemonSet pod, which has been asked of the maintainer as the fastest safe step. This issue needs a lasting fix before the mail store moves nodes again.
🤖 Generated with Claude Code
devantler commentedon Nov 15, 2026
Likely cause in the pinned source (verified statically)
The deployed node-agent is built from
kubescape/node-agent@4956ea26aabadb9daff42c4717842dd7bda26aa3, which usesinspektor-gadget@c91c23581ebf15ed6fb1faad6dc2700ceec783ca. In that inspektor-gadget commit:pkg/container-collection/options.go,WithTracerCollection: the enricher opens/proc/<pid>/ns/mntand/proc/<pid>/ns/netfor every container it sees, and stores the descriptors asmntNsFdandnetNsFd. An open namespace descriptor keeps that namespace alive.pkg/container-collection/container-collection.go,AddContainer: every enricher runs first, thencontainers.LoadOrStore. When the container ID is already stored (loaded), it returns withoutcontainer.close(). The rejected candidate is never registered or cached, so the cache's own closer never releases its descriptors either. Only an enricher that rejects a container closes it.A leaked mount-namespace descriptor keeps the container's mount namespace alive after the container exits. That namespace still has the pod's volume mounted, so the encrypted device stays open. That would explain all three observations of 2026-11-15:
luksClosereports the device in use after the host has unmounted it.HostToContaineron the agent's host mounts (fix(storage): let node agents see host unmounts so encrypted volumes can detach #6322) did not help.The node-agent logs show repeated callbacks for the same container (
callbackCount), which would reach the duplicate path.This is still not node-level proof. A process-level check would read the agent's
/proc/<pid>/fdformnt:[…]links that belong to exited containers.Fix direction:
/proc/self/fdaround a duplicate add, using the real enricher.This lead came from the Longhorn migration session.
🤖 Generated with Claude Code
fix(storage): identify the node agent's current template by its revision
fix(storage): read the whole CSI log when matching the stuck close
fix(storage): record the holder release's dispatch-input disposition
devantler commentedon Nov 15, 2026
Correction: the deployed node-agent builds Kubescape's inspektor-gadget fork
kubescape/node-agent@4956ea26…requiresinspektor-gadget@c91c2358…, but itsgo.modreplaces it:So the deployed agent compiles
kubescape/inspektor-gadget@06b0d12baca0(2026-08-26). The source analysis above, and the native descriptor proof, were againstc91c2358. In the fork:AddContainercallscontainer.close()on the discarded candidate whenLoadOrStorefinds the ID already stored.WithTracerCollectionenricher still warnsmntns reference already setand overwrites the open descriptor, which leaks the original. The duplicate branch then callsclose()on that same object, which closes the registered container's own namespace descriptors.Live evidence, read-only. The node-agent pod on the mail store's old worker logged no
reference already setwarning in its last six hours. It logged only its normal container callbacks for the mail pod, at 05:32 and 08:38. So the same-pointer path is not evidenced for this hold.The dependency defect proven natively is therefore not the deployed behaviour, and the process holding the mail device is still unidentified. Replacing the agent pod releases the device, which still points at the agent process. Other code in it that opens files through a container's process root would produce the same symptom: an open file keeps a lazily unmounted filesystem, and its encrypted device, busy.
Next: a read-only node diagnostic that lists which process holds the volume's filesystem or device, and the agent's open namespace and file descriptors, before any code fix is chosen. The fork's same-pointer defect is real and worth fixing too, but it is not established as this incident's cause.
🤖 Generated with Claude Code
devantler commentedon Nov 15, 2026
Mail restored, 2026-11-15. Email was down from 08:38 to 11:02 UTC. The mail store's encrypted volume could not detach from its old worker.
The reviewed recovery (#6335, run 43706667776) observed:
pvc-6a149990-…, attached on the old worker through one attachment;luksClosein its log;node-agent-xhzjm, at the DaemonSet's current revision.It deleted only that pod, with a UID precondition, and its replacement became Ready. The original attachment then went the ordinary way:
COMPLETE … detached=ORDINARY.Read-only afterwards:
private-mail-0is Running and Ready on the new worker, with no restarts.healthy, with three running replicas.That is the third time replacing a node-agent pod released an encrypted device today. Which process held it is still unproven at process level. #6338 fixes the inspektor-gadget fork's same-object descriptor defect, a real defect but not shown to be this cause. The secure-defaults adoption (#6333) moves the mail pod again, so it waits for #6338 to deploy. If it recurs, #6335 is the reviewed release path.
🤖 Generated with Claude Code
devantler commentedon Nov 15, 2026
The patched node-agent runs on every node. #6342 merged at 14:41:52 UTC as
5101bf589. Its merge-group CI run passed, and the rollout happened during that run.Read-only readback at 14:42 UTC:
sha256:b2965418…sha256:b2965418…sha256:b2965418…sha256:b2965418…sha256:b2965418…sha256:b2965418…79ff96c66c.imageID, is the published indexsha256:b296541856c36514a3faeb58e43f4a26460b869c12953e6ddb95d2086db6bd59.kubescapeHelmRelease and theinfrastructure-controllers,infrastructureandappsKustomizations are Ready.private-mail-0has been Running and Ready on prod-worker-1 with 0 restarts. Its encrypted volume is attached and healthy with 3 of 3 replicas running.Still open here: the fix stops the namespace-descriptor leak that we inferred held the device. It does not yet prove that a detach now succeeds. The first live test is the next time the mail pod is replaced, when #6333 deploys. If
luksClosereports the device still in use again, the reviewedrelease-encrypted-volume-holder.yamlrecovery stays available. This issue stays open until a replacement detaches cleanly without it.Next mail stop: #6333 entered the merge queue at 14:45 UTC, second behind #6341. Its merge-group deploy will replace
private-mail-0, expected between about 15:10 and 15:40 UTC. Please hold any storage cutover until I post the result here.🤖 Generated with Claude Code
devantler commentedon Nov 15, 2026
The hold came back with the patched node-agent running. The reviewed recovery released it, and mail is back.
Timeline, all UTC:
private-mail-0. Mail stopped. The new pod was scheduled to prod-worker-2.cryptsetup luksClose pvc-6a149990-5a81-496d-bf0a-54f347e9cd9dfailed withDevice … is still in use(exit 5), 13 times. The volume stayed attached to prod-worker-1 through its one VolumeAttachment, and no pod on that node used the claim.node-agent-fx5t4, which was running the patched imagesha256:b2965418…at the DaemonSet's current revision. It deleted that one pod with a UID precondition and waited for its Ready replacement,node-agent-f2g6t. It then saw the original attachment go away through the ordinary detach:ENCRYPTED_VOLUME_HOLDER_RELEASE=COMPLETE … detached=ORDINARY.private-mail-0was Ready on prod-worker-2 with the new bootstrap imagesha256:7626ff9d…, with 0 restarts. The volume is attached there and healthy, with 3 of 3 replicas running.Mail was down for about 11 minutes.
What this shows. The namespace-descriptor patch fixed a real descriptor leak in the gadget fork. It did not stop this hold. Replacing the node-agent pod on the volume's node has now released the device every time it was tried, including against the patched build. So the node-agent is still the likely holder, but it holds the device some other way than the leaked namespace descriptors. This is still an inference: no process on the node has been shown holding the device.
Next on this issue:
This issue stays open.
For storage work: the mail stop is over. Any planned move of an encrypted volume between nodes can hit this hold again, and the workflow above is its recovery.
🤖 Generated with Claude Code
devantler commentedon Nov 15, 2026
New evidence, read-only (21:00–21:30 UTC): when the hold lasts, and a descriptor leak that remains in the patched build.
1. The hold lasts only when the node-agent started after the volume's pod.
luksCloserefused for 10 minutes. Cleared only by replacing the node-agent pod.autoscale-longhorn-v1-48cc…luksCloserefused 4 times in 4 seconds, then closed on its own at 18:23:14. No pod was replaced.The 4-second case matches the gadget fork's normal removal path. It polls
/procevery second for the container's process, then closes the container's namespace descriptors after a 2-second cache delay, checked every 5 seconds. So a hold that lasts means that container's descriptors were never closed.2. The patched node-agent still holds a namespace with nothing left in it. Using Talos reader access, I compared each node-agent's open namespace descriptors with the namespaces of every live process on its node:
On prod-worker-2, the dead mount namespace is held at descriptor 172. That is inside the block of descriptors the node-agent opened in one go at startup (54–194); containers added later have higher numbers. The node-agent's own add and remove log does not account for it. Every container it logged as added and that has since exited was also logged as removed. So something other than the plain add/remove pairing opened, or kept, that descriptor.
What this means.
Next (one reviewed change): the release workflow will record, read-only, at the moment of the next hold and before it replaces anything:
It will print only counts and process names, never paths, so no mailbox data reaches the run log. That turns the inference into proof and shows where to fix it.
Until then, the standing remedy is the reviewed
release-encrypted-volume-holder.yamldispatch for the stuck volume. It refuses unless the CSI plugin has just logged the refusedluksClose, and it reports whether the ordinary detach followed. Moving an encrypted volume off a node whose node-agent started after the volume's pod should be expected to need it.🤖 Generated with Claude Code
fix(storage): never report unread or overdue holder evidence