Skip to content

qfile-unpacker: fix concurrent mount/unmount race via flock (#5950) - #675

Open
datenbar wants to merge 1 commit into
QubesOS:mainfrom
datenbar:fix/issue5950
Open

datenbar wants to merge 1 commit into
QubesOS:mainfrom
datenbar:fix/issue5950

Conversation

@datenbar

@datenbar datenbar commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Fixes QubesOS/qubes-issues/issues/5950

Problem

When two qvm-copy operations to the same target VM run concurrently,
each invocation of qfile-unpacker independently bind-mounts and later
unmounts incoming_dir. Since both mounts stack at the same path in
the same mount namespace, umount2() can remove the wrong process's mount layer, causing:

qfile-unpacker: Fatal error: Cannot umount incoming directory (error type: Invalid argument)

Fix

Instead of every invocation independently mounting and unmounting,
coordinate via flock() on incoming_dir itself :

  • Each worker holds a LOCK_SH on incoming_dir for its entire lifetime.

  • On entry: attempt LOCK_EX (non-blocking). Winner checks whether
    already mounted, mounts if not, downgrades to LOCK_SH. Loser blocks
    on LOCK_SH (waiting for the current holder's mount decision), then
    re-checks mount state, if still unmounted then releases SH and retries EX.

  • On exit: release LOCK_SH, blocking-acquire LOCK_EX, check
    mount state, unmount only if still mounted.

  • Crash safety comes from flock() itself since kernel cleans up after a process dies.

  • mount()/umount2() use the resolved absolute path throughout, since a process's "." can end up resolving through an lazily-detached mount reference.

Testing

Validated qfile-unpacker.c linked
against minimal local stubs of gui-fatal.h/libqubes-rpc-filecopy.h
(do_unpack_ext replaced with a sleep to simulate copy duration, no
change to any mount/lock logic under test).

Tested with N=64 concurrent qfile-unpacker, invocations and numerous iterations with varied timings:

  • Normal timing: clean, 0 failures.
  • --copy-us 0 (max mount/unmount churn): 0 incorrect outcomes. Some
    iterations show >1 mount/unmount cycle per batch (late joiner races a
    teardown) expected self-healing.
  • Random SIGKILL fault injection, incl. pre-mount-completion kills:
    recovers correctly every time.

Important Notes

The mount-status check uses statx(STATX_MNT_ID), which requires kernel
5.8+. I haven't been able to confirm Qubes' actual minimum supported
kernel floor for VMs/templates.

AI disclosure

Used Claude among some other models to discuss, research and write tests against my proposed fixes and review the final file before the files were reviewed and verified by me.

Coordinate concurrent invocations on incoming_dir via flock() instead
of each independently mount/unmounting, which could remove the wrong
process's bind-mount layer. First worker in mounts, last one out
unmounts.
@datenbar

datenbar commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

CI failed due to three runners getting StoragePoolExceptions

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.71%. Comparing base (9c58097) to head (3bbb89d).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #675   +/-   ##
=======================================
  Coverage   71.71%   71.71%           
=======================================
  Files           3        3           
  Lines         502      502           
=======================================
  Hits          360      360           
  Misses        142      142           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qfile-unpacker error when qvm-copying multiple files simultaneously

1 participant