EBUSY when remove thin device

[huang-jl]

I find that I encounter EBUSY of device remove of thin device, the error message seems like:

DM Core error: low-level ioctl error due to nix error; ioctl number: 4, input header: Some(DeviceInfo { version: Version { major: 4, minor: 0, patch      : 0 }, data_size: 16384, data_start: 312, target_count: 0, open_count: 0, flags: DmFlags(0x0), event_nr: 4222638, dev: Device { major: 0, minor: 0 }, name: Some(DmNameBuf { inn      er: \\\"thin_dev_1\\\" }), uuid: None }), header result: Some(DeviceInfo { version: Version { major: 4, minor: 48, patch: 0 }, data_size: 16384, data_start: 312,       target_count: 0, open_count: 0, flags: DmFlags(0x0), event_nr: 4222638, dev: Device { major: 0, minor: 0 }, name: Some(DmNameBuf { inner: \\\"thin_dev_1\\\" }), u      uid: None }), error: EBUSY: Device or resource busy

I find that the device mapper will try about 5 times when ioctl meet EBUSY errors, but it still fails in my scenario. When it fails, I monitor which process preventing it from delete the thin device, via the lsof tool. I find that a process named (udev-worker) holds reference to the thin device.

I am not familiar with udev (systemd-udevd), I guess the DM_DEV_REMOVE_CMD ioctl triggers the udev to handle events like "remove" events? But I am not sure that will udev receive and process any event before the ioctl has actually succeed (the DM_DEV_REMOVE_CMD failed with EBUSY and the thin device has not been deleted).

Does anyone have ideas on solve this problem?

[mulkieran]

@huang-jl Have you confirmed that nothing has the device open before you issue the remove ioctl?

[huang-jl]

Additional Information

1. My kernel version is 6.8.0-83-generic #83-Ubuntu.
2. The program that uses devicemapper-rs to call DM_DEV_REMOVE_CMD is running in a container. I bind mount /dev and /run/udev/control into the container. I also set --ipc=host to enable the container to use semaphore.

[huang-jl]

@mulkieran Yes, there is nothing has the device open. If there are other processes open the device, I should see them from lsof command. Actually, after waiting for more time, the device can be removed. Besides, I write a lot of data in the thin device (the capacity of thin dev is 80G, I write about 30G into thin devices) before remove it.

[huang-jl]

@mulkieran The same. If I use dmsetup to remove the device without using devicemapper-rs at all, it still shows EBUSY errors like following:

device-mapper: remove ioctl on thin_dev_7  failed: Device or resource busy
Command failed.

I suspect there might be some time-consuming udev rules?

[huang-jl]

Not sure if this info helps, I remove multiple thin devices concurrently.

[mulkieran]

The trick is to find out what udev activity is causing all this...you might want to try running "udevadm monitor" while attempting the removal.

[huang-jl]

After further digging what has happened. I find:

1. The remove get EBUSY typically when I execute a large amount of data write into the thin device.
2. The EBUSY will even last over minutes, even all userspace program that open the thin devices has exited.
3. I find that iostat shows substantial write traffic even after all userspace program using thin devices has exited.
4. There are many kworkers comsume CPU as following:

2973953 root      20   0       0      0      0 I  33.3   0.0   1:46.00 kworker/u180:5-loop4
2976143 root      20   0       0      0      0 I  33.3   0.0   1:22.55 kworker/u180:12-loop4
2976145 root      20   0       0      0      0 D  33.3   0.0   1:28.84 kworker/u180:14+dm-thin

The loop4 is the loop devices that uses as the backing data of thin pool. (i.e., the data device of thin pool is loop4).
5. I find that using dmsetup remove -f to remove the thin devices will not throw EBUSY immediately, but will get stuck.
6. Using strace on dmsetup remove -f, finding that it get stuck in DM_DEV_SUSPEND for long time (e.g., couple of minutes). After that, it start to remove the device.

I guess is probably due the the kernel tries to flush the data of thin devices to its backing storage? Since I write large amount of data (I am using fio to test performance, generating around 20-30GB of data write), it might takes kernel to flush for a long time, even after the userspace program using thin devices has exited.

@mulkieran Do you have any ideas or similar scenario (writing a lot of data, and remove the thin devices immediately)?

Really thanks for your follow and reply on my question.

[mulkieran]

I think your guess is exactly correct. Unless called explicitly with DM_NOFLUSH flag, suspend ioctl will try to flush all the data. When you're using the -f flag, that means go ahead with the removal even if the device is in use, so dmsetup will run the suspend to flush the data from the device first, to achieve a cleaner remove. Based on man page here:

              Removes a device.  It will no longer be visible to dmsetup.  Open
              devices  cannot  be  removed, but adding --force will replace the
              table with one that fails all I/O.  --deferred  will  enable  de‐
              ferred  removal of open devices - the device will be removed when
              the last user closes it. The deferred removal feature is support‐
              ed since version 4.27.0 of the device-mapper driver available  in
              upstream  kernel  version  3.13.   (Use  dmsetup version to check
              this.)  If an attempt to remove a device fails, perhaps because a
              process run from a quick udev rule temporarily opened the device,
              the --retry option will cause the operation to be retried  for  a
              few  seconds  before  failing.  Do NOT combine --force and --ude‐
              vcookie, as udev may start to process udev rules in the middle of
              error target replacement and result in nondeterministic result.

likely what is happening is that since you are using the -f flag, you're getting the table replacement described in the man page. A new table is loaded, and then to replace the running table with the new one it is necessary to call DM_DEV_SUSPEND which tries to do the flush. The replacement can't happen until the suspend is completed, and removal can't happen until replacement. Perhaps you should try to close the devices instead of using --force, but without knowing more about what you are trying to accomplish, I can not say.

For the devicemapper library things are a bit different, but since I do not know what you are doing in detail, I can not say.