Crazy values

[ant7511]

I started to use btrfs compression to store disk images.. The values seemed to me incorrect so i made a try..

Created a file (approx 100G) with full zeroes..
The values seemed ok.

wrote few data to this file, in this case i created a loobpack device and made an ext4 formatted partition inside this file.
i unmounted. loopback detached..

This is what i got:
ant@voyager:/storage/4tb-s76/work$ ls -la zeroes.img
-rw-r--r-- 1 root root 103131643904 okt 12 19:49 zeroes.img

ant@voyager:/storage/4tb-s76/work$ sudo compsize zeroes.img
Type Perc Disk Usage Uncompressed Referenced
TOTAL 99% 516M 518M 518M
none 100% 516M 516M 516M
zstd 3% 76K 2.1M 2.0M

[kilobyte]

What does du say? Is this file sparse?

[Zygo]

The method of creating the file wasn't included, but I'm guessing truncate -s 100G from the output given.

truncate -s 100G produces a file that occupies 0B of space and still allows compression on future writes, then mkfs.ext4 would add ~500MB of data to that. fallocate -l 100G would produce a file that occupies 100GB of space + space occupied by many later writes to the file, and prevents compression on future writes, but the file is smaller and compsize reports no prealloc extents. head -c 100G /dev/zero or similar would produce a file with minimum size ~3GB (100GB * 0.03 for the maximum compression ratio on btrfs) but the file is much smaller than that.

mke2fs would write highly compressible data, but sparsely, so the compression doesn't get much chance to work, hence the tiny amount of zstd data (or prealloc data for some versions of mkfs.ext4). There would be a large extent count from that, but the extent counts are missing from the output.

It looks fairly normal to me:

# truncate -s 100G test
# mkfs.ext4 test
mke2fs 1.46.2 (28-Feb-2021)
Discarding device blocks: done
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 8db74c2a-4415-46aa-aec8-3347e73e4a4c
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: 
done

# compsize test
Processed 1 file, 98 regular extents (99 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      517M         517M         517M
none       100%       38M          38M         5.0M
prealloc   100%      478M         478M         512M

mkfs.ext4 does discards which turn into prealloc extents, so there's no zstd if you do it this way, but a copy of the test file looks like this:

# cp --reflink=never test test_cp
# compsize /tmp/test_cp
Processed 1 file, 100 regular extents (100 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL        8%      464K         5.0M         5.0M       
none       100%      204K         204K         204K       
zstd         5%      260K         4.8M         4.8M       

[ant7511]

No, i copied from /dev/zero

dd if=/dev/zero of=./zeroimg.img bs=1M status=progress

102684950528 bytes (103 GB, 96 GiB) copied, 174 s, 590 MB/s^C
97936+0 records in
97936+0 records out
102693339136 bytes (103 GB, 96 GiB) copied, 174,014 s, 590 MB/s

compsize ./zeroimg.img

Type Perc Disk Usage Uncompressed Referenced
TOTAL 3% 2.9G 95G 95G
zstd 3% 2.9G 95G 95G

mkfs.ext4 ./zeroimg.img

compsize ./zeroimg.img

Type Perc Disk Usage Uncompressed Referenced
TOTAL 100% 517M 517M 517M
none 100% 517M 517M 517M

Now i checked the disk space after each step, the compsize shows the disk usage correctly, because after a dd copy 2.99M the disk usage,and after mkfs.ext2 the disk usage become only 432M.

[Zygo]

mkfs.ext4 does hole punching, so the end result will usually be the same as the truncate:

# strace mkfs.ext4 test
[...]
fsync(3)                                = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=107374182400, ...}) = 0
ioctl(3, BLKDISCARDZEROES, 0x7ffc1e027c10) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKROGET, 0x7ffc1e027c10)      = -1 ENOTTY (Inappropriate ioctl for device)
uname({sysname="Linux", nodename="tester39", ...}) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 0, 16777216) = 0
write(1, "Discarding device blocks: ", 26Discarding device blocks: ) = 26
write(1, "    4096/26214400", 17    4096/26214400)       = 17
write(1, "\10\10\10\10\10\10\10\10\10\10\10\10\10) = 17
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 16777216, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 2164260864, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 4311744512, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 6459228160, 2147483648) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 8606711808, 2147483648) = 0
[...]

mkfs.ext4 discards any previous extents in the file, so the end result is the same as the truncate case. truncate is much faster because it doesn't waste 100GB of iops.