Sample Header Ad - 728x90

Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

2 votes
0 answers
376 views
Why so much btrfs-send data for reflinks
I'm experiencing a very weird Btrfs "send" issue with file reflinks, and after so many trials, I found the details: First, I created a subvolume called "1": ```bash btrfs subvolume create 1 ``` and dd a file in it: ```bash dd if=/dev/urandom of=1/a bs=64 count=3110017 ``` Now make a read-only snapsh...
I'm experiencing a very weird Btrfs "send" issue with file reflinks, and after so many trials, I found the details: First, I created a subvolume called "1":
btrfs subvolume create 1
and dd a file in it:
dd if=/dev/urandom of=1/a bs=64 count=3110017
Now make a read-only snapshot by sudo btrfs subvolume snapshot -r 1 1_ro0, and then do cp --reflink=always 1/a 1/b, make a ro snapshot again: sudo btrfs subvolume snapshot -r 1 1_ro1. Now count the data size from "btrfs send":
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
It outputs 64864765, which is about 61.9MiB. This's so abnormal - I just copied it using reflink, I don't know why "btrfs send" outputs so much data. However, if the count above in dd is replaced with 3110016 (3110017-1), then did the routines above again, with wc -c on "btrfs send", you can get 604 (604 B), only 604 bytes, and this is what I expect. I also made a bash script to do those two tests:
#!/bin/bash

btrfs subvolume create 1

rm -rf 1/*
dd if=/dev/urandom of=1/a bs=64 count=3110017
sudo btrfs subvolume snapshot -r 1 1_ro0
cp --reflink=always 1/a 1/b
sudo btrfs subvolume snapshot -r 1 1_ro1
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
sudo btrfs subvolume delete 1_ro0 1_ro1


rm -rf 1/*
dd if=/dev/urandom of=1/a bs=64 count=3110016
sudo btrfs subvolume snapshot -r 1 1_ro0
cp --reflink=always 1/a 1/b
sudo btrfs subvolume snapshot -r 1 1_ro1
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
sudo btrfs subvolume delete 1_ro0 1_ro1

rm -r 1
If replace wc -c with btrfs receive --dump to inspect the "btrfs send" output data, with dd 3110016 value it will output a few "clone" entries like this:
clone           ./1_ro1/b                       offset=0 len=134217728 from=./1_ro1/a clone_offset=0
clone           ./1_ro1/b                       offset=134217728 len=64823296 from=./1_ro1/a clone_offset=134217728
...
But with dd value 3110017, it outputs:
write           ./1_ro1/b                       offset=197918720 len=49152
write           ./1_ro1/b                       offset=197967872 len=49152
write           ./1_ro1/b                       offset=198017024 len=49152
write           ./1_ro1/b                       offset=198066176 len=49152
write           ./1_ro1/b                       offset=198115328 len=49152
write           ./1_ro1/b                       offset=198164480 len=49152
... (so many more)
which contains a bunch of "write" entries. I also searched and noticed this site: https://www.spinics.net/lists/linux-btrfs/msg105951.html , and it's like talking about file holes, but in my case it obviously can't be. This is so weird and I don't know what I'm missing. My btrfs-progs version is 5.18.1-1 and the mount options from /proc/mount: rw,relatime,space_cache=v2,subvolid=5,subvol=/ Thanks!
bczhc (21 rep)
Jun 18, 2022, 03:58 PM • Last activity: Oct 10, 2024, 10:38 PM
0 votes
0 answers
118 views
what the `cp --reflink=always` will do if I try to copy directory that contains reflink to a diffrent disk?
I have a folder root1 on disk partition sda1 which contain reflink`/mnt/sda1/root1/a->/mnt/sda1/root1/raw/a` I have another folder root1 on disk partition sdb1, that has reflink `/mnt/sdb1/root1/a->/mnt/sdb1/backup/raw/a`. sda1 and sdb1 has same file system. It can either both be XFS or both be btrf...
I have a folder root1 on disk partition sda1 which contain reflink/mnt/sda1/root1/a->/mnt/sda1/root1/raw/a I have another folder root1 on disk partition sdb1, that has reflink /mnt/sdb1/root1/a->/mnt/sdb1/backup/raw/a. sda1 and sdb1 has same file system. It can either both be XFS or both be btrfs If I do copy -r --reflink=always /mnt/sda1/root1 /mnt/sdb1/root1.What will happen to my data? - case 1: the /mnt/sda1/root1/a and /mnt/sdb1/root1/a are exactly the same. - case 2: the /mnt/sda1/root1/a and /mnt/sdb1/root1/a are different in some blocks. but /mnt/sda1/root1/raw/a and /mnt/sdb1//backup/raw/a are the same. - case 3: all 4 a are different but most of the blocks are the same. Is the copy smart enough to maintain the block sharing on the /sdb1?
Wang (1395 rep)
Jul 5, 2024, 07:37 PM
21 votes
1 answers
8453 views
In Linux, which filesystems support reflinks?
`btrfs` supports reflinks, `XFS` supports reflinks [(since 2017 I think?)](https://blogs.oracle.com/linux/xfs-data-block-sharing-reflink#:~:text=Three%20years%20ago%2C%20I%20introduced,efficient%20use%20of%20storage%20hardware.). Are there any other filesystems that support it? ``` truncate -s 1G te...
btrfs supports reflinks, XFS supports reflinks [(since 2017 I think?)](https://blogs.oracle.com/linux/xfs-data-block-sharing-reflink#:~:text=Three%20years%20ago%2C%20I%20introduced,efficient%20use%20of%20storage%20hardware.) . Are there any other filesystems that support it?
truncate -s 1G test.file;
cp --reflink=always test.file ref.test.file;
hanshenrik (695 rep)
Jan 27, 2021, 10:21 AM • Last activity: Jan 17, 2024, 04:47 PM
5 votes
2 answers
1826 views
How to find out if the filesystem supports reflink copies?
I try to create a snapshot copy of a directory recursively, but it returns errors for every single file if the filesystem (custom shfs) does not support `reflink` copies: # cp -a --reflink=always /mnt/user/libvirt /mnt/user/libvirt_copy cp: failed to clone '/mnt/user/libvirt_copy/libvirt/libvirt.img...
I try to create a snapshot copy of a directory recursively, but it returns errors for every single file if the filesystem (custom shfs) does not support reflink copies: # cp -a --reflink=always /mnt/user/libvirt /mnt/user/libvirt_copy cp: failed to clone '/mnt/user/libvirt_copy/libvirt/libvirt.img' from '/mnt/user/libvirt/libvirt.img': Operation not supported cp: failed to clone '/mnt/user/libvirt_copy/libvirt/test.txt' from '/mnt/user/libvirt/test.txt': Operation not supported ... As it seems to be impossible to stop cp on the first error (?), which means useless looping through probably thousands of files, I instead like to pre-check if the filesystem supports reflinks. One idea is to search for a single file and do a testcopy: if ! find /mnt/user/libvirt -type f -print -quit | xargs -I {} sh -c 'dst_path=$(dirname $(echo {} | sed "s#^/mnt/user/libvirt#/mnt/user/libvirt_copy#")); mkdir -vp $dst_path && cp -a --reflink=always {} "$dst_path"'; then echo "Error: Filesystem does not support reflink" exit fi But this leaves the dir and an empty file which needs to be cleaned up afterwards: # find /mnt/user/libvirt_copy/ /mnt/user/libvirt_copy/ /mnt/user/libvirt_copy/libvirt.img # ls -lah /mnt/user/libvirt_copy/libvirt.img -rw------- 1 root root 0 Nov 1 08:58 /mnt/user/libvirt_copy/libvirt.img Isn't there an better way? Or maybe there is a solution to stop cp on the first error?
mgutt (547 rep)
Nov 1, 2022, 07:56 AM • Last activity: Jul 8, 2023, 03:47 AM
22 votes
1 answers
16826 views
Why does "cp -R --reflink=always" perform a standard copy on a Btrfs filesystem?
Btrfs support Copy-On-Write. I tried to use that feature to clone a directory: cp -R --reflink=always foo_directory foo_directory.mirror I expected the command to finish almost instantly (like a `btrfs subvolume snapshot`), but the `cp` command seems to perform a slow, standard copy. According to th...
Btrfs support Copy-On-Write. I tried to use that feature to clone a directory: cp -R --reflink=always foo_directory foo_directory.mirror I expected the command to finish almost instantly (like a btrfs subvolume snapshot), but the cp command seems to perform a slow, standard copy. According to the man page, I would expected --reflink=always to enforce Copy-On-Write: > When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if --reflink=auto is specified, fall back to a standard copy. **Questions:** - Do you know why --reflink=always doesn't work? - What options (or other commands) should I use instead?
Philipp Claßen (4967 rep)
Jul 30, 2015, 03:30 PM • Last activity: Jun 23, 2023, 08:29 AM
4 votes
1 answers
1594 views
GNU cp: What's the difference between `--link` and `--reflink=always`?
What's the difference between `--link` and `--reflink=always`? I use the following command as an `mv` substitute, and I am wondering if using `--reflink` is a better choice. ``` command gcp -r --link --archive --verbose "${opts[@]}" "$@" # delete the sources manually ```
What's the difference between --link and --reflink=always? I use the following command as an mv substitute, and I am wondering if using --reflink is a better choice.
command gcp -r --link --archive --verbose "${opts[@]}" "$@"
# delete the sources manually
HappyFace (1694 rep)
Jun 16, 2021, 08:57 AM • Last activity: Dec 4, 2021, 09:45 PM
8 votes
1 answers
1830 views
Detect btrfs reflinked files
It's clear, with `cp --reflink foo bar` I can copy a file on btrfs without wasting space. But how can I detect that the two files (foo and bar here) are using the same "space unit"? If they were hardlinks, I would use `[ foo -ef bar ]` for comparing or `stat -c%i foo` to get a unique id (inode in th...
It's clear, with cp --reflink foo bar I can copy a file on btrfs without wasting space. But how can I detect that the two files (foo and bar here) are using the same "space unit"? If they were hardlinks, I would use [ foo -ef bar ] for comparing or stat -c%i foo to get a unique id (inode in this case). Are there similar commands for btrfs reflinked files? I want to apply this on a 300G volume which is virtually 6T in size due to deduplication, so I want to avoid checksumming...
Daniel Alder (936 rep)
Oct 14, 2018, 01:26 PM • Last activity: Aug 24, 2021, 09:27 AM
11 votes
1 answers
8033 views
Is there a way to enable reflink on an existing XFS filesystem?
I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the `reflink` feature of XFS, but I did not do so when I first created the filesystem (I used the defaults). Is there a way that I can toggle the `reflink` feature on,...
I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the reflink feature of XFS, but I did not do so when I first created the filesystem (I used the defaults). Is there a way that I can toggle the reflink feature on, without having to backup and re-create the filesystem? And if not, can I expect that feature to be implemented in the future?
TheSola10 (231 rep)
Sep 14, 2019, 04:35 PM • Last activity: Aug 28, 2020, 01:58 PM
3 votes
0 answers
329 views
Combine mkfifo and cp --reflink to append files without copying data
I have huge files `{0..9}.bin` which I want concatenate into `out.bin`. I don't need the original files afterwards. So I was wondering, if this is possible by only modifying the filesystem index without copying the file contents (see https://unix.stackexchange.com/q/80429/234981 for efficient copy s...
I have huge files {0..9}.bin which I want concatenate into out.bin. I don't need the original files afterwards. So I was wondering, if this is possible by only modifying the filesystem index without copying the file contents (see https://unix.stackexchange.com/q/80429/234981 for efficient copy solutions). On modern file systems (e.g. btrfs) cp --reflink=always exists. Fifos are on the file system level (at least btrfs send also tracks fifos), so they should have information about the actual data blocks used. Therefore, cp --reflink=always should be able to determine the extend numbers on disk and re-use them. So I was wondering, if it is possible to use mkfifo in combination with cp --reflink=always? **Update** Currently, it is not working:
for i in {1..9}; do dd if=/dev/urandom of="in$i.bin" bs=5M count=200; done;
mkfifo fifo
cat in* >fifo &
cp --reflink=always fifo out.bin
results in cp: failed to clone 'out.bin' from 'fifo': Invalid argument Probably, it never will, since FIFOs have no information about the storage origin bug instead are just dumb pipes.
darkdragon (175 rep)
May 20, 2020, 08:46 AM • Last activity: May 20, 2020, 04:19 PM
0 votes
1 answers
292 views
Reflink copy diff
I want to make a behaviour like `btrfs send` and `btrfs receive`, but on a per-file basis. As far as I see it, I can use `cp -ax --reflink=always` to make a "snapshot" of the file. Let's assume that two such "snapshots" on two different hard disks are exactly equal. How do I `cp` the diff (and only...
I want to make a behaviour like btrfs send and btrfs receive, but on a per-file basis. As far as I see it, I can use cp -ax --reflink=always to make a "snapshot" of the file. Let's assume that two such "snapshots" on two different hard disks are exactly equal. How do I cp the diff (and only the diff) between two reflinked files like btrfs send -p would do it? Ideally, I would like to have a solution that works on any filesystem that is capable of making reflinks and not only on btrfs. Also ideally, I would like to run it without sudo. I know that I can look at the file extents with filefrag, but filefrag does not guarantee that the results are exact. I would rather copy some megabytes more than taking the risk of a corrupted file.
user1994405 (1 rep)
Apr 29, 2020, 02:58 PM • Last activity: Apr 29, 2020, 06:41 PM
8 votes
1 answers
1921 views
What does a rmlint's "clone" for btrfs do?
I was reading the [`rmlint`](http://rmlint.readthedocs.io/en/latest/index.html#) manual, and one of the duplicate handlers are clone and reflink: >· clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will physically delete duplicate extents. Needs at...
I was reading the [rmlint](http://rmlint.readthedocs.io/en/latest/index.html#) manual, and one of the duplicate handlers are clone and reflink: >· clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will physically delete duplicate extents. Needs at least kernel 4.2. > >· reflink: Try to reflink the duplicate file to the original. See also --reflink in man 1 cp. Fails if the filesystem does not support it. What exactly does this clone do, and how is it different from a reflink? What does the BTRFS_IOC_FILE_EXTENT_SAME ioctl do?
Dan (9622 rep)
Feb 13, 2018, 09:10 PM • Last activity: Mar 5, 2018, 12:49 PM
Showing page 1 of 11 total questions