Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
2
votes
0
answers
376
views
Why so much btrfs-send data for reflinks
I'm experiencing a very weird Btrfs "send" issue with file reflinks, and after so many trials, I found the details: First, I created a subvolume called "1": ```bash btrfs subvolume create 1 ``` and dd a file in it: ```bash dd if=/dev/urandom of=1/a bs=64 count=3110017 ``` Now make a read-only snapsh...
I'm experiencing a very weird Btrfs "send" issue with file reflinks, and after so many trials, I found the details:
First, I created a subvolume called "1":
btrfs subvolume create 1
and dd a file in it:
dd if=/dev/urandom of=1/a bs=64 count=3110017
Now make a read-only snapshot by sudo btrfs subvolume snapshot -r 1 1_ro0
, and then do cp --reflink=always 1/a 1/b
, make a ro snapshot again: sudo btrfs subvolume snapshot -r 1 1_ro1
. Now count the data size from "btrfs send":
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
It outputs 64864765
, which is about 61.9MiB. This's so abnormal - I just copied it using reflink, I don't know why "btrfs send" outputs so much data.
However, if the count
above in dd
is replaced with 3110016
(3110017-1), then did the routines above again, with wc -c
on "btrfs send", you can get 604 (604 B), only 604 bytes, and this is what I expect.
I also made a bash script to do those two tests:
#!/bin/bash
btrfs subvolume create 1
rm -rf 1/*
dd if=/dev/urandom of=1/a bs=64 count=3110017
sudo btrfs subvolume snapshot -r 1 1_ro0
cp --reflink=always 1/a 1/b
sudo btrfs subvolume snapshot -r 1 1_ro1
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
sudo btrfs subvolume delete 1_ro0 1_ro1
rm -rf 1/*
dd if=/dev/urandom of=1/a bs=64 count=3110016
sudo btrfs subvolume snapshot -r 1 1_ro0
cp --reflink=always 1/a 1/b
sudo btrfs subvolume snapshot -r 1 1_ro1
sudo btrfs send -p 1_ro0 1_ro1 | wc -c
sudo btrfs subvolume delete 1_ro0 1_ro1
rm -r 1
If replace wc -c
with btrfs receive --dump
to inspect the "btrfs send" output data, with dd 3110016
value it will output a few "clone" entries like this:
clone ./1_ro1/b offset=0 len=134217728 from=./1_ro1/a clone_offset=0
clone ./1_ro1/b offset=134217728 len=64823296 from=./1_ro1/a clone_offset=134217728
...
But with dd value 3110017, it outputs:
write ./1_ro1/b offset=197918720 len=49152
write ./1_ro1/b offset=197967872 len=49152
write ./1_ro1/b offset=198017024 len=49152
write ./1_ro1/b offset=198066176 len=49152
write ./1_ro1/b offset=198115328 len=49152
write ./1_ro1/b offset=198164480 len=49152
... (so many more)
which contains a bunch of "write" entries.
I also searched and noticed this site: https://www.spinics.net/lists/linux-btrfs/msg105951.html , and it's like talking about file holes, but in my case it obviously can't be.
This is so weird and I don't know what I'm missing.
My btrfs-progs
version is 5.18.1-1
and the mount options from /proc/mount
: rw,relatime,space_cache=v2,subvolid=5,subvol=/
Thanks!
bczhc
(21 rep)
Jun 18, 2022, 03:58 PM
• Last activity: Oct 10, 2024, 10:38 PM
0
votes
0
answers
118
views
what the `cp --reflink=always` will do if I try to copy directory that contains reflink to a diffrent disk?
I have a folder root1 on disk partition sda1 which contain reflink`/mnt/sda1/root1/a->/mnt/sda1/root1/raw/a` I have another folder root1 on disk partition sdb1, that has reflink `/mnt/sdb1/root1/a->/mnt/sdb1/backup/raw/a`. sda1 and sdb1 has same file system. It can either both be XFS or both be btrf...
I have a folder root1 on disk partition sda1 which contain reflink
/mnt/sda1/root1/a->/mnt/sda1/root1/raw/a
I have another folder root1 on disk partition sdb1, that has reflink /mnt/sdb1/root1/a->/mnt/sdb1/backup/raw/a
.
sda1 and sdb1 has same file system. It can either both be XFS or both be btrfs
If I do copy -r --reflink=always /mnt/sda1/root1 /mnt/sdb1/root1
.What will happen to my data?
- case 1: the /mnt/sda1/root1/a and /mnt/sdb1/root1/a are exactly the same.
- case 2: the /mnt/sda1/root1/a and /mnt/sdb1/root1/a are different in some blocks. but /mnt/sda1/root1/raw/a and /mnt/sdb1//backup/raw/a are the same.
- case 3: all 4 a are different but most of the blocks are the same.
Is the copy smart enough to maintain the block sharing on the /sdb1?
Wang
(1395 rep)
Jul 5, 2024, 07:37 PM
21
votes
1
answers
8453
views
In Linux, which filesystems support reflinks?
`btrfs` supports reflinks, `XFS` supports reflinks [(since 2017 I think?)](https://blogs.oracle.com/linux/xfs-data-block-sharing-reflink#:~:text=Three%20years%20ago%2C%20I%20introduced,efficient%20use%20of%20storage%20hardware.). Are there any other filesystems that support it? ``` truncate -s 1G te...
btrfs
supports reflinks, XFS
supports reflinks [(since 2017 I think?)](https://blogs.oracle.com/linux/xfs-data-block-sharing-reflink#:~:text=Three%20years%20ago%2C%20I%20introduced,efficient%20use%20of%20storage%20hardware.) .
Are there any other filesystems that support it?
truncate -s 1G test.file;
cp --reflink=always test.file ref.test.file;
hanshenrik
(695 rep)
Jan 27, 2021, 10:21 AM
• Last activity: Jan 17, 2024, 04:47 PM
5
votes
2
answers
1826
views
How to find out if the filesystem supports reflink copies?
I try to create a snapshot copy of a directory recursively, but it returns errors for every single file if the filesystem (custom shfs) does not support `reflink` copies: # cp -a --reflink=always /mnt/user/libvirt /mnt/user/libvirt_copy cp: failed to clone '/mnt/user/libvirt_copy/libvirt/libvirt.img...
I try to create a snapshot copy of a directory recursively, but it returns errors for every single file if the filesystem (custom shfs) does not support
reflink
copies:
# cp -a --reflink=always /mnt/user/libvirt /mnt/user/libvirt_copy
cp: failed to clone '/mnt/user/libvirt_copy/libvirt/libvirt.img' from '/mnt/user/libvirt/libvirt.img': Operation not supported
cp: failed to clone '/mnt/user/libvirt_copy/libvirt/test.txt' from '/mnt/user/libvirt/test.txt': Operation not supported
...
As it seems to be impossible to stop cp
on the first error (?), which means useless looping through probably thousands of files, I instead like to pre-check if the filesystem supports reflinks.
One idea is to search for a single file and do a testcopy:
if ! find /mnt/user/libvirt -type f -print -quit | xargs -I {} sh -c 'dst_path=$(dirname $(echo {} | sed "s#^/mnt/user/libvirt#/mnt/user/libvirt_copy#")); mkdir -vp $dst_path && cp -a --reflink=always {} "$dst_path"'; then
echo "Error: Filesystem does not support reflink"
exit
fi
But this leaves the dir and an empty file which needs to be cleaned up afterwards:
# find /mnt/user/libvirt_copy/
/mnt/user/libvirt_copy/
/mnt/user/libvirt_copy/libvirt.img
# ls -lah /mnt/user/libvirt_copy/libvirt.img
-rw------- 1 root root 0 Nov 1 08:58 /mnt/user/libvirt_copy/libvirt.img
Isn't there an better way? Or maybe there is a solution to stop cp
on the first error?
mgutt
(547 rep)
Nov 1, 2022, 07:56 AM
• Last activity: Jul 8, 2023, 03:47 AM
22
votes
1
answers
16826
views
Why does "cp -R --reflink=always" perform a standard copy on a Btrfs filesystem?
Btrfs support Copy-On-Write. I tried to use that feature to clone a directory: cp -R --reflink=always foo_directory foo_directory.mirror I expected the command to finish almost instantly (like a `btrfs subvolume snapshot`), but the `cp` command seems to perform a slow, standard copy. According to th...
Btrfs support Copy-On-Write. I tried to use that feature to clone a directory:
cp -R --reflink=always foo_directory foo_directory.mirror
I expected the command to finish almost instantly (like a
btrfs subvolume snapshot
), but the cp
command seems to perform a slow, standard copy.
According to the man page, I would expected --reflink=always
to enforce Copy-On-Write:
> When --reflink[=always]
is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if --reflink=auto
is specified, fall back to a standard copy.
**Questions:**
- Do you know why --reflink=always
doesn't work?
- What options (or other commands) should I use instead?
Philipp Claßen
(4967 rep)
Jul 30, 2015, 03:30 PM
• Last activity: Jun 23, 2023, 08:29 AM
4
votes
1
answers
1594
views
GNU cp: What's the difference between `--link` and `--reflink=always`?
What's the difference between `--link` and `--reflink=always`? I use the following command as an `mv` substitute, and I am wondering if using `--reflink` is a better choice. ``` command gcp -r --link --archive --verbose "${opts[@]}" "$@" # delete the sources manually ```
What's the difference between
--link
and --reflink=always
?
I use the following command as an mv
substitute, and I am wondering if using --reflink
is a better choice.
command gcp -r --link --archive --verbose "${opts[@]}" "$@"
# delete the sources manually
HappyFace
(1694 rep)
Jun 16, 2021, 08:57 AM
• Last activity: Dec 4, 2021, 09:45 PM
8
votes
1
answers
1830
views
Detect btrfs reflinked files
It's clear, with `cp --reflink foo bar` I can copy a file on btrfs without wasting space. But how can I detect that the two files (foo and bar here) are using the same "space unit"? If they were hardlinks, I would use `[ foo -ef bar ]` for comparing or `stat -c%i foo` to get a unique id (inode in th...
It's clear, with
cp --reflink foo bar
I can copy a file on btrfs without wasting space. But how can I detect that the two files (foo and bar here) are using the same "space unit"?
If they were hardlinks, I would use [ foo -ef bar ]
for comparing or stat -c%i foo
to get a unique id (inode in this case).
Are there similar commands for btrfs reflinked files?
I want to apply this on a 300G volume which is virtually 6T in size due to deduplication, so I want to avoid checksumming...
Daniel Alder
(936 rep)
Oct 14, 2018, 01:26 PM
• Last activity: Aug 24, 2021, 09:27 AM
11
votes
1
answers
8033
views
Is there a way to enable reflink on an existing XFS filesystem?
I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the `reflink` feature of XFS, but I did not do so when I first created the filesystem (I used the defaults). Is there a way that I can toggle the `reflink` feature on,...
I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the
reflink
feature of XFS, but I did not do so when I first created the filesystem (I used the defaults).
Is there a way that I can toggle the reflink
feature on, without having to backup and re-create the filesystem? And if not, can I expect that feature to be implemented in the future?
TheSola10
(231 rep)
Sep 14, 2019, 04:35 PM
• Last activity: Aug 28, 2020, 01:58 PM
3
votes
0
answers
329
views
Combine mkfifo and cp --reflink to append files without copying data
I have huge files `{0..9}.bin` which I want concatenate into `out.bin`. I don't need the original files afterwards. So I was wondering, if this is possible by only modifying the filesystem index without copying the file contents (see https://unix.stackexchange.com/q/80429/234981 for efficient copy s...
I have huge files
{0..9}.bin
which I want concatenate into out.bin
. I don't need the original files afterwards. So I was wondering, if this is possible by only modifying the filesystem index without copying the file contents (see https://unix.stackexchange.com/q/80429/234981 for efficient copy solutions).
On modern file systems (e.g. btrfs) cp --reflink=always
exists. Fifos are on the file system level (at least btrfs send
also tracks fifos), so they should have information about the actual data blocks used. Therefore, cp --reflink=always
should be able to determine the extend numbers on disk and re-use them.
So I was wondering, if it is possible to use mkfifo
in combination with cp --reflink=always
?
**Update**
Currently, it is not working:
for i in {1..9}; do dd if=/dev/urandom of="in$i.bin" bs=5M count=200; done;
mkfifo fifo
cat in* >fifo &
cp --reflink=always fifo out.bin
results in
cp: failed to clone 'out.bin' from 'fifo': Invalid argument
Probably, it never will, since FIFOs have no information about the storage origin bug instead are just dumb pipes.
darkdragon
(175 rep)
May 20, 2020, 08:46 AM
• Last activity: May 20, 2020, 04:19 PM
0
votes
1
answers
292
views
Reflink copy diff
I want to make a behaviour like `btrfs send` and `btrfs receive`, but on a per-file basis. As far as I see it, I can use `cp -ax --reflink=always` to make a "snapshot" of the file. Let's assume that two such "snapshots" on two different hard disks are exactly equal. How do I `cp` the diff (and only...
I want to make a behaviour like
btrfs send
and btrfs receive
, but on a per-file basis.
As far as I see it, I can use cp -ax --reflink=always
to make a "snapshot" of the file.
Let's assume that two such "snapshots" on two different hard disks are exactly equal.
How do I cp
the diff (and only the diff) between two reflinked files like btrfs send -p
would do it?
Ideally, I would like to have a solution that works on any filesystem that is capable of making reflinks and not only on btrfs. Also ideally, I would like to run it without sudo.
I know that I can look at the file extents with filefrag
, but filefrag
does not guarantee that the results are exact. I would rather copy some megabytes more than taking the risk of a corrupted file.
user1994405
(1 rep)
Apr 29, 2020, 02:58 PM
• Last activity: Apr 29, 2020, 06:41 PM
8
votes
1
answers
1921
views
What does a rmlint's "clone" for btrfs do?
I was reading the [`rmlint`](http://rmlint.readthedocs.io/en/latest/index.html#) manual, and one of the duplicate handlers are clone and reflink: >· clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will physically delete duplicate extents. Needs at...
I was reading the [
rmlint
](http://rmlint.readthedocs.io/en/latest/index.html#) manual, and one of the duplicate handlers are clone and reflink:
>· clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will physically delete duplicate extents. Needs at least kernel 4.2.
>
>· reflink: Try to reflink the duplicate file to the original. See also --reflink in man 1 cp. Fails if the filesystem does not support it.
What exactly does this clone do, and how is it different from a reflink? What does the BTRFS_IOC_FILE_EXTENT_SAME ioctl do?
Dan
(9622 rep)
Feb 13, 2018, 09:10 PM
• Last activity: Mar 5, 2018, 12:49 PM
Showing page 1 of 11 total questions