Piped dd command limits its output to 65536 bytes, why?
0
votes
1
answer
116
views
My colleague and I are both using a Docker container running Ubuntu 22.04.4 LTS (Jammy Jellyfish):
root@c2155d7af4a4:/# grep VERSION= /etc/os-release
VERSION="22.04.4 LTS (Jammy Jellyfish)"
Now, assume we have a binary file of 98299904 bytes in that Docker container:
root@c2155d7af4a4:/# dd if=/dev/urandom of=randomfile bs=1 count=98299904
98299904+0 records in
98299904+0 records out
98299904 bytes (98 MB, 94 MiB) copied, 101.014 s, 973 kB/s
root@c2155d7af4a4:/# ls -al randomfile
-rw-r--r-- 1 root root 98299904 May 29 19:17 randomfile
We can then extract a certain part of that binary file with dd
. Let's do that in 2619 blocks of size 12624 and let's skip the first block:
root@c2155d7af4a4:/# dd if=randomfile of=test1 ibs=12624 skip=1 count=2619 obs=33061608
2619+0 records in
1+1 records out
33062256 bytes (33 MB, 32 MiB) copied, 0.0745345 s, 444 MB/s
Note that although we specified obs=33061608
, we get an output file of size 2619*12624=33062256 bytes, which is larger then the 33061608 we requested. Therefore, we try to chop off the first 33061608 bytes of that first dd
output by piping its output to a second dd
command:
root@c2155d7af4a4:/# dd if=randomfile ibs=12624 skip=1 count=2619 obs=33061608 | dd bs=33061608 count=1 of=test2
0+1 records in
0+1 records out
65536 bytes (66 kB, 64 KiB) copied, 0.0526223 s, 1.2 MB/s
root@c2155d7af4a4:/# ls -al test2
-rw-r--r-- 1 root root 65536 May 29 19:23 test2
Unfortunately, what do we see? The resulting output file is only 65536 bytes large instead of our expected 33061608.
Now, the strange thing is that this on *my* system I get this faulty 65536 bytes output, but on my *colleague* his system (which uses exactly the same Docker container with the same version of dd
), he gets the expected 33061608 bytes output.
Also, when I do it in two steps with an intermediate file and not using the pipe operator, then things seem to work:
root@c2155d7af4a4:/# dd if=randomfile ibs=12624 skip=1 count=2619 obs=33061608 of=test1
2619+0 records in
1+1 records out
33062256 bytes (33 MB, 32 MiB) copied, 0.0321817 s, 1.0 GB/s
root@c2155d7af4a4:/# ls -al test1
-rw-r--r-- 1 root root 33062256 May 29 19:39 test1
root@c2155d7af4a4:/# dd if=test1 of=test2 bs=33061608 count=1
1+0 records in
1+0 records out
33061608 bytes (33 MB, 32 MiB) copied, 0.0264926 s, 1.2 GB/s
root@c2155d7af4a4:/# ls -al test2
-rw-r--r-- 1 root root 33061608 May 29 19:39 test2
Also, replacing the second dd
command with head -c
seems to work in combination with the pipe operator:
root@c2155d7af4a4:/# dd if=randomfile ibs=12624 skip=1 count=2619 obs=33061608 | head -c 33061608 > test2
2619+0 records in
1+1 records out
33062256 bytes (33 MB, 32 MiB) copied, 0.0691351 s, 478 MB/s
root@c2155d7af4a4:/# ls -al test2
-rw-r--r-- 1 root root 33061608 May 29 19:43 test2
We're currently in the dark what is causing this behavior. The 65536 size of the faulty output from the second piped dd
command is suspicious (65536=2^16), but we don't understand what's the root cause of this. It is a limitation from dd
or is it a limitation with the pipe operator that we use to pipe the output of the first dd
command to the second dd
? How can I make my second piped dd
command work?
The version of dd
that our Docker container has is:
root@c2155d7af4a4:/# dd --version | head -1
dd (coreutils) 8.32
The version of bash
that our Docker container has is:
root@c2155d7af4a4:/# /bin/bash --version | head -1
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
Asked by Bart Vandewoestyne
(101 rep)
May 29, 2024, 08:32 PM
Last activity: May 29, 2024, 08:37 PM
Last activity: May 29, 2024, 08:37 PM