Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
1
votes
1
answers
75
views
How do I build a statically linked copy of xz-utils?
I'm trying to build a statically linked copy of xz-utils, such that the `xz` do not have dependencies on libc. I tried the usual way, which is to set LDFLAGS=-static, and works for most other packages out there. In the case of xz-utils, however, the binary is built successfully but it is statically...
I'm trying to build a statically linked copy of xz-utils, such that the
xz do not have dependencies on libc.
I tried the usual way, which is to set LDFLAGS=-static, and works for most other packages out there. In the case of xz-utils, however, the binary is built successfully but it is statically linked, as can be seen here.
$ docker run --rm -it -v $PWD/downloads:/downloads --platform linux/amd64 alpine:3
/ # cd ~
~ # apk add build-base clang
...
~ # tar -xf /downloads/xz-5.8.1.tar.gz
~ # PREFIX=/opt/xz-5.8.1
~ # cd xz-5.8.1/
~ # export CC=clang
~ # export LDFLAGS=-static
~/xz-5.8.1 # LDFLAGS=-static CC=clang ./configure --prefix=$PREFIX --disable-nls --disable-doc --disable-microlzma
[See https://pastebin.com/Cj6ZX9sC]
~/xz-5.8.1 # make -j4
[See https://pastebin.com/79wmN4zS]
~/xz-5.8.1 # make install
[See https://pastebin.com/iCpCG3hQ]
~/xz-5.8.1 # ldd /opt/xz-5.8.1/bin/xz
/lib/ld-musl-x86_64.so.1 (0x7fe94c28b000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fe94c28b000)
rs3
(11 rep)
Apr 6, 2025, 02:32 PM
• Last activity: Apr 6, 2025, 04:18 PM
5
votes
3
answers
6813
views
How to check what compression type an RPM package uses?
I recently realized that the RPM packages shifted from gzip-compression to xz-compression a few years ago. I need to check is the compression type of an RPM package that I have. I also need to check what compression type is considered by my system when it is trying to unpack an RPM file.
I recently realized that the RPM packages shifted from gzip-compression to xz-compression a few years ago. I need to check is the compression type of an RPM package that I have. I also need to check what compression type is considered by my system when it is trying to unpack an RPM file.
Alchemist
(591 rep)
Aug 31, 2016, 07:07 PM
• Last activity: Jan 31, 2025, 05:54 PM
247
votes
4
answers
99374
views
Why are tar archive formats switching to xz compression to replace bzip2 and what about gzip?
More and more [`tar`][1] archives use the [`xz`][2] format based on LZMA2 for compression instead of the traditional [`bzip2(bz2)`][3] compression. In fact *kernel.org* made a late "*Good-bye bzip2*" [announcement, 27th Dec. 2013][4], indicating kernel sources would from this point on be released in...
More and more
tar archives use the xz format based on LZMA2 for compression instead of the traditional bzip2(bz2) compression. In fact *kernel.org* made a late "*Good-bye bzip2*" announcement, 27th Dec. 2013 , indicating kernel sources would from this point on be released in both tar.gz and tar.xz format - and on the main page of the website what's directly offered is in tar.xz.
Are there any specific reasons explaining why this is happening and what is the relevance of gzip in this context?
user44370
Jan 6, 2014, 06:39 PM
• Last activity: Jan 21, 2025, 02:31 PM
0
votes
2
answers
133
views
xz print crc checksum
I have two files, of several gigabytes in size, which have been compressed with `xz`. I suspect the original files before compression are identical, but they've been compressed with different `xz` options so the compressed files are different. How can I verify that the files are identical without un...
I have two files, of several gigabytes in size, which have been compressed with
xz. I suspect the original files before compression are identical, but they've been compressed with different xz options so the compressed files are different.
How can I verify that the files are identical without uncompressing them? As they're so large I would rather not uncompress them if I don't have to.
I know that xz stores a CRC checksum, so I thought it should be easy enough: just print and compare the CRC checksums of each file. However the xz -l (and xz -l -v) command just lists the checksum as type "CRC64" instead of what the actual checksum is.
k314159
(481 rep)
Nov 14, 2024, 12:07 PM
• Last activity: Nov 14, 2024, 12:25 PM
3
votes
1
answers
549
views
How to open a .tar.xz indexed archivefile with Midnight Commander?
`tar -I pixz -cf foo.tar.xz ./foo` compress the Stuff. `tar -I pixz -xf foo.tar.xz` decompress the Stuff. And with `pixz -l foo.tar.xz` comes a list the contents. How can i do this with (`mc`) Midnight Commander? If i select `foo.tar.xz` in `mc` and press Enter, it comes nothing. The CPU usage goes...
tar -I pixz -cf foo.tar.xz ./foo compress the Stuff.
tar -I pixz -xf foo.tar.xz decompress the Stuff.
And with pixz -l foo.tar.xz comes a list the contents.
How can i do this with (mc) Midnight Commander?
If i select foo.tar.xz in mc and press Enter, it comes nothing.
The CPU usage goes for a moment high but nothing comes than, no error.
Normaly mc can open archive files like this?
How can mc open and browse an indexed .tar.xz file?
user447274
(539 rep)
Oct 22, 2024, 04:11 AM
• Last activity: Oct 22, 2024, 05:51 AM
0
votes
1
answers
65
views
Is it possible to compress a tar ball with gzip/bzip2/xz after tar ball file has been created?
If we create a tar ball file by giving the following command tar -cvf Docs.tar $HOME/Documents/* then post creation of the tar ball is it possible to use gzip or bzip2 or xz or some other compression utility to compress the tar file? I know that we can give the option `--bzip2` or `--xz` or `--gzip`...
If we create a tar ball file by giving the following command
tar -cvf Docs.tar $HOME/Documents/*
then post creation of the tar ball is it possible to use gzip or bzip2 or xz or some other compression utility to compress the tar file?
I know that we can give the option
--bzip2 or --xz or --gzip while creating the tar along with -cvf option but what if that is not done. And after the tar is created then the compression is sought to be applied. Is it possible? If yes then how?
KDM
(116 rep)
Oct 2, 2024, 01:50 PM
• Last activity: Oct 2, 2024, 02:15 PM
0
votes
0
answers
58
views
browse, read, write and rename files in archives, not mount - mc & dolphin
the computer runs a linux kernel and kde6. i use the dolphin filemanager. mark files, add them to an archive is a very usefull option. dolphin can go into archivefiles, but it would be very usefull for me to rename files in the archives, add and or delete files in archives. How can i do that? do i n...
the computer runs a linux kernel and kde6. i use the dolphin filemanager.
mark files, add them to an archive is a very usefull option.
dolphin can go into archivefiles, but it would be very usefull for me to rename files in the archives, add and or delete files in archives. How can i do that? do i need any plugin or something else? is there any other filemanager for they can do this? is this what i am looking for with midnight commander possible? i know that mc can go into archivefiles and iso files, but rename, delete and add can i not make in mc.
i have found some soloutions like avfs, fuse-mount and archivemount. but i have very much archive files, thats not a good way for me to mount them all. i will browse into the files they i need. i will not mount an archive as an folder. i have many .tar and .zip and some .rar archivefiles, but i can all of them convert to any other archiveformat is the required.
user447274
(539 rep)
Sep 29, 2024, 07:17 PM
-3
votes
1
answers
624
views
tar: Exiting with failure status due to previous errors
I running ``` $ cat /etc/*-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/...
I running
$ cat /etc/*-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/ "
SUPPORT_URL="https://www.debian.org/support "
BUG_REPORT_URL="https://bugs.debian.org/ "$ wget -c https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_3.19.5-stable.tar.xz
--2024-04-17 11:21:57-- https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_3.19.5-stable.tar.xz
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.167.91, 142.250.71.91, 142.250.66.251, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.167.91|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 665630904 (635M) [application/x-tar]
Saving to: ‘flutter_linux_3.19.5-stable.tar.xz’
flutter_linux_3.19.5-stable.tar.xz 100%[===================================================================>] 634.79M 6.06MB/s in 1m 55s
2024-04-17 11:23:53 (5.51 MB/s) - ‘flutter_linux_3.19.5-stable.tar.xz’ saved [665630904/665630904]
$ tar xf flutter_linux_3.19.5-stable.tar.xz /usr/bin/
tar: /usr/bin: Not found in archive
tar: Exiting with failure status due to previous errors
$ sudo tar xf flutter_linux_3.19.5-stable.tar.xz /usr/bin/
tar: /usr/bin: Not found in archive
tar: Exiting with failure status due to previous errors
$ sudo tar -xf flutter_linux_3.19.5-stable.tar.xz /usr/bin/
tar: /usr/bin: Not found in archive
tar: Exiting with failure status due to previous errors
user3523406
(315 rep)
Apr 17, 2024, 02:03 AM
• Last activity: Apr 17, 2024, 12:02 PM
1
votes
2
answers
700
views
Why is 7-zip much faster than other LZMA implementations in Linux?
Until now, I used to backup my data using `tar` with one of the LZMA compression options (`--lzma`, `--xz` or `--lzip`) I recently noticed that 7-Zip has been ported to Linux in 2021 (https://www.xda-developers.com/7-zip-linux-official-release/). I'm not talking about the older P7Zip (https://p7zip....
Until now, I used to backup my data using
tar with one of the LZMA compression options (--lzma, --xz or --lzip)
I recently noticed that 7-Zip has been ported to Linux in 2021 (https://www.xda-developers.com/7-zip-linux-official-release/) . I'm not talking about the older P7Zip (https://p7zip.sourceforge.net/) , that doesn't seem to be maintained anymore, but about the official 7-Zip.
So, I tested it, and was very surprised to discover that it's A LOT faster than all the others Linux LZMA implementations, for the same compression ratio.
Below my tests (Debian 11). Please not that I emptied the RAM cache between every test (sync && echo 3 > /proc/sys/vm/drop_caches)
I am working on a 163M folder, containing several type of files, PDF, text, open office, and so on...
$ du -hs TEST/
163M TEST/$ time tar c -hp TEST/ | 7zz a -si test.tar.7z
real 0m14,565s
(...)
$ ll test.tar.7z
(...) 127M (...) test.tar.7z$ time tar -chp --lzma -f test.tar.lzma TEST/
real 1m13,159s
$ time tar -chp --xz -f test.tar.xz TEST/
real 1m12,889s
$ time tar -chp --lzip -f test.tar.lz TEST/
real 1m12,525s
$ ll test.tar.{7z,lz*,xz}
(...) 127M (...) test.tar.7z
(...) 127M (...) test.tar.lz
(...) 127M (...) test.tar.lzma
(...) 127M (...) test.tar.xztar, I did the same tests but piped tar's output to lzma|xz|lzip, instead of using the --lzma, --xz and --lzip switches. Same results.
So, basically, 7-Zip's Linux version makes all other LZMA implementations look rather bleak. I think 7-Zip doesn't support Linux owners and permissions, but that's irrelevant when compressing a .tar file.
So, does anybody know why 7-Zip's Linux version is so much faster than other LZMA implementations ?
```
ChennyStar
(1969 rep)
Mar 17, 2024, 10:25 AM
• Last activity: Apr 14, 2024, 08:49 AM
0
votes
1
answers
1250
views
How to XZ compress directory WITHOUT tar?
I know how to compress a directory with xz using tar, but is it possible to compress a directory using xz WITHOUT tar? If I try: ``` xz -z myDir ``` Then the following error is retured: ``` xz: myDir: Is a directory, skipping ```
I know how to compress a directory with xz using tar, but is it possible to compress a directory using xz WITHOUT tar?
If I try:
xz -z myDirxz: myDir: Is a directory, skipping
Fred
(345 rep)
May 16, 2023, 08:04 PM
• Last activity: May 16, 2023, 08:12 PM
0
votes
0
answers
775
views
CentOS 7: yum fails importing required Python module liblzma
I'm accessing a machine with CentOS Linux 7.9.2009, Python 2.7.5, and GCC 10.3.0 (the choice of python2 was not mine). I'm trying to install an rpm through the yum command, but got this error message: [cevangel@lxplus760 ~]$ yum install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-3-...
I'm accessing a machine with CentOS Linux 7.9.2009, Python 2.7.5, and GCC 10.3.0 (the choice of python2 was not mine). I'm trying to install an rpm through the yum command, but got this error message:
[cevangel@lxplus760 ~]$ yum install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-3-2.src.rpm
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
/cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_2_patch1/external/el8_amd64_gcc10/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Every time I use the yum command I got this same error, so I'm not able to install the xz module.
I would appreciate any help.
Cauê Evangelista
(1 rep)
Feb 18, 2023, 05:22 PM
• Last activity: Feb 18, 2023, 05:39 PM
9
votes
1
answers
5959
views
Is there any way to check the compression ratio of a xz compressed file?
In windows by default one can see the compression ratio of zip files and after installing WinRAR it also supports this feature but currently I am using a linux and have compressed a file using lzma2 and xz as the container but how can I check the compression ratio without comparing the original file...
In windows by default one can see the compression ratio of zip files and after installing WinRAR it also supports this feature but currently I am using a linux and have compressed a file using lzma2 and xz as the container but how can I check the compression ratio without comparing the original file size and the compressed file size? Is it even possible?
PSP
(345 rep)
Oct 29, 2016, 09:54 AM
• Last activity: Feb 3, 2023, 05:37 PM
0
votes
1
answers
1451
views
Compress Linux Kernel with xz
Just out of curiosity, is it possible to compress the Linux kernel with xz, and if so, how would I go about this? I actually tried compressing a raw kernel image with xz [a silly idea] and loading it from the grub prompt and got the following: grub> linux /vmlinuz-6.0.5.xz root=/dev/sda2 rootfstype=...
Just out of curiosity, is it possible to compress the Linux kernel with xz, and if so, how would I go about this? I actually tried compressing a raw kernel image with xz [a silly idea] and loading it from the grub prompt and got the following:
grub> linux /vmlinuz-6.0.5.xz root=/dev/sda2 rootfstype=ext4 rootwait ro Invalid magic number grub>Like I stated before, this was a hopeless idea and I had expected an error. But is there an actual method of compressing the Linux kernel with xz? Please note that I normally develop for Raspberry Pi, so some configuration options available in x86_64 may not be present there.
rando
(125 rep)
Dec 28, 2022, 01:12 PM
• Last activity: Dec 28, 2022, 05:55 PM
3
votes
1
answers
18219
views
Multithreaded xz, with gzip, pv, and pipes - is this the most efficient I can get?
I'm excited to learn that `xz` now supports multithreading: xz --threads=0 But now I want to utilise this as much as possible. For example, to recompress gzips as xz: gzip -d -k -c myfile.gz | pv | xz -z --threads=0 - > myfile.xz This results in my processor being more highly used (~260% CPU to xz,...
I'm excited to learn that
xz now supports multithreading:
xz --threads=0
But now I want to utilise this as much as possible. For example, to recompress gzips as xz:
gzip -d -k -c myfile.gz | pv | xz -z --threads=0 - > myfile.xz
This results in my processor being more highly used (~260% CPU to xz, yay!).
However:
- I realise that gzip is not (yet) multithreading,
- I think that either pv or the pipes may be restricting the number of (IO?) threads.
Is this true and, if so, is there a way to make this more efficient (other than to remove pv)?
tudor -Reinstate Monica-
(545 rep)
Oct 23, 2019, 11:25 PM
• Last activity: Sep 19, 2022, 02:17 PM
13
votes
1
answers
4397
views
Is there any way to verify the integrity of an xz archive without decompressing the entire archive?
Can I check the integrity of the archive without decompressing it as the archive is 64GB. It is a backup of my old laptop and before i format my old laptop I wanted to verify the backup but it will take too long to decompress and verify. So is there a faster way to do so?
Can I check the integrity of the archive without decompressing it as the archive is 64GB. It is a backup of my old laptop and before i format my old laptop I wanted to verify the backup but it will take too long to decompress and verify. So is there a faster way to do so?
PSP
(345 rep)
Oct 29, 2016, 08:43 AM
• Last activity: Aug 24, 2022, 10:50 AM
1
votes
0
answers
37
views
Reading from stdin `dd` copies more data than reading the same thing from file
I encountered weird behaviour of the `dd` tool: 1. I downloaded compressed "newsystem.iso.xz". 2. I ran `xz -d --stdout newsystem.iso.xz | dd of=/dev/sdb bs=1M conv=sync`. 3. The decompressed "*.iso" file weights about 4.5GB. I expect this to fit on the usb. Instead `dd` fails with "Not enough space...
I encountered weird behaviour of the
dd tool:
1. I downloaded compressed "newsystem.iso.xz".
2. I ran xz -d --stdout newsystem.iso.xz | dd of=/dev/sdb bs=1M conv=sync.
3. The decompressed "*.iso" file weights about 4.5GB. I expect this to fit on the usb. Instead dd fails with "Not enough space" error, saying that it wrote 8.1GB (the usb is about 8GB)
But then I downloaded the already decompressed "*.iso" and ran:
1. dd if=newsystem.iso of=/dev/sdb bs=1M conv=sync
2. Command succeeds after writing the expected 4.5GB.
I can verify that xz -d --stdout newsystem.iso.xz | shash512sum is the same as shash512sum newsystem.iso.
The sync option means
> sync pad every input block with NULs to ibs-size
The dd version is "GNU coreutils 9.0" and the iso image is "FreeBSD-13.0-RELEASE-amd64-dvd1.iso".
Thanks in advance!
zaabson
(33 rep)
Apr 16, 2022, 12:18 PM
• Last activity: Apr 16, 2022, 12:28 PM
1
votes
2
answers
1813
views
Tar pipe to xz results in broken pipe
I have a backup script that runs the following commands: tar -c dir1 dir2 | xz -9 -T0 | gpg -c --batch --passphrase xxx | aws s3 ... The return values are always the same: `tar` fails with `141` (`broken pipe` error) and `xz` returns `137` (no other error messages, even in verbose mode). The script...
I have a backup script that runs the following commands:
tar -c dir1 dir2 | xz -9 -T0 | gpg -c --batch --passphrase xxx | aws s3 ...
The return values are always the same:
tar fails with 141 (broken pipe error) and xz returns 137 (no other error messages, even in verbose mode).
The script is tested, runs as root and works well on other servers. Initially I thought that the data I'm backing up could be corrupt and deleted some socket files within the backup directory (which is a rsnapshot folder), but that hasn't helped as well.
Does anyone have an idea what the problem could be?
EDIT: If I remove xz from the pipe it works.
manifestor
(2563 rep)
Dec 1, 2021, 11:54 AM
• Last activity: Dec 1, 2021, 02:45 PM
3
votes
1
answers
3487
views
7z command to create a XZ file
The man page says 7z tool supports *7z, LZMA2, XZ, ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO,* most filesystem images and *DEB* formats. And it says we could pass the type as `-t{Type}`. The following command is success. ``` 7z a -tzip archive.zip dir1 dir2 file1 file2 ``` But I ca...
The man page says 7z tool supports *7z, LZMA2, XZ, ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO,* most filesystem images and *DEB* formats.
And it says we could pass the type as
-t{Type}. The following command is success.
7z a -tzip archive.zip dir1 dir2 file1 file2.xz using the following command.
7z a -txz archive.zip dir1 dir2 file1 file2$ 7z a -txz archive.xz dir1 dir2 file1 file2
7-Zip 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (806E9),ASM,AES-NI)
Scanning the drive:
2 folders, 7 files, 196192711 bytes (188 MiB)
Creating archive: archive.xz
Items to compress: 9
System ERROR:
E_INVALIDARG.xz file using 7z command?
**EDIT:**
The following command successful.
7z a -txz archive archive.tar-txz or -tbzip2 on only archive files as we can't directly use compression on files without archiving the files first?
NayabSD
(194 rep)
Sep 22, 2021, 04:46 PM
• Last activity: Sep 22, 2021, 05:08 PM
1
votes
1
answers
968
views
xz: OOM when compressing 1TB .tar
I'm trying to compress a large archive with multi-threading enabled, however, my system keeps freezing up and runs out of memory. ``` OS: Manjaro 21.1.0 Pahvo Kernel: x86_64 Linux 5.13.1-3-MANJARO Shell: bash 5.1.9 RAM: 16GB ``` ``` |swapon| NAME TYPE SIZE USED PRIO /swapfile file 32G 0B -2 ``` I've...
I'm trying to compress a large archive with multi-threading enabled, however, my system keeps freezing up and runs out of memory.
OS: Manjaro 21.1.0 Pahvo
Kernel: x86_64 Linux 5.13.1-3-MANJARO
Shell: bash 5.1.9
RAM: 16GB|swapon|
NAME TYPE SIZE USED PRIO
/swapfile file 32G 0B -2/swapfile 2x the amount of RAM I have (32GB) but the system would always freeze once >90% of total RAM has been used, and would seem to not make use of the /swapfile.
|xz --info-memory|
Total amount of physical memory (RAM) : 15910 MiB
Memory usage limit for compression: Disabled
Memory usage limit for decompression: Disabledxz so please bear with me, but is there a way to globally enable the memory usage limiter and for the Total amount of physical memory (RAM) to take into account the space made available by /swapfile?
JKleinne
(113 rep)
Jul 23, 2021, 10:20 PM
• Last activity: Jul 23, 2021, 10:55 PM
1
votes
0
answers
137
views
Tar archive overwrote original file
I was compressing a rosbag file using tar when I mistakenly used the same original file name for the the archive name and had it overwritten by the output archive , the compressed archive size is very small compared to the expected output had I used another name e.g., file.tar.xz. When I extract the...
I was compressing a rosbag file using tar when I mistakenly used the same original file name for the the archive name and had it overwritten by the output archive , the compressed archive size is very small compared to the expected output had I used another name e.g., file.tar.xz.
When I extract the archive , I get a file of type "STL 3D model (binary) (model/x.stl-binary)" instead of the original bag file of type "Binary (application/octet-stream)". The file size is the same but the format is different.
Is there a way to retrieve the original file or is this irreversible ? kindly find below the commands used to compress then extract the file.
tar -cJf
tar -xf
any help would be much appreciated , thanks in advance.
Amr
(11 rep)
May 20, 2021, 10:35 AM
• Last activity: May 20, 2021, 10:42 AM
Showing page 1 of 20 total questions