CentOS 9. 3 vCPU VM with 4Gb RAM. I run a cron job with 7z compressing 35Gb data in 150 files 7za a -mx=9 -mmt=3 ... RAM usage - 18%, disk queue is very small, CPU is 61% on average. Why not 100% ? How do I find the bottleneck?

sar -p -d 1 10
Linux 5.14.0-80.el9.x86_64 (logger)       30/04/22        _x86_64_        (3 CPU)

16:50:10          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:11         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:11          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:12         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:12          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:13        39.00  33832.00      0.00      0.00    867.49      0.04      0.95      1.90 sda

16:50:13          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:14         2.00      0.00     24.00      0.00     12.00      0.00      0.50      0.10 sda

16:50:14          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:15         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:15          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:16         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:16          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:17         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:17          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:18         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

16:50:18          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:19         2.00      0.00     12.00      0.00      6.00      0.00      0.50      0.20 sda

16:50:19          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
16:50:20         0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00 sda

Average:          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
Average:         4.30   3383.20      3.60      0.00    787.63      0.00      0.91      0.22 sda

sar -p -u 1 10
Linux 5.14.0-80.el9.x86_64 (logger)       30/04/22        _x86_64_        (3 CPU)

16:50:26        CPU     %user     %nice   %system   %iowait    %steal     %idle
16:50:27        all     60.20      0.00      0.99      0.00      0.00     38.82
16:50:28        all     61.54      0.00      0.67      0.00      0.00     37.79
16:50:29        all     60.87      0.00      0.33      0.00      0.00     38.80
16:50:30        all     59.26      0.00      1.01      0.00      0.00     39.73
16:50:31        all     60.20      0.00      1.00      0.00      0.00     38.80
16:50:32        all     62.79      0.00      0.00      0.00      0.00     37.21
16:50:33        all     63.46      0.00      1.00      0.00      0.00     35.55
16:50:34        all     64.88      0.00      0.67      0.00      0.00     34.45
16:50:35        all     63.04      0.00      0.66      0.00      0.00     36.30
16:50:36        all     62.88      0.00      0.33      0.00      0.00     36.79
Average:        all     61.91      0.00      0.67      0.00      0.00     37.42

**EDIT** I found this doc: https://documentation.help/7-Zip/method.htm and it says "LZMA compression uses only 2 threads." which would explain what I observe on CentOS. But on Windows it uses 24 threads with LZMA.. Why?

I run Linux Live CD and I need to extract a specific file from a wim-archive that is located on a disk drive. I know a full path to the file in the archive: xubuntu@xubuntu:~$ 7z l winRE.wim | grep -i bootrec.exe 2009-08-28 15:02:29 ....A 299008 134388 Windows/System32/BootRec.exe I am short on disk space and do not have a possibility to unpack the whole archive. How could I extract that specific file from the archive? I tried the -i option, but that did not work: xubuntu@xubuntu:~$ 7z x -i Windows/System32/BootRec.exe winRE.wim Error: Incorrect command line

It's possible to restore original file timestamps during extraction on linux? I'm tested .zip extraction using 7z, unzip, unar, but only modification date is restored. Environment: - Linux Mint 22 - Filesystem: ext4 - tested.zip: it's really keeps all original timestamps creation/modification/access (it's visible e.g. in PeaZip), archive created at Win via 7z.

I have an embedded Linux device (not a Raspberry Pi) which has two partitions on an eMMC drive. I have used a vendor-provided tool to copy these two partitions onto a USB drive whose size roughly equals the size of the eMMC drive. What I did next was mount the USB drive onto a Linux computer and use tools such as e2fsck and resize2fs -M to resize the second filesystem to a minimum viable size, and finally used gdisk to resize the partition itself to match. In all calculations I've been careful to ensure that the partition does not get smaller than the filesystem, even to a point where I've made the partition slightly larger, and then used resize2fs to enlarge the filesystem to match it. After these operations I have a 16 GB USB drive with a 118 MiB boot partition and a 3.1 GiB root filesystem partition. I then used dd to read data directly from the USB drive, utilizing bs= and count= parameters to limit the amount of data read to only extend up to where the second partition ends plus an extra sector to accommodate the "zero sector". The result is that I now have an image file in which the two partitions have exactly the size of their corresponding sources but when I use the "File" -> "Properties" option of the 7-Zip tool it claims that the "Physical size" (circled in red) is larger than these partitions, and larger than what I instructed dd to copy from the source. Where does this "Physical size" value come from? Is it part of the partition table? Or does dd read it from the disk somehow? Is there any way for me to modify it so as to get rid of the "Unexpected end of data" error?

**Questions:** * Which AES algorithm mode is used in 7z? (Is it ECB, CBC, CFB, OFB, or CTR?) * When executing the command 7z a ArchiveName SomeDir -p, how does 7z perform encryption? Does it treat all data as a single solid file, or does it encrypt files individually, or maybe it creates blocks of a specific size for encryption? * If 7z uses data-blocks, how are these blocks created and what is their size? * How does 7z validate a password during decryption? Like, when it displays ERROR: Wrong password?, how does it deduce that the password is incorrect? Does it add certain headers before encryption or something? * Does 7z arrange files in alphabetical order for encryption/archiving when using the command 7z a ArchiveName SomeDir -p? **Motivation:** I managed to recover a deleted 7z archive (see this question for more details). The archive was password-protected and contained 17 sub-archives, all in 7z format. I managed to decrypt and extract 14 of them, but three of them remain inaccessible (_one of these three "B.7z" is 90% decrypted/decompressed but lacks the final block, including file names and sizes_). I guess the data-blocks (?) for these files may be corrupted. Here are some additional details about the archive:

7-Zip  16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz (806EA),ASM,AES-NI)

Scanning the drive for archives:
1 file, 125245022698 bytes (117 GiB)

Listing archive: Archive.7z

--
Path = Archive.7z
Type = 7z
Physical Size = 125245022698
Headers Size = 890
Method = LZMA2:24 7zAES
Solid = -
Blocks = 17

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2024-07-31 00:58:29 D....            0            0  DirName
2023-12-19 05:09:59 ....A    709350805    709388448  DirName/A.7z
2024-07-31 00:01:33 ....A   1586475200   1586559392  DirName/B.7z
2024-02-16 05:23:20 ....A   5813424658   5813733152  DirName/C.7z
2023-11-14 21:52:41 ....A  25460362975  25461713984  DirName/D.7z
2023-11-14 22:19:30 ....A   1713616561   1713707504  DirName/E.7z
2023-12-19 03:45:36 ....A  10539749018  10540308304  DirName/F.7z
2023-12-27 05:36:51 ....A   3637788779   3637981824  DirName/G.7z
2024-02-27 04:20:54 ....A   2566526729   2566662928  DirName/H.7z
2024-03-21 04:59:37 ....A   1180988362   1181051040  DirName/I.7z
2024-03-19 06:42:05 ....A   3751427829   3751626896  DirName/J.7z
2023-08-28 13:37:06 ....A  22618868536  22620068768  DirName/K.7z
2024-01-28 16:44:37 ....A   2768329573   2768476480  DirName/L.7z
2023-09-26 03:42:19 ....A  12245436464  12246086256  DirName/M.7z
2023-09-07 07:34:45 ....A   8162568245   8163001392  DirName/N.7z
2024-02-24 04:26:55 ....A   3300729037   3300904192  DirName/O.7z
2024-03-27 03:32:53 ....A   4387445171   4387677984  DirName/P.7z
2023-07-02 06:10:47 ....A  14795288170  14796073264  DirName/Q.7z
------------------- ----- ------------ ------------  ------------------------
2024-07-31 00:58:29       125238376112 125245021808  17 files, 1 folders

###########################
# ERRORS WHEN EXTRACTING:
###########################

ERROR: Data Error in encrypted file. Wrong password? : DirName/B.7z
ERROR: Data Error in encrypted file. Wrong password? : DirName/C.7z
ERROR: Data Error in encrypted file. Wrong password? : DirName/D.7z

Considering that each of these sub-7z files has a constant signature (Magic number + some flags: 37 7a bc af 27 1c 00 04), and knowing the password allows us to calculate its SHA-256 value for AES, is it possible to implement a reverse approach to recover the damaged parts? Also, if I am certain that the corruption is within the first 8 GB of the file, can I decrypt/decompress some segments of DirName/D.7z? I would greatly appreciate any guidance on recovering the damaged sections. I apologize for the lengthy and detailed question, and I would also appreciate any references for further reading.

**Context** I want to (try) unpack an MSI, Zip (or any archive), or EXE to inspect the contents. I also want to recursively try unpack all extracted files. Using 7zip, I have found I can unpack MSI, Zip, EXE and it just fails if it can't treat the file like an archive. This is good enough for me. However, running 7z over every file is too slow. I want to quickly skip files which don't look like they're compressed. I have read this: https://unix.stackexchange.com/questions/63918/determine-whether-a-particular-file-is-compressed * Note: in contrast to comment here I actually would like to unpack anything that **could** be an archive (like an ODF document or EPUB) This is what I have so far. It is correct, but inefficient (my script `COMPRESSED=/home/wineuser/bin/compressed` always returns true):

#!/bin/bash
set -euo pipefail
COMPRESSED=/home/wineuser/bin/compressed

# Initial archive and output directory
initial_archive=$1
output_dir=$2

# Function to extract an archive
extract_archive() {
    local archive=$1
    local output_dir=$2

    # Extract the archive to the output directory
    7z x "$archive" -o"$output_dir" -y || return
    
    # Find all files in the output directory
    idx=0
    find "$output_dir" -type f -exec "$COMPRESSED" {} \; -print | while read line; do
        idx=$((idx+1))
        extract_archive "$line" "$output_dir"/"$idx" || true
    done
}

# Extract the initial archive recursively
extract_archive "$initial_archive" "$output_dir"

**Requirements** I want this to work in general, so thinking something like:

if (file "$f" | grep -q compressed ) ; then
     # try uncompress with 7zip
fi

BUT, `file doesn't have the word "compressed" anywhere for MSI. Neither does file -k`. Example output for MSI:

Composite Document File V2 Document, Little Endian, Os: Windows, Version 6.2, MSI Installer, Code page: 1252, Title: Installation Database, Subject: DB Browser for SQLite, Author: DB Browser for SQLite Team, Keywords: Installer, Comments: This installer database contains the logic and data required to install DB Browser for SQLite., Template: x64;1033, Revision Number: {E6668432-CAB0-416D-9422-E24C9E71DB68}, Create Time/Date: Sun May  2 16:30:00 2021, Last Saved Time/Date: Sun May  2 16:30:00 2021, Number of Pages: 405, Number of Words: 2, Name of Creating Application: Windows Installer XML Toolset (3.11.1.2318), Security: 2

Example output for self-extracting exe:

PE32 executable (GUI) Intel 80386, for MS Windows, Nullsoft Installer self-extracting archive

I could maintain a word list, but feel this is rather brittle. At this stage, I would be happy with a simple entropy check on first N bytes - ie, anything high entropy is "compressed". Wondering if there may be a more straight-forward solution. btw, I'm aware of this https://superuser.com/questions/307678/how-do-i-extract-files-from-an-msi-package - but I need to do this on linux not windows, and file names mean nothing for my use case so 7zip works as good as msiexec for my use case (and works for other archives too).

With the commands md5sum, sha1sum, sha256sum I can take a text file having an hash and a path per line and verify the entire list of files in a single command, like sha1sum -c mydir.txt. (Said text file is easy to produce with a loop in find or other.) Is there a way to do the same with a list of CRC/CRC32 hashes? Such hashes are often stored inside zip-like archives, like ZIP itself or 7z. For instance: $ unzip -v archive.zip Archive: archive.zip Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 8617812 Stored 8617812 0% 12-03-2015 15:20 13fda20b 0001.tif Or: $ 7z l -slt archive.7z Path = filename Size = 8548096 Packed Size = Modified = 2015-12-03 14:20:20 Attributes = A_ -rw-r--r-- CRC = B2F761E3 Encrypted = - Method = LZMA2:24 Block = 0

At best I would like to have a call like this: $searchtool /path/to/search/ -contained-file-name "*vacation*jpg" ... so that this tool + does a recursive scan of the given path + takes all files with supported archive formats which should at least be the "most common" like zip, rar, 7z, tar.bz, tar.gz ... + and scan the file list of the archive for the name pattern in question (here *vacation*jpg) I'm aware of how to use the find tool, tar, unzip and alike. I could combine these with a shell script but I'm looking for a simple solution that might be a shell one-liner or a dedicated tool (hints to GUI tools are welcome but my solution must be command line based).

Until now, I used to backup my data using tar with one of the LZMA compression options (--lzma, --xz or --lzip) I recently noticed that 7-Zip has been ported to Linux in 2021 (https://www.xda-developers.com/7-zip-linux-official-release/) . I'm not talking about the older P7Zip (https://p7zip.sourceforge.net/) , that doesn't seem to be maintained anymore, but about the official 7-Zip. So, I tested it, and was very surprised to discover that it's A LOT faster than all the others Linux LZMA implementations, for the same compression ratio. Below my tests (Debian 11). Please not that I emptied the RAM cache between every test (sync && echo 3 > /proc/sys/vm/drop_caches) I am working on a 163M folder, containing several type of files, PDF, text, open office, and so on...

$ du -hs TEST/
163M	TEST/

With 7-Zip it's compressed into a 127M file in 15 seconds :

$ time tar c -hp TEST/ | 7zz a -si test.tar.7z
real    0m14,565s
(...)

$ ll test.tar.7z
(...) 127M (...) test.tar.7z

Whereas with all the other implementations of LZMA, it takes almost 5 times longer (around 1'13"), for the same archive size !

$ time tar -chp --lzma -f test.tar.lzma TEST/
real    1m13,159s

$ time tar -chp --xz -f test.tar.xz TEST/
real    1m12,889s

$ time tar -chp --lzip -f test.tar.lz TEST/
real    1m12,525s

$ ll test.tar.{7z,lz*,xz}
(...) 127M (...) test.tar.7z
(...) 127M (...) test.tar.lz
(...) 127M (...) test.tar.lzma
(...) 127M (...) test.tar.xz

Just to be sure there's nothing wrong with tar, I did the same tests but piped tar's output to lzma|xz|lzip, instead of using the --lzma, --xz and --lzip switches. Same results. So, basically, 7-Zip's Linux version makes all other LZMA implementations look rather bleak. I think 7-Zip doesn't support Linux owners and permissions, but that's irrelevant when compressing a .tar file. So, does anybody know why 7-Zip's Linux version is so much faster than other LZMA implementations ? ```

How can I use the 7z compression with PPMD algorithm? It produces better compression than the default in 7z.

I did quite a bit of searching before asking this! (Even so I'm expecting to have a duplicate pointed out). So I collect a bunch of files, e.g.: find . -name "2020-*" ... is there some way to "pipe" all these files into a .7z archive? (NB I realise "pipe" may not be the right word as the operation I want is in fact a file-copy operation). I've tried things like: find . -name "2020-*" | 7z a -p pumple.7z ... this gives me ">" on the next line, as though it's expecting some console input. So does the following: find . -name "2020-* > 7z a -p pumple.7z

To extract all 7z archives into their own folder, using the archive name as the folder name, I run the command:

7z x "*.7z" -o*

E.g., if one has the two files a.7z and b.7z, then folders a and b will be created and they'll contain the content of a.7z and b.7z (keeping the directory structure), respectively. Sometimes, I have many 7z archives, and I'd like to extract them using multiple cores. I read this [answer](https://unix.stackexchange.com/a/690784/16704) by [Artem S. Tashkinov](https://unix.stackexchange.com/users/260833/artem-s-tashkinov) : > 7z decompression is single threaded and that's the limitation of the compression format Therefore, I am looking at using parallel, since in my case I have multiple 7z archives. I tried:

parallel -j4 '7z x {} -o{}' ::: *.7z

However, if we go back to the example with the two files a.7z and b.7z, the command will try to extract the two files a.7z and b.7z to the folders a.7z and b.7z instead of a and b, which will crash the execution since a.7z and b.7z are already taken and are files. How can I fix that? I want to have two folders, a and b. If that matters, I use Ubuntu.

When trying to create an archive with Debian GUI using **Create archive...** from right-click menu and **Create a new archive** window, is there an option to assign a password to 7z archive?

I have a bunch of large zip files on a Synology. The Synology built-in extractor is a bit fiddly, and I'd like to check whether the files were extracted correctly using the commandline. Installed is 7z, and the "standard" Linux tools. What are some methods I can make sure the files were all extracted correctly? I've tried asking GPT for some answers, but it only told me that I use -t to test whether the files *inside* the zip aren't corrupted. The 7z manual also didn't look like it supported this out of the box, or I missed it, which is totally possible. Obviously, I searched the internet and StackOverflow as well.

I'm using this for creating backups securely (? - is it really secure? with a good password?): # ENCRYPT ORIGDIR="DIRECTORYNAMEHERE"; tar cvf - "${ORIGDIR}/" 2>/dev/null | gzip -9 - 2>/dev/null | openssl aes-256-cbc -salt -out "${ORIGDIR}.tar.gz.aes" # DECRYPT openssl aes-256-cbc -d -salt -in "ENCDIRECTORYNAMEHERE" | tar -xz -f - **Q:** But how can I do this using 7z with max compression rate? Creating temporary files besides the only OUTPUT file is not good, because if I need to compress ~100 GByte sized files/directories on a 180 GByte FS I wouldn't have enough free space (if ex.: the compressed file would take ~60 GByte).

I have a 7z file that contains binary data and both large (>1Mb) and small ( tmplist1 num="6" grep -E "\s[0-9]{$num}\s" tmplist1 | grep -oE '[^[:space:]]+$' | sed "s/^/*/" > tmplist2 7z e -o/tmp test.7z -i@tmplist2 -aos grep -E "*string*" /tmp/*.xml > tmplist3 ```

As in title. I've got a lot of ZIP archives that I want to extract. All archives have their own unique name. All archives contain **files only** (inside archives there are **NOT** folder(s) at all: no parent / main folder). I'd like to process all these ZIP archives via GNU parallel. To sum up: - archivename(s).zip has **NOT** folder(s) inside - extract content of archivename(s).zip into archivename(s)/ folder (this folder needs to be created!) - **keep** archivename(s).zip after extracting it - repeat this for all the ZIP archivename(s).zip I was wondering about what utility fits best ZIP extraction: gunzip? unzip? bsdtar? 7z? P. S.: *I'd like to take advantage of GNU parallel for speeding up the whole operation (I'm using SATA SSD devices)*.

I have a couple tarballs that are many gigabytes large and contain several thousand files and am wondering if it is possible to convert these to more versatile 7z archives, because the current tarballs make pulling individual files out of them very tedious. If at all possible I would like to do this in-place, i.e. without having to fully extract the tarballs. I thought I could do it with the following command, but the resulting archive just contains a single file with all the data.

tar -xf 100GB.tar -O | 7za a -si -ms=16f32m 100GB.7z

Is this somehow possible?

Because of specifics of my archiving needs I am not comfortable with solid tar.gz archives and use 7z instead. I use the following command to do this: 7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=off ~/my/folder.7z ~/my/folder/* To create an archive of everything inside ~/my/folder/ as the ~/my/folder.7z file. But ~/my/folder/.hiddenFolder doesm't get into the archive then. How to fix this? Isn't * supposed to return all the files and folders?

I would like a shell (or other terminal based) script to convert zip files to 7z format. Some zip files contain folders with more zip files in them. This needs to work recursively since there are zip files in folders inside the origin zip files. Hope this makes sense. I am using linux not windows. Regards, Brian