Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

2 votes

3 answers

2809 views

Parallelize recursive deletion with find

find recursive parallelism multiprocessor

I want to recursively delete all files that end with `.in`. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From [this thread][1], it looks like it's possible to use `xargs` or `make` to parallelize `find`. Is this application of find possibl...

                                  I want to recursively delete all files that end with .in. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From this thread , it looks like it's possible to use xargs or make to parallelize find. Is this application of find possible to parallelize?

Here is my current serial command:

    find . -name "*.in" -type f -delete

kilojoules (169 rep)

Mar 6, 2017, 04:57 PM • Last activity: Jul 26, 2025, 03:16 PM

0 votes

1 answers

2150 views

how to find all pics in a directory and its subdirectories and run a command on them

find recursive imagemagick exif mogrify

how to find all images in a directory and its sub-directories, with types ```gif, jpg, jpeg, png, ico```, and run ```mogrify -strip your_filename.jpg``` on them? Can the command ```mogrify -strip``` corrupt an image? Also can the command ```mogrify -strip``` be run on a gif file?

how to find all images in a directory and its sub-directories, with types

, jpg, jpeg, png, ico

, and run

-strip your_filename.jpg

on them? Can the command

-strip

corrupt an image? Also can the command

-strip

be run on a gif file?

stacking and exchanging woohoo (63 rep)

Dec 13, 2022, 10:47 PM • Last activity: Jun 24, 2025, 02:04 PM

0 votes

0 answers

24 views

How does the search algorithm in Gnome Files traverse the directory tree?

files gnome recursive file-search

If I place myself in some directory C and press the search lens and type some keywords, are files matching the keywords in directory C displayed first? This is not obvious, for instance if the search algorithm operated by alphabetical order (hence if there's some subfolder starting by "a" then resul...

                                  If I place myself in some directory C and press the search lens and type some keywords, are files matching the keywords in directory C displayed first? This is not obvious, for instance if the search algorithm operated by alphabetical order (hence if there's some subfolder starting by "a" then results from that subfolder could be displayed before files from C which start by "b")
                                

user324831 (113 rep)

Jun 2, 2025, 06:28 PM • Last activity: Jun 2, 2025, 07:12 PM

1 votes

1 answers

54 views

How to define a recursive alias in csh (intentionally!)

alias recursive csh

It is known that attempting to invoke a custom alias inside itself (be it directly; or indirectly via intermediate aliases that call each other) would cause an "**`Alias loop.`**" error, and rightfully so. However, I would like to write a recursive `csh` alias, as I intend to include a base case to...

It is known that attempting to invoke a custom alias inside itself (be it directly; or indirectly via intermediate aliases that call each other) would cause an "**Alias loop.**" error, and rightfully so. However, I would like to write a recursive csh alias, as I intend to include a base case to the recursion, so in my use case it is not an infinite loop. Here's a simplified example:

alias countdown 'if ( \!* > 0 ) then; echo \!*; countdown expr \!* - 1; endif;'

The expected output from this alias, for example with input 3, should be:

$ countdown 3
3
2
1
0

Attempting to define such an alias leads to the aforementioned "**Alias loop.**" error on csh. How can I achieve this functionality correctly? I tried using a while loop in the alias instead of recursion, but [it doesn't seem to be supported in csh](https://stackoverflow.com/a/61298726/10679420) , and [other workarounds for this alias loop issue](https://unix.stackexchange.com/q/28704/459071) are not applicable since I do not have a built-in counterpart to invoke.

Vadrif Draco (111 rep)

May 11, 2025, 01:05 PM • Last activity: Jun 2, 2025, 06:50 AM

112 votes

14 answers

260767 views

How do I count all the files recursively through directories

disk-usage recursive inode

I want to see how many files are in subdirectories to find out where all the inode usage is on the system. Kind of like I would do this for space usage du -sh /* which will give me the space used in the directories off of root, but in this case I want the number of files, not the size.

                                  I want to see how many files are in subdirectories to find out where all the inode usage is on the system. Kind of like I would do this for space usage

    du -sh /*

which will give me the space used in the directories off of root, but in this case I want the number of files, not the size.

xenoterracide (61203 rep)

Nov 16, 2010, 11:02 AM • Last activity: May 27, 2025, 01:23 PM

19 votes

9 answers

25038 views

How do I recursively grep through compressed archives?

grep find zip recursive archive

I'm trying to find out what modules [`use Test::Version`](http://search.cpan.org/dist/Test-Version/lib/Version.pm) in cpan. So I've used [`minicpan`](http://search.cpan.org/dist/CPAN-Mini/bin/minicpan) to mirror it. My problem is that I need to iterate through the archives that are downloaded, and g...

                                  I'm trying to find out what modules [use Test::Version](http://search.cpan.org/dist/Test-Version/lib/Version.pm)  in cpan. So I've used [minicpan](http://search.cpan.org/dist/CPAN-Mini/bin/minicpan)  to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files that are in the archives. Can anyone tell me how I might do this? preferably in a way that tells me which file in the archive and what line it's on.

*(note: they aren't all tarballs some are zip files)*

xenoterracide (61203 rep)

May 25, 2011, 12:29 PM • Last activity: Mar 20, 2025, 11:10 AM

1 votes

2 answers

61 views

recursive call to gnu parallel

bash recursive gnu-parallel

I have a very slow samba share that i can access through wsl on my windows laptop. On windows accessing the share doesn't take much time but on linux it is in order of seconds. I basically replicate the file structure of the share so I made this poor-man rsync bash script. Given it takes several second to get the content of one directory, I thought it would be a good idea to 'enqueue' them in parallel to accelerate the process. So I made the following bash script. The replicate function get the current directory then call itself using parallel on all the subdirs. Running this program, I noticed the most used resources were memory and cpu usage. Disk and Network I/O are very little used. I understand that memory needs to be allocated for all the process that are created, but I supposed that the process will spend most of their time idling, either waiting for network I/O or for their children to terminate. My question is: What is the CPU doing during all this time, all my cores are used at 100%. For reference there is at all time around 800 to a 1000 process running using around 4~6G of memory.

create_file(){
    if [ ! -f "$1" ] ; then
        touch "$1";
    fi
}
export -f create_file

better_ls(){
    find "$1" -maxdepth 1 -type f ! -name '~*' -exec bash -c 'echo "- $(basename "{}")"' \; -o -type d -exec bash -c 'echo "d $(basename "{}")"' \; | tail -n+2 | sort
}
export -f better_ls


replicate() {
    DIR="realpath "$1""
    OUTPUT="$2"
    if [ ! -e "$DIR" ] ; then 
        echo "File not found: $DIR";
        return
    fi
    OUTPUT="$OUTPUT$DIR"

    echo "$DIR" | sed -e "s!/mnt/!!" -e "s;/;  ;g";

    mkdir -p "$OUTPUT" ;

    content="better_ls "$DIR"";
    alreadycopied="better_ls "$OUTPUT" | cut -d' ' -f2-";

    newcontent="echo "$content" | cut -d' ' -f2-";

    echo "$content" |
        grep '^-' |
        cut -d\  -f2- |
        xargs -I {} bash -c "create_file  \"$OUTPUT/{}\""

    diff <(echo "$alreadycopied") <(echo "$newcontent") | grep '^<' | cut -c 3- | xargs -I{} rm -r "$OUTPUT/{}"


    echo "$content" |
        grep '^d' |
        cut -d\  -f2- |
        parallel --line-buffer -I {} replicate "\"$DIR\"/{}" "$2"

}
export -f replicate;

PS: this idea was not so great and I went back to using rsync, but I still wanted to understand what was happening here.

Rouge a (11 rep)

Mar 7, 2025, 10:02 AM • Last activity: Mar 10, 2025, 03:47 PM

0 votes

3 answers

127 views

what is recursion when applied to the bash shell?

bash recursive terminology

I keep hearing people talk about "recursion" for example, when you copy and paste a file and a directory you need to also put in the `-r` flag to tell the `cp` command to copy the directory "recursively" I look up the word "recursion" and I get this > recursion is when a function can call itself and...

                                  I keep hearing people talk about "recursion" for example, when you copy and paste a file and a directory you need to also put in the -r flag to tell the cp command to copy the directory "recursively" 

I look up the word "recursion" and I get this

> recursion is when a function can call itself

and then people tell me about russian dolls and how recursion is like a program inside a program like a russian doll is like a doll inside a doll. 

So my question is, what does "recursion" mean when it's applied to the bash shell? 

I don't understand how the concept of "recursion" applies to bash or the programs in bash for example when I cp a file and a directory and I have to put the -r flag in with cp to make sure that the file AND the directory gets copied

how to bash (9 rep)

Feb 21, 2025, 02:12 PM • Last activity: Mar 7, 2025, 04:43 PM

0 votes

1 answers

145 views

why doesn't cp automatically copy directories recursively?

bash cp recursive

Why does `cp` need `-r` flag to copy "recursively"? Why doesn't `cp` automatically copy directories recursively? Would that make more sense? Is there some documentation that explains this decision? Why do I have to add this flag for `cp` to do what it should do by default?

                                  Why does cp need -r flag to copy "recursively"? Why doesn't cp automatically copy directories recursively? Would that make more sense?

Is there some documentation that explains this decision? Why do I have to add this flag for cp to do what it should do by default?

how to bash (9 rep)

Feb 11, 2025, 02:55 PM • Last activity: Feb 21, 2025, 04:18 PM

1 votes

1 answers

140 views

Recursively move and overwrite

bash mv recursive

I am using Ubuntu 24.10 and have a directory containing files and subdirectories: ``` main-dir file1 file2 file3 subdir1 file1-1 file1-2 subdir1-1 file1-1-1 file1-1-2 subdir1-2 file1-2-1 subdir1-2-1 ... ``` And I have a directory tree containing "corrected" files: ``` corrections-dir file2 subdir1 f...

I am using Ubuntu 24.10 and have a directory containing files and subdirectories:

main-dir
   file1
   file2
   file3
   subdir1
      file1-1
      file1-2
      subdir1-1
         file1-1-1
         file1-1-2
      subdir1-2
         file1-2-1
         subdir1-2-1
            ...

And I have a directory tree containing "corrected" files:

corrections-dir
   file2
   subdir1
      file1-1
      subdir1-1
         file1-1-1
      subdir1-2
         file1-2-1

I want to recursively move and overwrite all files from corrections-dir to main-dir. However, the mv command doesn't seem to have a recursive option. I know I could instead do this:

cp -R corrections-dir main-dir
rm -r corrections-dir

But the files are large and that command takes time and unnecessarily writes to new sectors of my SSD (thereby shortening its life). Is there a way to use mv recursively?

k314159 (481 rep)

Nov 25, 2024, 09:04 AM • Last activity: Nov 25, 2024, 03:40 PM

2 votes

2 answers

2855 views

How to create directories containing index.html with wget --recursive?

wget recursive

I'm pretty happy how `wget -r` works and downloads the things. I have set up a localhost server which serves a website and the pages look like this: http://localhost:8080/ http://localhost:8080/foo http://localhost:8080/bar http://localhost:8080/blog/ http://localhost:8080/blog/1-and-here-the-slug W...

                                  I'm pretty happy how wget -r works and downloads the things.

I have set up a localhost server which serves a website and the pages look like this:

    http://localhost:8080/
    http://localhost:8080/foo
    http://localhost:8080/bar
    http://localhost:8080/blog/
    http://localhost:8080/blog/1-and-here-the-slug

When I use wget -r localhost:8080 it creates the following structure:

    .
    ├── static-files
    │   └── ...
    ├── bar
    ├── blog
    │   └── 1-and-here-the-slug
    ├── foo
    └── index.html
    
bar, foo  and 1-and-here-the-slug are files. I want them to be directories with a single file in them, named index.html and still not breaking the paths to the resources (CSS, JS etc).

I expect something like this:

    .
    ├── static-files
    │   └── ...
    ├── bar
    │   └── index.html
    ├── foo
    │   └── index.html
    ├── blog
    │   ├── index.html // <---------- Also I want this one here to show the blog
    │   └── 1-and-here-the-slug
    │       └── index.html
    └── index.html

How can I do that?
                                

Ionică Bizău (3481 rep)

Oct 25, 2016, 02:03 PM • Last activity: Oct 2, 2024, 09:49 PM

5 votes

3 answers

4634 views

rsync with colons in filenames

rsync recursive

I have a fairly large directory that I need to sync to an external hard drive for both backup and portability reasons. /work I try to run the following rsync command in order to copy this folder to my external hard drive: rsync -avz /work /media/extern_drive --max-size '4G' Which seems to work fine,...

                                  I have a fairly large directory that I need to sync to an external hard drive for both backup and portability reasons.

    /work

I try to run the following rsync command in order to copy this folder to my external hard drive:

    rsync -avz /work /media/extern_drive --max-size '4G'

Which seems to work fine, EXCEPT that it does not copy any file with a : in it.

This post gives a solution for a single file:
https://unix.stackexchange.com/questions/11581/rsync-colon-in-file-names , but the problem is that I have so many of these files scattered in different directories that I cannot do it manually.

Is there any way to recursively rsync any files with a colon in the filename?

Manila Thrilla (153 rep)

Oct 13, 2014, 01:35 PM • Last activity: Aug 26, 2024, 01:22 PM

98 votes

14 answers

527998 views

How to remove all the files in a directory?

linux directory rm recursive

I am trying to remove all files and subdirectories in a directory. I used `rm -r` to remove all files, but I want to remove all files *and* subdirectories, excluding the top directory itself. For example, I have a top directory like `images`. It contains the files `header.png`, `footer.png` and a su...

                                  I am trying to remove all files and subdirectories in a directory. I used rm -r to remove all files, but I want to remove all files *and* subdirectories, excluding the top directory itself.

For example, I have a top directory like images. It contains the files header.png, footer.png and a subdirectory.

Now I want to delete header.png, footer.png and the subdirectory, but not images.

How can I do this in linux?

whiterose (1089 rep)

May 4, 2011, 07:36 AM • Last activity: Jul 13, 2024, 01:25 PM

2 votes

1 answers

126 views

rsync recursive jumps across directories

rsync recursive

I've used rsync recursively and, to my surprise, I witnessed this very strange behaviour in the output (I don't remember if it was deleting or creating or a combination, but I think it won't matter): A/B/file1 A/file2 A/B/file3 In other words, the recursion did not happen linearly (according to the...

                                  I've used rsync recursively and, to my surprise, I witnessed this very strange behaviour in the output (I don't remember if it was deleting or creating or a combination, but I think it won't matter):

A/B/file1

A/file2

A/B/file3

In other words, the recursion did not happen linearly (according to the tree structure of either the sender or the receiver). This is at least very counter-intuitive: I'd expect that every line which lies between any two lines of the form

A/B/...

would also have to be of the form A/B/...

Can someone explain this, please?

user624166

Jul 9, 2024, 10:21 AM • Last activity: Jul 9, 2024, 11:09 AM

2 votes

2 answers

8870 views

How to zip specific individual files in a directory

files filenames zip recursive

for example, this folder `~/folder1/folder2` contains a lot of files like: `abc_0101.txt` `abc_0102.txt` ... `abc_0201.txt` `abc_0202.txt` ... `abc_0301.txt` `abc_0302.txt` ... `abc_0401.txt` `abc_0402.txt` ... how to do zip each file in this folder for specific file names? like i only want to zip a...

                                  for example, this folder ~/folder1/folder2 contains a lot of files like: 

abc_0101.txt abc_0102.txt ... 

abc_0201.txt abc_0202.txt ... 

abc_0301.txt abc_0302.txt ... 

abc_0401.txt abc_0402.txt ... 

how to do zip each file in this folder for specific file names? 

like i only want to zip all files like abc_01*.txt and abc_03*.txt

dcdum2018 (37 rep)

Apr 8, 2022, 02:26 AM • Last activity: Jul 4, 2024, 03:08 AM

12 votes

3 answers

32654 views

recursive delete with sftp?

recursive sftp

I have a host that I can only access with sftp, scp, and rsync-- no ssh. I have a large tree of directories that I want to delete, but my sftp client apparently does not support recursive `rm`s or `rmdir`s. In the help screen, it shows I can do recursive `put`s, but that doesn't help me. How can I d...

                                  I have a host that I can only access with sftp, scp, and rsync-- no ssh. I have a large tree of directories that I want to delete, but my sftp client apparently does not support recursive rms or rmdirs. In the help screen, it shows I can do recursive puts, but that doesn't help me. 

How can I delete my files?

user394 (14722 rep)

Sep 23, 2013, 01:49 AM • Last activity: Jun 22, 2024, 11:58 PM

0 votes

2 answers

83 views

How to loop through multiple subdirectories and change file extension

rename filenames recursive

I was assigned to manage a legacy codeIgniter 3 code base. There is this error that the website doesn't load images that are `.JPG`. Rather than making it not case-sensitive we want to change every `.JPG` to `.jpg`. This is the file structure: ``` - 0 - 1 - 10 - 100 - 101 - 0 - 1 - 10 - 11 - 10111 -...

I was assigned to manage a legacy codeIgniter 3 code base. There is this error that the website doesn't load images that are .JPG. Rather than making it not case-sensitive we want to change every .JPG to .jpg. This is the file structure:

- 0
- 1
- 10
- 100
- 101
    - 0
    - 1
    - 10
    - 11
        - 10111
            - image.JPG
            - image-small.jpg
            - ...
    - 12
    - ...
- 102
- ...

I know how to loop through files, and replace the extensions, but not sure how to loop through this many folders. This code would be for replacing the extension

file in *.JPG; do mv -- "$file" "${f%.JPG}.jpg"; done

Džiugas Balčiūnas (3 rep)

Jun 1, 2024, 12:01 PM • Last activity: Jun 1, 2024, 01:59 PM

58 votes

6 answers

72714 views

Finding largest file recursively

bash shell-script recursive

I am trying to find the largest file in a directory recursively. If there is a subdirectory inside of that directory the function needs to go inside that directory and check to see if the largest file is there. Once the largest file is found the output is displayed with the relative path name and th...

                                  I am trying to find the largest file in a directory recursively. If there is a subdirectory inside of that directory the function needs to go inside that directory and check to see if the largest file is there. Once the largest file is found the output is displayed with the relative path name and the name and size of the largest file.

EX:

    dude@shell2 (~...assignment/solutions) % bash maxfile.sh ~/test
    class/asn
    dude.h.gch: 9481628
    
This is what I have:

    #!/bin/sh
    clear
    
    recursiveS() {
        for d in *; do
            if [ -d $d ]; then
                (cd $d; echo $(pwd)/$line; du -a; recursiveS;)
            fi
        done
    }
    recursiveS


I have been stuck for a while now. I cannot implement this by pipelining a number of existing Unix tools. Any ideas would be nice!
                                

user2419571 (771 rep)

Sep 29, 2014, 08:08 PM • Last activity: May 28, 2024, 12:51 PM

262 votes

14 answers

536708 views

How to recursively find the amount stored in directory?

directory disk-usage recursive

I know you are able to see the byte size of a file when you do a long listing with `ll` or `ls -l`. But I want to know how much storage is in a directory including the files within that directory and the subdirectories within there, etc. I don't want the number of files, but instead the amount of st...

                                  I know you are able to see the byte size of a file when you do a long listing with ll or ls -l. But I want to know how much storage is in a directory including the files within that directory and the subdirectories within there, etc. I don't want the number of files, but instead the amount of storage those files take up.

So I want to know how much storage is in a certain directory recursively? I'm guessing, if there is a command, that it would be in bytes.

Rob Avery IV (3265 rep)

Mar 13, 2013, 04:13 PM • Last activity: Jan 11, 2024, 07:33 PM

7 votes

3 answers

3058 views

id3v2 used recursively at command line?

shell command-line files recursive

Trying to get my id3 tags cleaned up and am loving id3v2 on the command line -- but I've been using it only with a *.mp3 wildcard and want to explore if there's a way to use it recursively so I can batch all of my MP3s. There doesn't seem to be an option to use it recursively. I'm pretty sure all yo...

                                  Trying to get my id3 tags cleaned up and am loving id3v2 on the command line -- but I've been using it only with a *.mp3 wildcard and want to explore if there's a way to use it recursively so I can batch all of my MP3s.  There doesn't seem to be an option to use it recursively.

I'm pretty sure all you awesome command line folks know of a sweet way to do this, but I am still learning my way through it...

Here's the commands I have aliased to brute force it:

    id3v2 --remove-frame "COMM" *.mp3
    id3v2 --remove-frame "PRIV" *.mp3
    id3v2 -s *.mp3

So -- is there a way to do this recursively, so I can run it a the root of my music folder?  Gravy points for: including other audio file types and collapsing all three commands above into one uber command (I can do this with the ; between commands... right?)

chow (471 rep)

Sep 12, 2012, 02:59 AM • Last activity: Jan 2, 2024, 09:48 PM

Showing page 1 of 20 total questions