Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

0 votes

0 answers

42 views

How to show make visible a tftp transfer progress in a script

I have a client app that launch in async mode a bash script that send a file to a TFTP server. The issue is that I don't know how to show user the progress of this transfer, worst of all this operation is about 2-3 minutes long. My workaround was to use --trace option with atftp command and put it i...

                                  I have a client app that launch in async mode a bash script that send a file to a TFTP server.
The issue is that I don't know how to show user the progress of this transfer, worst of all this operation is about 2-3 minutes long.

My workaround was to use --trace option with atftp command and put it in a file and every n-second I read this file to show user the progress in %.

    atftp -p -l test.png --verbose --trace 192.168.1.1 > progress.txt
    atftp -p --trace -l test.png 192.168.1.1 | tee progress.txt

There is a problem, strangely atftp doesn't seems to put anything on that file, meanwhile classic tftp did it.
Did I missing something about atftp or did you suggest other way to retrieve progress status of this operation? 
Unfortunately I'm on a Debian Squeeze on a very legacy configuration. Thanks in advance and have a nice day.

EDIT: first workaround found, I put tftp interactive mode (so I can active trace) in a script and pipe out the output of that script. Now I must find a way to read only the last row data every n-seconds.
                                

Mat BCF (1 rep)

Jun 18, 2025, 06:51 AM • Last activity: Jun 18, 2025, 07:59 AM

211 votes

14 answers

265068 views

How can I move files and view the progress (e.g. with a progress bar)?

command-line mv progress-information

When moving large directories using `mv`, is there a way to view the progress (%)? The `cp` command on gentoo had a `-g` switch that showed the progress.

                                  When moving large directories using mv, is there a way to view the progress (%)?
The cp command on gentoo had a -g switch that showed the progress.
                                

letronje (2305 rep)

Oct 1, 2010, 06:11 AM • Last activity: Mar 17, 2025, 04:37 PM

4 votes

3 answers

848 views

Display progress for rsync with `--info=progress2` and `--files-from`

rsync progress-information

So I've been looking around for a while now and saw that rsync offers the insanely useful option `--info=progress2` (from threads like [this](https://serverfault.com/questions/219013/showing-total-progress-in-rsync-is-it-possible)), which prints the overall update progress quite nicely. `--no-inc-re...

So I've been looking around for a while now and saw that rsync offers the insanely useful option --info=progress2 (from threads like [this](https://serverfault.com/questions/219013/showing-total-progress-in-rsync-is-it-possible)) , which prints the overall update progress quite nicely. --no-inc-recursive ensures that the progress is absolute as well. Now one issue I've come across is that this stops working when setting a list of files via --files-from. It transfers the files just fine but only shows the progress at the very end when the transfer has already finished. To me it seems like this option just completely skips building the internal file tree and starts transferring files as it reads them from the list. So I'm wondering if there's a way I can keep this continuous progress with --files-from, even if it means building the entire file list to transfer, as it would with a recursive transfer. As requested, this is the full command line I'm using:

rsync -a -h \
    -z --compress-choice=zstd --partial \
    --info=stats1 --no-inc-recursive --info=progress2 \
    --files-from=$file_list --ignore-missing-args \
    -- $server:$source $target

And my rsync Version is 3.2.7 on my local computer and the server. They do however slightly vary in their capabilities Local computer:

$ rsync --version
rsync  version 3.2.7  protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/ 
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Server:

$ rsync --version
rsync  version 3.2.7  protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/ 
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes, SLP
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

BrainStone (3784 rep)

Feb 19, 2025, 09:10 AM • Last activity: Mar 4, 2025, 06:48 PM

2 votes

1 answers

117 views

Is there any handy way to follow multiple piped commands' status output

terminal pipe output progress-information

Let's say I have serverA with a block device, the contents of which I want to move to serverB's block device via the computer I am working on as a relay. This is very handy with pipes, buffers, dd and similar tools. In order to monitor the progress and identify any bottlenecks, I like to follow the...

                                  Let's say I have serverA with a block device, the contents of which I want to move to serverB's block device via the computer I am working on as a relay. This is very handy with pipes, buffers, dd and similar tools.

In order to monitor the progress and identify any bottlenecks, I like to follow the progress of the individual status output of each command.

Aside from complicating this one-liner by running multiple terminals, backgrounding (&), looping the outputs through FIFOs or the like, is there any way to handily sort the status outputs?

    ssh serverA "sudo dd if=/dev/sdc status=progress | mbuffer" | mbuffer | ssh serverB "mbuffer | sudo dd of=/dev/sdc status=progress"

This gives me a single row that rapidly switches between the five different commands' status and also does not clean the row in between. For example alternating between the following (note the '00% full' on the alternate output):

    in @ 6137 kiB/s, out @ 6137 kiB/s, 26.3 GiB total, buffer 100% full
    30353129472 bytes (30 GB, 28 GiB) copied, 1158 s, 26.2 MB/s00% full

A neat output would keep each command's output consistently on a separate row.

Johan Ehnberg (119 rep)

Feb 22, 2022, 04:28 PM • Last activity: Oct 5, 2024, 02:35 PM

0 votes

0 answers

33 views

How can I get the _real_ progress of dd, ignoring fast/staging buffers?

dd usb-drive io buffer progress-information

I'm using the `dd` command to write to a USB stick. My command is pretty straightforward: ``` dd if=myimage.iso of=/dev/sdd bs=1M status=progress ``` and indeed, I seem to be getting the progress reported: ``` 1234567890 bytes (1.2 GB, 1.2 GiB) copied, 15 s, 101 MB/s ``` ... but after a few seconds,...

I'm using the dd command to write to a USB stick. My command is pretty straightforward:

dd if=myimage.iso of=/dev/sdd bs=1M status=progress

and indeed, I seem to be getting the progress reported:

1234567890 bytes (1.2 GB, 1.2 GiB) copied, 15 s, 101 MB/s

... but after a few seconds, I'm told that the full amount of data has been copied - without dd actually concluding. Likely, the data has been written into some temporary buffer (in memory perhaps?) - but the process of physically writing it to the USB drive is ongoing. Only when it is actually done, does the dd command return. Is there a way to obtain the real progress, in terms of amount of bytes really written and/or estimated time remaining, for such dd executions?

einpoklum (10753 rep)

Jul 12, 2024, 06:54 PM

17 votes

7 answers

37112 views

progress information via pv for directory copy

osx directory cp progress-information

I need to copy a very large directory (talking in terabytes here) and want to monitor the progress. I found that `pv` is a nice utility, but how can I use it for copying directories recursively? (`pv src > dst` doesn't work because they are directories) PS: Running on OS X Mountain Lion, `pv` was in...

                                  I need to copy a very large directory (talking in terabytes here) and want to monitor the progress.

I found that pv is a nice utility, but how can I use it for copying directories recursively? (pv src > dst doesn't work because they are directories)

PS: Running on OS X Mountain Lion, pv was installed from Mac Ports

F.P (295 rep)

Nov 18, 2012, 05:29 PM • Last activity: Mar 9, 2024, 09:53 AM

9 votes

5 answers

22316 views

Estimate time or work left to finish for rsync?

rsync progress-information

When running `rsync` on a directory with lots of files and directories at multiple levels, can we estimate the amount of work or time to finish? - the progress option only shows the progress of transferring a single file, not the progress of transferring all files and directories under the source di...

                                  When running rsync on a directory  with lots of files and directories at multiple levels, can we  estimate the amount of work or time to finish?

 - the progress option only shows the progress of transferring a single file, not the progress of transferring all files and directories under the source directory.

 - One way I guess is to look at the what directories it has
    transferred so far, and compare that to the source. 
    
    It will help a lot if I know the order of files and directories in
    which rsync transfers them. 

    I guess that it may be related to that rsync runs multiple threads  and what each thread does?  

    I am not sure what order it chooses, and my previous guess of bread-first order seems not correct (so I strike it through).

Tim (106420 rep)

Mar 7, 2015, 03:01 PM • Last activity: Mar 9, 2024, 09:37 AM

67 votes

8 answers

100200 views

When I use zip, how can I display the overall progress without flooding the command line?

bash terminal zip progress-information

A fixed-length progress bar, a file or byte count, or better yet a timer showing the estimated time remaining would be ideal. `zip`'s standard behavior seems to be to print a line for every file processed, but I don't want that information overload when I zip thousands of files. I want a guesstimate...

                                  A fixed-length progress bar, a file or byte count, or better yet a timer showing the estimated time remaining would be ideal.

zip's standard behavior seems to be to print a line for every file processed, but I don't want that information overload when I zip thousands of files. I want a guesstimate how long it's going to take.

I tried the -q (--quiet) option in combination with -dg (--display-globaldots) but that just floods stdout with multiple lines of dots and gives no useful indication.

I also tried -qdgds 10m as mentioned in the man page, but got the same result.

I then tried -db (--display-bytes) and -dc (--display-counts) but there doesn't seem to be a global option, so it again prints it for every filename.

Lastly, I tried it together with -q like -qdbdc, but that just outputs nothing.

Funnily enough, I found a man page on the info-zip site that mentions a -de (--display-est-to-go) option which should "Display an estimate of the time to finish the archiving operation."

That sounds exactly like what I want, but the problem is that my version of zip does not have that feature. I'm using Ubuntu 14.04.1 64bit, bash-4.3.30(1) and zip-3.00. According to Wikipedia, this is zip's latest stable release. 

There are unreleased beta versions on the info-zip sourceforge page, but I'd rather not entrust my data to a beta release.

bug (2698 rep)

Jan 17, 2015, 01:48 PM • Last activity: Jan 5, 2024, 07:41 PM

0 votes

1 answers

336 views

How to read a command's output while it is running?

bash shell-script progress-information

If a command were to output, say, a progress bar that updates live, until it stops, and I wanted to constantly obtain the percentage of said progress bar and display it in a GUI such as zenity, how would one go about to do this. I do know how to obtain it when the command ends, storing it in a varia...

                                  If a command were to output, say, a progress bar that updates live, until it stops, and I wanted to constantly obtain the percentage of said progress bar and display it in a GUI such as zenity, how would one go about to do this.

I do know how to obtain it when the command ends, storing it in a variable. This is what most questions on this site ask, but I want to do it while the command is running.

I imagine it is possible as many apps do it already, such as many game launchers, app stores, and such. Once again, how would I go on to do the same?

The Cool Man (1 rep)

Aug 30, 2023, 01:57 PM • Last activity: Aug 30, 2023, 06:35 PM

19 votes

4 answers

22202 views

Compare massive directories with progress report

rsync diff progress-information

I just rsync-ed 2,000,000 files (3TB) from one RAID to another. I want to make sure my data is intact. `rsync -c` takes a really long time. `diff` doesn't show me what it's doing. Is there an alternative that's (a) faster, and (b) will show me progress while it's comparing? (I'm on Mac, and `brew se...

                                  I just rsync-ed 2,000,000 files (3TB) from one RAID to another.

I want to make sure my data is intact.

rsync -c takes a really long time.

diff doesn't show me what it's doing.

Is there an alternative that's (a) faster, and (b) will show me progress while it's comparing?

(I'm on Mac, and brew search diff gives me `apgdiff		colordiff	diffstat	diffutils	fmdiff		libxdiff	open-vcdiff	podiff		rfcdiff		vbindiff
bsdiff		diffpdf		diffuse		dwdiff		kdiff3		ndiff		perceptualdiff	rdiff-backup	tkdiff		wdiff
` ... would one of these do the job?)

d0g (867 rep)

Jan 28, 2014, 04:38 AM • Last activity: Dec 6, 2022, 08:04 PM

0 votes

1 answers

102 views

Pipeline progress report

shell pipe progress-information

Suppose I'm running a pipeline like this: ``` cat input.txt | foo | bar | baz | last_step > result.txt ``` Is there a way to make the shell print each step to stderr as it is executed? I know that if the commands `foo`, `bar` etc themselves printed to stderr it would accomplish this. However I'd lik...

Suppose I'm running a pipeline like this:

cat input.txt | foo | bar | baz | last_step > result.txt

Is there a way to make the shell print each step to stderr as it is executed? I know that if the commands foo, bar etc themselves printed to stderr it would accomplish this. However I'd like to have a general solution that does not rely on the pipeline steps providing progress info.

Dommondke (143 rep)

Oct 5, 2022, 08:09 PM • Last activity: Oct 5, 2022, 09:25 PM

0 votes

0 answers

196 views

estimating cp -r time

linux rsync cp progress-information

For a `/data` type folder that has gigabytes and thousands of files and sub folders, I will often do tar -cf data.tar data {it was a 102gb tar file} rsync -P data.tar /some_new_location cd /some_new_location tar -xf data.tar the `rsync -P` I like because it provides the following output during copy...

                                  For a /data type folder that has gigabytes and thousands of files and sub folders, I will often do

    tar -cf   data.tar    data      {it was a 102gb tar file}

    rsync -P data.tar   /some_new_location

    cd /some_new_location

    tar -xf data.tar

the rsync -P I like because it provides the following output during copy of a single file for example

    # rsync -P   rhel-8.6-x86_64-dvd.iso   ~
      rhel-8.6-x86_64-dvd.iso                        {it is 11Gb in size}
      1,731,657,728  15%  413.16MB/s    0:00:23

Is there a way to do a cp -rp     /data/*    /some_new_location/  and get any kind of progress meter or estimated time to completion, rather than play numerous solitaire and freecell waiting for the linux prompt to come back?  And same question if using mv.

ron (8647 rep)

Sep 21, 2022, 05:26 PM • Last activity: Sep 21, 2022, 05:49 PM

0 votes

0 answers

40 views

How to wrap a process with an activity indicator?

shell terminal progress-information

I have a process which I'm running in a terminal session. The process prints nothing (or perhaps it prints a lot of junk which I redirect somewhere), and takes quite a while, e.g. a couple of minutes. Now, I want to wrap it somehow, so that while it's running, I'll get some sort of visual indication...

                                  I have a process which I'm running in a terminal session. The process prints nothing (or perhaps it prints a lot of junk which I redirect somewhere), and takes quite a while, e.g. a couple of minutes.

Now, I want to wrap it somehow, so that while it's running, I'll get some sort of visual indication of it not being stuck. It could be a rotating bar (-->/->|->\->-), or some point which goes back-and-forth on the line, or at worst even just some printouts with newlines every once in a while. 

Bonus points if this progress indicator becomes "stuck" if the process becomes stuck and its CPU usage drops to 0 etc.

einpoklum (10753 rep)

Jun 19, 2022, 11:30 AM

0 votes

1 answers

1462 views

How to add progress status of a downloading file in bash script

bash shell-script progress-information

I want to show (regularly update) my download/upload progress of a file on my telegram bot. I'm using aria2 for downloading and rclone for uploading. And all I'm using is bash/shell script. ``` aria2c $url | text=$(grep '%') | curl -s "https://api.telegram.org/bot${TG_TOKEN}/editMessageText" --data...

I want to show (regularly update) my download/upload progress of a file on my telegram bot. I'm using aria2 for downloading and rclone for uploading. And all I'm using is bash/shell script.

aria2c $url | text=$(grep '%') | curl -s "https://api.telegram.org/bot${TG_TOKEN}/editMessageText " --data "message_id=${msg_id}&text=DOWNLOADED-${text}&chat_id=${ch_id}&parse_mode=HTML&disable_web_page_preview=True"

Also I want it to update after every 5 seconds.

sauraj (21 rep)

Jan 18, 2022, 05:39 PM • Last activity: Jan 19, 2022, 09:32 AM

1 votes

1 answers

3692 views

Progress bar "dialog" to show progress based on command execution from function

linux bash ubuntu dialog progress-information

How to make "dialog" progress bar increasing the counter, when each echo command is executed from function? I have this code below as example, but I'm having problem of catching when command from function is executed. I can count number of "echo" in function and set as "items", but how to know when...

                                  How to make "dialog" progress bar increasing the counter, when each echo command is executed from function?

I have this code below as example, but I'm having problem of catching when command from function is executed. I can count number of "echo" in function and set as "items", but how to know when echo is finished and how to then increase the bar?

    #!/bin/bash
    function two() {
       echo "test2-1"; sleep 1;
       echo "test2-2"; sleep 1;
       echo "test2-3"; sleep 1;
       echo "test2-4"; sleep 1;
       echo "test2-5"; sleep 1;
    }

    (
        items=5
        processed=0
        while [ $processed -le $items ]; do
            pct=$(( $processed * 100 / $items ))
            echo "XXX"
            echo "Processing item $processed" # Here I wish instead $processed
                                              # to be value (test2-1, test2-2 etc.)
                                              # of processed echo
            echo "XXX"
            echo "$pct"
            processed=$((processed+1))
            sleep 3 # Instead of this it should be when echo is finished printing
        done
    ) | dialog --title "Gauge" --gauge "Wait please..." 10 60 0
                                

Sasha Milic (25 rep)

Dec 4, 2021, 01:45 PM • Last activity: Dec 4, 2021, 08:00 PM

3 votes

2 answers

2670 views

progress command doesn't show mv progress

mv coreutils progress-information

I installed `progress` by `brew install progress` % progress -v progress version 0.13 And while I'm using `mv` command I used `progress -w`, but it failed with wrong permission error. progress -w No command currently running: cp, mv, dd, tar, cat, rsync, grep, fgrep, egrep, cut, sort, md5sum, sha1su...

                                  I installed progress by brew install progress

    % progress -v
    progress version 0.13

And while I'm using mv command I used progress -w, but it failed with wrong permission error.

    progress -w
    No command currently running: cp, mv, dd, tar, cat, rsync, grep, fgrep, egrep, cut, sort, md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum, adb, gzip, gunzip, bzip2, bunzip2, xz, unxz, lzma, unlzma, zcat, bzcat, lzcat, or wrong permissions.

Should I change permission somewhere to use progress?
What am I doing wrong?

I'm using macOS Sierra version 10.12.6.

### More Info

I executed progress -w just after mv command that moved about 30GB data from internal SSD to external HDD. And I checked after the error message of progress, the mv command was still working.

    % which mv
    /usr/local/opt/coreutils/libexec/gnubin/mv
    % mv --version
    mv (GNU coreutils) 8.27
    Copyright (C) 2017 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later .
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Written by Mike Parker, David MacKenzie, and Jim Meyering.

                                

ironsand (5425 rep)

Aug 17, 2017, 03:46 PM • Last activity: Sep 29, 2021, 11:02 AM

1 votes

1 answers

1206 views

Hour glass/Progress while extracting large file from remote host in shell

shell-script zip progress-information

I have a shell script, which is using `unzip` to get a very huge file from a remote directory to local directory within the shell. This operations taking pretty long time, roughly from 20-30 min. ```sh #!/bin/sh … unzip RemoteHostNFSDirectory -d LocalHostDirectory > output.log ``` (it is a 6.2 GB fi...

I have a shell script, which is using unzip to get a very huge file from a remote directory to local directory within the shell. This operations taking pretty long time, roughly from 20-30 min.

#!/bin/sh

…

unzip RemoteHostNFSDirectory -d LocalHostDirectory > output.log

(it is a 6.2 GB file). How can I embed the above command in a progress or hour-glass bar so users don't think that it is just hanging and once extract to the local directory is done successfully. I can print success or failed error. (Newbie to shell, apologize for any inconvenience.)

Naveed Iftikhar (11 rep)

Apr 15, 2021, 12:05 AM • Last activity: Apr 16, 2021, 08:00 AM

0 votes

2 answers

1678 views

Rsync progress waits a long time after finish

rsync cache progress-information

When I am running `rsync` to copy big file (within directory on local hard drive) + rsync -P bigfile1.img bigfile2.img bigfile1.img 6,512,705,536 100% 413.76MB/s 0:00:15 (xfr#1, to-chk=0/1) it first shows progress up to 100% and then waits approximately the same time without any change and only then...

                                  When I am running rsync to copy big file (within directory on local hard drive)

    + rsync -P bigfile1.img bigfile2.img
    bigfile1.img
      6,512,705,536 100%  413.76MB/s    0:00:15 (xfr#1, to-chk=0/1)

it first shows progress up to 100% and then waits approximately the same time without any change and only then exits.

This probably due to some caching.

Can I display real progress?

Dims (3425 rep)

Jan 27, 2021, 09:29 PM • Last activity: Jan 28, 2021, 12:21 AM

0 votes

0 answers

60 views

How to show progress towards a directory reaching a certain size

rsync file-copy progress-information

I have a server to which I am copying large amounts of data using `rsync` and I would like to monitor progress on the server, not the client. I know, for a given source directory on the client how much data is due to be transferred, and I can also find the the current size of the target directory on...

                                  I have a server to which I am copying large amounts of data using rsync and I would like to monitor progress on the server, not the client.

I know, for a given source directory on the client how much data is due to be transferred, and I can also find the the current size of the target directory on the server.

Given these two pieces of information, is there any way I can display current progress, for example using pv?

j b (121 rep)

Jun 10, 2020, 10:20 AM • Last activity: Jun 10, 2020, 01:42 PM

3 votes

1 answers

1205 views

Show progress of a for loop after it was started

bash for job-control progress-information

I started a `for` loop in an interactive `bash` session. For this question we can assume the loop was something like for i in dir/*; do program "$i" done > log The command takes a lot longer than expected and still runs. How can I see the current progress of the running `for` loop. **Non-Solutions:*...

                                  I started a for loop in an interactive bash session. For this question we can assume the loop was something like

    for i in dir/*; do
        program "$i"
    done > log

The command takes a lot longer than expected and still runs. How can I see the current progress of the running for loop.

**Non-Solutions:**

- Look at log.  
*Doesn't work because:* program is expected to run silently. You can think of program as a validator. If the loop completes without any output then I'm happy.
- Stop the loop. Add some kind of progress indication (for instance echo "$i"). Start the loop again.  
*Doesn't work because:* The loop already ran for hours. I don't want to throw away all the time and energy invested in the current run. I assume everything works fine. I'm just curious and want to know the current progress.
- Ctrl+Z then set -x; fg  
*Doesn't work because:* bash doesn't continue the loop when using fg. After fg only the current/next command inside the loop will run, then the loop exits. You can try it yourself using for i in {1..200}; do printf '%s ' $i; /usr/bin/sleep 0.5; done.
                                

Socowi (645 rep)

Sep 20, 2019, 03:22 PM • Last activity: Sep 21, 2019, 01:00 PM

Showing page 1 of 20 total questions