Sample Header Ad - 728x90

Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

18 votes
2 answers
3502 views
zsh can't input to terminal when piping stdin and stdout with variable command that has tty output
System info: macOS Sierra 10.12.6 zsh 5.4.2 (x86_64-apple-darwin16.7.0) GNU bash, version 4.4.12(1)-release (x86_64-apple-darwin16.3.0) Scroll to the **EXAMPLES** at the bottom if you just want to dig in to the simplified examples that I made. NOTE: I am not a big `zsh` user. --- I was looking at th...
System info: macOS Sierra 10.12.6 zsh 5.4.2 (x86_64-apple-darwin16.7.0) GNU bash, version 4.4.12(1)-release (x86_64-apple-darwin16.3.0) Scroll to the **EXAMPLES** at the bottom if you just want to dig in to the simplified examples that I made. NOTE: I am not a big zsh user. --- I was looking at the [fzf](https://github.com/junegunn/fzf) keybindings for [bash](https://github.com/junegunn/fzf/blob/master/shell/key-bindings.bash#L59) and [zsh](https://github.com/junegunn/fzf/blob/master/shell/key-bindings.zsh#L73) . Notice how they both run a variable command $(__fzfcmd). __fzfcmd by default outputs fzf to stdout and the parameter substitution just runs command (fzf) resulting from the output. One difference between the bash and zsh script is that the bash one further pipes the output of $(__fzfcmd) but zsh just captures it inside an array. My guess is because of a problem in zsh when you further pipe the output of fzf where you can't input to fzf and the process piped to by fzf doesn't get any stdin. Your only choice is to ^Z or ^C. ^C seems to background the process for some reason. Or maybe they just wanted it in an array so they could could [run zle vi-fetch-history on it](https://github.com/junegunn/fzf/blob/master/shell/key-bindings.zsh#L78) . The bash version does some magic in the key binding with ["\e^": history-expand-line](https://github.com/junegunn/fzf/blob/master/shell/key-bindings.bash#L83) Now fzf isn't important. It seems like you just need a program that outputs to the tty to be called by parameter substitution to cause this problem. So I will show some simpler examples. Here are some other commands that output to the tty that can cause this problem in zsh: - [vipe](https://linux.die.net/man/1/vipe) (run editor in middle of a pipe) - 'vim -' (make vim read from stdin. similar to vipe but won't output to stdout) --- In the examples below, replace every occurrence of vipe with vim - if you don't want to do a separate install. Just remember that vim - won't output the editor contents to stdout like vipe does. **EXAMPLES:** 1) echo 1 | vipe | cat # works in both bash and zsh 2) echo 1 | $(echo vipe) | cat # works in bash only. zsh problem with no output until I hit ^C: ^C zsh: done echo 1 | zsh: suspended (tty output) $(echo vipe) | zsh: interrupt cat # seems like the process is backgrounded. I can still see it in jobs command 3) cat >(cat) # works in both bash and zsh # The following don't have and input pipe to vipe. # Type something then send EOF with ^D 6) vipe | cat # works for both 7) $(echo vipe) | cat # works for both Now, I'm mostly wondering why 2) has a problem for zsh but not for bash and why 4) and 5) fixes the problem for zsh. The requirements for zsh to have this problem to seem to be exactly what I put in the title: - input pipe - command run by variable/parameter substitution that has tty output - output pipe **UPDATE** I added another workaround that doesn't cause zsh to have this problem, 5). It's similar to 4) but instead of redirecting stdout directly into stin, I redirect it into a file that redirects into stdin using process substitution.
dosentmatter (536 rep)
Dec 18, 2017, 10:44 PM • Last activity: Jul 24, 2025, 04:48 AM
-1 votes
1 answers
81 views
Serving a file (e.g. in Apache) from a named pipe (made with mkfifo)
Let's say I use Apache, and that I am in `/var/www/html/`. I do: mkfifo test.tar tar cvf - ~/test/* > test.tar & In a browser, when trying to download `http://localhost/test.tar` I get: ERR_EMPTY_RESPONSE (didn’t send any data) Is there a specific parameter of `mkfifo` that would the pipe look reall...
Let's say I use Apache, and that I am in /var/www/html/. I do: mkfifo test.tar tar cvf - ~/test/* > test.tar & In a browser, when trying to download http://localhost/test.tar I get: ERR_EMPTY_RESPONSE (didn’t send any data) Is there a specific parameter of mkfifo that would the pipe look really like a regular file? Here the problems seems to come from the fact the file is a named pipe. In my real example, I might use different webservers (not Apache), like [CivetWeb](https://github.com/civetweb/civetweb) , but first I want to analyze if it works with Apache (that I use the most).
Basj (2579 rep)
Jul 13, 2025, 03:26 PM • Last activity: Jul 21, 2025, 07:11 AM
1 votes
1 answers
2285 views
Remove newline character after pipe
I want to do Base64 encode with a command echo -en "my_message" | openssl sha1 -hmac "secret_key" | base64 The output string of `openssl` is as expected, but the output of base64 is not as the output from openssl has a new line character. If I run the command echo -en "my_message" | openssl sha1 -hm...
I want to do Base64 encode with a command echo -en "my_message" | openssl sha1 -hmac "secret_key" | base64 The output string of openssl is as expected, but the output of base64 is not as the output from openssl has a new line character. If I run the command echo -en "my_message" | openssl sha1 -hmac "secret_key" | xargs echo -n | base64 Then the final output is correct. I wonder if there is a more elegant way for this command
Dino Tw (111 rep)
Mar 26, 2020, 07:34 AM • Last activity: Jul 14, 2025, 12:02 PM
16 votes
2 answers
5188 views
How to unbuffer cut?
I want to get just e-mail addresses that end in "@xyz.nl" from my mail logfile. To achieve this I do: # tail -f /var/log/mail.log | grep --i --line-buffered "@xyz.nl" | cut -d '@' -f 1 | cut -d ' , relay=123.456.123.456[123.456.123.456]:25, delay=2, delays=0.4/0/0.4/1.2, dsn=2.0.0, status=sent (250...
I want to get just e-mail addresses that end in "@xyz.nl" from my mail logfile. To achieve this I do: # tail -f /var/log/mail.log | grep --i --line-buffered "@xyz.nl" | cut -d '@' -f 1 | cut -d ', relay=123.456.123.456[123.456.123.456]:25, delay=2, delays=0.4/0/0.4/1.2, dsn=2.0.0, status=sent (250 2.0.0 u7T9twxN074009 Message accepted for delivery) The first cut then makes: Aug 29 11:56:01 localhost postfix/smtp: 05491500123: to=
Forkbeard (657 rep)
Sep 2, 2016, 06:53 AM • Last activity: Jul 1, 2025, 12:45 PM
0 votes
0 answers
26 views
How can I grep the output of ffprobe?
I'd like to only see those lines containing "Stream #" from an ffprobe output. But whatever I do, it continues to show the whole output. Neither "|" nor ">" pipes work. What’s the magic? Thanks a lot! ffprobe -v info video.mp4 | grep "Stream #"
I'd like to only see those lines containing "Stream #" from an ffprobe output. But whatever I do, it continues to show the whole output. Neither "|" nor ">" pipes work. What’s the magic? Thanks a lot! ffprobe -v info video.mp4 | grep "Stream #"
Gary U.U. Unixuser (339 rep)
Jun 28, 2025, 11:16 AM • Last activity: Jun 28, 2025, 12:51 PM
3 votes
2 answers
3891 views
How to write Docker logs to a file in real time (à la `tail -f`)
I docker that output logs stdout stderr, which can be viewed using: docker logs -f $LOGS_CONTAINER_ID I also added 'sed', which puts the container id on each line: docker logs -f $LOGS_CONTAINER_ID | sed "s/^/$LOGS_CONTAINER_ID /" If I run it, I get something like: container112 error 10:20:10 proble...
I docker that output logs stdout stderr, which can be viewed using: docker logs -f $LOGS_CONTAINER_ID I also added 'sed', which puts the container id on each line: docker logs -f $LOGS_CONTAINER_ID | sed "s/^/$LOGS_CONTAINER_ID /" If I run it, I get something like: container112 error 10:20:10 problem container112 info 10:20:09 not problem container112 error 10:20:01 problem where "container112" is $LOGS_CONTAINER_ID. SO FAR SO GOOD. Now I want to output the above command to a file (log.out), so I wrote the following command: docker logs -f $LOGS_CONTAINER_ID | sed "s/^/$LOGS_CONTAINER_ID /" >> log.out What happens is that it writes the logs to log.out, but it doesn't get new logs (if I open a new session and run tail -f log.out, I don't get output). So I also tried: tail -f $(docker logs -f $LOGS_CONTAINER_ID | sed "s/^/$LOGS_CONTAINER_ID /") >> log.out But it also didn't work. What is the problem?
Yagel (143 rep)
Mar 3, 2019, 08:09 PM • Last activity: Jun 21, 2025, 08:03 PM
91 votes
6 answers
209181 views
Bash: How to read one line at a time from output of a command?
I am trying to read the output of a command in bash using a `while loop`. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash test.sh ./test.py ./test1.py ./out1 ./test.sh ./out ./out2 ./hello ranveer@ranveer:~/tmp$ After this I tried $(find . -...
I am trying to read the output of a command in bash using a while loop. while read -r line do echo "$line" done <<< $(find . -type f) The output I got ranveer@ranveer:~/tmp$ bash test.sh ./test.py ./test1.py ./out1 ./test.sh ./out ./out2 ./hello ranveer@ranveer:~/tmp$ After this I tried $(find . -type f) | while read -r line do echo "$line" done but it generated an error test.sh: line 5: ./test.py: Permission denied. So, how do I read it line by line because I think currently it is slurping the entire line at once. Required output: ./test.py ./test1.py ./out1 ./test.sh ./out ./out2 ./hello
RanRag (6035 rep)
Oct 16, 2012, 08:35 PM • Last activity: Jun 20, 2025, 06:36 AM
37 votes
1 answers
42157 views
find and rsync?
I want to be able to search for files over 14 days and over 10k and than rsync those found files to a destination. Is there a way to combine these two commands? find ./ -mtime +14 -size +10k rsync --remove-sent-files -avz /src /dest
I want to be able to search for files over 14 days and over 10k and than rsync those found files to a destination. Is there a way to combine these two commands? find ./ -mtime +14 -size +10k rsync --remove-sent-files -avz /src /dest
mkrouse (959 rep)
Aug 16, 2013, 05:58 PM • Last activity: Jun 2, 2025, 12:31 PM
5 votes
3 answers
490 views
Can grep output the complete input verbatim only if there was a match?
I'm processing input data in a pipeline consisting of several programs. In one step of the pipeline, I want to know if there are certain lines in the input, but I want to output everything as-is without filtering for matching lines. *Edit: Unless no match is found, then the output should be empty.*...
I'm processing input data in a pipeline consisting of several programs. In one step of the pipeline, I want to know if there are certain lines in the input, but I want to output everything as-is without filtering for matching lines. *Edit: Unless no match is found, then the output should be empty.* grep can do sort of the opposite, when supplying -q where it outputs nothing but returns with 0 if there was a match or something else if there wasn't. Example of what I want:
`
# positive case
$ echo -e 'text\nthat\nI\nwant' | grep --output-input-as-is 'that'
text
that
I
want

$ echo $?
0

# negative case
$ echo -e 'text\nthat\nI\nwant' | grep --output-input-as-is 'foo'

$ echo $?
1
` Obviously --output-input-as-is is not a real grep argument. *Edit note: I retroactively changed what this question wants after realising that what I initially wanted to happen did not actually solve my problem.*
bitmask (1246 rep)
May 30, 2025, 05:14 PM • Last activity: May 31, 2025, 11:43 PM
17 votes
1 answers
8354 views
`docker logs foo | less` isn't searchable or scrollable but `docker logs foo 2>&1 | less` is
Using either `docker logs foo | less` or `docker logs foo 2>&1 | less` produces readable text, but only with the stderr redirect can one scroll or type `/somepattern` and obtain matches. Without it, searching gives "Nothing to search (press RETURN)" and a column of `~`'s. Given that stderr and stdou...
Using either docker logs foo | less or docker logs foo 2>&1 | less produces readable text, but only with the stderr redirect can one scroll or type /somepattern and obtain matches. Without it, searching gives "Nothing to search (press RETURN)" and a column of ~'s. Given that stderr and stdout aren't the same, why does less show them the same until I start doing something in less? This may be some weird multi-window vim thing that I just don't understand. Thoughts?
MagicWindow (311 rep)
Mar 29, 2016, 09:01 PM • Last activity: May 23, 2025, 08:37 AM
0 votes
2 answers
65 views
One-liner piping from find/xargs with paths including spaces
The following question likely does not relate specifically to Vim. I use a Vim example, as this is where I encounter the issue. Working on Ubuntu, I often open multiple files in Vim using tab pages: ``` $ vim --help | grep tab -p[N] Open N tab pages (default: one for each file) ``` I also use `find`...
The following question likely does not relate specifically to Vim. I use a Vim example, as this is where I encounter the issue. Working on Ubuntu, I often open multiple files in Vim using tab pages:
$ vim --help | grep tab
   -p[N]		Open N tab pages (default: one for each file)
I also use find with xargs and grep -l to obtain a list of files.
find . -type f -name "*.txt" | xargs grep -l "zod"
I can then quickly review the files output by find in vim:
vim -p find . -type f -name "*.txt" | xargs grep -l "zod"
The earlier grep command would fail if there are spaces in the files or directories, so -print0 can be added to the arguments to find; and -0 can be added to the arguments to xargs. The following creates a MWE set of sample files and directories including spaces:
echo zod > xx.txt && echo zod > 'x x.txt' && mkdir aa && echo zod > aa/xx.txt && echo zod > 'aa/x x.txt' && mkdir 'a a' && echo zod > 'a a/xx.txt' && echo zod > 'a a/x x.txt'
The following will then list the 6 text files we expect:
find . -type f -name "*.txt" -print0 | xargs -0 grep -l "zod"
But if I then try to pass the output of this command to vim tab pages (as below), the paths including spaces are split, and opened as 2 existent, and 9 non-existent files. Is there a way to get past the problem?
vim -p find . -type f -name "*.txt" -print0 | xargs -0 grep -l "zod"
I am keen to avoid side-effects such intermediate files or shell/environment variables (such as used in the top answer to a similar question [here](https://unix.stackexchange.com/a/597765)) ; and so I am looking specifically for a single-line command.
user7543 (274 rep)
May 21, 2025, 03:11 PM • Last activity: May 22, 2025, 01:23 PM
10 votes
3 answers
2041 views
bash: echo "hello" | > file.txt results in an empty file
I use the Git Bash on Windows, and stumbled over a strange behaviour. When I execute this command, it creates an empty file: echo hello | > hello.txt (hello.txt is empty after this command) I accidently combined a pipe with a redirection (This was originally a multi-line pipeline where I removed som...
I use the Git Bash on Windows, and stumbled over a strange behaviour. When I execute this command, it creates an empty file: echo hello | > hello.txt (hello.txt is empty after this command) I accidently combined a pipe with a redirection (This was originally a multi-line pipeline where I removed some lines for debugging purposes, and I didn't see that I put pipe symbol and redirection next to each other). Can someone explain this behaviour? I expected either an error or a file with content.. I tried to reproduce the problem on OSX with zsh, but here it does create a file with content. When switching to bash on OSX, the unexpected behaviour happens again. So it looks like it is something that bash does
daniel kullmann (9737 rep)
May 14, 2025, 12:48 PM • Last activity: May 21, 2025, 11:02 PM
1 votes
2 answers
2508 views
Linux nuisance: /dev/stdin doesn't work with sockets
Linux has this blitheringly annoying peculiarity that `/dev/stdin` won't work with sockets - it is hardcoded to return ENXIO. Try this: socat TCP-OPEN:localhost:1234 EXEC:cat\ /dev/stdin,nofork That's a perfectly reasonable command you'd expect to work, and does on basically every system, except Lin...
Linux has this blitheringly annoying peculiarity that /dev/stdin won't work with sockets - it is hardcoded to return ENXIO. Try this: socat TCP-OPEN:localhost:1234 EXEC:cat\ /dev/stdin,nofork That's a perfectly reasonable command you'd expect to work, and does on basically every system, except Linux. (I'm using cat as a general example of any tool that opens a filename as the only way for you to specify a specific fd to use.) The linux kernel is explicitly written to forbid sensible use of /dev/stdin in this way ­— see http://marc.info/?l=ast-users&m=120978595414993 . If you only need unidirectional capability, you can buffer the data in a separate process: socat TCP-OPEN:localhost:1234 SYSTEM:'cat | thingy /dev/stdin' It's wasteful, and worse, is useless if thingy is meant to be reading and writing to the same fd, because pipes are unidirectional in Linux. What are we meant to do? /dev/stdin simply can't be used on Linux for building pipelines with bidirectional pipes, as far as I can tell, because sockets are the only underlying mechanism on Linux that yield a bidi stream with a single fd to read and write from (unlike a pair of pipes).
Nicholas Wilson (1088 rep)
Nov 5, 2013, 03:38 PM • Last activity: May 21, 2025, 04:27 PM
1 votes
2 answers
3728 views
How to supply a command to 'stdin' of a running process from a second shell?
If you have a process that is waiting for user-input from the `stdin` scope, then how can you supply that user-input from a second terminal ? Specifically, if I run the c-program while(1){ fgets(string, len, stdin); string[strlen(string)-1] = 0; if(strcmp("Stop", string) == 0){ printf("Gotcha"); ret...
If you have a process that is waiting for user-input from the stdin scope, then how can you supply that user-input from a second terminal ? Specifically, if I run the c-program while(1){ fgets(string, len, stdin); string[strlen(string)-1] = 0; if(strcmp("Stop", string) == 0){ printf("Gotcha"); return 1; } } then how can I supply the string "Stop" to stdin of that process from another process, such that the first process will stop (and print "Gotcha") ? I've tried to run the c-program in terminal 'pts/0' and open a new terminal ('pts/1') with commands: $ echo "Stop" > /proc//fd/0 $ echo "Stop" > /dev/pts/0 where pid is the process id. The "Stop"-command is "repeated" in the first shell, but the process does not receive it.
drC1Ron (181 rep)
Dec 21, 2018, 03:05 PM • Last activity: May 19, 2025, 01:05 AM
419 votes
22 answers
840642 views
How do I trim leading and trailing whitespace from each line of some output?
I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like `trim` I could pipe my output into? Example file: test space at back test space at front TAB at end TAB at front sequence of some space in the middle some empty lines with differi...
I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into? Example file: test space at back test space at front TAB at end TAB at front sequence of some space in the middle some empty lines with differing TABS and spaces: test space at both ends
rubo77 (30435 rep)
Nov 21, 2013, 01:07 AM • Last activity: May 15, 2025, 08:01 AM
0 votes
1 answers
3259 views
Named pipe buffer after process end
I am creating named pipes in Ubuntu 18 and 16 environments in C language using gcc as compiler (`mkfifo()` and `open()`). One of the things I noticed that the named pipes remain in the filesystem after the process ends. My process is a endless process that runs in a `while(1)` loop because of my req...
I am creating named pipes in Ubuntu 18 and 16 environments in C language using gcc as compiler (mkfifo() and open()). One of the things I noticed that the named pipes remain in the filesystem after the process ends. My process is a endless process that runs in a while(1) loop because of my requirements and the only way to exit is ctrl-c or the kill command in linux. I might add a ctrl-c signal to properly handle these situations but this is not the question. Given that the named pipe remains in the filesystem (for example /tmp/named_pipe1), do I need to check if the named pipe exists in the filesystem and delete it in the beginning of the process (because the file persists in the system), or is it redundant because even the file stays in the filesystem, it's buffer is deleted and I can use it like a new fresh fifo ? Because I don't want the fifo buffers to be mixed when I ctrl-c exit the previous run of the code and start the new one. I require an empty buffer when I restart the code. **Note:** The system is not restarted between the runs of the process. Just the process is re-run. Thanks in advance.
Max Paython (101 rep)
Jan 25, 2020, 09:47 PM • Last activity: Apr 22, 2025, 09:05 AM
45 votes
7 answers
25495 views
Measure pipe throughput in the shell
There is a shell command that allows you to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you can do something like: $ somecommand | ??? | anothercommand And throughput stats (bytes/sec) are prin...
There is a shell command that allows you to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you can do something like: $ somecommand | ??? | anothercommand And throughput stats (bytes/sec) are printed to stderr, I think. But I can't for the life of me remember what that command was.
Alex B (4598 rep)
Sep 3, 2010, 06:06 AM • Last activity: Apr 14, 2025, 11:32 AM
0 votes
2 answers
134 views
How can I run if statements on a command output without terminating it?
I have to run four separate `if` statements (and their consequences) on one command output continuously as it updates (doing four different things based on four different strings which might be in the output). Here are some limitations: - It must function with the last TWO lines as the command I'm u...
I have to run four separate if statements (and their consequences) on one command output continuously as it updates (doing four different things based on four different strings which might be in the output). Here are some limitations: - It must function with the last TWO lines as the command I'm using always puts a line break at the end. - It must run without ever terminating the original command. - I've tried piping the output to a file and using entr, but that doesn't work as entr only detects when the file is closed and my method of piping into the file doesn't do that? Or something, I don't really understand it. My options, I think, are as follows: 1. Continuously save the last two lines of output to a variable, then run the if statements in a separate parallel process whenever the variable changes. 2. Run all the if statements in series as a one-liner, piping the original command's stdout unmodified through all of them. (Cursory research suggests running an if statement directly on stdin may not be possible.) 3. Pipe the same stdout directly into four separate commands at the same time. 4. Find a way to pipe a command into a file that closes the file with every update. 5. Pipe the command output to a file and do the bulk of the script in Python instead of Bash. If anyone knows a method of doing any of these I really appreciate the help. Note that I'm out of my depth here and I'll probably need a bit more explaining than usual. Thank you :)
Gridzbi Spudvetch (65 rep)
Apr 10, 2025, 01:27 AM • Last activity: Apr 11, 2025, 06:27 AM
0 votes
1 answers
72 views
What is (if any) the file descriptor of /dev/tty?
The urgent issue to read keyboard input in the pipeline is solved by the answer in https://stackoverflow.com/questions/15230289/read-keyboard-input-within-a-pipelined-read-loop: mycommand-outputpiped | while read line do # do stuff read confirm < /dev/tty done Why does it work? Aren't `tty` redirect...
The urgent issue to read keyboard input in the pipeline is solved by the answer in https://stackoverflow.com/questions/15230289/read-keyboard-input-within-a-pipelined-read-loop : mycommand-outputpiped | while read line do # do stuff read confirm < /dev/tty done Why does it work? Aren't tty redirected to standard input? Can I get a file descriptor of /dev/tty and use read -u fd instead? TIA
Martian2020 (1443 rep)
Apr 8, 2025, 04:02 AM • Last activity: Apr 8, 2025, 05:12 AM
87 votes
8 answers
21877 views
How to trick a command into thinking its output is going to a terminal
Given a command that changes its behaviour when its output is going to a terminal (e.g. produce coloured output), how can that output be redirected in a pipeline while preserving the changed behaviour? There must be a utility for that, which I am not aware of. Some commands, like `grep --color=alway...
Given a command that changes its behaviour when its output is going to a terminal (e.g. produce coloured output), how can that output be redirected in a pipeline while preserving the changed behaviour? There must be a utility for that, which I am not aware of. Some commands, like grep --color=always, have option flags to force the behaviour, but the question is how to work around programs that rely solely on testing their output file descriptor. If it matters, my shell is bash on Linux.
Amir (1891 rep)
Dec 16, 2015, 01:39 PM • Last activity: Mar 25, 2025, 03:01 PM
Showing page 1 of 20 total questions