Unix & Linux Stack Exchange
Q&A for users of Linux, FreeBSD and other Unix-like operating systems
Latest Questions
30
votes
7
answers
31505
views
How to cat files together, adding missing newlines at end of some files
I have a bunch of `.text` files, _most_ of which end with the standard nl. A couple don't have any terminator at end. The last physical byte is (generally) an alphameric character. I was using `cat *.text >| /tmp/joined.text`, but then noticed a couple of places in joined.text where the first line o...
I have a bunch of
.text files, _most_ of which end with the standard nl.
A couple don't have any terminator at end. The last physical byte is (generally) an alphameric character.
I was using cat *.text >| /tmp/joined.text, but then noticed a couple of places in joined.text where the first line of a file appeared at the end of the last line of a previous file. Inspecting the previous file, I saw there wasn't a line terminator -- concatenation explained.
That raised the question, what's the easiest way to concatenate, sticking in the missing newline? What about these options?
1. A solution that might effectively add a blank line to some input files. For me, that's not a problem as the processing of joined.text can handle it.
2. A solution that adds the cr/fl only to files that do not already end that way.
HiTechHiTouch
(991 rep)
Feb 16, 2017, 07:17 PM
• Last activity: Jul 23, 2025, 10:22 AM
3
votes
2
answers
543
views
Grep command with the side effect of adding a trailing newline character in the last line of file
I've been doing some research on how to correctly read lines from a file whose last line may not have a trailing newline character. Have found the answer in [Read a line-oriented file which may not end with a newline](https://unix.stackexchange.com/questions/418060/read-a-line-oriented-file-which-ma...
I've been doing some research on how to correctly read lines from a file whose last line may not have a trailing newline character. Have found the answer in [Read a line-oriented file which may not end with a newline](https://unix.stackexchange.com/questions/418060/read-a-line-oriented-file-which-may-not-end-with-a-newline) .
However, I have a second goal that is to exclude the comments at the beginning of lines and have found a [
grep](http://man7.org/linux/man-pages/man1/grep.1.html) command that achieves the goal
$ grep -v '^ *#' file
But I have noticed that this command has a (for me unexpected) side behavior: it adds a trailing newline character in the last line if it does not exist
$ cat file
# This is a commentary
aaaaaa
# This is another commentary
bbbbbb
cccccc
$ od -c file
0000000 # T h i s i s a c o m m
0000020 e n t a r y \n a a a a a a \n #
0000040 T h i s i s a n o t h e r
0000060 c o m m e n t a r y \n b b b b b
0000100 b \n c c c c c c \n
0000111
$ truncate -s -1 file
$ od -c file
0000000 # T h i s i s a c o m m
0000020 e n t a r y \n a a a a a a \n #
0000040 T h i s i s a n o t h e r
0000060 c o m m e n t a r y \n b b b b b
0000100 b \n c c c c c c
0000110
$ od -c <(grep -v '^ *#' file)
0000000 a a a a a a \n b b b b b b \n c c
0000020 c c c c \n
0000025
Paulo Tomé
(3832 rep)
Jan 17, 2020, 06:00 PM
• Last activity: Jul 18, 2025, 07:13 AM
1
votes
1
answers
2285
views
Remove newline character after pipe
I want to do Base64 encode with a command echo -en "my_message" | openssl sha1 -hmac "secret_key" | base64 The output string of `openssl` is as expected, but the output of base64 is not as the output from openssl has a new line character. If I run the command echo -en "my_message" | openssl sha1 -hm...
I want to do Base64 encode with a command
echo -en "my_message" | openssl sha1 -hmac "secret_key" | base64
The output string of
openssl is as expected, but the output of base64 is not as the output from openssl has a new line character. If I run the command
echo -en "my_message" | openssl sha1 -hmac "secret_key" | xargs echo -n | base64
Then the final output is correct.
I wonder if there is a more elegant way for this command
Dino Tw
(111 rep)
Mar 26, 2020, 07:34 AM
• Last activity: Jul 14, 2025, 12:02 PM
3
votes
1
answers
2676
views
nano - soft wrap at 80 characters
I would like `nano` to soft-wrap at 80 characters. That is, as I keep typing, the text should flow to the next line beyond 80 characters, without `nano` insert a `\n` newline at 80 characters. set softwrap set tabsize 4 set tabstospaces set fill 72 However, this inserts a newline after each line, wh...
I would like
nano to soft-wrap at 80 characters. That is, as I keep typing, the text should flow to the next line beyond 80 characters, without nano insert a \n newline at 80 characters.
set softwrap
set tabsize 4
set tabstospaces
set fill 72
However, this inserts a newline after each line, when tested with cat.
How do I softwrap and fill simultaneously?
Karthik Nishanth
(133 rep)
Jul 8, 2017, 04:27 PM
• Last activity: Jun 27, 2025, 07:00 AM
13
votes
1
answers
67238
views
No such file or directory but I can see it!
I'm trying to run a python script, on a headless Raspberry PI using winSCP and get the following error message: Command '"./areadetect_movie_21.py"' failed with return code 127 and error message /usr/bin/env: python : No such file or directory. When I try and run from terminal, I get: : No such file...
I'm trying to run a python script, on a headless Raspberry PI using winSCP and get the following error message:
Command '"./areadetect_movie_21.py"'
failed with return code 127 and error message
/usr/bin/env: python
: No such file or directory.
When I try and run from terminal, I get:
: No such file or directory.
I try a similar python script, in the same directory, with the same python shebang, the same permissions and using the same user pi, and it works.
I also do a
ls and I can see the file, so I don't know why it will not run.
reggie
(533 rep)
Mar 10, 2015, 12:34 PM
• Last activity: Jun 12, 2025, 04:42 PM
0
votes
1
answers
142
views
echo $’hi\nhi’ > /etc/list.list not working
When I use `echo $’hi\nhi’ > /etc/list.list` in an executable script text file I create comprising multiple commands on Kali Linux 2020.4 Live USB it: 1. Includes the `$` in the files text while disappearing the 2 `’’` symbols. 2. Does not work to create a new line, separating the 2 `hi`’s from each...
When I use
echo $’hi\nhi’ > /etc/list.list in an executable script text file I create comprising multiple commands on Kali Linux 2020.4 Live USB it:
1. Includes the $ in the files text while disappearing the 2 ’’ symbols.
2. Does not work to create a new line, separating the 2 hi’s from each other, instead just showing the \n with no separation at all.
but when I use the same command outside of the executable or in the plain terminal it works.
Did Kali 2020.4 change from 2020.3 to not allow the echo formatting I use within an executable text file, because it use to work flawlessly? Is there some other command I can use for echo that will create a new line in a 1 line command?
markbunting
(3 rep)
Feb 22, 2021, 05:27 PM
• Last activity: Jun 6, 2025, 12:34 AM
3
votes
3
answers
2431
views
Mapfile not removing trailing newline
Sample Data: Tab separated file (TSV) ``` a.1.58 fadado/CSV https://github.com/fadado/CSV a.1.63 jehiah/json2csv https://github.com/jehiah/json2csv a.1.80 stedolan/jq https://github.com/stedolan/jq/issues/370 ``` Following selects one record using `fzf`, and stores 2nd and 3rd column to an array Lin...
Sample Data: Tab separated file (TSV)
a.1.58 fadado/CSV https://github.com/fadado/CSV
a.1.63 jehiah/json2csv https://github.com/jehiah/json2csv
a.1.80 stedolan/jq https://github.com/stedolan/jq/issues/370 fzf, and stores 2nd and 3rd column to an array Link:
mapfile -d $'\t' -t Link < <(awk 'BEGIN{FS="\t"; OFS="\t"} {print $2,$3}' "${SessionP}" | fzf)-t option of mapfile, but echo "${Link}" prints a trailing new line!
**Why is it not getting eliminated?**
## Reference
- [mapfile Man Page - Linux - SS64.com](https://ss64.com/bash/mapfile.html)
Porcupine
(2156 rep)
Jul 26, 2021, 04:35 AM
• Last activity: May 30, 2025, 10:52 PM
9
votes
4
answers
7821
views
Determine whether a file has no EOL at the end from the command line
[If you open a file in `vim` and that file has no `EOL` at the end of its last line, then the editor will report it as `[noeol]`.][1] How can I determine this *before* opening it in `vim`? (Is there a command I can issue to determine this?) [1]: https://unix.stackexchange.com/questions/31807/what-do...
[If you open a file in
vim and that file has no EOL at the end of its last line, then the editor will report it as [noeol].][1] How can I determine this *before* opening it in vim? (Is there a command I can issue to determine this?)
user664833
(209 rep)
Sep 18, 2019, 08:19 PM
• Last activity: May 13, 2025, 05:06 PM
290
votes
21
answers
472953
views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says `No newline at end of file`. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
Using version control systems I get annoyed at the noise when the diff says
No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
k0pernikus
(16501 rep)
Feb 17, 2012, 12:44 PM
• Last activity: Apr 28, 2025, 12:53 PM
1
votes
4
answers
520
views
How do I print line numbers but reset line counter at empty line?
I have a file.txt containing: ``` this is the first second line not last line fourth but first second in list seventh in file seventh with nl ``` Normally I would just `cat` and pipe `|` it into `nl` like so: ```bash $> cat file.txt | nl 1 this is the first 2 second line 3 not last line 4 fourth but...
I have a file.txt containing:
this is the first
second line
not last line
fourth but first
second in list
seventh in file
seventh with nlcat and pipe | it into nl like so:
$> cat file.txt | nl
1 this is the first
2 second line
3 not last line
4 fourth but first
5 second in list
6 seventh in file
7 seventh with nl$> alias_or_function file.txt
1 this is the first
2 second line
3 not last line
1 fourth but first
2 second in list
3 seventh in file
4 seventh with nl~/.zshrc?
ntruter42
(197 rep)
Aug 22, 2020, 04:17 AM
• Last activity: Mar 3, 2025, 07:36 PM
0
votes
3
answers
266
views
How to add an empty line after lines starting with a timestamp
The log viewing experience is so bad in my current company. I want to add new line to the logs that I've trimmed for a specific time duration. I want a new line after every date-time. The date-time format is like this. 2023-10-03 15:34:37 I am aware `tr` can do this but I'm not sure how. I'll select...
The log viewing experience is so bad in my current company. I want to add new line to the logs that I've trimmed for a specific time duration. I want a new line after every date-time.
The date-time format is like this.
2023-10-03 15:34:37
I am aware
tr can do this but I'm not sure how. I'll select the answer best anyone who can tell me how to write that new to a new file. Because of performance reasons. I don't prefer using vi, I use less as a logs reader.
achhainsan
(543 rep)
Oct 10, 2023, 05:16 AM
• Last activity: Feb 26, 2025, 01:33 AM
3
votes
3
answers
649
views
How to remove 'newline' from 'here string'
The 'here string' (` | md5sum` or `printf ... | md5sum` instead. But my question specifically pertains to the 'here string'.
The 'here string' (
| md5sum or printf ... | md5sum instead. But my question specifically pertains to the 'here string'.
Seamus
(3772 rep)
Feb 7, 2025, 05:02 AM
• Last activity: Feb 9, 2025, 12:51 AM
2
votes
2
answers
692
views
Removing more than two consecutive newlines with ed
Input file: 1 line1\ 2 line2\ 3 line3\ 4 \ 5 line4\ 6 \ 7 \ 8 line5\ Desired output: 1 line1 2 line2 3 line3 4 5 line4 6 7 line5 Is it possible with POSIX `ed`? Removing the trailing backslashes is easy, but how can I also handle consecutive blackslashes as expected output?
Input file:
1 line1\
2 line2\
3 line3\
4 \
5 line4\
6 \
7 \
8 line5\
Desired output:
1 line1
2 line2
3 line3
4
5 line4
6
7 line5
Is it possible with POSIX
ed?
Removing the trailing backslashes is easy, but how can I also handle consecutive blackslashes as expected output?
user28512
(31 rep)
Dec 9, 2012, 05:05 PM
• Last activity: Dec 16, 2024, 08:56 AM
60
votes
10
answers
58084
views
Finding all "Non-Binary" files
Is it possible to use the find command to find all the "non-binary" files in a directory? Here's the problem I'm trying to solve. I've received an archive of files from a windows user. This archive contains source code and image files. Our build system doesn't play nice with files that have windows...
Is it possible to use the
find command to find all the "non-binary" files in a directory? Here's the problem I'm trying to solve.
I've received an archive of files from a windows user. This archive contains source code and image files. Our build system doesn't play nice with files that have windows line endings. I have a command line program (flip -u) that will flip line endings between *nix and windows. So, I'd like to do something like this
find . -type f | xargs flip -u
However, if this command is run against an image file, or other binary media file, it will corrupt the file. I realize I could build a list of file extensions and filter with that, but I'd rather have something that's not reliant on me keeping that list up to date.
So, is there a way to find all the non-binary files in a directory tree? Or is there an alternate solution I should consider?
Alana Storm
(1483 rep)
Aug 24, 2012, 06:46 PM
• Last activity: Sep 13, 2024, 09:52 AM
110
votes
11
answers
223756
views
How to test whether a file uses CRLF or LF without modifying it?
I need to periodically run a command that ensures that some _text_ files are kept in Linux mode. Unfortunately `dos2unix` always modifies the file, which would mess file's and folder's timestamps and cause unnecessary writes. The script I write is in Bash, so I'd prefer answers based on Bash.
I need to periodically run a command that ensures that some _text_ files are kept in Linux mode. Unfortunately
dos2unix always modifies the file, which would mess file's and folder's timestamps and cause unnecessary writes.
The script I write is in Bash, so I'd prefer answers based on Bash.
Adam Ryczkowski
(5859 rep)
Jun 17, 2013, 04:34 PM
• Last activity: Jun 10, 2024, 06:35 AM
0
votes
0
answers
71
views
Why do newline characters get lost when concatenating strings ending with newline characters?
Scenario: ``` $ cat xxx.txt | od -c 0000000 x \n x \n 0000004 $ value="" # 1st iteration $ value+="$(<xxx.txt)" ; echo "$value" | od -c 0000000 x \n x \n 0000004 # 2nd iteration $ value+="$(<xxx.txt)" ; echo "$value" | od -c 0000000 x \n x x \n x \n 0000007 ``` At the 2nd iteration we see that `\n`...
Scenario:
$ cat xxx.txt | od -c
0000000 x \n x \n
0000004
$ value=""
# 1st iteration
$ value+="$(\n after the 2nd x is lost. Why? How to prevent this?
The expected output is:
0000000 x \n x \n x \n x \n
0000007
pmor
(665 rep)
May 23, 2024, 10:46 AM
• Last activity: May 23, 2024, 12:18 PM
0
votes
1
answers
331
views
Redirect command output to file (append) and then append a newline character
I'm running the following one-liner: cat input.txt | xargs -I{} bash -c './script.sh "{}"' >> output.txt The script works and does exactly what I want it to do. The only issue is that the output is appended in the file all in a single "line". How can I modify this so that a new line character is als...
I'm running the following one-liner:
cat input.txt | xargs -I{} bash -c './script.sh "{}"' >> output.txt
The script works and does exactly what I want it to do.
The only issue is that the output is appended in the file all in a single "line".
How can I modify this so that a new line character is also appended after each line is processed?
dlyk1988
(103 rep)
May 10, 2024, 12:27 PM
• Last activity: May 10, 2024, 01:30 PM
25
votes
2
answers
127646
views
Carriage return with echo command
I was practicing echo command with option `\r` (carriage return) as below. echo -e "This is \r my college" output: my college but when I add one more word before `\r` as below echo -e "This is the \r my college" Then it gives me output like: my college the Another example echo -e "This is \r valid d...
I was practicing echo command with option
\r (carriage return) as below.
echo -e "This is \r my college"
output:
my college
but when I add one more word before \r as below
echo -e "This is the \r my college"
Then it gives me output like:
my college the
Another example
echo -e "This is \r valid data"
valid data
echo -e "This is not a \r valid data"
valid data a
So, I wanted to know that what is the actual purpose of carriage return here?
Dip
(740 rep)
Jul 1, 2017, 08:12 AM
• Last activity: Mar 29, 2024, 01:41 PM
1
votes
1
answers
133
views
Will transferring an SVN repository dump from Linux to Windows cause EOL problems?
I want to mirror an SVN repository on my local Windows system, but in order to save bandwidth I will have to do an svnrdump onto remote Linux system, zip it up and download it onto my Windows system. Does the fact that the repository was first downloaded to a Linux system then transferrred to Window...
I want to mirror an SVN repository on my local Windows system, but in order to save bandwidth I will have to do an svnrdump onto remote Linux system, zip it up and download it onto my Windows system.
Does the fact that the repository was first downloaded to a Linux system then transferrred to Windows mean there could be end of line conversion issues, as happens with Git systems which are not properly configured?
vfclists
(7909 rep)
Jul 9, 2016, 01:12 PM
• Last activity: Feb 21, 2024, 04:40 PM
14
votes
9
answers
6350
views
Concatenate files placing an empty line between them
I have a bunch of files with the same extension (let's say .txt) and I would like to concatenate them. I am using `cat *.txt > concat.txt`, but I would like to add a new line between each file so to distinguish them in concat.txt. Is it possible to do it with a single bash command rather than an imp...
I have a bunch of files with the same extension (let's say .txt) and I would like to concatenate them. I am using
cat *.txt > concat.txt, but I would like to add a new line between each file so to distinguish them in concat.txt.
Is it possible to do it with a single bash command rather than an implementation such as this ?
Thank you
Gigiux
(557 rep)
Jan 11, 2021, 12:29 PM
• Last activity: Feb 8, 2024, 10:57 AM
Showing page 1 of 20 total questions