Shell script to process latest log files and count submitted vs. not submitted entries
1
vote
5
answers
2275
views
A log file gets generated every minute in a directory called
data_logs
.
Log file names:
abc.log.2019041607
abc.log.2019041608
...
Contents of the log file look like this:
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:1
R_MT|D:1234|ID:413|S:0
R_MT|D:1234|ID:413|S:0
R_MT|D:1234|ID:413|S:0
R_MT|D:1234|ID:413|S:0
R_MT|D:1234|ID:413|S:0
k_MT|D:1234|ID:414|S:1
k_MT|D:1234|ID:414|S:1
k_MT|D:1235|ID:413|S:1
k_MT|D:1235|ID:413|S:1
I am writing a shell script which, when executed, looks for the files that were created in the last 5 minutes (1 file gets created every minute), opens each file one by one, and processes the contents. It creates an output.txt
file with the following structure:
For the combination R_MT|D:1234|ID:413
, the total count with S=0
is stored in the submitted
column, and S=1
is stored in the notsubmitted
column.
Expected output.txt
:
Type,Number,ID,submitted,notsubmitted
R_MT,D:1234,ID:413,5,10
R_MT,D:1234,ID:414,0,2
R_MT,D:1235,ID:413,0,2
I have used this command to get the submitted and notsubmitted values:
zcat abc.log.2019041607.gz | grep "R_MT" | awk -F"|" '{print $2","$3","$4}' | sort | uniq -c
Sample output:
5 D:1234,ID:413,S:0
10 D:1234,ID:413,S:1
2 D:1234,ID:414,S:1
2 D:1235,ID:413,S:1
With the above command I am getting the count, but I am not sure how to assign it to variables so I can write them into the submitted
and notsubmitted
fields in the output file. Also, I am not sure how to obtain the last 5 minutes' files.
Asked by user365760
(11 rep)
Aug 6, 2019, 11:54 AM
Last activity: Jun 22, 2025, 10:04 PM
Last activity: Jun 22, 2025, 10:04 PM