Unix & Linux Stack Exchange

Q&A for users of Linux, FreeBSD and other Unix-like operating systems

Latest Questions

16 votes

2 answers

5188 views

How to unbuffer cut?

I want to get just e-mail addresses that end in "@xyz.nl" from my mail logfile. To achieve this I do: # tail -f /var/log/mail.log | grep --i --line-buffered "@xyz.nl" | cut -d '@' -f 1 | cut -d ' , relay=123.456.123.456[123.456.123.456]:25, delay=2, delays=0.4/0/0.4/1.2, dsn=2.0.0, status=sent (250...

                                  I want to get just e-mail addresses that end in "@xyz.nl" from my mail logfile. To achieve this I do:

    # tail -f /var/log/mail.log | grep --i --line-buffered "@xyz.nl" | cut -d '@' -f 1 | cut -d ', relay=123.456.123.456[123.456.123.456]:25, delay=2, delays=0.4/0/0.4/1.2, dsn=2.0.0, status=sent (250 2.0.0 u7T9twxN074009 Message accepted for delivery)

The first cut then makes:

    Aug 29 11:56:01 localhost postfix/smtp: 05491500123: to=

Forkbeard (657 rep)

Sep 2, 2016, 06:53 AM • Last activity: Jul 1, 2025, 12:45 PM

1 votes

3 answers

2251 views

Bash awk/sed extract multiple strings from a single large string using keywords

awk sed cut

Please help me figure out how to extract multiple substrings based on keywords. I have been struggling with different methods using delimiters My Input: ``` Inventory for 30844-ap01 NAME: AP1800 , DESCR: Cisco Aironet 1800 Series (IEEE 802.11ac) Access Point PID: AIR-AP1832I-E-K9, VID: V03, SN: KWC2...

Please help me figure out how to extract multiple substrings based on keywords. I have been struggling with different methods using delimiters My Input:

Inventory for 30844-ap01 NAME: AP1800 , DESCR: Cisco Aironet 1800 Series (IEEE 802.11ac) Access Point PID: AIR-AP1832I-E-K9, VID: V03, SN: KWC21420CKU
Inventory for ckh.hq-ap99 NAME: AP2700 , DESCR: Cisco Aironet 2700 Series (IEEE 802.11n) Access Point PID: AIR-CAP2702I-E-K9, VID: V03, SN: FCW2007N0ZQ
Inventory for AP0042.6843.ab78 NAME:  , DESCR:  PID: AIR-CAP1702I-E-K9, VID: V, SN: FCZ201622NY

Desired output:

30844-ap01 AIR-AP1832I-E-K9 KWC21420CKU
ckh.hq-ap99 AIR-CAP2702I-E-K9 FCW2007N0ZQ
AP0042.6843.ab78 AIR-CAP1702I-E-K9 FCZ201622NY

First string is anything between "Inventory for " and the next space Second string is anything between "PID: " and the coma Third string is an 11 char string after "SN: "

Gunsmoke (11 rep)

May 28, 2021, 09:49 PM • Last activity: May 16, 2025, 09:08 PM

13 votes

5 answers

10564 views

Dynamically trim stdout line width in Bash

bash shell stdout cut

Lately, I have been experimenting with the `ps` command, and sometimes long paths wrap to the next line (or two) and make it hard to read. I want to pipe the `ps` output into another program to limit the output to `x` number of characters. Here is what I have so far, but it doesn't work quite right:...

                                  Lately, I have been experimenting with the ps command, and sometimes long paths wrap to the next line (or two) and make it hard to read. I want to pipe the ps output into another program to limit the output to x number of characters.

Here is what I have so far, but it doesn't work quite right:

    ps aux | cut -c1-$(stty size | cut -d' ' -f2)

$(stty size | cut -d' ' -f2) evaluates to 167, but doesn't seem to be valid input for cut.

Is there a way to get this type of syntax to work in bash?

lentils (133 rep)

Apr 18, 2014, 04:16 AM • Last activity: Apr 20, 2025, 05:00 PM

27 votes

9 answers

3944 views

Is piping, shifting, or parameter expansion more efficient?

shell-script pipe performance cut

I'm trying to find the most efficient way to iterate through certain values that are a consistent number of values away from each other in a space separated list of words(I don't want to use an array). For example, list="1 ant bat 5 cat dingo 6 emu fish 9 gecko hare 15 i j" So I want to be able to j...

                                  I'm trying to find the most efficient way to iterate through certain values that are a consistent number of values away from each other in a space separated list of words(I don't want to use an array). For example,

    list="1 ant bat 5 cat dingo 6 emu fish 9 gecko hare 15 i j"
So I want to be able to just iterate through list and only access 1,5,6,9 and 15.

***EDIT:*** I should have made it clear that the values I'm trying to get from the list don't have to be different in format from the rest of the list. What makes them special is solely their position in the list(In this case, position 1,4,7...). So the list could be1 2 3 5 9 8 6 90 84 9 3 2 15 75 55 but I'd still want the same numbers. And also, I want to be able to do it assuming I don't know the length of the list. 

The methods I've thought of so far are:

**Method 1**

    set $list
    found=false
    find=9
    count=1
    while [ $count -lt $# ]; do
        if [ "${@:count:1}" -eq $find ]; then
        found=true
        break
        fi
        count=expr $count + 3
    done
**Method 2**

    set list
    found=false
    find=9
    while [ $# ne 0 ]; do
        if [ $1 -eq $find ]; then
        found=true
        break
        fi
        shift 3
    done
**Method 3**
I'm pretty sure piping makes this the worst option, but I was trying to find a method that doesn't use set, out of curiosity.

    found=false
    find=9
    count=1
    num=echo $list | cut -d ' ' -f$count
    while [ -n "$num" ]; do
        if [ $num -eq $find ]; then
        found=true
        break
        fi
        count=expr $count + 3
        num=echo $list | cut -d ' ' -f$count
    done

----------
So what would be most efficient, or am I missing a simpler method?
                                

Levi Uzodike (458 rep)

Jan 31, 2019, 07:10 PM • Last activity: Mar 8, 2025, 08:24 AM

10 votes

8 answers

40764 views

grep first n and last n characters from a line in a file

text-processing awk sed grep cut

I have a log file which looks like: Mar 23 08:20:23 New file got created in sec: 235 Mar 23 08:21:45 New file got created in sec: 127 Mar 23 08:22:34 New file got created in sec: 875 Mar 23 08:25:46 New file got created in sec: 322 Mar 23 08:26:12 New file got created in sec: 639 I need the output t...

                                  I have a log file which looks like:

    Mar 23 08:20:23 New file got created in sec: 235
    Mar 23 08:21:45 New file got created in sec: 127
    Mar 23 08:22:34 New file got created in sec: 875
    Mar 23 08:25:46 New file got created in sec: 322
    Mar 23 08:26:12 New file got created in sec: 639

I need the output to look like:

    Mar 23 08:20:23 : 235
    Mar 23 08:21:45 : 127
    Mar 23 08:22:34 : 875
    Mar 23 08:25:46 : 322
    Mar 23 08:26:12 : 639

What I am able to do is just grep either first part or the last part of the line. I am not able to put the two together. How can I get the desired output from my input?

                                

IAmNoob (209 rep)

Mar 24, 2017, 03:42 AM • Last activity: Dec 12, 2024, 11:36 PM

3 votes

3 answers

358 views

Large file manipulation

text-processing awk csv cut

I have a Garmin Nuvi which uses OpenStreet maps. Garmin do postcodes, but is usually 2-3 years out of date for Scotland. OSM does not do British postcodes, but the Post office does, and it can be downloaded for free. This file is just under 1GB. It has 16 columns, of which I only want the first 3. I...

                                  I have a Garmin Nuvi which uses OpenStreet maps. Garmin do postcodes, but is usually 2-3 years out of date for Scotland. OSM does not do British postcodes, but the Post office does, and it can be downloaded for free. This file is just under 1GB. It has 16 columns, of which I only want the first 3.

I used cut to remove the extraneous columns, so I now have Postcode, Latitude and Longitude. Unfortunately the POI file is to be Latitude, Longitude and Postcode, i.e. column 1 is to be column 3. To add to the problem, the postcode must be in quotes e.g EH9 1QG and SW12 1AB has to be "EH9 1QG" and "SW12 1AB".

I used awk rather awkwardly (see what I did there?) with:

    awk 'BEGIN {FS="\t"; OFS=","} {print $2, $3, $1}' pc0.csv > pc.csv
and all it did was add 2 empty columns to the front.  
It would be nice to be able to use a spreadsheet on it but there are over 3 million rows.

Any ideas?  

----

This is what I get from the output of cut - pc0.csv.   
The awk command gives the same but with two commas at the front of each row to give 2 empty columns.  

    Postcode Latitude Longitude
    AB1 0AA,57.101474,-2.242851  
    AB1 0AB,57.102554,-2.246308  
    AB1 0AD,57.100556,-2.248342  
    AB1 0AE,57.084444,-2.255708  
    AB1 0AF,57.096656,-2.258102  
    AB1 0AG,57.097085,-2.267513  
    AB1 0AJ,57.099011,-2.252854  
    AB1 0AL,57.101765,-2.254688

So using the "cut" file above, which is now only 73MB, I need to convert it to:

    Latitude,Longitude,Postcode
    57.101474,-2.242851,"AB1 0AA"  
    57.102554,-2.246308,"AB1 0AB"  
    57.100556,-2.248342,"AB1 0AD"  
    57.084444,-2.255708,"AB1 0AE"  
    57.096656,-2.258102,"AB1 0AF"  
    57.097085,-2.267513,"AB1 0AG"  
    57.099011,-2.252854,"AB1 0AJ"  
    57.101765,-2.254688,"AB1 0AL"

Now I had to remove the tabs to display these lines, so that is another problem as there can only be commas and nothing else - not even spaces unless inside quotes.  

P.S. Linux (Ubuntu Mate) 22.04 LTS  


                                

user256787 (49 rep)

Aug 29, 2024, 07:21 PM • Last activity: Sep 16, 2024, 11:20 AM

0 votes

1 answers

108 views

Cut command failing

shell-script ksh cut

We have a ksh Script that is suppose to give the output as last day of the month but when ever we run it we get the below error: ``` cut: fields are numbered from 1 Try ‘cut —help’ for more information, ``` I have been trying to debug this but unable to do so. Here is the code: ``` #!/usr/bin/ksh if...

We have a ksh Script that is suppose to give the output as last day of the month but when ever we run it we get the below error:

cut: fields are numbered from 1
Try ‘cut —help’ for more information,

I have been trying to debug this but unable to do so. Here is the code:

#!/usr/bin/ksh

if (( $(#)  /dev/null | wc - l)
if (( whencecal != 1 )) ; then print "FATAL: \"cal\" command not found!" ; exit 102 ; fi

typeset -i wrdcount=$(cal ${month} ${year} | sed -e "/^$/ d" | tail -1 | wc -w)
typeset -Z2 momaxdate=$(cal ${month} ${year} | sed -e “/^$/ d” -e “s/[ ][ ]*/,/g” | tail -1 | cut -d”,” -f${wrdcount})

print ${momaxdate}

What can be done to fix this

ZyPH3R (3 rep)

Aug 20, 2024, 03:01 PM • Last activity: Aug 21, 2024, 07:54 PM

212 votes

10 answers

189267 views

How do I use cut to separate by multiple whitespace?

text-processing columns cut

I have this input, which is displayed in columns. I would like to get the second last column with the numbers of this sample: [ 3] 1.0- 2.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 2.0- 3.0 sec 768 KBytes 6.29 Mbits/sec [ 3] 3.0- 4.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 4.0- 5.0 sec 256 KBytes 2.10 Mbits/s...

                                  I have this input, which is displayed in columns. I would like to get the second last column with the numbers of this sample:


    [  3]  1.0- 2.0 sec  1.00 MBytes  8.39 Mbits/sec
    [  3]  2.0- 3.0 sec   768 KBytes  6.29 Mbits/sec
    [  3]  3.0- 4.0 sec   512 KBytes  4.19 Mbits/sec
    [  3]  4.0- 5.0 sec   256 KBytes  2.10 Mbits/sec
    ...

If I use 

    cut -d\  -f 13

I get 

    Mbits/sec
    6.29
    4.19
    2.10

because sometimes there are additional spaces in between.

                                

rubo77 (30435 rep)

Jan 17, 2014, 11:54 PM • Last activity: May 18, 2024, 09:05 PM

3 votes

5 answers

24788 views

How to use cut command to get the first and last elements of a row?

text-processing ksh cut

I've asked almost [the same question already][1], but this time, I want to retrieve the X **latest** elements of a row of a CSV file. For example, with an input file as this one: 1;foo;bar;baz;x;y;z 2;foo;bar;baz;x;y;z 3;foo;bar;baz;x;y;z What would be the command (eventually using `cut`) to get the...

                                  I've asked almost the same question already , but this time, I want to retrieve the X **latest** elements of a row of a CSV file. For example, with an input file as this one:

    1;foo;bar;baz;x;y;z
    2;foo;bar;baz;x;y;z
    3;foo;bar;baz;x;y;z

What would be the command (eventually using cut) to get the last 2 columns, so I get:

    y;z
    y;z
    y;z

In fact, my real target is to retrieve the first 3 **and** the last 2 fields of each row, so I get:

    1;foo;bar;y;z
    2;foo;bar;y;z
    3;foo;bar;y;z

Unfortunately, I cannot use a command like cut -d \; -f 1-3,10-11 (if there are 11 elements in the row), because the CSV file does not respect the *real* CSV format. Indeed, some fields in the middle of the rows are *encrypted*, and their encrypted value may sometimes contains a ; characters (and of course, they are not wrapped inside "). In others words, my lines may look like that:

    1;foo;bar;#@$"é&^l#;baz;x;y;z
    2;foo;bar;#¤=é;)o'#;baz;x;y;z
    3;foo;bar;#]]'~é{{#;baz;x;y;z

and as you can see, on the second line, there is an additional ; character, so I can't use here a command like cut -d \; -f 1-3,7-8, because if will return that, which is wrong:

    1;foo;bar;y;z
    2;foo;bar;x;y  (-> Wrong here, there is a shift)
    3;foo;bar;y;z

So how can I use cut to solve my problem?

Thanks

ps: I am specially in love with the cut command, so if you have a command that does what I want but that is not cut, then it's fine too :)

*Edit* It seems important to note that the machine is quite old: uname -a give this message:

    SunOS ###### 5.10 Generic_142900-05 sun4u sparc SUNW,Sun-Fire-V240

and some commands may not be present (like rev)

Romain Linsolas (235 rep)

Jan 7, 2013, 02:32 PM • Last activity: Mar 20, 2024, 05:14 AM

1 votes

1 answers

68 views

How do I format the output of cut to the following format?

text-formatting cut

last part of the command: ``` | cut -d' ' -f3 ``` output: ``` 4 10 ``` required format: ``` 4,10 ``` I'm guessing `echo` (and `sort`?) is to be used in some way, but I can't quite place my finger on it.

last part of the command:

| cut -d' ' -f3

output:

4
10

required format:

4,10

I'm guessing echo (and sort?) is to be used in some way, but I can't quite place my finger on it.

kesarling (159 rep)

Mar 4, 2024, 11:12 PM • Last activity: Mar 4, 2024, 11:17 PM

1 votes

0 answers

33 views

pfSense (FreeBSD) - tail -f not showing entire log when filtering with cut or sed

ubuntu sed cut tail pfsense

I have a strange problem when trying to display logs on pfSense (and I can reproduce the same problem on Ubuntu server also). The problem is this (with examples): I'm trying to display a running `dhcp` log with `tail -f`. This works without problem when I just use it, like this: $ tail -n 48 -f /var...

                                  I have a strange problem when trying to display logs on pfSense (and I can reproduce the same problem on Ubuntu server also).

The problem is this (with examples):

I'm trying to display a running dhcp log with tail -f. This works without problem when I just use it, like this:

    $ tail -n 48 -f /var/log/dhcpd.log 

    Feb  4 10:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0x2cbc27bf: lease 10.10.1.62 has been allocated for 7200 seconds
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path
    Feb  4 10:59:03 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 10:59:03 vault dhclient: DHCPACK from 147.78.28.48 
    Feb  4 10:59:03 vault dhclient: RENEW
    Feb  4 10:59:03 vault dhclient: Creating resolv.conf
    Feb  4 10:59:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 dc:a6:32:9a:15:72], cid=[ff:92:39:3b:55:00:02:00:00:ab:11:ac:47:9e:3e:13:09:39:5f], tid=0x656556fb: lease 10.10.2.4 has been allocated for 7200 seconds
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 04:d4:c4:76:16:b5], cid=[01:04:d4:c4:76:16:b5], tid=0x18d711b8: lease 10.10.2.22 has been allocated for 7200 seconds
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d058d0000] EVAL_RESULT Expression pool_opt5_0 evaluated to 1 
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 3c:ec:ef:06:a1:c1], cid=[01:3c:ec:ef:06:a1:c1], tid=0x26a022f: lease 10.10.2.1 has been allocated for 7200 seconds
    Feb  4 11:29:04 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67 
    Feb  4 11:29:04 vault dhclient: DHCPACK from 147.78.28.48  
    Feb  4 11:29:04 vault dhclient: RENEW
    Feb  4 11:29:04 vault dhclient: Creating resolv.conf  
    Feb  4 11:29:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05818900] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05818900] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05818900] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05818900] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0xac0108b7: lease 10.10.1.62 has been allocated for 7200 seconds
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path
    Feb  4 11:59:04 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67 
    Feb  4 11:59:04 vault dhclient: DHCPACK from 147.78.28.48   
    Feb  4 11:59:04 vault dhclient: RENEW  
    Feb  4 11:59:04 vault dhclient: Creating resolv.conf
    Feb  4 11:59:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 12:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 12:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 12:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 12:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 dc:a6:32:9a:15:72], cid=[ff:92:39:3b:55:00:02:00:00:ab:11:ac:47:9e:3e:13:09:39:5f], tid=0x656556fb: lease 10.10.2.4 has been allocated for 7200 seconds 
    Feb  4 12:03:44 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 12:03:44 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 12:03:44 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 12:03:44 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 04:d4:c4:76:16:b5], cid=[01:04:d4:c4:76:16:b5], tid=0x6ba6e88e: lease 10.10.2.22 has been allocated for 7200 seconds
    Feb  4 12:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_lan_0 evaluated to 1
    Feb  4 12:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt4_0 evaluated to 1
    Feb  4 12:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05817400] EVAL_RESULT Expression pool_opt5_0 evaluated to 1
    Feb  4 12:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 3c:ec:ef:06:a1:c1], cid=[01:3c:ec:ef:06:a1:c1], tid=0x26a022f: lease 10.10.2.1 has been allocated for 7200 seconds

I'll then filter the results with grep, which still works as expected:

    $ tail -n 48 -f /var/log/dhcpd.log | grep -v 'Expression pool'

    Feb  4 10:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0x2cbc27bf: lease 10.10.1.62 has been allocated for 7200 seconds
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path
    Feb  4 10:59:03 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 10:59:03 vault dhclient: DHCPACK from 147.78.28.48
    Feb  4 10:59:03 vault dhclient: RENEW
    Feb  4 10:59:03 vault dhclient: Creating resolv.conf
    Feb  4 10:59:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 dc:a6:32:9a:15:72], cid=[ff:92:39:3b:55:00:02:00:00:ab:11:ac:47:9e:3e:13:09:39:5f], tid=0x656556fb: lease 10.10.2.4 has been allocated for 7200 seconds
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 04:d4:c4:76:16:b5], cid=[01:04:d4:c4:76:16:b5], tid=0x18d711b8: lease 10.10.2.22 has been allocated for 7200 seconds
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 3c:ec:ef:06:a1:c1], cid=[01:3c:ec:ef:06:a1:c1], tid=0x26a022f: lease 10.10.2.1 has been allocated for 7200 seconds
    Feb  4 11:29:04 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 11:29:04 vault dhclient: DHCPACK from 147.78.28.48
    Feb  4 11:29:04 vault dhclient: RENEW
    Feb  4 11:29:04 vault dhclient: Creating resolv.conf
    Feb  4 11:29:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05818900] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0xac0108b7: lease 10.10.1.62 has been allocated for 7200 seconds
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path
    Feb  4 11:59:04 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 11:59:04 vault dhclient: DHCPACK from 147.78.28.48
    Feb  4 11:59:04 vault dhclient: RENEW
    Feb  4 11:59:04 vault dhclient: Creating resolv.conf
    Feb  4 11:59:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 12:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 dc:a6:32:9a:15:72], cid=[ff:92:39:3b:55:00:02:00:00:ab:11:ac:47:9e:3e:13:09:39:5f], tid=0x656556fb: lease 10.10.2.4 has been allocated for 7200 seconds
    Feb  4 12:03:44 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 04:d4:c4:76:16:b5], cid=[01:04:d4:c4:76:16:b5], tid=0x6ba6e88e: lease 10.10.2.22 has been allocated for 7200 seconds
    Feb  4 12:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 3c:ec:ef:06:a1:c1], cid=[01:3c:ec:ef:06:a1:c1], tid=0x26a022f: lease 10.10.2.1 has been allocated for 7200 seconds

Finally, I want to cut the column width with cut (or make string manipulation with sed), but then the problems begin. As you can see from the output, this results in the latest line of the log not being included.

    $ tail -n 48 -f /var/log/dhcpd.log | grep -v 'Expression pool' | cut -c -223
    
    Feb  4 10:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05817400] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0x2cbc27bf: lease 10.10.1.62 has been allocated for 7200 second
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 10:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp
    Feb  4 10:59:03 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 10:59:03 vault dhclient: DHCPACK from 147.78.28.48
    Feb  4 10:59:03 vault dhclient: RENEW
    Feb  4 10:59:03 vault dhclient: Creating resolv.conf
    Feb  4 10:59:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:00:53 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 dc:a6:32:9a:15:72], cid=[ff:92:39:3b:55:00:02:00:00:ab:11:ac:47:9e:3e:13:09:39:5f], tid=0x656556fb: lease 10.10.2.4
    Feb  4 11:03:43 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 04:d4:c4:76:16:b5], cid=[01:04:d4:c4:76:16:b5], tid=0x18d711b8: lease 10.10.2.22 has been allocated for 7200 second
    Feb  4 11:06:14 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d058d0000] DHCP4_LEASE_ALLOC [hwtype=1 3c:ec:ef:06:a1:c1], cid=[01:3c:ec:ef:06:a1:c1], tid=0x26a022f: lease 10.10.2.1 has been allocated for 7200 seconds
    Feb  4 11:29:04 vault dhclient: DHCPREQUEST on igc2 to 147.78.28.48 port 67
    Feb  4 11:29:04 vault dhclient: DHCPACK from 147.78.28.48
    Feb  4 11:29:04 vault dhclient: RENEW
    Feb  4 11:29:04 vault dhclient: Creating resolv.conf
    Feb  4 11:29:04 vault dhclient: bound to XX.XX.XX.XX -- renewal in 1800 seconds.
    Feb  4 11:43:01 vault kea-dhcp4: INFO  [kea-dhcp4.leases.0x313d05818900] DHCP4_LEASE_ALLOC [hwtype=1 dc:e5:5b:91:28:97], cid=[01:dc:e5:5b:91:28:97], tid=0xac0108b7: lease 10.10.1.62 has been allocated for 7200 second
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
    Feb  4 11:51:54 vault kea-dhcp4: INFO  [kea-dhcp4.dhcpsrv.0x313d05812000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp

As you can see, the timestamps with cut stop with 11:51:54, while the latest entry is in fact 12:06:14. This is completely reproducible, but it differs how much of the log tail is not included.

Also, this is both tested on pfSense and Ubuntu, with cut and sed, and I get the same behavior of the entire log tail not getting included most of the time.

Can anyone explain what's happening here? And any suggestions for a working method to do this? I would like to follow the log tail, while at the same time being able to filter the output with cut and/or sed, and be sure that the latest lines are always included.
                                

Artur Meinild (792 rep)

Feb 4, 2024, 11:21 AM • Last activity: Feb 4, 2024, 11:32 AM

0 votes

2 answers

68 views

Unexpected cut output

cut

I am getting an unexpected output from cut. I would expect no output since there is no third field based on the specified delimiter. The first two lines intentionally use commas as delimiters, and the third uses both commas and semicolons. I know how cut generally works, but I was trying to show tha...

Fred,Jones,sophomore
Mary,Adams,freshman
Sam,Fredrick;senior

cut command:

cut -f3 -d ';' students.txt

unexpected output:

Fred,Jones,sophomore
Mary,Adams,freshman

ayeieye (1 rep)

Jan 26, 2024, 08:51 PM • Last activity: Jan 26, 2024, 09:21 PM

0 votes

1 answers

216 views

cut -d – Delimiter is still shown in the output. How to remove it simply?

cut

Why the delimiter is still shown in the output? I thought field 2 and 4 would be shown without the `-` delimiter. How to use `cut` command to not show at the same time the delimiter itself used to separate fields? ```bash $ curl -s -m 2 https://am.i.mullvad.net/json | jq '.mullvad_exit_ip_hostname'...

Why the delimiter is still shown in the output? I thought field 2 and 4 would be shown without the - delimiter. How to use cut command to not show at the same time the delimiter itself used to separate fields?

$ curl -s -m 2 https://am.i.mullvad.net/json  | jq '.mullvad_exit_ip_hostname'
"nl-ams-wg-002"

$ curl -s -m 2 https://am.i.mullvad.net/json  | jq '.mullvad_exit_ip_hostname' | sd '\"' '' | cut -d '-' -f 2,4
ams-002

Faxopita (179 rep)

Dec 12, 2023, 04:01 PM • Last activity: Dec 12, 2023, 04:28 PM

0 votes

3 answers

100 views

How do I change the extension of a long filename quickly?

shell cut mv

original file: `a really really long name with spaces.ext` target file: `the same name.new-ext` command: `mv *part of the name that'll give a unique answer.ext $(echo $(ls -l | grep -i *.ext | cut -d ' ' -f 11-))` result: `mv: target 'last word of the name.ext': No such file or directory` What am I...

                                  original file: a really really long name with spaces.ext  
target file: the same name.new-ext  

command: mv *part of the name that'll give a unique answer.ext $(echo $(ls -l | grep -i *.ext | cut -d ' ' -f 11-))

result: mv: target 'last word of the name.ext': No such file or directory

What am I doing wrong?

PS: the remote machine only allows sh (not even bash) so tab-completion is out of the picture.

kesarling (159 rep)

Dec 10, 2023, 07:24 PM • Last activity: Dec 12, 2023, 05:58 AM

4 votes

1 answers

911 views

Cut command stored in variable will not work

bash shell-script cut

Can someone explain why I cannot store the cut command in a variable?? ``` user:~$ echo "foo bii doo" | cut -d ' ' -f 2 bii # as expected # now put cut command into a variable: user:~$ c="cut -d ' ' -f 2" user:~$ echo "foo bii doo" | $c cut: "'": No such file or directory user:~$ c="cut -d \ -f 2" u...

Can someone explain why I cannot store the cut command in a variable??

user:~$ echo "foo bii doo" | cut -d ' ' -f 2
bii # as expected
# now put cut command into a variable:
user:~$ c="cut -d ' ' -f 2"
user:~$ echo "foo bii doo" | $c
cut: "'": No such file or directory
user:~$ c="cut -d \  -f 2"
user:~$ echo "foo bii doo" | $c
foo bii doo

It is not a general quoting problem because tr works:

user:~$ t="tr -d 'oo'"
user:~$ echo "foo bii doo" | $t
f bii d

(I am on Debian 12 using bash) Solved the problem in other ways already. I ask just to know why I wasted 3 hours debugging :)

archygriswald (105 rep)

Nov 26, 2023, 11:48 AM • Last activity: Nov 26, 2023, 01:20 PM

4 votes

4 answers

20234 views

Use `cut` to extract a list from /etc/passwd

bash text-processing cut

How could I use the `cut` command to extract a list of usernames and login shells from the `/etc/passwd` file, where the resulting usernames and login shells are separated by a single space? Sort the resulting list in ascending alphabetical order, using the login shell as the primary key, and the us...

                                  How could I use the cut command to extract a list of usernames and login shells from the /etc/passwd file, where the resulting usernames and login shells are separated by a single space? 

Sort the resulting list in ascending alphabetical order, using the login shell as the primary key, and the username as a secondary key. Store the result in the newly created file ~/usershells.txt. 
 
This is the command I attempted:

    cut -d /etc/passwd -k7 -k1 | sort > ~/usershells.txt

user72510 (513 rep)

Jul 15, 2014, 09:31 PM • Last activity: Nov 23, 2023, 10:19 AM

-1 votes

2 answers

183 views

Grep a log file for SQL queries and their execution time

linux awk sed grep cut

I have an application log file that also captures the execution time for each executed SQL statement, like below:- **Sample logs:** ``` I 2023-11-15 07:18:00.743Z 1760463 37 ZVRwqItu6aw-b8eejMLI_gAAAAU 1@45077318@A PHP Request Summary: 7 warnings/errors on 6 lines I 2023-11-15 07:17:15.927Z 1773299...

I have an application log file that also captures the execution time for each executed SQL statement, like below:- **Sample logs:**

I 2023-11-15 07:18:00.743Z 1760463 37 ZVRwqItu6aw-b8eejMLI_gAAAAU 1@45077318@A PHP Request Summary: 7 warnings/errors on 6 lines
I 2023-11-15 07:17:15.927Z 1773299 99 ZVRwenUWDwrXl_9NnC-2vwAAAAM 1@45077318@A There is no cache key assigned for the current PID
I 2023-11-15 07:17:55.778Z 1773397 24 ZVRwoydG6PTd3KWR_MnfVAAAABU - Initiating db proxy socket to dev01:8080
I 2023-11-15 07:17:55.865Z 1773397 33 ZVRwoydG6PTd3KWR_MnfVAAAABU 1@45077318@A execStmtEx: time: 0.002 / 0 rows: 1 sql: 
I 2023-11-15 07:17:56.185Z 1773397 65 ZVRwoydG6PTd3KWR_MnfVAAAABU 1@45077318@A applicationConnection TRX Record Commit (execStmtEx) time: 0
I 2023-11-15 07:17:57.515Z 1755787 26 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.001 / 0 rows: 1 sql: 
I 2023-11-15 07:17:57.519Z 1755787 26 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.001 / 0 rows: 0 sql: 
I 2023-11-15 07:17:57.529Z 1755787 27 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.002 / 0 rows: 1 sql: 
I 2023-11-15 07:17:57.531Z 1755787 27 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.001 / 0 rows: 0 
I 2023-11-15 07:17:57.535Z 1755787 28 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.002 / 0 rows: 1 sql: 
I 2023-11-15 07:17:57.542Z 1755787 28 ZVRwpA5grkn1knR4gk6FMwAAABA 1@45077318@A queryResults: time: 0.002 / 0 rows: 1 sql: 
...

I want to grep for all these SQL queries along with their execution times, sorted in descending order(based on their execution times). **Desired output:**

time: 0.002 | sql: 
time: 0.002 | sql:

How can I achieve this with Linux commands? Could someone please help me to achieve this? Thanks in advance.

Arpit Jain (109 rep)

Nov 15, 2023, 08:23 AM • Last activity: Nov 15, 2023, 10:41 AM

1 votes

2 answers

219 views

How do I backslash-ignore a delimiter passed into cut?

cut

I have the following use case: ``` echo "some comment char '\;' embedded in strings ; along with inline comments" \ | cut -d';' -f 1 ``` I want: ``` some comment char ';' embedded in strings ``` I get: ``` some comment char ' ``` ---- How do I hide the delimiter configured for cut from cut, as in th...

I have the following use case:

echo "some comment char '\;' embedded in strings   ; along with inline comments" \
| cut -d';' -f 1

I want:

some comment char ';' embedded in strings

I get:

some comment char '

---- How do I hide the delimiter configured for cut from cut, as in this use case? Ideally, cut would read and respect the backslash, but if not that way, is there another?

Chris (1075 rep)

Jul 21, 2023, 08:30 PM • Last activity: Nov 7, 2023, 01:47 PM

1 votes

4 answers

548 views

Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs

text-processing awk cut bioinformatics

I have a `tsv` file with tab-separated columns. I want to obtain the 5th column, which has space-separated values. Convert the space-separation to tab-separation and save as a new file. Attempt: cut -d"\t" -f"4" input.tsv awk -v OFS="\t" '$1=$1' input.tsv > output.tsv Input: Composite_Element_REF Ge...

                                  I have a tsv file with tab-separated columns. I want to obtain the 5th column, which has space-separated values. Convert the space-separation to tab-separation and save as a new file.

Attempt:

    cut -d"\t" -f"4" input.tsv
    awk -v OFS="\t" '$1=$1' input.tsv > output.tsv


Input:

    Composite_Element_REF   Gene_Symbol     Chromosome      Genomic_Coordinate      TCGA-KL-8323-01A-21D-2312-05 TCGA-KL-8324-01A-11D-2312-05 TCGA-KL-8325-01A-11D-2312-05 
    cg00000027      RBL0    14      53468110        0.0545368833399913 0.635089208882213 0.0581022991274144
    cg00000028      RBL1    15      53468111        0.0545366588241415 0.635089205024173 0.0581085373336217
    cg00000029      RBL2    16      53468112        0.0545366588040571 0.635089205078394 0.0581085373332275

Expected output:

    Composite_Element_REF   TCGA-KL-8323-01A-21D-2312-05    TCGA-KL-8324-01A-11D-2312-05    TCGA-KL-8325-01A-11D-2312-05
    cg00000027     0.0545368833399913    0.635089208882213    0.0581022991274144
    cg00000028    0.0545366588241415    0.635089205024173    0.0581085373336217
    cg00000029    0.0545366588040571    0.635089205078394    0.0581085373332275
                                

Anon (133 rep)

Nov 2, 2023, 09:30 PM • Last activity: Nov 5, 2023, 06:47 PM

1 votes

4 answers

344 views

How to get first Character of every underscore delimiter using bash

bash text-processing cut

Hi Team there is a scenario where table name such as ``` head_must_report customer_report_sub_division country_code_universe_nation_part ``` However using cut command output should be as below. Note: table name would be underscore delimiter. ``` HMR CRSD CCUNP ``` using bash script

Hi Team there is a scenario where table name such as

head_must_report

customer_report_sub_division

country_code_universe_nation_part

However using cut command output should be as below. Note: table name would be underscore delimiter.

HMR

CRSD

CCUNP

using bash script

Gyari (15 rep)

Apr 11, 2022, 02:18 PM • Last activity: Sep 13, 2023, 09:18 PM

Showing page 1 of 20 total questions