I'm seeing a ton of the following two lines in my dmesg logs¹: [602956.308844] [iptables] (10): IN=eno1 OUT=eno2 MAC=xx:yy:..:zz SRC=10.174.26.245 DST=192.168.22.59 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=53 DPT=47150 WINDOW=28960 RES=0x00 ACK SYN URGP=0 [602956.652575] [iptables] (10): IN=eno1 OUT=eno2 MAC=xx:yy:..:zz SRC=10.172.0.22 DST=192.168.22.59 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=53 DPT=44204 WINDOW=28960 RES=0x00 ACK SYN URGP=0 My firewall blocks those because it does not recognized the 10.172.0.22 and 10.174.26.245 IP addresses. Actually, looking at the eno1 and eno2 lists of IP addresses, those two are not included. I have two 10.x.x.x that I use, but not the two listed above (hence the firewall blocking those two). My network looks something like this: +----------+ +--------+ +--------+ +--------+ | Internet || Router || Server || Laptop | +----------+ +--------+ +--------+ +--------+ Both, the Server and Laptop have firewalls. The Laptop is 192.168.22.59. It is not sent any of those UDP TCP packets. The eno1 and eno2 are on the Server. eno1 connects to the Router which connects to the Internet. The Router connection uses local network addresses (IPv4 & IPv6). eno2 is my local network (LAN). The Server is setup to FORWARD traffic between the Laptop and the Internet. The Laptop uses a VPN and I suspect it could come from that, but the Laptop also has a firewall and thus would ignore such traffic too. What I'm wondering is where those packets are coming from? Would it be a local system or is that coming from some hackers? Or could the VPN be the culprit? Either way, I don't understand how a UDP TCP packet could be using an IP address that is not present on a network interface and if local, I don't see how it could come from the outside. Is there a way to find out what sends those packets, assuming it is a local process that does so? _Side Note: I have libvirt installed, but I tried to stop the one VPN I am running and it does not make any difference. Also, the two bridges it creates do not use those 10.17.x.x IP addresses. Plus, there would be no reasons I can think of for the VPN to send UDP TCP packets to the wrong machine._ ### Update So, I went to my laptop and reconnected the VPN. After that, the two lines above stopped happening. That allowed me to see another line: [608974.298853] [iptables] (192): IN=eno1np0 OUT=eno2np1 MAC=xx:yy:...:zz SRC=192.168.19.2 DST=192.168.22.189 LEN=151 TOS=0x00 PREC=0x00 TTL=63 ID=8281 DF PROTO=UDP SPT=53 DPT=47512 LEN=131 This one **is** UDP, but the point is that, just like with the laptop, it wants data from what looks like a local IP that comes from the router (thus the Internet). Device 189 is my HP printer, so maybe it has a VPN like system too and fails DNS requests once in a while in this manner. ### Resolution I could actually see those two IP in the route table which you can get doing: $ ip route This means my graph would be more like this: +----------+ +-----+ +--------+ +--------+ +--------+ | Internet || VPN || Router || Server || Laptop | +----------+ +-----+ +--------+ +--------+ +--------+ Of course, as mentioned by telcoM, there is also the ISP in between the Router and the VPN, but that is not the culprit. I now DROP those packets without logging them first: -A bad_tcp_packets -i eno1 -s 10.172.0.0/16 -j DROP -A bad_tcp_packets -i eno1 -s 10.174.0.0/16 -j DROP One thing to be noted: it means using a VPN may open a set of _local_ IPs from the other side. So you have to pay attention to such a thing since that could affect your LAN setup. --- ¹ _I setup my firewall to log such accesses to make sure I can see such issues. At the moment, I'm not trying to avoid the log, but to understand it._

i have a single server which two VM(ubuntu) configured as ubuntu-edge(with frr module for routing - wan side) and ubuntu-fw(for lan side), between these machines there is a virtual internal interface. all the traffic form lan toward fw, and from ubuntu-fw with default route forward to ubuntu-edge. when someone try to reach from behind wan side of the ubuntu-edge to lan side(for example icmp or ssh) cant connect, the tcpdump output is here:

ubuntu@ubuntu-fw:~$ sudo tcpdump -n -i lan host 18.x.x.201
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    13:22:43.742256 IP 172.25.4.96 > 18.x.x.201: ICMP echo request, id 60418, seq 4404, length 40
    13:22:43.743299 ARP, Request who-has 172.25.4.96 tell 18.x.x.201, length 46
    13:22:48.861752 ARP, Request who-has 18.x.x.201 tell 18.x.x.1, length 28
    13:22:48.863665 ARP, Reply 18.x.x.201 is-at 74:86:0b:19:fe:c1, length 46
    13:25:42.226903 IP 172.25.4.96 > 18.x.x.201: ICMP echo request, id 60418, seq 4458, length 40
    13:25:42.231842 ARP, Request who-has 172.25.4.96 tell 18.x.x.201, length 46
    13:25:47.236840 IP 172.25.4.96 > 18.x.x.201: ICMP echo request, id 60418, seq 4459, length 40
    13:25:47.237899 ARP, Request who-has 172.25.4.96 tell 18.x.x.201, length 46
    13:25:47.549749 ARP, Request who-has 18.x.x.201 tell 18.x.x.1, length 28

and here is the local route table: ubuntu@ubuntu-fw:~$ ip route show default via 172.24.8.9 dev internal proto static 18.x.x.0/24 dev lan proto kernel scope link src 18.x.x.1 172.24.8.8/30 dev internal proto kernel scope link src 172.24.8.10 the ubuntu-edge, has the route of 172.25.4.96 via bgp dynamic route. what is the problem not forward packet ? by the way packet forwarding is enabled on the both machines. Thanks

I have a Linux box with multiple LTE connections, e.g. wwan0, wwan1, etc., all using the modems' qmi interfaces. If I send dozens of UDP packets, when a modem's antennae are disconnected, they get delivered (many seconds late) when I reconnect the antennae. This shows that the radio is queuing packets. How can I see the size and occupancy of LTE radios' queues from Linux (for LTE modems using the qmi interface)?

**How to mark all packets (inbound and outbound) for specific program/ cmd in Linux using iptables or any other firewall/ tool** Given that --cmd-owner option was deprecated ref:http://www.spinics.net/lists/netfilter/msg49716.html . For example, how to mark all Firefox's packets, knowing that Firefox can spawn processes so the PID option isn't feasible.

I try to measure UDP [%packet loss] between two machines. I'm getting different results while using two different tools: 1. iperf3: machine 1: iperf3 -s machine 2: iperf3 -c -u -b0 result: [ 5] local 172.25.12.25 port 5201 connected to 172.25.12.9 port 46605 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 31.2 MBytes 261 Mbits/sec 0.113 ms 566/4557 (12%) [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 0.113 ms 0/0 (0%) [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0.113 ms 0/0 (0%) [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 0.113 ms 0/0 (0%) [ 5] 4.00-5.00 sec 4.69 MBytes 39.4 Mbits/sec 0.162 ms 65056/65656 (99%) [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0.162 ms 0/0 (0%) [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0.162 ms 0/0 (0%) [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0.162 ms 0/0 (0%) [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0.162 ms 0/0 (0%) [ 5] 9.00-10.00 sec 6.20 MBytes 52.1 Mbits/sec 0.139 ms 64856/65650 (99%) [ 5] 10.00-10.25 sec 0.00 Bytes 0.00 bits/sec 0.139 ms 0/0 (0%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.25 sec 0.00 Bytes 0.00 bits/sec 0.139 ms 130478/135863 (96%) 2. nuttcp: machine 1: nuttcp -S machine 2: nuttcp -u -i -Ri50000m result: 83.5605 MB / 1.00 sec = 700.9434 Mbps 1581 / 87147 ~drop/pkt 1.81 ~%loss 102.1035 MB / 1.00 sec = 856.5107 Mbps 4434 / 108988 ~drop/pkt 4.07 ~%loss 108.7383 MB / 1.00 sec = 912.1637 Mbps 1651 / 112999 ~drop/pkt 1.46 ~%loss 109.0381 MB / 1.00 sec = 914.6668 Mbps 1863 / 113518 ~drop/pkt 1.64 ~%loss 108.7529 MB / 1.00 sec = 912.2939 Mbps 1534 / 112897 ~drop/pkt 1.36 ~%loss 109.3838 MB / 1.00 sec = 917.5759 Mbps 1515 / 113524 ~drop/pkt 1.33 ~%loss 107.4248 MB / 1.00 sec = 901.1491 Mbps 1337 / 111340 ~drop/pkt 1.20 ~%loss 108.1553 MB / 1.00 sec = 907.2559 Mbps 1620 / 112371 ~drop/pkt 1.44 ~%loss 110.3291 MB / 1.00 sec = 925.5187 Mbps 722 / 113699 ~drop/pkt 0.64 ~%loss I get similar bandwidth but different lost percentage. What can I do now? maybe try a third tool?

I have a linux box where we see a lot of (30%) TCP retransmission in tcpdump when receiving (downloading) files from outside. Using the dropwatch utility we see many packet drops in kernel function net_receive_skb(). That means data have been received on the NIC, but later some of them are dropped in the kernel when processing the packets. The many dropped packets can explain the necessity of retransmissions. dropwatch output is like the following: dropwatch -l kas Initalizing kallsyms db dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 1 drops at tcp_rcv_established+906 (0xffffffff814d0a66) 6 drops at unix_dgram_connect+4ac (0xffffffff8151890c) 6 drops at unix_dgram_connect+4ac (0xffffffff8151890c) 19 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 5 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 9 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 7 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 6 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 14 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 15 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 2 drops at __netif_receive_skb+49f (0xffffffff8147b4ef) 2 drops at inet_csk_reset_xmit_timer.clone.1+265 (0xffffffff814d9cb5) ^CGot a stop message dropwatch> exit Shutting down ... The system is a CentOS 6.2 with a 2.6.32 kernel (centOS package name 2.6.32-696.el6.x86_64). So I looked into the version of netif_receive_skb in kernel source code, trying to find the reason for the packet drops. And I see there is only one place calling kfree_skb (near the end of the function) that will leave a trace on dropped packets. The code is: int netif_receive_skb(struct sk_buff *skb) { struct packet_type *ptype, *pt_prev; struct net_device *orig_dev; struct net_device *null_or_orig; int ret = NET_RX_DROP; __be16 type; if (!skb->tstamp.tv64) net_timestamp(skb); if (skb->vlan_tci && vlan_hwaccel_do_receive(skb)) return NET_RX_SUCCESS; /* if we've gotten here through NAPI, check netpoll */ if (netpoll_receive_skb(skb)) return NET_RX_DROP; if (!skb->iif) skb->iif = skb->dev->ifindex; null_or_orig = NULL; orig_dev = skb->dev; if (orig_dev->master) { if (skb_bond_should_drop(skb)) null_or_orig = orig_dev; /* deliver only exact match */ else skb->dev = orig_dev->master; } __get_cpu_var(netdev_rx_stat).total++; skb_reset_network_header(skb); skb_reset_transport_header(skb); skb->mac_len = skb->network_header - skb->mac_header; pt_prev = NULL; rcu_read_lock(); #ifdef CONFIG_NET_CLS_ACT if (skb->tc_verd & TC_NCLS) { skb->tc_verd = CLR_TC_NCLS(skb->tc_verd); goto ncls; } #endif list_for_each_entry_rcu(ptype, &ptype_all, list) { if (ptype->dev == null_or_orig || ptype->dev == skb->dev || ptype->dev == orig_dev) { if (pt_prev) ret = deliver_skb(skb, pt_prev, orig_dev); pt_prev = ptype; } } #ifdef CONFIG_NET_CLS_ACT skb = handle_ing(skb, &pt_prev, &ret, orig_dev); if (!skb) goto out; ncls: #endif skb = handle_bridge(skb, &pt_prev, &ret, orig_dev); if (!skb) goto out; skb = handle_macvlan(skb, &pt_prev, &ret, orig_dev); if (!skb) goto out; type = skb->protocol; list_for_each_entry_rcu(ptype, &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) { if (ptype->type == type && (ptype->dev == null_or_orig || ptype->dev == skb->dev || ptype->dev == orig_dev)) { if (pt_prev) ret = deliver_skb(skb, pt_prev, orig_dev); pt_prev = ptype; } } if (pt_prev) { ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); } else { kfree_skb(skb); /* Jamal, now you will not able to escape explaining * me how you were going to use this. :-) */ ret = NET_RX_DROP; } out: rcu_read_unlock(); return ret; } It looks like the call to kfree_skb will only happen when the skb->dev is not registered in any of the protocol's ptype list, so that the pt_prev remains NULL after the 2 loops against the ptype lists. This does not make sense as the system is dropping only a small part of all packages - meaning the device is "most of time registered in protocol ptype list but sometime not there". So, question is - what mistakes did I make in understanding the dropwatch result and the netif_receive_skb code? And what is a more reasonable explanation of the packet drops being reported at this function?

I want to retrieve raw bytes of DNS response for debugging reason, for example:

dig  -t https clickhouse.com

;; ANSWER SECTION:
clickhouse.com.         242     IN      HTTPS   1 . alpn="h3,h3-29,h2" ipv4hint=172.66.40.249,172.66.43.7 ipv6hint=2606:4700:3108::ac42:28f9,2606:4700:3108::ac42:2b07

How can I get the bytes of those answer section, is there some existing tool that can show me those raw bytes before decoded into plain text? I tried with wireshark but it's encrypted, tried with another dns server but I don't get any response (seems blocked by ISP). **UPDATE**: so I do manual way, create a simple program in Go:

m := &dns.Msg{
		MsgHdr: dns.MsgHdr{
			Authoritative:     false,
			AuthenticatedData: false,
			CheckingDisabled:  true,
			RecursionDesired:  true,
			Opcode:            dns.OpcodeQuery,
		},
		Question: make([]dns.Question, 1),
	}
	q := &m.Question
	q.Qclass = dns.ClassINET
	q.Qtype = dns.TypeHTTPS
	q.Name = "clickhouse.com."

	// 0 = {uint8} 0

	r, err := dns.Exchange(m, "9.9.9.9:9953")

put breakpoint on dns.Answer, off, err = unpackRRslice(int(dh.Ancount), msg, off) on github.com/miekg/dns/msg.go:840 then copy paste the msg to text file.

I connected two Debian 10 systems with a direct link. On one of them I defined a dummy interface and on the other I gave a static route to access it. I see the traffic from the sender on the interface, but I don't see the traffic on the opposite system, and no response is received. Is it possible that the traffic was dropped by the opposite system? sudo ip link add lo1 type dummy sudo ip link set dev lo1 up sudo ip addr add 13.13.13.13/32 dev lo1

pc2~$ ip addr show lo1
   63: lo1:  mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ae:e2:d8:95:2b:ae brd ff:ff:ff:ff:ff:ff
    inet 13.13.13.13/32 brd 13.13.13.13 scope global lo1
       valid_lft forever preferred_lft forever
    inet6 fe80::ace2:d8ff:fe95:2bae/64 scope link 
       valid_lft forever preferred_lft forever


pc2~$ sudo sysctl -p
sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file or directory
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
kernel.printk = 1 4 1 7
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

and another system:

sudo ip route add 13.13.13.13/32 via 20.20.20.2

pc1~$ ip addr show port25
31: port25:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 04:f8:f8:76:53:59 brd ff:ff:ff:ff:ff:ff
    inet 20.20.20.1/24 brd 20.20.20.255 scope global port25
       valid_lft forever preferred_lft forever
    inet6 fe80::6f8:f8ff:fe76:5359/64 scope link 
       valid_lft forever preferred_lft forever

pc1(20.20.20.1)-------(20.20.20.2)pc2(lo2:13.13.13.13/32)

pc1~$ sudo tcpdump -ni port25 icmp

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on port25, link-type EN10MB (Ethernet), snapshot length 262144 bytes




22:54:42.098033 IP 20.20.20.1 > 13.13.13.13: ICMP echo request, id 3321, seq 0, length 64
22:54:43.107142 IP 20.20.20.1 > 13.13.13.13: ICMP echo request, id 3321, seq 1, length 64
22:54:44.117137 IP 20.20.20.1 > 13.13.13.13: ICMP echo request, id 3321, seq 2, length 64
22:54:45.127138 IP 20.20.20.1 > 13.13.13.13: ICMP echo request, id 3321, seq 3, length 64
22:54:46.137143 IP 20.20.20.1 > 13.13.13.13: ICMP echo request, id 3321, seq 4, length 64

pc2:~$ sudo tcpdump -ni port25 icmp
Password: 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on port25, link-type EN10MB (Ethernet), snapshot length 262144 bytes

pc1~$ ip route
default nhid 16 via 192.168.60.2 dev enp0 proto 196 metric 20 
13.13.13.13 nhid 79 via 20.20.20.2 dev port25 proto 196 metric 20 
20.20.20.0/24 dev port25 proto kernel scope link src 20.20.20.1 
50.50.60.60 nhid 16 via 192.168.60.2 dev enp0 proto 196 metric 20 
192.168.60.0/24 dev enp0 proto kernel scope link src 192.168.60.75 
192.168.121.0/24 dev port2 proto kernel scope link src 192.168.121.37 linkdown

pc2~$ ip route
default nhid 26 via 192.168.60.2 dev enp0 proto 196 metric 20 
20.20.20.0/24 dev port25 proto kernel scope link src 20.20.20.2 
192.168.22.0/24 dev port1 proto kernel scope link src 192.168.22.156 
192.168.60.0/24 dev enp0 proto kernel scope link src 192.168.60.76 
zharf-switch:~$

I'm configuring Netfilter Tables to queue packets to and from the userspace, and the table configuration I have so far looks like:

table inet filter {

        # protocols to allow
        set allowed_protocols {
                type inet_proto
                elements = { icmp, icmpv6 }
        }

        # interfaces to accept any traffic on
        set allowed_interfaces {
                type ifname
                elements = { "lo" }
        }

        # services to allow
        set allowed_tcp_dports {
                type inet_service
                elements = { ssh, 9090 }
        }

        # this chain gathers all accept conditions
        chain allow {
                ct state established,related accept

                meta l4proto @allowed_protocols accept
                iifname @allowed_interfaces accept
                tcp dport @allowed_tcp_dports accept
        }

        # base-chain for traffic to this host
        chain INPUT {
                type filter hook input priority filter + 20
                policy accept

                jump allow
                reject with icmpx type port-unreachable
        }

        chain input {
                type filter hook input priority 0;
        }

        chain forward {
                type filter hook forward priority 0;
        }

        chain output {
                type filter hook output priority 0;
        }
}

So far, this seems to load fine with nft -f. However, when I run either of these commands...

nft add inet filter input counter queue num 0

or

nft add inet filter output counter queue num 1

...my VM completely stops responding to input, and when I terminate the connection and vagrant reload, I'm told my VM has to be forcefully shut down before it can reboot. Any help on how I can properly configure these queues would be appreciated! OS: Linux fedora 5.19.8-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 8 19:02:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Vagrant: Vagrant 2.3.0

I transfer a series of test packet with IEC61850_SV protocal in my host as follows: > 01 0C CD 04 00 03 00 0C CD 04 00 00 81 00 80 01 88 BA 40 03 01 24 00 00 00 00 60 82 01 18 80 01 01 A2 82 01 11 30 82 01 0D 80 1B 50 53 53 43 36 30 31 4D 55 53 56 2F 4C 4C 4E 30 24 4D 53 24 4D 53 56 43 42 30 31 82 02 0B C6 83 04 00 00 00 01 85 01 00 87 81 E0 00 00 00 FA 00 00 00 00 FF EC C5 D9 00 00 00 00 FF EC C5 D9 00 00 00 00 00 12 19 12 00 00 00 00 00 12 19 12 00 00 00 00 00 01 21 24 00 00 00 00 00 01 21 24 00 00 00 00 FF EC C5 D9 00 00 00 00 00 12 19 12 00 00 00 00 00 01 21 24 00 00 00 00 FF 90 FD E8 00 00 00 00 FF 90 FD E8 00 00 00 00 00 68 7C DF 00 00 00 00 00 68 7C DF 00 00 00 00 00 06 85 39 00 00 00 00 00 06 85 39 00 00 00 00 FF 90 FD E8 00 00 00 00 FF 90 FD E8 00 00 00 00 00 00 00 0F 00 00 00 00 00 00 00 0F 00 00 00 00 FF 90 FD E8 00 00 00 00 FF 90 FD E8 00 00 00 00 00 68 7C DF 00 00 00 00 00 68 7C DF 00 00 00 00 00 06 85 39 00 00 00 00 00 06 85 39 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Num                                  000001
    Length                               314
    Capture Length                       310
    time difference                      0.000000000 sec
ethernet - II                                [0/14]
    target                               01:0C:CD:04:00:03  [0/6]
    source                               00:0C:CD:04:00:00  [6/6]
    protocal                             0x8100  (VLAN)  [12/2]
802.1Q vlan protocal                        [14/4]
    Label                                [14/2]
    User priority                        100. ....    [14/1]  0x00E0
    CFI                                  ...0 ....    [14/1]  0x0010
    VlAN num                             .... 0000 0000 0001  (1)  [14/2]  0x0FFF
    protocal                             0x88ba  (IEC61850_SV)  [16/2]
IEC61850 sample value                       [18/21]
    Id                                   0x4003  [18/2]
    length                               292  [20/2]
    reserved bit1                        0x0  [22/2]
    reserved bit2                        0x0  [24/2]
    savPdu                              [30/0]
        application service data units                       1  [32/1]
        Sequence ASDU                   [37/0]
            ASDU                        [37/0]
Additional data                                    [39/271]
    bytes                                271 bytes  [39/271]
FCs
    FCs                                  0x2F0CA2EA

Then I use tcpdump in my target machine to capture the packets. The log is shown as follows：

09:06:02.562846 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.574656 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.589684 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.605706 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.620610 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.635996 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.650857 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.665517 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.680210 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
09:06:02.695342 00:0c:cd:04:00:00 (oui Unknown) > 01:0c:cd:04:00:03 (oui Unknown), 802.1Q, length 310: 
10 packets captured
10 packets received by filter
0 packets dropped by kernel
10 packets dropped by interface

It's werid that all vlan protocal packets was dropped by interfaces. Then I use

add eth0 1

to add eth0.1 port, and use eth0.1 to receive the packet, and it shows packets was received properly. Since other packets like ARP will be received by interface properly. I wonder why packets with vlan protocal will be dropped automatically by interface rather than received by interface but can not be resolved by network layer. It sames that data link layer dropped it directly and kernel did not get any data.

cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  eth1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
    lo:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0

Besides adding vlan, is there any way for the kernel to directly receive these packets?

The following rule corrupts 5% of the packets by introducing a single bit error at a random offset in the packet: > sudo tc qdisc change dev ens8 root netem corrupt 5% But recently it gave me the following error: > Error: Qdisc not found. To create specify NLM_F_CREATE flag Could you kindly help me or provide me with some other methods to simulate packet corruption? I'm trying to simulate packet corruption to see how well my error detection mechanism works.

I'm interested to learn how libpcap reads network packets, as I am finding it is not possible when running on AWS Lambda From what I can understand, you need either or both of the CAP_NET_ADMIN and CAP_NET_RAW capabilities which would allow the relevant Linux kernel system calls to read packets from the network device However, I understand if you are operating in a VM environment, you would be reading packets from the physical network device , not the virtual one created inside the VM? This would result in the ability to read packets from _every VM_ running on the same host, which is clearly why the ability is removed from hosts, e.g. Lambda. I'm wondering why libpcap works this way, and why it cannot read packets from the virtual network device instead? Also - are there any other solutions that might work in a heavily isolated environment such as Firecracker MicroVMs?

I'm trying to track down some particular network paths which are slowing down to about 200KByte/sec. I see this performance through various tests including with scp, rsync and iperf3:

$ iperf3 -c 157.130.91.64 -R
Connecting to host 157.130.91.64, port 5201
Reverse mode, remote host 157.130.91.64 is sending
[  5] local 172.16.1.177 port 47862 connected to 157.130.91.64 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   274 KBytes  2.25 Mbits/sec
[  5]   1.00-2.00   sec   199 KBytes  1.63 Mbits/sec
[  5]   2.00-3.00   sec   202 KBytes  1.66 Mbits/sec
[  5]   3.00-4.00   sec   198 KBytes  1.62 Mbits/sec
[  5]   4.00-5.00   sec   195 KBytes  1.60 Mbits/sec
[  5]   5.00-6.00   sec   184 KBytes  1.51 Mbits/sec
[  5]   6.00-7.00   sec   195 KBytes  1.60 Mbits/sec
[  5]   7.00-8.00   sec   209 KBytes  1.71 Mbits/sec
[  5]   8.00-9.00   sec   192 KBytes  1.58 Mbits/sec
[  5]   9.00-10.00  sec   187 KBytes  1.53 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.31 MBytes  1.94 Mbits/sec   65             sender
[  5]   0.00-10.00  sec  1.99 MBytes  1.67 Mbits/sec                  receiver

iperf Done.

The host in question is on a third party hosting provider. And I am downloading data to a co-located data center. I haven't entirely been able to pin down the common element. There are two routers on my side of the network and then a VM host and then a virtual machine. The VM host and the inner-most router are using a VxLAN (i.e. via ip link add vxlan100 type vxlan...), which I suspect is part of the problem. However, I can get 1Gbit speeds (measured with iperf3) directly over the VxLAN to various locations within the rack. I can provide an example but it reads like the above only orders of magnitude faster. The only clue I have at this point is if I capture traffic while this 200KByte/sec transfer is running I do see a higher incidence of TCP retransmissions, TCP out-of-order, and TCP Dup ACK messages in Wireshark. These do seem to correlate with the slow. Captured traffic which runs at much higher speeds also has some TCP retransmissions but much fewer in relation to how much traffic is being sent. My question here is how do I debug this to find the cause of the missing packets? And are there any specific places I should be checking? It seems like there is some degree of packet loss which is causing this slow, but I'm at a loss as to where to try to find it. The packet loss itself does not exhibit at slower speeds, nor does it exhibit within machines within my own datacenter. There seems to be no exact single place where this predictably occurring, only that it definitely occurs between a VM in my datacenter and another machine in another data center. (And also this other machines does have higher transfer rates to other places like AWS, so it's the third party machine, I've checked, it only reproduces when sending to my network). Any ideas?

I am currently sending ~9mpps, each packet is 72bytes large, and is a UDP DNS request. The packets are being sent to a Dell server I have using an intel i40e interface. The server is using all of its 32 queues and has a queue size of 4096 per queue. All client source packets have random source MAC addresses and random IP addresses in a pool of about 150. Note, that none of the network cards show dropped / discarded packets. I can see all the unicast traffic is arriving to the interface ethtool -S shows rx_unicast at ~9mpps. However rx_packets shows 1.7mpps. The egress interface shows the same amount of packets being processed and sent out the other interface I am forwarding through. Note the far end IP the generated traffic arrives at is just a L3 interface, there are no DNS services listening on it, so the packets are dropped on the far end (this is ok). When I use dropwatch I can see the following messages:

224021 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
235277 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
232467 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
227083 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
228235 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
227216 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
221967 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]
225418 drops at kfree_skb+1e (0xffffffffa33f5cbe) [software]

I tried looking up what 0xffffffffa33f5cbe is in /boot/System.map-$(uname -r) but nothing is listed. I tried looking up what kfree_skb does, but this it frees the sk_buff memory. (not very helpful) I believe this maybe a limitation of the kernel (forwarding, netfilter, etc.), or CPU being able to process so much traffic, however I would like to know why / and demonstrate the reason why packets are being dropped in the Kernel. How can I determine why all traffic is not being sent? **UPDATE** I was able to remove the following modules

nf_conntrack
ip_tables
iptables_filter
x_tables

This increased the speed to processing more, I can see now I am able to send ~4.5mpps, it seems that netfilter is the bottle neck. Will work more with removing other modules. Curious if anyone else have details on why this happens?

I have recently installed Linux Mint 20.2 Cinnamon (5.0.4) (Kernel 5.4.0-74-generic) on my desktop on a separate drive from a Windows 10 install. On Linux I noticed very inconsistent connections when trying to download updates (often failing) and slow web page loading. After looking through a few similar questions on StackExchange sites I have checked the following: - The problem persists with both WiFi and Ethernet, and different Ethernet cables - Disabling IPv6 makes no difference - The issue only appears in Linux, not Windows 10 (same machine) - There are **zero** dropped packets when using my tethered phone for internet - When running "ping 8.8.8.8 -c 30" (or 139.130.4.5, or 8.8.4.4) I get pretty consistent behavior where it "switches" on and off so that a few pings in a row return successfully, then the next few have "Destination Host Unreachable" errors (avg ~50% packet loss across multiple runs). Example: $ ping 8.8.8.8 -c 30 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. From 192.168.1.1 icmp_seq=1 Destination Host Unreachable From 192.168.1.1 icmp_seq=2 Destination Host Unreachable From 192.168.1.1 icmp_seq=3 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=4 ttl=113 time=36.8 ms 64 bytes from 8.8.8.8: icmp_seq=5 ttl=113 time=30.2 ms From 192.168.1.1 icmp_seq=6 Destination Host Unreachable From 192.168.1.1 icmp_seq=7 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=8 ttl=113 time=38.9 ms 64 bytes from 8.8.8.8: icmp_seq=9 ttl=113 time=28.9 ms From 192.168.1.1 icmp_seq=10 Destination Host Unreachable From 192.168.1.1 icmp_seq=11 Destination Host Unreachable From 192.168.1.1 icmp_seq=12 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=13 ttl=113 time=38.1 ms 64 bytes from 8.8.8.8: icmp_seq=14 ttl=113 time=40.5 ms From 192.168.1.1 icmp_seq=15 Destination Host Unreachable From 192.168.1.1 icmp_seq=16 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=17 ttl=113 time=32.7 ms 64 bytes from 8.8.8.8: icmp_seq=18 ttl=113 time=31.8 ms From 192.168.1.1 icmp_seq=19 Destination Host Unreachable From 192.168.1.1 icmp_seq=20 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=21 ttl=113 time=39.5 ms 64 bytes from 8.8.8.8: icmp_seq=22 ttl=113 time=34.2 ms From 192.168.1.1 icmp_seq=23 Destination Host Unreachable From 192.168.1.1 icmp_seq=24 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=25 ttl=113 time=51.0 ms 64 bytes from 8.8.8.8: icmp_seq=26 ttl=113 time=30.6 ms From 192.168.1.1 icmp_seq=27 Destination Host Unreachable From 192.168.1.1 icmp_seq=28 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=29 ttl=113 time=48.8 ms 64 bytes from 8.8.8.8: icmp_seq=30 ttl=113 time=35.3 ms --- 8.8.8.8 ping statistics --- 30 packets transmitted, 14 received, +16 errors, 53.3333% packet loss, time 29275ms rtt min/avg/max/mdev = 28.926/36.946/50.983/6.365 ms Additionally, I noticed very similar behavior several months ago when I originally tried setting up a Linux install on a different computer (gave up and stuck with Windows). I thought that it was the physical network adapter since it had driver issues, but the new setup disputes that. It also doesn't seem to be dependent on the Linux version since I had similar issues with regular Ubuntu and Mint installs. All of this is pointing me towards the common factor of the router, which is an Orbi LTE LBR20 running off of a 4G signal (rural internet sucks). **My question is:** what can I do next to confirm the router as the culprit and is there a "simple" fix? Switching to another provider is difficult due to few options at this location and I don't think we're allowed to modify firmware since the router is provided with the internet service. Please let me know if you need more info. Edit 12/30/2021: adding Ethernet adapter info per request in comments: The driver was updated by downloading the latest R8125 driver from the manufacturer's website and installed using two methods: using the autorun.sh provided with the driver and using dkms per the instructions here: https://askubuntu.com/questions/1263363/2-5g-ethernet-linux-driver-r8125-installation-guide Both methods produced the same results. description: Ethernet interface product: RTL8125 2.5GbE Controller vendor: Realtek Semiconductor Co., Ltd. physical id: 0 bus info: pci@0000:2a:00.0 logical name: enp42s0 version: 05 serial: d8:bb:c1:69:fd:c6 size: 100Mbit/s capacity: 1Gbit/s width: 64 bits clock: 33MHz capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=r8125 driverversion=9.007.01-NAPI duplex=full ip=192.168.1.40 latency=0 link=yes multicast=yes port=twisted pair speed=100Mbit/s resources: irq:37 ioport:f000(size=256) memory:fc500000-fc50ffff memory:fc510000-fc513fff Update 1/19/2022: I have been able to "bypass" the issue by using a computer running Windows 10 as a wifi-hotspot, then connecting to the hotspot in Linux. This works reasonably well since I get the expected speeds with good stability, but I'm not fond of chaining connections like this. Does this provide any hints as to why Linux is seeing issues, but not Windows 10? Could the same thing be accomplished with a modem/router instead of another PC?

Sounds dumb, but can you extract application name from a packet/pcap. For eg: If a packet destination is to chrome process, can you extract that information from packet?

I need to merge keytab files and all tutorials mention kutil. But I can't find it: root@nfsserver:/etc# kutil bash: kutil: command not found But krb5-usr is already installed: krb5-user is already the newest version (1.18.3-6+deb11u1). I have to actually merge the keys on two machines, both running Debian, one on Debian 10 and the other on Debian 11. klist for example works; /usr/bin/ contains only kinit, klist and kmod. How do I get the kutil (or how do I merge the two keytab files at least)? Furthermore I'm missing read_kt and write_kt.. Thanks for any help!

I've been having one packet consistently dropping when initiating a flood ping in ubuntu I'll toss out that one with 2 dropped packets as randomness, is there something going on here within the network stack?

I have a trouble configuring iptables to do this: some clients send messages to a server. I want that the host server processes the messages locally (host A, as normal) but that additionally for each message (tcp packet) sends a copy of the packet to another host (host B, that runs a modified version of the server and I want see how is the behaviour with the same messages, so that I can compare both servers). It should be done with iptables. I have tried with the following commands. These send the packet to B but the message is not processed by the host A (should be done by the 2nd command?). iptables -t nat -A PREROUTING -p tcp --dport 31090 -j DNAT --to-destination IP_HOST_B:32090 iptables -t nat -A POSTROUTING -p tcp --dport 32090 -j SNAT --to-source IP_HOST_A:31090 What I've missed in my configuration to accomplish my goal? Thank you.

I have question similar to this one https://unix.stackexchange.com/questions/48402/dpkg-new-pre-installation-script-returned-error-exit-status-1 I'm getting error same as above when trying to install PacketTracer 7.3.1. I think I declined EULA. I know nothing about bash and debconf. Does anyone know how to modify this script?

#!/bin/sh -e


# Source debconf library.
. /usr/share/debconf/confmodule

remove_pt ()
{
if [ -e /opt/pt ]; then
  echo "Removing old version of Packet Tracer from /opt/pt"
  sudo rm -rf /opt/pt
  sudo rm -rf /usr/share/applications/cisco-pt7.desktop
  sudo rm -rf /usr/share/applications/cisco-ptsa7.desktop
  sudo rm -rf /usr/share/icons/hicolor/48x48/apps/pt7.png
fi
}




db_fset PacketTracer_731_amd64/show-eula seen false
db_fset PacketTracer_731_amd64/accept-eula seen false
STATE=1 
while [ "$STATE" != 0 -a "$STATE" != 4 ]; do
    case "$STATE" in
    1)
        db_input critical PacketTracer_731_amd64/show-eula || true
    ;;
    2)
        db_input critical PacketTracer_731_amd64/accept-eula || true
    ;;
    3)
        db_get PacketTracer_731_amd64/accept-eula
        if [ "$RET" = "false" ]; then
            exit 1
        fi
    ;;
    esac

    if db_go; then
        STATE=$(($STATE + 1))
    else
        STATE=$(($STATE - 1))
    fi
done

This is what i got after adding set -x to preinst script and trying to install the packet.

to unpack .../packet_tracer_modified.deb ...
+ . /usr/share/debconf/confmodule
+ [ !  ]
+ PERL_DL_NONLAZY=1
+ export PERL_DL_NONLAZY
+ [  ]
+ exec /usr/share/debconf/frontend /var/lib/dpkg/tmp.ci/preinst install 8.0.0 7.3.1
+ . /usr/share/debconf/confmodule
+ [ ! 1 ]
+ [ -z  ]
+ exec
+ [  ]
+ exec
+ DEBCONF_REDIR=1
+ export DEBCONF_REDIR
+ db_fset PacketTracer_731_amd64/show-eula seen false
+ _db_cmd FSET PacketTracer_731_amd64/show-eula seen false
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n FSET PacketTracer_731_amd64/show-eula seen false
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=false
+ return 0
+ db_fset PacketTracer_731_amd64/accept-eula seen false
+ _db_cmd FSET PacketTracer_731_amd64/accept-eula seen false
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n FSET PacketTracer_731_amd64/accept-eula seen false
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=false
+ return 0
+ STATE=1
+ [ 1 != 0 -a 1 != 4 ]
+ db_input critical PacketTracer_731_amd64/show-eula
+ _db_cmd INPUT critical PacketTracer_731_amd64/show-eula
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n INPUT critical PacketTracer_731_amd64/show-eula
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=question will be asked
+ return 0
+ db_go
+ _db_cmd GO 
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n GO 
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=ok
+ return 0
+ STATE=2
+ [ 2 != 0 -a 2 != 4 ]
+ db_input critical PacketTracer_731_amd64/accept-eula
+ _db_cmd INPUT critical PacketTracer_731_amd64/accept-eula
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n INPUT critical PacketTracer_731_amd64/accept-eula
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=question will be asked
+ return 0
+ db_go
+ _db_cmd GO 
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n GO 
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=ok
+ return 0
+ STATE=3
+ [ 3 != 0 -a 3 != 4 ]
+ db_get PacketTracer_731_amd64/accept-eula
+ _db_cmd GET PacketTracer_731_amd64/accept-eula
+ _db_internal_IFS= 	

+ IFS= 
+ printf %%s\n GET PacketTracer_731_amd64/accept-eula
+ IFS= 	

+ IFS=
 read -r _db_internal_line
+ RET=false
+ return 0
+ [ false = false ]
+ exit 1
dpkg: error processing archive /home/yanaz/Pobrane/packet_tracer_modified.deb (--install):
 new packettracer package pre-installation script subprocess returned error exit status 1
gtk-update-icon-cache: Cache file created successfully.