RHEL 8 IP/Kernel Routing Multi-Homed Server Issue - Cannot get a response to ping, when trying to ping from 2nd Interface

**Set up/configuration:** I have a RHEL 8 server, running Asterisk 15.x, that has 2 NICs. NMCLI is used for networking NIC0 (eno5np0) is on the trusted network and is configured as a static IPv4 and NIC1 (ens1f0) is on the untrusted side as a DHCP IPv4. Both are UP,BROADCAST,RUNNING,MULTICAST NIC0 is where I access the server from, is an internal network and has an IP of 10.38.149.244/32 (GW is 10.38.149.241) NIC1 is supposed to allow access to the internet (for SIP calling) and has an IP of 10.0.0.91 (GW is 10.0.0.1) Firewall status - inactive(dead) SE Linux status - disabled Server #1 interface configs: TYPE=Ethernet DEVICE=eno5np0 UUID=77c33e7a-7dba-4785-b749-dc0883b46cef ONBOOT=yes IPADDR=10.38.149.244 NETMASK=255.255.255.240 GATEWAY=10.38.149.241 NM_CONTROLLED=yes BOOTPROTO=none DOMAIN=comcast.net DNS1=69.252.80.80 DNS2=69.252.81.81 DEFROUTE=yes USERCTL=no IPV4_FAILURE_FATAL=yes TYPE=Ethernet BOOTPROTO=dhcp NM_CONTROLLED=yes PEERDNS=no DEFROUTE=no NAME=ens1f0 UUID=249b95f0-d490-4402-b654-43695317d738 DEVICE=ens1f0 ONBOOT=yes PROXY_METHOD=none BROWSER_ONLY=no IPV4_FAILURE_FATAL=no IPV6_DISABLED=yes IPV6INIT=no IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no **Kernel IP routing table:** | Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface | | :----------- | :------- | :------- | :----- | :------ | :--- | :--- | :------| | 0.0.0.0 | 10.38.149.241 | 0.0.0.0 | UG | 100 | 0 | 0 | eno5np0 | | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | U | 101 | 0 | 0 | ens1f0 | | 10.38.149.240 | 0.0.0.0 | 255.255.255.240 | U | 100 | 0 | 0 | eno5np0 | I do not have any nft tables/IP tables configured I am SSH'd to the 10.38.149.244 interface (NIC0, aka eno5np0), have sudo access I run the following command for NIC0: sudo traceroute -i eno5np0 8.8.8.8 and get a nice, completed trace to 8.8.8.8 I run the following command for NIC1: sudo traceroute -i ens1f0 8.8.8.8 and it times out, no packets received I cannot ping/traceroute to any ip address through NIC1 (sudo ping -I and sudo traceroute -i) except 10.0.0.1, which is the gateway. It is almost like if it isn't the gateway the packets are not making it back into the server for processing? **Issue/Problem** So, after trying both ping and traceroute and not receiving a response, I opened a second SSH session to the server and did a tcpdump while running a ping to 8.8.8.8 over the NIC1 interface in my first SSH session: **TCP Dump** sudo tcpdump -vv --interface ens1f0 -c 10 dropped privs to tcpdump tcpdump: listening on ens1f0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:21:09.450739 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120 hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms rdnss option (25), length 40 (5): lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net 0x0000: 0000 0000 00b4 2001 0558 feed 0000 0000 0x0010: 0000 0000 0001 2001 0558 feed 0000 0000 0x0020: 0000 0000 0002 prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s 0x0000: 40c0 0000 012c 0000 012c 0000 0000 2601 0x0010: 0000 0200 0080 0000 0000 0000 0000 route info option (24), length 24 (3): ::/0, pref=medium, lifetime=180s 0x0000: 0000 0000 00b4 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 source link-address option (1), length 8 (1): 10:56:11:86:6e:92 0x0000: 1056 1186 6e92 15:21:10.415419 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28 15:21:11.439570 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28 15:21:12.453262 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120 hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms rdnss option (25), length 40 (5): lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net 0x0000: 0000 0000 00b4 2001 0558 feed 0000 0000 0x0010: 0000 0000 0001 2001 0558 feed 0000 0000 0x0020: 0000 0000 0002 prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s 0x0000: 40c0 0000 012c 0000 012c 0000 0000 2601 0x0010: 0000 0200 0080 0000 0000 0000 0000 route info option (24), length 24 (3): ::/0, pref=medium, lifetime=180s 0x0000: 0000 0000 00b4 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 source link-address option (1), length 8 (1): 10:56:11:86:6e:92 0x0000: 1056 1186 6e92 15:21:12.463417 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28 15:21:13.487416 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28 15:21:13.546246 IP (tos 0x0, ttl 4, id 8382, offset 0, flags [DF], proto UDP (17), length 219) 169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 191 15:21:13.546273 IP (tos 0x0, ttl 4, id 8383, offset 0, flags [DF], proto UDP (17), length 223) 169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 195 15:21:13.546320 IP (tos 0x0, ttl 4, id 8384, offset 0, flags [DF], proto UDP (17), length 227) 169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 199 15:21:13.546419 IP (tos 0x0, ttl 4, id 8385, offset 0, flags [DF], proto UDP (17), length 220) 169.254.100.1.50759 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 192 10 packets captured 10 packets received by filter 0 packets dropped by kernel I am not understanding why, if the server is doing an ARP request, am I not getting a response? Is the issue on my server not knowing how to respond back to NIC0 with my ping request (where I am SSH'd into)? Is it the gateway being misconfigured? Do I need a NFT table/IP Table configured? I am familiar with how to do this in RHEL 6.x, but not in RHEL 8 (configuration using IP route and IP tables was simpler I think?) At the end of the day (for a broader picture) - I have Softphone clients to register to the Asterisk PBX on the internal/trusted network coming in over NIC0 (which works). They need to make phone calls to endpoints on the Internet, but only over NIC1 - and right now I cannot even ping to any location on the internet over the NIC1 interface. Any help/guidance would be very much appreciated at this point - I am lost and desperate. **Edit/additional clarification:** I have a RHEL 6.x server, with exact same physical connections and NICs that this does work on. I have tried to use the iptable and routing table from this Server #2 on Server #1 above and it will not work (I get booted when I turn the interface back up, and have to reboot the device to clear out any unsaved changes before I can get back in) I did use the iptables to nft translate function just as an FYI. I have plugged my Server #1 NIC1 into the known good modem/internet access port that Server #2 is using and still no change. Server #2 interface configs: DEVICE=eth0 BOOTPROTO=none NM_CONTROLLED=yes ONBOOT=yes TYPE=Ethernet UUID="da71293d-4351-481e-a794-bc5850e29391" IPADDR=10.38.149.243 DNS1=10.168.241.223 DOMAIN=comcast.net DEFROUTE=no IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0" #HWADDR=00:1C:23:CF:BC:E3 HWADDR=00:1c:23:cf:bc:e3 NETMASK=255.255.255.240 USERCTL=no PEERDNS=yes GATEWAY=10.38.149.241 DEVICE=eth1 BOOTPROTO=dhcp HWADDR=00:1c:23:cf:bc:e5 NM_CONTROLLED=yes ONBOOT=yes DEFROUTE=yes TYPE=Ethernet UUID="78bc69cb-80ca-41d1-af9c-66703eb952d5" USERCTL=no PEERDNS=yes IPV6INIT=no **Kernel Routing Table on Server #2** | Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface | | :----------- | :------- | :------- | :----- | :------ | :--- | :--- | :------| | 0.0.0.0 | 10.0.0.1 | 255.255.255.255 | UGH | 0 | 0 | 0 | eth1 | | 10.38.149.240 | 0.0.0.0 | 255.255.255.240 | U | 0 | 0 | 0 | eth0 | | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | eth1 | | 10.0.0.0 | 10.38.149.241 | 255.0.0.0 | UG | 0 | 0 | 0 | eth0 | | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | UG | 0 | 0 | 0 | eth1 | iptables -L on Server #2 **Chain INPUT (policy ACCEPT)** | target | prot | opt | source | destination | status? | | :------| :----| :----| :-----| :-----------| :-------| | DROP | all | -- | c-67-164-235-175.devivce1.mi.inetprovider.net | anywhere | | | DROP | all | -- | c-67-164-235-175.devivce1.mi.inetprovider.net | anywhere | | | ACCEPT | all | -- | anywhere | anywhere | | | ACCEPT | all | -- | anywhere | anywhere | state RELATED,ESTABLISHED | | ACCEPT | tcp | -- | anywhere | anywhere | tcp dpt:ssh | | ACCEPT | udp | -- | anywhere | anywhere | udp dpt:sip | | ACCEPT | udp | -- | anywhere | anywhere | udp dpts:ndmp:dnp | | DROP | all | -- | 106.0.0.0/8 | anywhere | | | DROP | all | -- | 106.0.0.0/8 | anywhere | | | DROP | all | -- | host-87-0-0-0.retail.blockeddomain.notus/8 | anywhere | | | DROP | all | -- | 113.0.0.0/8 | anywhere | | | DROP | all | -- | 117.0.0.0/8 | anywhere | | | DROP | all | -- | p5b000000.dip0.blockeddomain.notus/8 | anywhere | | **Chain FORWARD (policy ACCEPT)** | target | prot | opt | source | destination | | :------| :----| :---| :------| :-----------| | ACCEPT | all | -- | anywhere | anywhere | **Chain OUTPUT (policy ACCEPT)** | target | prot | opt | source | destination | | :------| :----| :---| :------| :-----------|