libvirt kvm virtual routed network: cannot ping gateway itself or beyond

I'm having trouble with libvirt kvm's routed networks where a VM inside a routed virtual network can ping every VM in my home subnet except the default gateway... or any gateway for that matter. I believe my situation is not a duplicate of https://unix.stackexchange.com/questions/611580/libvirt-kvm-virtualhost-cannot-ping-router-address as I am able to ping machines in the host's subnet (unlike in the aforementioned post where the author says no other hosts in the 192.168.2.0/24 network can be pinged) ### The environment - my home subnet is 192.168.1.0/24 - The default gateway is your standard consumer router, located at 192.168.1.1. Its static route table is as follows: - dst: 192.168.100.0 - gw: 192.168.1.100 - genmask: 255.255.255.0 - dst: 172.16.0.0 - gw: 192.168.1.4 - genmask: 255.240.0.0 - My machine (**server-01**, runs EndeavourOS) has the following IP address 192.168.1.100 - I am running libvirt kvm, I have a virtual *routed* network for my VMs. As such server-01 is a router for the 192.168.100.0/24 subnet. - On that subnet there is a Debian 12 VM **vm-alpha** (192.168.100.200) - A test subnet 172.20.200.0/24 exists (default gw 172.20.200.1). It can be reached from the home subnet via 192.168.1.4 (OPNsense) - There's a lonely Debian 12 VM **vm-beta** whose IP is 172.20.200.2 ### The problem I am unable to ping the default gw from **vm-alpha**. Hence vm-alpha has no "Internet connection". Below is a table outlining the pings I have attempted, with empty cells representing those that I have not tried | from =>
to ↓ | **vm-alpha**
(192.168.100.200) | server-01 home net IP (192.168.1.100) | vm-beta
(172.20.200.2) | | ---------------------------------------- | --------------------------------- | ------------------------------------- | ------------------------- | | **vm-alpha**
(192.168.100.200) | | YES | YES | | server-01 gw
(192.168.100.1) | YES | YES | YES | | server-01 home net IP
(192.168.1.100) | YES | | YES | | OPNsense home net
(192.168.1.4) | YES | YES | YES | | default gw home net
(192.168.1.1) | no response | YES | YES | | OPNsense other side
(172.20.200.1) | no response | YES | YES | | vm-beta
(172.20.200.2) | no response | YES | | | Cloudflare
(1.1.1.1) | no response | YES | YES | - vm-alpha (192.168.100.200) to home net gateway (192.168.1.1) - **no response** - packets are simply lost, there is no "destination host unreachable" or "network is unreachable" errors that I could see with wireshark running on server-01, listening on interface virbr1. #### mtr mtr from 192.168.100.200 to 172.20.200.2

My traceroute  [v0.95]
vm-alpha (192.168.100.200) -> 172.20.200.2025-03-30T00:00:24+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   qui
t                             Packets               Pings
 Host                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.100.1            0.0%     7    0.2   0.2   0.1   0.3   0.1
 2. (waiting for reply)

^ when that mtr was done, the routing table of the host (server-01) was as follows:

Kernel IP routing table  
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface  
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 enp9s0  
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp9s0  
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr1

mtr from 172.20.200.2 to 192.168.100.200

My traceroute  [v0.95]
vm-beta (172.20.200.2) -> 12025-03-29T22:59:07+0000
Keys:  Help   Display mode   Restart statistics   Order of field
s   quit               Packets               Pings
 Host                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.20.200.1      0.0%    16    0.5   0.6   0.3   1.1   0.2
 2. 192.168.1.1       0.0%    15    1.3   1.1   0.9   1.4   0.1
 3. 192.168.1.100     0.0%    15    1.1   1.2   0.7   1.9   0.3
 4. 192.168.100.200   0.0%    15    1.3   1.3   0.8   1.7   0.2

route -n for 172.20.200.2:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.20.200.1    0.0.0.0         UG    0      0        0 eth0
172.20.200.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

### What I've tried - disabling firewalld which was running - the problem remains and the results from the ping table above still stand. - changing the routes on server-01 - I thought maybe the 192.168.1.1 gw was the problem so I removed it completely. I manually set the route for the 172.20.200.0/24 subnet. With the routing table of server-01 now being:

Kernel IP routing table  
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface  
172.16.0.0      192.168.1.4     255.240.0.0     UG    0      0        0 enp9s0  
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp9s0  
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr1

server-01 is able to ping 172.20.200.2. However, vm-alpha still can't. mtr shows the new hop:

My traceroute  [v0.95]
vm-alpha (192.168.100.200) -> 172.20.200.2025-03-30T00:15:08+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   qui
t                             Packets               Pings
 Host                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.100.1            0.0%     4    0.1   0.2   0.1   0.2   0.1
 2. 192.168.1.4              0.0%     4    0.7   0.7   0.5   0.9   0.1
 3. (waiting for reply)

### Other info iptables -L:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
FWD-vm-alpha  all  --  anywhere             192.168.100.200     
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FWD-vm-alpha (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             192.168.100.200      tcp dpts:tcpmux:65535
ACCEPT     udp  --  anywhere             192.168.100.200      udp dpts:tcpmux:65535

cat /proc/sys/vm/ipv4/ip_forward returns 1. ip -c a on server-01:

(...)
2: enp9s0:  mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:ff:e0:3d:ab:c5 brd ff:ff:ff:ff:ff:ff
    altname enx10ffe03dabc5
    inet 192.168.1.100/24 brd 192.168.1.255 scope global dynamic noprefixroute enp9s0
       valid_lft 83966sec preferred_lft 83966sec
    inet6 fe80::6f46:313c:4159:fde4/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
(...)
11: virbr1:  mtu 1500 qdisc htb state UP group default qlen 1000
    link/ether 52:54:00:0d:2c:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr1
       valid_lft forever preferred_lft forever

firewalld runs on the system. Some config info:

sudo firewall-cmd --get-active-zones 

libvirt
  interfaces: virbr0
libvirt-routed
  interfaces: virbr1
public (default)
  interfaces: enp9s0

sudo firewall-cmd --list-all --zone=libvirt-routed

libvirt-routed (active)
  target: default
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: virbr1
  sources: 
  services: 
  ports: 
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Virtual network configuration:

routed-network-100
  a4b18fac-87e8-4f6c-aebd-04ca31b8c7f7