IPIP tunnel between two docker containers on separate servers

Thank you in advance for your attention to my question and your help. I have a rather specific task. I need to set up an IPIP tunnel between two Docker containers located on different physical servers. The network type for the containers is bridge. The IP of the first physical server (cluster01): 10.130.0.10 The IP of the second physical server (cluster02): 10.129.0.16 I created the ipip0 interface in the first container:

3: ipip0@NONE:  mtu 1430 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 172.21.0.2 peer 10.129.0.16
    inet 10.3.0.1/24 scope global ipip0
       valid_lft forever preferred_lft forever
    inet6 fe80::5efe:ac15:2/64 scope link 
       valid_lft forever preferred_lft forever

In the second container:

3: ipip0@NONE:  mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 172.23.0.3 peer 10.130.0.10
    inet 10.3.0.2/24 scope global ipip0
       valid_lft forever preferred_lft forever
    inet6 fe80::5efe:ac17:3/64 scope link 
       valid_lft forever preferred_lft forever

Next, I set up DNAT on both servers for packets with protocol 4 (IPIP): On cluster01: iptables -t nat -I PREROUTING -p 4 -d 10.130.0.10 -j DNAT --to-destination 172.21.0.2 On cluster02: iptables -t nat -I PREROUTING -p 4 -d 10.129.0.16 -j DNAT --to-destination 172.23.0.3 172.21.0.2 and 172.23.0.3 are the IPs of the Docker containers. When I try to ping from the first container to the second, there is no response:

bash-5.1# ping 10.3.0.2
PING 10.3.0.2 (10.3.0.2): 56 data bytes
^C
--- 10.3.0.2 ping statistics ---
94 packets transmitted, 0 packets received, 100% packet loss

However, on cluster02, I can see IPIP packets:

cluster02:~# tcpdump -i any -nn proto 4
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:38:07.255829 eth0  In  IP 10.130.0.10 > 10.129.0.16: IP 10.3.0.1 > 10.3.0.2: ICMP echo request, id 56, seq 0, length 64
15:38:08.255317 eth0  In  IP 10.130.0.10 > 10.129.0.16: IP 10.3.0.1 > 10.3.0.2: ICMP echo request, id 56, seq 1, length 64
15:38:09.255415 eth0  In  IP 10.130.0.10 > 10.129.0.16: IP 10.3.0.1 > 10.3.0.2: ICMP echo request, id 56, seq 2, length 64
15:38:10.255605 eth0  In  IP 10.130.0.10 > 10.129.0.16: IP 10.3.0.1 > 10.3.0.2: ICMP echo request, id 56, seq 3, length 64

However, there are no packets visible in the second container through tcpdump:

# tcpdump -i any -nn not tcp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

The packets from the iptables rule on cluster02 are not captured:

cluster02:~# iptables -t nat -L PREROUTING -n -v
Chain PREROUTING (policy ACCEPT 432 packets, 20869 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       4    --  *      *       0.0.0.0/0            10.129.0.16          to:172.23.0.3
  999 54889 DOCKER     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Both servers are running Ubuntu 24.04 /proc/sys/net/ipv4/ip_forward is set to 1 Firewall is disabled with the command ufw disable What I have tried: 1) Adding the rule iptables -t raw -A PREROUTING -p 4 -d 10.129.0.16 -j NOTRACK 2) Marking packets and doing policy routing

iptables -t mangle -I PREROUTING -p 4 -d 10.129.0.16 -j MARK --set-mark 1
echo "100 ipip_route" >> /etc/iproute2/rt_tables
ip rule add fwmark 1 table ipip_route
ip route add default dev docker0 table ipip_route

But at least the packets are captured by the rule:

cluster02:~# iptables -t mangle -L PREROUTING -n -v
Chain PREROUTING (policy ACCEPT 5714 packets, 538K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  386 40144 MARK       4    --  *      *       0.0.0.0/0            10.129.0.16          MARK set 0x1

This didn't help – the packets still don't reach the second Docker container.