Unclear delays pinging over IPsec VPN

What could be the cause if every second ping is delayed for a second when pinging through the host-to-network VPN while at the same time pinging the security gateway outside the VPN takes 13 ms on average and no more than 20 ms maximum, even when using largest possible ping -s 1464 ? (Larger pings result in icmp_seq=1 Frag needed and DF set (mtu = 1492).)

bytes from 192.168.178.1: icmp_seq=2300 ttl=64 time=1031 ms
64 bytes from 192.168.178.1: icmp_seq=2301 ttl=64 time=17.6 ms
64 bytes from 192.168.178.1: icmp_seq=2302 ttl=64 time=1034 ms
64 bytes from 192.168.178.1: icmp_seq=2303 ttl=64 time=19.8 ms
64 bytes from 192.168.178.1: icmp_seq=2304 ttl=64 time=1032 ms
64 bytes from 192.168.178.1: icmp_seq=2305 ttl=64 time=18.6 ms
64 bytes from 192.168.178.1: icmp_seq=2306 ttl=64 time=803 ms
64 bytes from 192.168.178.1: icmp_seq=2307 ttl=64 time=19.6 ms
64 bytes from 192.168.178.1: icmp_seq=2308 ttl=64 time=455 ms
64 bytes from 192.168.178.1: icmp_seq=2309 ttl=64 time=19.9 ms
64 bytes from 192.168.178.1: icmp_seq=2310 ttl=64 time=18.4 ms
64 bytes from 192.168.178.1: icmp_seq=2311 ttl=64 time=1052 ms
64 bytes from 192.168.178.1: icmp_seq=2312 ttl=64 time=1019 ms
64 bytes from 192.168.178.1: icmp_seq=2313 ttl=64 time=18.2 ms
64 bytes from 192.168.178.1: icmp_seq=2314 ttl=64 time=1023 ms

That is getting worse with even slightly larger packets:

ping -s 234 192.168.178.1
242 bytes from 192.168.178.1: icmp_seq=8 ttl=64 time=23.7 ms
242 bytes from 192.168.178.1: icmp_seq=9 ttl=64 time=18.6 ms
242 bytes from 192.168.178.1: icmp_seq=10 ttl=64 time=2060 ms
242 bytes from 192.168.178.1: icmp_seq=11 ttl=64 time=1043 ms
242 bytes from 192.168.178.1: icmp_seq=12 ttl=64 time=1024 ms
242 bytes from 192.168.178.1: icmp_seq=13 ttl=64 time=41.2 ms
242 bytes from 192.168.178.1: icmp_seq=14 ttl=64 time=2047 ms
242 bytes from 192.168.178.1: icmp_seq=15 ttl=64 time=1042 ms
242 bytes from 192.168.178.1: icmp_seq=16 ttl=64 time=2034 ms
242 bytes from 192.168.178.1: icmp_seq=17 ttl=64 time=1032 ms
242 bytes from 192.168.178.1: icmp_seq=18 ttl=64 time=18.7 ms

Is it possible that there are packet losses on the network in the protocols under IPsec while ICMP ping goes through undisturbed ? If yes, how to debug these ? Edit: Could this have been introduced by the recent fix by AVM for a yet undisclosed vulnerability in all kinds of Fritz-Box ? Client is vpnc 0.5.3r550-3.1 on LUbuntu 20.04, Security Gateway is Fritz-Box 7530 with up to date software. Rebooting these did not help. The same client software and hardware did work with the same gateway hardware on ADSL without these delays. Switching to the current fibre-optic network seems to have caused the problems, but there might also have been a software update on the Fritz-Box causing the problems.