L4 balancing using ipvs: drop RST packets - failover

I have a L4 ipvs load balancer with L7 envoy balancers setup. Let's say one of my L4 balancers goes down and thanks to consistent hashing the traffic which is now handled (thanks to BGP) by another L4 balancer is proxied to the same L7 node. This should work without any problems and I would think is a common setup. Problem is with long-running connections. When new L4 node receives the traffic (just data - ACK/PUSH packets) and no SYN packet has been received by the node, the node just sends RST packet to the client which terminates the connection. Picture below illustrates this. This should not be happening and my question is, is there a way (a sysctl config or something) which is the reason for this? I know I can perhaps drop RST packets using iptables, but that doesn't sound right.