Ubuntu HWE kernel showing network poor performance

I have two Ubuntu 22.04.05 LTS server with Mellanox CX-6 nic with 25Gbps link. I found very strange behavior with ipef3 when using HWE kernel. I have default MTU 1500 # uname -a Linux mcc-ceph207-02 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux iperf3 result # iperf3 -c 10.230.35.130 Connecting to host 10.230.35.130, port 5201 [ 5] local 10.230.35.131 port 34384 connected to 10.230.35.130 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.56 GBytes 22.0 Gbits/sec 0 3.08 MBytes [ 5] 1.00-2.00 sec 2.24 GBytes 19.2 Gbits/sec 361 1.55 MBytes [ 5] 2.00-3.00 sec 2.72 GBytes 23.4 Gbits/sec 0 1.57 MBytes [ 5] 3.00-4.00 sec 2.73 GBytes 23.5 Gbits/sec 0 1.81 MBytes [ 5] 4.00-5.00 sec 2.73 GBytes 23.5 Gbits/sec 0 1.86 MBytes [ 5] 5.00-6.00 sec 2.73 GBytes 23.5 Gbits/sec 0 1.86 MBytes [ 5] 6.00-7.00 sec 2.31 GBytes 19.9 Gbits/sec 0 1.98 MBytes [ 5] 7.00-8.00 sec 2.51 GBytes 21.6 Gbits/sec 0 2.03 MBytes [ 5] 8.00-9.00 sec 2.73 GBytes 23.5 Gbits/sec 0 2.03 MBytes [ 5] 9.00-10.00 sec 2.73 GBytes 23.5 Gbits/sec 0 2.03 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 26.0 GBytes 22.3 Gbits/sec 361 sender [ 5] 0.00-10.04 sec 26.0 GBytes 22.2 Gbits/sec receiver Now installed HWE kernel on same box and result is very different # uname -a Linux mcc-ceph207-01 6.8.0-52-generic #53~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jan 15 19:18:46 UTC 2 x86_64 x86_64 x86_64 GNU/Linux iperf3 result # iperf3 -c 10.230.35.130 Connecting to host 10.230.35.130, port 5201 [ 5] local 10.230.35.131 port 44400 connected to 10.230.35.130 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.87 GBytes 16.1 Gbits/sec 472 1.33 MBytes [ 5] 1.00-2.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 2.00-3.00 sec 1.86 GBytes 16.0 Gbits/sec 0 1.33 MBytes [ 5] 3.00-4.00 sec 1.86 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 4.00-5.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 5.00-6.00 sec 1.86 GBytes 16.0 Gbits/sec 0 1.33 MBytes [ 5] 6.00-7.00 sec 1.86 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 7.00-8.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 8.00-9.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.33 MBytes [ 5] 9.00-10.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.33 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 18.6 GBytes 15.9 Gbits/sec 472 sender [ 5] 0.00-10.04 sec 18.6 GBytes 15.9 Gbits/sec receiver