hostapd: slow performance unless android phone connects

I have an host computer, set up with hostapd as access point and dnsmasq for DHCP server. I'm running under ArchLinux, with Linux kernel 6.1 (as I'm on an embedded ARM board which only supports up to this kernel - all the rest is freshly up-to-date). I'm using a 802.11 Wi-Fi dongle: 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter This host broadcasts through the AP messages to a set of 4 devices (ESP32s controlling various hardware devices) at a 20 milliseconds interval (by sending to 192.168.1.255). The devices send a heartbeat message to the host every second. iwconfig reports: wlan0 IEEE 802.11 Mode:Master Tx-Power=20 dBm Retry short limit:7 RTS thr=2347 B Fragment thr:off Power Management:off My Linux cmdline has: power_save=0 usbcore.autosuspend=-1 The behaviour I observe is: if I just have these four devices connected to the system, then there's a 8 - 10 second latency between the time the message gets sent (observed through wireshark) and the time I observe the message being received. Also, receiving is very erratic and jittery. The mystery part is: the very moment I connect my Android phone to the host AP, then suddenly everything becomes instantaneous, no more jitter, latency or performance issues. It goes back to being bad the moment I disconnect my phone. What could be the cause of this, and how can I make sure that my setup always perform optimally without having to let my phone in the network?