How to troubleshoot random latency spikes in WiFi traffic?

I am building an embedded device that runs Linux (Armbian) and receives a steady stream of UDP data. It has both Ethernet and WiFi connectivity. Ethernet works fine, however when using WiFi, the data stream will occasionally appear to suddenly pause for up to a second, no data is received during this period. The packets are not lost, however, they seem to just be queued in the network driver stack somewhere. And after the pause is up, the entire backlog is dumped on the application. This would be fine if I had a reasonable buffer to work with, but my stream is very time-sensitive, I cannot afford jitter more than 50ms or so. So these pauses lead to big buffer underruns in my application. The delay is at the receiver, earlier than userspace, since tshark running there shows a gap in packets when it happens. But this gap is not present in wireshark on the client PC. It is hard to tell whether the spikes correlate with activity, since activity is very evenly distributed on the device, there is nothing else of note running on there than my application. It is not correlated with time, it seems random. Where can I begin to troubleshoot this?