Why would USB video be dropping frames in Realtime Linux?

We have a problem of losing about one frame every 60 seconds or so with four USB cameras hooked up to Ubuntu 20.04 with the Realtime Linux patches applied. From the user code ioctl(VIDIOC_DQBUF) call level we see that v4l2_buffer.sequence skips a buffer, but with no error reported. What makes it odd is that one camera doesn't skip, but three do, even though they are all on separate USB ports. Looking at the kernel debug information we see info like this:

Jan 21 08:48:52 kernel: [ 612.290354] uvcvideo: frame 1955 stats: 0/151/151 packets, 0/0/151 pts (!early initial), 0/151 scr, last pts/stc/sof 0/0/0
Jan 21 08:48:52 kernel: [ 612.291017] uvcvideo: frame 1940 stats: 0/151/151 packets, 0/0/151 pts (!early initial), 0/151 scr, last pts/stc/sof 0/0/0
Jan 21 08:48:52 kernel: [ 612.294264] uvcvideo: frame 1956 stats: 0/9/9 packets, 0/0/9 pts (!early initial), 0/9 scr, last pts/stc/sof 0/0/0
Jan 21 08:48:52 kernel: [ 612.294269] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.294270] uvcvideo: Frame complete (FID bit toggled).
Jan 21 08:48:52 kernel: [ 612.294270] uvcvideo: frame 1957 stats: 0/1/1 packets, 0/0/0 pts (!early !initial), 0/1 scr, last pts/stc/sof 0/1217480818/18547
Jan 21 08:48:52 kernel: [ 612.294272] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.294678] uvcvideo: frame 1958 stats: 0/2/2 packets, 0/0/0 pts (!early !initial), 0/1 scr, last pts/stc/sof 0/1217480818/18547
Jan 21 08:48:52 kernel: [ 612.294682] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.294682] uvcvideo: Frame complete (FID bit toggled).
Jan 21 08:48:52 kernel: [ 612.294683] uvcvideo: frame 1959 stats: 0/1/1 packets, 0/0/0 pts (!early !initial), 0/1 scr, last pts/stc/sof 0/1267616628/19316
Jan 21 08:48:52 kernel: [ 612.294685] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.294686] uvcvideo: Frame complete (EOF found).
Jan 21 08:48:52 kernel: [ 612.294888] uvcvideo: Dropping payload (out of sync).
Jan 21 08:48:52 kernel: [ 612.295094] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.295094] uvcvideo: Dropping payload (out of sync).
Jan 21 08:48:52 kernel: [ 612.295299] uvcvideo: Dropping payload (out of sync).
Jan 21 08:48:52 kernel: [ 612.295509] uvcvideo: Marking buffer as bad (error bit set).
Jan 21 08:48:52 kernel: [ 612.295510] uvcvideo: Dropping payload (out of sync).
Jan 21 08:48:52 kernel: [ 612.295715] uvcvideo: frame 1960 stats: 0/5/5 packets, 2/4/3 pts (!early !initial), 2/3 scr, last pts/stc/sof 1284525428/1284525171/19827

Looking at the source code, Frame complete (FID bit toggled) means that the USB driver hasn't sent the complete frame (otherwise we would get an (EOF found) message), which is backed up by the log showing 0/2/2 packets instead of 0/151/151. How do I proceed with the debugging now? I find it hard to believe that the USB driver is buggy, but is there some not quite RTLinux-ready component in the stack?