vlc playback issue after using pulseaudio jack sink

Distro: Debian Bullseye Audio interface: M-Audio M-Track 2X2M, Focusrite Scarlett 2i4 I can play any sound fine with vlc, mpv, quodlibet, firefox. I start jack and I can still plays sound by telling each app to use [Pulseaudio jack sink](https://askubuntu.com/a/1213554/419514) . When stopping jack, each software goes back from jack sink to audio interface. They all play fine (after a short transition glitch when stopping jack) except for vlc which has audio playback issues (image seems fine). The symptoms vary slightly between the two audio interfaces. When using the M-Audio M-Track 2X2M, the sounds keeps stopping every few seconds and the console is filled with those errors:

[0000559265fdea20] pulse audio output warning: starting late (-17920 us)
[0000559265fdea20] main audio output warning: playback way too early (-512867): playing silence
[0000559265fdea20] main audio output warning: playback too late (60253): up-sampling
[0000559265fdea20] main audio output warning: timing screwed (drift: 121333 us): stopping resampling
[0000559265fdea20] main audio output warning: playback too late (121292): up-sampling
[0000559265fdea20] main audio output warning: playback way too late (182033): flushing buffers
[0000559265fdea20] main audio output warning: playback way too early (-508497): playing silence
[0000559265fdea20] main audio output warning: playback too late (75859): up-sampling
[0000559265fdea20] main audio output warning: timing screwed (drift: 154957 us): stopping resampling
[0000559265fdea20] main audio output warning: playback too late (154937): up-sampling
[0000559265fdea20] main audio output warning: playback way too late (194625): flushing buffers
[0000559265fdea20] pulse audio output warning: starting late (-1470 us)
[0000559265fdea20] main audio output warning: playback way too early (-508119): playing silence
[0000559265fdea20] main audio output warning: playback too late (60012): up-sampling
[0000559265fdea20] main audio output warning: timing screwed (drift: 120048 us): stopping resampling
[0000559265fdea20] main audio output warning: playback too late (120331): up-sampling
[0000559265fdea20] main audio output warning: playback way too late (180143): flushing buffers
[0000559265fdea20] pulse audio output warning: starting late (-3654 us)
[0000559265fdea20] main audio output warning: playback way too early (-548874): playing silence
[0000559265fdea20] main audio output warning: playback too late (77824): up-sampling
[0000559265fdea20] main audio output warning: timing screwed (drift: 159831 us): stopping resampling
[0000559265fdea20] main audio output warning: playback too late (159790): up-sampling
[0000559265fdea20] main audio output warning: playback way too late (180711): flushing buffers
[0000559265fdea20] pulse audio output warning: starting late (-19826 us)
[0000559265fdea20] main audio output warning: playback way too early (-513173): playing silence
[0000559265fdea20] main audio output warning: playback too late (60207): up-sampling

When using the Scarlett, I don't get those errors and the sound doesn't stop but it sounds dirty, as if it clipped. vlc seems to be the only affected player. Restarting vlc doesn't help. Changing output to internal soundcard then back doesn't help either. Issue stops when selecting internal sound card then starts again when selecting audio interface. Restarting pulseaudio (then restarting vlc) does solve the problem (until next time).

systemctl --user restart pulseaudio

I'm clueless. I don't see anything relevant in syslog. The only log I can find that shows something related is the vlc log quoted above, but I'm afraid it only shows the consequence, not the root cause. -------------------------------------- I just noticed those errors in QjackCtl when stopping jack. They only appear when using pulseaudio jack sink.

Sat Dec  3 18:13:29 2022: Client 'qjackctl' with PID 5618 is out
Sat Dec  3 18:13:29 2022: Stopping jack server...
Sat Dec  3 18:13:29 2022: Client 'system' with PID 0 is out
Sat Dec  3 18:13:29 2022: Client 'PulseAudio JACK Sink' with PID 5393 is out
Sat Dec  3 18:13:29 2022: Client 'PulseAudio JACK Source' with PID 5393 is out
Sat Dec  3 18:13:29 2022: ERROR: Cannot write socket fd = 47 err = Broken pipe
Sat Dec  3 18:13:29 2022: ERROR: CheckRes error
Sat Dec  3 18:13:29 2022: ERROR: Could not write notification
Sat Dec  3 18:13:29 2022: ERROR: ClientNotify fails name = system notification = 1 val1 = 0 val2 = 0
Sat Dec  3 18:13:29 2022: ERROR: Cannot write socket fd = 51 err = Broken pipe
Sat Dec  3 18:13:29 2022: ERROR: CheckRes error
Sat Dec  3 18:13:29 2022: ERROR: Could not write notification
Sat Dec  3 18:13:29 2022: ERROR: ClientNotify fails name = system notification = 1 val1 = 0 val2 = 0
Sat Dec  3 18:13:29 2022: Released audio card Audio1
Sat Dec  3 18:13:29 2022: ERROR: Cannot write socket fd = 47 err = Broken pipe
Sat Dec  3 18:13:29 2022: ERROR: CheckRes error
Sat Dec  3 18:13:29 2022: ERROR: Could not write notification
Sat Dec  3 18:13:29 2022: ERROR: ClientNotify fails name = freewheel notification = 1 val1 = 0 val2 = 0
Sat Dec  3 18:13:29 2022: ERROR: Cannot write socket fd = 51 err = Broken pipe
Sat Dec  3 18:13:29 2022: ERROR: CheckRes error
Sat Dec  3 18:13:29 2022: ERROR: Could not write notification
Sat Dec  3 18:13:29 2022: ERROR: ClientNotify fails name = freewheel notification = 1 val1 = 0 val2 = 0

Is it possible that when stopping jack, pulseaudio jack sink is interrupted in a not so nice way and this screws up pulseaudio?