Home Sharing from Mac to Mac is stuck at song start

I have a Mac Mini serving as my home media server. I setup Home Sharing, and I can see my library on all my other devices at home. My entire collection is without any DRM, and it's all ALAC (some 24 bit, some 16 bit). I am able to stream music to my iPhone, iPad and my Apple TVs. However, I am unable to stream to other Macs in my house, I tried both a 5k iMac and a Macbook Air. The Mac Mini is running Catalina. The iMac is running Big Sur. The Macbook Air was running Mojave (didn't work), I then upgraded to Big Sur (still didn't work). I'm running out of ideas what I can check and would love some suggestions.