iTunes Home Sharing doesn't work with second library

I have two libraries on my Mac, one of them has all of my Apple Music, while the other has all of my iTunes purchases and CD rips. I'd like to share the second library with my Apple TV and iPhone. However, I cannot seem to get this to work. The first library shares just fine, and I can access it just fine from my phone/TV, but when I restart iTunes and select the second library, I cannot access it. I have tried restarting everything, disabling and re enabling sharing, tried with and without a password, and nothing I do seems to make it work. Yet the second I restart iTunes with my Apple Music library, it shows up under home sharing almost immediately.