Want to verify it's IMPOSSIBLE to use a Mac mini 2018 or later with ARD without connecting display, keyboard, & mouse

I've already done tons of experimentation with this and feel mostly confident I know the answer and I have done searches on here to make sure this is not a duplicate query. I recently replaced my Mac mini 2018 with the new Mac mini M4. My Plex server has been running on a Mac Pro Tower (2009) for years and it's not IF but when it gives up the ghost. So I figured I'd repurpose the 2018 Mac mini to be my new Plex server with an external 4 drive raid box. Since it will be pretty much just an always running server I see no need to have a display, keyboard, and mouse always attached after I get it all set up. I have several Macs in the house in different locations and they all work great with ARD but the are full systems with display, keyboards, and pointing devices. All of them give me pretty much full control including being able to reboot them remotely, and log back in remotely. But after several days of trying different ways of doing this including plugging in a HDMI dummy connector to simulate a display it appears the only way I can do this on this Mac mini is to keep a mouse, keyboard, and display plugged in all the time. I know I could keep all that stuff unplugged and nearby after getting it going but the pain of pulling them out and using them to reset the system would be such a pain and I really don't have the space for the display and keyboard and mouse to be sitting around. I've disabled much of the built-in security thinking that would be the issue -- but no. And by default every time a remote access attempt is made that does not work because everything is not plugged into it, the Mac goes into recovery and I have to plug all that stuff back in to get it up and running back to the login screen. **SO... is it the case I'm reduced to leaving a keyboard and mouse and display plugged into it all the time and just hide it away behind the rack we have everything supporting our home office in?** I just want to be sure I've not overlooked something. I've tried every variation and combination of setups I can and every time I'm having to reboot the Mac with all its accessories to get it to work again.