Eliminate inclusion of X11 libs in initrd

I'm running Debian Bookworm on a TS412 (QNAP); it stores the kernel & initrd in flash. It has a total of 16MB of flash. So even after tweaking with the MD partitions the best I can get is a 12MB initrd . I'm using xz compression. But even so, the initial initrd was 12.5MB. I know the common fix is to reduce modules, but they are not the real issue.

2984    .../usr/lib/modules
6036    .../usr/share
18824   .../usr/lib/arm-linux-gnueabi

Looking in arm-linux-gnueabi it had stuff like libX11... This machine does not even have a console (even getting a serial line is difficult) it certainly does not need "X". So I removed lots of packages totally (apt-get remove ...) in fact rather more than I should really. But the result was a much smaller initrd (about 5MB IIRC) which boots and runs just fine. Problem is, if I ssh onto the box and run say emacs(1) it'd be nice to project the GUI version back to my normal desktop (which has lots of memory and big displays, etc.) but this would require installing the X client libraries on the TS412. Now I don't see why just because I have the package with the X client libraries installed, that a copy needs to be kept on initrd, they aren't needed during boot. Is there a way to limit other stuff from getting onto initrd, not just modules (which are not the culprit) I've already tried document searches for guides to limit initrd.