Writing to the framebuffer while X or Wayland is running?

This is a pretty straightforward question. I am writing a Qt program that looks to render graphics in the terminal. To do that, I utilize Sixel Graphics which has a large overhead. I would rather render directly to the frame buffer, but can that be done while X and Wayland are running? Or do I have to strictly be in a headless environment, meaning no active desktop environment, to be able to access this? Does X or Wayland perhaps have functions allowing me to use it to render graphics through it? Thanks.