Need small distro without a desktop or windows manager, just to run a single graphical app

I need a small distro, that is stable. I don't need a full X server or window manager, I only need it to run one single application with a basic UI that consists of a viewport. I would like for the distro to be as small as possible. 700 mb or less would be ideal. Is their a base distro of ubuntu or similar that I can add whatever I need to it from the command line. Which basically is the kernel and some way of graphical output. I was thinking of putting Direct FB on it to render the application. Even a live distro would work.