Enable a generic user to start X

I have Debian, Sid if it matters, and I'd like to start X from the console, rather than using a display manager, but when i type startx I receive the following message xinit: connection to X server lost If I'm logged as root, or if I use sudo, I'm able to start X so I suspect that permissions are at the root of the problem . What should I do to enable generic users to start X, possibly via startx ? I'm interested both in a simple solution, achieved probably manipulating permissions, that has other drawbacks and in a possibly more complicated solution, maybe a proxy like a sort of _non-graphical display manager_, or whatever else…