Xorg taking huge amounts of memory

Lately, there is a lot of memory leak on my (Arch) Linux laptop. A command named

-nolisten tcp :0 vt1 --keeptty -auth /tmp/serverauth.mWgFYYiRdF

is continually taking 27.2 % of my 8GB RAM (also around 2G is consumed of swap). How do I troubleshoot ? (I use no login manager, just