Moving Linux install to a new computer

I know that it can, in some circumstances, be difficult to move a Windows installation from one computer to another (physically move the hard drive), but how does that work on Linux? Aren't most of the driver modules loaded at bootup? So theoretically would it be that much of a hassle? Obviously, xorg configs would change and proprietary ATI drivers and such would have to be recompiled (maybe?). Is there more to it than I'm thinking of? Assume the two computers are from the same era, e.g. both i7s but slightly different hardware. Update: Thanks for the answers. This is mostly for my own curiosity. I have my Linux system up and running at work, but eventually I'd like to move to a computer that I can get dual video cards into so I can run more than two monitors. But not any time soon.