When will Finder use large amount of memory?

Sometimes I see the iMac with 4GB of RAM having a few MB of swap file only, and after I left for a few hours and come back, the swap file is 2.7GB. Looking into the Activities Monitor, the Finder is using 796MB. How can it use so much memory -- could it be due to some "Search" that was left over in a Finder window? I then chose File -> Close All (by pressing down the Option key while clicking File, and closed all Finder windows, only that it still won't close but just hidden, and looks like the only way is to reboot. How can this situation be prevented?