The computer is likely paging, not enough free memory (you can check this via Activity Monitor.app).
The total amount of physical RAM has to be shared between OS X and Windows.
You have 4 GB RAM, don't assign more than 2 GB to the VM, in fact, I recommend 1 GB only. Of course, the apps you run on the OS X side, at the same time, have influence on free memory, in particular, browsers, for instance, Safari can easily eat up 1 GB RAM.
Last edited: Jan 10, 2012