I haven't specifically tried this, but it would make sense, especially if you don't have a lot of memory. Rosetta is a RAM hog, and obviously, so is Parallels. Also, even with beta 3 some people are experiencing 20% CPU at idle.
How much total memory do you have in the mini, and how much is allocated to the VM? Also, what applications are you running under Rosetta?
1 gig memory in Mini, 256 allocated to VM and the programs are MS Word, Firefox etc.. (not run all at the same time). It sounds logical that there may be page swapping going on with the virtual memory. So when I switch, it loads the rosetta memory from the HD. The only thing that is weird though is that I have no delay going back to the VM.
Have you tried adjusting the memory allocation for the VM to 512Mb? Word normally performs pretty well under Rosetta (unlike some of the Adobe apps), and it should do ok with only 512Mb. On the VM, 256 is pretty tight. I haven't even run a hardware PC with that little in years. Try raising the memory for the VM to 512, and re-start Windows. I think that will work better.
Using this thread and reducing the overall memory fixed the slow down http://forum.parallels.com/thread595.html Great job!!!