Assigning Mouse+Keyboard to VM

Can I connect a 2nd mouse and keyboard to my iMac and have them control only one of my virtual machines?

I have the same video game installed on my Windows VM and my Ubuntu VM. I want to run both at the same time off of my iMac, each on a different display (this I can do), but I want my friend and I to be able to play at the same time. So we need Mouse A and Keyboard A to only effect Windows and Mouse B and Keyboard B to only effect Ubuntu.

Anyone know how to do this?