On a Mac Studio M3 Ultra (the most powerful Mac that exists) and host hangs for a few seconds

I have a single VM running the same MacOS as host (both on latest macOS). I am passing half of the cores to the VM so that rules out that it is stealing too much power from the host. And yet, youtube playback on host, zoom meetings, etc all have periodic hangs for a couple of seconds when the VM is running. Also, the VM has time-tracking software that periodically takes screen snapshots, and every time it takes a screenshot of the VM, it causes the host to hang/stutter a few seconds. What is going on? I can run MacOs in proxmox as a VM and it works perfectly fine, but real Mac hardware using official options like Parallels is glitchty?