Weird CAC behavior with M1 MAX Macbook Pro, Parallels Desktop 18, and Win 11 ARM64

OK community, I got a poser for ya. I use a CAC with an Identiv SCR 3300 USB-C smart card reader. It shows up in my Win 11 VM Devices->USB & Bluetooth menu. When I turn it on for Windows, it works perfectly inside Windows. When I turn it off, it doesn't work at all for the Mac. Here's the weird thing - I have another USB-C CAC reader, this one is an ACR39U folder that I used to use but it got jostled all the time so I switched to the larger Identiv. I plug that one in, and voila! Mac sees it just fine and the CAC works perfectly in Mac OS Ventura 13.1.

I really would rather just use the Identiv SCR, but I can't figure out how to get the Mac recognize it again. In my System Report, it shows up as a connected Smart Card reader under USB, so electrically it shows up.

Any ideas?