Unfortunately, my OpenStep 4.2 VM that worked fine under Beta4 hangs at startup under Beta5, even with VT-X disabled....
Which guest resolution you have in your Openstep VM? We implemented dynamic screen resolution feature in Beta5 and some videomodes numbers were changed...
My openstep image hangs up as well, and it does so at boot time while trying to load some config file (probably it hangs after trying to apply things it read I think). Nothing to do with videomode apparently, as even booting in Default (eg 16 colors vga mode) hangs, as well as single user boot...
I'd love to play with it, but have no media or license. History can teach us a lot, and I like toying with older OS's.