A Report on Performance Between Version 3 and Version 4

J

jmanos3

Member
Parallels,

I, like many others, upgraded to version 4 with the expectation that performance would be better than version 3. This expectation was set by Parallels in their press release announcing version 4. However, I have not been able to achieve these performance gains in my environment.

I have been participating in many threads regarding version 4 and have been encouraged by Parallels' increased level of involvement. However, as it relates to performance, I feel that both Parallels and the user community have been taking "shots in the dark" when attempting to resolve the problems.

With this in mind, I completed a performance comparison between version 3 and version 4 and have posted a detailed report of my findings. The link to my public iDisk area is provided below where the report, in PDF format, can be downloaded. The file is 2MB in size and is listed first on the iDisk file listing.

http://idisk.mac.com/jmanos3-Public/?view=web

I also posted a 9MB zip file containing all of the configuration information used to create the report. Both files have been posted externally because of attachment file size limitations on these forums.

I am posting this information via the forum (instead of a problem ticket) because I believe it is critical that other users participate in the discussion. However, if you would like a problem ticket created, I would be happy to do so.

I have invested many hours in completing the comparison and writing the report. I would ask that you invest a similar level of time reviewing the information provided and respond back to the user community with your interpretation. As an alternative, if you have existing information comparing performance of version 3 and version 4, please make it available or point me to where it can be found.

Please understand that I am a fan of Parallels Desktop and want the product and company to succeed. But, I am also a customer who, at the moment, is not receiving the stated benefits from the product.

Users of Parallels Desktop,

Some of you are experiencing great performance with version 4 while others are not. As I am one who is not, I would love to know what you have done and/or more information on your configuration. If you are so inclined, please review the report and provide information as you see fit.

Thanks,

James
 
James,

Please also make Help->Report a problem and write here the ID. Ideally this should be done while VM is running. This will provide us with some logs and configurations necessary to investigate the problem.

Thank you for your effort, truly appreciated!
 
STim said:
James,

Please also make Help->Report a problem and write here the ID. Ideally this should be done while VM is running. This will provide us with some logs and configurations necessary to investigate the problem.

Thank you for your effort, truly appreciated!
Click to expand...

I have created and submitted a problem report - #17274.

Thank you for investigating this.

James
 
Additional Testing / Updated Summary Reported

Based upon other performance related threads, I completed additional testing by introducing a freshly built, Windows XP Professional VM created in Parallels Desktop version 4 build 3540. I have updated my report and summarized the level of technical detail in order to add it directly to this post (see attachment).

As you read the report, pay special attention to the conclusions section. It clearly shows it is possible to support the performance improvement claims made by Parallels for version 4. However, in my findings, it is only possible when creating a new virtual machine and for one of thirteen test actions that I performed.

James
 

Attachments

I really don't understand your report, James. It seems like the only tests you are doing, other than startup and IE, are on the OSX side of things. Am I reading the report wrong?

Further, I don't really see any improvement in your "new VM" scenario so I don't understand why you now claim it is (only) possible to support the performance improvement claim by using a new VM. The new VM numbers actually look slower to me.

Is your primary concern the performance of OSX, or of windows?

My timed tests, as posted in a different thread last week, conclude PD4 is indeed slower for windows related activities. My startup times for V3 were much faster than yours, while my startup times for V4 are about the same as yours. Once the system starts up, performance of V4 is slower, but not by much.

I've tried the various suggestions with no affect, so I assume my virtual machine is not "damaged" like that of so many others' seems to be.

We know that parallels has a new virtual CPU in V4, probably for various reasons (primarily of which might have been to support multiple processors). You can see from both your tests, and from mine, that the dual processors makes things slightly worse. I suspect over time that parallels will improve the performance of their virtual CPU.
 
James, per my previous post (I cannot change it until the moderators approve it) - I have now read your latest report and understand that you are indeed testing windows apps. What tripped me up was your mention of "text pad", safari, and itunes. I think the windows text apps are either "notepad" or "wordpad", not "text pad"? When I saw this, I figured it had to be OSX apps that you were testing. From your latest report, I clearly see you are testing windows apps (safari on windows, itunes on windows, etc.). I am sorry for the confusion.

But, my other questions and observations about your timings still stand (for the moment).

Paul
 
Paul,

First, thanks for taking the time to review the report and for catching the "TextPad" error - my bad. I admit the report could use another round of editing. I will update it to reflect Notepad.

Second, let me address the other questions and comments you made.

phesopheon said:
Further, I don't really see any improvement in your "new VM" scenario so I don't understand why you now claim it is (only) possible to support the performance improvement claim by using a new VM. The new VM numbers actually look slower to me.
Click to expand...

You are correct - the majority of tests show that, in the new VM scenario, version 4 is not faster. However, there was one result, Internet Explorer, that showed significant performance improvement in the best case. As such, when comparing version 4 to version 3, Parallels could support the claim they made on performance.

I did write the follow-up post in a "teasing" way to get peoples interest in the thread up. But, being a "geek" that has also done sales and marketing, I wanted to be fair to Parallels because the test results showed it was possible.

Parallels is aware there are performance problems with both upgraded and newly created VMs and they are working on it. I personally would like to see more information come out from them because the user community could assist in the process.

phesopheon said:
Is your primary concern the performance of OSX, or of windows?
Click to expand...

Windows. I use my VM for work purposes. Because of this, I am in the VM 95% of the time during working hours. I use the standard Windows applications, including Office, along with some work specific applications. I did not include those in my tests because I wanted to keep the applications tested as generic as possible so everyone had access to them.

phesopheon said:
My timed tests, as posted in a different thread last week, conclude PD4 is indeed slower for windows related activities. My startup times for V3 were much faster than yours, while my startup times for V4 are about the same as yours. Once the system starts up, performance of V4 is slower, but not by much.
Click to expand...

I am really interested in why your V3 startup times are faster than my V3 startup times. The VM I used for testing is very vanilla - no third party applications, no anti-virus, no internet security, etc... As such, I would not think it could get better unless the VM is run on faster hardware.

phesopheon said:
You can see from both your tests, and from mine, that the dual processors makes things slightly worse.
Click to expand...

I am glad to see that other test results are showing that version 4 performance is worse when more than one processors is allocated to the VM. I was not expecting the results of my tests to show better performance under multiple processors because the applications used in the test were not developed for that purpose. However, the fact that results were worse is a concern.

James
 
Hi James,

Thanks for your reply, and for the excellent work you have done thus far. I hope the people at parallels take it seriously for it is the first real, objective, documented set of performance tests that I have seen. The tests I ran myself are specific to one of our windows based applications at where I work, so describing those results to the level of detail that you did in your report would not make sense to anyone unfamiliar with our app.

jmanos3 said:
I am really interested in why your V3 startup times are faster than my V3 startup times. The VM I used for testing is very vanilla - no third party applications, no anti-virus, no internet security, etc... As such, I would not think it could get better unless the VM is run on faster hardware.
Click to expand...
I have no explanation, but I had consistent results with several VMs. I do observe now that with V4 there is about 10 seconds before the white windows progress bar appears on the screen. I seem to remember (I'm being purely subjective in this) that it appeared much faster in V3.

Overall, my guess is that the new "hardware" in V4 is much more accurate and complete, and this is the reason for the delayed startup time and perhaps also for the slower overall times. The new "hardware" supports network boot, power management, multiple processors, and who knows what else. All that comes with price. I wonder if any of us really wanted all that extra (more correct) virtual hardware layer, but now we have it.

On the other hand, I thought the Intel VT chip was supposed to make all of this better....

Paul
 
phesopheon said:
Hi James,
Overall, my guess is that the new "hardware" in V4 is much more accurate and complete, and this is the reason for the delayed startup time and perhaps also for the slower overall times. The new "hardware" supports network boot, power management, multiple processors, and who knows what else. All that comes with price. I wonder if any of us really wanted all that extra (more correct) virtual hardware layer, but now we have it.
Paul
Click to expand...

Unfortunately you are right to some extent. When running in PD3 Windows uses "Standard PC" HAL, while switching to PD4 changes HAL to "ACPI" which is only slightly slower. Booting VM with multiple processors upgrades HAL to more slower "ACPI Multiprocessor HAL" (and never downgrades it back!). Unfortunately such "effects" in software are very hard to deal with and it's very easy to decrease performance.

We take it VERY seriously and I'm contacting James right now for checking with him together on what's going on with his VMs, whether he has some errors in benchmarking methodology or has some real troubles and needs help.
 
[email protected] said:
When running in PD3 Windows uses "Standard PC" HAL, while switching to PD4 changes HAL to "ACPI" which is only slightly slower. Booting VM with multiple processors upgrades HAL to more slower "ACPI Multiprocessor HAL" (and never downgrades it back!).
Click to expand...
GREAT answer! This is very helpful. Thank you for posting.

I took a look and I can see the different platforms in device manager in windows. Too bad there is no way to go back.

Paul
 
So I have to ask, as a Multi-core desktop user, is there any good reason to have my VM set to 2 CPU's? Does this slow down anything other than boot times?
 
benhanson said:
So I have to ask, as a Multi-core desktop user, is there any good reason to have my VM set to 2 CPU's? Does this slow down anything other than boot times?
Click to expand...

At this point, I would say no unless you have applications that can take advantage of the the multiple cores within the VM.

One example that I can think of is Handbrake - a video conversion tool. While this is an open source tool that could be run directly on the OS X side, it is multi-threaded and, when I run it on the Mac, it utilizes multiple cores. I would suspect that it would do the same within the VM if multiple cores were available.

James
 
[email protected] said:
We take it VERY seriously and I'm contacting James right now for checking with him together on what's going on with his VMs, whether he has some errors in benchmarking methodology or has some real troubles and needs help.
Click to expand...

I just want to let everyone following this thread know that the Parallels team follows through - they have contacted me and I continue to work with them on my particular issues.

Nice job Parallels!

James
 
Reducing Windows VM Startup Time

All,

Over the past few weeks I have been working with Parallels to identify and resolve performance issues experienced with version 4. One of the areas that was a focal point was the longer than desired startup times for my Windows VM. I am happy to report that the Parallels team identified a method to improve the startup time and developed a knowledge base article for it - http://kb.parallels.com/en/5859.

In my own testing, I found that optimizing the boot sequence (using the methods described in the article) improved my Windows VM startup times significantly for both newly created VM's and for VM's upgraded from version 3 to version 4. The summarized results are presented below.

New VM

Version 3.0.5626 - Pre Optimized ~ 55 seconds
Version 3.0.5626 - Post Optimized ~ 40 seconds
Version 4.0.3540 - Pre Optimized ~ 45 seconds
Version 4.0.3540 - Post Optimized ~ 32 seconds

Upgraded VM

Version 4.0.3540 - Pre Optimized ~ 60 seconds
Version 4.0.3540 - Post Optimized ~ 38 seconds

What the results above show is that version 4 does perform faster than version 3.

I would like to publicly thank the Parallels performance architects that I worked with during this effort. They were very helpful, considerate and professional and I would work with them again any day.

Thanks,

James
 
You must log in or register to reply here.
Back
Top