I have been contacted by Kevin Tolly the founder of http://CommonTestPlan.org to evaluate their Virtual Server Performance Test Plan & provide suggestions in that regard about two weeks ago. Kevin & his team were kind enough to provide me with a one year license to their Virtual Server Performance Test Plan Document for my own personal use without any string attached. I believe they are offering the same to many other Virtualization Experts around the globe in order to improve their document & get some honest feed backs from professionals who hit the virtualization field on daily basis. Below is my honest review & thoughts of the document.
As no product is ever perfect and Tolly Common Test Plan for Virtual Server Performance is no exception for that I will cover the points I like & the points I believe are missing of the report in here.
I will start with what I like about the Tolly Common Test Plan for Virtual Server Performance. The document is focused on helping IT Professionals in identifying the key concepts for virtual servers benchmarking as well helping in avoiding common mistakes in virtual servers benchmarking. Its one of these topics that has barely any standards & many misleading concepts are all over the place, that make a similar document essential for enterprise environment that is evaluating multiple virtualization solutions.
The document can save enterprises the expensive time of developing their own Virtualization benchmarking from scratch. In addition, it can help avoid making expensive mistakes by evaluating virtualization solutions performance based on misleading measurements. I am planning to adjust my performance testing procedures according to some of the advices in the document, as I was over-looking some of the points that can make a difference in certain environments.
The size of the document is 39 pages, which is not too lengthy. Though it covers most of what you need to develop a performance benchmarking for your virtualization solution. In addition, it can help you with crafting RFP for your Virtualization Solution tender.
Below are few things I would hope they will include in the next release as they were missing in the current one:
– As Different virtualization solutions waste a different amount of resources when building High Availability. I did not notice that covered any where in the document. I believe it should be included.
– The document has very nice tables that can nicely be used to evaluate virtualization solutions performance, but I had no way of copying these & insert my data into them without having to build the table from scratch. I am not sure if it was only a limitation with my license type (I could not complain as I got it for free). I understand the fact they need to protect their document, though I still believe they should find away to supply their users with these tables even if it goes into a separate excel sheets. As providing these table to customers can save them plenty of time constructing it from scratch, & this is quite important as time saving is one important benefit of this document.
– Although performance is a major part when comparing between Virtualization solutions, there is much more to it. I wish if the document discussed solutions reliability testing, solution features (specially features related to virtual machines high availability). Though its fair to mention that performance is all what the document claim to cover. Adding extra sections covering these extra pieces to it will be great, or maybe release a second document covering these bits.
Over all I am quite happy with the document & I believe it well deserve its price due the time you would need to build it from scratch.