First lets start by answering “Do I agree with the VMworld 2010 Awards winners?”. My answer to the previous question to the most part I agree with most of the winners, though I believe the judging process might need a bit of improvement. I have just came across Eric Siebert, which discuss the judging process for VMworld Award winners. You can find Eric article at:
Although the process is quite fair to every vendor, I still believe without installing and testing the product running we will allow vendors to promise the sky and deliver nothing close. I am sure the judges like Eric are Virtualization experts that are not easy to fool about the products functionality, but still there is a gray area that vendors can play around. I believe it would be more fair if the judges get to test these products in the lab & ensure it meets the promised functionality. As the installation of all the products to be tested can be time consuming for the judges, it might make sense to let these app vendors install the application and keep it ready for the judges test drive.
Further, I was wondering if the judges declare them selves to the competing vendors as judges when they go to evaluate and ask about the products. I believe it will be quite interesting to get the judges to act as they are just a normal attendees and detect how of a professional reaction the vendor at the booth offer.
One extra tip that I think it would be nice to let VMworld attendees votes count for a percentage of the judging process. As what is the most innovative, creative, & the most important to fill market gap will highly differ from a customer view , an engineer at VMware, or Vendor view. It will help to gather the full vote. Let’s the attendees vote count for something!!
As VMworld Award is the most prestige award in the virtualization market today, keeping it fair and raising the bar for it is quite critical. Many customers will factor the award in their buying decision and are not aware of the judging process & will assume the product perform as advertised being a VMworld Award winner. It’s important to test these products to death before declaring them a winner not to let down customers on the long run.
2 responses to “My VMworld Awards Critique”
Hi Eiad,
Thanks for the feedback. Indeed it is not an easy process and I think there is always room for improvement. Often times I have little experience with some of the products that get nominated so I have to come up to speed quickly on them. The problem with installing and testing apps is it is very time-consuming and also many products in the same category are dis-similar so its hard to compare them with other products. All the judges are very experienced virtualization veterans that really know their stuff. Judges are typically paired up with a category that they have a lot of experience in as well.
I can’t speak for the other judges but I always identify myself when visting the vendor booths. Doing this ensures I speak with their best and most technical people, I’m sure the other judges do this as well. The decisions we make are not always easy which is why we rely on group deliberation at the end of the day to select the winners. This gives all the judges an opportunity to provide feedback as one may have valuable insight/experience with a product that the others may not.
I like the idea of having customer voting as part of the decision process. The one thing you have to be careful about though is many vote based on personal preference and their likes/dislikes of certain vendors may not be an accurate reflection of how good their product is. I’ll pass along your feedback to the folks at Tech Target.
Eric
Hi Eric,
Thanks for your contribution back & thanks again for being ready to share the judging process.
Enjoy,
Eiad