Microsoft argues benchmark results can’t be relied on to assess cloud PCs

Executives at Microsoft believe that performance testing is not a reliable assessment of a cloud PC as it's usually done in a sterile environment that does not take into account all the variables that may impact real-world performance.  

In an official blog post, Ron Martinsen, senior product manager of Windows 365 Cloud PC/Azure Virtual Desktop, said that to get reproducible numbers, testers remove as many variables as possible, which is then used by product makers to boast the best possible numbers, even though they may not be applicable in real-world usage.  

“Such numbers would not be very useful or reliable, since a cloud PC is not a fixed hardware device but a dynamic SaaS offering that continuously evolves,” said Martinsen. He urged that IT professionals looking for the best cloud PC configuration, to focus on relative performance charts for different cloud PCs, released by Microsoft, instead of test numbers that vary.  

Martinsen's comments were made on the same day ML Commons, an engineering consortium that maintains testing benchmarks used by the chip industry, released test results on Wednesday, which show that Qualcomm’s AI 100 chip is more power efficient than Nvidia’s H100 chip.  

The Nvidia chip powers the Microsoft supercomputer used for training OpenAI’s GPT-4 model, the underlying technology behind the popular AI chatbot ChatGPT. Microsoft has deployed several new AI products such as AI-powered Bing, which is based on GPT-4, which has helped Microsoft revive the search engine battle. Microsoft also announced Azure OpenAI services which give its enterprise customer access to GPT-4 and other OpenAI products including Codex and DALL-E.  

The big tech firm also announced a multi-billion dollar investment in OpenAI.  

However, the use of AI-powered chatbots to answer search engine queries has raised environmental concerns. According to a February report in Wired, which cites several experts, using AI for search will require more processing power, which will lead to a significant increase in the power and cooling resources. Microsoft has tried to address these concerns by limiting the number of chats users can have with AI-powered Bing.  

Chipmakers on their part are trying to address this by making their AI inference chips more power efficient. According to ML Commons’ test results, Qualcomm’s AI chips beat Nvidia in two out of three power efficiency tests. For instance, in tests to measure how many server queries a chip can carry per watt, the Qualcomm chip registered 197 queries per watt as compared to Nvidia’s 108 queries per watt.  

Martinsen argues that the number of variables increases when it comes to clouds.   

“Imagine if you take all these variables and complexities and put them into an environment where the host computer is running on a cluster and serving a variable number of active cloud PCs, which then, in turn, could host their own virtualisation workloads,” said Martinsen.  

He added that the number of variables impacting performance in such a scenario is going to be overwhelming.  

Martinsen also noted that hardware configurations for cloud PCs change over time to optimise performance and efficiency. 

“The underlying hardware of cloud PCs may vary over time as Microsoft updates and upgrades its infrastructure. Therefore, it is not possible to provide exact performance metrics for each cloud PC configuration as they may differ slightly depending on hardware configuration,” he explained.