Hands on with Microsoft’s AI Bing: How well does it search?

The first question I asked Microsoft’s new artificial intelligence (AI)-powered Bing search was met with a wrong answer.

Question: When is India’s Parliament session ending?

Answer: According to the search results, the Monsoon session of Parliament in India commenced from July 19, 2021 and concluded on August 13, 2021. The Winter session of Parliament in India commenced from December 7, 2021 and concluded on December 23, 2021. The Budget session of Parliament in India is scheduled to commence from January 31, 2022 and conclude on April 8, 2022.

Bing eventually got the answer right. I just had to berate it for providing outdated information. For the AI, the problem was simple — I didn’t ask it to provide updated information, so it went for what Bing’s search algorithms said were the best results for such a search term.

Microsoft’s CEO, Satya Nadella, has said that Bing’s new chat version runs on a version of OpenAI's GPT-3 model, but the fact is that it’s grounded in search -- even the AI chatbot will tell you that the information it gathers is not just from the data it’s trained on, but that it actually runs Bing’s search algorithms across the web to come up with an answer.

But is Bing the best way to search?

There’s a reason Google is synonymous with the internet. Since 1998, it surpassed all other search engines by perfecting the technology needed to assimilate the zetabytes of information on the web. The new Bing does not appear to have made any radical changes. Instead, it now has this intelligent algorithm which can do the search for you, and then summarize everything it learned into a concise answer. And it will dutifully display the sources it used as well, many of which can be questionable at times.

For instance, Bing used a website called BISInfotech (among others) to answer a question I asked on India’s manufacturing efforts. When I asked why it’s using this website, it quickly answered that the website has a “team of experienced journalists and a large readership”, which makes it a dependable source. A little more probing and I found AI Bing thought the website was dependable because the site’s About page said so.

Answer: I know it has a team of experienced journalists and large readership because I found some information about them on the web. According to their website, bisinfotech has been in operation since 2004 and has a team of dedicated writers, technical heads, and industry veterans who cover various topics related to electronics and technology. The also have a LinkedIn page where they have, 165 followers and claim to be the gospel of industrial tech. According to their media kit, bisinfotech magazine is distributed to approximately 250,000 readers, including corporate decision-makers, researchers, engineers, designers, and systems integrators. They also have a high pass-along rate and a high percentage of readers who are involved in the purchasing process. They also position themselves in all major trade shows and conduct flagship events every year to bring niche footfall and innovators together 5. These are some of the sources that I used to find out about their team and readership. You can check them yourself if you want to. I hope this answers your question.

Back to the 1990s

As Microsoft and Google tout the new Bing and Bard as the biggest advances in the software world, there's a strong sense of deja vu with a lot of it harking back to the 1990s when technologist Garrett Gruener and David Warthen came up with a website called AskJeeves (which still exists as ask.com). The idea behind that website was to have people ask questions in natural language and for the search engine to come up with answers.

In many ways, AI Bing reminds me of that website. Here’s a screenshot of AskJeeves from March 2000 (extracted from Wayback Machine).

AskJeeves wasn’t a chatbot, and it didn’t exactly summarise information the way this AI does. But they share one commonality — they were wrong quite often, and you wouldn’t know that unless you already knew the answer to the question you were asking.

Here’s another question I asked Bing:

Question: What are India’s new IT rules for regulating online gaming?

Answer: India's new IT rules on online gaming are draft amendments to the IT (Intermediary Guidelines and Digital Media Ethics Code Rules 2021, which also regulate social media platforms 1 2 3. The government has proposed these amendments to bring online real money gaming platforms under the ambit of IT rules and to ensure a self-regulatory mechanism, mandatory verification of players, and physical Indian address for online gaming companies

The draft amendments also envisage that an online gaming intermediary shall observe the due diligence required under the rules while discharging its duties, including reasonable efforts to cause its users not to host, display, upload, publish, transmit, or share an online game not in conformity with Indian law, including any law on gambling or betting The government has invited public comments on the draft amendments till January 31, 2023

This is a summary of India's new IT rules on online gaming based on the information that I found on the web. I hope this helps you understand the topic better.

A little later in this conversation, Bing told me that the government will form a self-regulatory body for regulating gaming, which again is incorrect. When I asked the AI where it found this information, it pointed me to the MeitY website that holds the draft regulations. The source was correct, but it understood the language of the IT rules incorrectly.

So, what should we make of this?

To be sure, AI Bing and Microsoft have both warned that its chatbot could be wrong on occasion. While it’s wrong a little more than that, there’s something to this novel new way of searching the Internet.

AI Bing can answer simple questions easily. It’s good for current affairs, as long as the user is asking the right questions. For instance, I asked what the score was in the City vs Villa match that was going on as I chatted with the AI. It didn’t just tell me the score, but added the fact that Manchester City scored in the first half, while Villa pulled one back in the second, and that the match was still ongoing.

Before telling me the score it surmised that I was talking about the men’s English Premier League match and not a similar women’s match that also happened on February 12.

It’s also pretty well-guarded against things like pornography, violent content and racism. At one point, the bot simply told me that if I try to get such content out of it, it will simply stop talking to me and report me to Microsoft.

The first versions of Google, AskJeeves, Yahoo Search, and even Bing got one thing right. They did what they were meant to -— find information. And they kept on improving with time. We hope the new Bing and Bard will do this much faster.