Google’s new AI tools will allow you to Search for what you see

Technology giant Google is using artificial intelligence (AI) to put smartphone cameras at the center of its Search platform. In a blog post today, the company said it has reached a critical milestone in understanding the Multimask Unified Model (MUM), which is an AI powered system that essentially allows people to Search for what they see in front of them. Google said it will be introducing ways to “search visually” in the coming months “with the ability to ask questions about what you see”. MUM was a system Google first unveiled at the I/O conference earlier this year, and it essentially extends the Google Lens feature that the company has been promoting for some years now.

With the new features, Google will allow users to tap the Lens icon on Search and ask Google to look for patterns on the Internet. This includes things like searching for an article of clothing, which could be useful for shopping online. “You could type ‘white floral Victorian socks,’ but you might not find the exact pattern you’re looking for. By combining images and text into a single query, we’re making it easier to search visually and express your questions in more natural ways,” the company said in its blog post.

Shopping aside, the new feature could also be used to use Google Search for videos. The company says users will be able to use the point-and-ask mode to find the exact moment in a video that’s relevant to how they can fix their broken bike. Which is another feature that sounds very useful for people who search for how to fix things on Google, or how to cook a particular recipe, etc.

AI advances like MUM will also help the company redesign Search. It will add a new “things to know” section, which will offer more information on a search topic. “If you search for ‘acrylic painting,’ Google understands how people typically explore this topic, and shows the aspects people are likely to look at first. For example, we can identify more than 350 topics related to acrylic painting, and help you find the right path to take,” the company said in the post.

Further, MUM will also be used to provide a new “browsable results page”, which will include images, videos, articles and more about a search topic. Google says this new page is designed for those looking for “inspiration”, like ‘Halloween decorating ideas’ or ‘indoor vertical garden ideas’. The company can also link directly to specific points in a video and understand that a search topic is related to a video, even if the specific words aren’t mentioned in the video itself.