How Indian LLMs are gaining momentum

As global attention focuses on the intense competition in the realm of large language models (LLMs), Indian companies are also actively engaging in this space by announcing their ambitious plans and showing unique capabilities. Despite some uncertainties, India's recent progress in LLM development paints a promising picture. The scepticism expressed by OpenAI CEO Sam Altman during his visit to India last year regarding the country's competitiveness in LLMs ignited a sense of determination among Indian tech leaders.

As India approaches its 78th Independence Day on August 15, and with significant advancements in science and technology, we explore the nation’s standing in the LLM landscape.

India's LLM moment

There’s no doubt that India does not figure in the world of LLMs, which is almost entirely dominated by the USA and China. However, Pranav Mistry, founder and CEO of Two Platforms believes that Indian companies are launching AI LLMs in local languages and that’s where they can differentiate.

“Indian startups, working on fine-tuning the existing models, can drive this change by rethinking their approach and developing new multi-lingual models that do not need training from the ground up can help it to become the leader of AI in non-English markets,” he said at the Mint Digital Innovation Summit 2024 held in May in Mumbai.

There are several such examples. In December 2023, Ola co-founder Bhavish Aggarwal entered the Indian LLM market with Krutrim Si Designs, introducing "Krutrim," which means "artificial" in Sanskrit. Krutrim’s foundational model trained on 2 trillion tokens and specialised datasets, understands 22 Indian languages and generates responses in 10, claiming superiority over OpenAI's GPT-4 in handling Indic languages.

Read more: How LLM adoption has impacted AI job roles?

CoRover.ai, the company behind BharatGPT, developed this language model in collaboration with Bhashini under the National Language Translation Mission. Bhashini is an AI-driven language translation tool that marks a transformative step in digital inclusivity for Indian languages. Known for its human-centric conversational AI, it supports over 12 Indian languages and provides AI-driven virtual assistants to organisations like IRCTC and the Government of India, serving over 1 billion users.

Another LLM, OpenHathi 7B Hi, developed by Sarvam AI, is touted to compete with GPT-3.5 for Indic language performance. It employs a bilingual training process and is built on Meta’s Llama2-7B. OpenHathi 7B Hi surpassed GPT-3.5 and GPT-4 in translating Devanagari Hindi to English, as measured by the Bilingual Evaluation Understudy metric.

Additionally, Tech Mahindra launched ‘Project Indus', an LLM focused on Indic languages, with an initial training phase in Hindi and 37 dialects. This model is designed to be culturally sensitive, helping to preserve these languages and addressing areas like education and rural financing.

Besides, Nandan Nilekani backed and based in IIT Madras, AI4Bharat develops open-source language AI for Indian languages, including IndicBart, which supports 11 Indian languages and is trained on a vast corpus of content. Furthermore, Two Platforms, Mistry's Silicon Valley-based deep tech startup, backed by Mukesh Ambani’s Jio Platforms and South Korea’s Naver Corp., released Sutra, a multilingual large language model designed specifically for the Indian market. These are only just a few examples.

Overcoming the hurdles

However, significant challenges persist. Most of the online content is in English, making it difficult for many to engage with the digital realm. Kesava Reddy, CRO at E2E Networks, notes that LLMs struggle with the complex structures of Indian languages, emphasising the need for tailored models. Reddy noted that LLMs encounter difficulties with Indian languages, as they generally exhibit superior performance in English.

Additionally, developers of Indian LLMs must address issues related to inadequacy, scalability, and access gaps. S Anjani Kumar, Partner, Consulting, Deloitte India, highlighted challenges such as sourcing high-quality training data and achieving a balance between accuracy and computational efficiency.

Further, he said, that establishing guidelines and regulations for the ethical development and deployment of LLMs is necessary to address issues related to data privacy, security, and potential biases in the output. The Trustworthy AI framework also aims to help businesses increase brand equity and trust, which can lead to new customers, employee retention, and more customers opting to share data.

The Ministry of Electronics and Information Technology (MeitY) has reminded LLM developers of their regulatory responsibilities, requiring government permission for deploying untested AI models and ensuring transparency about their reliability.

The silver lining

Experts believe that the successful deployment of robust Indian LLMs relies on collaboration between researchers, developers, and businesses to address the unique requirements of the Indian market.

Moving ahead in 2024 and beyond, more companies will create India-specific LLMs that tackle local challenges, promoting inclusivity and benefiting regional economies. These models will support applications and specifically tailored services in education, health, finance, and governance, aligning with India's vision for a digital society enhanced by AI.

Already local governments and businesses are encouraged to adopt Indian LLMs. For instance, IRCTC’s AskDisha chatbot has effectively managed over 150,000 passenger inquiries daily with 90% accuracy, while Indraprastha Gas’s Ask Maitri chatbot has reduced complaint calls by over 35% within six weeks of launch. Asian Tobacco Company’s Ask ATC bot, supporting vernacular languages, has decreased machine downtime by more than 30%.

By taking a calculated and collaborative approach, India can leverage the power of AI and build indigenous LLMs to drive its economic and social development while ensuring that the benefits are equitably distributed across the country.