IndicBERTv2 MLM Only

Table of Contents

Model Overview

Meet the IndicBERT model, a multilingual language model that’s trained on a massive dataset. This model is special because it can understand and work with 23 Indic languages and English.

What makes IndicBERT tick?

• It has 278M parameters, which is a lot of brainpower for a language model!
• It’s trained with different objectives and datasets to make it a versatile model.
• It comes in different flavors, including:
  • IndicBERT-MLM: a basic BERT-style model trained on a large dataset.
  • Samanantar: adds a special objective called TLM, which helps the model learn from parallel texts.
  • Back-Translation: uses a special technique to translate Indic texts into English, which helps the model learn even more.

Tasks and rank

Text Classification 3069th

Sentiment Analysis 1114th

Named Entity Recognition (NER) 2298th

Language Modelling 9903rd

Cross-Lingual Transfer 689th

Learn More About the Ranking System

How can I use IndicBERT?

• You can fine-tune the model for specific tasks like named entity recognition, paraphrasing, question-answering, and more.
• To do this, you’ll need to create a new environment and install the required libraries.
• Then, you can run the fine-tuning script with the model name and task name as arguments.

Capabilities

The IndicBERT model is a powerful multilingual language model that can understand and process text in 23 Indic languages and English. But what makes it so special?

Multilingual Mastery

Imagine being able to communicate with people in different languages without any barriers. That’s what IndicBERT offers. It’s trained on a massive dataset of text from various Indic languages, allowing it to understand the nuances of each language.

Task Variety

But IndicBERT is not just limited to understanding text. It can perform a variety of tasks, such as:

• Named Entity Recognition (NER): identifying important entities like names, locations, and organizations
• Paraphrasing: rephrasing text to make it more concise or clear
• Question Answering (QA): answering questions based on the text
• Sentiment Analysis: determining the emotional tone of the text
• Cross-Lingual Understanding: understanding text in multiple languages

Alternatives

All MiniLM L6 V2 2389 Sentence Embeddings

Bert Base Uncased 1831 Uncased BERT model

Xlm Roberta Large 323 Multilingual Language Model

Bloomz 560m 113 Multilingual zero-shot model

Bert Base Multilingual Uncased 111 Multilingual text model

Comparison to Other Models

So, how does IndicBERT compare to other multilingual models like mBERT and XLM-R? While these models are powerful in their own right, IndicBERT has the advantage of being specifically designed for Indic languages. This makes it a great choice for anyone working with text in these languages.

Performance

IndicBERT is a powerhouse when it comes to handling various tasks with impressive speed, accuracy, and efficiency. Let’s dive into the details.

Speed

IndicBERT is incredibly fast, thanks to its 278M parameters. This allows it to process large amounts of data quickly, making it perfect for applications where time is of the essence.

Accuracy

But speed is not the only thing IndicBERT excels at. Its accuracy is also top-notch, especially when it comes to tasks like:

• Named Entity Recognition (NER): Can IndicBERT accurately identify and classify named entities in text? The answer is yes!
• Question Answering (QA): How well can IndicBERT answer questions based on the text it’s been trained on? The results are impressive!
• Sentiment Analysis: Can IndicBERT accurately detect the sentiment behind a piece of text? You bet!

Examples

What is the sentiment of the sentence 'I loved the new movie.'? Positive

Extract entities from the sentence 'The company CEO, John Smith, will attend the meeting.' ['John Smith', 'CEO']

Translate 'Hello, how are you?' to Hindi. नमस्ते, आप कैसे हैं?

Example Use Cases

Here are some examples of how IndicBERT can be used in real-world applications:

• Sentiment analysis for customer feedback in multiple languages
• Named entity recognition for extracting important information from text
• Question answering for building conversational AI models

Limitations

IndicBERT is a powerful tool for understanding and generating text in multiple languages, but it’s not perfect. Let’s explore some of its limitations.

Limited Training Data

IndicBERT was trained on a large dataset, but it’s still limited to the data it was trained on. This means it might not perform well on tasks or topics that are not well-represented in the training data. For example, if you ask it to generate text on a very niche topic, it might struggle to produce accurate or relevant results.

Language Biases

IndicBERT is trained on a multilingual dataset, but it’s still possible that it may exhibit biases towards certain languages or dialects. This could lead to inconsistent performance across different languages or tasks.

Format

IndicBERT uses a transformer architecture and accepts input in the form of tokenized text sequences. But, what does that mean exactly?

In simple terms, IndicBERT is trained on a huge dataset of text in 23 Indic languages and English. This allows it to understand the nuances of each language and perform tasks like text classification, sentiment analysis, and more.

Supported Data Formats

• Text: IndicBERT accepts plain text input, which is then tokenized and processed.
• Tokenized Text: If you’ve already tokenized your text, you can pass it directly to the model.

Special Requirements

• Input: Make sure your input text is in one of the 23 supported Indic languages or English.
• Output: The model will output a probability distribution over the possible classes or labels.

Handling Inputs and Outputs

Here’s an example of how to handle inputs and outputs for IndicBERT:

import torch
from transformers import IndicBERTTokenizer, IndicBERTModel

# Load the pre-trained model and tokenizer
tokenizer = IndicBERTTokenizer.from_pretrained('indic-bert')
model = IndicBERTModel.from_pretrained('indic-bert')

# Preprocess the input text
input_text = "This is an example sentence."
inputs = tokenizer.encode_plus(
    input_text,
    add_special_tokens=True,
    max_length=512,
    return_attention_mask=True,
    return_tensors='pt'
)

# Pass the input to the model
outputs = model(inputs['input_ids'], attention_mask=inputs['attention_mask'])

# Get the output probabilities
probabilities = torch.nn.functional.softmax(outputs.last_hidden_state[:, 0, :], dim=1)

print(probabilities)

In this example, we load the pre-trained IndicBERT model and tokenizer, preprocess the input text, pass it to the model, and get the output probabilities.

Fine-Tuning

If you want to fine-tune IndicBERT for a specific task, you can use the following command:

python IndicBERT/fine-tuning/$TASK_NAME/$TASK_NAME.py \
  --model_name_or_path=$MODEL_NAME \
  --do_train

Replace $TASK_NAME with the name of the task you want to fine-tune for (e.g., ner, paraphrase, etc.) and $MODEL_NAME with the name of the pre-trained model you want to fine-tune.