T5 Base Dutch

Table of Contents

Model Overview

The Current Model is a Dutch language model designed for natural language processing tasks. It was created to process and understand Dutch language inputs.

Key Attributes

• Model Type: T5
• Parameters: 222M
• Pre-training Objective: Masked language modeling (denoise token span corruption)
• Dataset: mc4_nl_cleaned (Dutch mC4)
• Sequence Length: 512
• Batch Size: 128
• Total Steps: 527500
• Epochs: 1
• Duration: 2d9h

Functionalities

• Language Understanding: The model is designed to understand and process Dutch language inputs.
• Text Generation: The model can be fine-tuned for text generation tasks such as summarization and translation.
• Downstream Tasks: The model can be used for various downstream tasks such as language translation, text summarization, and more.

Tasks and rank

Text Generation 12705th

Text Classification 3906th

Summarization 811th

Machine Translation 505th

Language Modelling 11127th

Learn More About the Ranking System

Capabilities

The Current Model is capable of performing a variety of tasks, including:

• Text Summarization: The model can summarize long pieces of text into shorter, more digestible versions.
• Translation: The model can translate text from one language to another, with a focus on Dutch and English.
• Masked Language Modeling: The model can fill in missing words or phrases in a sentence.

Strengths

• High Accuracy: The model has been fine-tuned to achieve high accuracy on a variety of tasks.
• Efficient Training: The model was trained on a large dataset, but with a relatively small number of parameters.
• Flexibility: The model can be used for a variety of tasks, including text summarization, translation, and more.

Examples

Vertaal het Engelse zin 'The cat sat on the mat' naar het Nederlands. De kat zat op het kleed.

Summariseer het volgende artikel: 'De afgelopen week is de temperatuur in Nederland flink gestegen. De temperaturen zijn in enkele dagen tijd met meer dan 10 graden gestegen. Dit is een ongekende stijging.' De temperatuur in Nederland is de afgelopen week flink gestegen.

Vertaal het Nederlandse zin 'De hond liep om de hoek' naar het Engels. The dog walked around the corner.

Comparison to Other Models

The Current Model is not the only model that has been trained on Dutch text. Let’s compare its performance to other models:

Model	Rouge1 Score	Bleu Score
Current Model	33.38	45.88
==t5-v1.1-base-dutch-uncased==	33.97	51.21
==t5-v1.1-base-dutch-cased==	34.39	48.31

Alternatives

NeMo SFT with Llama 2 10500 Fine-tuned language model

All MiniLM L6 V2 2389 Sentence Embeddings

Bert Base Uncased 1831 Uncased BERT model

Bloomz 560m 113 Multilingual zero-shot model

Qwen2.5 1.5B Instruct 51 Multilingual instruct model

Fine-Tuning

The Current Model can be fine-tuned for specific tasks, such as language translation. In fact, two models, t5-small-24L-dutch-english and t5-base-36L-dutch-english, have been fine-tuned for both language directions on the CCMatrix dataset. The results are impressive:

Model	Source Language	Target Language	Bleu Score
t5-base-36L-ccmatrix-multi	English	Dutch	56.8
t5-base-36L-ccmatrix-multi	Dutch	English	62.8
t5-small-24L-ccmatrix-multi	English	Dutch	57.4
t5-small-24L-ccmatrix-multi	Dutch	English	63.1

Limitations

The Current Model has several limitations that are important to consider when using it for downstream tasks.

Limited Training Data

The model was trained on a limited dataset, which may not be representative of all possible scenarios. This can lead to biases and inaccuracies in the model’s outputs.

Lack of Common Sense

While the Current Model is good at understanding language, it sometimes lacks common sense or real-world experience. This can result in outputs that are not practical or realistic.

Inability to Reason Abstractly

The model is not capable of abstract reasoning or complex problem-solving. It is best suited for tasks that require language understanding and generation.

Format

The Current Model uses a transformer architecture, specifically designed for text-to-text tasks. This model accepts input in the form of tokenized text sequences.

Input Format

• The model expects input text to be pre-processed using a SentencePiece tokenizer.
• The tokenizer is configured with the following normalizers:
  • Nmt
  • NFKC
  • Replace multi-space to single-space
• The model has a vocabulary size of 32,003 tokens.

Output Format

• The model generates output in the form of tokenized text sequences.
• The output can be converted back to plain text using the same SentencePiece tokenizer.

Special Requirements

• The model requires a specific sequence length of 512 tokens.
• The model uses a batch size of 128.

Example Code

Here’s an example of how to use the Current Model in Python:

import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

# Load pre-trained model and tokenizer
tokenizer = T5Tokenizer.from_pretrained('t5-base-dutch')
model = T5ForConditionalGeneration.from_pretrained('t5-base-dutch')

# Pre-process input text
input_text = "Dit is een voorbeeld zin."
inputs = tokenizer.encode_plus(input_text, 
                                add_special_tokens=True, 
                                max_length=512, 
                                return_attention_mask=True, 
                                return_tensors='pt')

# Generate output
outputs = model.generate(inputs['input_ids'], 
                          attention_mask=inputs['attention_mask'], 
                          max_length=512)

# Convert output to plain text
output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(output_text)