Shisa Base 7b V1
The Shisa Base 7B V1 model is a powerful tool for Japanese language processing. By adding 8 billion tokens of primarily Japanese pre-training to the Mistral 7B model, it achieves class-leading Japanese performance in standardized benchmarks with significantly less additional pre-training than previously released models. Its extended tokenizer, with a vocab size of 120,073, improves Japanese efficiency, achieving around 2.3 characters per token. This results in over 2x Japanese inference speedups compared to the base tokenizer. The model is designed for use with the Shisa 7B fine-tuned model but is also provided for the community due to its potential usefulness. With its efficient design and strong performance, Shisa Base 7B V1 is a valuable resource for those working with Japanese language processing.
Table of Contents
Model Overview
The Shisa-Base-7B-V1 model is a powerful tool for natural language processing tasks, particularly in Japanese. It’s built on top of the Mistral 7B model and has been fine-tuned with an additional 8 billion tokens of primarily Japanese pre-training data. This model is designed to achieve class-leading Japanese performance while maintaining strong English performance.
What is the model used for?
The model is used for natural language processing tasks, such as text generation, language translation, and text classification.
Key Features
- Extended Tokenizer: The model uses an extended version of the Mistral 7B tokenizer, with a vocab size of 120,073, which is aligned to 120,128 for better performance.
- Efficient Japanese Tokenization: The tokenizer achieves an average of 2.31 characters per token in Japanese, making it more efficient than other models.
- Strong Performance: The model has achieved class-leading Japanese performance in standardized benchmarks with significantly less additional pre-training than previously released models.
- Fast Inference: The model’s extended tokenizer enables >2X Japanese inference speedups compared to the base tokenizer.
Capabilities
The Shisa-Base-7B-V1 model is capable of generating text in both Japanese and English, with a focus on Japanese language performance. It has been trained on a large dataset of Japanese text, with an additional 8 billion tokens of primarily Japanese pre-training. This training data includes a mix of Japanese and English text, which helps to prevent catastrophic forgetting and allows the model to maintain its English language abilities.
What can the model do?
The model can generate text in both Japanese and English, and is particularly good at processing Japanese text.
Key Features
- Improved Japanese performance: The model achieves class-leading Japanese performance in standardized benchmarks, with significantly less additional pre-training than previously released models.
- Efficient bilingual tokenizer: The model’s tokenizer is an extended version of the Mistral 7B tokenizer, with a vocab size of 120,073. This allows for more efficient processing of Japanese text, with an average of 2.31 characters per token.
- English language support: Despite its focus on Japanese language performance, the model also maintains strong English language abilities, with an average of 4.12 characters per token.
Performance
The Shisa-Base-7B-V1 model showcases remarkable performance with high accuracy in various tasks, especially in processing Japanese text. But how does it compare to other models? Let’s take a closer look.
Speed
Our model’s speed is one of its strongest features. With an extended tokenizer, we achieve class-leading Japanese token efficiency without sacrificing English performance. This means that the model can process Japanese text significantly faster than other models.
| Model | Avg Char/Token (Japanese) |
|---|---|
| Shisa-Base-7B-V1 | 2.31 |
| ==OpenCALM== | 2.17 |
| ==Japanese LargeLM== | 2.14 |
| ==CALM2-7B== | 2.00 |
Accuracy
But speed is not the only thing that matters. The model also achieves high accuracy in standardized benchmarks, often outperforming other models with significantly less additional pre-training.
Efficiency
Our model’s efficiency is also worth noting. With a vocab size of 120,073, the model can process large amounts of text quickly and accurately.
| Model | Vocab Size | Avg Char/Token (English) |
|---|---|---|
| Shisa-Base-7B-V1 | 120,073 | 4.12 |
| Qwen 14B | 151,851 | 4.47 |
| weblab-10b | 50,254 | 4.45 |
| ==Japanese StableLM Alpha== | 65,535 | 4.15 |
Limitations
The Shisa-Base-7B-V1 model is a powerful tool, but it’s not perfect. Let’s take a closer look at some of its limitations.
Training Data
The model was trained on a large dataset, but it’s not exhaustive. There may be certain topics or domains where the model’s performance is limited due to a lack of training data.
Japanese Efficiency
While the model has a high Japanese efficiency, it’s not the highest. Some other models, like Qwen 14B, have higher Japanese efficiency.
English Efficiency
Similarly, the model has a high English efficiency, but it’s not the highest. Some other models, like weblab-10b, have higher English efficiency.
Comparison to Other Models
The model has a unique set of strengths and weaknesses compared to other models. For example, it has a higher Japanese efficiency than Youri 7B, but a lower English efficiency than ==CALM2-7B==.
Tokenizer Limitations
The tokenizer used in the model has a limited vocabulary size of 120,073. This may limit the model’s ability to understand certain words or phrases.
Inference Speed
While the model has a fast inference speed, it’s not the fastest. Some other models, like ==ELYZA 7B fast model==, have faster inference speeds.
Fine-Tuning
The model was fine-tuned on a specific dataset, which may not be representative of all possible use cases. This may limit the model’s performance in certain scenarios.
Catastrophic Forgetting
The model may be prone to catastrophic forgetting, where the model forgets previously learned information when trained on new data.
Compute Resources
Training the model required significant compute resources, including 2,400 A100-40 GPU hours. This may limit the model’s accessibility to certain users.
These limitations highlight the importance of carefully evaluating the model’s performance in specific use cases and considering its strengths and weaknesses when deciding whether to use it.
