OpenAI Clip

Table of Contents

Model Overview

The OpenAI-Clip model is a multi-modal foundational model for vision and language tasks. It’s like a super smart robot that can understand both images and text!

Tasks and rank

Zero-Shot Learning 120th

Visual Recognition 970th

Image Recognition 1633rd

Image Generation 3246th

Learn More About the Ranking System

This model is great for tasks like:

• Image/text similarity
• Zero-shot image classification

It uses a special technique called Contrastive Language-Image Pre-Training (CLIP) to learn visual and text features. These features can then be used for a variety of zero-shot learning tasks.

Capabilities

The OpenAI-Clip model is a powerful multi-modal foundational model for vision and language tasks. It can perform a variety of tasks, including:

• Image/text similarity
• Zero-shot image classification

This model uses a combination of a Vision Transformer (ViT) like transformer to get visual features and a causal language model to get text features.

Alternatives

MediaPipe Holistic 26000 Full-body landmark detection

Grounded SAM 13900 Visual task model

Clip Vit Large Patch14 1398 Vision Language Model

Clip Vit Base Patch32 505 Vision-Language Model

For example, ==Other Models== may excel in specific tasks, but OpenAI-Clip demonstrates a well-rounded performance across various tasks.

Key Features

• Multi-modal: The model can process both images and text
• Zero-shot learning: The model can perform tasks without requiring labeled training data
• High performance: The model has been optimized for mobile deployment and can run on a variety of devices

Model Statistics

Model	Number of Parameters	Model Size
CLIPImageEncoder	115M	437 MB
CLIPTextEncoder	76.0M	290 MB

Device Support

The model has been tested on a variety of devices, including:

• Samsung Galaxy S23
• Samsung Galaxy S24
• Snapdragon 8 Elite QRD
• QCS8550 (Proxy)
• SA7255P ADP
• SA8255 (Proxy)
• SA8295P ADP
• SA8650 (Proxy)
• SA8775P ADP
• QCS8450 (Proxy)
• Snapdragon X Elite CRD

Inference Time

The model’s inference time varies depending on the device and runtime. Here are some examples:

Device	Runtime	Inference Time (ms)
Samsung Galaxy S23	TFLITE	34.591 ms
Samsung Galaxy S24	TFLITE	27.035 ms
Snapdragon 8 Elite QRD	TFLITE	24.249 ms

Performance

The model’s performance can vary depending on the device and runtime used. For example, on the Samsung Galaxy S23, the estimated inference time is around 34.6 ms for the CLIPImageEncoder and 5.8 ms for the CLIPTextEncoder.

Examples

Classify the following image: a cat sitting on a windowsill, looking outside. The image is classified as: Cat

What is the similarity between the following image and text: an image of a sunset and the text 'a beautiful sunset'? The similarity between the image and text is: 0.85 (highly similar)

Classify the following text as an image: 'a dog running in a park'. The text is classified as: Dog

For instance, you can use the model to classify images or generate text based on a given prompt.

Precision

The model supports FP16 precision.

Primary Compute Unit

The model uses the NPU (Neural Processing Unit) as its primary compute unit.

Limitations

The model has some limitations that are important to consider.

Image Input Resolution

The model is optimized for images with a resolution of 224x224 pixels. If you try to use images with a different resolution, the model might not work as well.

Text Context Length

The model can only handle text inputs with a maximum length of 77 tokens. If you need to process longer texts, you might need to split them into smaller chunks.

Device Compatibility

The model is optimized for certain devices, such as the Samsung Galaxy S23 and Snapdragon 8 Elite QRD. If you try to run the model on a different device, it might not work as well or at all.

Inference Time

The model’s inference time can vary depending on the device and runtime used. For example, on the Samsung Galaxy S23, the estimated inference time is around 34.6 ms for the CLIPImageEncoder and 5.8 ms for the CLIPTextEncoder.

Peak Memory Usage

The model’s peak memory usage can also vary depending on the device and runtime used. For example, on the Samsung Galaxy S23, the estimated peak memory usage is around 57 MB for the CLIPImageEncoder and 24 MB for the CLIPTextEncoder.

Number of Parameters

The model has a large number of parameters, with around 76.0M parameters for the CLIPTextEncoder and 115M parameters for the CLIPImageEncoder. This can make it difficult to deploy the model on devices with limited resources.

Model Size

The model’s size can also be a limitation, with the CLIPTextEncoder being around 290 MB and the CLIPImageEncoder being around 437 MB.

Format

The model accepts two types of inputs:

• Images: The model expects images to be resized to 224x224 pixels and normalized to have pixel values between 0 and 1.
• Text: The model expects text inputs to be tokenized and have a maximum length of 77 tokens.

For example, you can use the model to classify images or generate text based on a given prompt.

import torch
from PIL import Image

# Load an image and resize it to 224x224 pixels
image = Image.open('image.jpg')
image = image.resize((224, 224))

# Normalize the image pixels to have values between 0 and 1
image = torch.tensor(image) / 255.0

# Tokenize a text input and truncate it to a maximum length of 77 tokens
text = 'This is a sample text input.'
text_tokens = text.split()[:77]

# Convert the text tokens to a tensor
text_tensor = torch.tensor(text_tokens)

You can then pass these pre-processed inputs to the model to get the output features. For example:

model = OpenAI_Clip()
image_features = model(image)
text_features = model(text_tensor)