|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- TNILab/DomainNet_FL |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- google/siglip2-base-patch16-224 |
|
|
pipeline_tag: image-classification |
|
|
library_name: transformers |
|
|
tags: |
|
|
- Multisource-121-DomainNet |
|
|
--- |
|
|
|
|
|
 |
|
|
|
|
|
# **Multisource-121-DomainNet** |
|
|
|
|
|
> **Multisource-121-DomainNet** is an image classification vision-language encoder model fine-tuned from **google/siglip2-base-patch16-224** for a single-label classification task. It is designed to classify images into 121 domain categories using the **SiglipForImageClassification** architecture. |
|
|
|
|
|
 |
|
|
|
|
|
*Moment Matching for Multi-Source Domain Adaptation* : https://arxiv.org/pdf/1812.01754 |
|
|
|
|
|
*SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features* https://arxiv.org/pdf/2502.14786 |
|
|
|
|
|
```py |
|
|
Classification Report: |
|
|
precision recall f1-score support |
|
|
|
|
|
barn 0.7483 0.8370 0.7902 270 |
|
|
baseball_bat 0.9197 0.9333 0.9265 270 |
|
|
basket 0.8302 0.8148 0.8224 270 |
|
|
beach 0.7059 0.7556 0.7299 270 |
|
|
bear 0.7500 0.7444 0.7472 270 |
|
|
beard 0.5496 0.5741 0.5616 270 |
|
|
bee 0.9004 0.9037 0.9020 270 |
|
|
bird 0.7352 0.7815 0.7576 270 |
|
|
blueberry 0.7230 0.7926 0.7562 270 |
|
|
bowtie 0.8726 0.8370 0.8544 270 |
|
|
bracelet 0.7328 0.7111 0.7218 270 |
|
|
brain 0.8925 0.9222 0.9071 270 |
|
|
bread 0.5573 0.6667 0.6071 270 |
|
|
broccoli 0.9200 0.7667 0.8364 270 |
|
|
bus 0.8442 0.8630 0.8535 270 |
|
|
butterfly 0.9321 0.9148 0.9234 270 |
|
|
circle 0.6038 0.8185 0.6950 270 |
|
|
cloud 0.8201 0.8444 0.8321 270 |
|
|
cruise_ship 0.8545 0.8481 0.8513 270 |
|
|
dolphin 0.8286 0.8593 0.8436 270 |
|
|
dumbbell 0.8705 0.8963 0.8832 270 |
|
|
elephant 0.8598 0.8630 0.8614 270 |
|
|
eye 0.8603 0.8667 0.8635 270 |
|
|
eyeglasses 0.8425 0.7926 0.8168 270 |
|
|
feather 0.8413 0.7852 0.8123 270 |
|
|
fish 0.8169 0.8593 0.8375 270 |
|
|
flower 0.7973 0.8741 0.8339 270 |
|
|
foot 0.8152 0.8333 0.8242 270 |
|
|
frog 0.9270 0.8000 0.8588 270 |
|
|
giraffe 0.9026 0.8926 0.8976 270 |
|
|
goatee 0.5171 0.5037 0.5103 270 |
|
|
golf_club 0.6466 0.6778 0.6618 270 |
|
|
grapes 0.8731 0.8407 0.8566 270 |
|
|
grass 0.7359 0.6296 0.6786 270 |
|
|
guitar 0.8386 0.8852 0.8613 270 |
|
|
hamburger 0.8535 0.8630 0.8582 270 |
|
|
hand 0.7824 0.6926 0.7348 270 |
|
|
hat 0.7333 0.7741 0.7532 270 |
|
|
headphones 0.8971 0.9037 0.9004 270 |
|
|
helicopter 0.8992 0.8259 0.8610 270 |
|
|
hexagon 0.9113 0.8370 0.8726 270 |
|
|
hockey_stick 0.8419 0.8481 0.8450 270 |
|
|
horse 0.8081 0.8889 0.8466 270 |
|
|
hourglass 0.9161 0.9296 0.9228 270 |
|
|
house 0.7524 0.8778 0.8103 270 |
|
|
ice_cream 0.8821 0.8593 0.8705 270 |
|
|
jacket 0.8621 0.7407 0.7968 270 |
|
|
ladder 0.7051 0.8148 0.7560 270 |
|
|
leg 0.5916 0.5741 0.5827 270 |
|
|
lipstick 0.8889 0.8000 0.8421 270 |
|
|
megaphone 0.8710 0.9000 0.8852 270 |
|
|
monkey 0.8370 0.8556 0.8462 270 |
|
|
moon 0.8527 0.8148 0.8333 270 |
|
|
mushroom 0.8774 0.8481 0.8625 270 |
|
|
necklace 0.8670 0.7481 0.8032 270 |
|
|
owl 0.9179 0.9111 0.9145 270 |
|
|
panda 0.9490 0.8963 0.9219 270 |
|
|
pear 0.8832 0.8963 0.8897 270 |
|
|
peas 0.7743 0.8259 0.7993 270 |
|
|
penguin 0.8618 0.8778 0.8697 270 |
|
|
pig 0.6767 0.8296 0.7454 270 |
|
|
pillow 0.7359 0.6296 0.6786 270 |
|
|
pineapple 0.9213 0.9111 0.9162 270 |
|
|
pizza 0.9173 0.9444 0.9307 270 |
|
|
pool 0.6717 0.6593 0.6654 270 |
|
|
popsicle 0.7390 0.8074 0.7717 270 |
|
|
rabbit 0.8345 0.8778 0.8556 270 |
|
|
rhinoceros 0.9219 0.9185 0.9202 270 |
|
|
rifle 0.9256 0.8296 0.8750 270 |
|
|
river 0.6067 0.7370 0.6656 270 |
|
|
sailboat 0.8606 0.9148 0.8869 270 |
|
|
sandwich 0.7638 0.7667 0.7652 270 |
|
|
sea_turtle 0.8794 0.9185 0.8986 270 |
|
|
shark 0.8114 0.8444 0.8276 270 |
|
|
shoe 0.8097 0.8667 0.8372 270 |
|
|
skyscraper 0.7727 0.8185 0.7950 270 |
|
|
snorkel 0.8238 0.6926 0.7525 270 |
|
|
snowman 0.8736 0.8444 0.8588 270 |
|
|
soccer_ball 0.9395 0.8630 0.8996 270 |
|
|
speedboat 0.7649 0.7593 0.7621 270 |
|
|
spider 0.9212 0.8222 0.8689 270 |
|
|
spoon 0.8165 0.8074 0.8119 270 |
|
|
square 0.4669 0.6259 0.5348 270 |
|
|
squirrel 0.8394 0.7741 0.8054 270 |
|
|
stethoscope 0.8566 0.8630 0.8598 270 |
|
|
strawberry 0.8629 0.7926 0.8263 270 |
|
|
streetlight 0.5000 0.6852 0.5781 270 |
|
|
submarine 0.6850 0.6926 0.6888 270 |
|
|
suitcase 0.8259 0.7556 0.7892 270 |
|
|
sun 0.8082 0.6556 0.7239 270 |
|
|
sweater 0.5912 0.6963 0.6395 270 |
|
|
sword 0.8258 0.8074 0.8165 270 |
|
|
table 0.5502 0.5481 0.5492 270 |
|
|
teapot 0.9019 0.8852 0.8935 270 |
|
|
teddy-bear 0.7906 0.8111 0.8007 270 |
|
|
telephone 0.7836 0.7778 0.7807 270 |
|
|
tent 0.7579 0.7074 0.7318 270 |
|
|
The_Eiffel_Tower 0.8633 0.8889 0.8759 270 |
|
|
The_Great_Wall_of_China 0.8893 0.8333 0.8604 270 |
|
|
The_Mona_Lisa 0.8152 0.9148 0.8621 270 |
|
|
tiger 0.8577 0.8259 0.8415 270 |
|
|
toaster 0.6788 0.6889 0.6838 270 |
|
|
tooth 0.8807 0.7926 0.8343 270 |
|
|
tornado 0.7530 0.7000 0.7255 270 |
|
|
tractor 0.9372 0.8296 0.8802 270 |
|
|
train 0.7692 0.7407 0.7547 270 |
|
|
tree 0.7639 0.8148 0.7885 270 |
|
|
triangle 0.8852 0.8000 0.8405 270 |
|
|
trombone 0.6653 0.5963 0.6289 270 |
|
|
truck 0.7049 0.7963 0.7478 270 |
|
|
trumpet 0.7463 0.5667 0.6442 270 |
|
|
umbrella 0.9144 0.8704 0.8918 270 |
|
|
vase 0.8148 0.7333 0.7719 270 |
|
|
violin 0.8966 0.7704 0.8287 270 |
|
|
watermelon 0.7970 0.8000 0.7985 270 |
|
|
whale 0.7769 0.6963 0.7344 270 |
|
|
windmill 0.8963 0.8963 0.8963 270 |
|
|
wine_glass 0.8996 0.8630 0.8809 270 |
|
|
yoga 0.7406 0.8037 0.7709 270 |
|
|
zebra 0.9144 0.7519 0.8252 270 |
|
|
zigzag 0.6502 0.6333 0.6417 270 |
|
|
|
|
|
accuracy 0.7995 32670 |
|
|
macro avg 0.8052 0.7995 0.8006 32670 |
|
|
weighted avg 0.8052 0.7995 0.8006 32670 |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The model categorizes images into the following 121 classes: |
|
|
- **Class 0:** "barn" |
|
|
- **Class 1:** "baseball_bat" |
|
|
- **Class 2:** "basket" |
|
|
- **Class 3:** "beach" |
|
|
- **Class 4:** "bear" |
|
|
- **Class 5:** "beard" |
|
|
- **Class 6:** "bee" |
|
|
- **Class 7:** "bird" |
|
|
- **Class 8:** "blueberry" |
|
|
- **Class 9:** "bowtie" |
|
|
- **Class 10:** "bracelet" |
|
|
- **Class 11:** "brain" |
|
|
- **Class 12:** "bread" |
|
|
- **Class 13:** "broccoli" |
|
|
- **Class 14:** "bus" |
|
|
- **Class 15:** "butterfly" |
|
|
- **Class 16:** "circle" |
|
|
- **Class 17:** "cloud" |
|
|
- **Class 18:** "cruise_ship" |
|
|
- **Class 19:** "dolphin" |
|
|
- **Class 20:** "dumbbell" |
|
|
- **Class 21:** "elephant" |
|
|
- **Class 22:** "eye" |
|
|
- **Class 23:** "eyeglasses" |
|
|
- **Class 24:** "feather" |
|
|
- **Class 25:** "fish" |
|
|
- **Class 26:** "flower" |
|
|
- **Class 27:** "foot" |
|
|
- **Class 28:** "frog" |
|
|
- **Class 29:** "giraffe" |
|
|
- **Class 30:** "goatee" |
|
|
- **Class 31:** "golf_club" |
|
|
- **Class 32:** "grapes" |
|
|
- **Class 33:** "grass" |
|
|
- **Class 34:** "guitar" |
|
|
- **Class 35:** "hamburger" |
|
|
- **Class 36:** "hand" |
|
|
- **Class 37:** "hat" |
|
|
- **Class 38:** "headphones" |
|
|
- **Class 39:** "helicopter" |
|
|
- **Class 40:** "hexagon" |
|
|
- **Class 41:** "hockey_stick" |
|
|
- **Class 42:** "horse" |
|
|
- **Class 43:** "hourglass" |
|
|
- **Class 44:** "house" |
|
|
- **Class 45:** "ice_cream" |
|
|
- **Class 46:** "jacket" |
|
|
- **Class 47:** "ladder" |
|
|
- **Class 48:** "leg" |
|
|
- **Class 49:** "lipstick" |
|
|
- **Class 50:** "megaphone" |
|
|
- **Class 51:** "monkey" |
|
|
- **Class 52:** "moon" |
|
|
- **Class 53:** "mushroom" |
|
|
- **Class 54:** "necklace" |
|
|
- **Class 55:** "owl" |
|
|
- **Class 56:** "panda" |
|
|
- **Class 57:** "pear" |
|
|
- **Class 58:** "peas" |
|
|
- **Class 59:** "penguin" |
|
|
- **Class 60:** "pig" |
|
|
- **Class 61:** "pillow" |
|
|
- **Class 62:** "pineapple" |
|
|
- **Class 63:** "pizza" |
|
|
- **Class 64:** "pool" |
|
|
- **Class 65:** "popsicle" |
|
|
- **Class 66:** "rabbit" |
|
|
- **Class 67:** "rhinoceros" |
|
|
- **Class 68:** "rifle" |
|
|
- **Class 69:** "river" |
|
|
- **Class 70:** "sailboat" |
|
|
- **Class 71:** "sandwich" |
|
|
- **Class 72:** "sea_turtle" |
|
|
- **Class 73:** "shark" |
|
|
- **Class 74:** "shoe" |
|
|
- **Class 75:** "skyscraper" |
|
|
- **Class 76:** "snorkel" |
|
|
- **Class 77:** "snowman" |
|
|
- **Class 78:** "soccer_ball" |
|
|
- **Class 79:** "speedboat" |
|
|
- **Class 80:** "spider" |
|
|
- **Class 81:** "spoon" |
|
|
- **Class 82:** "square" |
|
|
- **Class 83:** "squirrel" |
|
|
- **Class 84:** "stethoscope" |
|
|
- **Class 85:** "strawberry" |
|
|
- **Class 86:** "streetlight" |
|
|
- **Class 87:** "submarine" |
|
|
- **Class 88:** "suitcase" |
|
|
- **Class 89:** "sun" |
|
|
- **Class 90:** "sweater" |
|
|
- **Class 91:** "sword" |
|
|
- **Class 92:** "table" |
|
|
- **Class 93:** "teapot" |
|
|
- **Class 94:** "teddy-bear" |
|
|
- **Class 95:** "telephone" |
|
|
- **Class 96:** "tent" |
|
|
- **Class 97:** "The_Eiffel_Tower" |
|
|
- **Class 98:** "The_Great_Wall_of_China" |
|
|
- **Class 99:** "The_Mona_Lisa" |
|
|
- **Class 100:** "tiger" |
|
|
- **Class 101:** "toaster" |
|
|
- **Class 102:** "tooth" |
|
|
- **Class 103:** "tornado" |
|
|
- **Class 104:** "tractor" |
|
|
- **Class 105:** "train" |
|
|
- **Class 106:** "tree" |
|
|
- **Class 107:** "triangle" |
|
|
- **Class 108:** "trombone" |
|
|
- **Class 109:** "truck" |
|
|
- **Class 110:** "trumpet" |
|
|
- **Class 111:** "umbrella" |
|
|
- **Class 112:** "vase" |
|
|
- **Class 113:** "violin" |
|
|
- **Class 114:** "watermelon" |
|
|
- **Class 115:** "whale" |
|
|
- **Class 116:** "windmill" |
|
|
- **Class 117:** "wine_glass" |
|
|
- **Class 118:** "yoga" |
|
|
- **Class 119:** "zebra" |
|
|
- **Class 120:** "zigzag" |
|
|
|
|
|
# **Run with Transformers🤗** |
|
|
|
|
|
```python |
|
|
!pip install -q transformers torch pillow gradio |
|
|
``` |
|
|
|
|
|
```python |
|
|
import gradio as gr |
|
|
from transformers import AutoImageProcessor, SiglipForImageClassification |
|
|
from transformers.image_utils import load_image |
|
|
from PIL import Image |
|
|
import torch |
|
|
|
|
|
# Load model and processor |
|
|
model_name = "prithivMLmods/Multisource-121-DomainNet" |
|
|
model = SiglipForImageClassification.from_pretrained(model_name) |
|
|
processor = AutoImageProcessor.from_pretrained(model_name) |
|
|
|
|
|
def multisource_classification(image): |
|
|
"""Predicts the domain category for an input image.""" |
|
|
# Convert the input numpy array to a PIL Image and ensure it is in RGB format |
|
|
image = Image.fromarray(image).convert("RGB") |
|
|
|
|
|
# Process the image and convert it to model inputs |
|
|
inputs = processor(images=image, return_tensors="pt") |
|
|
|
|
|
# Get model predictions without gradient calculations |
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
logits = outputs.logits |
|
|
# Convert logits to probabilities using softmax |
|
|
probs = torch.nn.functional.softmax(logits, dim=1).squeeze().tolist() |
|
|
|
|
|
# Mapping from class indices to domain labels |
|
|
labels = { |
|
|
"0": "barn", "1": "baseball_bat", "2": "basket", "3": "beach", "4": "bear", |
|
|
"5": "beard", "6": "bee", "7": "bird", "8": "blueberry", "9": "bowtie", |
|
|
"10": "bracelet", "11": "brain", "12": "bread", "13": "broccoli", "14": "bus", |
|
|
"15": "butterfly", "16": "circle", "17": "cloud", "18": "cruise_ship", "19": "dolphin", |
|
|
"20": "dumbbell", "21": "elephant", "22": "eye", "23": "eyeglasses", "24": "feather", |
|
|
"25": "fish", "26": "flower", "27": "foot", "28": "frog", "29": "giraffe", |
|
|
"30": "goatee", "31": "golf_club", "32": "grapes", "33": "grass", "34": "guitar", |
|
|
"35": "hamburger", "36": "hand", "37": "hat", "38": "headphones", "39": "helicopter", |
|
|
"40": "hexagon", "41": "hockey_stick", "42": "horse", "43": "hourglass", "44": "house", |
|
|
"45": "ice_cream", "46": "jacket", "47": "ladder", "48": "leg", "49": "lipstick", |
|
|
"50": "megaphone", "51": "monkey", "52": "moon", "53": "mushroom", "54": "necklace", |
|
|
"55": "owl", "56": "panda", "57": "pear", "58": "peas", "59": "penguin", |
|
|
"60": "pig", "61": "pillow", "62": "pineapple", "63": "pizza", "64": "pool", |
|
|
"65": "popsicle", "66": "rabbit", "67": "rhinoceros", "68": "rifle", "69": "river", |
|
|
"70": "sailboat", "71": "sandwich", "72": "sea_turtle", "73": "shark", "74": "shoe", |
|
|
"75": "skyscraper", "76": "snorkel", "77": "snowman", "78": "soccer_ball", "79": "speedboat", |
|
|
"80": "spider", "81": "spoon", "82": "square", "83": "squirrel", "84": "stethoscope", |
|
|
"85": "strawberry", "86": "streetlight", "87": "submarine", "88": "suitcase", "89": "sun", |
|
|
"90": "sweater", "91": "sword", "92": "table", "93": "teapot", "94": "teddy-bear", |
|
|
"95": "telephone", "96": "tent", "97": "The_Eiffel_Tower", "98": "The_Great_Wall_of_China", |
|
|
"99": "The_Mona_Lisa", "100": "tiger", "101": "toaster", "102": "tooth", "103": "tornado", |
|
|
"104": "tractor", "105": "train", "106": "tree", "107": "triangle", "108": "trombone", |
|
|
"109": "truck", "110": "trumpet", "111": "umbrella", "112": "vase", "113": "violin", |
|
|
"114": "watermelon", "115": "whale", "116": "windmill", "117": "wine_glass", "118": "yoga", |
|
|
"119": "zebra", "120": "zigzag" |
|
|
} |
|
|
|
|
|
# Create a dictionary mapping each label to its corresponding probability (rounded) |
|
|
predictions = {labels[str(i)]: round(probs[i], 3) for i in range(len(probs))} |
|
|
return predictions |
|
|
|
|
|
# Create Gradio interface |
|
|
iface = gr.Interface( |
|
|
fn=multisource_classification, |
|
|
inputs=gr.Image(type="numpy"), |
|
|
outputs=gr.Label(label="Prediction Scores"), |
|
|
title="Multisource-121-DomainNet Classification", |
|
|
description="Upload an image to classify it into one of 121 domain categories." |
|
|
) |
|
|
|
|
|
# Launch the app |
|
|
if __name__ == "__main__": |
|
|
iface.launch() |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
# **Intended Use:** |
|
|
|
|
|
The **Multisource-121-DomainNet** model is designed for multi-source image classification. It can categorize images into a diverse set of 121 domains, covering various objects, scenes, and landmarks. Potential use cases include: |
|
|
|
|
|
- **Cross-Domain Image Analysis:** Enabling robust classification across a wide range of visual domains. |
|
|
- **Multimedia Retrieval:** Assisting in content organization and retrieval in multimedia databases. |
|
|
- **Computer Vision Research:** Serving as a benchmark for evaluating domain adaptation and transfer learning techniques. |
|
|
- **Interactive Applications:** Enhancing user interfaces with diverse, real-time image recognition capabilities. |
