File size: 2,036 Bytes
3a8f3ae 5cbbc44 3a8f3ae 5cbbc44 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
---
tags:
- model_hub_mixin
- pytorch_model_hub_mixin
language:
- en
base_model:
- google/vit-base-patch16-224
- google-bert/bert-base-uncased
- facebook/wav2vec2-base-960h
---
# 🖼️📝 OneEncoder: A Unified Text & Image & Audio Model
**OneEncoder** is a lightweight framework for cross-modal alignment, focusing on efficiently integrating **text, image and audio** (with future extensions to other modalities). Unlike traditional methods relying on massive modality-specific encoders, OneEncoder progressively aligns different data types, making it cost-effective and performant even on small paired datasets.
## 🚀 Key Features
✅ **Multimodal Alignment**: Initially supports **text & image & audio**, with extension to other modalities.
✅ **Lightweight & Efficient**: Avoids full retraining when adding new modalities.
✅ **Superior Performance**: Outperforms models that require large specialized datasets.
## 🎯 Applications
- **Visual Question Answering (VQA)**
- **Image-Text-Audio Retrieval**
- **Multimodal Content Understanding**
## 📄 Research Paper
📜 **arXiv**: [OneEncoder: Progressive Cross-Modal Alignment](https://arxiv.org/abs/2409.11059)
## 📌 Resources
🔗 **GitHub Repo**: [OneEncoder](https://github.com/b-faye/OneEncoder)
🚀 **Hugging Face Demo**: [OneEncoder Retriever](https://huggingface.co/spaces/bilalfaye/OneEncoder-retriever)
📓 **Demo Notebook**: [OneEncoder Demos](https://github.com/b-faye/OneEncoder/tree/main/demo)
🔊 **OneEncoder for Text, Image**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image)
🔊 **OneEncoder for Text, Image & Video**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image-video)
🔊 **OneEncoder for Text, Image & X-ray**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image-xray)
## 📝 Authors
📌 **Bilal FAYE**, Hanane AZZAG, Mustapha LEBBAH, Djamel BOUCHAFFRA
**Note: This model is training with temperature=2.5 and addition as fusion operation** |