|
|
--- |
|
|
title: Multimodal Image Search Engine |
|
|
emoji: 🔍 |
|
|
colorFrom: yellow |
|
|
colorTo: yellow |
|
|
sdk: gradio |
|
|
sdk_version: 4.36.1 |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
<p align="center"> |
|
|
<h1 align="center">Multi-Modal Image Search Engine</h1> |
|
|
<p align="center"> |
|
|
A Semantic Search Engine that understands the Content & Context of your Queries. |
|
|
<br> |
|
|
Use Multi-Modal inputs like Text-Image or a Reverse Image Search to Query a Vector Database of over 15k Images. <a href="https://huggingface.co/spaces/Snehil-Shah/Multimodal-Image-Search-Engine">Try it Out!</a> |
|
|
<br><br> |
|
|
<img src="https://github.com/Snehil-Shah/Multimodal-Image-Search-Engine/blob/main/assets/demo.gif?raw=true"> |
|
|
</p> |
|
|
</p> |
|
|
|
|
|
<h3>• About The Project</h3> |
|
|
|
|
|
At its core, the Search Engine is built upon the concept of **Vector Similarity Search**. |
|
|
All the Images are encoded into vector embeddings based on their semantic meaning using a Transformer Model, which are then stored in a vector space. |
|
|
When searched with a query, it returns the nearest neighbors to the input query which are the relevant search results. |
|
|
|
|
|
<p align="center"><img src="https://raw.githubusercontent.com/Snehil-Shah/Multimodal-Image-Search-Engine/main/assets/encoding_flow.png"></p> |
|
|
|
|
|
We use the Contrastive Language-Image Pre-Training (CLIP) Model by OpenAI which is a Pre-trained Multi-Modal Vision Transformer that can semantically encode Words, Sentences & Images into a 512 Dimensional Vector. This Vector encapsulates the meaning & context of the entity into a *Mathematically Measurable* format. |
|
|
|
|
|
<p align="center"><p align="center"><img src="https://raw.githubusercontent.com/Snehil-Shah/Multimodal-Image-Search-Engine/main/assets/Visualization.png" width=1000></p> |
|
|
<p align="center"><i>2-D Visualization of 500 Images in a 512-D Vector Space</i></p></p> |
|
|
|
|
|
The Images are stored as vector embeddings in a Qdrant Collection which is a Vector Database. The Search Term is encoded and run as a query to Qdrant, which returns the Nearest Neighbors based on their Cosine-Similarity to the Search Query. |
|
|
|
|
|
<p align="center"><img src="https://raw.githubusercontent.com/Snehil-Shah/Multimodal-Image-Search-Engine/main/assets/retrieval_flow.png"></p> |
|
|
|
|
|
**The Dataset**: All images are sourced from the [Open Images Dataset](https://github.com/cvdfoundation/open-images-dataset) by Common Visual Data Foundation. |
|
|
|
|
|
<h3>• Technologies Used</h3> |
|
|
|
|
|
- Python |
|
|
- Jupyter Notebooks |
|
|
- Qdrant - Vector Database |
|
|
- Sentence-Transformers - Library |
|
|
- CLIP by OpenAI - ViT Model |
|
|
- Gradio - UI |
|
|
- HuggingFace Spaces - Deployment |
