Upload 3 files

README.md

@@ -0,0 +1,109 @@
+---
+title: MediSync - Multi-Modal Medical Analysis System
+emoji: 🩺
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: "3.41.0"
+app_file: app.py
+pinned: false
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# MediSync: Multi-Modal Medical Analysis System
+
+MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing to provide comprehensive medical insights.
+
+Introduction
+MediSync is a multi-modal AI system that combines X-ray image analysis with medical report text processing to provide comprehensive medical insights. By leveraging state-of-the-art deep learning models for both vision and language understanding, MediSync can:
+
+Analyze chest X-ray images to detect abnormalities
+Extract key clinical information from medical reports
+Fuse insights from both modalities for enhanced diagnosis support
+Provide comprehensive visualization of analysis results
+This AI system demonstrates the power of multi-modal fusion in the healthcare domain, where integrating information from multiple sources can lead to more robust and accurate analyses.
+
+System Architecture
+MediSync follows a modular architecture with three main components:
+
+Image Analysis Module: Processes X-ray images using pre-trained vision models
+Text Analysis Module: Analyzes medical reports using NLP models
+Multimodal Fusion Module: Combines insights from both modalities
+The system uses the following high-level workflow:
+
+                      ┌─────────────────┐
+                      │    X-ray Image  │
+                      └────────┬────────┘
+                               │
+                               ▼
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│  Preprocessing  │───▶│  Image Analysis │───▶│                 │
+└─────────────────┘    └─────────────────┘    │                 │
+                                               │   Multimodal    │
+┌─────────────────┐    ┌─────────────────┐    │     Fusion      │───▶ Results
+│ Medical Report  │───▶│  Text Analysis  │───▶│                 │
+└─────────────────┘    └─────────────────┘    │                 │
+                                               └─────────────────┘
+
+
+## Features
+
+- **X-ray Image Analysis**: Detects abnormalities in chest X-rays using pre-trained vision models from Hugging Face.
+- **Medical Report Processing**: Extracts key information from patient reports using NLP models.
+- **Multi-modal Integration**: Combines insights from both image and text data for more accurate diagnosis suggestions.
+- **User-friendly Interface**: Simple web interface for uploading images and reports.
+
+## Project Structure
+
+```
+mediSync/
+├── app.py                    # Main application with Gradio interface
+├── models/
+│   ├── image_analyzer.py     # X-ray image analysis module
+│   ├── text_analyzer.py      # Medical report text analysis module
+│   └── multimodal_fusion.py  # Fusion of image and text insights
+├── utils/
+│   ├── preprocessing.py      # Data preprocessing utilities
+│   └── visualization.py      # Result visualization utilities
+├── data/
+│   └── sample/               # Sample data for testing
+└── tests/                    # Unit tests
+```
+
+## Setup Instructions
+
+1. Clone this repository:
+```bash
+git clone [repository-url]
+cd MediSync
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+3. Run the application:
+```bash
+python app.py
+```
+
+4. Access the web interface at `http://localhost:7860`
+
+## Models Used
+
+- **X-ray Analysis**: facebook/deit-base-patch16-224-medical-cxr
+- **Medical Text Analysis**: medicalai/ClinicalBERT
+- **Additional Support Models**: Medical question answering and entity recognition models
+
+## Use Cases
+
+- Preliminary screening of chest X-rays
+- Cross-validation of radiologist reports
+- Educational tool for medical students
+- Research tool for studying correlation between visual findings and written reports
+
+## Note
+
+This system is designed as a support tool and should not replace professional medical diagnosis. Always consult with healthcare professionals for medical decisions. 

app.py

@@ -0,0 +1,104 @@
+import gradio as gr
+
+# Import modules
+from models.image_analyzer import ImageAnalyzer
+from models.multimodal_fusion import MultimodalFusion
+from models.text_analyzer import TextAnalyzer
+from utils.visualization import create_combined_visualization
+
+
+def process_input(image, medical_report):
+    """Process the image and text inputs and provide combined analysis"""
+    # Initialize analyzers
+    image_analyzer = ImageAnalyzer()
+    text_analyzer = TextAnalyzer()
+    fusion_model = MultimodalFusion()
+
+    # Analyze inputs
+    image_results = image_analyzer.analyze(image)
+    text_results = text_analyzer.analyze(medical_report)
+
+    # Fuse insights
+    combined_results = fusion_model.fuse_insights(image_results, text_results)
+
+    # Visualize results
+    visualization = create_combined_visualization(
+        image, image_results, text_results, combined_results
+    )
+
+    # Format results for display
+    image_findings = "\n".join(
+        [
+            f"{k}: {v:.2f}" if isinstance(v, float) else f"{k}: {v}"
+            for k, v in image_results.items()
+        ]
+    )
+    text_findings = "\n".join(
+        [f"{k}: {v}" for k, v in text_results.items() if k != "Entities"]
+    )
+    entities = ", ".join(text_results.get("Entities", []))
+
+    recommendation = combined_results.get("Recommendation", "No recommendation")
+    confidence = combined_results.get("Confidence", "N/A")
+
+    return (
+        visualization,
+        image_findings,
+        text_findings,
+        entities,
+        recommendation,
+        confidence,
+    )
+
+
+# Create Gradio interface
+with gr.Blocks(title="MediSync: Multi-Modal Medical Analysis System") as demo:
+    gr.Markdown("# MediSync: Multi-Modal Medical Analysis System")
+    gr.Markdown(
+        "Upload a chest X-ray image and provide a medical report for comprehensive analysis"
+    )
+
+    with gr.Row():
+        with gr.Column():
+            image_input = gr.Image(type="pil", label="Upload X-ray Image")
+            text_input = gr.Textbox(lines=10, label="Enter Medical Report")
+            submit_btn = gr.Button("Analyze")
+
+        with gr.Column():
+            output_image = gr.Image(type="pil", label="Visualization")
+
+            with gr.Tab("Summary"):
+                recommendation_output = gr.Textbox(label="Recommendation")
+                confidence_output = gr.Textbox(label="Confidence Score")
+
+            with gr.Tab("Detailed Results"):
+                image_output = gr.Textbox(label="Image Analysis")
+                text_output = gr.Textbox(label="Text Analysis")
+                entities_output = gr.Textbox(label="Detected Medical Entities")
+
+    submit_btn.click(
+        process_input,
+        inputs=[image_input, text_input],
+        outputs=[
+            output_image,
+            image_output,
+            text_output,
+            entities_output,
+            recommendation_output,
+            confidence_output,
+        ],
+    )
+
+    gr.Markdown("""
+    ## About MediSync
+    
+    MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing 
+    to provide comprehensive medical insights.
+    
+    **Note:** This system is designed as a support tool and should not replace professional medical diagnosis. 
+    Always consult with healthcare professionals for medical decisions.
+    """)
+
+# Launch the app
+if __name__ == "__main__":
+    demo.launch()

requirements.txt

@@ -0,0 +1,9 @@
+gradio==3.41.0
+torch>=1.10.0
+transformers>=4.20.0
+Pillow>=9.0.0
+numpy>=1.20.0
+sentencepiece>=0.1.96
+protobuf>=3.19.0
+accelerate>=0.15.0
+matplotlib>=3.5.0