Upload 5 files
Browse files- DOCUMENTATION.md +335 -0
- README.md +64 -0
- mediSync.log +80 -0
- requirements.txt +11 -0
- run.py +71 -0
DOCUMENTATION.md
ADDED
@@ -0,0 +1,335 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# MediSync: Multi-Modal Medical Analysis System
|
2 |
+
|
3 |
+
## Comprehensive Technical Documentation
|
4 |
+
|
5 |
+
### Table of Contents
|
6 |
+
1. [Introduction](#introduction)
|
7 |
+
2. [System Architecture](#system-architecture)
|
8 |
+
3. [Installation](#installation)
|
9 |
+
4. [Usage](#usage)
|
10 |
+
5. [Core Components](#core-components)
|
11 |
+
6. [Model Details](#model-details)
|
12 |
+
7. [API Reference](#api-reference)
|
13 |
+
8. [Extending the System](#extending-the-system)
|
14 |
+
9. [Troubleshooting](#troubleshooting)
|
15 |
+
10. [References](#references)
|
16 |
+
|
17 |
+
---
|
18 |
+
|
19 |
+
## Introduction
|
20 |
+
|
21 |
+
MediSync is a multi-modal AI system that combines X-ray image analysis with medical report text processing to provide comprehensive medical insights. By leveraging state-of-the-art deep learning models for both vision and language understanding, MediSync can:
|
22 |
+
|
23 |
+
- Analyze chest X-ray images to detect abnormalities
|
24 |
+
- Extract key clinical information from medical reports
|
25 |
+
- Fuse insights from both modalities for enhanced diagnosis support
|
26 |
+
- Provide comprehensive visualization of analysis results
|
27 |
+
|
28 |
+
This AI system demonstrates the power of multi-modal fusion in the healthcare domain, where integrating information from multiple sources can lead to more robust and accurate analyses.
|
29 |
+
|
30 |
+
## System Architecture
|
31 |
+
|
32 |
+
MediSync follows a modular architecture with three main components:
|
33 |
+
|
34 |
+
1. **Image Analysis Module**: Processes X-ray images using pre-trained vision models
|
35 |
+
2. **Text Analysis Module**: Analyzes medical reports using NLP models
|
36 |
+
3. **Multimodal Fusion Module**: Combines insights from both modalities
|
37 |
+
|
38 |
+
The system uses the following high-level workflow:
|
39 |
+
|
40 |
+
```
|
41 |
+
┌─────────────────┐
|
42 |
+
│ X-ray Image │
|
43 |
+
└────────┬────────┘
|
44 |
+
│
|
45 |
+
▼
|
46 |
+
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
47 |
+
│ Preprocessing │───▶│ Image Analysis │───▶│ │
|
48 |
+
└─────────────────┘ └─────────────────┘ │ │
|
49 |
+
│ Multimodal │
|
50 |
+
┌─────────────────┐ ┌─────────────────┐ │ Fusion │───▶ Results
|
51 |
+
│ Medical Report │───▶│ Text Analysis │───▶│ │
|
52 |
+
└─────────────────┘ └─────────────────┘ │ │
|
53 |
+
└─────────────────┘
|
54 |
+
```
|
55 |
+
|
56 |
+
## Installation
|
57 |
+
|
58 |
+
### Prerequisites
|
59 |
+
- Python 3.8 or higher
|
60 |
+
- Pip package manager
|
61 |
+
|
62 |
+
### Setup Instructions
|
63 |
+
|
64 |
+
1. Clone the repository:
|
65 |
+
```bash
|
66 |
+
git clone [repository-url]
|
67 |
+
cd mediSync
|
68 |
+
```
|
69 |
+
|
70 |
+
2. Install dependencies:
|
71 |
+
```bash
|
72 |
+
pip install -r requirements.txt
|
73 |
+
```
|
74 |
+
|
75 |
+
3. Download sample data:
|
76 |
+
```bash
|
77 |
+
python -m mediSync.utils.download_samples
|
78 |
+
```
|
79 |
+
|
80 |
+
## Usage
|
81 |
+
|
82 |
+
### Running the Application
|
83 |
+
|
84 |
+
To launch the MediSync application with the Gradio interface:
|
85 |
+
|
86 |
+
```bash
|
87 |
+
python run.py
|
88 |
+
```
|
89 |
+
|
90 |
+
This will:
|
91 |
+
1. Download sample data if not already present
|
92 |
+
2. Initialize the application
|
93 |
+
3. Launch the Gradio web interface
|
94 |
+
|
95 |
+
### Web Interface
|
96 |
+
|
97 |
+
MediSync provides a user-friendly web interface with three main tabs:
|
98 |
+
|
99 |
+
1. **Multimodal Analysis**: Upload an X-ray image and enter a medical report for combined analysis
|
100 |
+
2. **Image Analysis**: Upload an X-ray image for image-only analysis
|
101 |
+
3. **Text Analysis**: Enter a medical report for text-only analysis
|
102 |
+
|
103 |
+
### Command Line Usage
|
104 |
+
|
105 |
+
You can also use the core components directly from Python:
|
106 |
+
|
107 |
+
```python
|
108 |
+
from mediSync.models import XRayImageAnalyzer, MedicalReportAnalyzer, MultimodalFusion
|
109 |
+
|
110 |
+
# Initialize models
|
111 |
+
fusion_model = MultimodalFusion()
|
112 |
+
|
113 |
+
# Analyze image and text
|
114 |
+
results = fusion_model.analyze("path/to/image.jpg", "Medical report text...")
|
115 |
+
|
116 |
+
# Get explanation
|
117 |
+
explanation = fusion_model.get_explanation(results)
|
118 |
+
print(explanation)
|
119 |
+
```
|
120 |
+
|
121 |
+
## Core Components
|
122 |
+
|
123 |
+
### Image Analysis Module
|
124 |
+
|
125 |
+
The `XRayImageAnalyzer` class is responsible for analyzing X-ray images:
|
126 |
+
|
127 |
+
- Uses the DeiT (Data-efficient image Transformers) model fine-tuned on chest X-rays
|
128 |
+
- Detects abnormalities and classifies findings
|
129 |
+
- Provides confidence scores and primary findings
|
130 |
+
|
131 |
+
Key methods:
|
132 |
+
- `analyze(image_path)`: Analyzes an X-ray image
|
133 |
+
- `get_explanation(results)`: Generates a human-readable explanation
|
134 |
+
|
135 |
+
### Text Analysis Module
|
136 |
+
|
137 |
+
The `MedicalReportAnalyzer` class processes medical report text:
|
138 |
+
|
139 |
+
- Extracts medical entities (conditions, treatments, tests)
|
140 |
+
- Assesses severity level
|
141 |
+
- Extracts key findings
|
142 |
+
- Suggests follow-up actions
|
143 |
+
|
144 |
+
Key methods:
|
145 |
+
- `extract_entities(text)`: Extracts medical entities
|
146 |
+
- `assess_severity(text)`: Determines severity level
|
147 |
+
- `extract_findings(text)`: Extracts key clinical findings
|
148 |
+
- `suggest_followup(text, entities, severity)`: Suggests follow-up actions
|
149 |
+
- `analyze(text)`: Performs comprehensive analysis
|
150 |
+
|
151 |
+
### Multimodal Fusion Module
|
152 |
+
|
153 |
+
The `MultimodalFusion` class combines insights from both modalities:
|
154 |
+
|
155 |
+
- Calculates agreement between image and text analyses
|
156 |
+
- Determines confidence-weighted findings
|
157 |
+
- Provides comprehensive severity assessment
|
158 |
+
- Merges follow-up recommendations
|
159 |
+
|
160 |
+
Key methods:
|
161 |
+
- `analyze_image(image_path)`: Analyzes image only
|
162 |
+
- `analyze_text(text)`: Analyzes text only
|
163 |
+
- `analyze(image_path, report_text)`: Performs multimodal analysis
|
164 |
+
- `get_explanation(fused_results)`: Generates comprehensive explanation
|
165 |
+
|
166 |
+
## Model Details
|
167 |
+
|
168 |
+
### X-ray Analysis Model
|
169 |
+
|
170 |
+
- **Model**: facebook/deit-base-patch16-224-medical-cxr
|
171 |
+
- **Architecture**: Data-efficient image Transformer (DeiT)
|
172 |
+
- **Training Data**: Chest X-ray datasets
|
173 |
+
- **Input Size**: 224x224 pixels
|
174 |
+
- **Output**: Classification probabilities for various conditions
|
175 |
+
|
176 |
+
### Medical Text Analysis Models
|
177 |
+
|
178 |
+
- **Entity Recognition Model**: samrawal/bert-base-uncased_medical-ner
|
179 |
+
- **Classification Model**: medicalai/ClinicalBERT
|
180 |
+
- **Architecture**: BERT-based transformer models
|
181 |
+
- **Training Data**: Medical text and reports
|
182 |
+
|
183 |
+
## API Reference
|
184 |
+
|
185 |
+
### XRayImageAnalyzer
|
186 |
+
|
187 |
+
```python
|
188 |
+
from mediSync.models import XRayImageAnalyzer
|
189 |
+
|
190 |
+
# Initialize
|
191 |
+
analyzer = XRayImageAnalyzer(model_name="facebook/deit-base-patch16-224-medical-cxr")
|
192 |
+
|
193 |
+
# Analyze image
|
194 |
+
results = analyzer.analyze("path/to/image.jpg")
|
195 |
+
|
196 |
+
# Get explanation
|
197 |
+
explanation = analyzer.get_explanation(results)
|
198 |
+
```
|
199 |
+
|
200 |
+
### MedicalReportAnalyzer
|
201 |
+
|
202 |
+
```python
|
203 |
+
from mediSync.models import MedicalReportAnalyzer
|
204 |
+
|
205 |
+
# Initialize
|
206 |
+
analyzer = MedicalReportAnalyzer()
|
207 |
+
|
208 |
+
# Analyze report
|
209 |
+
results = analyzer.analyze("Medical report text...")
|
210 |
+
|
211 |
+
# Access specific components
|
212 |
+
entities = results["entities"]
|
213 |
+
severity = results["severity"]
|
214 |
+
findings = results["findings"]
|
215 |
+
recommendations = results["followup_recommendations"]
|
216 |
+
```
|
217 |
+
|
218 |
+
### MultimodalFusion
|
219 |
+
|
220 |
+
```python
|
221 |
+
from mediSync.models import MultimodalFusion
|
222 |
+
|
223 |
+
# Initialize
|
224 |
+
fusion = MultimodalFusion()
|
225 |
+
|
226 |
+
# Multimodal analysis
|
227 |
+
results = fusion.analyze("path/to/image.jpg", "Medical report text...")
|
228 |
+
|
229 |
+
# Get explanation
|
230 |
+
explanation = fusion.get_explanation(results)
|
231 |
+
```
|
232 |
+
|
233 |
+
## Extending the System
|
234 |
+
|
235 |
+
### Adding New Models
|
236 |
+
|
237 |
+
To add a new image analysis model:
|
238 |
+
|
239 |
+
1. Create a new class that follows the same interface as `XRayImageAnalyzer`
|
240 |
+
2. Update the `MultimodalFusion` class to use your new model
|
241 |
+
|
242 |
+
```python
|
243 |
+
class NewXRayModel:
|
244 |
+
def __init__(self, model_name, device=None):
|
245 |
+
# Initialize your model
|
246 |
+
pass
|
247 |
+
|
248 |
+
def analyze(self, image_path):
|
249 |
+
# Implement analysis logic
|
250 |
+
return results
|
251 |
+
|
252 |
+
def get_explanation(self, results):
|
253 |
+
# Generate explanation
|
254 |
+
return explanation
|
255 |
+
```
|
256 |
+
|
257 |
+
### Custom Preprocessing
|
258 |
+
|
259 |
+
You can extend the preprocessing utilities in `utils/preprocessing.py` for custom data preparation:
|
260 |
+
|
261 |
+
```python
|
262 |
+
def my_custom_preprocessor(image_path, **kwargs):
|
263 |
+
# Implement custom preprocessing
|
264 |
+
return processed_image
|
265 |
+
```
|
266 |
+
|
267 |
+
### Visualization Extensions
|
268 |
+
|
269 |
+
To add new visualization options, extend the utilities in `utils/visualization.py`:
|
270 |
+
|
271 |
+
```python
|
272 |
+
def my_custom_visualization(results, **kwargs):
|
273 |
+
# Create custom visualization
|
274 |
+
return figure
|
275 |
+
```
|
276 |
+
|
277 |
+
## Troubleshooting
|
278 |
+
|
279 |
+
### Common Issues
|
280 |
+
|
281 |
+
1. **Model Loading Errors**
|
282 |
+
- Ensure you have a stable internet connection for downloading models
|
283 |
+
- Check that you have sufficient disk space
|
284 |
+
- Try specifying a different model checkpoint
|
285 |
+
|
286 |
+
2. **Image Processing Errors**
|
287 |
+
- Ensure images are in a supported format (JPEG, PNG)
|
288 |
+
- Check that the image is a valid X-ray image
|
289 |
+
- Try preprocessing the image manually using the utility functions
|
290 |
+
|
291 |
+
3. **Performance Issues**
|
292 |
+
- For faster inference, use a GPU if available
|
293 |
+
- Reduce image resolution if processing is too slow
|
294 |
+
- Use the text-only analysis for quicker results
|
295 |
+
|
296 |
+
### Logging
|
297 |
+
|
298 |
+
MediSync uses Python's logging module for debug information:
|
299 |
+
|
300 |
+
```python
|
301 |
+
import logging
|
302 |
+
logging.basicConfig(level=logging.DEBUG)
|
303 |
+
```
|
304 |
+
|
305 |
+
Log files are saved to `mediSync.log` in the application directory.
|
306 |
+
|
307 |
+
## References
|
308 |
+
|
309 |
+
### Datasets
|
310 |
+
|
311 |
+
- [MIMIC-CXR](https://physionet.org/content/mimic-cxr/2.0.0/): Large dataset of chest radiographs with reports
|
312 |
+
- [ChestX-ray14](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community): NIH dataset of chest X-rays
|
313 |
+
|
314 |
+
### Papers
|
315 |
+
|
316 |
+
- He, K., et al. (2020). "Vision Transformers for Medical Image Analysis"
|
317 |
+
- Irvin, J., et al. (2019). "CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison"
|
318 |
+
- Johnson, A.E.W., et al. (2019). "MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs"
|
319 |
+
|
320 |
+
### Tools and Libraries
|
321 |
+
|
322 |
+
- [Hugging Face Transformers](https://huggingface.co/docs/transformers/index)
|
323 |
+
- [PyTorch](https://pytorch.org/)
|
324 |
+
- [Gradio](https://gradio.app/)
|
325 |
+
|
326 |
+
---
|
327 |
+
|
328 |
+
## License
|
329 |
+
|
330 |
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
331 |
+
|
332 |
+
## Acknowledgments
|
333 |
+
|
334 |
+
- The development of MediSync was inspired by recent advances in multi-modal learning in healthcare.
|
335 |
+
- Special thanks to the open-source community for providing pre-trained models and tools.
|
README.md
ADDED
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# MediSync: Multi-Modal Medical Analysis System
|
2 |
+
|
3 |
+
MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing to provide comprehensive medical insights.
|
4 |
+
|
5 |
+
## Features
|
6 |
+
|
7 |
+
- **X-ray Image Analysis**: Detects abnormalities in chest X-rays using pre-trained vision models from Hugging Face.
|
8 |
+
- **Medical Report Processing**: Extracts key information from patient reports using NLP models.
|
9 |
+
- **Multi-modal Integration**: Combines insights from both image and text data for more accurate diagnosis suggestions.
|
10 |
+
- **User-friendly Interface**: Simple web interface for uploading images and reports.
|
11 |
+
|
12 |
+
## Project Structure
|
13 |
+
|
14 |
+
```
|
15 |
+
mediSync/
|
16 |
+
├── app.py # Main application with Gradio interface
|
17 |
+
├── models/
|
18 |
+
│ ├── image_analyzer.py # X-ray image analysis module
|
19 |
+
│ ├── text_analyzer.py # Medical report text analysis module
|
20 |
+
│ └── multimodal_fusion.py # Fusion of image and text insights
|
21 |
+
├── utils/
|
22 |
+
│ ├── preprocessing.py # Data preprocessing utilities
|
23 |
+
│ └── visualization.py # Result visualization utilities
|
24 |
+
├── data/
|
25 |
+
│ └── sample/ # Sample data for testing
|
26 |
+
└── tests/ # Unit tests
|
27 |
+
```
|
28 |
+
|
29 |
+
## Setup Instructions
|
30 |
+
|
31 |
+
1. Clone this repository:
|
32 |
+
```bash
|
33 |
+
git clone [repository-url]
|
34 |
+
cd MediSync
|
35 |
+
```
|
36 |
+
|
37 |
+
2. Install dependencies:
|
38 |
+
```bash
|
39 |
+
pip install -r requirements.txt
|
40 |
+
```
|
41 |
+
|
42 |
+
3. Run the application:
|
43 |
+
```bash
|
44 |
+
python app.py
|
45 |
+
```
|
46 |
+
|
47 |
+
4. Access the web interface at `http://localhost:7860`
|
48 |
+
|
49 |
+
## Models Used
|
50 |
+
|
51 |
+
- **X-ray Analysis**: facebook/deit-base-patch16-224-medical-cxr
|
52 |
+
- **Medical Text Analysis**: medicalai/ClinicalBERT
|
53 |
+
- **Additional Support Models**: Medical question answering and entity recognition models
|
54 |
+
|
55 |
+
## Use Cases
|
56 |
+
|
57 |
+
- Preliminary screening of chest X-rays
|
58 |
+
- Cross-validation of radiologist reports
|
59 |
+
- Educational tool for medical students
|
60 |
+
- Research tool for studying correlation between visual findings and written reports
|
61 |
+
|
62 |
+
## Note
|
63 |
+
|
64 |
+
This system is designed as a support tool and should not replace professional medical diagnosis. Always consult with healthcare professionals for medical decisions.
|
mediSync.log
ADDED
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
2025-03-10 22:44:41,854 - __main__ - INFO - Starting MediSync setup
|
2 |
+
2025-03-10 22:44:41,854 - __main__ - INFO - Checking for sample data
|
3 |
+
2025-03-10 22:44:42,626 - __main__ - INFO - No sample images found. Downloading...
|
4 |
+
2025-03-10 22:44:42,626 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/53448173/322830a37f0fa0852773ca2db3e8d8_big_gallery.jpeg to D:\Multi Model AI Problem Solving\mediSync\data\sample\normal_chest_xray.jpg
|
5 |
+
2025-03-10 22:44:43,532 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/53448173/322830a37f0fa0852773ca2db3e8d8_big_gallery.jpeg: HTTP Error 404: Not Found
|
6 |
+
2025-03-10 22:44:43,532 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/52465460/e4d8791bd7502ab72af8d9e5c322db_big_gallery.jpg to D:\Multi Model AI Problem Solving\mediSync\data\sample\pneumonia_xray.jpg
|
7 |
+
2025-03-10 22:44:43,988 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/52465460/e4d8791bd7502ab72af8d9e5c322db_big_gallery.jpg: HTTP Error 404: Not Found
|
8 |
+
2025-03-10 22:44:43,988 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/556520/cf17c05750adb04b2a6e23afb47c7d_big_gallery.jpg to D:\Multi Model AI Problem Solving\mediSync\data\sample\cardiomegaly_xray.jpg
|
9 |
+
2025-03-10 22:44:44,605 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/556520/cf17c05750adb04b2a6e23afb47c7d_big_gallery.jpg: HTTP Error 404: Not Found
|
10 |
+
2025-03-10 22:44:44,605 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/19972291/41eed1a2cdad06d26c3f415a6ed65a_big_gallery.jpeg to D:\Multi Model AI Problem Solving\mediSync\data\sample\nodule_xray.jpg
|
11 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/19972291/41eed1a2cdad06d26c3f415a6ed65a_big_gallery.jpeg: HTTP Error 404: Not Found
|
12 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - INFO - Downloaded 0 out of 4 images
|
13 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - INFO - Created sample info file: D:\Multi Model AI Problem Solving\mediSync\data\sample\sample_info.txt
|
14 |
+
2025-03-10 22:44:45,107 - __main__ - INFO - Launching MediSync application
|
15 |
+
2025-03-10 22:44:48,918 - httpx - INFO - HTTP Request: GET https://api.gradio.app/gradio-messaging/en "HTTP/1.1 200 OK"
|
16 |
+
2025-03-10 22:45:02,402 - mediSync.app - INFO - Initializing MediSync application
|
17 |
+
2025-03-10 22:45:03,005 - httpx - INFO - HTTP Request: GET https://api.gradio.app/pkg-version "HTTP/1.1 200 OK"
|
18 |
+
2025-03-10 22:45:03,294 - httpx - INFO - HTTP Request: GET http://127.0.0.1:7860/gradio_api/startup-events "HTTP/1.1 200 OK"
|
19 |
+
2025-03-10 22:45:03,722 - httpx - INFO - HTTP Request: HEAD http://127.0.0.1:7860/ "HTTP/1.1 200 OK"
|
20 |
+
2025-03-10 22:46:47,162 - mediSync.app - INFO - Loading models...
|
21 |
+
2025-03-10 22:46:47,162 - models.multimodal_fusion - INFO - Using device: cpu
|
22 |
+
2025-03-10 22:46:47,178 - models.image_analyzer - INFO - Using device: cpu
|
23 |
+
2025-03-10 22:46:47,391 - models.image_analyzer - ERROR - Failed to load model: facebook/deit-base-patch16-224-medical-cxr is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
24 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
25 |
+
2025-03-10 22:46:47,391 - models.multimodal_fusion - ERROR - Failed to initialize image analyzer: facebook/deit-base-patch16-224-medical-cxr is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
26 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
27 |
+
2025-03-10 22:46:47,391 - models.text_analyzer - INFO - Using device: cpu
|
28 |
+
2025-03-10 22:46:47,490 - models.text_analyzer - ERROR - Failed to load NER model: samrawal/bert-base-uncased_medical-ner is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
29 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
30 |
+
2025-03-10 22:46:58,283 - models.text_analyzer - INFO - Successfully loaded classifier model: medicalai/ClinicalBERT
|
31 |
+
2025-03-10 22:46:58,283 - models.multimodal_fusion - INFO - Successfully initialized text analyzer
|
32 |
+
2025-03-10 22:46:58,283 - mediSync.app - INFO - Models loaded successfully
|
33 |
+
2025-03-10 22:46:58,379 - mediSync.app - INFO - Performing multimodal analysis
|
34 |
+
2025-03-10 22:46:58,379 - models.multimodal_fusion - WARNING - Image analyzer not available
|
35 |
+
2025-03-10 22:46:58,379 - models.text_analyzer - WARNING - NER model not available
|
36 |
+
2025-03-10 22:46:58,379 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
37 |
+
2025-03-10 22:47:44,547 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp19zqepvi\upload.png
|
38 |
+
2025-03-10 22:47:55,128 - mediSync.app - INFO - Analyzing medical report text
|
39 |
+
2025-03-10 22:47:55,128 - models.text_analyzer - WARNING - NER model not available
|
40 |
+
2025-03-10 22:48:20,220 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpowd6qwty\upload.png
|
41 |
+
2025-03-10 22:55:22,019 - mediSync.app - INFO - Performing multimodal analysis
|
42 |
+
2025-03-10 22:55:22,019 - models.multimodal_fusion - WARNING - Image analyzer not available
|
43 |
+
2025-03-10 22:55:22,025 - models.text_analyzer - WARNING - NER model not available
|
44 |
+
2025-03-10 22:55:22,025 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
45 |
+
2025-03-10 22:55:35,606 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpipzmsxch\upload.png
|
46 |
+
2025-03-10 22:55:45,116 - mediSync.app - INFO - Analyzing medical report text
|
47 |
+
2025-03-10 22:55:45,116 - models.text_analyzer - WARNING - NER model not available
|
48 |
+
2025-03-10 23:01:32,009 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpo88f1uaw\upload.png
|
49 |
+
2025-03-10 23:05:24,892 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpnckk4w27\upload.png
|
50 |
+
2025-03-10 23:05:45,332 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpxab_m30m\upload.png
|
51 |
+
2025-03-10 23:06:01,387 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpcit78i2e\upload.png
|
52 |
+
2025-03-10 23:06:04,157 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpivli_abx\upload.png
|
53 |
+
2025-03-10 23:11:47,429 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp4cpwvo5h\upload.png
|
54 |
+
2025-03-10 23:22:27,118 - mediSync.app - INFO - Performing multimodal analysis
|
55 |
+
2025-03-10 23:22:27,118 - models.multimodal_fusion - WARNING - Image analyzer not available
|
56 |
+
2025-03-10 23:22:27,118 - models.text_analyzer - WARNING - NER model not available
|
57 |
+
2025-03-10 23:22:27,118 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
58 |
+
2025-03-10 23:24:22,378 - mediSync.app - INFO - Performing multimodal analysis
|
59 |
+
2025-03-10 23:24:22,378 - models.multimodal_fusion - WARNING - Image analyzer not available
|
60 |
+
2025-03-10 23:24:22,378 - models.text_analyzer - WARNING - NER model not available
|
61 |
+
2025-03-10 23:24:22,378 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
62 |
+
2025-03-10 23:24:35,252 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp947ie5e1\upload.png
|
63 |
+
2025-03-10 23:24:37,554 - mediSync.app - INFO - Performing multimodal analysis
|
64 |
+
2025-03-10 23:24:37,554 - models.multimodal_fusion - WARNING - Image analyzer not available
|
65 |
+
2025-03-10 23:24:37,554 - models.text_analyzer - WARNING - NER model not available
|
66 |
+
2025-03-10 23:24:37,554 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
67 |
+
2025-03-10 23:24:58,770 - mediSync.app - INFO - Analyzing medical report text
|
68 |
+
2025-03-10 23:24:58,770 - models.text_analyzer - WARNING - NER model not available
|
69 |
+
2025-03-10 23:25:25,652 - mediSync.app - INFO - Analyzing medical report text
|
70 |
+
2025-03-10 23:25:25,652 - models.text_analyzer - WARNING - NER model not available
|
71 |
+
2025-03-10 23:27:06,100 - mediSync.app - INFO - Performing multimodal analysis
|
72 |
+
2025-03-10 23:27:06,100 - models.multimodal_fusion - WARNING - Image analyzer not available
|
73 |
+
2025-03-10 23:27:06,100 - models.text_analyzer - WARNING - NER model not available
|
74 |
+
2025-03-10 23:27:06,100 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
75 |
+
2025-03-10 23:27:46,183 - mediSync.app - INFO - Performing multimodal analysis
|
76 |
+
2025-03-10 23:27:46,183 - models.multimodal_fusion - WARNING - Image analyzer not available
|
77 |
+
2025-03-10 23:27:46,183 - models.text_analyzer - WARNING - NER model not available
|
78 |
+
2025-03-10 23:27:46,183 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
79 |
+
2025-03-10 23:28:15,394 - mediSync.app - INFO - Analyzing medical report text
|
80 |
+
2025-03-10 23:28:15,394 - models.text_analyzer - WARNING - NER model not available
|
requirements.txt
ADDED
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
torch>=2.0.0
|
2 |
+
transformers>=4.30.0
|
3 |
+
datasets>=2.12.0
|
4 |
+
pillow>=9.0.0
|
5 |
+
matplotlib>=3.7.0
|
6 |
+
pandas>=2.0.0
|
7 |
+
numpy>=1.24.0
|
8 |
+
scikit-learn>=1.2.0
|
9 |
+
gradio>=4.0.0
|
10 |
+
huggingface_hub>=0.16.0
|
11 |
+
opencv-python>=4.7.0
|
run.py
ADDED
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
"""
|
2 |
+
MediSync: Multi-Modal Medical Analysis System - Runner Script
|
3 |
+
============================================================
|
4 |
+
This script downloads sample data and launches the MediSync application.
|
5 |
+
"""
|
6 |
+
|
7 |
+
import logging
|
8 |
+
import sys
|
9 |
+
from pathlib import Path
|
10 |
+
|
11 |
+
# Set up logging
|
12 |
+
logging.basicConfig(
|
13 |
+
level=logging.INFO,
|
14 |
+
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
15 |
+
handlers=[logging.StreamHandler(), logging.FileHandler("mediSync.log")],
|
16 |
+
)
|
17 |
+
logger = logging.getLogger(__name__)
|
18 |
+
|
19 |
+
|
20 |
+
def main():
|
21 |
+
"""
|
22 |
+
Main function to set up and run the MediSync application.
|
23 |
+
"""
|
24 |
+
logger.info("Starting MediSync setup")
|
25 |
+
|
26 |
+
# Add the current directory to Python path
|
27 |
+
current_dir = Path(__file__).resolve().parent
|
28 |
+
sys.path.append(str(current_dir))
|
29 |
+
|
30 |
+
# Download sample data if needed
|
31 |
+
try:
|
32 |
+
logger.info("Checking for sample data")
|
33 |
+
from mediSync.utils.download_samples import (
|
34 |
+
create_sample_info_file,
|
35 |
+
download_sample_images,
|
36 |
+
)
|
37 |
+
|
38 |
+
# Check if sample directory exists and contains images
|
39 |
+
sample_dir = current_dir / "mediSync" / "data" / "sample"
|
40 |
+
sample_dir.mkdir(parents=True, exist_ok=True)
|
41 |
+
|
42 |
+
image_files = list(sample_dir.glob("*.jpg")) + list(sample_dir.glob("*.png"))
|
43 |
+
|
44 |
+
if not image_files:
|
45 |
+
logger.info("No sample images found. Downloading...")
|
46 |
+
download_sample_images()
|
47 |
+
create_sample_info_file()
|
48 |
+
else:
|
49 |
+
logger.info(f"Found {len(image_files)} existing sample images")
|
50 |
+
|
51 |
+
except Exception as e:
|
52 |
+
logger.error(f"Error setting up sample data: {e}")
|
53 |
+
print(f"Warning: Could not set up sample data: {e}")
|
54 |
+
|
55 |
+
# Launch the application
|
56 |
+
try:
|
57 |
+
logger.info("Launching MediSync application")
|
58 |
+
from mediSync.app import create_interface
|
59 |
+
|
60 |
+
create_interface()
|
61 |
+
|
62 |
+
except Exception as e:
|
63 |
+
logger.error(f"Error launching application: {e}")
|
64 |
+
print(f"Error: Failed to launch MediSync application: {e}")
|
65 |
+
return 1
|
66 |
+
|
67 |
+
return 0
|
68 |
+
|
69 |
+
|
70 |
+
if __name__ == "__main__":
|
71 |
+
sys.exit(main())
|