README.md

---
title: LoRACaptioner
emoji: 🤠
colorFrom: red
colorTo: green
sdk: gradio
sdk_version: 5.25.2
app_file: demo.py
pinned: false
---

# LoRACaptioner

- **Image Captioning**: Automatically generate detailed and structured captions for your LoRA dataset.
- **Prompt Optimization**: Enhance prompts during inference to achieve high-quality outputs.

<table>
  <tr>
    <td><img src="examples/sukuna_4.png" alt="Sukuna example 4" width="400"></td>
    <td><img src="examples/sukuna_5.png" alt="Sukuna example 5" width="400"></td>
  </tr>
  <tr>
    <td><img src="examples/sukuna_6.png" alt="Sukuna example 6" width="400"></td>
    <td><img src="examples/sukuna_7.png" alt="Sukuna example 7" width="400"></td>
  </tr>
</table>

## Installation

### Prerequisites
- Python 3.10 or higher
- [Together AI](https://together.ai/) account and API key

### Setup

1. Create the virtual environment:
   ```bash
   python -m venv venv
   source venv/bin/activate
   python -m pip install -r requirements.txt
   ```

2. Run inference on one set of images:

   ```bash
   python main.py --input examples/ --output output/
   ```

   <details>
   <summary>Arguments</summary>

   - `--input` (str): Directory containing images to caption.
   - `--output` (str): Directory to save images and captions (defaults to input directory).
   - `--batch_images` (flag): Caption images in batches by category.
   </details>


## Gradio Demo

Launch a user-friendly web interface for captioning and prompt optimization:
```bash
python demo.py
```

### Notes
- Images are processed individually in standard mode
- For large collections, batch processing by category is recommended
- Each caption is saved as a .txt file with the same name as the image

### Troubleshooting

- **API errors**: Ensure your Together API key is set and has funds
- **Image formats**: Only .png, .jpg, .jpeg, and .webp files are supported

## Manual Captioning with ChatGPT
Follow the instructions in my [blog post](https://rishidesai.github.io/posts/character-lora/) and use `system_prompt.txt` as the system prompt.


## Examples 

### Sukuna from Jujutsu Kaisen


**User Prompt:**  
holding a bow and arrow in a dense forest

**Optimized Prompt:**  
`tr1gg3r anime-style, pink spiky hair and black markings on face, shirtless with dark arm bands, holding bow and arrow, focused expression, dense forest, soft dappled lighting, three-quarter view`

<img src="examples/sukuna_1.png" alt="Sukuna with bow and arrow" width="500">

---

**User Prompt:**  
drinking coffee in a san francisco cafe, white cloak, side view

**Optimized Prompt:**  
`tr1gg3r anime-style, spiky pink hair and facial markings, white cloak, sitting with cup in hand, neutral expression, cafe interior with san francisco view, soft natural lighting, side profile`

<img src="examples/sukuna_2.png" alt="Sukuna drinking coffee" width="500">

---

**User Prompt:**  
playing pick-up basketball on a sunny day

**Optimized Prompt:**  
`tr1gg3r photorealistic, athletic build, sleeveless basketball jersey and shorts, jumping with ball, focused expression, outdoor basketball court with spectators, bright sunlight, low-angle view`

<img src="examples/sukuna_3.png" alt="Sukuna playing basketball" width="500">

---

### A character generated by Flux.1-dev

**User Prompt:**  
riding a horse on a prairie during sunset

**Optimized Prompt:**  
`tr1gger photorealistic, curly shoulder-length hair, floral button-up shirt, riding a horse, neutral expression, prairie during sunset, warm directional lighting, three-quarter view`

<img src="examples/woman_1.png" alt="Woman riding a horse" width="500">

---

**User Prompt:**  
painting on a canvas in an art studio, side-view

**Optimized Prompt:**  
`tr1gg3r photorealistic, curly shoulder-length hair, floral button-up shirt, standing at an angle with brush in hand, neutral expression, art studio with canvas and paints, soft natural lighting, right side profile`

<img src="examples/woman_2.png" alt="Woman painting in studio" width="500">

---

**User Prompt:**  
standing on a skyscraper in a dense city, dramatic stormy lighting, rear view

**Optimized Prompt:**  
`tr1gg3r photorealistic, curly shoulder-length hair, floral button-up shirt, standing upright, neutral expression, skyscraper rooftop in dense city, dramatic stormy lighting, back view`

<img src="examples/woman_3.png" alt="Woman on skyscraper" width="500">