File size: 2,253 Bytes
2b49482 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
# Evaluation
We evaluate GeoChat on a variety of tasks, including scene classification, region captioning, visual grounding, grounding description and VQA.
Converted files in the input format for GeoChat are available at [GeoChat-Bench](https://huggingface.co/datasets/MBZUAI/GeoChat-Bench/tree/main)
Below we provide a general guideline for evaluating datasets.
1. LRBEN/HRBEN.
Images and ground truth for evaluation need to be downloaded from the following sources: [LRBEN](https://zenodo.org/records/6344334), [HRBEN](https://zenodo.org/records/6344367)
Give the path to the extracted image folder in the evaluation script. We add the following text after each question during our evaluation.
```
<question>
Answer the question using a single word or phrase.
```
```Shell
python geochat/eval/batch_geochat_vqa.py \
--model-path /path/to/model \
--question-file path/to/jsonl/file \
--answer-file path/to/output/jsonl/file \
--image_folder path/to/image/folder/
```
2. Scene Classification.
Download the images from the following sources, [UCmerced](http://weegee.vision.ucmerced.edu/datasets/landuse.html), [AID](https://drive.google.com/drive/folders/1-1D9DrYYWMGuuxx-qcvIIOV1oUkAVf-M). We add the following text after each question during our evaluation.
```
<question>
Classify the image from the following classes. Answer in one word or a short phrase.
```
```Shell
python geochat/eval/batch_geochat_scene.py \
--model-path /path/to/model \
--question-file path/to/jsonl/file \
--answer-file path/to/output/jsonl/file \
--image_folder path/to/image/folder/
```
3. Region-Captioning/Visual grounding.
The evaluation images are present in the image.zip folder in [GeoChat_Instruct](https://huggingface.co/datasets/MBZUAI/GeoChat_Instruct/blob/main/images.zip).
```Shell
python geochat/eval/batch_geochat_grounding.py \
--model-path /path/to/model \
--question-file path/to/jsonl/file \
--answer-file path/to/output/jsonl/file \
--image_folder path/to/image/folder/
```
```Shell
python geochat/eval/batch_geochat_referring.py \
--model-path /path/to/model \
--question-file path/to/jsonl/file \
--answer-file path/to/output/jsonl/file \
--image_folder path/to/image/folder/
``` |