File size: 3,711 Bytes
506da10 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
# Test Server Evaluation on Cityscapes dataset
This page walks through the steps required to convert DeepLab2 predictions for
test server evaluation on [Cityscapes](https://www.cityscapes-dataset.com/).
A high-level overview of the whole process:
1. Save raw panoptic prediction in the two-channel format.
2. Create images json file.
3. Convert predictions in the two-channel format to the panoptic COCO format.
4. Run local validation set evaluation or prepare test set evaluation.
We also define some environmental variables for simplicity and convenience:
`BASE_MODEL_DIRECTORY`: variables set in textproto file, which defines where all
checkpoints and results are saved.
`DATA_ROOT`: where the original Cityscapes dataset is located.
`PATH_TO_SAVE`: where the converted results should be saved.
`IMAGES_SPLIT`: *val* or *test* depending on the target split.
## Save Raw Panoptic Prediction
Save the raw panoptic predictions in the
[two-channel panoptic format](https://arxiv.org/pdf/1801.00868.pdf) by ensuring
the following fields are set properly in the textproto config file.
```
eval_dataset_options.decode_groundtruth_label = false
evaluator_options.save_predictions = true
evaluator_options.save_raw_predictions = true
evaluator_options.convert_raw_to_eval_ids = true
```
Then run the model in evaluation modes (with `--mode=eval`), the results will be
saved at
*semantic segmentation*: ${BASE_MODEL_DIRECTORY}/vis/raw_semantic/\*.png
*panoptic segmentation*: ${BASE_MODEL_DIRECTORY}/vis/raw_panoptic/\*.png
## Create Images JSON
Create images json file by running the following commands.
```bash
python deeplab2/utils/create_images_json_for_cityscapes.py \
--image_dir=${DATA_ROOT}/leftImg8bit/${IMAGES_SPLIT} \
--output_json_path=${PATH_TO_SAVE}/${IMAGES_SPLIT}_images.json \
--only_basename \
--include_image_type_suffix=false
```
## Convert the Prediction Format
Convert prediction results saved in the
[two-channel panoptic format](https://arxiv.org/pdf/1801.00868.pdf) to the
panoptic COCO format.
```bash
python panopticapi/converters/2channels2panoptic_coco_format.py \
--source_folder=${BASE_MODEL_DIRECTORY}/vis/raw_panoptic \
--images_json_file=${PATH_TO_SAVE}/${IMAGES_SPLIT}_images.json\
--categories_json_file=deeplab2/utils/panoptic_cityscapes_categories.json \
--segmentations_folder=${PATH_TO_SAVE}/panoptic_cocoformat \
--predictions_json_file=${PATH_TO_SAVE}/panoptic_cocoformat.json
```
## Run Local Evaluation Scripts (for *validation* set)
Run the [official scripts](https://github.com/mcordts/cityscapesScripts) to
evaluate validation set results.
For *semantic segmentation*:
```bash
CITYSCAPES_RESULTS=${BASE_MODEL_DIRECTORY}/vis/raw_semantic/ \
CITYSCAPES_DATASET=${DATA_ROOT} \
CITYSCAPES_EXPORT_DIR=${PATH_TO_SAVE} \
python cityscapesscripts/evaluation/evalPixelLevelSemanticLabeling.py
```
For *panoptic segmentation*:
```bash
python cityscapesscripts/evaluation/evalPanopticSemanticLabeling.py \
--prediction-json-file=${PATH_TO_SAVE}/panoptic_cocoformat.json \
--prediction-folder=${PATH_TO_SAVE}/panoptic_cocoformat \
--gt-json-file=${DATA_ROOT}/gtFine/cityscapes_panoptic_val.json \
--gt-folder=${DATA_ROOT}/gtFine/cityscapes_panoptic_val
```
Please note that our prediction fortmat does not support instance segmentation
prediction format yet.
## Prepare Submission Files (for *test* set)
Run the following command to prepare a submission file for test server
evaluation.
```bash
zip -r cityscapes_test_submission_semantic.zip ${BASE_MODEL_DIRECTORY}/vis/raw_semantic
zip -r cityscapes_test_submission_panoptic.zip ${PATH_TO_SAVE}/panoptic_cocoformat ${PATH_TO_SAVE}/panoptic_cocoformat.json
```
|