|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- |
|
|
Initializing training for seed 42 |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- No private macro file found! |
|
|
- It is recommended to use a private macro file |
|
|
- To setup, run: python /home/hk-project-robolear/ft4740/miniconda3/envs/flower_cal/lib/python3.9/site-packages/robosuite/scripts/setup_macros.py |
|
|
- No private macro file found! |
|
|
- No private macro file found! |
|
|
- It is recommended to use a private macro file |
|
|
- It is recommended to use a private macro file |
|
|
- To setup, run: python /home/hk-project-robolear/ft4740/miniconda3/envs/flower_cal/lib/python3.9/site-packages/robosuite/scripts/setup_macros.py |
|
|
- To setup, run: python /home/hk-project-robolear/ft4740/miniconda3/envs/flower_cal/lib/python3.9/site-packages/robosuite/scripts/setup_macros.py |
|
|
- No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' |
|
|
- No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' |
|
|
- No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' |
|
|
- No private macro file found! |
|
|
- It is recommended to use a private macro file |
|
|
- To setup, run: python /home/hk-project-robolear/ft4740/miniconda3/envs/flower_cal/lib/python3.9/site-packages/robosuite/scripts/setup_macros.py |
|
|
- No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- Training config for seed 42: |
|
|
{'callbacks': {'ema': {'_target_': 'flower.callbacks.ema.EMA', 'decay': 0.999, 'start_step': 0, 'save_ema_weights_in_callback_state': True, 'evaluate_ema_weights_instead': True, 'power': 0.6666666666666666, 'inv_gamma': 1.0, 'min_value': 0.0, 'max_value': 0.9999}, 'rollout_lh': {'_target_': 'flower.rollout.libero_rollout.RolloutLibero', '_recursive_': False, 'env_cfg': {'_target_': 'flower.wrappers.hulc_wrapper.HulcWrapper'}, 'skip_epochs': '${rollout_lh_skip_epochs}', 'benchmark_name': '${libero_benchmark}', 'rollout_freq': 10, 'num_videos': 0, 'num_sequences': 50, 'max_steps': 520, 'empty_cache': False, 'debug': False, 'n_eval': 50, 'num_procs': 10, 'use_mp': False, 'task_embedding_format': 'clip', 'device': '${device}'}, 'checkpoint': {'_target_': 'pytorch_lightning.callbacks.ModelCheckpoint', 'save_top_k': 1, 'verbose': True, 'monitor': 'eval_lh/avg_seq_len', 'mode': 'max', 'dirpath': 'saved_models', 'filename': '{epoch:02d}_{eval_lh/avg_seq_len:.2f}', 'every_n_epochs': '${callbacks.rollout_lh.rollout_freq}'}}, 'datamodule': {'transforms': {'train': {'rgb_static': , 'rgb_gripper': }, 'val': {'rgb_static': , 'rgb_gripper': }}, '_target_': 'flower.datasets.libero_data_module.LiberoDataModule', '_recursive_': False, 'root_data_dir': '${root_data_dir}', 'action_space': 7, 'shuffle_val': False, 'benchmark_name': '${libero_benchmark}', 'observation_space': {'rgb_obs': , 'depth_obs': , 'state_obs': , 'actions': , 'language': }, 'proprioception_dims': 'None', 'datasets': {'lang_dataset': {'_target_': 'flower.datasets.libero_dataset.LiberoMultitaskDataset', 'key': 'lang', 'benchmark_name': '${libero_benchmark}', 'batch_size': '${batch_size}', 'proprio_state': '${datamodule.proprioception_dims}', 'obs_space': '${datamodule.observation_space}', 'num_workers': '${num_workers}', 'action_seq_len': '${act_seq_len}', 'obs_seq_len': '${obs_seq_len}', 'split_ratio': 0.0}}}, 'model': {'_target_': 'flower.models.flower.FLOWERVLA', '_recursive_': False, 'vlm_path': 'microsoft/Florence-2-large', 'freeze_florence': False, 'freeze_vision_tower': False, 'vlm_prompt_style': 'default', 'token_dropout': 0.1, 'multistep': '${multistep}', 'num_sampling_steps': 4, 'lowdim_obs_dim': 7, 'action_dim': 7, 'act_window_size': 10, 'load_pretrained': True, 'pretrained_model_path': '/home/hk-project-sustainebot/ft4740/code/flower_vla_policy/logs/runs/2025-02-05/10-17-02/360000_model_weights.pt', 'use_second_view': True, 'second_view_key': 'image_wrist', 'action_type_adaln': True, 'use_causal_attention': True, 'use_cross_attn': True, 'use_adaln_cond': False, 'use_readout_token': False, 'use_proprio': False, 'return_act_chunk': False, 'sampling_type': 'uniform', 'dit_dim': 1024, 'n_heads': 16, 'n_layers': 18, 'attn_pdrop': 0.1, 'resid_pdrop': 0.1, 'mlp_pdrop': 0.1, 'use_rope': True, 'use_nope': False, 'query_seq_len': 100, 'rope_theta': 32.0, 'optimizer_type': 'adamw', 'optimizer': {'_target_': 'torch.optim.AdamW', 'transformer_weight_decay': 0.05, 'learning_rate': 2e-05, 'betas': }, 'lr_scheduler': {'lr_scheduler': {'init_lr': 2e-05, 'init_lr_scale': 0.1, 'final_lr_scale': 0.5, 'total_steps': 50000, 'phase_ratio': '(0.05, 0.1, 0.85)', 'lr': 2e-05}}}, 'root_data_dir': '/home/yagmurlu/code/MoDE_Calvin/dataset/task_ABC_D', 'lang_folder': 'lang_clip_resnet50', 'log_dir': './logs', 'slurm': False, 'seed': 42, 'device': 'cuda', 'batch_size': 8, 'devices': 4, 'goal_window_size': 1, 'act_dim': 7, 'proprio_dims': 9, 'obs_dim': 512, 'goal_dim': 512, 'obs_seq_len': 1, 'act_seq_len': 10, 'multistep': '${act_seq_len}', 'p_last_state': 0, 'max_epochs': 30, 'rollout_lh_skip_epochs': 9, 'num_workers': 1, 'benchmark_name': '${libero_benchmark}', 'libero_benchmark': 'libero_spatial', 'trainer': {'devices': '${devices}', 'precision': 'bf16-mixed', 'max_epochs': '${max_epochs}', 'sync_batchnorm': True, 'accelerator': 'gpu', 'strategy': 'ddp', 'limit_train_batches': 1000, 'limit_val_batches': 4}, 'logger': {'_target_': 'pytorch_lightning.loggers.WandbLogger', 'save_dir': '.', 'name': '2025-03-16/18-00-19', 'group': '2025-03-16', 'log_model': False, 'project': '${libero_benchmark}', 'entity': 'bennoq', 'id': '2025-03-16_18-00-19'}} |
|
|
- Git commit: cd15964f487ea13d3176eaf3b9a6b7202fbbfa5a |
|
|
- * CUDA: |
|
|
- GPU: |
|
|
- NVIDIA A100-SXM4-40GB |
|
|
- NVIDIA A100-SXM4-40GB |
|
|
- NVIDIA A100-SXM4-40GB |
|
|
- NVIDIA A100-SXM4-40GB |
|
|
- available: True |
|
|
- version: 12.1 |
|
|
* Packages: |
|
|
- numpy: 1.26.4 |
|
|
- pyTorch_debug: False |
|
|
- pyTorch_version: 2.2.2+cu121 |
|
|
- pytorch-lightning: 2.0.8 |
|
|
- tqdm: 4.67.1 |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- Creating EMA weights copy. |
|
|
- Creating EMA weights copy. |
|
|
- Creating EMA weights copy. |
|
|
- Creating EMA weights copy. |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- eval_lh/avg_seq_len success rate 0.9677884578704834 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate with success 0.9791666666666666 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate with success 0.8012820512820513 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate with success 0.9391025641025641 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate with success 0.9583333333333333 |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- eval_lh/avg_seq_len success rate 0.9681089520454407 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate with success 0.9791666666666666 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate with success 0.9807692307692308 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate with success 0.9807692307692308 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate with success 0.8621794871794872 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate with success 0.9166666666666666 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate with success 0.9615384615384616 |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate |
|
|
- GPU 0 starting to evaluate: pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate |
|
|
- eval_lh/avg_seq_len success rate 0.9621794819831848 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate with success 0.9807692307692308 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate with success 0.9599358974358975 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate with success 0.9615384615384616 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate with success 0.8990384615384616 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate with success 0.8798076923076923 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate with success 1.0 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate with success 0.9791666666666666 |
|
|
- eval_lh/sr_pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate with success 0.9615384615384616 |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 3.73GB, Reserved: 3.83GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 3.73GB, Reserved: 3.79GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 3.73GB, Reserved: 3.83GB |
|
|
- GPU 0 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 1 Memory: Allocated: 3.73GB, Reserved: 3.79GB |
|
|
- GPU 2 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
- GPU 3 Memory: Allocated: 0.00GB, Reserved: 0.00GB |
|
|
|
