lemexp-task2-unordered_template_small-deepseek-coder-1.3b-base-ddp-8lr

This model is a fine-tuned version of deepseek-ai/deepseek-coder-1.3b-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2327

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0008
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 12
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.4681	0.2001	629	0.3776
0.379	0.4001	1258	0.3465
0.3592	0.6002	1887	0.3382
0.334	0.8003	2516	0.3192
0.3264	1.0003	3145	0.3150
0.3122	1.2004	3774	0.3127
0.3073	1.4004	4403	0.3033
0.2982	1.6005	5032	0.2978
0.2974	1.8006	5661	0.2936
0.2911	2.0006	6290	0.2884
0.2871	2.2007	6919	0.2861
0.2803	2.4008	7548	0.2783
0.277	2.6008	8177	0.2820
0.2778	2.8009	8806	0.2738
0.272	3.0010	9435	0.2729
0.2578	3.2010	10064	0.2702
0.2596	3.4011	10693	0.2718
0.2586	3.6011	11322	0.2648
0.257	3.8012	11951	0.2595
0.2577	4.0013	12580	0.2591
0.2431	4.2013	13209	0.2589
0.2401	4.4014	13838	0.2598
0.2445	4.6015	14467	0.2537
0.2403	4.8015	15096	0.2544
0.2405	5.0016	15725	0.2535
0.2311	5.2017	16354	0.2533
0.227	5.4017	16983	0.2508
0.2259	5.6018	17612	0.2450
0.2238	5.8018	18241	0.2445
0.2254	6.0019	18870	0.2419
0.2201	6.2020	19499	0.2426
0.2096	6.4020	20128	0.2432
0.2088	6.6021	20757	0.2440
0.2078	6.8022	21386	0.2386
0.2114	7.0022	22015	0.2388
0.1919	7.2023	22644	0.2388
0.1939	7.4024	23273	0.2351
0.1947	7.6024	23902	0.2330
0.1939	7.8025	24531	0.2283
0.1916	8.0025	25160	0.2262
0.179	8.2026	25789	0.2330
0.1784	8.4027	26418	0.2311
0.1781	8.6027	27047	0.2327
0.1772	8.8028	27676	0.2260
0.1783	9.0029	28305	0.2234
0.1688	9.2029	28934	0.2323
0.1607	9.4030	29563	0.2269
0.1613	9.6031	30192	0.2249
0.1622	9.8031	30821	0.2253
0.1602	10.0032	31450	0.2239
0.1444	10.2032	32079	0.2283
0.1461	10.4033	32708	0.2281
0.1449	10.6034	33337	0.2282
0.1455	10.8034	33966	0.2232
0.1431	11.0035	34595	0.2277
0.1334	11.2036	35224	0.2302
0.1304	11.4036	35853	0.2322
0.1304	11.6037	36482	0.2313
0.1291	11.8038	37111	0.2327

Framework versions

• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0