Llama3-OpenBioLLM-8B-PsyCourse-fold4

This model is a fine-tuned version of aaditya/Llama3-OpenBioLLM-8B on the course-train-fold4 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0349

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.4799	0.0763	50	0.2933
0.1053	0.1527	100	0.0766
0.0594	0.2290	150	0.0722
0.0586	0.3053	200	0.0537
0.066	0.3816	250	0.0550
0.0553	0.4580	300	0.0551
0.0468	0.5343	350	0.0476
0.0515	0.6106	400	0.0481
0.0562	0.6870	450	0.0466
0.0347	0.7633	500	0.0401
0.0417	0.8396	550	0.0408
0.0466	0.9159	600	0.0393
0.0346	0.9923	650	0.0404
0.035	1.0686	700	0.0389
0.0408	1.1449	750	0.0390
0.0324	1.2213	800	0.0427
0.0331	1.2976	850	0.0382
0.0335	1.3739	900	0.0433
0.034	1.4502	950	0.0374
0.0253	1.5266	1000	0.0399
0.0299	1.6029	1050	0.0382
0.0534	1.6792	1100	0.0424
0.0318	1.7556	1150	0.0402
0.0484	1.8319	1200	0.0385
0.0263	1.9082	1250	0.0355
0.0329	1.9845	1300	0.0349
0.0171	2.0609	1350	0.0365
0.0204	2.1372	1400	0.0374
0.0248	2.2135	1450	0.0413
0.0132	2.2899	1500	0.0397
0.0152	2.3662	1550	0.0383
0.0222	2.4425	1600	0.0387
0.0187	2.5188	1650	0.0374
0.0177	2.5952	1700	0.0415
0.0164	2.6715	1750	0.0380
0.0212	2.7478	1800	0.0395
0.0248	2.8242	1850	0.0357
0.0187	2.9005	1900	0.0384
0.0315	2.9768	1950	0.0372
0.006	3.0531	2000	0.0423
0.0077	3.1295	2050	0.0459
0.0073	3.2058	2100	0.0493
0.0096	3.2821	2150	0.0523
0.0086	3.3585	2200	0.0449
0.0057	3.4348	2250	0.0469
0.0098	3.5111	2300	0.0460
0.0086	3.5874	2350	0.0493
0.0073	3.6638	2400	0.0471
0.0086	3.7401	2450	0.0468
0.0079	3.8164	2500	0.0455
0.0042	3.8928	2550	0.0473
0.0088	3.9691	2600	0.0474
0.004	4.0454	2650	0.0474
0.0033	4.1217	2700	0.0500
0.0009	4.1981	2750	0.0520
0.0021	4.2744	2800	0.0532
0.0042	4.3507	2850	0.0546
0.0017	4.4271	2900	0.0566
0.002	4.5034	2950	0.0577
0.0021	4.5797	3000	0.0583
0.0017	4.6560	3050	0.0585
0.0036	4.7324	3100	0.0587
0.0021	4.8087	3150	0.0588
0.0028	4.8850	3200	0.0588
0.0021	4.9614	3250	0.0588

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3