Llama3-OpenBioLLM-8B-PsyCourse-fold5

This model is a fine-tuned version of aaditya/Llama3-OpenBioLLM-8B on the course-train-fold5 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0349

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.5262	0.0758	50	0.3222
0.0888	0.1517	100	0.0871
0.0821	0.2275	150	0.0828
0.0718	0.3033	200	0.0627
0.0622	0.3791	250	0.0543
0.0509	0.4550	300	0.0504
0.0576	0.5308	350	0.0506
0.066	0.6066	400	0.0472
0.0593	0.6825	450	0.0450
0.0316	0.7583	500	0.0445
0.0318	0.8341	550	0.0472
0.0436	0.9100	600	0.0449
0.0454	0.9858	650	0.0416
0.0339	1.0616	700	0.0424
0.0369	1.1374	750	0.0448
0.034	1.2133	800	0.0374
0.0332	1.2891	850	0.0419
0.029	1.3649	900	0.0428
0.031	1.4408	950	0.0432
0.0353	1.5166	1000	0.0388
0.0278	1.5924	1050	0.0380
0.0261	1.6682	1100	0.0366
0.0294	1.7441	1150	0.0390
0.0337	1.8199	1200	0.0366
0.0359	1.8957	1250	0.0350
0.0358	1.9716	1300	0.0363
0.0233	2.0474	1350	0.0363
0.0157	2.1232	1400	0.0363
0.0227	2.1991	1450	0.0357
0.018	2.2749	1500	0.0389
0.0237	2.3507	1550	0.0354
0.025	2.4265	1600	0.0365
0.015	2.5024	1650	0.0392
0.0265	2.5782	1700	0.0360
0.0181	2.6540	1750	0.0368
0.0228	2.7299	1800	0.0380
0.0224	2.8057	1850	0.0366
0.0162	2.8815	1900	0.0356
0.0284	2.9573	1950	0.0349
0.0069	3.0332	2000	0.0351
0.0065	3.1090	2050	0.0394
0.0069	3.1848	2100	0.0438
0.006	3.2607	2150	0.0446
0.0088	3.3365	2200	0.0436
0.0056	3.4123	2250	0.0424
0.0086	3.4882	2300	0.0445
0.0058	3.5640	2350	0.0438
0.0092	3.6398	2400	0.0426
0.0082	3.7156	2450	0.0433
0.01	3.7915	2500	0.0422
0.0082	3.8673	2550	0.0426
0.012	3.9431	2600	0.0433
0.0054	4.0190	2650	0.0437
0.0037	4.0948	2700	0.0452
0.0049	4.1706	2750	0.0474
0.0024	4.2464	2800	0.0492
0.0023	4.3223	2850	0.0507
0.0048	4.3981	2900	0.0512
0.0073	4.4739	2950	0.0514
0.0014	4.5498	3000	0.0518
0.0021	4.6256	3050	0.0521
0.0043	4.7014	3100	0.0525
0.005	4.7773	3150	0.0528
0.0021	4.8531	3200	0.0531
0.0032	4.9289	3250	0.0529

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3