File size: 2,179 Bytes
a020885 d2b71ac a020885 0b7c192 a020885 b2e59e0 a020885 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
---
license: mit
tags:
- tinyllama
- sciq
- multiple-choice
- peft
- lora
- 4bit
- quantization
- instruction-tuning
datasets:
- allenai/sciq
language:
- en
library_name: transformers
pipeline_tag: text-generation
---
# 🧠 TinyLLaMA-1.1B LoRA Fine-tuned on SciQ Dataset
This is a **TinyLLaMA-1.1B** model fine-tuned using **LoRA (Low-Rank Adaptation)** on the [SciQ](https://huggingface.co/datasets/allenai/sciq) multiple-choice question answering dataset. It uses **4-bit quantization** via `bitsandbytes` to reduce memory usage and improve inference efficiency.
## 🧪 Use Cases
This model is suitable for:
- Educational QA bots
- MCQ-style reasoning
- Lightweight inference on constrained hardware (e.g., GPUs with <8GB VRAM)
## 🛠️ Training Details
- Base Model: `TinyLlama/TinyLlama-1.1B-Chat-v1.0`
- Dataset: `allenai/sciq` (Science QA)
- Method: Parameter-Efficient Fine-Tuning using LoRA
- Quantization: 4-bit using `bitsandbytes`
- Framework: 🤗 Transformers + PEFT + Datasets
## 🧬 Model Architecture
- Model: Causal Language Model
- Fine-tuned layers: `q_proj`, `v_proj` (via LoRA)
- Quantization: 4-bit (bnb config)
## 📊 Evaluation
- Accuracy: **100%** on a 1000-sample SciQ subset
- Eval Loss: ~0.19
## 💡 How to Use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("TechyCode/tinyllama-sciq-lora")
tokenizer = AutoTokenizer.from_pretrained("TechyCode/tinyllama-sciq-lora")
prompt = """Question: What is the boiling point of water?\nChoices:\nA. 50°C\nB. 75°C\nC. 90°C\nD. 100°C\nAnswer:"""
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## 🔐 License
This model is released under the MIT License.
## 🙌 Credits
FineTuned By - [Uditanshu Pandey](https://huggingface.co/TechyCode)\
Linkedin - [UditanshuPandey](https://www.linkedin.com/in/uditanshupandey)\
GitHub - [UditanshuPandey](https://github.com/UditanshuPandey)\
Based on - [TinyLLaMA-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|