|
--- |
|
license: mit |
|
tags: |
|
- tinyllama |
|
- sciq |
|
- multiple-choice |
|
- peft |
|
- lora |
|
- 4bit |
|
- quantization |
|
- instruction-tuning |
|
datasets: |
|
- allenai/sciq |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# 🧠 TinyLLaMA-1.1B LoRA Fine-tuned on SciQ Dataset |
|
|
|
This is a **TinyLLaMA-1.1B** model fine-tuned using **LoRA (Low-Rank Adaptation)** on the [SciQ](https://huggingface.co/datasets/allenai/sciq) multiple-choice question answering dataset. It uses **4-bit quantization** via `bitsandbytes` to reduce memory usage and improve inference efficiency. |
|
|
|
## 🧪 Use Cases |
|
|
|
This model is suitable for: |
|
|
|
- Educational QA bots |
|
- MCQ-style reasoning |
|
- Lightweight inference on constrained hardware (e.g., GPUs with <8GB VRAM) |
|
|
|
## 🛠️ Training Details |
|
|
|
- Base Model: `TinyLlama/TinyLlama-1.1B-Chat-v1.0` |
|
- Dataset: `allenai/sciq` (Science QA) |
|
- Method: Parameter-Efficient Fine-Tuning using LoRA |
|
- Quantization: 4-bit using `bitsandbytes` |
|
- Framework: 🤗 Transformers + PEFT + Datasets |
|
|
|
## 🧬 Model Architecture |
|
|
|
- Model: Causal Language Model |
|
- Fine-tuned layers: `q_proj`, `v_proj` (via LoRA) |
|
- Quantization: 4-bit (bnb config) |
|
|
|
## 📊 Evaluation |
|
|
|
- Accuracy: **100%** on a 1000-sample SciQ subset |
|
- Eval Loss: ~0.19 |
|
|
|
## 💡 How to Use |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("TechyCode/tinyllama-sciq-lora") |
|
tokenizer = AutoTokenizer.from_pretrained("TechyCode/tinyllama-sciq-lora") |
|
|
|
prompt = """Question: What is the boiling point of water?\nChoices:\nA. 50°C\nB. 75°C\nC. 90°C\nD. 100°C\nAnswer:""" |
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
outputs = model.generate(**inputs, max_new_tokens=20) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
``` |
|
## 🔐 License |
|
This model is released under the MIT License. |
|
|
|
## 🙌 Credits |
|
FineTuned By - [Uditanshu Pandey](https://huggingface.co/TechyCode)\ |
|
Linkedin - [UditanshuPandey](https://www.linkedin.com/in/uditanshupandey)\ |
|
GitHub - [UditanshuPandey](https://github.com/UditanshuPandey)\ |
|
Based on - [TinyLLaMA-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) |
|
|
