|
|
|
|
|
--- |
|
|
language: en |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- intellite |
|
|
- "small-language-model" |
|
|
- transformers |
|
|
- pytorch |
|
|
- conversational-ai |
|
|
- huggingface |
|
|
- low-resource |
|
|
--- |
|
|
|
|
|
# IntellIte Chat |
|
|
|
|
|
IntellIte Chat is a lightweight conversational AI model (~45M parameters) designed for warm, engaging dialogue and basic reasoning. Part of the *IntellIte* series, it delivers efficient performance on modest hardware, complete with streaming data loading, episodic memory buffers, and RAG-based knowledge augmentation. |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚙️ Key Features |
|
|
|
|
|
- **Small & Efficient**: ~45M parameters, ideal for edge devices and academic projects. |
|
|
- **Streaming Data**: Uses Hugging Face `IterableDataset` for on-the-fly data without local storage constraints. |
|
|
- **Memory Buffer**: Maintains the last 200 messages for coherent multi-turn conversations. |
|
|
- **RAG Integration**: FAISS-based retrieval for up-to-date knowledge augmentation. |
|
|
- **Content Safety**: Built-in filters to enforce conversational guidelines. |
|
|
- **Extensible API**: Hook into `generate_with_plugins()` for custom prompts or downstream tasks. |
|
|
|
|
|
--- |
|
|
|
|
|
## 💾 Installation |
|
|
|
|
|
```bash |
|
|
pip install transformers datasets faiss-cpu torch huggingface-hub |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 Quick Start |
|
|
|
|
|
```python |
|
|
from il import generate_with_plugins |
|
|
|
|
|
response = generate_with_plugins( |
|
|
prompt="Hello, how's it going?", |
|
|
source="wiki", |
|
|
k=3, |
|
|
max_new_tokens=100 |
|
|
) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 🛠️ Training Pipeline |
|
|
|
|
|
Run the main training script: |
|
|
|
|
|
```bash |
|
|
export HF_TOKEN=<your_hf_token> |
|
|
python il.py --hf_token $HF_TOKEN --seed 42 |
|
|
``` |
|
|
|
|
|
The script will: |
|
|
|
|
|
1. Stream Wikipedia, CodeParrot, and grade-school math datasets. |
|
|
2. Apply cosine LR scheduling, weight decay, and label smoothing. |
|
|
3. Run simple evals (2 chat, 1 code prompt) at each epoch end. |
|
|
4. Save & push the best model to `ProCreations/IntellIte` on Hugging Face. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📊 Evaluation & Monitoring |
|
|
|
|
|
A `SimpleEvalCallback` runs designated chat/code prompts each epoch, logging outputs for quick sanity checks. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔧 Configuration Options |
|
|
|
|
|
Edit `il.py` to customize: |
|
|
|
|
|
- **Batch Sizes, LR, Scheduler** via `TrainingArguments`. |
|
|
- **Retrieval Sources**: adjust `k` and index sources. |
|
|
- **Memory Buffer**: change size or filter rules. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🌱 Fine‑Tuning on Custom Data |
|
|
|
|
|
1. Prepare your dataset as a Hugging Face `Dataset` or `IterableDataset`. |
|
|
2. Interleave with base streams and pass to the Trainer. |
|
|
3. Use `--resume_from_checkpoint` to continue an interrupted run. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🤝 Contributing |
|
|
|
|
|
Contributions welcome! Steps: |
|
|
|
|
|
1. Fork the repo. |
|
|
2. Create a feature branch. |
|
|
3. Submit a PR with clear descriptions and tests. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📜 License |
|
|
|
|
|
This project is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0). |
|
|
|
|
|
--- |
|
|
|
|
|
❤️ Developed by ProCreations under the *IntellIte* brand. |
|
|
|
|
|
|
