|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- code |
|
|
- peft |
|
|
- sql-generation |
|
|
- text-generation-inference |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- gretelai/synthetic_text_to_sql |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# DeepSeek R1 Distill Qwen 1.5B finetuned for SQL query generation |
|
|
This model is a fine-tuned version of DeepSeek R1 Distill Qwen 1.5B, specifically optimized for SQL query generation. It has been trained on the GretelAI Synthetic Text-to-SQL dataset to enhance its ability to convert natural language prompts into accurate SQL queries. |
|
|
|
|
|
Due to its lightweight architecture, this model can be deployed efficiently on local machines without requiring a GPU, making it ideal for on-premises inference in resource-constrained environments. It offers a balance between performance and efficiency, making it suitable for businesses and developers looking for a cost-effective SQL generation solution. |
|
|
|
|
|
## Training Methodology |
|
|
1. Fine-tuning approach: LoRA (Low-Rank Adaptation) for efficient parameter tuning. |
|
|
2. Precision: bfloat16 (bf16) to reduce memory consumption while maintaining numerical stability. |
|
|
3. Gradient Accumulation: Used to handle larger batch sizes within GPU memory limits. |
|
|
4. Optimizer: AdamW with learning rate scheduling. |
|
|
5. Cosine Scheduler: Used cosine learning rate scheduler for training stability. (500 warm-up steps, 2000 steps for the cosine schedule.) |
|
|
|
|
|
## Use Cases |
|
|
1. Assisting developers and analysts in writing SQL queries. |
|
|
2. Automating SQL query generation from user prompts in chatbots. |
|
|
3. Enhancing SQL-based retrieval-augmented generation (RAG) systems. |
|
|
|
|
|
## Limitations & Considerations |
|
|
1. The model may generate incorrect or suboptimal SQL queries for complex database schemas. |
|
|
2. It does not perform schema reasoning and requires clear table/column references in the input. |
|
|
3. Further fine-tuning on domain-specific SQL data may be required for better accuracy. |
|
|
|
|
|
## How to Use |
|
|
You can load the model using 🤗 Transformers: |
|
|
|
|
|
```python |
|
|
from peft import AutoPeftModelForCausalLM |
|
|
from transformers import AutoTokenizer |
|
|
import torch |
|
|
|
|
|
model = AutoPeftModelForCausalLM.from_pretrained("NotShrirang/DeepSeek-R1-Distill-Qwen-1.5B-SQL-Coder-PEFT") |
|
|
tokenizer = AutoTokenizer.from_pretrained("NotShrirang/DeepSeek-R1-Distill-Qwen-1.5B-SQL-Coder-PEFT") |
|
|
|
|
|
prompt = "Write a SQL query to get the total revenue from the sales table." |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Total Steps:** 25,000 |
|
|
- **Batch Size:** 4 |
|
|
- **Optimizer:** AdamW |
|
|
- **Learning Rate:** 5e-5 |
|
|
|
|
|
### Training and Validation Loss Progression |
|
|
|
|
|
| Step | Training Loss | Validation Loss | |
|
|
|-------|--------------|----------------| |
|
|
| 1000 | 1.0017 | 1.0256 | |
|
|
| 2000 | 1.1644 | 0.8818 | |
|
|
| 3000 | 0.7851 | 0.8507 | |
|
|
| 4000 | 0.7416 | 0.8322 | |
|
|
| 5000 | 0.6960 | 0.8184 | |
|
|
| 6000 | 1.0118 | 0.8068 | |
|
|
| 7000 | 0.9897 | 0.7997 | |
|
|
| 8000 | 0.9165 | 0.7938 | |
|
|
| 9000 | 0.8048 | 0.7875 | |
|
|
| 10000 | 0.8869 | 0.7822 | |
|
|
| 11000 | 0.8387 | 0.7788 | |
|
|
| 12000 | 0.8117 | 0.7746 | |
|
|
| 13000 | 0.7259 | 0.7719 | |
|
|
| 14000 | 0.8100 | 0.7678 | |
|
|
| 15000 | 0.6901 | 0.7626 | |
|
|
| 16000 | 0.9630 | 0.7600 | |
|
|
| 17000 | 0.6599 | 0.7571 | |
|
|
| 18000 | 0.6770 | 0.7541 | |
|
|
| 19000 | 0.7360 | 0.7509 | |
|
|
| 20000 | 0.7170 | 0.7458 | |
|
|
| 21000 | 0.7993 | 0.7446 | |
|
|
| 22000 | 0.5846 | 0.7412 | |
|
|
| 23000 | 0.8269 | 0.7411 | |
|
|
| 24000 | 0.5817 | 0.7379 | |
|
|
| 25000 | 0.5772 | 0.7357 | |
|
|
|
|
|
- **Developed by:** [NotShrirang](https://huggingface.co/NotShrirang) |
|
|
- **Language(s) (NLP):** [en] |
|
|
- **License:** [apache-2.0] |
|
|
- **Finetuned from model :** [deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) |
