2-bit Qwen2.5 72B Instruct
Collection
6 items
•
Updated
This is Qwen/Qwen2.5-72B-Instruct quantized with AutoRound in 2-bit (symmetric + gptq format) with a group size of 32 and calibration samples of 4096 tokens. The model has been created, tested, and evaluated by The Kaitchup. The model is compatible with vLLM and Transformers.
More details in this article: Accurate 2-bit Quantization: Run Massive LLMs on a Single Consumer GPU
- Developed by: The Kaitchup
- License: Apache 2.0 license
How to Support My Work
Subscribe to The Kaitchup. This helps me a lot to continue quantizing and evaluating models for free.
- Downloads last month
- 54
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support