Files changed (1) hide show
  1. README.md +111 -97
README.md CHANGED
@@ -1,97 +1,111 @@
1
- ---
2
- license: apache-2.0
3
- license_link: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/LICENSE
4
- base_model:
5
- - Qwen/Qwen2.5-7B-Instruct
6
- ---
7
- # Qwen2.5-7B-Instruct-fp16-ov
8
- * Model creator: [Qwen](https://huggingface.co/Qwen)
9
- * Original model: [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
10
-
11
- ## Description
12
- This is [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2025/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to FP16.
13
-
14
-
15
- ## Compatibility
16
-
17
- The provided OpenVINO™ IR model is compatible with:
18
-
19
- * OpenVINO version 2025.1.0 and higher
20
- * Optimum Intel 1.24.0 and higher
21
-
22
- ## Running Model Inference with [Optimum Intel](https://huggingface.co/docs/optimum/intel/index)
23
-
24
- 1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
25
-
26
- ```
27
- pip install optimum[openvino]
28
- ```
29
-
30
- 2. Run model inference:
31
-
32
- ```
33
- from transformers import AutoTokenizer
34
- from optimum.intel.openvino import OVModelForCausalLM
35
-
36
- model_id = "OpenVINO/qwen2.5-7b-instruct-fp16-ov"
37
- tokenizer = AutoTokenizer.from_pretrained(model_id)
38
- model = OVModelForCausalLM.from_pretrained(model_id)
39
-
40
- inputs = tokenizer("What is OpenVINO?", return_tensors="pt")
41
-
42
- outputs = model.generate(**inputs, max_length=200)
43
- text = tokenizer.batch_decode(outputs)[0]
44
- print(text)
45
- ```
46
-
47
- For more examples and possible optimizations, refer to the [Inference with Optimum Intel](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-optimum-intel.html).
48
-
49
- ## Running Model Inference with [OpenVINO GenAI](https://github.com/openvinotoolkit/openvino.genai)
50
-
51
-
52
- 1. Install packages required for using OpenVINO GenAI.
53
- ```
54
- pip install openvino-genai huggingface_hub
55
- ```
56
-
57
- 2. Download model from HuggingFace Hub
58
-
59
- ```
60
- import huggingface_hub as hf_hub
61
-
62
- model_id = "OpenVINO/qwen2.5-7b-instruct-fp16-ov"
63
- model_path = "qwen2.5-7b-instruct-fp16-ov"
64
-
65
- hf_hub.snapshot_download(model_id, local_dir=model_path)
66
-
67
- ```
68
-
69
- 3. Run model inference:
70
-
71
- ```
72
- import openvino_genai as ov_genai
73
-
74
- device = "CPU"
75
- pipe = ov_genai.LLMPipeline(model_path, device)
76
- print(pipe.generate("What is OpenVINO?", max_length=200))
77
- ```
78
-
79
- More GenAI usage examples can be found in OpenVINO GenAI library [docs](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-genai.html) and [samples](https://github.com/openvinotoolkit/openvino.genai?tab=readme-ov-file#openvino-genai-samples)
80
-
81
- You can find more detaild usage examples in OpenVINO Notebooks:
82
-
83
- - [LLM](https://openvinotoolkit.github.io/openvino_notebooks/?search=LLM)
84
- - [RAG text generation](https://openvinotoolkit.github.io/openvino_notebooks/?search=RAG+system&tasks=Text+Generation)
85
- - [Convert models from ModelScope to OpenVINO](https://openvinotoolkit.github.io/openvino_notebooks/?search=Convert+models+from+ModelScope+to+OpenVINO)
86
-
87
- ## Limitations
88
-
89
- Check the original [model card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) for limitations.
90
-
91
- ## Legal information
92
-
93
- The original model is distributed under [Apache License Version 2.0](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/LICENSE) license. More details can be found in [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
94
-
95
- ## Disclaimer
96
-
97
- Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ license_link: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/LICENSE
4
+ base_model:
5
+ - Qwen/Qwen2.5-7B-Instruct
6
+ language:
7
+ - zho
8
+ - eng
9
+ - fra
10
+ - spa
11
+ - por
12
+ - deu
13
+ - ita
14
+ - rus
15
+ - jpn
16
+ - kor
17
+ - vie
18
+ - tha
19
+ - ara
20
+ ---
21
+ # Qwen2.5-7B-Instruct-fp16-ov
22
+ * Model creator: [Qwen](https://huggingface.co/Qwen)
23
+ * Original model: [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
24
+
25
+ ## Description
26
+ This is [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2025/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to FP16.
27
+
28
+
29
+ ## Compatibility
30
+
31
+ The provided OpenVINO™ IR model is compatible with:
32
+
33
+ * OpenVINO version 2025.1.0 and higher
34
+ * Optimum Intel 1.24.0 and higher
35
+
36
+ ## Running Model Inference with [Optimum Intel](https://huggingface.co/docs/optimum/intel/index)
37
+
38
+ 1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
39
+
40
+ ```
41
+ pip install optimum[openvino]
42
+ ```
43
+
44
+ 2. Run model inference:
45
+
46
+ ```
47
+ from transformers import AutoTokenizer
48
+ from optimum.intel.openvino import OVModelForCausalLM
49
+
50
+ model_id = "OpenVINO/qwen2.5-7b-instruct-fp16-ov"
51
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
52
+ model = OVModelForCausalLM.from_pretrained(model_id)
53
+
54
+ inputs = tokenizer("What is OpenVINO?", return_tensors="pt")
55
+
56
+ outputs = model.generate(**inputs, max_length=200)
57
+ text = tokenizer.batch_decode(outputs)[0]
58
+ print(text)
59
+ ```
60
+
61
+ For more examples and possible optimizations, refer to the [Inference with Optimum Intel](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-optimum-intel.html).
62
+
63
+ ## Running Model Inference with [OpenVINO GenAI](https://github.com/openvinotoolkit/openvino.genai)
64
+
65
+
66
+ 1. Install packages required for using OpenVINO GenAI.
67
+ ```
68
+ pip install openvino-genai huggingface_hub
69
+ ```
70
+
71
+ 2. Download model from HuggingFace Hub
72
+
73
+ ```
74
+ import huggingface_hub as hf_hub
75
+
76
+ model_id = "OpenVINO/qwen2.5-7b-instruct-fp16-ov"
77
+ model_path = "qwen2.5-7b-instruct-fp16-ov"
78
+
79
+ hf_hub.snapshot_download(model_id, local_dir=model_path)
80
+
81
+ ```
82
+
83
+ 3. Run model inference:
84
+
85
+ ```
86
+ import openvino_genai as ov_genai
87
+
88
+ device = "CPU"
89
+ pipe = ov_genai.LLMPipeline(model_path, device)
90
+ print(pipe.generate("What is OpenVINO?", max_length=200))
91
+ ```
92
+
93
+ More GenAI usage examples can be found in OpenVINO GenAI library [docs](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-genai.html) and [samples](https://github.com/openvinotoolkit/openvino.genai?tab=readme-ov-file#openvino-genai-samples)
94
+
95
+ You can find more detaild usage examples in OpenVINO Notebooks:
96
+
97
+ - [LLM](https://openvinotoolkit.github.io/openvino_notebooks/?search=LLM)
98
+ - [RAG text generation](https://openvinotoolkit.github.io/openvino_notebooks/?search=RAG+system&tasks=Text+Generation)
99
+ - [Convert models from ModelScope to OpenVINO](https://openvinotoolkit.github.io/openvino_notebooks/?search=Convert+models+from+ModelScope+to+OpenVINO)
100
+
101
+ ## Limitations
102
+
103
+ Check the original [model card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) for limitations.
104
+
105
+ ## Legal information
106
+
107
+ The original model is distributed under [Apache License Version 2.0](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/LICENSE) license. More details can be found in [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
108
+
109
+ ## Disclaimer
110
+
111
+ Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.