add eos token and the end of assistant content
Browse files1. **Original Implementation Flaw**: Failed to append End-of-Sequence (EOS) tokens to language model assistant responses in each conversation turn.
2. **Resulting Issue**: During inference, model couldn't recognize termination signals, leading to continuous text generation beyond the expected output end.
3. **Impact**: Compromised the integrity and controllability of conversational interactions.
- tokenizer_config.json +1 -1
tokenizer_config.json
CHANGED
@@ -10,7 +10,7 @@
|
|
10 |
"<|number_end|>"
|
11 |
],
|
12 |
"bos_token": "<|startoftext|>",
|
13 |
-
"chat_template": "{% for message in messages %}{% set role = message['role'] | lower %}{% if role == 'user' %}{% set role = 'HUMAN' %}{% endif %}{% set role = role | upper %}{{ '<role>' + role + '</role>' + message['content'] }}{% endfor %}{% if add_generation_prompt %}{{ '<role>ASSISTANT</role>' }}{% endif %}",
|
14 |
"clean_up_tokenization_spaces": false,
|
15 |
"cls_token": "[CLS]",
|
16 |
"eos_token": "<|endoftext|>",
|
|
|
10 |
"<|number_end|>"
|
11 |
],
|
12 |
"bos_token": "<|startoftext|>",
|
13 |
+
"chat_template": "{% for message in messages %}{% set role = message['role'] | lower %}{% if role == 'user' %}{% set role = 'HUMAN' %}{% endif %}{% set role = role | upper %}{{ '<role>' + role + '</role>' + message['content'] }}{% if role == 'ASSISTANT' %}{{ eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<role>ASSISTANT</role>' }}{% endif %}",
|
14 |
"clean_up_tokenization_spaces": false,
|
15 |
"cls_token": "[CLS]",
|
16 |
"eos_token": "<|endoftext|>",
|