rbao2018 commited on
Commit
8dcffca
·
verified ·
1 Parent(s): e2cad8a

add eos token and the end of assistant content

Browse files

1. **Original Implementation Flaw**: Failed to append End-of-Sequence (EOS) tokens to language model assistant responses in each conversation turn.
2. **Resulting Issue**: During inference, model couldn't recognize termination signals, leading to continuous text generation beyond the expected output end.
3. **Impact**: Compromised the integrity and controllability of conversational interactions.

Files changed (1) hide show
  1. tokenizer_config.json +1 -1
tokenizer_config.json CHANGED
@@ -10,7 +10,7 @@
10
  "<|number_end|>"
11
  ],
12
  "bos_token": "<|startoftext|>",
13
- "chat_template": "{% for message in messages %}{% set role = message['role'] | lower %}{% if role == 'user' %}{% set role = 'HUMAN' %}{% endif %}{% set role = role | upper %}{{ '<role>' + role + '</role>' + message['content'] }}{% endfor %}{% if add_generation_prompt %}{{ '<role>ASSISTANT</role>' }}{% endif %}",
14
  "clean_up_tokenization_spaces": false,
15
  "cls_token": "[CLS]",
16
  "eos_token": "<|endoftext|>",
 
10
  "<|number_end|>"
11
  ],
12
  "bos_token": "<|startoftext|>",
13
+ "chat_template": "{% for message in messages %}{% set role = message['role'] | lower %}{% if role == 'user' %}{% set role = 'HUMAN' %}{% endif %}{% set role = role | upper %}{{ '<role>' + role + '</role>' + message['content'] }}{% if role == 'ASSISTANT' %}{{ eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<role>ASSISTANT</role>' }}{% endif %}",
14
  "clean_up_tokenization_spaces": false,
15
  "cls_token": "[CLS]",
16
  "eos_token": "<|endoftext|>",