Falcondette / tokenizer_config.json
Raiff1982's picture
Create tokenizer_config.json
c9bf3f7 verified
raw
history blame contribute delete
191 Bytes
{
"tokenizer_class": "AutoTokenizer",
"model_max_length": 2048,
"padding_side": "right",
"truncation_side": "right",
"special_tokens_map_file": "special_tokens_map.json"
}