-
- Downloads
Fix: SentencePiece underscore fixed
Showing
- resources/qwen2tokenizer_fast/added_tokens.json 24 additions, 0 deletionsresources/qwen2tokenizer_fast/added_tokens.json
- resources/qwen2tokenizer_fast/merges.txt 176450 additions, 0 deletionsresources/qwen2tokenizer_fast/merges.txt
- resources/qwen2tokenizer_fast/special_tokens_map.json 31 additions, 0 deletionsresources/qwen2tokenizer_fast/special_tokens_map.json
- resources/qwen2tokenizer_fast/tokenizer.json 871526 additions, 0 deletionsresources/qwen2tokenizer_fast/tokenizer.json
- resources/qwen2tokenizer_fast/tokenizer_config.json 208 additions, 0 deletionsresources/qwen2tokenizer_fast/tokenizer_config.json
- resources/qwen2tokenizer_fast/vocab.json 1 addition, 0 deletionsresources/qwen2tokenizer_fast/vocab.json
- src/tokenization/bpe.py 10 additions, 1 deletionsrc/tokenization/bpe.py
- src/tokenization/configure_special_tokens.py 3 additions, 3 deletionssrc/tokenization/configure_special_tokens.py
- src/tokenization/localisation_mn.py 12 additions, 32 deletionssrc/tokenization/localisation_mn.py
- src/tokenization/merges.py 0 additions, 31 deletionssrc/tokenization/merges.py
- src/tokenization/qwen_post_tokenizer/merges.txt 68 additions, 68 deletionssrc/tokenization/qwen_post_tokenizer/merges.txt
- src/tokenization/qwen_post_tokenizer/vocab.json 1 addition, 1 deletionsrc/tokenization/qwen_post_tokenizer/vocab.json
Loading
Please register or sign in to comment