sg>dZddlmZmZddlmZddlmZddlm Z ddl m Z e je Zd d d d Zd diZGddeZy)zTokenization classes for Qwen2.)OptionalTuple) AddedToken)PreTrainedTokenizerFast)logging)Qwen2Tokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filezqwen/qwen-tokenizericdeZdZdZeZddgZeZ d fd Z d de de e de e fdZ xZS) Qwen2TokenizerFastac Construct a "fast" Qwen2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import Qwen2TokenizerFast >>> tokenizer = Qwen2TokenizerFast.from_pretrained("Qwen/Qwen-tokenizer") >>> tokenizer("Hello world")["input_ids"] [9707, 1879] >>> tokenizer(" Hello world")["input_ids"] [21927, 1879] ``` This is expected. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Not applicable to this tokenizer. bos_token (`str`, *optional*): The beginning of sequence token. Not applicable for this tokenizer. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc Dt|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t |d|||||||d|y)NFT)lstriprstripspecial normalized)r r r unk_token bos_token eos_token pad_token) isinstancestrrsuper__init__) selfr r r rrrrkwargs __class__s d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.pyrzQwen2TokenizerFast.__init__Ss$)S) yudW\ ] )S) yudW\ ] )S) yudW\ ] )S) yudW\ ]   !#)  save_directoryfilename_prefixreturncf|jjj||}t|S)N)name) _tokenizermodelsavetuple)r r%r&filess r#save_vocabularyz"Qwen2TokenizerFast.save_vocabularys+%%**>*PU|r$)NNN <|endoftext|>Nr0r0)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr slow_tokenizer_classrrrrr/ __classcell__)r"s@r#rr%se'R*$&67)!!!. bcHSM]bcf]gr$rN)r4typingrrtokenization_utilsrtokenization_utils_fastrutilsrtokenization_qwen2r get_loggerr1loggerr5MAX_MODEL_INPUT_SIZESrrr$r#rBs\&",>.   H %&/6a0ar$