sgZdZddlmZmZmZddlmZddlmZddl m Z ddl m Z e jeZd d d d ZGd deZy)z$Tokenization classes for OpenAI GPT.)ListOptionalTuple)pre_tokenizers)PreTrainedTokenizerFast)logging) CLIPTokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_fileceZdZdZeZddgZeZ dfd Z dZ dde e de e e de e fd Z dde e de e e de e fd Zdd ed e edeefd ZxZS)CLIPTokenizerFasta Construct a "fast" CLIP tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): The path to a tokenizer file to use instead of the vocab file. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc t |||f|||||d|t|jjt j s td|jy)N)r unk_token bos_token eos_token pad_tokenaThe `backend_tokenizer` provided does not match the expected format. The CLIP tokenizer has been heavily modified from transformers version 4.17.0. You need to convert the tokenizer you are using to be compatible with this version.The easiest way to do so is `CLIPTokenizerFast.from_pretrained("path_to_local_folder_or_hub_repo, from_slow=True)`. If you want to use your existing tokenizer, you will have to revert to a version prior to 4.17.0 of transformers.) super__init__ isinstancebackend_tokenizer pre_tokenizerrSequence ValueError%_wrap_decode_method_backend_tokenizer) selfr r rrrrrkwargs __class__s b/var/www/html/venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip_fast.pyrzCLIPTokenizerFast.__init__=sv    *  $00>>@W@WX!  224c|jj|jjjfd}||j_y)NcX|i|}|jdj}|S)N )replacestrip)argsr!textend_of_word_suffixorig_decode_methods r#new_decode_methodzRCLIPTokenizerFast._wrap_decode_method_backend_tokenizer..new_decode_methodfs1%t6v6D<< 2C8>>@DKr$)rdecodemodelr,)r r.r,r-s @@r#rz7CLIPTokenizerFast._wrap_decode_method_backend_tokenizer_sD!33::"3399LL  ):%r$ token_ids_0 token_ids_1returncl|jg}|jg}|||z|zS||z|z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idr r1r2rrs r# build_inputs_with_special_tokensz2CLIPTokenizerFast.build_inputs_with_special_tokensmsU(&&' &&'  {*Y6 6;&2Y>LyXXr$c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)r5r6lenr7s r#$create_token_type_ids_from_sequencesz6CLIPTokenizerFast.create_token_type_ids_from_sequencessp &&' &&'  y;.:;qcA A9{*Y6B[PS\\]ab`cccr$save_directoryfilename_prefixcf|jjj||}t|S)N)name) _tokenizerr0savetuple)r r<r=filess r#save_vocabularyz!CLIPTokenizerFast.save_vocabularys+%%**>*PU|r$)NNN <|endoftext|>z<|startoftext|>rErE)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr slow_tokenizer_classrrrintrr8r;strrrD __classcell__)r"s@r#rrs2*$&67(!#!!5D :JNY9Y3;DI3FY cY8JNd9d3;DI3Fd cd.cHSM]bcf]gr$rN)rItypingrrr tokenizersrtokenization_utils_fastrutilsr tokenization_clipr get_loggerrFloggerrJrr$r#rYsM+((%>,   H %#/ `pqB/Br$