sgwdZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl mZmZmZddlmZddlmZdd lmZd d lmZmZd d lmZmZmZmZmZej@e!Z"d ddddZ#GddeZ$y)z!Tokenization classes for Whisper.N) lru_cache)ListOptionalTuple) AddedTokenpre_tokenizers processors) BatchEncoding)PreTrainedTokenizerFast)logging)BasicTextNormalizerEnglishTextNormalizer) LANGUAGESTASK_IDSTO_LANGUAGE_CODEWhisperTokenizer _decode_asrz vocab.jsonztokenizer.jsonz merges.txtznormalizer.json) vocab_filetokenizer_file merges_filenormalizer_fileceZdZdZeZddgZeZ d3fd Z de ffd Z de ffd Z d4de fd Zd5d Zed6d Zd7d efd ZdZ d8d edededededededede ffd Zdddddededede ffdZdZd7dZdZed7dZd9de dee dee fdZd:d e d!e d"efd#Z e!de"e#fd$Z$d9de"e#fd%Z% d;d&e"e#d'ee"e#d(ede"e#ffd) Z&dWhisperTokenizerFastag Construct a "fast" Whisper tokenizer (backed by HuggingFace's *tokenizers* library). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Whisper tokenizer detect beginning of words by the preceding space). language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsattention_maskFc 4t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t|||f|||||d| | j dd|_tj|jjj} | jd||k7r;tt| j d}|| d<|d i| |j_ |1t|d 5}tj ||_dddnd|_||_t'j(d |_| |_| |_| |_y#1swY@xYw) NFT)lstriprstrip normalizedspecial)r unk_token bos_token eos_tokenadd_prefix_space add_bos_tokenr&typeutf-8encodingz<\|(\d+\.\d+)\|>) isinstancestrrsuper__init__popr'jsonloadsbackend_tokenizer pre_tokenizer __getstate__getgetattrropenloadenglish_spelling_normalizerr&recompile timestamp_patlanguagetaskpredict_timestamps)selfrrrrr#r$r%r&r?r@rAkwargs pre_tok_state pre_tok_class vocab_handle __class__s h/var/www/html/venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper_fast.pyr0zWhisperTokenizerFast.__init__Ws")S) yuX\ ] )S) yuX\ ] )S) yuX\ ]     *-  $ZZ? 4#9#9#G#G#T#T#VW   /1A BFV V#NM4E4Ef4MNM0@M, -3@3Q=3QD " " 0  &o8 KL3799\3J0 K K04D , 0ZZ(;<   "4 K Ks ,FFreturnc|jdd}|js!|rJd|jjdt ||i|SNis_split_into_wordsFzYou need to instantiate z? with add_prefix_space=True to use it with pretokenized inputs.)r7r&rG__name__r/_batch_encode_plusrBargsrCrLrGs rHrNz'WhisperTokenizerFast._batch_encode_plussb$jj)>F$$,? &t~~'>'>&?@2 2 ? w)4:6::c|jdd}|js!|rJd|jjdt ||i|SrK)r7r&rGrMr/ _encode_plusrOs rHrSz!WhisperTokenizerFast._encode_plussb$jj)>F$$,? &t~~'>'>&?@2 2 ? w#T4V44rQc|jddz}gg}d}d}d} t|D]\} } | |k\r|t| |z |z} | |kr7| dk\xr|| dz |k\xr || dz |k\ } | r |||zz }n | }|| z }|dd}|} | }|jd| |zdd |jg|dj| |Dcgc]'}t |t r|n|j || )}}d j|Scc}w) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". rgNz<|z.2fz|>skip_special_tokens)all_special_ids enumeratefloatappendr-r.decodejoin)rB token_idsrYtime_precision segment_sizetimestamp_beginoutputscur_max_timestampprev_segments_lenpenultimate_timestampitoken timestamplast_was_single_endingss rH_decode_with_timestampsz,WhisperTokenizerFast._decode_with_timestampssr..r2Q6$ #!), *HAu'!5?#:n"LM 00-.!V.!!a%(O;c !a%@PTc@c9*.)^l-JJ),A))-BB)")#2,(9%$-!Y1B%BC#HKLr" ""5)- *0kr efAs#AQL_)` `  www s,Dcg}dtt|vr1t|dr%t|jr|j }t j |}|jddkDr#t|jdkDr td|jddz}||k\}t j|dd|ddzddz}|jddk(r|jdkrgSt j|dddz|vr2t j|t j|dddz}t j|dd}d} d} |D]} ||| } t| dkDr| dj|z } | dj|z }| | kr-|dk\xr||dz |k\xr ||dz |k\ }|r| |z } n| | z } |} |j| } |j!| }|j#|}|j|| |z| |zz||z| |zzfd | }|S) a Compute offsets for a given tokenized input Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. segment_size (`int`, *optional*, defaults to 1500): The number of features in the input mel spectrogram. torchcpurrz)Can only process a single input at a timerUNrV)textrk)r.r(hasattrcallablerqnparrayshapelen ValueErrorr[wheresumr^item_preprocess_token_ids_decode_filter_timestamp_ids)rBrarbrcoffsetsrdtimestamp_tokens consecutive last_slicerfrg current_slice sliced_tokensstart_timestamp_positionend_timestamp_positionis_single_endingrrs rH_compute_offsetsz%WhisperTokenizerFast._compute_offsetss c$y/* * 50IhW`WdWdNe! IHHY' ??1  !c)//&:Q&>HI I..r2Q6$7hh/47G7KKLQORSS   Q 1 $)9)=)=)?1)DI XX& ' *2 . 2+ E))K:J1KA1Nr1RUV1VWKXX./215 ( 'M%j?M=!A%+8+;+@+@+B_+T()6r):)?)?)AO)S&+.??'1Q(!*q.1_DuS]`aSaIbfuIu<$()\9))->>)$:!!% : := I ||M211$7 $4~EHY\jHjj2^CFWZhFhh&'J? 'BrQch|jtdDcgc] }d||zz c}Scc}w)a Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. iz<|%.2f|>)convert_tokens_to_idsrange)rBrbris rH timestamp_idsz"WhisperTokenizerFast.timestamp_idss5))X]^fXg*hSTJ!n:L,M*hii*hs/rYct|r5|jd}|jd}|j|||}|S)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|><|startoftranscript|>)r _strip_prompt)rBrarYprompt_token_iddecoder_start_token_ids rHr}z*WhisperTokenizerFast._preprocess_token_ids(sE "889JKO%)%?%?@W%X "**9oG]^IrQcDtj|jd|S)NrZ)r<subr>)rBras rHrz*WhisperTokenizerFast._filter_timestamp_ids;svvd(("i88rQclean_up_tokenization_spacesoutput_offsetsrbdecode_with_timestamps normalizebasic_normalizeremove_diacriticsc |j||} t|| f||||| d| } |r|j| ||} n|j | } |r|j ||} | | dS| S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Will remove the previous tokens (pre-prompt) if present. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. If there are previous tokens (pre-prompt) to decode, they will only appear in the decoded text if they contain timestamp tokens. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. rX)rYrrrr)rbrY)rb)rrr)r}r/r_rnrr)rBrarYrrrbrrrrrC filtered_idsrrrrGs rHr_zWhisperTokenizerFast.decode?sd11  32 w~  3)E+/    "//^Qd0D--d3D ++In+UG W5 5 rQ)rrrc~t||i|}|r|j|}|S|r|j||}|S|S)Nr)r/r~ _normalize_basic_normalize) rBrrrrPrCrr clean_textrGs rHr~zWhisperTokenizerFast._decodesSw// .J  ..tGX.YJ KrQcNtjd|j|S)NzThe private method `_normalize` is deprecated and will be removed in v5 of Transformers.You can normalize an input string using the Whisper English normalizer using the `normalize` method.)warningswarnr)rBrrs rHrzWhisperTokenizerFast._normalizes%  s ~~d##rQcRtjd|j||S)NzThe private method `_basic_normalize` is deprecated and will be removed in v5 of Transformers.You can normalize an input string using the Whisper basic normalizer using the `basic_normalize` method.r)rrr)rBrrrs rHrz%WhisperTokenizerFast._basic_normalizes-  w ##D*P'',, o_s22QbctQuu   + + 7osW= JJt??UYhmnquu  U|000   s 36CC r?r@rAc ||n |j|_||n |j|_||n |j|_|j}|j |}|j }|j }dj|Dcgc]}|d c}} tj| d|d| d|d||fgt|||j_ ycc}w)a Override the prefix tokens appended to the start of the label sequence. This method can be used standalone to update the prefix tokens as required when fine-tuning. Example: ```python >>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizerFast.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. N z:0z $A:0 z $A:0 $B:1 z:1)singlepairspecial_tokens) r?r@rA prefix_tokensconvert_ids_to_tokensr% eos_token_idr`r TemplateProcessingzipr4post_processor) rBr?r@rAprefix_token_idsprefixeseosrrjprefix_templates rHset_prefix_tokensz&WhisperTokenizerFast.set_prefix_tokenss(%-$8dmm  ,D$)) 8J8V"4\`\s\s----.>?nn(( ((h#GUugRL#GH0:0M0M%&fSE4#$KuB7l#X/01 -$Hs Cc |jd}|jd}|jd}|jd}ttj}|j|jj |_|jt vrt |j}n|jt jvr |j}nnt|jdk(}td|jd|rtt jntt jd|j1|jtvrtd |jd t|g}|j&|j|d z|jz|j"|j|jd k(r|n||js|j||S) Nrz <|translate|>z<|transcribe|>z<|notimestamps|>rVzUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r transcribe)rrrkeysr?lowerrvaluesrxrylistr@rr^indexrA) rB bos_token_idtranslate_token_idtranscribe_token_idnotimestamps_token_idlangs language_idis_language_code bos_sequences rHrz"WhisperTokenizerFast.prefix_tokenss112IJ !77H"889IJ $ : :;M Ninn&' == $ MM//1DM}} 00.t}}= "2"9"9";;"mm #&t}}#5#:  ,T]]O<;K-4467QUVfVkVkVmQnoopr 99 yy( #5dii[@UV^U_!`aa$~ == $    q 05;;{3K K L 99   tyyL7P 3Vh i&&    5 6rQc||j|z|jgzS|j|z|z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.)rr)rB token_ids_0 token_ids_1s rH build_inputs_with_special_tokensz5WhisperTokenizerFast.build_inputs_with_special_tokenssK  %% 3t7H7H6II I!!K/+=ARAR@SSSrQrralready_has_special_tokensc|rt|||dSdgt|jz}dg}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)r/get_special_tokens_maskrxr)rBrrr prefix_ones suffix_onesrGs rHrz,WhisperTokenizerFast.get_special_tokens_masks& &72'[]a3 cC 2 233 c  1#K(8"89KG GqcC $445!s;?O9OPS^^^rQc|j||| |jdd}t|Dcgc] \}}|dz|f}}}|Scc}}w)N)r@r?rAr)rrr\)rBr@r? no_timestamps forced_tokensrankrjforced_decoder_idss rHget_decoder_prompt_idsz+WhisperTokenizerFast.get_decoder_prompt_ids>sa D8TaPab **12. CL]C[\KD%tax/\\!!]sA c"t|||||S)N)return_timestampsreturn_languagerb)r)rB model_outputsrrrbs rHrz WhisperTokenizerFast._decode_asrHs  /+)   rQrrcdd|jzd}|ddd}tfd|Dd}| j|}td |d |j | |dS) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].rrF)add_special_tokensrrNc3HK|]}|jdk\s|yw)rN)r[).0xrBs rH z6WhisperTokenizerFast.get_prompt_ids..Xs$ \qqDDXDXYZD[?[ \s""zJEncountered text in the prompt corresponding to disallowed special token: r) tensor_type)stripnextrryconvert_to_tensors)rBrrreturn_tensorsbatch_encodingprompt_text_idsspecial_token_idrjs` rHget_prompt_idsz#WhisperTokenizerFast.get_prompt_idsRs/tzz|1CX]^)5ab9 \O \^bc  '../?@Eijoippqrs s))n)Ek**rQrarrct|ts|j|}|s|S|d|k(}|r||vr||j|dSgS|S)Nr)r-r_convert_to_listr)rBrarr has_prompts rHrz"WhisperTokenizerFast._strip_promptasc)T*--i8I q\_4 %2 1G!H!JKK rQcvt|dr\dtt|vr|jj }nMdtt|vr7|j }n&dtt|vr|j }t |tjr|j }|S)Nnumpyrp tensorflowjaxlib) rsr.r(rqrtolistr-rundarray)ras rHrz%WhisperTokenizerFast._convert_to_listss 9g &#d9o..%MMO113 T)_!55%OO- T)_- -!((*I i ,!((*IrQ) NNNN <|endoftext|>r r FNNF)F{Gz?)r r )r )F)FNFr FFFF)N)NNN)NF)NNT)ru)-rM __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrslow_tokenizer_classr0r rNrSr.rnrrrboolr}rr]r_r~rrr staticmethodrrrrrpropertyrintrrrrrrrr __classcell__)rGs@rHrr,s$L*$&67+!!!  =5@;];5}5W[( ( VBHjjD&9%*-1$ $', %"'L"L'+ L  L  L!%LLL L L^(-egl  $ ?C `d   $O    1c 1HSM 1]bcf]g 1$ #$ C$ \`$ LtCyDTQUVYQZTsx_9_3;DI3F_ko_ c_>"  +3 +tCy3`c$  rQr)%rr2rr<r functoolsrtypingrrrrru tokenizersrrr tokenization_utils_baser tokenization_utils_fastr utilsr english_normalizerrrtokenization_whisperrrrrr get_loggerrMloggerrrr,rQrHr#ss( ((==4>Jff   H %&( U 2U rQ