sg$dZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZmZerddlZerddlZej(eZddiZd ZGd d e Zy) zTokenization class for VITS.N)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsoncZtjd}|j|}|du}|S)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romans ]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr%s3 ?3  $ $\ 2E%M c eZdZdZeZddgZ d dfd ZedZ dZ d Z d Z dd e d ed eedee ee efffdZd e dee fdZdee de fdZdZdZdde dee deee dffdZxZS) VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNreturnc dt|d5} tj| |_ddd|jj D cic]\} } | |  c} } |_||_||_||_||_ ||_ t |0d|||||||d| y#1swYwxYwcc} } w)Nutf-8encoding) pad_token unk_tokenlanguage add_blank normalize phonemize is_uroman) openjsonloadencoderitemsdecoderr%r&r'r(r)super__init__)selfrr#r$r%r&r'r(r)kwargs vocab_handlekv __class__s rr2zVitsTokenizer.__init__Hs*w / 3<99\2DL 3*.););)=>A1>   """"     3 3?sB  B, B)c,t|jSN)lenr.)r3s r vocab_sizezVitsTokenizer.vocab_sizejs4<<  rct|jDcic]}|j||}}|j|j|Scc}wr:)ranger<convert_ids_to_tokensupdateadded_tokens_encoder)r3ivocabs r get_vocabzVitsTokenizer.get_vocabnsK;@;QRa++A.1RR T../ SsAcxt|jjt|jjz}d}d}|t |kr`d}|D]-}|||t |z|k(s||z }|t |z }d}n|s|||j z }|dz }|t |kr`|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr.keysrAr;lower)r3rall_vocabulary filtered_textrB found_matchwords rnormalize_textzVitsTokenizer.normalize_textssdll//12T$:S:S:X:X:Z5[[ #l##K& AD M2d:!T)MTNA"&K  a!6!6!88 Q#l##rcH|jdk(r|jdd}|S)z4Special treatment of characters in certain languagesronuțuţ)r%replace)r3texts r_preprocess_charzVitsTokenizer._preprocess_chars# ==E !<<d+D rrSis_split_into_wordsr'c H||n j}|rj|}j|}t|rQjrEt st jdn%tj}|j|}jrKts tdtj|ddddd}tj dd|}||fS|r5d j#t%t'fd |j)}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. aCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r%backendstrippreserve_punctuation with_stressz\s+ rFc |jvSr:)r.)charr3s rz8VitsTokenizer.prepare_for_tokenization..sTT\\=Qr)r'rOrTrr)r loggerwarningurUromanromanize_stringr(r ImportError phonemizerrsubjoinrHfilterrY)r3rSrUr'r4rLuromans` rprepare_for_tokenizationz&VitsTokenizer.prepare_for_tokenizations 4"+!6IDNN &&t,D--d3 #M 2t~~&(y & 6 6} E >>*,!"ijj&00  %) MFF63 >M f$$ GGD0QS`)a$bciikMf$$rct|}|jr.|jdgt|dzdzz}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrGN)rHr&_convert_id_to_tokenr;)r3rStokens intersperseds r _tokenizezVitsTokenizer._tokenizesQd >> 55a89S[1_q=PQL!'LA !F rroch|jrt|dkDr|ddd}dj|S)NrGrmrF)r&r;rh)r3ros rconvert_tokens_to_stringz&VitsTokenizer.convert_tokens_to_strings0 >>c&kAoADqD\Fwwvrc|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)r.getr$)r3tokens r_convert_token_to_idz"VitsTokenizer._convert_token_to_ids,||t||'7'7'GHHrc8|jj|S)z=Converts an index (integer) in a token (str) using the vocab.)r0ru)r3indexs rrnz"VitsTokenizer._convert_id_to_tokens||&&rsave_directoryfilename_prefixc tjj|stj d|dytjj ||r|dzndt dz}t|dd5}|jtj|jd d d d zddd|fS#1swY|fSxYw)NzVocabulary path (z) should be a directory-rFrwr r!rmTF)indent sort_keys ensure_ascii ) ospathisdirr`errorrhVOCAB_FILES_NAMESr+writer,dumpsr.)r3rzr{rfs rsave_vocabularyzVitsTokenizer.save_vocabularysww}}^, LL,^,<zNTTTF)rN)FNr:)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr2propertyr<rDrOrTstrboolrrrrrkrrqrsrwrnrr __classcell__)r8s@rrr/s**$&67      D!! *Y]?%?%.2?%GOPT~?% sDcN" #?%B c d3i tCyS I' c HSM ]bchilcmoscs]t rr)rr,rrtypingrrrrrrtokenization_utilsr utilsr r r rfrjrb get_loggerrr`rrrr*rrrsg# ::5JJ   H %!<0D'Dr