sg" d Z ddlZddlmZ ddlmZmZmZmZm Z ddl
Z ddl m
Z
ddlmZ dd lmZ ej$ e Zd
d iZ G d d
e
Zy)z Tokenization class for SpeechT5. N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)logging )EnglishNumberNormalizer
vocab_filezspm_char.modelc
J e Zd ZdZeZddgZ d deee e
f ddf fd
Z d dZ e
d Ze
d
Zej d Zd Zd
Zd Zde dee fdZd Zd Zd Zd dee fdZ d dee deee dedee f fd
Z d de dee de e fdZ xZ S ) SpeechT5Tokenizera
Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The begin of sequence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
normalize (`bool`, *optional*, defaults to `False`):
Whether to convert numeric quantities in the text to their spelt-out english counterparts.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc
|i n|| _ || _ || _ d | _ t j
di | j | _ | j j | t | $ d|||||| j d| y )N) bos_token eos_token unk_token pad_token normalizer )
r r r _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__)
selfr r r r r r r kwargs __class__s
e/var/www/html/venv/lib/python3.12/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr" zSpeechT5Tokenizer.__init__O s &5%|j | d}R || j j | z
}|j S )z:Converts a sequence of tokens (string) in a single string. Fr) T)all_special_tokensr decodeappendstrip)r# tokenscurrent_sub_tokens
out_stringprev_is_specialrO s r& convert_tokens_to_stringz*SpeechT5Tokenizer.convert_tokens_to_string s
(E///& #%Jdmm 2 23E F NN
"&%'""))%0"'
( dmm**+=>>
!!r' c L ||| j gz S ||z | j gz S )z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r# token_ids_0 token_ids_1s r& build_inputs_with_special_tokensz2SpeechT5Tokenizer.build_inputs_with_special_tokens s5 $"3"3!44 4[(D,=,=+>>>r' rb rc already_has_special_tokensc |rt | ||d S dg}|dgt | z |z S dgt | z dgt | z z |z S )NT)rb rc re r r )r! get_special_tokens_masklen)r# rb rc re suffix_onesr% s r& rg z)SpeechT5Tokenizer.get_special_tokens_mask su &72 '[]a 3
c C#k**k9 9c+&&A3[1A+AB[PPr' save_directoryfilename_prefixc t j j | st j d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 rBt j j | j rt | j | |fS t j j | j sCt |d 5 }| j j }|j | d d d |fS |fS # 1 sw Y |fS xY w)NzVocabulary path (z) should be a directory-rV r wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfiler openr serialized_model_protowrite)r# rj rk out_vocab_fileficontent_spiece_models r& save_vocabularyz!SpeechT5Tokenizer.save_vocabulary s% ww}}^, LL,^,<zzzFN)Fr0 )NF) __name__
__module__ __qualname____doc__ru vocab_files_namesmodel_input_namesr r rK r r" r. propertyr3 r+ setterr> rC rG r rL rP rT r_ intrd boolrg r r~
__classcell__)r% s @r& r r ! sM (T *$&67
48
"$sCx.1
: . .
! !
,8c 8d3i 80 "&?QUVYQZ ? sx Q 9 Q3;DI3F Qko Q
c Q!c !HSM !]bcf]g !r' r )r ro shutilr typingr r r r r
sentencepiecer tokenization_utilsr
utilsr number_normalizerr
get_loggerr rr ru r r r' r& r sQ ' 3 3 5 6
H %!#34 y!+ y!r'