sg d Z ddlZddlmZ ddlmZmZmZ ddlm Z ddl
m Z m Z e rdd l
mZ ndZ e j e Zd
d d Z G d
de Zy)z Tokenization classes for XGLM. N)copyfile)ListOptionalTuple )PreTrainedTokenizerFast)is_sentencepiece_availablelogging )
XGLMTokenizerzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZdZeZddgZeZ d fd Z e
de fd Z dde
e dee
e de
e fd Z dde
e dee
e de
e fd
Zdd ed ee dee fd
Z xZS )XGLMTokenizerFasta{
Construct a "fast" XGLM tokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`RobertaTokenizer`]
and [`XLNetTokenizer`]. Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
additional_special_tokens (`List[str]`, *optional*, defaults to `["NOTUSED", "NOTUSED"]`):
Additional special tokens used by the tokenizer.
input_idsattention_maskc * d| _ t | j D
cg c] }
d|
d
} }
| j dg xs g | d< | dxx | D cg c] } | | d vs | c} z
cc< t
| |f|||||||d| || _ y c c}
w c c} w )N z additional_special_tokens)r bos_token eos_token sep_token cls_token unk_token pad_token)num_madeup_wordsrangegetsuper__init__r
)selfr
r r r r r r r kwargsi madeup_wordsword __class__s b/var/www/html/venv/lib/python3.12/site-packages/transformers/models/xglm/tokenization_xglm_fast.pyr! zXGLMTokenizerFast.__init__X s !" 49$:O:O4PQq+aS*Q Q.4jj9TVX.Y._]_*+*+ )0
T@[9\-\D0
+
)
%' R0
s
B B
B
returnc p | j r)t j j | j S dS )NF)r
ospathisfile)r" s r( can_save_slow_tokenizerz)XGLMTokenizerFast.can_save_slow_tokenizer{ s$ 26//rww~~doo.LuL token_ids_0 token_ids_1c \ || j g|z S | j g}||z |z |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An XLM-RoBERTa sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr" r0 r1 seps r( build_inputs_with_special_tokensz2XGLMTokenizerFast.build_inputs_with_special_tokens sF ( %%&4 4 ![ 3&,{::r/ c z | j g}|t ||z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )r3 lenr4 s r( $create_token_type_ids_from_sequencesz6XGLMTokenizerFast.create_token_type_ids_from_sequences sU $ ! s[()QC/ /3$s*S0;>?1#EEr/ save_directoryfilename_prefixc | j s t d t j j | st
j
d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 rt | j | |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory.- r
) r.
ValueErrorr+ r, isdirloggererrorjoinVOCAB_FILES_NAMESabspathr
r )r" r: r; out_vocab_files r( save_vocabularyz!XGLMTokenizerFast.save_vocabulary s ++
ww}}^, LL,^,<rI rH zz)N)__name__
__module__ __qualname____doc__rD vocab_files_namesmodel_input_namesr slow_tokenizer_classr! propertyboolr. r intr r6 r9 strr rG
__classcell__)r' s @r( r r $ s -^ *$&67( !%F M M M JN; 9;3;DI3F;
c;4 JNF 9F3;DI3FF
cF0!c !HSM !]bcf]g !r/ r )rM r+ shutilr typingr r r tokenization_utils_fastr utilsr r
tokenization_xglmr
get_loggerrJ rA rD r r/ r( r] s[ % ( ( > 8 0M
H %#