sg ' d Z ddlZddlmZ ddlmZmZmZ ddlm Z ddl
m Z ddl m
Z
mZ e
rd d
lmZ ndZ ej" e Zd d d
ZdZ G d de Zy)z'Tokenization classes for RemBERT model. N)copyfile)ListOptionalTuple )
AddedToken)PreTrainedTokenizerFast)is_sentencepiece_availablelogging )RemBertTokenizerzsentencepiece.modelztokenizer.json)
vocab_filetokenizer_fileu ▁c
e Zd ZdZeZeZ d fd Ze de
fd Z dde e
dee e
de e
fdZ dde e
dee e
de
de e
fd Z dde e
dee e
de e
fd
Zdd ed ee dee fd
Z xZS )RemBertTokenizerFasta
Construct a "fast" RemBert tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This
tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
do_lower_case (`bool`, *optional*, defaults to `True`):
Whether or not to lowercase the input when tokenizing.
remove_space (`bool`, *optional*, defaults to `True`):
Whether or not to strip the text when tokenizing (removing excess spaces before and after the string).
keep_accents (`bool`, *optional*, defaults to `False`):
Whether or not to keep accents when tokenizing.
bos_token (`str`, *optional*, defaults to `"[CLS]"`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `"[SEP]"`):
The end of sequence token. .. note:: When building a sequence using special tokens, this is not the token
that is used for the end of sequence. The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c
t | t rt | dd n| } t | |f|||||||| |
| | d |
|| _ || _ || _ || _ y )NTF)lstriprstrip) r
do_lower_case remove_space keep_accents bos_token eos_token unk_token sep_token pad_token cls_token
mask_token)
isinstancestrr super__init__r r r r )selfr r r r r r r r r r r r kwargs __class__s h/var/www/html/venv/lib/python3.12/site-packages/transformers/models/rembert/tokenization_rembert_fast.pyr" z RemBertTokenizerFast.__init__X s " KUU_adJeZ
4Fku
)'%% !
+ ( ($ returnc p | j r)t j j | j S dS NF)r ospathisfile)r# s r& can_save_slow_tokenizerz,RemBertTokenizerFast.can_save_slow_tokenizer s$ 26//rww~~doo.LuLr' token_ids_0 token_ids_1c f | j g}| j g}|||z |z S ||z |z |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A RemBERT sequence has the following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added
token_ids_1 (`List[int]`, *optional*, defaults to `None`):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_id cls_token_idr# r/ r0 sepclss r& build_inputs_with_special_tokensz5RemBertTokenizerFast.build_inputs_with_special_tokens sP & ! ! $s* *[ 3&4s::r' already_has_special_tokensc
|r9| t d |D cg c] }|| j | j fv rdnd" c}S |+dgdgt | z z dgz dgt | z z dgz S dgdgt | z z dgz S c c}w )a
Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of ids.
token_ids_1 (`List[int]`, *optional*, defaults to `None`):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Set to True if the token list is already formatted with special tokens for the model
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
zYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.r r )
ValueErrorr2 r3 len)r# r/ r0 r8 xs r& get_special_tokens_maskz,RemBertTokenizerFast.get_special_tokens_mask s & && R VaaPQt 0 0$2C2C DDA!Ka a "31#K 001QC7A3[AQ;QRVWUXX XsqcC ,,-33 bs %B c | j g}| j g}|t ||z |z dgz S t ||z |z dgz t ||z dgz z S )a
Creates a mask from the two sequences passed to be used in a sequence-pair classification task. A RemBERT
sequence pair mask has the following format:
```
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence | second sequence |
```
if token_ids_1 is None, only returns the first portion of the mask (0s).
Args:
token_ids_0 (`List[int]`):
List of ids.
token_ids_1 (`List[int]`, *optional*, defaults to `None`):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
r r )r2 r3 r; r4 s r& $create_token_type_ids_from_sequencesz9RemBertTokenizerFast.create_token_type_ids_from_sequences st . ! ! s[(3./1#5 53$s*+qc1C c8I4JaS4PPPr' save_directoryfilename_prefixc t j j | s%t j dj | y t j j
||r|dz ndt d z }t j j | j t j j | k7 rt | j | |fS )Nz*Vocabulary path ({}) should be a directory- r ) r+ r, isdirloggererrorformatjoinVOCAB_FILES_NAMESabspathr r )r# r@ rA out_vocab_files r& save_vocabularyz$RemBertTokenizerFast.save_vocabulary s ww}}^, LLELL^\ ] o_s 22QbcoQp p
77??4?? +rww~/N N T__n 5 r' ) NNTTF[CLS][SEP]zrO zrN z[MASK])Nr* )__name__
__module__ __qualname____doc__rJ vocab_files_namesr
slow_tokenizer_classr" propertyboolr. r intr r7 r= r? r r rM
__classcell__)r% s @r& r r & s2 ,\ *+ &%P M M M JN; 9;3;DI3F;
c;4 sx 4 9 43;DI3F 4ko 4
c 4@ JN Q 9 Q3;DI3F Q
c Q< !c !HSM !]bcf]g !r' r )rS r+ shutilr typingr r r tokenization_utilsr tokenization_utils_fastr utilsr
r tokenization_rembertr
get_loggerrP rF rJ SPIECE_UNDERLINEr r' r& rc se . ( ( , > 8 6 H %#8L\] !2 !r'