sg d Z ddlZddlmZmZmZmZmZ ddlm Z m
Z
ddl m Z m
Z
mZmZmZmZmZmZ ddlmZ ddlmZmZ d d
lmZmZm Z ej: e Z d d d
dZ G d de Z!y)z
Fast tokenization class for LayoutLMv3. It overwrites 2 methods of the slow tokenizer class, namely _batch_encode_plus
and _encode_plus, in which the Rust tokenizer is used.
N)DictListOptionalTupleUnion)pre_tokenizers
processors )
BatchEncoding EncodedInputPaddingStrategyPreTokenizedInput
TensorType TextInput
TextInputPairTruncationStrategy)PreTrainedTokenizerFast)add_end_docstringslogging )"LAYOUTLMV3_ENCODE_KWARGS_DOCSTRING2LAYOUTLMV3_ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGLayoutLMv3Tokenizerz
vocab.jsonz
merges.txtztokenizer.json)
vocab_file merges_filetokenizer_filec * e Zd ZdZeZddgZeZdddddddddd d
d d g d g d g d d
d f fd Z e
e e d9de
eeee ee f dee
eee f de
eee eeee f dee
ee eee f dede
eeef de
eeef dee dedee dee d ee
eef d ee d ee d ed ed!ed"ed#ed$ef(d% Z e
e e d9d&e
ee ee ee f d'edeeeee dee
ee eee f dede
eeef de
eeef dee dedee dee d ee
eef d ee d ee d ed ed!ed"ed#ed$ef(d( Zd:ded)ee ded$ee fd*Z e
e e d9de
eef dee deeee deee dede
eeef de
eeef dee dedee dee d ee
eef d ee d ee d ed ed!ed"ed#ed$ef(d+ Z dddd ej< ej> dddddddddddd fd&e
ee ee ee f d'edeeeee deeee ded,ed-edee dedee dee d ee d ee d ee d ed ed!ed"ed#ed$ef(d.Z dddd ej< ej> dddddddddddd fde
eef dee deeee deee ded,ed-edee dedee dee d ee d ee d ee d ed ed!ed"ed#ed$ef(d/Z!dej< dddfd0e
e"ee#f ef dee d,edee dee d ee d$e$fd1Z%d;d2ed3ee d$e&e fd4Z'd;d5Z( d;d6ee d7eee d$ee fd8Z) xZ*S )<LayoutLMv3TokenizerFasta<
Construct a "fast" LayoutLMv3 tokenizer (backed by HuggingFace's *tokenizers* library). Based on BPE.
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
errors (`str`, *optional*, defaults to `"replace"`):
Paradigm to follow when decoding bytes to UTF-8. See
[bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
trim_offsets (`bool`, *optional*, defaults to `True`):
Whether the post processing step should trim offsets to avoid including whitespaces.
cls_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
The bounding box to use for the special [CLS] token.
sep_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
The bounding box to use for the special [SEP] token.
pad_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
The bounding box to use for the special [PAD] token.
pad_token_label (`int`, *optional*, defaults to -100):
The label to use for padding tokens. Defaults to -100, which is the `ignore_index` of PyTorch's
CrossEntropyLoss.
only_label_first_subword (`bool`, *optional*, defaults to `True`):
Whether or not to only label the first subword, in case word labels are provided.
input_idsattention_maskNreplacezzzzzT)r r r r ic t | ||fi d|d|d|d|d|d|d| d|
d | d
| d |
d |d
|d|d|d|| t j | j j
j
}|j d
| | k7 r;t t |j d }| |d
< |di || j _ | | _ d}t | j |d }|rt j |j
}d|v rt |d |d< d|v rt |d |d< d}|j d
| | k7 r| |d
< d}|j d |
|
k7 r|
|d < d}|r>t t |j d } |di |}t | j || || _ || _ || _ || _ || _ y )Nr errors bos_token eos_token sep_token cls_token unk_token pad_token
mask_tokenadd_prefix_space trim_offsets
cls_token_box
sep_token_box
pad_token_boxpad_token_labelonly_label_first_subwordtypepost_processorsepclsFT )super__init__jsonloadsbackend_tokenizer
pre_tokenizer __getstate__getgetattrr popr+ tupler setattrr- r. r/ r0 r1 ) selfr r r r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 kwargs
pre_tok_state
pre_tok_classtokenizer_component tokenizer_component_instancestatechanges_to_applycomponent_class new_value __class__s n/var/www/html/venv/lib/python3.12/site-packages/transformers/models/layoutlmv3/tokenization_layoutlmv3_fast.pyr8 z LayoutLMv3TokenizerFast.__init__y sV ,
*
"
.
&
(
(
(!
" ,#
$ &>'
,
4#9#9#G#G#T#T#VW
/1A BFV V #NM4E4Ef4M NM0@M, -3@3Q=3QD " " 0 0 .'.t/E/EGZ\`'a$ 'JJ ; H H JKE ~ $U5\ 2e ~ $U5\ 2e $ yy+-=>BRR,<()#' yy 6,F(4n%#' ")*eii6G"H + 4e 4 ..0CYO + * * . (@% Fr text text_pairboxes word_labelsadd_special_tokenspadding
truncation
max_lengthstridepad_to_multiple_of padding_sidereturn_tensorsreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping
return_lengthverbosereturnc h d }|4 || s t d t |t t f s,t d t |t t f s t d |t |t t f }n5t |t t f xr |xr t |d t t f }||n|}| t d |rYt | t | k7 r t d t || D ]' \ }}t | t | k7 s t d n"t | t | k7 r t d |r|;t | t | k7 r$t d t | d
t | d |t t || n|}t
|du } | j d!i d |d
|d|d|d|d|d|d|d| d|
d| d| d|
d|d|d|d |d |d ||S | j d!i d |d |d|d|d|d|d|d|d| d|
d| d| d|
d|d|d|d |d |d ||S )"a3
Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
sequences with word-level normalized bounding boxes and optional labels.
Args:
text (`str`, `List[str]`, `List[List[str]]`):
The sequence or batch of sequences to be encoded. Each sequence can be a string, a list of strings
(words of a single example or questions of a batch of examples) or a list of list of strings (batch of
words).
text_pair (`List[str]`, `List[List[str]]`):
The sequence or batch of sequences to be encoded. Each sequence should be a list of strings
(pretokenized string).
boxes (`List[List[int]]`, `List[List[List[int]]]`):
Word-level bounding boxes. Each bounding box should be normalized to be on a 0-1000 scale.
word_labels (`List[int]`, `List[List[int]]`, *optional*):
Word-level integer labels (for token classification tasks such as FUNSD, CORD).
c t | t ryt | t t f rft | dk( ryt | d t ryt | d t t f r)t | d dk( xs t | d d t S yy)NTr F)
isinstancestrlistrA len)ts rN _is_valid_text_inputz>LayoutLMv3TokenizerFast.__call__.._is_valid_text_input sz !S!Ae}-q6Q; !c* !tUm4 qt9>EZ!Q-EE rO NzStext input must of type `str` (single example) or `List[str]` (batch of examples). zwWords must be of type `List[str]` (single pretokenized example), or `List[List[str]]` (batch of pretokenized examples).r z-You must provide corresponding bounding boxesz@You must provide words and boxes for an equal amount of examplesz:You must provide as many words as there are bounding boxeszbatch length of `text`: z- does not match batch length of `text_pair`: .batch_text_or_text_pairsis_pairrR rS rT rU rV rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rP rQ r6 )
ValueErrorrf rh rA ri zipboolbatch_encode_plus encode_plus) rC rP rQ rR rS rT rU rV rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rD rk
is_batchedwords
words_example
boxes_examplerm rn s rN __call__z LayoutLMv3TokenizerFast.__call__ sa X ( '- !vww i$ 7 M dT5M2 M
#D4-8J#D4-8hThjQUVWQX[_afZgFhJ!)y = LM M 5zSZ' !cdd03E50A
c,
}}%]);;$%abb
c 5zSZ' !]^^ $Tc)n)D .s4yk :I'q* FOEZtCi,@'A`d $9D01G)4)) )A (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
. $4## # (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
rO rm rn c | j d||||
|d|\ }}}} | j di d|d|d|d|d|d|d|d |d
| d |
d | d
| d|
d|d|d|d|d|d||S )NrU rV rW rY rb rm rn rR rS rT padding_strategytruncation_strategyrW rX rY rZ r[ r\ r] r^ r_ r` ra rb r6 )"_get_padding_truncation_strategies_batch_encode_plus)rC rm rn rR rS rT rU rV rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rD r{ r| s rN rr z)LayoutLMv3TokenizerFast.batch_encode_plusg s < ElDDkDk E
!! 1 E
E
A-z6 't&&
%=
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
rO pairc t |r||fgn|g} | j j |f|dd|}|d j S )NFrT is_pretokenizedr )
_tokenizer encode_batchtokens)rC rP r rT rD
batched_input encodingss rN tokenizez LayoutLMv3TokenizerFast.tokenize sQ *.$ TF
0DOO00
.@RW
[a
|"""rO c | j d||||
|d|\ }}}} | j di d|d|d|d|d|d|d|d |d
| d |
d | d
| d|
d|d|d|d|d|d||S )aj
Tokenize and prepare for the model a sequence or a pair of sequences. .. warning:: This method is deprecated,
`__call__` should be used instead.
Args:
text (`str`, `List[str]`, `List[List[str]]`):
The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
text_pair (`List[str]` or `List[int]`, *optional*):
Optional second sequence to be encoded. This can be a list of strings (words of a single example) or a
list of list of strings (words of a batch of examples).
rz rP rR rQ rS rT r{ r| rW rX rY rZ r[ r\ r] r^ r_ r` ra rb r6 )r} _encode_plus)rC rP rQ rR rS rT rU rV rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rD r{ r| s rN rs z#LayoutLMv3TokenizerFast.encode_plus s L ElDDkDk E
!! 1 E
E
A-z6 !t
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
rO r{ r| c
t |t st dt | d | j |||| |
| |r$|D cg c] \ }}|j |f }}}| j j ||d }|D cg c] }| j ||
||||dn||| ! }}i }|d d j D ]' }|D cg c] \ }} || D ] } | } } }} | ||< ) |D cg c] \ } }|D ] } | } }} } |r2g } t | D ] \ }!\ }"} | |!gt |"d z z
} | |d < |d D ] }#| j |#|| g }$t t |d D ]! }%|r |d |% }&n|%}&g }'t |d |% | |% j | |% j D ] \ }(})}*|*;|r!|)dk( r |'j# | j$ ,|'j# ||& |* D|(| j& k( r |'j# | j( o|(| j* k( r |'j# | j, |(| j. k( r |'j# | j$ t1 d
|$j# |' $ |$|d < |g }+t t |d D ] }%|r |d |% }&n|%}&g },d }-t |d |% |d
|% | |% j D ] \ }(}.}*|*l| j2 rH|.d dk( r|-s|,j# ||& |* n|,j# | j4 |.dk( rd}-Zd }-]|,j# ||& |* u|,j# | j4 |+j# |, |+|d< |s|d
= t7 || | S c c}}w c c}w c c} } }w c c} }} w )Nz/batch_text_or_text_pairs has to be a list (got ))r{ r| rW rX rY rZ Tr )encodingr\ r] r^ r_ r` ra rb r r overflow_to_sample_mappingzId not recognizedbboxFoffset_mapping)r r labels) tensor_type) rf rh TypeErrorr2 set_truncation_and_paddingsplitr r _convert_encodingkeys enumerateri &_eventual_warn_about_too_long_sequencerangerp sequence_idsword_idsappendr/ cls_token_idr- sep_token_idr. pad_token_idro r1 r0 r )/rC rm rn rR rS rT r{ r| rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rP rQ r r tokens_and_encodingssanitized_tokenskeyitem_estacksanitized_encodingsr itoksr token_boxes batch_indexoriginal_indextoken_boxes_exampleid sequence_idword_idr labels_exampleprevious_token_emptyoffsets/ rN r~ z*LayoutLMv3TokenizerFast._batch_encode_plus s 4 2D9 MdSkNlMmmn op p
'' - 3! 1%
(
Yq'rodI y(A'r $'rOO00 $ 1 1
2 &
" "!&;&;*C+E * (,+ +
#
, '*1-224 *C&:NN74DINqQNQNEN$) S ! * 1E S SWQd Sq Sq S S %)+ & )*> ?
K 9D!*qcC[8I4J.JJ*
K=W 9 :)+6 XI 7 7 :w W X %5k%B!CD 4K(!12N!OP[!\!,"$ ,/ -k:#K0==#K099-
>(K
& ;!#3+2243E3EF+225 3H3QRT . ..+2243E3EFt 0 00+2243E3EFt 0 00+2243E3EF ()< ==%
>&
2 33 46 $/ "F $S)9+)F%G H
. ,%56R%ST_%`N%0N!#',$+.$[1+>$%56{C' 4==, D'B
* 88 %ayA~6J . 5 5k.6QRY6Z [ . 5 5d6J6J K % /7; 47< 4 * 1 1+n2Mg2V W&--d.B.BC'D(
n-7
.: *0 X &)$%56 -/BP^__s (s
0 O Ss N4$N:N?
6Oc |r||fgn|g}|g}||gnd } | j |fi dt |d u d|d|d|d|d|d|d| d |
d
| d | d |
d
|d|d|d|d|d||}| `|s^t |j D ci c]. \ }}|t | dkD rt |d t r|d n|0 c}}|j }| j |d || |S c c}}w )Nrn rR rS rT r{ r| rW rX rY rZ r[ r\ r] r^ r_ r` ra rb r r ) r~ rq r itemsri rf rh r r )rC rP rQ rR rS rT r{ r| rW rX rY rZ r[ r\ r] r^ r_ r` ra rb rD r
batched_boxesbatched_word_labelsbatched_outputr values rN r z$LayoutLMv3TokenizerFast._encode_plus s 6 09$ *+tf
/:/F{mD000
$./
,
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
2 !*C * '5&:&:&<"U SZ!^
58T8RqX]] (( N
33N;4OQ[]de s 3C#
encoded_inputsc . |d| j v }|| j d }|t j k( r t | }||||z dk7 r ||z dz |z }|t j k7 xr t | |k7 }|rd|vrdgt | z |d< |r|t | z
} ||n | j
}|dk( r|r|d dg| z z |d< d|v r|d | j g| z z |d< d|v r|d | j g| z z |d< d|v r|d | j g| z z |d< d|v r|d dg| z z |d< || j g| z z || j d < |S |d k( r|rdg| z |d z |d< d|v r| j g| z |d z |d< d|v r| j g| z |d z |d< d|v r| j g| z |d z |d< d|v rdg| z |d z |d< | j g| z |z || j d < |S t d
t | z |S ) a#
Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
Args:
encoded_inputs:
Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
max_length: maximum length of the returned list and optionally padding length (see below).
Will truncate by taking into account the special tokens.
padding_strategy: PaddingStrategy to use for padding.
- PaddingStrategy.LONGEST Pad to the longest sequence in the batch
- PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
- PaddingStrategy.DO_NOT_PAD: Do not pad
The tokenizer padding sides are defined in self.padding_side:
- 'left': pads on the left of the sequences
- 'right': pads on the right of the sequences
pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
`>= 7.5` (Volta).
padding_side:
The side on which the model should have padding applied. Should be selected between ['right', 'left'].
Default value is picked from the class attribute of the same name.
return_attention_mask:
(optional) Set to False to avoid returning attention mask (default: set to model specifics)
r r r righttoken_type_idsr r special_tokens_maskleftzInvalid padding strategy:) model_input_namesr
LONGESTri
DO_NOT_PADrZ pad_token_type_idr/ r0 r ro rg )
rC r rW r{ rY rZ r] required_inputneeds_to_be_padded
differences
rN _padz LayoutLMv3TokenizerFast._pad s H ! ($48N8N$N !'(>(>q(AB 6 6 6 ^,J !&8&D*WiJimnJn%);;q@DVVJ - 1K1K K qPSTbPcgqPq !%5^%K01sS 5H/HN+ , #c.&99J+7+C<IZIZLw&(7EFV7W[\Z]`jZj7jN#34#~5&'78DC**PU|rO c | j g|z | j gz }||S || j gz |z | j gz S N) bos_token_id eos_token_id)rC token_ids_0 token_ids_1outputs rN build_inputs_with_special_tokensz8LayoutLMv3TokenizerFast.build_inputs_with_special_tokens; sU ##${2d6G6G5HH M**++k9T=N=NDEKKrO )NNNTFNNr NNNNNFFFFT)NFr )+__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr r slow_tokenizer_classr8 r r r r r r r r intrq rg r
r r r rx r rr r rs r DO_NOT_TRUNCATEr~ r r r dictr r r r r
__classcell__)rM s @rN r r 1 s AF *$&67. """ !%'SAj : ?@ U !
U tS/12U $%778U SMU U %SMU tnU !sJ!78U (~ U (~ U $(!U" %)#U$ !%%U& 'U( )U,
-U pUn : ?@:
!:
tS/12:
$%778:
SM:
:
%SM :
tn!:
" !sJ!78#:
$ (~%:
& (~':
( $():
* %)+:
, !%-:
. /:
0 1:
4
5:
p:
z#S #
#RV #mqrumv # :r sl
5 5 1 ? 0
H %#/ `pq d L 5 d LrO