sgz
d Z ddlZddlZddlZddlmZmZmZmZ ddl m
Z
ddl m Z m
Z
ddlmZmZmZmZmZ ddlmZ dd lmZmZ eeeeef eeeeeef eeeeef eeeeeef f Z G d
d ed
Z G d ded
Z G d ded
Z G d de Z deeeeef dedeeef fdZ dededefdZ!d Z"d Z#d Z$d d Z%y) z Processor class for KOSMOS-2. N)ListOptionalTupleUnion ) BatchFeature)
ImageInput
is_batched) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInputc D e Zd ZU eee ed< ee ed< ee ed< y)Kosmos2ImagesKwargsbboxesnum_image_tokensfirst_image_token_idN)__name__
__module__ __qualname__r r float__annotations__int a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr r % s% T%[ !! sm#"3-'r r F)totalc e Zd ZU ee ed< y)Kosmos2TextKwargs
add_eos_tokenN)r r r r boolr r r r r# r# + s D>!r r# c
D e Zd ZU eed< eed< dddddddddd ddid Zy
) Kosmos2ProcessorKwargs text_kwargs
images_kwargsTFr ) add_special_tokenspaddingstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser$ r @ )r( r) N)r r r r# r r _defaultsr r r r' r' / sC ""&& #' ).*/&+%* "
Ir r' c d e Zd ZdZddgZdgZdZdZd fd Z d de d e
e e e f d
e
e d efd Zd
Zd Z d de
e e e f de dedee d e
ee e f f
dZd Zd Zd dZd Ze d Z d ede
e e e e e e f d efdZ de
e eef e e e e e f f d e eef fdZ! xZ"S ) Kosmos2Processora,
Constructs an KOSMOS-2 processor which wraps a KOSMOS-2 image processor and a KOSMOS-2 tokenizer into a single
processor.
[`Kosmos2Processor`] offers all the functionalities of [`CLIPImageProcessor`] and some functionalities of
[`XLMRobertaTokenizerFast`]. See the docstring of [`~Kosmos2Processor.__call__`] and [`~Kosmos2Processor.decode`]
for more information.
Args:
image_processor (`CLIPImageProcessor`):
An instance of [`CLIPImageProcessor`]. The image processor is a required input.
tokenizer (`XLMRobertaTokenizerFast`):
An instance of ['XLMRobertaTokenizerFast`]. The tokenizer is a required input.
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
image_processor tokenizernum_patch_index_tokensCLIPImageProcessor
AutoTokenizerc d|_ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d | _ d
| _ d | _
d | _ | j | j | j | j | j
| j | j | j | j | j | j g | _ || _
t | j D cg c] }d
t | j! d d" }}g }| j |z D ] }|j# t% |ddd " |j' | t( | U || y c c}w )NFzzzzzzz z z T)lstriprstrip
normalized)r0 eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensr8 rangestrzfillappendr
add_tokenssuper__init__)
selfr6 r7 r8 kwargsxpatch_index_tokens
tokens_to_addtoken __class__s
r rU zKosmos2Processor.__init__[ sM */ '!"##"#$#$9&
NN NN NN NN NN NN NN NN NN NN NN
'=#JOPTPkPkJl mQ
c!fll1o->a @ m m
__'99 aE E$uY^!_ ` a ]+
)4 ns 4%E1imagestextrW returnc
<
|
| t d | j t fd| j j i|}|d j dd }|d j dd }|d j dd } |d j d
d }
|d d } |d d
} |d j
dd }
t }|' | j |fi |d }|j | || j |||| }| rd|
sbt |t r| j j | }n7t |t r'|D cg c] }| j j | }}|d d xr |
|d d < || nd |d d
< ||
nd|d d< | j d d|i|d }|j | | |d d < | |d d
< |
|d d< | | | | j j dz } | }t! | dz }t t# | | |z }dgdg|z z dgz }g }g }|d }t |t r |g}|d g|d< |D ]p }|d| |z |||z d z }|j% | t' j& | }|rdg|z }|dgt) | t) | z
z z
}|j% | r t |t rt+ t- j. D cg c] \ }} |t) | f c} }d } | d \ } } | d \ }} |d d xr |
|d d < d|d d< | j d d|| gi|d }t) |j. d } | | k7 r5| j j0 dk( r|D cg c]+ } | | j j2 g| t) | z
z z - }} |D cg c] } | dg| t) | z
z z }} |d D cg c] } | dg| t) | z
z z c} |d< n| j j0 dk( r|D cg c]+ } | j j2 g| t) | z
z | z - }} |D cg c] } dg| t) | z
z | z }} |d D cg c] } dg| t) | z
z | z c} |d< t |t r|
|d }|d d |d< |d }|j t5 ||d |d|
|S c c}w c c} }w c c} w c c} w c c} w c c} w c c} w c c} w ) a
This method uses [`CLIPImageProcessor.__call__`] method to prepare image(s) for the model, and
[`XLMRobertaTokenizerFast.__call__`] to prepare text for the model.
Please refer to the docstring of the above two methods for more information.
The rest of this documentation shows the arguments specific to `Kosmos2Processor`.
Args:
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
Nz*You have to specify either images or text.tokenizer_init_kwargsr) r r r2 r r( r$ Fr* r+ return_tensors)r r^ r input_idsattention_maskc | d S Nr )rX s r z+Kosmos2Processor.__call__.. s
defhdi r )keyrh rightleft)rd re image_embeds_position_mask)data tensor_typer )
ValueError
_merge_kwargsr' r7 init_kwargspop
setdefaultr r6 updatepreprocess_examples
isinstancerP bos_tokenlist unk_token_idr rO rR copylensorted enumeraterd padding_side pad_token_idr )!rV r] r^ audiovideosrW
output_kwargsr r r r$ r* r+ rb encodingimage_encodings
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrd rm
all_input_idstext_idsmaskidxrX
sorted_length_min_len_not_paddedmax_len_paddeds! r __call__zKosmos2Processor.__call__ sY 8 >dl IJ J*** "
"&.."<"<
/33HdC( 9==>PRTU ,_ = A ABXZ^ _ %m488 %P
*= 9:N O
.y9&}5@@AQSWX > 1T 1 1& [M/D d+FJKt~~ 7 7 8 <KDK m,-ABT}
- ()= > BH gUZM- ( 3OU~^cgM- ()9 : *DNN U U
m8T UM OOM *=O
m$%9:29
m$Y/9G
m$%56 2#+'+~~'B'BQ'F$ *H h-!+K #5)=?SVf?f#g hO/0cQC:J4J.JaS.P + I)+ & $[ 1M$$!.
.67G.H-I)* )
8#L[1OCh{]mOmOoFpp *yy!@A 3:Ds8}s4y 899*11$7
8 $% &1:=;R;R1STvsAc3q6]TZi!
)6a(8%%&r*Q!-01EFX= m,-AB BF
m,-=> . `T#YK `=Q^C_ `
!$]%<% ? ?62 6 JRRbIc6DEA~A'> ? ?6!12 44>lu$vghdnn&A&A%BnWZ[\W]F]%^ab%b$v $vIc6DEQC>CF#: ;a ?62 6 JRRbIc6DEQC>CF#: ;a ?6!12
$$ )? %aL -56F-G-J)*-G-J*
OO %.*23C*D6P
!/
K Lj U %w66 %w66s0 !S5/S:
0T T, T
,0T" T Tc @ |yt |t s t d |D ]{ }|t |t s|g}|D ]^ }t |t rBt | dk( rt d |D r4t | dk( rt d |D rUt d } y)a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c 3 < K | ] }t |t y wN)rw r .0rX s r zAKosmos2Processor._check_bboxes_for_single_text..C s .Saz!S/A.S r> c 3 < K | ] }t |t y wr )rw r r s r r zAKosmos2Processor._check_bboxes_for_single_text..D s 1X1*Q2F1Xr a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)rw ry rp tupler| all)rV r bboxelements r _check_bboxes_for_single_textz.Kosmos2Processor._check_bboxes_for_single_text* s > FD) _` ` D| d+ v
!'51\Q&3.S7.S+SG )c1XPW1X.X$@
r c \ |j }|| d| }| j || }|S )N )strip_insert_patch_index_tokens)rV r^ imager img_info_tokenss r _preprocess_single_examplez+Kosmos2Processor._preprocess_single_exampleM s= zz| %&av.D ..tV< r textsr r c | j g|z }dj | j g|z | j gz }d}t |t rd}|g}|dgt | z }nt
| s|g}t | t | k7 r$t dt | dt | d |s| j | |g}nE|4t |t s t d |D ] }| j | ndgt | z }t | t | k7 r$t d t | dt | d t ||| D
cg c] \ } }
} | j | |
| | } }
} } |s| d
} | S c c} }
} w ) a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, List[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, List[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got r ) rD joinrE rw rP r| r
rp r ry zipr )
rV r r] r r
img_tokensr batchedrX r^ r r results
r rv z$Kosmos2Processor.preprocess_examplesW s , nn%(88
((DNN#3j#@DNNCS#ST eS !GGE >Vc%j(FF# XF u:V $ YZ]^cZdYeeklopvlwkx yB C
. .v 6 XF
fd+ !vww
62215
6 Vc%j(F v;#e* $ YZ]^cZdYeeklopvlwkx yB C
&)%?
!eT
+ +D% O
AYF
s Fc : | j j |i |S )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
refer to the docstring of this method for more information.
)r7 batch_decoderV argsrW s r r z Kosmos2Processor.batch_decode s
+t~~**D;F;;r c : | j j |i |S )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
the docstring of this method for more information.
)r7 decoder s r r zKosmos2Processor.decode s
%t~~$$d5f55r c \ |j | j d }|r t | S |S rg )splitrE +clean_text_and_extract_entities_with_bboxes)rV r^ cleanup_and_extractcaptions r post_process_generationz(Kosmos2Processor.post_process_generation s, **T^^,R0 >wG Gr c t | j |d }|D cg c] }| j |d c}S c c}w )a
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
Returns:
`List[str]`: The decoded text.
T)skip_special_tokensF)r )r r )rV generated_outputsgenerated_textsr^ s r post_process_image_text_to_textz0Kosmos2Processor.post_process_image_text_to_text s@ ++,=SW+XZijRV,,Tu,Mjjjs 5c | j j }| j j }t t j ||z S r )r7 model_input_namesr6 ry dictfromkeys)rV tokenizer_input_namesimage_processor_input_namess r r z"Kosmos2Processor.model_input_names sA !% @ @ &*&:&:&L&L#DMM"7:U"UVWWr c |t | dk( r|S t t j d| }t | t | k7 r$t dt | dt | d d}g }t || D ] \ }}|j
\ }} |j ||| | }|2t |t r|g}g }
t d |D s t d |D ], } | j | \ } }
|
j | d |
. t |
dk( rd
j |
}|j d | d |t | k r|j ||d d
j | }|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c 3 $ K | ] }|d u
y wr r )r boxs r r z>Kosmos2Processor._insert_patch_index_tokens.. s 73s$7s zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
r| ry refinditerrp r spanrR rw r r #_convert_bbox_to_patch_index_tokensr )rV r^ r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strs r r z+Kosmos2Processor._insert_patch_index_tokens s >S[A -K r{{+B4 PQ 3v; . H IL M\ I] H^ ^d eh io ep dq qz {
& 9 @MGT \\^FAs MM$x, -H|$& v"$ 7$77 j
O/3/W/WX[/\,
}#**m_Am_+MN
O &'1,=BBCVWL MMIl^:> ?/ @2 c$i MM$xy/ *wwv r r c t | dk( r|\ }}n7t t j | j }t || \ }}dt
| j d d}dt
| j d d}||fS )Nr r= r> r? )r| r mathsqrtr8 coordinate_to_patch_indexrP rQ )rV r idx_1idx_2num_patches_per_sidetoken_1token_2s r r z4Kosmos2Processor._convert_bbox_to_patch_index_tokens s t9> LE5 $'tyy1L1L'M#N 4T;OPLE5!#e*"2"21"5!6a8!#e*"2"21"5!6a8 r )i )NNNN)NNr2 )T)#r r r __doc__
attributes valid_kwargsimage_processor_classtokenizer_classrU r r r r r r' r r r r BboxInputr r rP rv r r r r propertyr r r r r
__classcell__)r\ s @r r5 r5 D s " $[1J,-L 0%O+5^ "26 `` ItI./` /0
`
`D!F " *, @YY/0@ @ @
#3- @
sDI~
@F<6
k X X
+s +E$uSzBRTXY^_dYeTfBf x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rc )rp r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxs r r r s RR GRopp ::b// 0D ::b// 0D 99R..2 3D 99R..2 3D
(
(4
/F
(
(4
/F 6>r r r c " d|z }| |z }| |z }||z }||z }| |k( r||z }||z } ||z |z }
||z |z } nQ||k( s||k( r||z }||z } ||z |z }
||z |z } n,||z |dz z }||z |dz z } ||z |dz z }
||z |dz z } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r ) r r r cell_sizer r r r r r r r s r patch_index_to_coordinater # s **I ( (D ) )D ( (D ) )D
I
I
I
)
I
)
I
I
I
)
I
)
I
A
-
I
A
-
I
A
-
I
A
-
r2r>r c
$ d}t j || }g }|D ]o }|j d }|j \ }}}|s*d}|j d d |j d d f}|j d } g }
| D ] } t j
d| } t j
d| dd }
| s5|
s8|rD|
j
t | j d t |
j d f ~|
j
t | j d t |
j d f |r|j
|||
f E|
D ]& }d|d d |d d
}|j
|||gf ( r |S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This functioin is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr r< zrc r= z>r N)r| r sub)r r^ entity_namestartr adjusted_start adjusted_endadjusted_entitys r adjust_entity_positionsr s_ & K %T&5\:;Nrvvgr4:67L"^\$BCO r c | j }t | t | j z
}g }|D ] \ }\ }}}t | t |j z
} t | t |j z
}
||z
| z }||z
|
z
}|j }|j |||f|f ||fS )z9Remove the spaces around the text and the entities in it.)r r| r@ rA rR ) r^ entitiesnew_textleading_spaces new_entitiesr r r r entity_name_leading_spacesentity_name_trailing_spacess r _cleanup_spacesr s zz|HYT[[]!33NL-5 A) \eS6%(%5K>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r r rc )r r r
r r rR r ) r^ r processed_textr r itemr r r r bboxes_in_coordss r r r s VVGR.N"Ed"K H+ ? aDG1&$?jpqbf 5d1gtAwH\ ]qq+;*==> ? >8 44 rs B) )&r r{ r r typingr r r r image_processing_utilsr image_utilsr r
processing_utilsr r r
r r tokenization_utilsr tokenization_utils_baser r r r r r r# r' r5 r r r
r r r r r r r* s0 $ / / 2 1 b b , ?
sCx ueUE) *+eCHo eE5%'( )*,
(,e ( "
% " -U *} ~ } @E%u*D$E ]` ejknpskset >(c (3 (c (Z7't"*5r