sgĂLdZddlZddlmZmZmZmZddlZddl m Z m Z m Z ddl mZmZmZddlmZmZmZmZmZmZmZmZmZmZddlmZmZmZm Z m!Z!m"Z"m#Z#erddl$Z$e"jJe&Z'd eeeeeeefd eeefd Z(Gd d e Z)Gdde Z*y)zImage processor class for Fuyu.N)DictListOptionalUnion)BaseImageProcessor BatchFeature get_size_dict)padresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imageis_valid_imagemake_list_of_imagesto_numpy_arrayvalidate_preprocess_arguments) TensorTypefilter_out_non_signature_kwargsis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsimagesreturnct|r|ggSt|trtd|Dr|St|tr|Dcgc] }t |c}St dcc}w)Nc3<K|]}t|tywN) isinstancelist).0images a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/fuyu/image_processing_fuyu.py z.make_list_of_list_of_images..@s'TE 5$(?'TszHimages must be a list of list of images or a list of images or an image.)rr$r%allr ValueError)rr's r(make_list_of_list_of_imagesr,:sefz&$C'TV'T$T &$8>?u#E*?? _ ``@sA(c6eZdZdZddeeeeffdZddZ y)FuyuBatchFeaturez BatchFeature class for Fuyu image processor and processor. The outputs dictionary from the processors contains a mix of tensors and lists of tensors. N tensor_typec  ||S|j|\fd fd}|jD]\ }t|tr=t|dtr*|Dcgc]}|Dcgc] }|| c}c}}| <St|tr|Dcgc] }|| c}| <}||| <|Scc}wcc}}wcc}w)a5 Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. )r/c(|r|S|Sr#)elem as_tensor is_tensors r(_convert_tensorz._convert_tensor^s T? "cV |S#dk(r tdtdxYw)Noverflowing_valueszKUnable to create tensor returning overflowing values of different lengths. zUnable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.)r+)r3r6keys r(_safe_convert_tensorzAFuyuBatchFeature.convert_to_tensors.._safe_convert_tensorcsB &t,, ..$%rss Xs (r)_get_is_as_tensor_fnsitemsr$r%) selfr/r;valueelemsr3r6r4r5r:s @@@@r(convert_to_tensorsz#FuyuBatchFeature.convert_to_tensorsPs  K#99k9R 9 # **, 8JC%&:eAh+EY^_PUUKT248K_S E4(DIJD1$7JS 17S  8 L_Ks( C 1CC !CC c  t|dgddl i}jd etdkDrWd}t |rnFt |t st|st |tr| ntdt |d  fd}|jD]\}}t |trGt |dtr4g}|D]'} |j| D cgc] } ||  c} )|||<]t |tr|D cgc] } ||  c} ||<||||<||_ |Scc} wcc} w)a  Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in different `dtypes` and sending the `BatchFeature` to a different `device`. Args: args (`Tuple`): Will be passed to the `to(...)` function of the tensors. kwargs (`Dict`, *optional*): Will be passed to the `to(...)` function of the tensors. Returns: [`BatchFeature`]: The same instance after modification. torchrNdevicez*Attempting to cast a BatchFeature to type z. This is not supported.cxj|r|jiS|jS|S)N)rD)is_floating_pointto)r3argsrDkwargsrCs r(_toz FuyuBatchFeature.to.._tosE&u&&t,tww///!wwfw--Kr7)rrCgetlenrr$strrintr+r=r%appenddata) r>rHrInew_dataargrJkvnew_vr@r3rDrCs `` @@r(rGzFuyuBatchFeature.to{s? $ *H% >c$i!mq'Cc"C%)=CQTAU!#McRUhZWo!pqq JJL %DAq!T"z!A$'=@ELL!>#d)!>?@# At$567Ts4y7 !!f  %  "?8s 'E E r#)r r ) __name__ __module__ __qualname____doc__rrrMrrArGr2r7r(r.r.Is' )huS*_7M.N)V8r7r.c!peZdZdZgdZddej dddddddddf d ed ee e e fd ed ed e de dede e ee fde e ee fdede dee e e fffd Zej ddfdej"d e e e fd edee e efdee e efdej"f dZ d-dej"d e e e fde de dee e efdee e efdej"fdZeddddddddddddej,ddfd eed ee e e fd eed eed ee dee deedee dee deedee dee e e fdee e efdee e efdeefdZd.d e d!e de e e fde fd"Zd.dd#dee e e fdd#fd$Z d.d%d#d&d#d'd#d(d#d)e d*e d+edee e e fdefd,ZxZS)/FuyuImageProcessora This class should handle the image processing part before the main FuyuForCausalLM. In particular, it should handle: - Processing Images: Taking a batch of images as input. If the images are variable-sized, it resizes them based on the desired patch dimensions. The image output is always img_h, img_w of (1080, 1920) Then, it patches up these images using the patchify_image function. - Creating Image Input IDs: For each patch, a placeholder ID is given to identify where these patches belong in a token sequence. For variable-sized images, each line of patches is terminated with a newline ID. - Image Patch Indices: For each image patch, the code maintains an index where these patches should be inserted in a token stream. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image to `size`. size (`Dict[str, int]`, *optional*, defaults to `{"height": 1080, "width": 1920}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `True`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to 1.0): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `"constant"`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to 0.5): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to 0.5): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `1 / 255`): The factor to use when rescaling the image. patch_size (`Dict[str, int]`, *optional*, defaults to `{"height": 30, "width": 30}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rimage_input_ids image_patchesimage_patch_indices_per_batch#image_patch_indices_per_subsequenceTN?constantg?gp? do_resizesizeresampledo_pad padding_value padding_mode do_normalize image_mean image_std do_rescalerescale_factor patch_sizec  t|di| ||_||nddd|_||_||_||_||_||_||_ | |_ | |_ | |_ | | |_ yddd|_ y)Ni8i)heightwidthr2)super__init__rcrdrerfrgrhrirjrkrlrmrn)r>rcrdrerfrgrhrirjrkrlrmrnrI __class__s r(rtzFuyuImageProcessor.__init__s "6"" ,DTD2Q    *(($"$,(2(>*r\^D_r7r' data_formatinput_data_formatr c t||\}}|d|d} } || kr|| kr|S| |z } | |z } t| | } t|| z}t|| z}td|||f|||d|}|S)a Resize an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to resize. size (`Dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Returns: `np.ndarray`: The resized image. rprq)r'rdrervrwr2)rminrNr )r>r'rdrervrwrI image_height image_width target_height target_widthheight_scale_factorwidth_scale_factoroptimal_scale_factor new_height new_width scaled_images r(r zFuyuImageProcessor.resize sF%35:K$L! k&*8nd7m| , &<=+HL+l:)K7"#68JK(<<=  &::;  i(#/    r7modeconstant_valuesct||\}}|d|d} } d} d} | |z } | |z }t|| | f| |ff||||}|S)a Pad an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to pad. size (`Dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The data format of the output image. If unset, the same format as the input image is used. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. rprqr)paddingrrrvrw)rr )r>r'rdrrrvrwrzr{r|r} padding_top padding_leftpadding_bottom padding_right padded_images r( pad_imagezFuyuImageProcessor.pad_imageDsx,%35:K$L! k&*8nd7m|   &5${2  !>2\=4QR+#/  r7return_tensorsc||n |j}||n |j}||n |j}||n |j}| | n |j} | | n |j } ||n |j }| | n |j} | | n |j} ||n |j}||n |j}| | n |j} | | n |j } | | n |j} t|trtd|Dr tdt!|}t#| | || | ||||| |Dcgc]}|Dcgc] }t%|c}}}}t'|ddr| rt(j+d|t-|dd}|Dcgc]}t/|d|}}t1|}|r1|Dcgc]$}|Dcgc]}|j3|||c}&}}}|Dcgc]}t/|d|}}|Dcgc]}|dg }}|Dcgc]}|dg }}t5||Dcgc]\}}|d|dz g}}}|r3|Dcgc]&}|Dcgc]}|j7||||| c}(}}}| r1|Dcgc]$}|Dcgc]}|j9|| | c}&}}}|r2|Dcgc]%}|Dcgc]}|j;|| | | c}'}}}|*|Dcgc]}|Dcgc]}t=|||c}}}}||||d }t?|| Scc}wcc}}wcc}wcc}wcc}}wcc}wcc}wcc}wcc}}wcc}wcc}}wcc}wcc}}wcc}wcc}}wcc}wcc}}w)a Utility function to preprocess the images and extract necessary information about original formats. Args: images (`ImageInput`): Images to preprocess. Expects a single image, a list or images or a list of lists of images. Pixel values range from 0 to 255, or between 0 and 1 if `do_rescale` is `False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image to `size`. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `self.do_pad`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to `self.padding_value`): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `self.padding_mode`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to `self.image_mean`): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to `self.image_std`): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): The factor to use when rescaling the image. patch_size (`Dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format of the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. c3\K|]$}t|txrt|dk\&yw)N)r$r%rL)r&r3s r(r)z0FuyuImageProcessor.preprocess..s)+iZ^JtT,B,Us4yTU~,U+is*,z:Multiple images for a single sample are not yet supported.) rlrmrirjrkrfsize_divisibilityrcrdrerzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.) channel_dim)rdrw)rdrrrw)scalerw)meanstdrw)rimage_unpadded_heightsimage_unpadded_widthsimage_scale_factors)rPr/) rcrdrerfrlrmrirjrkrgrhrnr$r%anyr+r,rrrlogger warning_oncerrr r ziprrescale normalizer r.)r>rrcrdrerfrgrhrirjrkrlrmrnrvrwr batch_imagesr'original_image_sizes image_sizes image_sizerr original_size resized_sizerrPs r( preprocesszFuyuImageProcessor.preprocessjsLL"+!6IDNN 'tTYY'38!-4;;#-#9Zt +9+E4K^K^'3'?|TEVEV #-#9Zt !*!6IDNN )6)B HZHZ '3'?|TEVEV #-#9Zt +9+E4K^K^#-#9Zt fd #+ibh+i(iYZ Z26: %!)%!" S__FC5.C_ _ <?1- .:   s   $ >|Aq?Q R gst]cvayFW XttT" +bhhX]UARShL _kkTZ~fQi=NOk kDO!Pj:a=/!P!PCN OZ*Q- O O 033G/U + |!_}Q/ / 0  + "( NN!)(5*; #  L  +nttdie>M^_tL  +  "(NN5zy\mNnL  "+bhhX],UKARShL #&<%:#6    T~FFSD_ u i l!P O    uis3 L8<L3 L8L> M M $M0M M M>M M('M# M( M3M. 7M3 M>M9 +M>: N N N 3L8M#M(.M39M>N rzr{c||n |j}|jd|jd}}||zdk7rtd|d|||zdk7rtd|d|||z}||z}||z}|S)a Calculate number of patches required to encode an image. Args: image_height (`int`): Height of the image. image_width (`int`): Width of the image. patch_size (`Dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rprqrz image_height=z must be divisible by z image_width=)rnr+) r>rzr{rn patch_height patch_widthnum_patches_per_dim_hnum_patches_per_dim_w num_patchess r(get_num_patchesz"FuyuImageProcessor.get_num_patchess$.#9Zt $(OOH$=tw?Wk , &! + .D\NST T  $ ) ~-CK=QR R , < +{ :+.CC r7z torch.Tensorcht|dg||n |j}|d|d}}|j\}}}}|jd||}|jd||} | j } | j ||d||} | j ddddd } | j|d||z|z} | S) a| Convert an image into a tensor of patches. Args: image (`torch.Tensor`): Image to convert. Shape: [batch, channels, height, width] patch_size (`Dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rCrprqrrrr)rrnshapeunfold contiguousviewpermutereshape) r>r'rnrr batch_sizechannels_unfolded_along_heightpatchess r(patchify_imagez!FuyuImageProcessor.patchify_image5s $ *#-#9Zt $.x$8*W:Mk &+[[" Ha % Q l K'..q+{K$$&,,z8R{S//!Q1a0//*b(\2IK2WXr7 image_input image_presentimage_unpadded_himage_unpadded_wimage_placeholder_idimage_newline_idvariable_sizedc t|dg||n |j}|d|d} } g} g} g} t|jdD]\}g}g}t|jdD]}|||fr|||f}|jd|jd}}|rft |t j |||f| z | z}t |t j |||f| z | z}|ddd|d|f}||}}|j||}tj|g|tj|j }|j|jd jd}||jdk(sJ|r|jd || z}tj|jddg|tj|j }tj ||gd }|jd }| j#|g|j#||j#||j#tj$gtj|j | j#|| j#|_g}g}| D]}d}g}g} |D]}!|!|k(}"tj&|"}tj(|tj*|!j j-|!}#tj.|!d }$tj.|!d }%tj0|"d d}&|#|z|$|&<|#|%|&<|j#|$| j#|%||z }|j#||j#| t3| | | ||dS)aProcess images for model input. In particular, variable-sized images are handled here. Args: image_input (`torch.Tensor` of shape [batch_size, subsequence_size, num_channels, height, width]): Tensor of images padded to model input size. image_present (`torch.Tensor` of shape [batch_size, subsequence_size, num_images]): Tensor of 1s and 0s indicating whether an image is present. image_unpadded_h (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image heights. image_unpadded_w (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image widths. image_placeholder_id (int): The id of the image placeholder token. Comes from an associated tokenizer. image_newline_id (int): The id of the image newline token. Comes from an associated tokenizer. variable_sized (bool): Whether to process images as variable-sized. patch_size (`Dict[str, int]`, *optional*, defaults to `self.patch_size`): Size of the patches. rCNrprqrrr)rzr{)dtyperD)r'r)dimT)as_tupler\)rP)rrnrangerrymathceilrrCfullint32rDr unsqueezesqueezercatrOtensor count_nonzeroarangeint64type_as full_likenonzeror.)'r>rrrrrrrrnrrrbatch_image_patchesbatch_image_input_ids batch_indexr]r^ subseq_indexr'rzr{new_hnew_wrtensor_of_image_idsr newline_idsr_r`sample_image_input_ids index_offsetper_batch_indicesper_subsequence_indicessubseq_image_input_ids patches_maskindicesindices_in_stream_per_batch!indices_in_stream_per_subsequence patches_indss' r(preprocess_with_tokenizer_infoz1FuyuImageProcessor.preprocess_with_tokenizer_infoOs> $ *#-#9Zt $.x$8*W:Mk ,.8::< !2!21!56/ 6K OM %k&7&7&: ;) k  l!:;' \(ABE05 A A+L%!$( II&6{L7P&QT`&`adpp!!$' II&6{L7P&QT_&_`cnn!!&a%%&7 8495k "&"6"6L^i"6"jK*/**$ ';5;;WbWiWi+'#118J1KSSTUVG&'--*::::%.A.I.I"k]hNh.i+&+jj066q91=,"'++#.#5#5 ' /4ii9Lk8Z`a.b+.A.I.I".M+MM5'*#**+>?!((1#**5<<%++VaVhVh+ijS) kV " ( ( 9  & &} 5_/ 6fCE%HJ+&; P "L " &( #*@ ,&59MM #11,? ,,{%++NdNkNkltt* /4oo>TVX.Y+49OODZ\^4_1$}}\DI!L BDH 8zz838n8% 8 eC)9$9:; 8 $E#/?*?$@A 8 8|!$>BDH$zz$38n$ $  $ eC)9$9:; $$E#/?*?$@A$ $L%&%))-15!%)-&*'+&*%)%)*./3>N>T>TDH/3#oGD>oGtCH~& oG -. oG  oG oGsmoGtnoGUOoGE?oGTNoG!oGT#s(^,oGeC)9$9:;oG $E#/?*?$@A!oG"!,#oG'oGbCctTWY\T\~il2NcSVh@XdrF04B #B &B ) B ) B " B B B T#s(^,B  B r7r[)+rYrtypingrrrrnumpyrimage_processing_utilsrr r image_transformsr r r image_utilsrrrrrrrrrrutilsrrrrrrrrC get_loggerrVrr,r.r[r2r7r(rs& ..UU      H % a $tJ'($z*:JF G a $z  aj|jZ[ +[ r7