sgddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z ddlmZmZmZmZddlmZmZmZmZmZmZmZmZmZmZddlmZm Z er ddl!Z!dd l!m"Z"e jFe$Z%ed d e&d eee&e&ffdZ'de&de&de&de&de&d ee&e&ff dZ(ed de&de&d e&de&d ee&e&ff dZ)de jTde&de&d e jTfdZ+deeee&e&fd e&d e jTfdZ,deee jTd e&d ee jTeee&ffdZ-d.deeee&e&fde&d e jTfd Z.deeee&e&fd e&d e jTfd!Z/ d/de jTd"eee0fd#eeee0fd e jTfd$Z1ded efd%Z2d&ed eeee jTfd'Z3d&efd(Z4d)ee0e&fd dfd*Z5d+Z6Gd,d-e Z7y)0N) lru_cache)DictListOptionalTupleUnion)BaseImageProcessor BatchFeature) PaddingModeget_image_sizepadresize) IMAGENET_STANDARD_MEANIMAGENET_STANDARD_STDChannelDimension ImageInputPILImageResamplinginfer_channel_dimension_formatis_valid_imageis_vision_availableto_numpy_arrayvalidate_preprocess_arguments) TensorTypelogging)Image )maxsizemax_image_tilesreturncg}td|dzD]2}td|dzD]}||z|ks |j||f 4|S)a Computes all allowed aspect ratios for a given maximum number of input tiles. This function calculates all possible arrangements of tiles that can be formed within the constraint of the maximum number of tiles. Each arrangement is represented by its aspect ratio (width/height) and the corresponding tile configuration. Args: max_image_tiles (`int`): The maximum number of tiles allowed. Returns: `List[Tuple[int, int]]`: A list of tuples, each tuple representing a valid (width, height) configuration in terms of number of tiles. Example: >>> get_all_supported_aspect_ratios(4) [(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (2, 2), (3, 1), (4, 1)] )rangeappend)r aspect_ratioswidthheights e/var/www/html/venv/lib/python3.12/site-packages/transformers/models/mllama/image_processing_mllama.pyget_all_supported_aspect_ratiosr)4se,Mq/A-.6A23 6Fv~0$$eV_5 66  image_height image_width canvas_height canvas_width tile_sizectj|||}tj|||}||z }||z }||kr(|} ttj||z|} | | fS|} ttj||z|} | | fS)a Calculates the new size of an image to fit within a canvas while maintaining aspect ratio. This function calculates the optimal size for an image to fit within a canvas defined by canvas_height and canvas_width, while ensuring that the image dimensions are not smaller than tile_size. If the image is larger than the canvas, the returned size will fit within the canvas. If the image already fits within the canvas, the size remains unchanged. The aspect ratio of the original image is preserved. Args: image_height (`int`): The height of the original image. image_width (`int`): The width of the original image. canvas_height (`int`): The height of the canvas. canvas_width (`int`): The width of the canvas. tile_size (`int`): The tile size. Returns: `Tuple[int, int]`: A tuple containing the new height and width of the image. )npclipminmathfloor) r+r,r-r.r/ target_width target_heightscale_hscale_w new_width new_heights r(get_image_size_fit_to_canvasr<RsB77; <@LGGL)]CMl*G[(G L7$:;]K y  #  ;#89<H y  r*dct|}tj||z}tj|j\}}||z }||z } tj| |kD|| } | | dk\} t | dkDrtj | } n| | dk} tj| } || | k(}t |dkDr/|dddf|dddfz}tj|}||}|S|d}|S)a Determines the best canvas based on image and tile size and maximum number of tiles. First, calculates possible resolutions based on the maximum number of tiles and tile size. For example for max_image_tiles=2, tile_size=100, possible tile arrangements are: [(1, 1), (1, 2), (2, 1)] and corresponding canvas sizes are: [(100, 100), (100, 200), (200, 100)] For each possible resolution, calculates the scaling factors for width and height, and selects the smallest one, which is the limiting side. E.g. to match the canvas you can upscale height by 2x, and width by 1.5x, therefore, the maximum upscaling you can do is min(2, 1.5) = 1.5. If upscaling is possible (any of the scaling factors is greater than 1), then picks the smallest upscaling factor > 1. If upscaling is not possible, then picks the largest scaling factor <= 1, i.e. reduce downscaling as much as possible. If there are multiple resolutions with the same max scale, we pick the one with the lowest area, to minimize padding. E.g., the same image can be upscaled to 224x224 and 224x448, but the latter has more padding. Example of canvases made from tiles: To visualize how the image can fit onto different tile grids, let's try fitting an ASCII cat into the tiles. Here's an ASCII cat image you want to fit into the tiles: /\_/\ ( o.o ) > ^ < If `num_tiles=6`, possible tile grids would look like this: **2x3 Canvas (2 tiles wide, 3 tiles tall)**: -> total of 6 tiles +-------+-------+ | /\_/\ | 0 | <- Cat image split across two tiles horizontally +-------+-------+ | > ^ < | 0 | <- Remaining part of the cat occupies the left tile +-------+-------+ |( o.o )| 0 | +-------+-------+ **3x2 Canvas (3 tiles wide, 2 tiles tall)**: -> total of 6 tiles +-------+-------+-------+ | /\_/\ |( o.o )| 0 | <- Cat image occupies the first two tiles, 1 tile remains empty +-------+-------+-------+ | > ^ < | 0 | 0 | <- Remaining part of the cat occupies the left tile +-------+-------+-------+ **1x6 Canvas (1 tile wide, 6 tiles tall)**: -> total of 6 tiles +-------+ | /\_/\ | <- Top part of the cat +-------+ |( o.o )| <- Middle part of the cat +-------+ | > ^ < | <- Bottom part of the cat +-------+ | 0 | +-------+ | 0 | +-------+ | 0 | +-------+ Given that the tiles you get depend on the chosen aspect ratio, you have to add embedding in the modeling code to help it know if it got a 3x2 or a 1x6 or a 2x3 aspect ratio. The function tests these arrangements to find the smallest canvas where the image fits. If multiple canvases fit, it selects the one where the dimensions are closest to the image size. In this case the first canvas is the closest to the original image. You then feed all of the tiles to the model: +-------+-------+-------+-------+-------+-------+ - | /\_/\ |( o.o )| > ^ < | 0 | 0 | 0 | <- Last canvas +-------+-------+-------+-------+-------+-------+ +-------+-------+-------+-------+-------+-------+ - | /\_/\ | 0 |( o.o )| 0 | > ^ < | 0 | <- First canvas +-------+-------+-------+-------+-------+-------+ +-------+-------+-------+-------+-------+-------+ - | /\_/\ |( o.o )| 0 | > ^ < | 0 | 0 | <- second canvas +-------+-------+-------+-------+-------+-------+ For each tile, you have num_channels (usually RGB so 3), tile_width, tile_height Args: image_height (`int`): The height of the image. image_width (`int`): The width of the image. max_image_tiles (`int`): The maximum number of tiles any image can be split into. tile_size (`int`): The tile size. Returns: `Tuple[int, int]`: The best canvas resolution [height, width] for the given image. r"rN) r)r1arrayTwherelenr3maxargmin)r+r,rr/possible_tile_arrangementspossible_canvas_sizestarget_heights target_widthsr8r9scalesupscaling_optionsselected_scaledownscaling_options chosen_canvasareas optimal_idxoptimal_canvass r(get_optimal_tiled_canvasrQs-^"A!QHH%?@9L%'HH-B$C$E$E!NM|+Gk)GXXg'' :Fv{+ ! 12%VaZ0 34*&N*BCM =Aad#mAqD&99ii& &{3 'q) r*imagenum_tiles_heightnum_tiles_widthc|j\}}}||z}||z}|j|||||}|jddddd}|j||z|||}tj|S)a6 Split an image into a specified number of tiles along its width and height dimensions. Args: image (`np.ndarray`): Input image with shape (num_channels, height, width). num_tiles_height (`int`): Number of tiles to split the image into along its height. num_tiles_width (`int`): Number of tiles to split the image into along its width. Returns: `np.ndarray`: Array of image tiles with shape (num_tiles_width * num_tiles_height, num_channels, tile_height, tile_width). r"r r)shapereshape transposer1ascontiguousarray)rRrSrT num_channelsr'r& tile_height tile_widths r(split_to_tilesr_s #(++L&%,,K/)J MM,(8+Xb cE OOAq!Q *E MM/,<t|}t|Dcgc] }t|c}}tj|||ftj}d|dddddf<t |D](\}}t |D]\}\} } d|||d| | zf<*|Scc}w)aP Builds a mask for the aspect ratios of the images. Args: aspect_ratios (`List[List[Tuple[int, int]]]`): A list of lists containing aspect ratios for each image in the batch. Each aspect ratio is represented as a tuple of (width, height) in terms of number of tiles. max_image_tiles (`int`): The maximum number of tiles any image can be split into. Returns: `np.ndarray`: A 3D numpy array of shape (batch_size, max_num_images, max_image_tiles). The mask contains 1s for valid tiles and 0s for padding. dtyper"Nr)rBrCr1zerosint64 enumerate) r%r batch_sizerowmax_num_imagesaspect_ratio_maskisample_aspect_ratiosj num_tiles_w num_tiles_hs r(build_aspect_ratio_maskro8s]#Jmz&make_list_of_images..sK 8dE]3Ks "c32K|]}t|ywr)is_valid_list_of_imagesrs r(rz&make_list_of_images..sIh'1Iz]Invalid input type. Must be a single image, a list of images, or a list of batches of images.)rrrrrallanyr)rs output_imagess r(make_list_of_imagesrsf    FT5M */Fv/N   6D%=) KFK K I&I I  k  r*c.|xrtd|DS)Nc32K|]}t|ywr)r)rrRs r(rz*is_valid_list_of_images..&sDE./Dr)r)rss r(rr%s  DcDVDDDr*rcbd|vrd|vstd||d|dk7rtd|y)Nr'r&zJArgument `size` must be a dictionary with keys 'height' and 'width'. Got: z9Argument `size` must have the same height and width, got )r)rs r(_validate_sizer)sO  Defjeklmm H~g&TUYTZ[\\'r*cz|s td|s td||dkrtd|dt|y)Nz9MllamaImageProcessor doesn't support `do_pad=False` mode.z*DZ&/&;AV .-dnndiiVZVjVjkr*rsinput_data_formatreturn_tensorsc ||n |j}||n |j}||n |j}||n |j}||n |j}||n |j }||n |j }| | n |j} | | n |j} | | n |j} | | n |j} t|||| | |||t||| | t|}|jr(|Dcgc]}|Dcgc] }t|c}}}}|Dcgc]}|Dcgc] }t|c}}}}g}g}|D]}g}g}|D]}t j"}t%||| }|j'|||| ||\}}|j)|||||}|r|j+||| |}|r|j-|| | | |}|\}}t/|||}|j1||j1||f|j1||j1|t3|| \}}t5|| }t7|| }t9|||d| }||d <|Scc}wcc}}wcc}wcc}}w) a Preprocess a batch of images. Args: images (`ImageInput`): A list of images to preprocess. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image tile. Should be a dictionary containing 'height' and 'width' keys, both with integer values. The height and width values should be equal. resample (`int`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. do_pad (`bool`, *optional*, defaults to `self.do_pad`): Whether or not to pad the images to the largest height and width in the batch. max_image_tiles (`int`, *optional*, defaults to `self.max_image_tiles`): The maximum number of tiles to split the image into. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. Returns: `BatchFeature` of the following structure: - **pixel_values** (`TensorType`): The preprocessed pixel values. - **aspect_ratio_ids** (`TensorType`): The aspect ratio ids of the images. - **num_tiles** (`List[List[int]]`): The number of tiles for each image in the batch. )rrrrrrrr)r)rRrrrr data_format)rRr aspect_ratiorr)rRscalerr)rRmeanstdrr)r)rrri)data tensor_typerz)rrrrrrrrrrrrrrrrrrrrrrescale normalizer_r$r{rror )rrsrrrrrrrrrrrrr images_listrRrpbatch_aspect_ratios sample_imagesrkrrrSrTrzrriencoded_inputss r( preprocesszMllamaImageProcessor.preprocess{s H,:+E4K^K^!*!6IDNN 'tTYY'38#-#9Zt +9+E4K^K^'3'?|TEVEV #-#9Zt !*!6IDNN !-4;;-<-H/dNbNb%!)%!  .ivW)&1   U`a6vFeN51FaKaQ\]v6B%u-B] ]  "6 =FM#%  . Q/44 3E;Zkl'+kk%$3&1 + '2'#|!-&1 + ! LL#,*;$/ )E  NN#'%*;$/ +E5A1 /&u.>P$$U+$++-=,OP]. Q`    .  & &'; 9 &g6  45<+;U7VW  %%#/   r*ct|t||\}}|d} t|||| \} } | | z} | | z} t||| | | \}}t |||f|||}|| | ffS)a Resizes an image to fit within a tiled canvas while maintaining its aspect ratio. The optimal canvas size is calculated based on the maximum number of tiles and the tile size. The function first determines the best tile arrangement for the image, then resizes the image to fit within this canvas. The resized image and the number of tiles along the height and width dimensions are returned. Args: image (`np.ndarray`): Image to resize. size (`Dict[str, int]`): Size of the output image. max_image_tiles (`int`): The maximum number of tiles to split the image into. resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`): Resampling filter to use when resizing the image. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format of the image. If not provided, it will be the same as the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. Returns: `Union[np.ndarray, Tuple[int, int]]`: The resized image and a tuple containing the number of tiles along the height and width dimensions. rr')r+r,rr/)r+r,r-r.r/)rrr)rr rQr<r)rrRrrrrrr+r,r/r-r.rSrTr;r:s r(rzMllamaImageProcessor.resize`sH t$25FW$X! kN &>%#+ ' # | )I5&)3 <%#'% !  I   ##/  '999r*) NNNNNNNNNNNNN)NN)__name__ __module__ __qualname____doc__model_input_namesrBILINEARboolrrstrintfloatrrrrrrrr1rrrr __classcell__)rs@r(rr:sW@_ $)-'9'B'B '!:>9= llltCH~& l % l  lllU5$u+#567lE%e"456lll lB*.$()-15%)*.'+:>9=!%)-DH;?tt!tD> t tCH~& t -. tTNt!ttntU5$u+#567tE%e"456tt"#t$E#/?*?$@At!sJ!78tv?CDH -zz-38n-CHo - eC)9$9:; - $E#/?*?$@A - -h(:'B'B>BDHB:zzB:38nB: B: % B: eC)9$9:; B:$E#/?*?$@AB: rzz5c?* +B:r*r)r"r)8r4 functoolsrtypingrrrrrnumpyr1image_processing_utilsr r image_transformsr r rr image_utilsrrrrrrrrrrutilsrrrr get_loggerrloggerrr)r<rQrr_ror{rrrrrrrrrrrr*r(rs  55F   )   H % 2ST%S/=R:.!.!.!.! .!  .!  38_ .!b 3RRRR R  38_ RRj'"**''c'VXV`V`'>4U38_0E+FY\acakakB2)tBJJ'(2)2) 2::tDI &'2)j!d4c3h+@&A!c!Z\ZdZd!0T%S/5J0K^afhfpfpDAE% ::%',-% &6&; <=%ZZ %R*0 tD"**9M4N/O>EDE]c3h]D]h:-h:r*