sg\<dZddlmZmZmZmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZddlmZmZddlmZmZdd lmZerdd lZerdd lZd ZGd d e dZGddedZGdde dZ ddZ!dZ"dZ#dZ$dZ%GddeZ&y )z Processor class for IDEFICS. )CallableDictListOptionalUnion)urlparse) BatchFeature) ImagesKwargsProcessingKwargsProcessorMixin TextKwargsUnpack!_validate_images_text_input_order)PreTokenizedInput TextInput)is_tf_availableis_torch_available)deprecate_kwargNcxeZdZUeeed<eeeefed<ee e e e fed<ee e e e fed<y)IdeficsImagesKwargs transform image_size image_mean image_stdN) __name__ __module__ __qualname__rr__annotations__rstrintrfloatra/var/www/html/venv/lib/python3.12/site-packages/transformers/models/idefics/processing_idefics.pyrr-sR!!c3h((ud5k1233eT%[0122r%rF)totalc.eZdZUeeed<eeed<y)IdeficsTextKwargs add_eos_tokenadd_end_of_utterance_tokenN)rrrrboolr r$r%r&r)r)4sD>! (.r%r)c:eZdZUeed<eed<ddddiddidZy ) IdeficsProcessorKwargs text_kwargs images_kwargsFlongest)add_special_tokenspaddingr*return_tensorspt)r/r0 common_kwargsN)rrrr)r r _defaultsr$r%r&r.r.9s2""&&#( " *D1Ir%r.c|dk7r-|dk(r d|||k\<n|dk(rtj||k\d|}|dk(r@|dk(}d||<tjjj ||}d||ddf<|S|dk(rtj |d}tj|d|}tj ||}tj|d}tj|tj||}S)Nr5tfr num_classes)depth) r:wheretorchnn functionalone_hotequal expand_dims zeros_like)incremental_maskr4r< negatives attn_masknegatives_expandeds r&$incremental_to_binary_attention_maskrJHs b T !@B -< = t #!xx(8K(GM]^ $* &'#HH''//0@k/Z "# )Q,  4 HH-r2 88Iq2BCJJ/{C ^^Ir:HH/y1I9U r%cH|dk(r t||S|dk(r t||Sy)Nr5r:),image_attention_mask_for_packed_input_ids_pt,image_attention_mask_for_packed_input_ids_tf) input_ids tokenizerr4s r&)image_attention_mask_for_packed_input_idsrPbs1;IyQQ 4 ;IyQQ r%ctj|d}tj|d}|jt}|j}t |j dD]K}d}d}t||D]4\} } | |k(r|dz }|||| <d}n|||| <|rd||| <| |k(s3d}6Mt |j dD]}d}d}t ||j ddz ddD]9} ||| } | |k(r|dz }|||| <d}n|||| <| |k(rd}|s2d||| <;||dk7} ||| xx|zcc<||| xxdzcc<||fS)Nr9) fill_valuerFT)r? full_likeconvert_tokens_to_ids IMAGE_TOKEN eos_token_idrangesize enumerate) rNrOimage_attention_masknext_image_attention_maskimage_token_id eod_token_id batch_idxcountseen_eodidxtoken_idnon_negative_indicess r&rLrLis ??9D % b I44[AN))L9>>!,- &y';< MC>) 7<$Y/4 7<$Y/479$Y/4<'  "9>>!,-I 9-221592rB ?C +C0H>) ))4S9 ? 9CrI!),-ABeKB!),-ABbHB)I, !: ::r%c|jt}|j}tj|d}tj tj|d}tj tj|d}t |D]}d}d} tj|d} t | dz ddD]} ||| fj} | |k(r<|dz }|| gg} |g}tj|| |}tj|| |}n*| |k(r%| s#d} d}|| gg} |g}tj|| |}| s| |k7s|| gg} dg}tj|| |}||fS)Nrr9FrST) rUrVrWr:shapefillrXnumpytensor_scatter_nd_update)rNrOr]r^ batch_sizer[r\r_r`ra seq_lengthrbrcindicesupdatess r&rMrMs44[AN))L)$Q'J77288I#6; "(;R @:&u XXi(+ aR0 uC C0668H>) %s+, '')'B'BCWY`bi'j$,.,G,GHacjls,t)\)(%s+, ',.,G,GHacjls,t)H 4%s+,$,.,G,GHacjls,t)# u u. !: ::r%cdd|vryt|}t|j|jgS)zChecks if the passed string contains a valid url and nothing else. e.g. if space is included it's immediately invalidated the url F)rallschemenetloc)stringresults r&is_urlrus0 f} f F  v}}- ..r%c eZdZdZddgZddgZdZdZdfd Ze d d d d  dd e e e e e e e e e e e e e fdeedefdZdZdZedZxZS)IdeficsProcessora Constructs a IDEFICS processor which wraps a LLama tokenizer and IDEFICS image processor into a single processor. [`IdeficsProcessor`] offers all the functionalities of [`IdeficsImageProcessor`] and [`LlamaTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`IdeficsImageProcessor`): An instance of [`IdeficsImageProcessor`]. The image processor is a required input. tokenizer (`LlamaTokenizerFast`): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. image_size (`int`, *optional*, defaults to 224): Image size (assuming a square image) image_processorrOrr+IdeficsImageProcessorLlamaTokenizerFastc | td| tdt| |||j|_t |dr |j n|jt|_|jj|jj|jjf|_ d|jjjdgvrd|_yd|_y)Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`. image_tokenadditional_special_tokensTF) ValueErrorsuper__init__rxcurrent_processorhasattrr]rUrVimage_num_channelsrdefault_image_dimsrOspecial_tokens_mapget1tokenizer_was_trained_with_end_of_utterance_token)selfrxrOrr+kwargs __class__s r&rzIdeficsProcessor.__init__s  "HI I  AB B )4!%!5!5y-0  $ $00=   3 3  + +  + +# $t~~'H'H'L'LMhjl'mm  > >r%promptsz5.0.0textT)old_nameversionnew_nameraise_if_both_namesrreturnc L ./| | tdt||\}}||}n|t|ttfs|g}t|t r|g}t|ttfr"t |t |k7r tdtd|Ds tdt|dttfr|Dcgc]}|g}}tt||}|jtfd|jji|}|djd d } |djd d} | |j} td Ds|g}d .d/d} ./fd} g} g}|D]&}|jj }g}d }d }t#|D]\}}|dkDr|sdnd }t|t rg|j%d}t'|r:|j(j+|}|| |z }|j-|d}w| r|r|| z }||z }d }|| |z }|j-|d}| r||jj.z }|j(|fi|d}| j-||j-|)|djdd}|j| fi|d}|d}|d}t1d|D}t1d|}t3d|DdkD}g}g}g}t|||D]2\}} }!|}"|"j5|j6}#t9|#|}$|!d|$}%t |%dkDr|dk(r=t;j<|g|%j?dd}&|%|&d|%j?dn|dk(rtAjB|%dd}'tAjD|g|'gd}(tAj<|(|%jF}&tAjB|%d})tAjHtAjJ|)d}*|%}+tAjL|&|*|+}&nN|dk(r!t;j<|g|jN}&n(|dk(r#tAj<|g|jN}&|j-&|dk(rJ|j-t;jP|"|j-t;jP| |dk(s|j-tAjR|"t@jT|j-| 5|dk(r@t;jV|}t;jV|}t;jV|}nD|dk(r?tAjV|}tAjV|}tAjV|}|r)tY||j|\},}-t[|,||},n|dk(rBt;j<|jBd|jBddt:j\},nG|dk(rBtAj<|jBd|jBddft@j\},t_|||,d !Scc}w)"aThis method takes batched or non-batched prompts made of text and images and converts them into prompts that the model was trained on and prepares the image pixel values for the model to process. Args: images (`Union[PIL.Image, str, List[PIL.Image], List[str]]`): either a single image or a batched list of images - can be passed in when text contains only text prompts, in order to use the image-text-to-text behavior. text (`Union[List[TextInput], [List[List[TextInput]]]]`): either a single prompt or a batched list of prompts - see the detailed description immediately after the end of the arguments doc section. return_tensors (`str` or `TensorType`, *optional*, defaults to `TensorType.PYTORCH`): The type of tensors to return. Can be one of: - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. Returns: a dict with entries: `input_ids`, `attention_mask`, `pixel_values`, `image_attention_mask` which can be directly passed to `model.generate` Detailed explanation: Each entry in `text` is either a text to be passed as is or an image that will be processed. An image can be either an image object (`PIL.Image`) or a url from which the image can be retrieved. When the processor encounters an image it'll inject `` entry into the prompt. Example: ```python checkpoint = "HuggingFaceM4/idefics-9b" processor = AutoProcessor.from_pretrained(checkpoint) url = "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg" img = processor.image_processor.fetch_images([url])[0] prompts = [ "User:", img, "Describe this image. Assistant: An image of two kittens in grass. ", "User:", "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg", "Describe this image. Assistant:", ] inputs = processor(text=prompts, return_tensors="pt") generated_ids = model.generate(**inputs, max_length=100) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` In this example the `prompts` will be converted into: ``` User:Describe this image. Assistant: An image of two kittens in grass. User:Describe this image. Assistant:' ``` and the two images will be massaged using [`IdeficsImageProcessor.__call__`] method and placed inside the `pixel_values` dict entry of the return value. This example also examplifies that images can be passed as objects or as text urls. It can be seen that the first image is passed as object and the second one as a url. To do training do: ```python image_transform = transforms.Compose( [ transforms.RandomResizedCrop( (w, h), scale=(0.9, 1.0), interpolation=transforms.InterpolationMode.BICUBIC ), transforms.ToTensor(), transforms.Normalize(mean=self.image_mean, std=self.image_std), ] ) inputs = processor(text=prompts, transform=image_transform, return_tensors="pt") ``` In order to help debug prompt generation enable `debug=True` which will show you what's happening. Nz9You need to specify either `text` or `images` and `text`.aWhen providing both images and text arguments, the number of text prompts should be the same as the number of images.If you want to have several images per prompt, images should be nested as such: images=[[img1, img2], [img3, img4], ...] for text=[prompt1, prompt2, ...].c3<K|]}t|tywN) isinstancer!.0is r& z,IdeficsProcessor.__call__..hs8az!S)8szQWhen using the image-text-to-text behavior, the prompts should only contain text.rtokenizer_init_kwargsr/r*Fr+c3HK|]}t|ttfywr)rlisttuplers r&rz,IdeficsProcessor.__call__..|sAA:a$/As "zrr}c"|rzSzzSrr$)last_was_image fake_tokenr|s r& image_tokensz/IdeficsProcessor.__call__..image_tokenss!"Z//!K/*<.s8SV8rSc32K|]}t|ywrrrs r&rz,IdeficsProcessor.__call__..s dlXY Y8F  >G  ftUm4 $$v$u .3t9F 3K q 8488 !tuu&)dE]3%)***3vt,-G*** " "&.."<"<  &m488%P %2=%A%E%EFbdh%i" & -)-)_)_ &AAAiG0  !5 =   % -F>>334IM"N!M$V, *4q50>DEMdC(::c?Dd| $ 4 4 A A$ G!\.%AA %,,U3)-6-%)??I!T) ).n!==I!((.%)N+ *.T^^555 0D00a-P_B`aM   y )   m ,K% -P'}599:JDQ&{SmM6RS !+. +,<=8Z88Q/ < < 9K)9* *001D1DEK#&{N#C -.C/CDN>"Q&!T)*/++n*a~GZGZG\]^]_G`*a'DR'(@.*=*=a*@A#t+#%((>":12">K#%99~.> -LST#UL*,((<~G[G[*\'!#.!9!!.JK4' ''(<(<=MUWU]U](^_&--n=K% >N T !${{+;< !KK 6M%*[[1G%H " t #!xx(89 HH]3M%'XX.D%E " &O $...' # !$H$n.$ %',{{$**1-/?/E/Ea/H!SXS]S]($ 4'')xx%++A.0@0F0Fq0I1MUWU\U\($-"8 -(<   m+s Z!c:|jj|i|S)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rO batch_decoderargsrs r&rzIdeficsProcessor.batch_decode s +t~~**D;F;;r%c:|jj|i|S)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rOdecoders r&rzIdeficsProcessor.decodes %t~~$$d5f55r%c|jj}|jj}ttj ||zSr)rOmodel_input_namesrxrdictfromkeys)rtokenizer_input_namesimage_processor_input_namess r&rz"IdeficsProcessor.model_input_namess? $ @ @&*&:&:&L&L#DMM"7:U"UVWWr%)NN)NNNN)rrr__doc__ attributes valid_kwargsimage_processor_classtokenizer_classrrrrrrrr.r rrrpropertyr __classcell__)rs@r&rwrws $[1J ">?L3*O 4i6_cd X    O " # i ! '( )  + X /0X  X eX t<6XXr%rw)r9)'rtypingrrrrr urllib.parserfeature_extraction_utilsr processing_utilsr r r rrrtokenization_utils_baserrutilsrrutils.deprecationrr? tensorflowr:rVrr)r.rJrPrLrMrurwr$r%r&rs98!4D80 3,e3/ %/ -U 4R,;^;B/[X~[Xr%