sgxxdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#d dl$m%Z%e"jLe'Z(dZ)dZ*eGddeZ+Gdde jXZ-Gdde jXZ.Gdde jXZ/Gdde jXZ0Gdde jXZ1Gdde jXZ2Gdd e jXZ3Gd!d"e jXZ4Gd#d$e jXZ5Gd%d&e jXZ6Gd'd(eZ7d)Z8d*Z9d+Z:e d,e8Gd-d.e7Z;Gd/d0e jXZ<e d1e8Gd2d3e7Z=Gd4d5e jXZ>Gd6d7e jXZ?e d8e8Gd9d:e7Z@e d;e8Gd<d=e7ZAe d>e:Gd?d@e7ZBe dAe8GdBdCe7ZCy)DzPyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViltConfigrzdandelin/vilt-b32-mlmceZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee e ejed<y)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr rrr!Y/var/www/html/venv/lib/python3.12/site-packages/transformers/models/vilt/modeling_vilt.pyrr4sl$)-D(5$$ %, $FE  $>BM8Du'8'8!9:;B;?JeE$5$5678?r*rc4eZdZdZfdZddZ ddZxZS)ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. c,t|t||_t j t jdd|j|_ t||_ |jj}t j t jd|dz|j|_ t j|j|j|_t j"|j$|_||_yNr)super__init__TextEmbeddingstext_embeddingsr Parameterr&zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfrBr: __class__s r+r1zViltEmbeddings.__init__Ws  .f5ekk!Q8J8J&KL 3F ;++77 #%<< A{QPVPbPb0c#d %'\\&2Q2QSYSeSe%f"zz&"<"<=  r*cL |jjjj\}}}}|j|}|dddddddfj }t j j||jd|jdfj}|dddfjddddf} |dddfjddddf} |j\} } } }|jj|jjz}|jddddddfjddjd| ||}t!j"t%| | Dcgc]R\}}t j j't j j|||fdd d||z d| |z fTc}}d}|j)djdd}|j)djdd}t!j*t-t!j.|jd t!j.|jd d d j1|j2}|ddddddddf}|j5|jd|jdd d d }|j)dd}|j)d}|dks|t7|t8s| | z}|j;}n| | z}t=|j;|}|j?d}d|z j?d}|dddfjA}|Dcgc]}||dddf|k(}}|Dcgc]}||dddf|k(}}|Dcgc]}|jCd}}|Dcgc]}|jCd}}|Dcgc]}||z  }}g} tEt%|||D]\}!\}}"}#|#dkrOt!jFt!jH|j |}$| jK||!|$^t!jFt!jH|"j |#d}%| jKt!j"||!||!|%gdt!j"| d} || dddf| dddffj| d | }|| dddf| dddffj| d }|| dddf| dddffj| d d}|| dddf| dddffj| d | }|jLj5| d d }&t!j"|&|fd}t!j"|jdddddfdddddfj5| d d |fd}||z}|jO|}t!j"t!jH|jddj1||gd}|||| |fffScc}}wcc}wcc}wcc}wcc}wcc}w)Nr )sizerrdimbilinearT)rGmode align_cornersij)indexingdeviceF)as_tuple) replacement)(r9 projectionweightshapefloatr functional interpolatelongsumrB image_size patch_sizer; transposeviewr&catzippadflattenstackrarangetorRexpand isinstanceintmaxminnonzerouniquerG enumerate multinomialonesappendr7rA)'rC pixel_values pixel_maskmax_image_length_phpwxx_maskx_hx_w batch_size num_channelsheightwidth patch_dim spatial_poshw pos_embed patch_indexeffective_resolution valid_idx non_valid_idx unique_rowsu valid_row_idxnon_valid_row_idxv valid_numsnon_valid_numspad_numsselectinvp valid_choice pad_choice cls_tokenss' r+ visual_embedzViltEmbeddings.visual_embedfsR,,77>>DD 1b"  ! !, /AtQM*002**6QWWQZ8P*QVVXQTl1%ad+QTl1%ad+23''/ L&%KK**dkk.D.DD ..q!"ax8BB1aHMMaQ]_hjst II SM Aq !!MM--#V'&* .  1fqj1  %%a(221a8 IIaL " "1a (kk U\\&,,r"23U\\&,,rBR5S^b cik "FMM" " "$aA"56 !((a&,,q/2rSUV !))!Q/ " a #3#;:N^`cCd $'9 3779 #&9 "#7#;#;#=?OP NNEN2 V,,e,< 1o,,. BMNQ9QT?a#78N NNYZ]=A+>!+CDZZ)67AaffQi7 7->?!&&)??2<=Q$q(==&s:~x'PQ fMAz2qAv$00A1D1D1FHXY  mA.|<="..uzz"~/C/C/EqVZ[  eiiq)9;LQ;OPZ;[(\bcde  f6q) fQTlF1a4L( ) . .z2| Lq!t fQTl2388RH!&A,q!t "<=BB:rSTU fQTlF1a4L89>>z2|\ ^^**:r2> IIz1o1 -II  % %aAg .q$z : A A*bRT UW` agh   M LLOEJJv||A:==fEvNTUV&;888S POZ7?=s%?AZ Z +ZZ%Z Z!c (|j|||} |-|j|||jj\}} } n|j d} |d}| |j t j|t j| jz} ||j t j| |t j| jz}t j| |gd} t j|| gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rurdtyperRrH) r3rrBrurdr>r& zeros_liker[rR full_likera)rCrattention_maskrrsrtr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r+forwardzViltEmbeddings.forwards**m+  595F5Fj4;;;W;W6G6 2L+{%,,Q/K '#$ !D$>$>   ^5::kFXFX Y%  $d&@&@ OOK)=UZZXcXjXj k'  YY \:B  >;7Q?5  r*))r)r"r#r$r%r1rr __classcell__rDs@r+r-r-Ns V9B'!r*r-c*eZdZdZfdZddZxZS)r2zGConstruct the embeddings from word, position and token_type embeddings.c>t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrNF) persistentrr)r0r1rr< vocab_sizer6 pad_token_idword_embeddingsmax_position_embeddingsr;type_vocab_sizer> LayerNormlayer_norm_epsr?r@rAgetattrrregister_bufferr&rfrhr5rrGr[rCrBrDs r+r1zTextEmbeddings.__init__s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r*cT||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |jdk(r|j|} | | z } |j| } |j| } | S)NrNrrrrr)rGrhasattrrrhr&r5r[rRrr>rr;rrA) rCrrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr>rr;s r+rzTextEmbeddings.forwards=  #..*K',,.s3K ^  ,,Q  ^ J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r*)NNNNr"r#r$r%r1rrrs@r+r2r2sQ & r*r2c(eZdZdZfdZdZxZS)r8z# Image to Patch Embedding. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r0r1r]r^r~r6ri collectionsabcIterabler:rConv2drU)rCrBr]r^r~r6r:rDs r+r1zViltPatchEmbeddings.__init__&s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir*c|j\}}}}||jk7r td|jjj }|j|j |}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rWr~ ValueErrorrUrVrrg)rCrsr}r~rr target_dtyperys r+rzViltPatchEmbeddings.forward5si2>2D2D/ L&% 4,, ,w --33 OOLOO,O? @r*rrs@r+r8r8!s jr*r8c,eZdZfdZdZddZxZS)ViltSelfAttentionct||j|jzdk7r3t |ds't d|jfd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r0r1r6num_attention_headsrrrjattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer?attention_probs_dropout_probrArs r+r1zViltSelfAttention.__init__As1     : : :a ?PVXhHi"6#5#5#6"78334A7  $*#=#= #&v'9'9F,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 6G=/2O\M]r*NNF)r"r#r$r1rrrrs@r+rr@sG$% !r*rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rBreturnNct|tj|j|j|_tj |j|_yN) r0r1rrr6denser?r@rArs r+r1zViltSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r*r input_tensorcJ|j|}|j|}|SrrrArCr rs r+rzViltSelfOutput.forwards$ =1  ]3 r*) r"r#r$r%rr1r&Tensorrrrs@r+rr}sD >z>d> U\\RWR^R^r*rc,eZdZfdZdZddZxZS) ViltAttentionct|t||_t ||_t |_yr)r0r1r attentionroutputset pruned_headsrs r+r1zViltAttention.__init__s0 *62$V, Er*c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrH)lenrrrrr rrrrrrrunion)rCheadsindexs r+ prune_headszViltAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r*cj|j||||}|j|d|}|f|ddz}|S)Nrr)rr)rCr rrr self_outputsattention_outputrs r+rzViltAttention.forwardsE~~m^YPab ;;|A F#% QR(88r*r)r"r#r$r1rrrrs@r+rrs" ;$r*rc`eZdZdeddffd Zdej dej fdZxZS)ViltIntermediaterBrNct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r0r1rrr6intermediate_sizerri hidden_actstrr intermediate_act_fnrs r+r1zViltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r*r cJ|j|}|j|}|Sr)rrrCr s r+rzViltIntermediate.forwards& =1 00? r* r"r#r$rr1r&rrrrs@r+rrs19z9d9U\\ellr*rcxeZdZdeddffd Zdej dej dej fdZxZS) ViltOutputrBrNct|tj|j|j |_tj|j|_ yr) r0r1rrrr6rr?r@rArs r+r1zViltOutput.__init__sB YYv779K9KL zz&"<"<= r*r rcT|j|}|j|}||z}|Srrrs r+rzViltOutput.forwards. =1  ]3 % 4 r*rrs@r+rrs?>z>d> U\\RWR^R^r*rc*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.crt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nrr)r0r1chunk_size_feed_forward seq_len_dimrrr intermediaterrrrr6rlayernorm_beforelayernorm_afterrs r+r1zViltLayer.__init__s '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr*c|j|j||||}|d}|dd}||j|jz}|j |}|j |}|j ||}|f|z}|S)N)rrr)rr(rgrRr)r'r) rCr rrrself_attention_outputsrr layer_outputs r+rzViltLayer.forwards!%  ! !- 0  / "0"  2!4(,)=+;+;z&ViltEncoder.forward..%smq_`_lms)last_hidden_stater r!)ror3r4training_gradient_checkpointing_func__call__tupler ) rCr rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r+rzViltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!"#% ! !-]NO]n o )!,M &9]1=M.  - KK   " " $ MM   $ $S ).r*N) r"r#r$r%r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesrRr)r*r+rGrG-s+ L&*#)+>?*r*rGaH This model is a PyTorch `torch.nn.Module `_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_images, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, num_images, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_images, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare ViLT Model transformer outputting raw hidden-states without any specific head on top.c eZdZdfd ZdZdZdZeee e e  dde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede ede edee ee jffdZxZS) ViltModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jyNr)r0r1rBr-rr.encoderrrr6r layernorm ViltPoolerpooler post_init)rCrBadd_pooling_layerrDs r+r1zViltModel.__init__si   (0"6* f&8&8f>S>ST,=j(4  r*cB|jjjSrrr3rrCs r+get_input_embeddingszViltModel.get_input_embeddingss..>>>r*c:||jj_yrrb)rCrs r+set_input_embeddingszViltModel.set_input_embeddingss:?''7r*c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr[r3rr)rCheads_to_pruner3rs r+ _prune_headszViltModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr* output_typerSrrrrsrtrrrrrr>r?rc ~| | n|jj} | | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \}}| |jn |j}|tj||f|}| | t d| | t d||jdn|jd}||k7r t d |Btj||jj|jjf|}|j||jj}|j|||||||| \}}|j|| }|j!|||| | | }|d}|j#|}|j$|j%|nd}| s ||f|d dzSt'|||j(|j* S)a Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerNz5You have to specify either input_ids or inputs_embedsrQzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrr>r?r)r9 pooler_outputr r!)rBrr>use_return_dictr%warn_if_padding_and_no_attention_maskrGrRr&rqrWr] get_head_maskr2rget_extended_attention_maskr[r\r^r r r!)rCrrrrsrtrrrrrr>r?rtext_batch_sizerrRimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputs r+rzViltModel.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U&1#%.%:!! @T@T  !"ZZ/:)FPVWN  # (@ef f  !l&:VW W4@4L<--a0R^RdRdefRg  .`a a  %5t{{7M7Mt{{OeOe$fouvJ&&y$++2O2OP +/??       !5,; , (.150P0PQ_al0m,, 2/!5# ' *!,..98< 8O O4UY #]3oab6II I)-')77&11   r*)T NNNNNNNNNNNN)r"r#r$r1rdrfrjrVILT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr& LongTensorr'rjboolrrrrrs@r+rXrXsy ?@C++@A+ETcd156:594815155948.2,0/3&*p E,,-p !!2!23p !!1!12 p u001 p U--. p E--.p   1 12p u001p 'smp $D>p 'tnp d^p  )51B1B+CC Dp eBp r*rXc$eZdZfdZdZxZS)r]ct|tj|j|j|_tj |_yr)r0r1rrr6rTanh activationrs r+r1zViltPooler.__init__Ys9 YYv1163E3EF '')r*c\|dddf}|j|}|j|}|S)Nr)rr)rCr first_token_tensorrys r+rzViltPooler.forward^s6+1a40 #56  6 r*rErs@r+r]r]Xs $ r*r]zU ViLT Model with a language modeling head on top as done during pretraining. c eZdZddgZfdZdZdZeejde e e  dde ejd e ej d e ejd e ej d e ejd e ej de ej de ej de ejde ede ede edee eej ffdZxZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biasct||t||_t ||_|j yr)r0r1rXrH ViltMLMHead mlm_scorer_rs r+r1zViltForMaskedLM.__init__ps4  f% $V, r*c.|jjSr)rdecoderrcs r+get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingsys~~%%%r*c\||j_|j|j_yr)rrr)rCnew_embeddingss r+set_output_embeddingsz%ViltForMaskedLM.set_output_embeddings|s !/,11r*zbatch_size, sequence_lengthrkrrrrsrtrrrlabelsrr>r?rc F| | n|jj} |j||||||||| | |  } | dd\}}||jdn|jd}|ddd|f|dd|df}}|j |}d}| at }| j |j} ||jd|jj| jd}| s|f| ddz}||f|zS|St||| j| jS)a labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrsrtrrrrr>r?rFrrNrrr r!) rBrorHrWrr rgrRr`rrr r!)rCrrrrsrtrrrrrr>r?rrxry text_seq_len text_featuresrv mlm_logitsmasked_lm_lossloss_fctrs r+rzViltForMaskedLM.forwards]R&1%X156:594815155948-1,0/3&*n E,,-n !!2!23n !!1!12 n u001 n U--. n E--.n   1 12n u001n ))*n $D>n 'tnn d^n  ~uU%6%677 8n Yhn r*rc$eZdZfdZdZxZS)ViltPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yrZ) r0r1rrr6rrirrr transform_act_fnrrrs r+r1z$ViltPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr*cl|j|}|j|}|j|}|Sr)rrrrs r+rz#ViltPredictionHeadTransform.forwards4 =1 --m< }5 r*rErs@r+rrsUr*rc,eZdZdfd ZdZdZxZS)rc|t|||_t||_t j |j|jd|_ t jtj|j|_ |||j_|j|j_ y)NFr)r0r1rBr transformrrr6rrr4r&r5rrV)rCrBrVrDs r+r1zViltMLMHead.__init__s  4V<yy!3!3V5F5FUS LLV->->!?@  "(DLL !II r*c:|j|j_yr)rrrcs r+ _tie_weightszViltMLMHead._tie_weightss II r*cJ|j|}|j|}|Sr)rr)rCrys r+rzViltMLMHead.forwards" NN1  LLOr*r)r"r#r$r1rrrrs@r+rrs &&r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. c eZdZfdZeeeee dde e jde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e d e e de e deeee jffdZxZS)ViltForQuestionAnsweringc t|||j|_t||_t j t j|j|jdzt j|jdzt jt j|jdz|j|_ |jy)NrF) r0r1 num_labelsrXrHr Sequentialrr6rGELU classifierr_rs r+r1z!ViltForQuestionAnswering.__init__"s   ++f% -- IIf((&*<*r?rc | | n|jj} |j||||||||| | |  } | r | jn| d}|j |}d}| K| j |j } tjj|| | jdz}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrFr)rBrorHrnrrgrRrrY binary_cross_entropy_with_logitsrWrr r!)rCrrrrsrtrrrrrr>r?rrnrrrs r+rz ViltForQuestionAnswering.forward3s b&1%S 'tnS d^S  'u/@/@)AA BS cBS r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. c eZdZfdZeeeee dde e jde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e d e e de e deeee jffdZxZS)ViltForImageAndTextRetrievalct||t||_t j |j d|_|jyr/) r0r1rXrHrrr6 rank_outputr_rs r+r1z%ViltForImageAndTextRetrieval.__init__sC  f% 99V%7%7; r*rkrrrrsrtrrrrrr>r?rc D| | n|jj} d} | td|j||||||||| | |  }| r |jn|d}|j |}| s|f|ddz}| | f|zS|St | ||j|jS)a' labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrrFr) rBroNotImplementedErrorrHrnrrr r!)rCrrrrsrtrrrrrr>r?rrrnrrs r+rz$ViltForImageAndTextRetrieval.forwardsZ&1%L 'tnL d^L  'u/@/@)AA BL cBL r*rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. c eZdZfdZeeeee dde e jde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e d e e de e deeee jffdZxZS)"ViltForImagesAndTextClassificationc t|||j|_t||_|j }t jt j|j|z|j|zt j|j|zt jt j|j|z|j|_ |jyr)r0r1rrXrH num_imagesrrrr6rrrr_)rCrBrrDs r+r1z+ViltForImagesAndTextClassification.__init__s   ++f% && -- IIf((:5v7I7IJ7V W LL++j8 9 GGI IIf((:5v7H7H I   r*rkrrrrsrtrrrrrr>r?rc | | n|jj} | | n|jj} | | n|jj} | |jdk(r|j d}| |jdk(r|j d}||j dnd} | ||j dnd} | |jjk7r tdg}| rgnd}| rgnd}t| D]}|j|||||dd|ddddddfnd||dd|ddddfnd||||dd|ddddfnd|dz| | |  }| r |jn|d}|j|| r|j|j| s|j|jtj |d}|j#|}d}| Wt%}| j'|j(} ||j+d|j,| j+d}| s|||f}||f|zS|St/|||| S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrrsrtrrrrrr>r?rNrHr)rBrr>rondim unsqueezerWrrr1rHrnrrr r!r&rarr rgrRr`rr)rCrrrrsrtrrrrrr>r?rpooler_outputsr r!rrrnryrrrrs r+rz*ViltForImagesAndTextClassification.forwards^2C1N-TXT_T_TqTq$8$D $++JjJj &1%'11!4L  # (9(9Q(>'11!4L.:.F\''*D  2>2J++A.PTJ // /n 2 ,R$ z" 6Aii--o 'tno d^o  7u?P?P9QQ Ro sBo r*rz ViLT Model with a token classification head on top (a linear layer on top of the final hidden-states of the text tokens) e.g. for Named-Entity-Recognition (NER) tasks. c eZdZfdZeeeee dde e jde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e d e e de e deeee jffdZxZS)ViltForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jy)NF)r`) r0r1rrXrHrr?r@rArr6rr_rs r+r1z#ViltForTokenClassification.__init__sk   ++f> zz&"<"<= ))F$6$68I8IJ r*rkrrrrsrtrrrrrr>r?rc 4| | n|jj} |j||||||||| | |  } | d}||jdn|jd}|j |}|j |ddd|f}d}| Wt }| j|j} ||jd|j| jd}| s|f| ddz}||f|zS|St||| j| jS)z labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: NrrrrNrFr)rBrorHrWrArr rgrRr`rrr r!)rCrrrrsrtrrrrrr>r?rrxtext_input_sizerrrrs r+rz"ViltForTokenClassification.forwards?0&1%= 'tn= d^=  $eE,=,=&>> ?= `B= r*r)Dr%collections.abcrr dataclassesrtypingrrrrr&torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r rrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_viltr get_loggerr"loggerr|_CHECKPOINT_FOR_DOCrModuler-r2r8rrrrrr#r.rGVILT_START_DOCSTRINGr{4VILT_IMAGES_AND_TEXT_CLASSIFICATION_INPUTS_DOCSTRINGrXr]rrrrrrrr)r*r+rs !// %!. ut*   H %- @{@ @2W!RYYW!t6RYY6r"))>9 9zRYY$BIIF ryy "  # #L2 "))2 j*/*8 5n584pdN #N N b    C )C  C L"))"")), g 2g g T Z #6Z Z z9  D )<D  D N L !4L L r*