sgydZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZdd lmZmZdd lmZmZmZdd lmZmZmZmZmZm Z m!Z!d d l"m#Z#m$Z$m%Z%m&Z&m'Z'ejPe)Z*dZ+dZ,dZ-dZ.dZ/gdZ0dZ1dZ2e e'e%e&fZ3eGddeZ4eGddeZ5eGddeZ6GddejnZ8GddejnZ9Gdd ejnZ:Gd!d"ejnZ;Gd#d$ejnZ<Gd%d&ejnZ=Gd'd(ejnZ>Gd)d*ejnZ?Gd+d,ejnZ@Gd-d.ejnZAGd/d0ejnZBd1ZCd2ZDd3ZEeEeDzZFd4ZGeGeDzZHd5eDzZId6ZJeEeGzeDzeJzZKd7eGzeEzd8zeDzZLd9ZMGd:d;eZNedd?eNZPed@eCjd=GdAdBeNZQedCeCjd=GdDdEeNZRedFeCjdG=GdHdIeNZSGdJdKejnZTGdLdMejnZUGdNdOejnZVedPeCjdQ=GdRdSeNZWGdTdUejnZXGdVdWejnZYGdXdYejnZZGdZd[ejnZ[ed\eCjdG=eMzGd]d^eNZ\y)_zPyTorch FLAVA model.N) OrderedDict) dataclass)AnyDictListOptionalSetTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-fullzfacebook/flava-image-codebookrr r)rg$(~k@ceZdZUdZdZeejed<dZ ee ed<dZ eejed<dZ ee ed<dZ eejed<dZee ed<d eefd Zy) FlavaModelOutputa Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. Args: image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))r(r&r*Ngetattrto_tuple).0kselfs [/var/www/html/venv/lib/python3.12/site-packages/transformers/models/flava/modeling_flava.py z,FlavaModelOutput.to_tuple..bs=  TTDGZabfhiZjZsZsZu u -0tuplekeysr3s`r4r0zFlavaModelOutput.to_tupleas# YY[   )__name__ __module__ __qualname____doc__r%rtorch FloatTensor__annotations__r&rr'r(r)r*r rr0r;r4r$r$Bs,59hu00189=L(56=37OXe//078Bx :;B %* r;r$ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<d efd Zy) FlavaLossesa"Class representing pretraining losses from FLAVA model Args: mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.: Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.: Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.: Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.: Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.: Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.: Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr+cBd}|jD] }|d}|S|S)NTF)values)r3all_nonevs r4rNzFlavaLosses.all_nones5 A}   r;)r<r=r>r?rFrr@rArBrGrHrIrJrKboolrNrCr;r4rErEhs&(,C%## $+'+C%## $+'+C%## $+6:!2!23:-1Ix))*1,0Hhu(()0$r;rEceZdZUdZdZeejed<dZ e ed<dZ eejed<dZ ee ed<dZeejed<dZee ed<dZeejed <dZee ed <dZeejed <dZee ed <dZeejed <dZee ed<dZeejed<dZee ed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dee fdZ!y)FlavaForPreTrainingOutputa Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. Args: loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. Nloss loss_infor%r&r'r(r)r*image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr+cTgdtfdjDS)N)r(r&r*rXrVrZc3dK|]'}|vr|nt|j)ywNr.)r1r2r3transformer_outputss r4r5z5FlavaForPreTrainingOutput.to_tuple..s4sbc)< r?rSrr@rArBrTrEr%r&rr'r(r)r*rUrVrWrXrYrZr[r\r]r^r_r`rar rr0rCr;r4rRrRs>@)-D(5$$ %,!I{!48hu00189=L(56=37OXe//078Bx :;B;?Xe&7&78?@D"<=D:>HU%6%67>?C!;<C@D (5+<+<"=DEIh'ABI.2J**+2.2J**+2.2J**+2@D (5+<+<"=D?C%*;*;!<C48hu001837OXe//07 t%* tr;rRc eZdZdZddededdffd Zdejde d e dejfd Z dd ejd e ejd edejfdZ xZS)FlavaImageEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configuse_mask_tokenr+Nct||xs |j}tjt j dd|j|_|r4tjt j dd|jnd|_t|j|j|j|j|_ |jj}tjt j d|dz|j|_tj |j"|_|j|_ ||_y)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr@zeros hidden_size cls_tokenPatchEmbeddingsrkrlrmpatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutrh)r3rhrirx __class__s r4rpzFlavaImageEmbeddings.__init__s '<6+<+<ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei /((((,,(( !  ++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ r; embeddingsheightwidthc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr bicubicF)sizemode align_cornersdim)shaperyr@jit is_tracingrlrreshapepermuter functional interpolateviewcat) r3r~rrrx num_positionsclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r4interpolate_pos_encodingz-FlavaImageEmbeddings.interpolate_pos_encoding s`!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr; pixel_valuesbool_masked_posrcV|j\}}}}|j||}|j\}} } ||jj || d} |j dk(r!|j |jdd}|jdj| } |d| z z| | zz}|jj |dd} tj| |fd}|r||j|||z}n||jz}|j|}|S)N)rrr r?rr)rrwrrqexpandrr unsqueezetype_asrur@rrryr|)r3rrr batch_sizermrrr~seq_len_ mask_tokensmask cls_tokenss r4forwardzFlavaImageEmbeddings.forward3s4 3?2D2D/ L&%** YY J7Q?  $#d&C&CJPVX]&^^J#d&>&>>J\\*- r;FNF)r<r=r>r?rrPrpr@Tensorintrr BoolTensorr __classcell__r}s@r4rgrgs/RV&&D5<<&D&DUX&D]b]i]i&DV7;). ll"%"2"23#'   r;rgc eZdZdZ d dedeeeeeffdedeffd Zd dejde d ejfd Z xZ S) rvz# Image to Patch Embedding. rkrlrmrncVt|t|tjj s||f}t|tjj s||f}|d|dz|d|dzz}||_||_||_tj|||||_ y)Nrr) kernel_sizestride) rorp isinstance collectionsabcIterablerkrlrxr Conv2d projection)r3rkrlrmrnrxr}s r4rpzPatchEmbeddings.__init__\s *koo&>&>?$j1J*koo&>&>?$j1J!!} 1 5*Q-:VW=:XY $$&))L)\fgr;rrr+c 8|j\}}}}|sV||jdk7s||jdk7r2td|d|d|jdd|jdd |j|j dj dd}|S)NrrzInput image size (*z) doesn't match model (z).r)rrk ValueErrorrflatten transpose)r3rrrrmrrxs r4rzPatchEmbeddings.forwardos2>2D2D/ L&%'++u8J/J (%9+,Adooa.@-AE OOL ) 1 1! 4 > >q! Dr;)r r"r) r<r=r>r?rr r rpr@rrPrrrs@r4rvrvWs} 24 hh#uS#X./h h  h& ELL D ]b]i]i r;rvceZdZdZfdZ ddeejdeejdeejfdZxZ S)FlavaTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c>t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrF) persistenttoken_type_ids)dtype)rorpr Embedding vocab_sizert pad_token_idword_embeddingsmax_position_embeddingsrytype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrzr{r|r/rregister_bufferr@arangerrsrrlongr3rhr}s r4rpzFlavaTextEmbeddings.__init__~s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r; input_idsrrc$|j}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}|j|}|j|} || z} |jdk(r|j|} | | z } |j| } |j| } | S)Nrrr)rdevicer)rrhasattrrrr@rsrrrrrryrr|) r3rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrr~rys r4rzFlavaTextEmbeddings.forwards  nn&  ^  ,,Q  ^ J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r;)NNN) r<r=r>r?rprr@rrrrs@r4rr{sRQ *-115/3 ELL) !. u||, r;rc"eZdZdeddffd Zdej dej fdZ d dej deej d eej d e de e ej ej fe ej ff d Z xZ S) FlavaSelfAttentionrhr+Nct||j|jzdk7r3t |ds't d|jfd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)rorprtnum_attention_headsrrrattention_head_size all_head_sizer Linearqkv_biasquerykeyvaluerzattention_probs_dropout_probr|rs r4rpzFlavaSelfAttention.__init__s1     : : :a ?PVXhHi"6#5#5#6"78334A7  $*#=#= #&v'9'9FS"S* **,CD 6G=/2O\M]r;NNF)r<r=r>FlavaPossibleConfigsrpr@rrrrPr r rrrs@r4rrsG3GG$%ell%u||%26,0"' (||(!.(ELL) (  ( uU\\5<</0% 2EE F (r;rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) FlavaSelfOutputz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. rhr+Nct|tj|j|j|_tj |j|_yrd) rorpr rrtdenserzr{r|rs r4rpzFlavaSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r;r input_tensorcJ|j|}|j|}|Srdrr|r3rrs r4rzFlavaSelfOutput.forwards$ =1  ]3 r;) r<r=r>r?r rpr@rrrrs@r4r r sE >3>> U\\RWR^R^r;r ceZdZdeddffd ZdeeddfdZ d dejde ejd e ejd e de e ejejfe ejff d ZxZS) FlavaAttentionrhr+Nct|t||_t ||_t |_yrd)rorpr attentionr outputset pruned_headsrs r4rpzFlavaAttention.__init__ s0 +F3%f- Er;headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrrrrrrrrrrrunion)r3rindexs r4 prune_headszFlavaAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r;rrrrcl|j||||}|j|d|}|f|ddz}|SN)rrrrr)rr)r3rrrr self_outputsattention_outputr s r4rzFlavaAttention.forward"sQ~~ .Iar&  ;;|A F#% QR(88r;r )r<r=r>r rpr rr r@rrrPr r rrrs@r4rr s"3"" ;S;d;*26,0"' ||!.ELL)    uU\\5<</0% 2EE F r;rc`eZdZdeddffd Zdej dej fdZxZS)FlavaIntermediaterhr+Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yrd) rorpr rrtintermediate_sizerr hidden_actstrrintermediate_act_fnrs r4rpzFlavaIntermediate.__init__4s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r;rcJ|j|}|j|}|Srd)rr+r3rs r4rzFlavaIntermediate.forward=s& =1 00? r; r<r=r>r rpr@rrrrs@r4r&r&3s29399U\\ellr;r&cxeZdZdeddffd Zdej dej dej fdZxZS) FlavaOutputrhr+Nct|tj|j|j |_tj|j|_ yrd) rorpr rr(rtrrzr{r|rs r4rpzFlavaOutput.__init__EsB YYv779K9KL zz&"<"<= r;rrcT|j|}|j|}||z}|Srdrrs r4rzFlavaOutput.forwardKs. =1  ]3 % 4 r;r.rs@r4r0r0Ds@>3>> U\\RWR^R^r;r0ceZdZdZdeddffd Z d dejdeejdeejd e de e ejejfe ejff d Z xZ S) FlavaLayerz?This corresponds to the Block class in the timm implementation.rhr+Ncrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_yNrr)rorpchunk_size_feed_forward seq_len_dimrrr& intermediater0rr rrtrlayernorm_beforelayernorm_afterrs r4rpzFlavaLayer.__init__Ws '-'E'E$'/-f5!&) !# V-?-?VEZEZ [!||F,>,>FDYDYZr;rrrrc|j|j||||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|Sr")rr:r;r9r) r3rrrrself_attention_outputsr$r  layer_outputs r4rzFlavaLayer.forwardcs"&  ! !- 0)/ "0"  2!4(,)=8 ++M: ((6 {{<? /G+r;r )r<r=r>r?r rpr@rrrPr r rrrs@r4r4r4TsI [3 [ [26,0"' ||!.ELL)    uU\\5<</0% 2EE F r;r4ceZdZdeddffd Z d dej deej deej ded ed ede e e ffd Z xZ S) FlavaEncoderrhr+Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rorprhr ModuleListrangenum_hidden_layersr4layergradient_checkpointing)r3rhrr}s r4rpzFlavaEncoder.__init__sN  ]]fF^F^@_#`1Jv$6#`a &+#$asA#rrrroutput_hidden_states return_dictcx|rdnd}|rdnd}t|jD]j\} } |r||fz}||| nd} |jr,|jr |j | j ||| |} n | ||| |} | d}|sb|| dfz}l|r||fz}|st d|||fDSt|||S)NrCrrc3&K|] }|| ywrdrC)r1rOs r4r5z'FlavaEncoder.forward..smq_`_lm)last_hidden_stater attentions) enumeraterErFtraining_gradient_checkpointing_func__call__r8r) r3rrrrrGrHall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r4rzFlavaEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!"#% ! !-]NO]n o )!,M &9]1=Mrrpr@rrrPr r8rrrrs@r4r@r@s,{,t,26,0"'%* ) ||) !.) ELL) )  ) # ) )  uo% &) r;r@cDeZdZdeffd Zdej fdZxZS) FlavaPoolerrhct|tj|j|j|_tj |_yrd)rorpr rrtrTanh activationrs r4rpzFlavaPooler.__init__s9 YYv1163E3EF '')r;rc\|dddf}|j|}|j|}|SNr)rr\)r3rfirst_token_tensor pooled_outputs r4rzFlavaPooler.forwards6+1a40 #56  6 r;r.rs@r4rYrYs $3$ U\\r;rYaD This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`{config}`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a; Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`FlavaImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). interpolate_pos_encoding (`bool`, *optional*): Whether to interpolate the pre-trained position encodings. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) z Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. z Args: skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. a Args: input_ids_masked (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) a image_attention_mask (`torch.FloatTensor` of shape `({1})`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. z Parameters: image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise. it will be initialized using the image_codebook_config defined in the config first as the first parameter. cteZdZdZeZdZdZdee je je jfddfdZ y)FlavaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. flavaTmoduler+Nc"t|tjtjfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyy)zInitialize the weightsg)meanstdNr)rr rrweightdatanormal_rhinitializer_rangerzero_rrrfill_)r3rds r4 _init_weightsz"FlavaPreTrainedModel._init_weights^s fryy"))4 5 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r;)r<r=r>r?r config_classbase_model_prefixsupports_gradient_checkpointingr r rrrrnrCr;r4rbrbTsF L&*#*E"))RYY *L$M*RV*r;rbzeThe bare FLAVA Image Model transformer outputting raw hidden-states without any specific head on top.)rhceZdZeZdZdZddedeffd Zde jfdZ de jfd Z d e eeefdd fd Zeej'd eeeede ddeej6deej8deedeej6deej6deedeedeedeeeffdZxZ S)FlavaImageModelzflava.image_modelrrhadd_pooling_layerct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jyNr)rorprhrgr~r@encoderr rrtr layernormrYpooler post_initr3rhrtr}s r4rpzFlavaImageModel.__init__ysg   .v6#F+ f&8&8f>S>ST->k&)D  r;r+c.|jjSrdr~rwr:s r4get_input_embeddingsz$FlavaImageModel.get_input_embeddingss///r;rc&||j_yrdr}r3rs r4set_input_embeddingsz$FlavaImageModel.set_input_embeddingss+0(r;heads_to_pruneNc|jD]7\}}|jj|jj |9yz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel NitemsrwrErr r3rrErs r4 _prune_headszFlavaImageModel._prune_headsE +002 CLE5 LL  u % / / ; ;E B Cr;batch_size, image_num_patchesvision) checkpoint output_typeromodalityexpected_outputrrrrrrGrHc "||n|jj}||n|jj}||n|jj}| t d|j ||jj }|j|||} |j| |||||} | d} |j| } |j|j| nd} |s | | f| ddzSt| | | j| jS)Nz You have to specify pixel_values)rrrrrrGrHrrrL pooler_outputrrM)rhrrGuse_return_dictr get_head_maskrDr~rwrxryrrrM) r3rrrrrrrGrHembedding_outputencoder_outputssequence_outputr`s r4rzFlavaImageModel.forwards8&2C1N-TXT_T_TqTq$8$D $++JjJj &1%rrorpmain_input_namerPrpr Moduler~rrrrrrFLAVA_IMAGE_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr!_CONFIG_CLASS_FOR_IMAGE_MODEL_DOC_EXPECTED_IMAGE_OUTPUT_SHAPErr@rrr r8rrrs@r4rsrsosb $L+$O / D 0bii01"))1C4T#Y+?CDC++G+N+NOn+op&.64 046:3715,0,0/3&*3 u||,3 "%"2"233 #+4. 3 !. 3 ELL) 3 $D>3 'tn3 d^3  u00 13 q3 r;rszdThe bare FLAVA Text Model transformer outputting raw hidden-states without any specific head on top.ceZdZeZdZddedeffd ZdefdZ de jfdZ d e eeefdd fd Zeej'd eeee  ddeej4deej4deej4deej4deej4deedeedeedeeeffdZxZS)FlavaTextModelzflava.text_modelrhrtct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jyrv)rorprhrr~r@rwr rrtrrxrYryrzr{s r4rpzFlavaTextModel.__init__sg   -f5#F+ f&8&8f>S>ST->k&)D  r;r+c.|jjSrdr~rr:s r4r~z#FlavaTextModel.get_input_embeddingss...r;rc&||j_yrdrrs r4rz#FlavaTextModel.set_input_embeddingss*/'r;rNc|jD]7\}}|jj|jj |9yrrrs r4rzFlavaTextModel._prune_headsrr;batch_size, text_seq_lengthrrrorrrrrrrGrHc ||n|jj}||n|jj}||n|jj}| t d|j } |!t j| |j}|j||jj}|j|| |j} |j|||} |j| | ||||} | d} |j| } |j|j| nd}|s | |f| ddzSt!| || j"| j$S)NzYou have to specify input_idsr)rrrrrrr)rhrrGrrrr@onesrrrDget_extended_attention_maskr~rwrxryrrrM)r3rrrrrrrGrHrextended_attention_maskrrrr`s r4rzFlavaTextModel.forwards"2C1N-TXT_T_TqTq$8$D $++JjJj &1%r rorprPrprvr~r rrrrrrrFLAVA_TEXT_INPUTS_DOCSTRINGrrrr _CONFIG_CLASS_FOR_TEXT_MODEL_DOCrr@rr r8rrrs@r4rrsU #L*  4 /o/0"))0C4T#Y+?CDC++F+M+MNk+lm&.5-11515/3,0,0/3&*= ELL)= !.= !. = u||, = ELL) = $D>= 'tn= d^=  u00 1= n = r;rzjThe bare FLAVA Multimodal Model transformer outputting raw hidden-states without any specific head on top.c6eZdZeZdZdZddeffd Zdee e e fddfdZ e e jd eeee  ddej(d eej(d eej(d eedeedeedeeeffdZxZS)FlavaMultimodalModelzflava.multimodal_modelrrhct||||_|jj|_|jr9t j t jdd|j|_ t||_ t j|j|j|_|r t|nd|_|j#yr6)rorprh use_cls_tokenr rrr@rsrtrur@rwrrrxrYryrzr{s r4rpzFlavaMultimodalModel.__init__Es   ![[66   \\%++aFS>ST->k&)D  r;rr+Nc|jD]7\}}|jj|jj |9yrrrs r4rz!FlavaMultimodalModel._prune_headsSrr;,batch_size, image_num_patches + text_seq_lenrrrrrGrHc||n|jj}||n|jj}||n|jj}|j \}}} |j r;|j j|dd} tj| |fd}|dz }|#tj||f|j}|j||jj}|j|||f|j} |j|| ||||} | d} |j!| } |j"|j#| nd}|s | |f| ddzSt%| || j&| j(S)Nrrrrrrr)rhrrGrrrrurr@rrrrrDrrwrxryrrrM)r3rrrrrGrHrrrrrrrr`s r4rzFlavaMultimodalModel.forward[s"2C1N-TXT_T_TqTq$8$D $++JjJj &1%rrorprrprrrrr!FLAVA_MULTIMODAL_INPUTS_DOCSTRINGrrrr&_CONFIG_CLASS_FOR_MULTIMODAL_MODEL_DOCr@rrrPr r8rrrs@r4rr;s )L0%O 4 C4T#Y+?CDC+)001_` &.;26,0,0/3&*7 ||7 !.7 ELL) 7 $D> 7 'tn 7 d^7  u00 17 7 r;rz_The bare FLAVA Model transformer outputting raw hidden-states without any specific head on top.rceZdZeZdeffd Zeejd dde e jde e jde e jde e jde e d e e d e e d e jfd Zeejd  dde e jde e j de e de e jde e jde e d e e d e e d e jfdZeejdeee dde e j*de e jde e jde e jde e jde e j*de e jde e de e d e d e e d eeeffdZxZS) FlavaModelrhct||t|jts"t dt |jdt|jts"t dt |jdt|jts%t ddt |jdz|j}|j}|j}|j|_ |j|_ |j|_|j|_t!||_t%||_t)||_t-j.|j|j|_t-j.|j|j|_t-j4t7j8|j:j<|_t-j.|j|j|_ t-j.|j|j|_!|jEy)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#rorpr text_configr TypeErrortype image_configrmultimodal_configrprojection_dimrttext_hidden_sizeimage_hidden_sizemm_hidden_sizer text_modelrs image_modelrmultimodal_modelr rimage_projectiontext_projectionrrr@tensorrhlogit_scale_init_value logit_scaleimage_to_mm_projectiontext_to_mm_projectionrz)r3rhrrrr}s r4rpzFlavaModel.__init__s  &,,o>++,-Q0  &--/?@,,-.a1  &224IJ_V%=%= >?qAB  (( ** "44$33 + 7 7!-!9!9/;;(5*<8 45F G " $*@*@$BUBU V!yy)>)>@S@ST<< T[[5W5W(XY&(ii0F0FH[H[&\#%'YYt/D/DdFYFY%Z" r;rrrrrrrGrHr+c djt|j|||||||}|d} |j| } | S)Na Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`FlavaTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ```)rrrrrrGrHr)rrrr) r3rrrrrrGrH text_outputsr` text_featuress r4get_text_featureszFlavaModel.get_text_featuress] "v)*))%/!5#' %Q ,,]; r;rrrrrc djt|j||||||||} | d} |j| } | S)Na Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`FlavaImageModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```)rrrrrrGrrHr)rrrr) r3rrrrrrrGrH image_outputsr`image_featuress r4get_image_featureszFlavaModel.get_image_featuressc *v)*((%+)/!5%=#)  &a( ..}=r;rrroimage_attention_maskskip_multimodal_encoderc | | n|jj} | s tdd} d} d}d}|5|j|||| | | }|d|d} } |j | d}d}d}d}d}|6|j ||||| | | }|d|d}}|j |d}d}d}|||s|g|j\}}}|jjr|dz }tj|||j }tj||gd }nd}tj||gd }|j||| }|d}| s| |||||fSt| ||||| S) a  Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrrGrHrrr)rrrrrrGrHrrr)rrH)r%r&r'r(r)r*)rhrHrrrrrrrrr@rrrr$)r3rrrrrrrrrrGrHr% image_statesimage_mm_projectionr&r' text_statestext_mm_projectionr(r)r*rrrattention_mask_imageattention_multimodalmultimodal_inputs r4rzFlavaModel.forward-s j&1%;q>[O!%!;!;KO!L  $   */A/MVm))<)B)B& GQ((66qLG',zz*gNaNhNh'i$',yy2F1W]^'_$'+$$yy*=?Q)RXYZ  $ 5 5 1ES^!6! %6a$8 ! %!   -%+#"7/   r;)NNNNNNNr) NNNNNNNNNTN)r<r=r>rrorprrrrr@rrPrArrrrFLAVA_MODEL_INPUTS_DOCSTRINGrr$ LongTensorr r r0rrrs@r4rrs L){)V++F+M+MNk+lm-11515/3,0/3&*)ELL))!.)!. ) u||, ) $D> )'tn)d^)   )n)V++G+N+NOn+op046:3715,0,0/3&*/u||,/"%"2"23/#+4. / !. / ELL) /$D>/'tn/d^/   /q/b+$++,Z[+;+V1548151526377;26,0%)&*z E,,-z u001z !. z !. z "%,,/ z u//0z 'u||4z "*$z $D>z #z d^z  uk! "z Wz r;rc`eZdZdedeffd Zdej dej fdZxZS)FlavaImageCodebookResPathin_sizeout_sizec t||dz}t}tj|d<tj ||dd|d<tj|d<tj ||dd|d<tj|d <tj ||dd|d <tj|d <tj ||dd |d <tj ||_y)Nrelu_1r rrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)rorprr ReLUr Sequentialpath)r3rrkwargshid_sizerr}s r4rpz"FlavaImageCodebookResPath.__init__s q=}X7H!QOXX8X1aPXX8X1aPXX8X1aPXMM$' r;rr+c$|j|Srd)rr3rs r4rz!FlavaImageCodebookResPath.forwardsyy|r; r<r=r>rrpr@rrrrs@r4rrs1((s( %,,r;rcdeZdZdededeffd Zdej dej fdZxZS)FlavaImageCodebookBlockrr num_layersc t|d|dzz |_||k7rtj||dd|_ntj |_t|||_y)Nrrrr) rorp post_gainr rid_pathIdentityrres_path)r3rrrrr}s r4rpz FlavaImageCodebookBlock.__init__sW j!m, h 99WhAqQDL;;=DL1'8D r;rr+cb|j||j|j|zzSrd)rrrrs r4rzFlavaImageCodebookBlock.forwards'||A$--2B!BBBr;rrs@r4rrs? E Es E ECC%,,Cr;rc neZdZd dededededef fd Zdejdejfd ZxZ S) FlavaImageCodebookLayerGroup num_blocksrrruse_poolc$t|t}t|D]4}|dk(rt ||||d|dz<t ||||d|dz<6|rt j d|d<t j||_y)Nrblock_rr)rpool) rorprrCrr MaxPool2drgroup) r3r rrrr blocksrTr}s r4rpz%FlavaImageCodebookLayerGroup.__init__s z" aAAv)@(T^)_!u~&)@8U_)`!u~&  a \\a8F6N]]6* r;rr+c$|j|Srd)rrs r4rz$FlavaImageCodebookLayerGroup.forwardszz!}r;r) r<r=r>rrPrpr@rrrrs@r4rrsH +3 +C +# +QT +`d +%,,r;ra" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. rceZdZdZeZdZdZdedeffd Z de jde jfdZ de jde jfd Z de jde jfd ZxZS) FlavaImageCodebookrFrhrc t||||_|j|_|j|_|j |_|j |_|j|_|j|j z}t}tj|d<tjd|j z|jdd|d<t}tj|jd|j zdd|d <t|j |d|j zd|j z|d <t|j |d|j zd |j z|d <t|j |d |j zd |j z|d<t|j |d |j zd|j zd|d<tj||d<tj||_|j|jj r|j#D] }d|_ yy)Nrelurrrconvr inputgroup_1rgroup_2rgroup_3F)r group_4r)rorprh num_groupsinput_channelsnum_blocks_per_grouprtrrr rrrrrrzfreeze parameters requires_grad)r3rhrr output_blocksrparamr}s r4rpzFlavaImageCodebook.__init__s    ++$33$*$?$?!!-- ++__t'@'@@ # " f " !d.>.>*>]^hi j f))D$7$7T=M=M9M[\fghw8  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I]hm y==7xmmF+   ;;  * ,&+# , r;r+c|djt|j|}tj|dS)Na Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("{0}") >>> image_processor = AutoImageProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)r_CHECKPOINT_FOR_CODEBOOK_DOCrr@argmaxr3rz_logitss r4get_codebook_indicesz'FlavaImageCodebook.get_codebook_indices"s3 . F/ 0;;|,||H1--r;c\|j|}tjd|S)Nrr)rr Softmaxr+s r4get_codebook_probsz%FlavaImageCodebook.get_codebook_probs>s&;;|, rzza **r;c8djtt|jdk7rt d|jd|jd|j k7r(t d|jdd|j |j |S)Na Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("{0}") >>> image_processor = AutoImageProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )rr)rrrr r)r3rs r4rzFlavaImageCodebook.forwardBs 4 F/ 0 |!! "a '|L,>,>+?zJK K   a D$7$7 7z,*<*rprrorrqrrpr@rr-r0rArrrs@r4rrs+L$O&+#*,(*,*,X..%,,.8+u||+ + )E$5$5 )%,, )r;rc$eZdZfdZdZxZS)FlavaPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yrv) rorpr rrtrrr)r*rtransform_act_fnrrrs r4rpz%FlavaPredictionHeadTransform.__init__fs{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr;cl|j|}|j|}|j|}|Srd)rr5rr-s r4rz$FlavaPredictionHeadTransform.forwardos4 =1 --m< }5 r;r<r=r>rprrrs@r4r3r3esUr;r3c,eZdZdfd ZdZdZxZS)FlavaMaskedPredictionHeadc|t|||_t||_t j |j|jd|_ t jtj|j|_ |||j_|j|j_ y)NFr)rorprhr3 transformr rrtrdecoderrrr@rsrrh)r3rhrhr}s r4rpz"FlavaMaskedPredictionHead.__init__ws  5f=yy!3!3V5F5FUS LLV->->!?@  "(DLL !II r;c:|j|j_yrd)rr<r:s r4 _tie_weightsz&FlavaMaskedPredictionHead._tie_weightss II r;cJ|j|}|j|}|Srd)r;r<rs r4rz!FlavaMaskedPredictionHead.forwards" NN1  LLOr;rd)r<r=r>rpr>rrrs@r4r9r9vs &&r;r9c$eZdZfdZdZxZS) FlavaITMHeadct|||_t||_t j |jd|_y)Nr) rorprhrYryr rrtseq_relationshiprs r4rpzFlavaITMHead.__init__s:  !&) " &*<*#ii(;< <<(8:M:W:WXY[\:]^all,,8L8V8VWXZ[8\]`kk&88(f&es 9J$J r7rs@r4rFrFsN 9r;rFzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c)eZdZgdZddedeejffd Zde jfdZ e e jddeee  dd ee j"d ee j"d ee j$d ee j$dee jdee jdee jdee j"dee jdedee jdee jdee jdeededeedeedeee jeff$dZxZS)FlavaForPreTraining)zmmm_text_head.decoder.biaszmmm_image_head.decoder.biaszmlm_head.decoder.biaszmim_head.decoder.biasrhimage_codebookcbt||t||_||_|j&|j rt |j|_t|j|_ t|j|_ t||_t|j|_t|j|_t#||_|jj&|_|jj&|_|j,|_|j.|_|j0|_|j2|_|j4|_|j6|_|j8|_|j:|_|j=yrd)rorprrcr\ init_codebookrimage_codebook_configr9rmim_headrmlm_headrAitm_headmmm_image_head mmm_text_headrFglobal_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderrz)r3rhr\r}s r4rpzFlavaForPreTraining.__init__sO  ' ,    &6+?+?"4V5Q5Q"RD 2&2E2EF 1&2D2DE $V, 78K8KL6v7I7IJ'A&'I$ & 3 3 > >%11<< ++ ++)/)I)I&%55 ++ & 7 7%55060W0W- r;rcn|jdkDr!|j|jdd}|S)Nrrr)rrrrs r4 _resize_to_2dz!FlavaForPreTraining._resize_to_2ds, 557Q;qvvay"%Ar;zbatch_size, text_seq_lenrrrinput_ids_maskedrcodebook_pixel_valuesrrrrrro mlm_labels mim_labels itm_labelsrrGrH return_lossr+c||n|jj}||n|jj}| | n |j} ||tj d|}|j |||||| | ||d }|j ||||| |||d }d}|j}|j}|j}|j}|j}dx}x}x}x}x}x}} dx}!x}"x}#}$dx}%x}&}'||C| A|r?|j td| td|jj|} |jdkDr| ||}(| |j| } |j|}|j | |j#d<|(dd| j%d  dddf}(| j#|j })| |)}*|(|)ddf}(|j'|(}!|rjt(j*j-|!j/d |j0|*j/d }||jz}n|j'|(}!|j2dkDr|||}+| |j| } |+dd| j%d  dddf}+| j#|j })| |)},|+|)ddf}+|j5|+}"|rjt(j*j-|"j/d |j6|,j/d }||j2z}n|j5|+}"|j8dkDr||j;|}%| | j#d}-t=j>|-jA|-|-jCdg}|r/t(j*j-|%| } | |j8z} |||}| | |} | | |} ||}||jDdkDr|}(|j%d d z }.|(ddd d |.zddf}(| |j| } |j|}|j | |j#d<| j#|j })| |)}*|(|)ddf}(|jG|(}$|rjt(j*j-|$j/d |j0|*j/d }||jDz}n|jG|(}$||jHdkDr|}+|+dd|j%d  dddf}+| |j| } | j#|j })| |)},|+|)ddf}+|jK|+}#|rjt(j*j-|#j/d |j6|,j/d }||jHz}n|jK|+}#|l|i|jLdkDrY|j jO|dddddf}/t(j*jQ|/d }/|j jS|dddddf}0t(j*jQ|0d }0|j jTjVjYtZt\|j_|0|/|j jT\}&}'}1||&|}&|'|}'|1|}1|rWt(j*j-|&|1}2t(j*j-|'|1}3|2|3zd z }||jLz}ta||| ||| }4|r0|4jcs ted|4jgD}|s.||jh|jhjknd||jl|jljknd|j|jn|jnjknd||jh|jhjknd||jl|jljknd||jn|jnjknd|!|"|%|&|&|$|#f}5|r|4jcs||4f|5z}5tqd|5DStsd%id|d|4d|d|jhd|d|jld|jd|jnd|d|jhd|d|jld|d|jnd|!d|"d |%d!|&d"|'d#|$d$|#S)&ai Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Return: Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrrGrH) rrrrrrrrGrHz`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrrrr)rFrGrHrIrJrKc3(K|] }||nd ywr^rC)r1rSs r4r5z.FlavaForPreTraining.forward..s_T%5T1<_sc3&K|] }|| ywrdrC)r1rs r4r5z.FlavaForPreTraining.forward.. s8qai8rKrSrTr%r&r'r(r)r*rUrVrWrXrYrZr[r\r]r^r_r`rarC):rhrrwrologgerwarningrcr%r'r)r\ RuntimeErrorrr-rirqrknerr`r r cross_entropyrrfrhrargrlrbr@whereanynewrmrcrnrdrjr normalizerrriclamp_LOGIT_SCALE_CLAMP_MINLOGIT_SCALE_CLAMP_MAXrerErNsumrMr&r0r(r*r8rR)6r3rrrrrsrrrrrrortrurvrrGrHrw flava_outputflava_masked_outputpos_maskr%r'rUrWrY total_lossmim_lossmlm_loss mmm_text_lossmmm_image_lossgc_lossitm_lossr[r\rar`r]rXrYsequence_for_image masked_tokensmim_labels_filteredsequence_for_textmlm_labels_filtered pos_pairs end_indextext_embeddingimage_embedding gc_labels gc_loss_image gc_loss_text flava_lossesrs6 r4rzFlavaForPreTraining.forwards p&1%eGV^GKK KZK/4D:>> >% # .2N2Z!k&&.&; )0$Y"00EEF[\  ??Q #:#FKgKo!8 %!// ; "&"4"4_"E7;7K7K ?--d34%7JOOA !}}::"D,A,ABDWD\D\]_D` H/H!]]+=>  ??Q #9#EJfJn 6 %!// ; $5a*//!:L9L9NPQ6Q$R! * d.B.B C &0&?#$5mQ6F$G!!]]+<= !}}::"D,@,@ACVC[C[\^C_ H/H!]]+<=  ??Q #?#K'CDJ%&MM!,  ;;y}} 9==RVQWCXY!}}:::zRH/H/;3OPX3Y0)!+H!5J)!+H!5J&5h&?O ( 38M8MPQ8Q!= /44Q7!;I!3Aq1y=7H!4K!L %!// ; "&"4"4_"E7;7K7K ?--d34 * d.B.B C &0&?#%7 q8H%I"#'#6#67I#J %']]%@%@(--b$2G2GHJ]JbJbceJf&N#d&;&;;N#'#6#67I#J  ( 38L8Lq8P <  1!6L6Q6QRS6T5T5VXY2Y Z %!// ; * d.B.B C &0&?#$5mQ6F$G!"&"4"45F"G$&MM$?$?',,R1E1EFH[H`H`acHd%M"T%9%99M"&"4"45F"G  'O,GDLjLjmnLn!ZZ771a8PQN]]44^4LN"jj99:J1aQR7:STO mm55o25NO JJ " " ' ' . ./DF[ \;?;W;W1G1G< 8 oy ##3H#= "1(";%h/  " ; ;
  • Q>]>]>i#//88:os,&88D$55>>@   +F.<#8#8#:  8F88 8(  " . &22  ,  %00  #/"D"D +<< %< !4 @ @ $:  3>> *F &9%J%J " "! ""# $*:% &)8' (.) *,+  r;rd)NNNNNNNNNNNNNNTNN)r<r=r>_tied_weights_keysrrr rrpr@rrqr"FLAVA_PRETRAINING_INPUTS_DOCSTRINGrrrRrrArPr r rrrs@r4r[r[s{HRYYA #!A "d^#A $d^%A & uU\\"$== >'A `A r;r[)]r?rrr dataclassesrtypingrrrrr r r r@torch.utils.checkpointr activationsrmodeling_outputsrrmodeling_utilsrrrutilsrrrrrrrconfiguration_flavarrrrr get_loggerr<r{rr)rrrrrrr r$rErRrrgrvrrr rr&r0r4r@rYFLAVA_START_DOCSTRINGFLAVA_INPUTS_DOCSTRING_COMMON!FLAVA_IMAGE_INPUTS_DOCSTRING_BASEr FLAVA_TEXT_INPUTS_DOCSTRING_BASErr!FLAVA_MODEL_INPUTS_DOCSTRING_BASErr'FLAVA_PRETRAINING_START_DOCSTRING_EXTRArbrrsrrrrrrrr3r9rArFr[rCr;r4rs^ #!??? !Kcc   H %+ ?$6!#4 )@&,_.>@UUV " {"  " J !+! !H `t `t `tJ_299_H!bii!H6"))6r@@Fbii$'RYY'T " "))  ++\0 2990 f "))  !0 %! AC`` $ ?A^^ $ $"%!&&'#$((' '( (*V$W+$#^+'*?*6k  (: ;\ *\ \ ~j  (9 :b )b b Jp  (? @[ /[ [ |e   6J %J J Z *CbiiC"299(   (B C r)-r)r)j299" , 299 %9%9P   69``  q .q  q r;