sgIBdZddlZddlmZddlmZmZddlmZmZddl m Z ddl m Z m Z mZmZej eZd ZGd d ej(Zd Zd Ze deGddej(e Ze deeGddej(Zy)zPyTorch MMBT model.N)nn)CrossEntropyLossMSELoss)BaseModelOutputWithPoolingSequenceClassifierOutput)ModuleUtilsMixin)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings MMBTConfigc*eZdZdZfdZddZxZS)ModalEmbeddingszPGeneric Modal Embeddings which takes in an encoder, and a transformer embedding.cnt|||_||_t j |j |j|_|j|_ |j|_ |j|_ |j|_ t j|j|_y)N)p)super__init__configencoderrLinearmodal_hidden_size hidden_sizeproj_embeddingsposition_embeddingstoken_type_embeddingsword_embeddings LayerNormDropouthidden_dropout_probdropout)selfrr embeddings __class__s d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/deprecated/mmbt/modeling_mmbt.pyrzModalEmbeddings.__init__#s   !yy)A)A6CUCUV#-#A#A %/%E%E")99#--zzF$>$>? cJ|j|j|}|jd}|>|j|}|dz }t j |j d|gd}|>|j|} |dz }t j || j dgd}|`t j|tj|j}|j dj|jd|}|At j|jd|ftj|j}|j|} |j|} || z| z} |j| } |j| } | S)Ndimdtypedevicer)rrsizertorchcat unsqueezearangelongr-expandzerosrrrr!) r" input_modal start_token end_token position_idstoken_type_idstoken_embeddings seq_lengthstart_token_embedsend_token_embedsrrr#s r%forwardzModalEmbeddings.forward.s// [0IJ%**1-  "!%!5!5k!B  !OJ$yy*<*F*Fq*IK[)\bcd   #33I>  !OJ$yy*: J%(;;>SS ^^J/ \\*- r&)NNNN__name__ __module__ __qualname____doc__rr? __classcell__r$s@r%rr sZ @r&ra MMBT model was proposed in [Supervised Multimodal Bitransformers for Classifying Images and Text](https://github.com/facebookresearch/mmbt) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine. It's a supervised multimodal bitransformer model that fuses information from text and other image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MMBTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. transformer (`nn.Module`): A text transformer that is used by MMBT. It should have embeddings, encoder, and pooler attributes. encoder (`nn.Module`): Encoder for the second modality. It should take in a batch of modal inputs and return k, n dimension embeddings. a4 Args: input_modal (`torch.FloatTensor` of shape `(batch_size, ***)`): The other modality data. It will be the shape that the encoder for that type expects. e.g. With an Image Encoder, the shape would be (batch_size, channels, height, width) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. It does not expect [CLS] token to be added as it's appended to the end of other modality embeddings. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) modal_start_tokens (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Optional start token to be added to Other Modality Embedding. [CLS] Most commonly used for classification tasks. modal_end_tokens (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Optional end token to be added to Other Modality Embedding. [SEP] Most commonly used. attention_mask (*optional*) `torch.FloatTensor` of shape `(batch_size, sequence_length)`: Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (*optional*) `torch.LongTensor` of shape `(batch_size, sequence_length)`: Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) modal_token_type_ids (*optional*) `torch.LongTensor` of shape `(batch_size, modal_sequence_length)`: Segment token indices to indicate different portions of the non-text modality. The embeddings from these tokens will be summed with the respective token embeddings for the non-text modality. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) modal_position_ids (`torch.LongTensor` of shape `(batch_size, modal_sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings for the non-text modality. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, embedding_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zRThe bare MMBT Model outputting raw hidden-states without any specific head on top.ceZdZfdZeeeee ddZ dZ dZ xZ S) MMBTModelcvt|||_||_t |||j |_yN)rrr transformerrr# modal_encoderr"rrKrr$s r%rzMMBTModel.__init__s4  &,VWk>T>TUr&) output_type config_classc B||n|jj}||n|jj}||n|jj}| | t d||j }n!| | j dd}n t d| |j n | j }|j|||| |}|j dd}|&tj|tj|}|jj|||| }tj||gd}|j dd}|tj|| }n=tjtj||tj |gd }| tj|| } n.tjtj|| | gd } |j||}|j| }|j!| |jj"} |jj%||| | |||| }|d }|jj'|}|s ||f|ddzSt)|||j*|j,S)a Returns: Examples: ```python # For example purposes. Not runnable. transformer = BertModel.from_pretrained("google-bert/bert-base-uncased") encoder = ImageEncoder(args) mmbt = MMBTModel(config, transformer, encoder) ```NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)r7r8r9r:r+) input_idsr9r: inputs_embedsr()r-)r-r,r))attention_mask head_maskencoder_hidden_statesencoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr)last_hidden_state pooler_output hidden_states attentions)rrXrYuse_return_dict ValueErrorr.r-rLr/onesr3rKr#r0get_extended_attention_maskinvert_attention_mask get_head_masknum_hidden_layersrpoolerrr]r^)r"r6rRmodal_start_tokensmodal_end_tokensrTr:modal_token_type_idsr9modal_position_idsrUrSrVrWrXrYrZinput_txt_shaper-modal_embeddingsinput_modal_shapetxt_embeddingsembedding_output input_shapeextended_attention_maskencoder_extended_attention_maskencoder_outputssequence_output pooled_outputs r%r?zMMBTModel.forwards@2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "'nn.O  &+0023B7OTU U%.%:!! @T@T-- *&+/ . -113CR8  !"ZZuzzRXYN))44l>iv5 !99&6%GK&++-cr2  !"ZZ FCN"YY-fEJJOQ_`fgN " )%*ZZ F%K "%*YY-f=?UV\]& "#'"B"B>S^"_*.*D*DE[*\'&&y$++2O2OP **22 2"7#B/!5#3  *!,((//@ #]3oab6II I)-')77&11   r&c.|jjSrJr#r)r"s r%get_input_embeddingszMMBTModel.get_input_embeddings*s...r&c&||j_yrJrw)r"values r%set_input_embeddingszMMBTModel.set_input_embeddings-s*/'r&)NNNNNNNNNNNNNNN) rArBrCrr MMBT_INPUTS_DOCSTRINGr r_CONFIG_FOR_DOCr?rxr{rErFs@r%rHrHsr V ++@A+ETcd!"#!#k eBk Z/0r&rHzw MMBT Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) cBeZdZdZfdZ ddZxZS)MMBTForClassificationa **labels**: (*optional*) `torch.LongTensor` of shape `(batch_size,)`: Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: *Tuple* comprising various elements depending on the configuration (config) and inputs: **loss**: (*optional*, returned when `labels` is provided) `torch.FloatTensor` of shape `(1,)`: Classification (or regression if config.num_labels==1) loss. **logits**: `torch.FloatTensor` of shape `(batch_size, config.num_labels)` Classification (or regression if config.num_labels==1) scores (before SoftMax). **hidden_states**: (*optional*, returned when `output_hidden_states=True`) list of `torch.FloatTensor` (one for the output of each layer + the output of the embeddings) of shape `(batch_size, sequence_length, hidden_size)`: Hidden-states of the model at the output of each layer plus the initial embedding outputs. **attentions**: (*optional*, returned when `output_attentions=True`) list of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples: ```python # For example purposes. Not runnable. transformer = BertModel.from_pretrained("google-bert/bert-base-uncased") encoder = ImageEncoder(args) model = MMBTForClassification(config, transformer, encoder) outputs = model(input_modal, input_ids, labels=labels) loss, logits = outputs[:2] ```ct||j|_t||||_t j |j|_t j|j|j|_ yrJ) rr num_labelsrHmmbtrrr r!rr classifierrMs r%rzMMBTForClassification.__init__Vs_  ++fk7; zz&"<"<= ))F$6$68I8IJr&c0| | n|jj} |j||||||||| | | |  }|d}|j|}|j |}d}| }|j dk(r2t }||jd| jd}nrs  .U/uu   H %*bii*Z0HVXz0 +z0 z0z _ BII_ _ r&