sgFdZddlmZddlZddlmZddlmZmZm Z m Z ddl Z ddl ZddlmZddlmZmZmZmZmZmZmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"d d l#m$Z$e!jJe&Z'd Z(dZ)eGddeZ*eGddeZ+GddejXjZZ.GddejXjZZ/GddejXjZZ0GddejXjZZ1GddejXjZZ2GddejXjZZ3Gdd ejXjZZ4Gd!d"ejXjZZ5Gd#d$ejXjZZ6Gd%d&ejXjZZ7Gd'd(ejXjZZ8eGd)d*ejXjZZ9Gd+d,eZ:d-Z;d.Z<ed/e;Gd0d1e:Z=Gd2d3ejXjZZ>Gd4d5ejXjZZ?Gd6d7ejXjZZ@Gd8d9ejXjZZAGd:d;ejXjZZBGd<d=ejXjZZCGd>d?ejXjZZDed@e;GdAdBe:ZEy)CzTF 2.0 LXMERT model.) annotationsN) dataclass)DictOptionalTupleUnion)get_tf_activation)TFModelInputTypeTFPreTrainedModelget_initializerkeraskeras_serializable shape_list unpack_inputs)check_embeddings_within_boundsstable_softmax) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) LxmertConfigzunc-nlp/lxmert-base-uncasedrceZdZUdZdZded<dZded<dZded<dZded<dZ ded <dZ ded <dZ ded <dZ ded <y) TFLxmertModelOutputa Lxmert's outputs that contain the last hidden states, pooled outputs, and attention probabilities for the language, visual, and, cross-modality encoders. (note: the visual encoder in Lxmert is referred to as the "relation-ship" encoder") Args: language_output (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the language encoder. vision_output (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the visual encoder. pooled_output (`tf.Tensor` of shape `(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification, CLS, token) further processed by a Linear layer and a Tanh activation function. The Linear language_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for input features + one for the output of each cross-modality layer) of shape `(batch_size, sequence_length, hidden_size)`. vision_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for input features + one for the output of each cross-modality layer) of shape `(batch_size, sequence_length, hidden_size)`. language_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. vision_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. cross_encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Ntf.Tensor | Nonelanguage_output vision_output pooled_outputTuple[tf.Tensor] | Nonelanguage_hidden_statesvision_hidden_stateslanguage_attentionsvision_attentionscross_encoder_attentions) __name__ __module__ __qualname____doc__r__annotations__r r!r#r$r%r&r'`/var/www/html/venv/lib/python3.12/site-packages/transformers/models/lxmert/modeling_tf_lxmert.pyrr8sg D)-O%,&*M#*&*M#*6:3:4818370715.58<5>vG\G\ct>ull((   .v/G/GH)  $ll==fF[F[br=s||++F,F,FG .. ,,  r.c|\}}|j|}|j|}|j|}|j|}||zdz }|j ||}|S)Ntraining)r9r=r@rArL)rS visn_inputrYfeatsboxesxyoutputs r/callz!TFLxmertVisualFeatureEncoder.callsk! u LL    # KK     "a%1fx8 r.c|jryd|_t|ddZtj|jj 5|jj dd|jgdddt|dddtj|jj 5|jj dd|jjgdddt|ddZtj|jj 5|jj dd|jgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swYUxYw#1swYxYw#1swYxYw#1swYyxYw)NTr9r=r@rA)builtgetattrtf name_scoper9r<buildrNr=rQrFr@rPrArS input_shapes r/rfz"TFLxmertVisualFeatureEncoder.builds ::  4D ) 5t||001 @ ""D$ #>? @ 4*D 1 =t33889 R$$**D$ 8O8O+PQ R 44 ( 4t{{//0 > !!4t||"<= > 4)4 0 <t22778 Q##))4t{{7N7N*OP Q Q = @ @ R R > > Q Qs0 )G23G#)G  3G,GG G),G5FNr(r)r*rCr`rf __classcell__rUs@r/r7r7s0 Qr.r7c2eZdZdZfdZddZddZxZS)TFLxmertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c Vt|di|||_|j|_|j|_|j |_t jj|jd|_ t jj|j|_ y)N LayerNormr>)rater-)rBrCrQrFmax_position_embeddingsrGrrDrHrIrqrJrKrLrRs r/rCzTFLxmertEmbeddings.__init__s "6" !--'-'E'E$!'!9!988AVAV]h8i||++1K1K+L r.ctjd5|jd|jj|j gt |j|_dddtjd5|jd|jj|j gt |j|_ dddtjd5|jd|j|j gt |j|_ ddd|jryd|_ t|d detj|jj 5|jj#dd|jj gdddyy#1swY]xYw#1swYxYw#1swYxYw#1swYyxYw) Nword_embeddingsweight)rG)r<shape initializertoken_type_embeddings embeddingsposition_embeddingsTrq)rdre add_weightrQ vocab_sizerFr rGrvtype_vocab_sizeryrsr{rbrcrqr<rfrgs r/rfzTFLxmertEmbeddings.builds ]], - //{{--t/?/?@+d>T>TU*DK ]]2 3 )-!{{22D4D4DE+d>T>TU*9*D & ]]0 1 '+!33T5E5EF+d>T>TU(7(D $  ::  4d + 7t~~223 L$$dD$++2I2I%JK L L 81       L Ls2AF?AG ,AG3G$?G  GG!$G-c"||J|At||jjtj|j |}t |dd}|tj|d}tjtjd|dd}tj|j|}tj|j|}||z|z} |j| } |j| | } | S) z Applies embedding based on inputs tensor. Returns: final_embeddings (`tf.Tensor`): output embedding tensor. N)paramsindicesr)dimsvalue)startlimitaxisinputs)rrY)rrQr}rdgatherrvrfill expand_dimsranger{ryrqrL) rS input_idstoken_type_ids inputs_embedsrYrh position_idsposition_embedstoken_type_embedsfinal_embeddingss r/r`zTFLxmertEmbeddings.calls%-*?@@  *9dkk6L6L MIIT[[)LM /4  !WW+Q?N~~bhhQk"o&NUVW ))4+C+C\ZIIT-G-GQ_`(?:=NN>>1A>B<$>>!CCC#&v'9'9F>  %WW^;K;Q;QRN/.@ ))9C,,,J /;?  ]F  JD,>,>? 7H=/2O\M]r.c|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddZtj|jj 5|jj dd|jgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYxYw#1swY~xYw#1swYyxYw)NTrrr) rbrcrdrerr<rfrQrFrrrrgs r/rfzTFLxmertAttention.build`s$ ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4 % 1txx}}- ;dDLL9: ; 4$ ' 3tzz/ =   $dll!;< = = 4  H H ; ; = =s$ 3E<)E"#)E.E"E+.E7rirj)r(r)r*rCrr`rfrlrms@r/rrsB2 $L =r.rc,eZdZfdZdZddZxZS)TFLxmertIntermediatec Tt|di|tjj |j t |jd|_t|jtr"t|j|_ ||_y|j|_ ||_y)Ndenser:r-)rBrCrrDrEintermediate_sizer rGr isinstance hidden_actstrr intermediate_act_fnrQrRs r/rCzTFLxmertIntermediate.__init__ps "6"\\''  $ $.v/G/GH(  f'' -'89J9J'KD $ (.'8'8D $ r.cJ|j|}|j|}|Srj)rrrSrs r/r`zTFLxmertIntermediate.call}s& =1 00? r.c(|jryd|_t|ddetj|jj 5|jj dd|jjgdddyy#1swYyxYwNTr rbrcrdrerr<rfrQrFrgs r/rfzTFLxmertIntermediate.build} ::  4$ ' 3tzz/ H   $dkk.E.E!FG H H 4 H H 3BBrjrkrms@r/rros  Hr.rc.eZdZfdZddZddZxZS)TFLxmertOutputc vt|di|tjj |j t |jd|_tjj|jd|_ tjj|j|_||_yNrr:rqr>r-rBrCrrDrErFr rGrrHrIrqrJrKrLrQrRs r/rCzTFLxmertOutput.__init__s "6"\\''   .v/G/GH(  88AVAV]h8i||++F,F,FG  r.ct|j|}|j||}|j||z}|SrjrrLrqrSr input_tensorrYs r/r`zTFLxmertOutput.calls9 =1  ]H= }|'CD r.c"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYwNTrrq) rbrcrdrerr<rfrQrrqrFrgs r/rfzTFLxmertOutput.builds ::  4$ ' 3tzz/ N   $dkk.K.K!LM N 4d + 7t~~223 L$$dD$++2I2I%JK L L 8 N N L L 3C9<3D9DDrirjrkrms@r/rrs  Lr.rc.eZdZfdZddZddZxZS)TFLxmertAttentionOutputc vt|di|tjj |j t |jd|_tjj|jd|_ tjj|j|_||_yrrrRs r/rCz TFLxmertAttentionOutput.__init__s "6"\\''   .v/G/GH(  88AVAV]h8i||++F,F,FG  r.cv|j|}|j||}|j||z}|S)NrXrrs r/r`zTFLxmertAttentionOutput.calls; =1  ]X F }|'CD r.c"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYwr rbrcrdrerr<rfrQrFrqrgs r/rfzTFLxmertAttentionOutput.build ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4d + 7t~~223 L$$dD$++2I2I%JK L L 8 H H L Lrrirjrkrms@r/rrs  Lr.rc.eZdZfdZddZddZxZS)TFLxmertSelfAttentionLayerc lt|di|t|d|_t |d|_y)NrSr<r_r-)rBrCrrSrattention_outputrRs r/rCz#TFLxmertSelfAttentionLayer.__init__s0 "6"%f6: 7X Nr.ct|j||||}|r|d}|j|d|}|r|fS|fS)Nrr)rSr)rSrrrrY self_outputrrs r/r`zTFLxmertSelfAttentionLayer.callsQii lNL]^ )!nO00QN6G /2`N^M``r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTrSr)rbrcrdrerSr<rfrrgs r/rfz TFLxmertSelfAttentionLayer.builds ::  4 & 2tyy~~. & % & 4+T 2 >t4499: 2%%++D1 2 2 ? & & 2 2 C %C CC rirjrkrms@r/rrsO a 2r.rc2eZdZfdZ ddZddZxZS)TFLxmertCrossAttentionLayerc lt|di|t|d|_t |d|_y)Nattrr_r-)rBrCrrrrrRs r/rCz$TFLxmertCrossAttentionLayer.__init__s0 "6"$V%8 7X Nr.c|j|||||}|r|d}|j|d||}|r|f} | S|f} | SNrXrr)rr) rSr ctx_tensor ctx_att_maskrrYr_rrrs r/r`z TFLxmertCrossAttentionLayer.callsj, LBS^fg $QiO00LS[0\9J#_5RbPcr.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTrr)rbrcrdrerr<rfrrgs r/rfz!TFLxmertCrossAttentionLayer.builds ::  4 % 1txx}}- %t$ % 4+T 2 >t4499: 2%%++D1 2 2 ? % % 2 2r)FFrjrkrms@r/rrsO   2r.rc.eZdZfdZddZddZxZS) TFLxmertLayerc t|di|t|d|_t |d|_t |d|_y)N attentionr intermediater_r-)rBrCrrrrrtransformer_outputrRs r/rCzTFLxmertLayer.__init__s? "6"3FM0nM"0h"Gr.c|j||||}|d}|j|}|j|||}|f|ddz} | S)NrXrr)rrr) rSrrrrYattention_outputsrintermediate_output layer_outputrs r/r`zTFLxmertLayer.call sk NN=.J[fnNo,Q/"//0@A../BDT_g.h /$5ab$99r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYqxYw#1swYyxYw)NTrrr) rbrcrdrerr<rfrrrgs r/rfzTFLxmertLayer.builds  ::  4d + 7t~~223 +$$T* + 4 . :t00556 .!!''- . 4-t 4 @t66;;< 4''--d3 4 4 A  + + . . 4 4$ D%%D1?D=%D.1D:=Erirjrkrms@r/rrsH  4r.rcLeZdZfdZ ddZ ddZddZ ddZddZxZ S) TFLxmertXLayerc  t|d i|t|d|_t |d|_t |d|_t|d|_t|d|_ t|d|_ t|d|_ y) Nvisual_attentionr lang_self_att visn_self_att lang_inter lang_output visn_inter visn_outputr-) rBrCrrrrrrr rr r r rRs r/rCzTFLxmertXLayer.__init__"s "6" ;FI[ \8_U7_U/vLI)&}E.vLI)&}Er.c tj|}tj|}tj|} tj|} |j|| |||} |j| ||||} | | fS)N)rrY)rdidentityr) rS lang_inputlang_attention_maskrZvisn_attention_maskrrYlang_attention_lang_inputvisn_attention_lang_inputlang_attention_visn_inputvisn_attention_visn_inputlang_att_outputvisn_att_outputs r/ cross_attzTFLxmertXLayer.cross_att0s%'KK $;!$&KK $;!$&KK $;!$&KK $;!// % % / 0 // % % / 0 //r.cnd}|j||||}|j||||}|d|dfS)NFrXr)rr) rSrrrZrrYrrrs r/self_attzTFLxmertXLayer.self_attQsW",,Z9LN_jr,s,,Z9LN_jr,sq!?1#555r.c|j|}|j|}|j|||}|j|||}||fSrj)r r r r )rSrrZrYlang_inter_outputvisn_inter_outputr r s r/ output_fczTFLxmertXLayer.output_fc_sW OOJ7 OOJ7&&'8*hO &&'8*hO K''r.c|}|}|j||||||\}}|dd} |j|d||d||\}}|j|||\} } |r| | | dfS| | fSr)rrr) rS lang_featsr visn_featsrrrYrrrr r s r/r`zTFLxmertXLayer.callis%$+/>>      ,:, (*!"-+/== A   A   ,9, ($(>>/?]e>#f [AR [/!*<=rYdfqXrrr.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swY(xYw#1swYxYw#1swYxYw#1swYAxYw#1swYxYw#1swYxYw#1swYyxYw) NTrrrr r r r ) rbrcrdrerr<rfrrr r r r rgs r/rfzTFLxmertXLayer.buildsF ::  4+T 2 >t4499: 2%%++D1 2 4$ / ;t11667 /""((. / 4$ / ;t11667 /""((. / 4t , 8t334 ,%%d+ , 4 - 9t//445 -  &&t, - 4t , 8t334 ,%%d+ , 4 - 9t//445 -  &&t, - - :# 2 2 / / / / , , - - , , - -sT J %J?J'J43K K 'K JJ$'J14J>K  KK"rirj) r(r)r*rCrrrr`rfrlrms@r/rr!s4 F*0N 6("s@-r.rc<eZdZfdZ ddZddZxZS)TFLxmertEncoderc t|di|t|d|_|j|_|j |_|j|_ t|j Dcgc]}t|d|c}|_ t|jDcgc]}t|d|c}|_t|jDcgc]}t|d|c}|_||_ycc}wcc}wcc}w)Nr9rzlayer_._z x_layers_._z r_layers_._r-)rBrCr7r9l_layers num_l_layersx_layers num_x_layersr_layers num_r_layersrrlayerrrQ)rSrQrTirUs r/rCzTFLxmertEncoder.__init__s "6"3FK #OO"OO"OOKPPTPaPaJbcQmF8A3@c QVW[WhWhQijA{1#5FGj PUVZVgVgPhi1vk!4EFi  djis-D#DD c ^d}d} |s|jjrdnd} |s|jjrdnd} |s|jjrdnd} |j||g|}|jD]%} | ||||}|d}| |fz} | | |dfz} '|jD]%} | ||||}|d}||fz}| | |dfz} '|j D]0} | ||||||}|dd\}}||fz}| |fz} | (| |dfz} 2||r| ndf}| |r| ndf}|||r| fSdfS)Nr-rXrrrW)rQrr9r,r*r()rSr r visual_feats visual_posvisual_attention_maskrrYr$r#r&r%r' layer_module l_outputs v_outputs x_outputsvisual_encoder_outputslang_encoder_outputss r/r`zTFLxmertEncoder.calls "!#"3t{{7T7TBZ^$59V9Vb\`):dkk>[>[2ae ||\:$>|R !JJ LL$Z1DFWbjkI"1J%;zm%K "".&9Yq\O&K#  L!MM HL$%!! I %QrRs r/rCzTFLxmertMainLayer.__init__s "6" "OO"OO"OO!'!9!9!'!9!9$*$?$?!!11,V,G&vI> $V(;  r.c|jSrj)rzrSs r/get_input_embeddingsz&TFLxmertMainLayer.get_input_embeddings%s r.c`||j_t|d|j_yNr)rzrvrr}rSrs r/set_input_embeddingsz&TFLxmertMainLayer.set_input_embeddings(s$!&%/%6q%9"r.ctrj)NotImplementedError)rSheads_to_prunes r/ _prune_headszTFLxmertMainLayer._prune_heads,s!!r.c | | td| t|} n|t|dd} n td|| td|tj| d}|tj| d}|j |||| } tj || ddd| df}tj || j}tjd| j}tjd | j}tjtj|||}|tj || ddd| df}tjtj|d d }tj || j}tjtj|||}nd}|j| |||||| }|dd \}}|d}|d}d }|r|d}|d}|d }|||f}| r||fnd }|d}|d}|j|}| s |||f|z|zSt|||| r|nd| r|nd|rnd|rnd|r Sd S)NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedszGvisual_feats and visual_pos cannot be `None` in LXMERT's `call` method.rrr?grrWr-)r!rr r#r$r%r&r')rrrdrrzrrrconstantmultiplysubtractrr=r>r)rSrr/r0rr1rrrr?rArYrhembedding_outputextended_attention_maskone_cstten_thousand_cstextended_visual_attention_maskencoder_outputsr6r7r$r#all_attentionsr%r&r'r visual_outputr r!s r/r`zTFLxmertMainLayer.call/s  ]%>cd d  "$Y/K  &$]3CR8KTU U  !5fg g  !WW[!4N  !WW[!4N ??9nmU]^#%**^k!naQRT_`aTb=c"d#%''*AIYI_I_"`++c)9)?)?@;;x7G7M7MN"$++bkk'CZ.[]m"n ,-/ZZ8MP[\]P^`acdfqrsftOu-v *-/^^BNNK`ghNQ__ _"'''=Q#9W[9M!5SW7H 3d3D/$AR%=  Y]  r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYqxYw#1swYyxYw)NTrzr=r>) rbrcrdrerzr<rfr=r>rgs r/rfzTFLxmertMainLayer.builds ::  4t , 8t334 ,%%d+ , 4D ) 5t||001 ) ""4( ) 44 ( 4t{{//0 ( !!$' ( ( 5  , , ) ) ( (r NNNNNNNNNNFrj) r(r)r*r config_classrCrErIrMrr`rfrlrms@r/r;r;s[L :""!j j X (r.r;c8eZdZdZeZdZedZedZ y)TFLxmertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. lxmertcd}d}tjgdgdgtj}tjj |||j j f}tjj ||df}|||dS)n Dummy inputs to build the network. Returns: tf.Tensor with dummy inputs rW r rWr rrhrr/r0)rdrPint32randomuniformrQrM)rSrnum_visual_featuresrr/r0s r/ dummy_inputsz$TFLxmertPreTrainedModel.dummy_inputss  KKI 6bhhG yy((*6I4;;KfKf)gh YY&& 4G'KL #($  r.c tjdtjdtjdtjdtjdd|jjftj dtjdtj dtjdtjdtjdtjd d S) N)NNrrrr/)NNrhr0r1r)rrr/r0r1r)rd TensorSpecrjrQrMfloat32rDs r/input_signaturez'TFLxmertPreTrainedModel.input_signatures|RXXKP mmL"((IYZMM4t{{7R7R*SUWU_U_ftu--,W%']]<Pg%h mmL"((IYZ   r.N) r(r)r*r+rr]base_model_prefixpropertyrnrrr-r.r/r_r_s:  L    &  r.r_a The LXMERT model was proposed in [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490) by Hao Tan and Mohit Bansal. It's a vision and language transformer model, pre-trained on a variety of multi-modal datasets comprising of GQA, VQAv2.0, MCSCOCO captions, and Visual genome, using a combination of masked language modeling, region of interest feature regression, cross entropy loss for question answering attribute prediction, and object tag prediction. This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. TensorFlow models and layers in `transformers` accept two formats as input: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should "just work" for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument: - a single Tensor with `input_ids` only and nothing else: `model(input_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})` Note that when creating models and layers with [subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) then you don't need to worry about any of this, as you can just pass inputs like you would to any other Python function! Parameters: config ([`LxmertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`np.ndarray` or `tf.Tensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) visual_feats (`tf.Tensor` of shape `(batch_size, num_visual_features, visual_feat_dim)`): This input represents visual features. They ROI pooled object features from bounding boxes using a faster-RCNN model) These are currently not provided by the transformers library. visual_pos (`tf.Tensor` of shape `(batch_size, num_visual_features, visual_feat_dim)`): This input represents spacial features corresponding to their relative (via index) visual features. The pre-trained LXMERT model expects these spacial features to be normalized bounding boxes on a scale of 0 to 1. These are currently not provided by the transformers library. attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) visual_attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): MMask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True. training (`bool`, *optional*, defaults to `False`): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). z`The bare Lxmert Model transformer outputting raw hidden-states without any specific head on top.ceZdZfdZeeeeee e  d ddZ ddZ xZ S) TFLxmertModelcPt||g|i|t|d|_y)Nr`r)rBrCr;r`)rSrQrrTrUs r/rCzTFLxmertModel.__init__Ds( 3&3F3'X> r.) checkpoint output_typer]c <|j||||||||| | | } | Srj)r`) rSrr/r0rr1rrrr?rArYrs r/r`zTFLxmertModel.callHs:*++     !       r.c|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYw)NTr`)rbrcrdrer`r<rfrgs r/rfzTFLxmertModel.buildmsi ::  44 ( 4t{{//0 ( !!$' ( ( 5 ( ( A11A:r\)rTFModelInputType | Noner/rr0rrnp.ndarray | tf.Tensor | Noner1r~rr~rr~rOptional[bool]r?rrArrYboolreturnz!Union[Tuple, TFLxmertModelOutput]rj)r(r)r*rCrrLXMERT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCr`rfrlrms@r/rvrv?s ?*+BC&'$.2)-'+8r-)rBrCrrDrErFr rGrrrrr transform_act_fnrHrIrqrQrRs r/rCz(TFLxmertPredictionHeadTransform.__init__s "6"\\''$$.v/G/GH(  f'' -$5f6G6G$HD !$*$5$5D !88AVAV]h8i r.cp|j|}|j|}|j|}|S)Nr)rrrqrs r/r`z$TFLxmertPredictionHeadTransform.calls8 - 8 --m< m< r.c"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYwrrrgs r/rfz%TFLxmertPredictionHeadTransform.buildrr)rQrr tf.Tensorrrrjrkrms@r/rrs" Lr.rcPeZdZdfd Zd dZd dZd dZd dZd dZd dZ xZ S)TFLxmertLMPredictionHeadc t|di|||_|j|_t |d|_||_y)N transformrr-)rBrCrQrFrrinput_embeddingsrSrQrrTrUs r/rCz!TFLxmertLMPredictionHead.__init__s@ "6" !--8kR!1r.cX|j|jjfddd|_|jryd|_t |ddNt j|jj5|jjddddyy#1swYyxYw)NzerosTbias)rwrx trainabler<r) r|rQr}rrbrcrdrerr<rfrgs r/rfzTFLxmertLMPredictionHead.buildsOO4;;+A+A*CQXdhouOv ::  4d + 7t~~223 +$$T* + + 8 + +s :B  B)c|jSrj)rrDs r/get_output_embeddingsz.TFLxmertLMPredictionHead.get_output_embeddingss$$$r.c`||j_t|d|j_yrG)rrvrr}rHs r/set_output_embeddingsz.TFLxmertLMPredictionHead.set_output_embeddingss(',$+5e+r.rc0eZdZdfd ZddZddZxZS)TFLxmertMLMHeadc Jt|di|t||d|_y)N predictionsrr-)rBrCrrrs r/rCzTFLxmertMLMHead.__init__s& "6"3Ft4499: S%%++T49P9P,QR S S ? - - S Ss C"%3C."C+.C7rjrkrms@r/rrs 9 Sr.rc,eZdZfdZdZddZxZS)TFLxmertVisualAnswerHeadc t|di||j}tjj |dzt |jd|_td|_ tjj|jd|_ tjj |t |jd|_||_y) NrWz logit_fc_._0r:geluz logit_fc_._2r>z logit_fc_._3r-)rBrCrFrrDrEr rGrr rrHrI layer_normdense_1hid_dim)rSrQ num_labelsrTrrUs r/rCz!TFLxmertVisualAnswerHead.__init__s "6"$$\\'' aK.v/G/GH(  ,F3,,99&BWBW^l9m||)) .v/G/GH*   r.c|j|}|j|}|j|}|j|}|Srj)rrrrrs r/r`zTFLxmertVisualAnswerHead.call,s@ =1  6  6  ]3 r.c|jryd|_t|ddZtj|jj 5|jj dd|jgdddt|dd\tj|jj 5|jj d|jdzgdddt|dd^tj|jj 5|jj dd|jdzgdddyy#1swYxYw#1swYxYw#1swYyxYw)NTrrrWr) rbrcrdrerr<rfrrrrgs r/rfzTFLxmertVisualAnswerHead.build4s1 ::  4$ ' 3tzz/ =   $dll!;< = 4t , 8t334 @%%tT\\A-=&>? @ 4D ) 5t||001 C ""D$ q0@#AB C C 6  = = @ @ C Cs$ )E2+E,E)EE&)E2rjrkrms@r/rrs" Cr.rc,eZdZfdZdZddZxZS)TFLxmertVisualObjHeadc t|d i|t|d|_i}|jrd|j d|d<|j rd|jd|d<|jrd|jd|d<||_ |jDcic]K}|tjj|j|d t|jd | Mc}|_||_ycc}w) Nrrr)rwnumobjattr)rifeatrz decoder_dict.r:r-)rBrCrrvisual_obj_lossnum_object_labelsvisual_attr_lossnum_attr_labelsvisual_feat_lossrM visual_lossesrrDrEr rG decoder_dictrQ)rSrQrTrrrUs r/rCzTFLxmertVisualObjHead.__init__Ds "6"8kR  ! !-26;S;S#TM%  " ".3F ,FDKK4J4JQVW  $9&GY$ZD ! <<7@R@RYfgD ,,$$3\$B||AAdAS,,<<<N    ! !//!$M%  " "--!%M& !  " "f445--%M& ! +r.cd}d}tjgdgdgtj}tjj |||j j f}tjj ||df}|j jri}|j jrG|j jr1tj||gtj||gfd<|j jr\|j jrFtj|||j j gtj||gfd<|j jrG|j jr1tj||gtj||gfd <i|||d |j jrd iSiS) rbrWrcrdrgrrhrrrri obj_labels) rdrPrjrkrlrQrMrronesrr)rSrrmrr/r0rs r/rnz#TFLxmertForPreTraining.dummy_inputss  KKI 6bhhG yy((*6I4;;KfKf)gh YY&& 4G'KL ;; ' 'J ;; ' 'DKK,H,H%89:%89:"Jv  ;; ' 'DKK,H,H%8$++:U:UVW%89:"Jv  ;; & &4;;+G+G%89:%89:!Ju   & ,( .2[[-I-I j)   PR   r.c.|jjSrj)rrrDs r/ get_lm_headz"TFLxmertForPreTraining.get_lm_headsxx###r.ctjdt|jdz|jjzdz|jj jzS)NzMThe method get_prefix_bias_name is deprecated. Please use `get_bias` instead./)warningswarn FutureWarningr<rrrDs r/get_prefix_bias_namez+TFLxmertForPreTraining.get_prefix_bias_namesG egtuyy3.4txx7K7K7P7PPPr.)ryr]c |j|||||||| | || }|d|d|d}}}|j||\}}|jr|j|}n|dd}|| | | dnt j d}d}|l|j r`|jdt j|dgt j|d|jjg}||z }||fz }| X|jrL|jdt j| dgt j|ddg}||z }||fz }| .|jr!d}|j|}|jjD]\}}| |\}} |d }!|d }"|d }#|j }$|j|"}%||}&|%t j||#t j|&d|!g}'|'j"dkDrt j$|'}'t j$|'t j&t j| dg|'j(z|$z}'||'z }||'fz }||z }| b|jrV|jdt j| dgt j|d|j*g}(||(z }||(fz }|s|||f|d dz})||f|)zS|)St-|||||j.|j0|j2|j4|j6 S)ag masked_lm_labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` obj_labels (`Dict[Str: Tuple[tf.Tensor, tf.Tensor]]`, *optional*, defaults to `None`): each key is named after each one of the visual losses and each element of the tuple is of the shape `(batch_size, num_features)` and `(batch_size, num_features, visual_feature_dim)` for each the label id and the label score respectively matched_label (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the whether or not the text input matches the image (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates that the sentence does not match the image, - 1 indicates that the sentence does match the image. ans (`tf.Tensor` of shape `(batch_size)`, *optional*, defaults to `None`): a one hot representation hof the correct answer *optional* Returns: rrrWNgr-rrrr2rwr ) r2r3r4r5r#r$r%r&r')r`rrrrdrPrrrrQr}rrrritemsrndim reduce_meanrrrr1r#r$r%r&r')*rSrr/r0rr1rrmasked_lm_labelsr matched_labelansrr?rArY lxmert_outputr rZr!lang_prediction_scoresr4 answer_score total_lossrmasked_lm_loss matched_losstotal_visn_lossvisn_prediction_scores_dictrkey_infolabel mask_conf output_dim loss_fct_name label_shaperv visn_loss_fctvisn_prediction_scores visn_loss answer_lossr_s* r/r`zTFLxmertForPreTraining.callsT     !        !  !  ! %2] <@88KQ^;_8 8 <<++M:L(+A.L!(]-BzGY^a^i S!    'D,=,=1T^^D1 +bT2 1B 8N8N3OPN . (J ~' 'F  $):):/4>>$/ =2$/ 3b!W=L , &J |o %F  !d&;&;!O*.*?*? *N '!%!3!3!9!9!; ' X#-c? y%e_ ( 0 &w/ 44 $} = )DS)I&)JJuk2JJ5J7GH >>A% "y 9INN9rwwrzz)VXUY?Z\e\k\k7l+lmpvv 9,9,&# '$ / )J ?t||.$... 3%rzz,TEWEW@X'YK + %J {n $F&(ab! "F 0:/EZMF* Q6 Q+4%=%1#0#G#G!.!C!C - A A+==%2%K%K  r.c`|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYxYw#1swY~xYw#1swYyxYw)NTr`rrr) rbrcrdrer`r<rfrrrrgs r/rfzTFLxmertForPreTraining.buildesR ::  44 ( 4t{{//0 ( !!$' ( 4 % 1txx}}- %t$ % 4+T 2 >t4499: 2%%++D1 2 4 - 9t//445 -  &&t, - - : ( ( % % 2 2 - -s0 E?%F ?FF$?F  FF!$F-)NNNNNNNNNNNNNNF) rr}r/rr0rrrr1rrrrrrrrz-Dict[str, Tuple[tf.Tensor, tf.Tensor]] | Nonerrrrr bool | Noner?rrArrYrrz/Tuple[tf.Tensor] | TFLxmertForPreTrainingOutputrj)r(r)r*rCrtrnrrrrrrr1rr`rfrlrms@r/rrrsD1+f& & P$Q*+BC+GVef.2)-'++/26+/*.-1DH*. $)-,0#'!I *I 'I % I ) I 0 I )I (I +I BI (I I 'I *I !I !I " 9#I gDI V-r.r)Fr+ __future__rr dataclassesrtypingrrrrnumpynp tensorflowrdactivations_tfr modeling_tf_utilsr r r rrrrtf_utilsrrutilsrrrrrrconfiguration_lxmertr get_loggerr(loggerrrrr1rDLayerr7rorrrrrrrrr$r;r_LXMERT_START_DOCSTRINGrrvrBrrrrrrrr-r.r/r's""!///G/   H %3  *=+*= *=Z ,=;,= ,=^4Q5<<#5#54QnC ++C LY= **Y=xH5<<--H8LU\\''L>Lell00L<2!3!3242%,,"4"42B4ELL&&4<@-U\\''@-Fj&ell((j&ZV( **V(V(r& /& R*X<~f0(+0( 0(fHU\\''H8"Lell&8&8"LL-u||11-b-ell((-(Su||11S:&Cu||11&CR,GELL..,G^PRhiA-4A-jA-r.