sgfdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZmZdd lmZmZmZmZmZmZdd lmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e%jPe)Z*dZ+dZ,dZ-Gdde j\Z/GddeZ0Gdde j\Z1Gdde j\Z2Gdde j\Z3Gdde j\Z4Gdde j\Z5Gd d!e j\Z6Gd"d#e j\Z7Gd$d%e j\Z8Gd&d'e j\Z9Gd(d)e j\Z:d*Z;d+Z<e#d,e;Gd-d.e0Z=Gd/d0e j\Z>e#d1e;Gd2d3e0Z?Gd4d5e j\Z@e#d6e;Gd7d8e0ZAe#d9e;Gd:d;e0ZBe#de0ZCe#d?e;Gd@dAe0ZDy)BzPyTorch ConvBERT model.N) attrgetter)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelSequenceSummary)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)ConvBertConfigzYituTech/conv-bert-baserc ddl}tj j |}tjd||jj|}i}|D]A\}}tjd|d||jj||} | ||<Cddd d d d d d} |jdkDrd} nd} t|jD]:} d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d | d| d!<d| d"| d| d#<d| d$| d| d%<d| d&| d| d'<d| d(| d| d)<d| d*| d| d+<d| d,| d| d-<d| d.| d| d/<d| d0| d| d1<d| d2| d| d3<d| d4| d| d5<d| d6| d7| d| d8<d| d6| d9| d| d:<d| d;| d7| d| d<<d| d;| d9| d| d=<d| d>| d| d?<d| d@| d| dA<=|jD]} | d}t|}||}| |}t!j"||}tjdB|dC|dD|j%d7r.|j%dEs|j%dFs |j&}|j%dGr|j)ddHd}|j%dIr|j)dHdd}|j%dJr|j+dK}||_|S#t$rtjdwxYw)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias) tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variable num_groupsrangenum_hidden_layersnamed_parametersrtorch from_numpyendswithTpermute unsqueezedata)modelconfigtf_checkpoint_pathtftf_path init_varstf_datanameshapearray param_mappinggroup_dense_namejparam param_name retrieverresulttf_namevalues a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbertrO0soggoo01G KK8 BC''0IG  e (l5'BC&&w5  .R1Y3]'K%H%H#DM1$" 6++ ,Cw$QC'C D qc)EFG%QC'A B qc)CDE%QC'A B qc)CDE%QC'? @ qc)ABC%QC'C D qc)EFG%QC'A B qc)CDE%QC'U V qc)]^_%QC'U V qc)]^_%QC'I J qc)QRS%QC'N O qc)QRS%QC'L M qc)OPQ%QC'M N qc)NOP%QC'K L qc)LMN%QC'E F qc)GHI%QC'H I qc)KLM%QC'C D qc)EFG%QC'G H qc)IJK%QC~6F5Gw O qc)CDE%QC~6F5Gu M qc)ABC%QCx0@/A I qc)=>?%QCx0@/A G qc);<=%QC'> ? qc)ABCG]]^\__uDv qc)?@AGCwJ'')1X z* 5! +  !12 d7)6*Q78   I &##$BC''(@A!GGE   / 0MM!Q*E   / 0MM!Q*E   1 2OOB'E #$ Lk   Q   s L++ M c eZdZdZfdZ d deejdeejdeejdeejdejf dZ xZ S) ConvBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_ids)rr$F) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr4arangeexpandzerosrVsizelongselfr< __class__s rNr[zConvBertEmbeddings.__init__s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsrXrV inputs_embedsreturnc2||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} |j|} || z| z} |j| } |j| } | S)Nr$rrXrrYdevice)rnrVhasattrrXrlr4rmroryr`rbrdreri) rqrtrXrVru input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrbrd embeddingss rNforwardzConvBertEmbeddings.forwards,  #..*K',,.s3K ^  ,,Q  ^ J"%88;PP ^^J/ \\*- rs)NNNN) __name__ __module__ __qualname____doc__r[rr4 LongTensor FloatTensorr __classcell__rrs@rNrQrQsQ (15593759 $E,,-$!!1!12$u//0 $   1 12 $    $rsrQc&eZdZdZeZeZdZdZ dZ y)ConvBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. convbertTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsmeanstdNg?) isinstancerLinearweightr:normal_r<initializer_rangebiaszero_r\rSrefill_)rqmodules rN _init_weightsz%ConvBertPreTrainedModel._init_weightss fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rsN) rrrrr config_classrOload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrsrNrrs$ "L1O"&*#*rsrcZeZdZdZfdZdej dej fdZxZS)SeparableConv1DzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc t|tj|||||dzd|_tj||dd|_tj tj|d|_ |jjjjd|j|j jjjd|jy)Nr#F) kernel_sizegroupspaddingrr)rrrr)rZr[rConv1d depthwise pointwise Parameterr4rmrrr:rr)rqr< input_filtersoutput_filtersrkwargsrrs rNr[zSeparableConv1D.__init__s   # 1$  =.aV[\LL^Q!?@  ""**9Q9Q*R ""**9Q9Q*Rrs hidden_statesrvch|j|}|j|}||jz }|SN)rrr)rqrxs rNrzSeparableConv1D.forwards0 NN= ) NN1  TYYrs rrrrr[r4Tensorrrrs@rNrrs']S U\\ellrsrceZdZfdZdZ d dej deejdeejdeej dee de ej eej ff d Z xZ S) ConvBertSelfAttentioncjt||j|jzdk7r2t |ds&t d|jd|jd|j|j z}|dkr|j|_d|_n||_|j |_|j|_|j|jzdk7r t d|j|jzdz|_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_t||j|j|j|_tj|j|j|jz|_tj|j|j|_tj&|jdgt)|jdz dz dg |_tj,|j.|_y) Nrr^zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr#)rr)rZr[ hidden_sizenum_attention_headsrz ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerrquerykeyrMrkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldrgattention_probs_dropout_probri)rqr<new_num_attention_headsrrs rNr[zConvBertSelfAttention.__init__s>     : : :a ?PVXhHi#F$6$6#78 445Q8  #)"<"<@Q@Q"Q "Q &$88DO'(D $'>D $$//DO & 7 7    8 8 8A =UV V$*$6$6$:R:R$RWX#X !558P8PPYYv1143E3EF 99V//1C1CDYYv1143E3EF #2 F&&(:(:D //0AB ..)BDUV 22?C!MM*;b$BWBWYZ=[\!MM*;C,,];~ BHZHZ7[\'11!Q7BBDNNrR-- ..2++a/A5q9 . (11!Q7?? D..0E0E ~D P u||Xell33 4PrsrcneZdZfdZdej dej dej fdZxZS)ConvBertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrT) rZr[rrrr!rerfrgrhrirps rNr[zConvBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rsr input_tensorrvcr|j|}|j|}|j||z}|Srr!rirerqrrs rNrzConvBertSelfOutput.forward7 =1  ]3 }|'CD rsrrrr[r4rrrrs@rNrrs1> U\\RWR^R^rsrceZdZfdZdZ d dej deejdeejdeej dee de ej eejff d Z xZ S) ConvBertAttentionct|t||_t ||_t |_yr)rZr[rrqroutputset pruned_headsrps rNr[zConvBertAttention.__init__s0 )&1 (0 Ersc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrqrrrrrrrMrr!runion)rqheadsindexs rN prune_headszConvBertAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rsrrrrrrvcl|j|||||}|j|d|}|f|ddz}|S)Nrr)rqr) rqrrrrr self_outputsattention_outputrs rNrzConvBertAttention.forwardsQyy    !    ;;|A F#% QR(88rsr) rrrr[rr4rrrrrrrrs@rNrrs" ;*7;158<,1 ||!!2!23E--.  ( 5  $D>  u||Xe&7&788 9rsrcVeZdZfdZdej dej fdZxZS)GroupedLinearLayerct|||_||_||_|j|jz|_|j|jz|_tjtj|j|j |j |_ tjtj||_ yr) rZr[ input_size output_sizer0 group_in_dim group_out_dimrrr4emptyrr)rqr rr0rrs rNr[zGroupedLinearLayer.__init__s $&$ OOt>!--@ll5;;t@Q@QSWSeSe#fg LL[!9: rsrrvct|jd}tj|d|j|j g}|j ddd}tj||j}|j ddd}tj||d|jg}||jz}|S)Nrr$rr#) listrnr4rr0rr8rrrr)rqrrrs rNrzGroupedLinearLayer.forwards-,,./2 MM-"doot?P?P)Q R IIaA  LLDKK ( IIaA  MM!j"d.>.>? @  Mrsrrs@rNr r s#;U\\ellrsr cVeZdZfdZdej dej fdZxZS)ConvBertIntermediatect||jdk(r0tj|j |j |_n1t|j |j |j|_t|jtrt|j|_ y|j|_ y)Nrr rr0)rZr[r0rrrintermediate_sizer!r r hidden_actstrr intermediate_act_fnrps rNr[zConvBertIntermediate.__init__s     !6#5#5v7O7OPDJ+!--6;S;S`f`q`qDJ f'' -'-f.?.?'@D $'-'8'8D $rsrrvcJ|j|}|j|}|Sr)r!rrqrs rNrzConvBertIntermediate.forwards& =1 00? rsrrs@rNrrs# 9U\\ellrsrcneZdZfdZdej dej dej fdZxZS)ConvBertOutputct||jdk(r0tj|j |j |_n1t|j |j |j|_tj|j |j|_ tj|j|_ y)NrrrT)rZr[r0rrrrr!r rerfrgrhrirps rNr[zConvBertOutput.__init__s     !6#;#;V=O=OPDJ+!33ASAS`f`q`qDJf&8&8f>S>STzz&"<"<= rsrrrvcr|j|}|j|}|j||z}|Srrrs rNrzConvBertOutput.forwardrrsrrs@rNrrs1 >U\\RWR^R^rsrceZdZfdZ d dej deejdeejdeej deej deede ej eejffd Z d Z xZ S) ConvBertLayercbt||j|_d|_t ||_|j |_|j|_|jr*|j st|dt ||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is added)rZr[chunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrrps rNr[zConvBertLayer.__init__s '-'E'E$*62 ++#)#=#=  # #??4&(f ghh"3F";D 08$V, rsrrrrencoder_attention_maskrrvc>|j||||}|d}|dd} |jr?|=t|dstd|d|j |||||} | d}| | ddz} t |j |j|j|} | f| z} | S)N)rrrr+z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) r'r(rzAttributeErrorr+rfeed_forward_chunkr%r&) rqrrrrr-rself_attention_outputsr rcross_attention_outputs layer_outputs rNrzConvBertLayer.forward%s"&   / "0"  2!4(, ??4@4!12$=dVDDD'+&9&9 &%! ' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+rscL|j|}|j||}|Sr)r,r)rqr intermediate_outputr3s rNr0z ConvBertLayer.feed_forward_chunkMs,"//0@A{{#68HI rs)NNNNF) rrrr[r4rrrrrrr0rrs@rNr#r#s -"7;158<9=,1&||&!!2!23&E--. & ( 5 & !) 6 &$D>& u||Xe&7&788 9&Prsr#ceZdZfdZ d dej deejdeejdeej deej deedeed eed e e e ffd Z xZ S) ConvBertEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rZr[r<r ModuleListr1r2r#layergradient_checkpointing)rqr<_rrs rNr[zConvBertEncoder.__init__TsN  ]]5IaIaCb#caM&$9#cd &+#$dsA#rrrrr-routput_hidden_states return_dictrvc |rdnd} |rdnd} |r|jjrdnd} t|jD]\} } |r| |fz} ||| nd}|jr.|j r"|j | j||||||}n | ||||||}|d}|sf| |dfz} |jjs| |dfz} |r| |fz} |std|| | | fDSt|| | | S)Nrrrr#c3$K|]}|| ywrr).0vs rN z*ConvBertEncoder.forward..s=s)last_hidden_stater attentionscross_attentions) r<r) enumerater:r;training_gradient_checkpointing_func__call__tupler)rqrrrrr-rr=r>all_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rNrzConvBertEncoder.forwardZs`#7BD$5b4%64;;;Z;Zr`d(4 VOA|#$58H$H!.7.CilO**t}} $ A A ))!"#)*%! !-!"#)*% ! *!,M &9]1=M 1]4D D '):; 'tn; d^;  u88 9; rsr7cVeZdZfdZdej dej fdZxZS)ConvBertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rZr[rrrr!rrrr transform_act_fnrerfrps rNr[z(ConvBertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrsrrvcl|j|}|j|}|j|}|Sr)r!rVrers rNrz'ConvBertPredictionHeadTransform.forwards4 =1 --m< }5 rsrrs@rNrTrTs$UU\\ellrsrTaK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ConvBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a8 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare ConvBERT Model transformer outputting raw hidden-states without any specific head on top.c|eZdZfdZdZdZdZeejde e e e  ddeej deej"d eej d eej d eej"d eej"d eedeedeedeee ffdZxZS) ConvBertModelc"t||t||_|j|j k7r/t j|j|j |_t||_ ||_ |jyr) rZr[rQrr^rrrembeddings_projectr7encoderr< post_initrps rNr[zConvBertModel.__init__sl  ,V4  F$6$6 6&(ii0E0EvGYGY&ZD #&v.   rsc.|jjSrrr`rqs rNget_input_embeddingsz"ConvBertModel.get_input_embeddingss...rsc&||j_yrr_)rqrMs rNset_input_embeddingsz"ConvBertModel.set_input_embeddingss*/'rsc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr\r:r'r)rqheads_to_pruner:rs rN _prune_headszConvBertModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Crsbatch_size, sequence_length checkpoint output_typerrtrrXrVrrurr=r>rvc ||n|jj}||n|jj}| | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j} |tj| | }|pt|jdr4|jjddd| f}|j| | }|}n&tj| tj| }|j!|| }|j#||jj$}|j||||}t|dr|j'|}|j)|||||| }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer$z5You have to specify either input_ids or inputs_embeds)ryrXrx)rtrVrXrur[)rrrr=r>)r<rr=use_return_dictr%warn_if_padding_and_no_attention_maskrnryr4onesrzrrXrlrmroget_extended_attention_mask get_head_maskr2r[r\)rqrtrrXrVrrurr=r>r{rr|ryr}r~extended_attention_maskrs rNrzConvBertModel.forwards$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZ FCN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z"&"B"B>S^"_&&y$++2O2OP l>iv(  4- . 33MBM 2/!5# % rs) NNNNNNNNN)rrrr[rarcrgrCONVBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr4rrrrrrrrs@rNrYrYs- /0C++D+K+KLi+jk&6$156:59371559,0/3&*<E,,-<!!2!23<!!1!12 < u//0 < E--. <  1 12<$D><'tn<d^< u88 9< l <rsrYcZeZdZdZfdZdej dej fdZxZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.ct|td|_t j |j |j|_t j|j|j |_ y)NgelurT) rZr[r activationrrer^rfrrr!rps rNr[z%ConvBertGeneratorPredictions.__init__PsV (0f&;&;AVAVWYYv1163H3HI rsgenerator_hidden_statesrvcl|j|}|j|}|j|}|Sr)r!r{re)rqr|rs rNrz$ConvBertGeneratorPredictions.forwardWs3 #:;  6 }5 rs) rrrrr[r4rrrrs@rNrxrxMs+KJu/@/@UEVEVrsrxz6ConvBERT Model with a `language modeling` head on top.ceZdZdgZfdZdZdZeejde e e e  ddeej deej"d eej d eej d eej"d eej"d eej deedeedeedeee ffdZxZS)ConvBertForMaskedLMzgenerator.lm_head.weightct||t||_t ||_t j|j|j|_ |jyr) rZr[rYrrxgenerator_predictionsrrr^r]generator_lm_headr]rps rNr[zConvBertForMaskedLM.__init__csR  %f- %A&%I"!#6+@+@&BSBS!T rsc|jSrrr`s rNget_output_embeddingsz)ConvBertForMaskedLM.get_output_embeddingsms%%%rsc||_yrr)rqr`s rNset_output_embeddingsz)ConvBertForMaskedLM.set_output_embeddingsps !0rsrhrirtrrXrVrrulabelsrr=r>rvc | | n|jj} |j|||||||| | } | d} |j| } |j | } d}|Pt j }|| jd|jj|jd}| s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr$rlosslogitsrrE) r<rmrrrrr rr]rrrE)rqrtrrXrVrrurrr=r>r|generator_sequence_outputprediction_scoresrloss_fctrs rNrzConvBertForMaskedLM.forwardss2&1%! 667PQ 223DE  **,H-222t{{7M7MNPVP[P[\^P_`D'),CAB,GGF)-)9TGf$ Ev E$1??.99   rs NNNNNNNNNN)rrr_tied_weights_keysr[rrrrsrtrrurrvrr4rrrrrrrrs@rNrr_sG45&1++D+K+KLi+jk&"$156:59371559-1,0/3&*4 E,,-4 !!2!234 !!1!12 4 u//0 4 E--. 4   1 124 ))*4 $D>4 'tn4 d^4  un$ %4 l 4 rsrcZeZdZdZfdZdej dej fdZxZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.cht|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ ||_ yr) rZr[rrrr!classifier_dropoutrhrgri num_labelsout_projr<rqr<rrrs rNr[z#ConvBertClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH  rsrrvc |dddddf}|j|}|j|}t|jj|}|j|}|j |}|S)Nr)rir!r r<rr)rqrrrs rNrz"ConvBertClassificationHead.forwardse !Q' " LLO JJqM 4;;)) *1 - LLO MM! rsrrs@rNrrs&7 U\\ rsrz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edeee ffdZxZS)!ConvBertForSequenceClassificationct|||j|_||_t ||_t ||_|jyr) rZr[rr<rYrr classifierr]rps rNr[z*ConvBertForSequenceClassification.__init__sH   ++ %f- 4V< rsrhrirtrrXrVrrurrr=r>rvc | | n|jj} |j|||||||| |  } | d} |j| } d}||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&|| j|j}n|| |}n|jjdk(r=t}|| jd|j |jd}n,|jjdk(rt}|| |}| s| f| ddz}||f|zS|St|| | j | j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrXrVrrurr=r>rr regressionsingle_label_classificationmulti_label_classificationr$r)r<rmrr problem_typerrYr4rorr squeezer rrrrrErqrtrrXrVrrurrr=r>rsequence_outputrrrrs rNrz)ConvBertForSequenceClassification.forwards2&1%D 'tnD d^D  u.. /D l D rsrz ConvBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edeee ffdZxZS)ConvBertForMultipleChoicect||t||_t ||_t j|jd|_ |jy)Nr) rZr[rYrrsequence_summaryrrrrr]rps rNr[z"ConvBertForMultipleChoice.__init__/sM  %f- / 7))F$6$6: rsz(batch_size, num_choices, sequence_lengthrirtrrXrVrrurrr=r>rvc L| | n|jj} ||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||| |  } | d} |j | }|j|}|jd| }d}|t}|||}| s|f| ddz}||f|zS|St||| j| jS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr$rrrr) r<rmrCrrnrrrr rrrE)rqrtrrXrVrrurrr=r> num_choicesrr pooled_outputrreshaped_logitsrrrs rNrz!ConvBertForMultipleChoice.forward9s6&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  -- ))%'/!5#  "!*--o> / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rsr)rrrr[rrsrtrrurrvrr4rrrrrrrrs@rNrr'sE+!(()ST &-$156:59371559-1,0/3&*@ E,,-@ !!2!23@ !!1!12 @ u//0 @ E--. @   1 12@ ))*@ $D>@ 'tn@ d^@  u// 0@ @ rsrz ConvBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edeee ffdZxZS)ConvBertForTokenClassificationc`t|||j|_t||_|j |j n |j }tj||_ tj|j|j|_ |jyr)rZr[rrYrrrhrrgrirrrr]rs rNr[z'ConvBertForTokenClassification.__init__s   ++%f- )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rsrhrirtrrXrVrrurrr=r>rvc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|2 'tn2 d^2  u++ ,2 l 2 rsrz ConvBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeee ffdZxZS)ConvBertForQuestionAnsweringct|||j|_t||_t j |j|j|_|jyr) rZr[rrYrrrr qa_outputsr]rps rNr[z%ConvBertForQuestionAnswering.__init__sS   ++%f- ))F$6$68I8IJ rsrhrirtrrXrVrrustart_positions end_positionsrr=r>rvc (| | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr$r) ignore_indexr#)r start_logits end_logitsrrE)r<rmrrsplitrrrrnclampr rrrE)rqrtrrXrVrrurrrr=r>rrrrr total_loss ignored_indexr start_lossend_lossrs rNrz$ConvBertForQuestionAnswering.forwards<&1%H 'tnH d^H  u22 3H l H rsr)Errr)operatorrtypingrrrr4torch.utils.checkpointrtorch.nnrr r activationsr r modeling_outputsrrrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrconfiguration_convbertr get_loggerrr'rurvrOModulerQrrrrrr rrr#r7rTCONVBERT_START_DOCSTRINGrsrYrxrrrrrrrrsrNrsu )) AA1?lluu2   H %/"yx99x*o*8bii4}BII}@  * *Z,299(RYY&:BII:zB biiB Jbii" 2jh]+] ]@299$RTlmM 1M nM `0 U (?U U p S 7S S l G %<G G T Y #:Y Y rs