sgjdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZmZdd lmZmZmZmZdd lmZdd lmZmZmZmZd dl m!Z!ejDe#Z$dZ%dZ&GddejNZ(GddejNZ)GddejTZ+Gdde+Z,e+e,dZ-GddejTZ.GddeZ/dZ0dZ1ed e0Gd!d"e/Z2ed#e0Gd$d%e/eZ3ed&e0Gd'd(e/Z4ed)e0Gd*d+e/Z5y),zPyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)!_prepare_4d_causal_attention_mask*_prepare_4d_causal_attention_mask_for_sdpa))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) BioGptConfigzmicrosoft/biogptrcVeZdZdZdedeffd Zddejdeffd ZxZ S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|y)N)offsetsuper__init__)selfrr __class__s ]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/biogpt/modeling_biogpt.pyr"z)BioGptLearnedPositionalEmbedding.__init__9s$  $++5}Eattention_maskpast_key_values_lengthc|j}tj|dj||zjdz }|dd|df}t|||j zS)z3`input_ids_shape` is expected to be [bsz x seqlen].rdimN)longtorchcumsumtype_asr!forwardr )r#r'r( positionsr$s r%r0z(BioGptLearnedPositionalEmbedding.forward?sp',,.\\.a8@@PSaaggilmm a!7!889 wy4;;677r&)r) __name__ __module__ __qualname____doc__intr"r- LongTensorr0 __classcell__r$s@r%rr4s=FsF3F 8e&6&6 8PS 8 8r&rc `eZdZdZd dedededeeffd Zdejffd Z xZ S) BioGptScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scalec6t||||||_yN)r!r"r=)r#rrr<r=r$s r%r"z"BioGptScaledWordEmbedding.__init__Rs  D&r& input_idsc<t|||jzSr?)r!r0r=)r#r@r$s r%r0z!BioGptScaledWordEmbedding.forwardVswy)D,<,<<.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > Ct{{<#>CCZP 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<AAr&)FTFNNNNNF)r2r3r4r5r6rCboolrrr"r-rDr_rr0r8r9s@r%rFrF[sMG  )-CCC C  C  CC&C>eU\\eCece 488<1526"'vB||vB#5<<0vB!u||!45 vB !. vB "%,,/ vB vB u||Xell3XeELL>Q5RR SvBr&rFc$eZdZ d dejdeejdeeejdeejdeejdedeejeejeeejfffd ZxZ S) BioGptSdpaAttentionr`rarbr'rcrdrecz|s|*tjdt| ||||||S|du}|j \}} } |j |} |r0|.|dj d|j dk(r |d} |d} n |rE|j|j|d|} |j|j|d|} n|}|j|j|d|} |j|j|d|} tj|d| gd} tj|d| gd} nD|j|j|d|} |j|j|d|} |jr| | f}|j| | |} |jr || dkDrd nd }tjjj!| | | ||j"r |j$nd | }|j ||j&| |j(fk7r7t+d ||j&| |j(fd|j |j-dd}|j/|| |j0}|j3|}|d|fS)rgNaBioGptModel is using BioGptSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rarbr'rcrdrrrrhr*TFr) attn_mask dropout_prLrmri)logger warning_oncer!r0rnrVror_rTrUr-rprJrLrrsscaled_dot_product_attentionrlrIrHrPrQr]rqrGrW)r#r`rarbr'rcrdrurZrvrwrxryrzrLrr$s r%r0zBioGptSdpaAttention.forwards  ;   l 7?!1-- /"3 # .T9',,.Wa{{=1 *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7N{{<#> !NN~/E'TU+D[` hh))FF   $&*mmdll G     #t~~w !N N2CRVR_R_3`2ab$$&')  "++Aq1 "))#wG mmK0 D.00r&r) r2r3r4r-rDrrrr0r8r9s@r%rrs488<1526"'f1||f1#5<<0f1!u||!45 f1 !. f1 "%,,/ f1 f1 u||Xell3XeELL>Q5RR Sf1f1r&r)eagersdpaceZdZdeffd Z d dej deej deej deeej dee dee d eejeeejejfffd Z xZ S) BioGptDecoderLayerrMct||j|_t |j |j|j |jdd|_|j|_ t|j|_ |j|_tj |j|_tj$|j|j&|_tj$|j&|j|_tj |j|_y)NT)rGrHrIrJrL)r!r" hidden_sizerGBIOGPT_ATTENTION_CLASSES_attn_implementationnum_attention_headsattention_probs_dropout_prob self_attnhidden_dropout_probrIr hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrSintermediate_sizefc1fc2final_layer_normr#rMr$s r%r"zBioGptDecoderLayer.__init__ks ++1&2M2MNnn0077  11 #F$5$56"(";";$&LL$@!99T^^V-E-EF99V55t~~F " T^^ <r&r`r'rcrbrd use_cacherec||}|j|}||ddnd}|j|||||\}} } tjj ||j|j }||z}|}|j |}|j|}|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|f} |r| | fz } |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nr)r`rbr'rcrdrj) rrrrsrIrlrrrrr) r#r`r'rcrbrdrresidualself_attn_past_key_valueself_attn_weightspresent_key_valueoutputss r%r0zBioGptDecoderLayer.forwardsY0!11-@ :H9S>"1#5Y] >Bnn'3)+/ ?M? ; (*; --mt||VZVcVc-d  =0 !--m< / **=9  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0  "  )+ +G  )+ +Gr&)NNNFT) r2r3r4rr"r-rDrrr FloatTensorr0r8r9s@r%rrjs=|=026268<,1$(<||<!.<"%,,/ < !u||!45 < $D> <D>< u  (51B1BEDUDU1U+V"WW X<r&rc&eZdZdZeZdZdZdZdZ y)BioGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. biogptTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsr)meanstdNrB) isinstancerrSweightdatanormal_rMinitializer_rangerKzero_ Embeddingr<rfill_)r#modules r% _init_weightsz#BioGptPreTrainedModel._init_weightss fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r&N) r2r3r4r5r config_classbase_model_prefixsupports_gradient_checkpointing_supports_sdparr&r%rrs$  L &*#N*r&raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~BioGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare BioGPT Model transformer outputting raw hidden-states without any specific head on top.cteZdZdeffd ZdZdZeejde e e e  ddeej deej"d eej"d eej"d eeeej&d eed eedeedeedeee ffdZxZS) BioGptModelrMct||||_|j|_|j|_|j |_|j|_ |jrtj|j nd}t|j|j|j||_t!|j"|j|_t'j(t+|j,Dcgc] }t/|c}|_t'j2|j|_d|_|j8dk(|_|j=ycc}w)NrB)r=Fr)r!r"rM layerdroprrIrrG pad_token_idr<scale_embeddingmathsqrtr; vocab_size embed_tokensrmax_position_embeddingsembed_positionsr ModuleListrangenum_hidden_layersrlayersr layer_normgradient_checkpointingr _use_sdpa post_init)r#rMr=rwr$s r%r"zBioGptModel.__init__"s   ))11 ++!..7=7M7Mdii 2 23SV 5   t~~t/?/?[  @@^@^`d`n`nommvOgOgIh$iA%7%?$ij ,,t~~6&+#44>  %jsE4c|jSr?rr#s r%get_input_embeddingsz BioGptModel.get_input_embeddings8s   r&c||_yr?rr#values r%set_input_embeddingsz BioGptModel.set_input_embeddings;s !r&batch_size, sequence_length checkpoint output_typerr@r' head_mask inputs_embedspast_key_valuesrrdoutput_hidden_states return_dictrec ||n|jj}||n|jj}||n|jj}| | n|jj} | | t d||} | j } n-| |j dd} |dddddf} n t d||ddjdnd} ||j| }|Ptj|jd|jd| zftj|j}n=|jd| | dzk7r%t d|jdd | | dzd |j|| } |jr|s|t|| || }nt!|| || }|| z}t"j$j'||j&|j( }|j*r%|j(r|rt,j/d d }|rdnd}|rdnd}d}|rdnd}t1|j2D]\}}|r||fz }|j(r%tj4g}||j6kr?|||nd}|j*r5|j(r)|j9|j:|||||ndd||}n||||||nd|||}|d}|r |||rdndfz }|s||dfz }|r||fz }|j=|}|r|nd}| st?d|||||fDStA|||||S)NzDYou cannot specify both input_ids and inputs_embeds at the same timerhz5You have to specify either input_ids or inputs_embedsrrr)dtypedevicez'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rjzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr)r'rcrbrdrc3$K|]}|| ywr?r).0vs r% z&BioGptModel.forward..s=s)last_hidden_staterr` attentionscross_attentions)!rMrdrruse_return_dictrQrnrorr-onesrrrrrr rrsrIrlrrr enumeraterrandr_gradient_checkpointing_func__call__rtupler)r#r@r'rrrrrdrrinput input_shaper(r1r`all_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cacheidx decoder_layerdropout_probabilityrb layer_outputs next_caches r%r0zBioGptModel.forward>s$2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "E**,K  &',,.s3K!!Q(+ETU UDSC^!3A!6!Q(O O9.:N:Nq:Q9RRm)KN:;;km  ((9OP >>"3 8IH ]J $D>J 'tnJ d^J  u?? @J j J r&rzHBioGPT Model with a `language modeling` head on top for CLM fine-tuning.ceZdZdgZfdZdZdZeejde e e e  ddeej deej"d eej"d eej"d eeeej&d eej d eedeedeedeedeee ffdZedZxZS)BioGptForCausalLMzoutput_projection.weightct||t||_t j |j |jd|_|jyNFrO) r!r"rrrrSrroutput_projectionrrs r%r"zBioGptForCausalLM.__init__sJ  !&) !#6+=+=v?P?PW\!] r&c|jSr?rrs r%get_output_embeddingsz'BioGptForCausalLM.get_output_embeddingss%%%r&c||_yr?r )r#new_embeddingss r%set_output_embeddingsz'BioGptForCausalLM.set_output_embeddingss !/r&rrr@r'rrrlabelsrrdrrrec 0| | n|jj} |j|||||||| |  } | d} |j| } d}|{| ddddddfj }|ddddfj }t }||j d|jj|j d}| s| f| ddz}||f|zS|St|| | j| j| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r'rrrrrdrrrrhr)losslogitsrr`rr) rMrrrr^rr\rrrr`rr)r#r@r'rrrrrrdrrrsequence_outputprediction_scoreslm_lossshifted_prediction_scoresloss_fctoutputs r%r0zBioGptForCausalLM.forwardsB2&1%.*s.nU_j--aZ=N=N1OPns58)r)rrreordered_past layer_pasts ` r%_reorder_cachez BioGptForCausalLM._reorder_cache%s=) J ncmnn N r& NNNNNNNNNN)r2r3r4_tied_weights_keysr"r rrrrrrrrrr-r7rrrDrrr0 staticmethodr!r8r9s@r%rrs`55&0++B+I+IJg+hi&5$156:1559@D-1$(,0/3&*7 E,,-7 !!2!237 E--. 7   1 12 7 "%ell(;"<= 7 ))*7 D>7 $D>7 'tn7 d^7  u77 87 j 7 rr&rz BioGPT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeeeeee  dde e jde e jde e jde e jde eee jde e jd e e jd e ed e ed e ed e edeeeffdZxZS)BioGptForTokenClassificationczt|||j|_t||_t |dr|j |j }n |j}tj||_ tj|j|j|_ |jy)Nclassifier_dropout)r!r" num_labelsrrhasattrr(rrDropoutrIrSr classifierr)r#rMr(r$s r%r"z%BioGptForTokenClassification.__init__7s   ++!&) 6/ 0V5N5N5Z!'!:!: !'!;!; zz"45 ))F$6$68I8IJ r&rr@token_type_idsr'rrrrrrdrrrec | | n|jj} |j||||||| | |  } | d} |j| } |j | }d}|t }||j ddk(}|j d|j}tj||j dtj|jj|}|||}n2||j d|j|j d}| s|f| ddz}||f|zS|St||| j| jS) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr'rrrrdrrrrhrr)rrr`r)rMrrrIr,rr\r)r-whererX ignore_indexr/rr`r)r#r@r-r'rrrrrrdrrtransformer_outputsr`rrr active_loss active_logits active_labelsrs r%r0z$BioGptForTokenClassification.forwardEso4&1%= $D>= 'tn= d^=  u++ ,= D = r&r&a The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cveZdZdeffd Zeeeee e  dde e jde e jde e jde eee j de e jd e e jd e ed e ed e ed e edeee ffdZdZdZxZS)BioGptForSequenceClassificationrMct|||j|_t||_t j |j|jd|_|jyr) r!r"r)rrrrSrscorerrs r%r"z(BioGptForSequenceClassification.__init__sS   ++!&) YYv114??O  r&rr@r'rrrrrrdrrrec  | | n|jj} |j|||||||| |  } | d} |j| } ||jdd\}}n|jdd\}}|jj d}n|Vt j||jj jddz j| j}n.d}tj|jjd| t j|| j|f}d}||jj |j"dk(rd |j_nl|j"dkDrL|j$t j&k(s|j$t j(k(rd |j_nd |j_|jj d k(rIt+}|j"dk(r&||j-|j-}n|||}n|jj d k(r=t/}||j1d|j"|j1d}n,|jj d k(rt3}|||}| s|f| ddz}||f|zS|St5||| j6| j8| j: S) r/Nr0rrrhrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr`r)rMrrr:rorr-nesumrrrrr$r2arange problem_typer)rr,r6r squeezerr\rrrr`r)r#r@r'rrrrrrdrrr3r`r batch_sizesequence_length pooled_logitsrrrs r%r0z'BioGptForSequenceClassification.forwards2&1%V $D>V 'tnV d^V  u66 7V D V p()r&r8)6r5rtypingrrrr-torch.utils.checkpointrtorch.nnrrr activationsr generationr modeling_attn_mask_utilsr rmodeling_outputsrrrrmodeling_utilsrutilsrrrrconfiguration_biogptr get_loggerr2rrrrrr;ModulerFrrrrBIOGPT_START_DOCSTRINGrrrr&r8rr&r%rWs )) AA!)u . /   H %(  8r||82 = =[Bbii[B~g1/g1V  RRj*O*8 4nfm 'm m `RTjX-XXv R #8R R j l)&;l)l)r&