sgdZddlmZddlmZmZmZddlZddl Z ddl m Z ddl mZmZddlmZmZmZmZmZmZmZdd lmZmZmZdd lmZmZmZm Z m!Z!d d l"m#Z#e jHe%Z&d Z'dZ(gdZ)dZ*dZ+d)d*dZ,d+d,dZ-Gddej\j^Z0Gddej\jbZ2Gddej\jbZ3dZ4ede4GddeZ5dZ6eGdd ej\jbZ7eGd!d"ej\jbZ8ed#e4eGd$d%e5Z9ed&e4eGd'd(e5eZ:y)-zTF 2.0 OPT model.) annotations)OptionalTupleUnionN)get_tf_activation)TFBaseModelOutputWithPastTFCausalLMOutputWithPast)TFCausalLanguageModelingLossTFModelInputTypeTFPreTrainedModelTFSharedEmbeddingskeraskeras_serializable unpack_inputs)check_embeddings_within_bounds shape_liststable_softmax)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) OPTConfigzfacebook/opt-350mr)rizaHey, are you conscious? Can you talk to me? I'm not conscious. I'm just a little bit of a weirdo.gחc|d}|d}tj||ftjttj}tj j |ddtj j |ddz }|dkDr.tjtj||f|gd}tj|ddddddf|dddfS)zB Make causal mask used for bi-directional self-attention. rraxisN) tffillcastLARGE_NEGATIVEfloat32linalg band_partconcatzerostile)input_ids_shapepast_key_values_lengthbsztgt_lenmasks Z/var/www/html/venv/lib/python3.12/site-packages/transformers/models/opt/modeling_tf_opt.py_make_causal_maskr1@s ! Ca G 77GW%rww~rzz'J KD 99  tQ +bii.A.A$1.M MD!yy"((G-C#DEtLSUV 774dAq()CAq> ::ct|d}||n|}tjd}tj||j}tj |ddddddfdd|df}||z t zS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. rNg?dtype)rr!constantr#r5r*r$)r/r.src_lenone_cst expanded_masks r0 _expand_maskr:Qsxq!G ,g'Gkk#G 774w}} -DGGDD$!12Q7A4FGM m #~ 55r2c2eZdZdZdfd Zddfd ZxZS)TFOPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. c Pd|_t|||jz|fi|y)N)offsetsuper__init__)selfnum_embeddings embedding_dimkwargs __class__s r0rAz(TFOPTLearnedPositionalEmbedding.__init__cs)  $++5}OOr2ctj|tj}tjj |d|zdz }|dd|df}t |||jzS)z3`input_ids_shape` is expected to be [bsz x seqlen].rrN)r!r#int64mathcumsumr@callr?)rBattention_maskr, positionsrFs r0rKz$TFOPTLearnedPositionalEmbedding.callisf:GGNN>N:^KaO a!7!889 w|I 344r2)rCintrDrNr)r,rN)__name__ __module__ __qualname____doc__rArK __classcell__rFs@r0r<r<^sP 5 5r2r<c|eZdZdZ d dfd ZddZ d d dZd dZxZS) TFOPTAttentionz6Multi-headed attention from "Attention Is All You Needc zt|d i|||_||_tj j ||_||z|_|j|z|jk7rtd|jd|d|jdz|_ ||_ tj j||d|_ tj j||d|_tj j||d|_tj j||d |_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gk_proj)use_biasnameq_projv_projout_proj)r@rA embed_dim num_headsrlayersDropoutdropouthead_dim ValueErrorscaling is_decoderDenserYr\r]r^)rBr`rardrhbiasrErFs r0rAzTFOPTAttention.__init__zs "6"""||++G4 !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $ll((T(Q ll((T(Q ll((T(Q  **9t**U r2c tjtj||||j|jfdS)Nrr>rr)r! transposereshaperare)rBtensorseq_lenr-s r0_shapezTFOPTAttention._shapes0||BJJvWdnndmm/\]_kllr2c  |du}t|\}} } |j||jz} |r| |d} |d} n |rE|j|j |d|} |j|j |d|} n|}|j|j |d|} |j|j |d|} t j|d| gd} t j|d| gd} nD|j|j |d|} |j|j |d|} |jr| | f}||jzd|jf}t j|j| | ||} t j| |} t j| |} t| d}t j| | d}t jjt|||jz| |gd ||jz| |fd t| |t jjt||d| |gd |d| |fd t| t j||j }t j|||j| |f|z}t j|||jz| |f}t#|d}|t jjt||jgd|jd t| t j|dt j|||j| |fz}t j|||jz| |f}|j%||}t j|| }t jjt|||jz| |jgd||j| |jfd t| t j&t j|||j| |jfd}t j||| | f}|j)|}t j|||j| |f}|||fS)z#Input shape: Batch x Time x ChannelNrrrr>rT) transpose_bz$Attention weights should be of size z , but is messagez!Attention mask should be of size r4z/Head mask for a single layer should be of size )rrrrtrainingz `attn_output` should be of size rl)rr\rgrqrYr]r!r(rhrarernmatmul debugging assert_equalr#r5rrdrmr^)rB hidden_stateskey_value_statespast_key_valuerLlayer_head_maskrwis_cross_attentionr-r.r` query_states key_states value_states proj_shaper7 attn_weights attn_probs attn_outputs r0rKzTFOPTAttention.calls.T9",]";Wi{{=1DLL@ ."<'*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BKJ99nQ&7%FQOLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > zz$++lGS"I:V ZZ J7 zz, ; Z(+yyztL  !! | $ 4>> !7G 46dnn8LgW^7_6`a|,-/ "   % LL % %>*a'*7a'8R7ST">235 &  WW^<;M;MNN::lS$..'SZ4[\_mmL::lS4>>5I7T[4\]L%l<  & LL % %?+ Et~~EWX"?346 & ::o}E sDNNGWEIL::lS4>>5I7T[4\]L\\,\B ii L9  !! { # 4>> !7DMM :2CRVR_R_3`2ab{+,. " ll JJ{S$..'4==$Q RT` jjsGY.GH mmK0 "$**\CQXZa;b"c L.88r2c|jryd|_t|ddZtj|jj 5|jj dd|jgdddt|ddZtj|jj 5|jj dd|jgdddt|ddZtj|jj 5|jj dd|jgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYAxYw#1swYxYw#1swYxYw#1swYyxYw)NTrYr\r]r^) builtgetattrr! name_scoperYr[buildr`r\r]r^rB input_shapes r0rzTFOPTAttention.builds ::  44 ( 4t{{//0 @ !!4t~~">? @ 44 ( 4t{{//0 @ !!4t~~">? @ 44 ( 4t{{//0 @ !!4t~~">? @ 4T * 6t}}112 B ##T4$@A B B 7 @ @ @ @ @ @ B Bs0 )F32)G)G )G3F=G  GG!)gFT) r`rNrarNrdfloatrhboolrjr)ro tf.TensorrprNr-rN)NNNNF)r{rr|tf.Tensor | Noner}zTuple[Tuple[tf.Tensor]] | NonerLrr~rrwOptional[bool]returnz"Tuple[tf.Tensor, tf.Tensor | None]N) rPrQrRrSrArqrKrrTrUs@r0rWrWws@   VVV V  V  V8m .29=+/,0#(t9 t9+t97 t9 ) t9 * t9!t9 ,t9lBr2rWc\eZdZdfd Z d ddZddZxZS)TFOPTDecoderLayerc t|d i||j|_|j|_t |j|j |jdd|_tjj|j|_ t|j|_tjj!dd|_tjj%|j&d|_tjj%|jd |_tjj!dd |_||_y) N self_attnT)r`rardr[rhh㈵>self_attn_layer_normepsilonr[fc1r[fc2final_layer_normr_)r@rAdo_layer_norm_before hidden_sizer`rWnum_attention_headsattention_dropoutrrrbrcrdractivation_function activation_fnLayerNormalizationrriffn_dimrrrconfigrBrrErFs r0rAzTFOPTDecoderLayer.__init__"s "6"$*$?$?!++'nn00,,  ||++FNN; .v/I/IJ$)LL$C$CDWm$C$n!<<%%fnn5%A<<%%dnn5%A % ? ?Se ? f r2c|}|jr|j|}||ddnd} |j|| ||\}} } |j||}||z}|js|j|}|}|jr|j |}|j |}|j |}|j|}|j||}||z}|js|j |}|| | fS)a( Args: hidden_states (`tf.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`tf.Tensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`tf.Tensor`, *optional*): mask for attention heads in a given layer of size `(decoder_attention_heads,)` past_key_value (`Tuple(tf.Tensor)`, *optional*): cached past key and value projection states training (`bool`, *optional*, defaults to `False`): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). Nr>)r{r}rLr~rv)rrrrdrrrr) rBr{rLr~r}rwoutput_attentions use_cacheresidualself_attn_past_key_valueself_attn_weightspresent_key_values r0rKzTFOPTDecoderLayer.call6s3,!  $ $ 55mDM:H9S>"1#5Y] ?Cnn'3)+ ?M? ; (*;  ]X F  =0 (( 55mDM!  $ $ 11-@M/ **=9 /  ]X F  =0 (( 11-@M02CDDr2c|jryd|_t|ddMtj|jj 5|jj ddddt|ddZtj|jj 5|jj dd|jgdddt|ddZtj|jj 5|jj dd|jgdddt|dddtj|jj 5|jj dd|jjgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYxYw#1swYXxYw#1swYxYw#1swYxYw#1swYyxYw)NTrrrrr)rrr!rrr[rrr`rrrrrrs r0rzTFOPTDecoderLayer.buildws ::  4d + 7t~~223 +$$T* + 4/ 6 Bt88==> N))//tT^^0LM N 4 % 1txx}}- =dDNN;< = 4 % 1txx}}- BdDKK,?,?@A B 4+T 2 >t4499: J%%++T4,HI J J ? + + N N = = B B J Js< H%)H$ )H133H=$)I H!$H.1H:=I Irr)NNNFFF)r{rrLnp.ndarray | tf.Tensor | Noner~rr}4Optional[Tuple[Tuple[Union[np.ndarray, tf.Tensor]]]]rwrrrrrrz4Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]r)rPrQrRrArKrrTrUs@r0rr!s.9=,0OS#(,1$)?E ?E6?E* ?E M ?E ! ?E*?E"?E >?EBJr2rat This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. TensorFlow models and layers in `transformers` accept two formats as input: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should "just work" for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument: - a single Tensor with `input_ids` only and nothing else: `model(input_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})` Note that when creating models and layers with [subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) then you don't need to worry about any of this, as you can just pass inputs like you would to any other Python function! Args: config ([`OPTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. zQThe bare OPT Model outputting raw hidden-states without any specific head on top.ceZdZdZeZdZy)TFOPTPreTrainedModelzz TFOPT Pretrained Model that inheritates from transformers.TFPreTrainedModel Args: config: OPTConfig modelN)rPrQrRrSr config_classbase_model_prefixr_r2r0rrs Lr2ra= Args: input_ids (`tf.Tensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`tf.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Tuple[Tuple[tf.Tensor]]` of length `config.n_layers`) contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Set to `False` during training, `True` during generation output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True. training (`bool`, *optional*, defaults to `False`): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). ceZdZeZd fd ZdZdZdZdZ dZ e d d dZ d dZ xZS) TFOPTDecoderc t|d i|||_|j|_|j |_|j }t|j|j|jd|_ t||jd|_ |jr3|js't j"j%dd|_nd|_|j|jk7rct j"j)|jdd |_t j"j)|jd d |_nd|_d|_t/|j0Dcgc]}t3|d |c}|_t j"j5|j6|_ycc}w) N embed_tokensrembed_positionsrrr project_outF)r[rZ project_inzlayers.r_)r@rAr pad_token_id padding_idx layerdropmax_position_embeddingsr vocab_sizeword_embed_proj_dimrr<rrr_remove_final_layer_normrrbrrrirrrangenum_hidden_layersrrcrd)rBrrErCirFs r0rAzTFOPTDecoder.__init__sz "6" !..))77.   v996;N;NUc  ?    "   & &v/N/N$)LL$C$CDWi$C$jD !$(D !  % %);); ;$||11&2L2LS`kp1qD #ll001C1C,af0gDO#DO#D NSTZTlTlNmn(smDn ||++FNN; os<G c|jSrrrBs r0get_embed_tokenszTFOPTDecoder.get_embed_tokens   r2c||_yrr)rBrs r0set_embed_tokenszTFOPTDecoder.set_embed_tokenss (r2cb|jd|j_||j_y)Nr)shaperrweightrBnew_embeddingss r0set_input_embeddingsz!TFOPTDecoder.set_input_embeddings!s)'5';';A'>$#1 r2c|jSrrrs r0get_input_embeddingsz!TFOPTDecoder.get_input_embeddings%rr2c |\}}tjj||zt|ddt|dd|d|dt ||d}|dkDrt || |z}|S|}|S) NrzXAttention mask shape should be (batch_size, seq_length + past_key_values_length) but is z with input_ids shape z and past length .rtr)r.)r,)r!ryrzrr:r1)rBrLrr,_ seq_lengthexpanded_attn_maskcombined_attention_masks r0_prepare_decoder_attention_maskz,TFOPTDecoder._prepare_decoder_attention_mask(s$ : !! / / ~ &q )!.1!455KK=Y&'q* " *.+b/R >!+F\]`rr $ '&'9 #&&r2c ||n|jj}||n|jj}||n|jj}| | n|jj} | | t d| t |} n|t |dd} n t d|t |dddnd} |1t||jj|j|}|1tj| d| d| zftj}nWtjjt |d| | dzd tj|dd | | dzd  |j!|| } |j#|| | }|j$|j%|}|| z}|rd nd}|rd nd}|rd nd}d|ffD]r\}}| tjjt |dt'|j(d|dt'|j(dt |dd tt+|j(D]>\}}|r||fz }|||nd}||||||nd|\}}}|r||fz }|s9||fz }@|j,|j-|}|j.|j/|}|r||fz }| st1d||||fDSt3||||S)a Args: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`tf.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Tuple[Tuple[tf.Tensor]]` of length `config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. training (`bool`, *optional*, defaults to `False`): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timerzEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr>rr4z'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rtr_ head_maskzThe z should be specified for z layers, but it is for r)rLr~r}c3&K|] }|| ywrr_).0vs r0 z$TFOPTDecoder.call..seferslast_hidden_statepast_key_valuesr{ attentions)rroutput_hidden_statesruse_return_dictrfrrrrr!onesrryrzrrrrlenrb enumeraterrtupler )rB input_ids inputs_embedsrLrrrrr return_dictrwrr, pos_embedsr{all_hidden_statesall_self_attnspresent_key_valuesattn_mask_name attn_maskidx decoder_layerr}layer_self_attnrs r0rKzTFOPTDecoder.call>s|2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "$Y/K  &$]3CR8Kde eIXIdOA,>q,A!B1!Ejk  *9d6G6G6R6R S --i8M  !WWk!nk!nG]6]%^fhfmfmnN LL % %>*1-&Q7=bhh~>VWX>Y=ZZu- A>??oq & )).:PQ ==nk[qr ?? & OOM:M% 2 #7BD0d#,R$,7 *B)C  %NI$ ))y)!, $~..GDKKHXGYZ&y1!45Q8 * #,DKK"8 5 C#!m%55!5D5P_S1VZN@M-2;2G #T- A =M?,="'8&::" ?"44# 5&  , 11-@M    ' ,,];M  -!1 1 )+=?PR`a  -"/ 2/)  r2c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|dddtj|jj 5|jj dd|jjgdddt|dddtj|jj 5|jj dd|jjgdddt|dddtj|jj 5|jj dd|jjgdddt|ddK|jD];}tj|j 5|j dddd=yy#1swYxYw#1swYxYw#1swY_xYw#1swYxYw#1swYxYw#1swYxYw)NTrrrrrrb)rrr!rrr[rrrrrrrrrb)rBrlayers r0rzTFOPTDecoder.builds= ::  4 . :t00556 .!!''- . 4*D 1 =t33889 1$$**40 1 4+T 2 >t4499: S%%++T49P9P,QR S 4 - 9t//445 N  &&dDKK4K4K'LM N 4t , 8t334 U%%tT4;;3R3R&ST U 44 ( 4 &]]5::.&KK%&& & 5 . . 1 1 S S N N U U&&sH I5%J?3J03J!3J(J45I?J JJ%(J14J= r NNNNNNNNNF)rTFModelInputType | NonerrrLrrrrrrrrrrrrrrwrr2Union[TFBaseModelOutputWithPast, Tuple[tf.Tensor]]r)rPrQrRrrrArrrrrrrKrrTrUs@r0rrsL