sgаTdZddlZddlZddlmZmZmZmZddlZ ddl m Z m Z m Z ddlmZmZmZmZmZmZmZmZddlmZmZmZddlmZmZmZd d lm Z ejBe"Z#d Z$d,d Z%d-d Z&GddejNjPZ)GddejNjPZ*dZ+d.dZ,GddejNjPZ-de j\de/de j\fdZ0GddejNjPZ1GddejNjPZ2eGddejNjPZ3d Z4ed!e4Gd"d#eZ5d$Z6ed!e4Gd%d&e5Z7Gd'd(e5eZ8ed)e4Gd*d+e5eZ9y)/zTF 2.0 Mistral model.N)ListOptionalTupleUnion)TFBaseModelOutputWithPastTFCausalLMOutputWithPast"TFSequenceClassifierOutputWithPast)TFCausalLanguageModelingLossTFPreTrainedModelTFSequenceClassificationLossget_initializerget_tf_activationkeraskeras_serializable unpack_inputs)check_embeddings_within_bounds shape_liststable_softmax)add_start_docstrings%add_start_docstrings_to_model_forwardlogging) MistralConfigrcV|\}}tj||ftjj|j}tj |}tj |dddf|dddfk\d|}|dkDr0tjtj||f||gd}|Htj|d}tj|d}tj||dddg}|Stj|ddddddf|d|||zf}|S)zm Make causal mask used for bi-directional self-attention, supporting both static and dynamic shapes. Ngrdtypeaxisr) tffilldtypesas_dtypeminrangewhereconcatzeros expand_dimstile broadcast_to)input_ids_shaperpast_key_values_lengthbsztgt_lenmask mask_conds b/var/www/html/venv/lib/python3.12/site-packages/transformers/models/mistral/modeling_tf_mistral.py_make_causal_maskr49s#LC 77GW%ryy'9'9%'@'D'D ED!I 88Iag&)D!G*<>$#:A>MOOMCGW3MNM"''-77M 88  t$bgg:m3LTVT^T^TbTb&cer r5c.eZdZdfd ZddZdZxZS)TFMistralRMSNormc @t|di|||_||_y)z? TFMistralRMSNorm is equivalent to T5LayerNorm N)super__init__ hidden_sizevariance_epsilon)selfrGepskwargs __class__s r3rFzTFMistralRMSNorm.__init__fs% "6"& #r5cr|jd|jd|_|jryd|_y)Nweightones)nameshape initializerT) add_weightrGrNbuiltrI input_shapes r3buildzTFMistralRMSNorm.buildns9oo""&  ::  r5cj|j}tj|tj}tjtj |dd}tj |tj||jz}|jtj||zS)NrT)r keepdims) rr!r:r< reduce_meansquaredividesqrtrHrN)rI hidden_states input_dtypevariances r3callzTFMistralRMSNorm.callxsz#))  rzz: >>"))M":dS -DDYDY9Y1Z[ {{RWW]K@@@r5)gư>N)__name__ __module__ __qualname__rFrWra __classcell__rLs@r3rBrBes$Ar5rBc(eZdZdfd ZddZxZS)TFMistralRotaryEmbeddingc t|di|||_||_||_d|jt j d|jdt j|jz zz |_y)Nr7r)startlimitdeltarrD) rErFdimmax_position_embeddingsbaser!r&r<inv_freq)rIrorprqrKrLs r3rFz!TFMistralRotaryEmbedding.__init__se "6"'>$ tyyRXXATXXUV^`^h^h-ilpltlt-tuv r5c|tjtj|tj|jj }tj d||j}tj||gd}tjtj||j }tjtj||j }|d|}tj||j }|d|}tj||j }||fS)Nrzi,j->ijrr) r!r:r&int64rrreinsumr(cossin)rIxseq_lentfreqsemb cos_values sin_valuess r3razTFMistralRotaryEmbedding.calls GGBHHWBHH5t}}7J7J K )Q 6iiR0WWRVVC[!''2 WWRVVC[!''2 ) WWZqww7 ) WWZqww7 J''r5)ii'rb)rcrdrerFrarfrgs@r3ririsw (r5ric|t|ddz}|dd|f}|d|df}tj| |gdS)z*Rotates half the hidden dims of the input.rrk.Nr)rr!r()rx mid_lengthx1x2s r3 rotate_halfrsPAr"a'J 3   B 3  B 99rc2YR ((r5ctjtj|||}tjtj|||}||zt||zz}||zt||zz}||fS)a\Applies Rotary Position Embedding to the query and key tensors. Args: q (`tf.Tensor`): The query tensor. k (`tf.Tensor`): The key tensor. cos (`tf.Tensor`): The cosine part of the rotary embedding. sin (`tf.Tensor`): The sine part of the rotary embedding. position_ids (`tf.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(tf.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r!r*gatherr)qkrvrw position_ids unsqueeze_dimq_embedk_embeds r3apply_rotary_pos_embrss* ..3 5} EC ..3 5} EC3w;q>C/0G3w;q>C/0G G r5c,eZdZfdZdZddZxZS) TFMistralMLPc t|di|||_|j|_|j|_t j j|jdd|_t j j|jdd|_ t j j|jdd|_ t|j|_ y)NF gate_projuse_biasrPup_proj down_projrD)rErFconfigrGintermediate_sizerlayersDenserrrr hidden_actact_fn)rIrrKrLs r3rFzTFMistralMLP.__init__s "6" !--!'!9!9++D,B,BUYd+e||))$*@*@5W`)a ++D,<,? @ @ 8  : : 8 8 @ @s$ 'E0'E'EEEE'rb)rcrdrerFrarWrfrgs@r3rrs;P @r5rr^n_repreturnct|\}}}}|dk(r|Stj|d}tj||d}tj||||z||fS)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rrk)repeatsr )rr!r*repeatreshape)r^rbatchnum_key_value_headsslenhead_dims r3 repeat_kvrsh 2>4==$QRf<8 r5r^attention_maskrpast_key_valueoutput_attentions use_cacherc d|vrtjdt|\} } } |j|} |j |} |j |}t jt j| | | |j|jfd} t jt j| | | |j|jfd} t jt j|| | |j|jfd}t| d}|||djdz }|j||\}}t| | |||\} } |8t j|d| gd } t j|d |gd }|r| |fnd}t!| |j"} t!||j"}t j$| | d t'j(|jz }|||z}t+|d }t j,|| j.}|j1||}t j$||}t j|d}t j|| | |j2f}|j5|}|sd}|||fS)N padding_maskrPassing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`rrr)rxry)rrrvrwrrkrrT) transpose_br)training)warningswarnrrrrr!rrrrrrQrrr(rrmatmulmathr]rr:rrrGr)rIr^rrrrrrrKr/q_len_ query_states key_states value_states kv_seq_lenrvrw attn_weights attn_outputs r3razTFMistralAttention.calls V # MME #=1 UA{{=1 [[/ {{=1 || JJ|c5$..$--%P QXd \\ JJzC0H0H$--#X Y`l || JJ|c5$2J2JDMM%Z [bn  +B/  % .+11"5 5J??# S$8% $  j  %N1$5z#BKJ99nQ&7%FQOL7@*l3dz4+D+DE  t/H/HI yyztLtyyY]YfYfOgg  %'.8L&l< ww|\-?-?@ || $ ii l; ll;\B jjsE4;K;K.LM kk+.  LL.88r5c|jryd|_t|ddXtj|jj 5|jj |jfdddt|ddXtj|jj 5|jj |jfdddt|ddXtj|jj 5|jj |jfdddt|ddftj|jj 5|jj |j|jzfdddyy#1swYHxYw#1swYxYw#1swYxYw#1swYyxYw)NTrrrr) rTrr!rrrPrWrGrrrrrrUs r3rWzTFMistralAttention.buildks} ::  44 ( 4t{{//0 7 !!4#3#3"56 7 44 ( 4t{{//0 7 !!4#3#3"56 7 44 ( 4t{{//0 7 !!4#3#3"56 7 44 ( 4t{{//0 E !!4>>DMM#A"CD E E 5 7 7 7 7 7 7 E Es0 'F80'G'G:4G8GGGG&rb)NNNFFN)rcrdre__doc__rrintrFr!Tensorrrr;rarWrfrgs@r3rrs %I}%I#%INRYY3/3,059,1$)O9yyO9!+O9ryy) O9 !ryy!12 O9 $D> O9D>O9 ryy(299-xbii8H/II JO9bEr5rc*eZdZdedeffd Z d dejdeejdeejdee ejdee d ee d e ejee ejejfffd Z dd Z xZ S)TFMistralDecoderLayerrrc ,t|di||j|_t||d|_t |d|_t|j|jd|_ t|j|jd|_ y)N self_attnrPmlpinput_layernormrJrPpost_attention_layernormrD) rErFrGrrrrrB rms_norm_epsrrrs r3rFzTFMistralDecoderLayer.__init__~s~ "6"!--+FIKPU3/0B0BH[H[bst(8   F$7$7>X) %r5r^rrrrrrc d|vrtjd|}|j|}|j||||||\}} } ||z}|}|j |}|j |}||z}|f} |r| | fz } |r| | fz } | S)a( Args: hidden_states (`tf.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`tf.Tensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(tf.Tensor)`, *optional*): cached past key and value projection states rr)r^rrrrr)rrrrrr) rIr^rrrrrrKresidualself_attn_weightspresent_key_valueoutputss r3razTFMistralDecoderLayer.calls. V # MME !,,]; ?Cnn')%)/ ?M? ; (*;!=0 !55mD /  =0  "  )+ +G  )+ +Gr5c`|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYxYw#1swY~xYw#1swYyxYw)NTrrrr) rTrr!rrrPrWrrrrUs r3rWzTFMistralDecoderLayer.buildsT ::  4d + 7t~~223 +$$T* + 4 % 1txx}}- %t$ % 4*D 1 =t33889 1$$**40 1 43T : Ft<<AAB :--33D9 : : G + + % % 1 1 : :s0 E?%F ?FF$?F  FF!$F-)NNNFFrb)rcrdrerrrFr!rrrr;rarWrfrgs@r3rr}s  }    /3,059,1$)9yy9!+9ryy) 9 !ryy!12 9 $D> 9D>9 ryy(5BII)=#>?? @9v:r5rc8eZdZdZeZdeffd ZdZdZdZ e dde jde e jd e e jd e ee jd e e jd e ed e ede ede edeeeffdZddZxZS)TFMistralMainLayerz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`] Args: config: MistralConfig rc t|di||j|_|j|_|j |_t jj|j|j d|_ t|jDcgc]}t||d|c}|_|j|_ t|j |jd|_||_ycc}w)N embed_tokens) input_dim output_dimrPzlayers.rnormrrD)rErF pad_token_id padding_idx vocab_sizerGrr Embeddingrr&num_hidden_layersr_attn_implementationrBrrr)rIrrKrrLs r3rFzTFMistralMainLayer.__init__s "6"!.. ++!--"LL22''))3 #6#;#;<  "&)GI;:O P  %+$?$?!$V%7%7V=P=PW]^   sC6c|jSrbrrIs r3get_input_embeddingsz'TFMistralMainLayer.get_input_embeddingss   r5c||_yrbrrIr9s r3set_input_embeddingsz'TFMistralMainLayer.set_input_embeddingss !r5cd}t||j|}|$t||j|d}||n||z}|S)N)r.r)r0)r4rr@)rIrrV inputs_embedsr.combined_attention_maskexpanded_attn_masks r3_prepare_decoder_attention_maskz2TFMistralMainLayer._prepare_decoder_attention_masksg#'"3    #9#   %!-nm>Q>Q[fgi[j!k &=&E"K]`wKw $'&r5 input_idsrrpast_key_valuesrrroutput_hidden_states return_dictrc .| | td|t|\} } n|t|\} } } n td| } d}|t|ddd}| |z} |Vtj|| |ztj}tj tj |dd| f}n:tjtj |d| ftj}|1t||jj|j|}|'tj| | ftj}|j|| | f||}|}|rdnd}|rdnd}|rdnd}t|j D]D\}}|r||fz }|||nd}||||||| }|d}|r |||rdnd fz }|s<||d fz }F|j#|}|r||fz }|r|nd}| st%d ||||fDSt'|||| S) NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timezEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrk)rlrmrrrrD)rrrrrrc3&K|] }|| ywrbrD).0vs r3 z*TFMistralMainLayer.call..gstqfgfsts)last_hidden_stater r^ attentions)rrr!r&rtrr*r:rrrrrOr;r  enumeraterrtupler)rIr rrr rrrr r  batch_size seq_lengthrseq_length_with_pastr.r^all_hidden_statesall_self_attnsnext_decoder_cacheidx decoder_layerr layer_outputs next_caches r3razTFMistralMainLayer.calls  ]%>st t  "%/ %: "J  &(2=(A %J Ade e)!"  &%/0B10E%Fq%I "#7:P#P  88,JAW4W_a_g_gL::bnn\1&EJGWXL772::lR\ $D>\ 'tn\ d^\  u// 0\ \ | &r5rax This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. TensorFlow models and layers in `model` accept two formats as input: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should "just work" for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument: - a single Tensor with `input_ids` only and nothing else: `model(input_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})` Note that when creating models and layers with [subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) then you don't need to worry about any of this, as you can just pass inputs like you would to any other Python function! Parameters: config ([`MistralConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. zUThe bare Mistral Model outputting raw hidden-states without any specific head on top.ceZdZeZdZy)TFMistralPreTrainedModelmodelN)rcrdrerr$base_model_prefixrDr5r3r&r&s !Lr5r&a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(tf.Tensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. One formats is allowed: - Tuple of `tuple(tf.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c4eZdZdeffd Zeee ddejde ejde ejde e ejde ejde e d e e d e e d e e d e eeffd ZddZxZS)TFMistralModelrcPt||g|i|t|d|_y)Nr'r)rErFrr'rIrinputsrKrLs r3rFzTFMistralModel.__init__s( 3&3F3'W= r5r rrr rrrr r rc :|j|||||||||  } | S)N r rrr rrrr r r') rIr rrr rrrr r rs r3razTFMistralModel.calls9**)%+'/!5#  r5c|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYw)NTr')rTrr!rr'rPrWrUs r3rWzTFMistralModel.buildsg ::  4$ ' 3tzz/ '   & ' ' 4 ' 's A11A:r#rb)rcrdrerrFrrMISTRAL_INPUTS_DOCSTRINGr!rrrr;rrrrarWrfrgs@r3r*r*s >}>*+CD $.2,059-1$(,0/3&*99!+ryy)  "$ryy/2    * D>$D>'tnd^ u// 0E2'r5r*ceZdZfdZdZdZdZdZdZdZ e e e jd dd ejd eejd eejd eeejd eejdeejdeedeedeedeedeeeffdZ ddZddZxZS)TFMistralForCausalLMc t||g|i|t|d|_|j|_t j j|jdt|jd|_ ||_ y)Nr'rFlm_headrkernel_initializerrP) rErFrr'rrrrrinitializer_ranger6rr,s r3rFzTFMistralForCausalLM.__init__&sv 3&3F3'W=  ++||))   .v/G/GH *   r5c.|jjSrbr'rrs r3rz)TFMistralForCausalLM.get_input_embeddings2zz&&&r5c&||j_yrbr;rs r3rz)TFMistralForCausalLM.set_input_embeddings5"' r5c|jSrbr6rs r3get_output_embeddingsz*TFMistralForCausalLM.get_output_embeddings8s ||r5c||_yrbr@)rInew_embeddingss r3set_output_embeddingsz*TFMistralForCausalLM.set_output_embeddings;s % r5c||_yrbr0)rIdecoders r3 set_decoderz TFMistralForCausalLM.set_decoder>s  r5c|jSrbr0rs r3 get_decoderz TFMistralForCausalLM.get_decoderAs zzr5batch_size, sequence_lengthr rrr rlabelsrrr r rc |j|||||||| |  } | d} |j| } tj| tj} d}|(| ddddf}|ddddf}|j ||}| s| f| ddz}||f|zS|St || | j| j| jS)a Args: labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. r/rNrrlosslogitsr r^r) r'r6r!r:r<hf_compute_lossr r r^r)rIr rrr rrKrrr r rr^rOrNshifted_logitsoutputs r3razTFMistralForCausalLM.callDs0**)%+'/!5#    m,,  #AssF^NAqrE]F''?DY,F)-)9TGf$ Ev E'#33!//))   r5c |rtj|dddfd}|jdd}|C|Atjj |dd}|rtj|dddfd}|||||jddS)NrrT)r exclusiver)r rrr r)r!r*getrcumsum)rIr r rrrKrs r3prepare_inputs_for_generationz2TFMistralForCausalLM.prepare_inputs_for_generations yB'777>>.rT>RL!~~l1b5.A2F #,(.K0   r5c|jryd|_t|ddMtj|jj 5|jj ddddt|ddctj|jj 5|jj |jjfdddyy#1swYzxYw#1swYyxYw)NTr'r6) rTrr!rr'rPrWr6rrGrUs r3rWzTFMistralForCausalLM.builds ::  4$ ' 3tzz/ '   & ' 4D ) 5t||001 ? ""DKK$;$;#=> ? ? 6 ' ' ? ? C %1C, C),C5 NNNNNNNNNN)NNNrb)rcrdrerFrrrArDrGrIrrr2formatr!rrrr;rrr rarWrWrfrgs@r3r4r4%sC '(&*+C+J+JKh+ij $.2,059-1&*$(,0/3&*7 997 !+7 ryy) 7 "$ryy/2 7   * 7 #7 D>7 $D>7 'tn7 d^7  u.. /7 k7 tSW * ?r5r4a The Mistral Model transformer with a sequence classification head on top (linear layer). [`MistralForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cxeZdZfdZdZdZeeejd dde jde e jde e jde e e jd e e jd e e jd e ed e ed e ede edeeeffdZddZxZS)"TFMistralForSequenceClassificationc t||g|i||j|_t|d|_t j j|jdt|jd|_ ||_ y)Nr'rFscorer7) rErF num_labelsrr'rrrrr9r_rr,s r3rFz+TFMistralForSequenceClassification.__init__st 3&3F3 ++'W= \\'' OO.v/G/GH (   r5c.|jjSrbr;rs r3rz7TFMistralForSequenceClassification.get_input_embeddingsr<r5c&||j_yrbr;rs r3rz7TFMistralForSequenceClassification.set_input_embeddingsr>r5rJr rrr rrKrrr r rc `|j|||||||| |  } | d} |j| } t| }d}|jjd}n|t j t jt jj||jj|jddz }t j|dk\|t jt|d|jdz }t j| |dd}n.d}tj|jj dd}||jj|ddk7r t#d t j$|s | d|d|f}|j't j(|dgt j(|d|j*g}||n| }| s|f| ddz}||f|zS|St-||| j.| j0| j2 S) a Args: labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. r/rNrrr) batch_dimsr z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`z=Cannot handle batch sizes > 1 if no padding token is defined.rM)r'r_rrrr!argmaxr:requalrr'rrrrLrcr is_tensorrPrr`r r r^r)rIr rrr rrKrrr r transformer_outputsr^rO logits_shape in_logitssequence_lengthsrN pooled_logitsrRs r3raz'TFMistralForSequenceClassification.callsA.#jj)%+'/!5#)  ,A. M*!&)  ;; # # +! $IIbggbggmmIt{{?W?W&XZcZiZijqst!$&88$)$GGJy}57G7M7MNQRR$ IIf.>1STU #% ##~~../0^^  {{''/LOq4H !`aa<< 01"1|A#68H#HI '' 6B4(@"**YY[]a]l]lXmBnoD%.%:  #%(;AB(??F)-)9TGf$ Ev E1 /??-;;*55   r5c|jryd|_t|ddMtj|jj 5|jj ddddt|ddctj|jj 5|jj |jjfdddyy#1swYzxYw#1swYyxYw)NTr'r_) rTrr!rr'rPrWr_rrGrUs r3rWz(TFMistralForSequenceClassification.builds ::  4$ ' 3tzz/ '   & ' 4$ ' 3tzz/ =   $++"9"9!;< = = 4 ' ' = =rYrZrb)rcrdrerFrrrrr2r[r!rrrr;rrr rarWrfrgs@r3r]r]s0 '(*+C+J+JKh+ij $.2,059-1&*$(,0/3&*O 99O !+O ryy) O "$ryy/2 O   * O #O D>O $D>O 'tnO d^O  u88 9O kO b =r5r])rrb)r):rrrtypingrrrr tensorflowr!modeling_tf_outputsrr r modeling_tf_utilsr r r rrrrrtf_utilsrrrutilsrrrconfiguration_mistralr get_loggerrcr_CONFIG_FOR_DOCr4r@rLayerrBrirrrrrrrrrMISTRAL_START_DOCSTRINGr&r2r*r4r]rDr5r3rys( //   SR 1   H %!6"Au||))A8(u||11(.)8@5<<%%@< [RYY [s [ryy [RE++REjW:ELL..W:td&++d&d&N(V[ 0  BJ[&'-&' &'Rx?35Qx?v o=)AC_o=o=r5