sg;ZddlZddlmZmZddlZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZdd lmZmZmZmZmZmZmZmZmZe rdd lmZe j<eZ Gd deZ!GddeZ"e jFe"GddeZ$Gddee$Z%Gddee$Z&GddeZ'GddeZ(GddeZ)GddeZ*gdZ+y) N)OptionalTuple)nn)Cache)ALL_LAYERNORM_LAYERS)is_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10logging) LlamaRMSNorm) OlmoConfig) OlmoAttentionOlmoDecoderLayerOlmoFlashAttention2OlmoForCausalLM OlmoModelOlmoPreTrainedModelOlmoSdpaAttentionapply_rotary_pos_emb repeat_kv)_flash_attention_forwardcNeZdZdZdZ dfd ZxZS) Olmo2Configa This is the configuration class to store the configuration of a [`Olmo2Model`]. It is used to instantiate an OLMo2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the [allenai/Olmo2-7B-1124-hf](https://huggingface.co/allenai/Olmo2-7B-1124-hf). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50304): Vocabulary size of the Olmo2 model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`Olmo2Model`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 11008): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 1): Padding token id. bos_token_id (`int`, *optional*): Beginning of stream token id. eos_token_id (`int`, *optional*, defaults to 50279): End of stream token id. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update `max_position_embeddings` to the expected new maximum. See the following thread for more information on how these scaling strategies behave: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an experimental feature, subject to breaking API changes in future versions. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. ```python >>> from transformers import Olmo2Model, Olmo2Config >>> # Initializing a Olmo2 7B style configuration >>> configuration = Olmo2Config() >>> # Initializing a model from the Olmo2 7B style configuration >>> model = Olmo2Model(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` olmo2c t|did|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|d||||_|`y)N vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_heads hidden_actmax_position_embeddingsinitializer_range use_cache pad_token_id bos_token_id eos_token_idtie_word_embeddings rope_theta rope_scalingattention_biasattention_dropout)super__init__ rms_norm_epsclip_qkv)selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r2kwargs __class__s Z/var/www/html/venv/lib/python3.12/site-packages/transformers/models/olmo2/modular_olmo2.pyr1zOlmo2Config.__init__os.  ! # 0 0  !4  !4  " %< 0   & & & !4 " &! "*# $0' ,) M)iii+ r9Nsiluig{Gz?TNigFg@NFgh㈵>)__name__ __module__ __qualname____doc__ model_typer1 __classcell__r6s@r7rrsVKZJ  $!)..r8rc eZdZy) Olmo2RMSNormNr=r>r?r/r8r7rErEr8rEc(eZdZd dedeeffd Z ddejdeejdeejdee de d e d eejd e ejeejee ejffd Z xZS)Olmo2Attentionconfig layer_idxct|||t|j|jz|j |_t|j|jz|j |_y)NrK) r0r1rE num_headshead_dimr2q_normr"k_normr4rJrKr6s r7r1zOlmo2Attention.__init__sY 95"4>>DMM#A6CVCVW "4#;#;dmm#KVM`M`a r8 hidden_statesattention_mask position_idspast_key_valueoutput_attentionsr&cache_positionreturnc |j\} } } |j|j|} |j|j |} |j |}| j | | |j|jjdd} | j | | |j|jjdd} |j | | |j|jjdd}|j||\}}t| | ||\} } |'|||d}|j| ||j|\} }t| |j } t||j }t#j$| | jddt'j(|jz }|#|ddddddd| j*df}||z}t,j.j1|dt"j2j5| j6}t,j.j9||j:|j<}t#j$||}|j| |j| |jfk7r7t?d | |j| |jfd |j|jddjA}|jC| | |jD}|jG|}|sd}|||fS) Nr;r sincosrXr)dimdtype)ptrainingz `attn_output` should be of size z , but is )$sizerPq_projrQk_projv_projviewrNrO transposer" rotary_embrupdaterKrnum_key_value_groupstorchmatmulmathsqrtshaper functionalsoftmaxfloat32toradropoutr.rc ValueError contiguousreshapero_proj)r4rSrTrUrVrWr&rXr5bszq_len_ query_states key_states value_statesr]r\ cache_kwargs attn_weights causal_mask attn_outputs r7forwardzOlmo2Attention.forwards&**, UA{{4;;}#=> [[]!;< {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm ??<>S#7 jRUWZ#[ j  %#&snUL'5'<'    #t~~udmm!L L2CPTP]P]3^2_`$$&')  "++Aq1<<> !))#ud6F6FG kk+.  LL.88r8NNNNFFN)r=r>r?rrintr1rmTensor LongTensorrboolrrrBrCs@r7rIrIsb{bx}b2637*."'59:9||:9!.:9u//0 :9 ! :9  :9:9!!1!12:9 u||Xell3XeELL>Q5RR S:9r8rIceZdZdZdZ d dej deejdeejdee de d e d eejd e ej eej ee ej ffd Z y)Olmo2FlashAttention2aF OLMo2 flash attention module. This module inherits from `Olmo2Attention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cNtj|i|t |_yr)rIr1r _flash_attn_uses_top_left_mask)r4argsr5s r7r1zOlmo2FlashAttention2.__init__s&00 3V2W.W+r8NrSrTrUrVrWr&rXrYc d}|j\} } } |j|j|} |j|j |} |j |}| j | | |j|jjdd} | j | | |j|jjdd} |j | | |j|jjdd}|j||\}}t| | ||\} } |'|||d}|j| ||j|\} }| jdd} | jdd} |jdd}|jr |j nd}| j"}|t$j&k(rt%j(rt%j*}nMt-|j.dr|j.j0}n |jj2j"}t4j7d|d| j9|} | j9|} |j9|}t;| | ||| |||j<|j> }|jA| | |jBjE}|jG|}|sd}||fS) NFr;r r[r<_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rUrvuse_top_left_mask is_causal)$rdrPrerQrfrgrhrNrOrir"rjrrkrKrcr.rarmrtis_autocast_enabledget_autocast_gpu_dtypehasattrrJrweightlogger warning_oncerurrrryrrxrz)r4rSrTrUrVrWr&rXr5r{r|r}r~rrr]r\r dropout_rate input_dtype target_dtyperrs r7rzOlmo2FlashAttention2.forwards"%**, UA{{4;;}#=> [[]!;< {{=1 $((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm ??<>S#7 jRUWZ#[ j  %#&snUL'5'<'$  (??<8L#|4J'??<8L.     % "AAnn  "))#ud6F6FGRRT kk+.  LL.88r8r) r=r>r?r@r1rmrrrrrrrr/r8r7rrs X6:37*."'59V9||V9!!1!12V9u//0 V9 ! V9  V9V9!!1!12V9 u||Xell3XeELL>Q5RR SV9r8rceZdZ d dejdeejdeej deedededeej de ejeejee ejfffd Z xZ S) Olmo2SdpaAttentionrSrTrUrVrWr&rXrYc |r+tjdt| |||||||S|j \}} } |j |j |} |j|j|} |j|} | j|| |j|jjdd} | j|| |j|jjdd} | j|| |j|jjdd} |j| |\}}t!| | ||\} } |'|||d}|j#| | |j$|\} } t'| |j(} t'| |j(} |}||ddddddd| j*df}| j,j.dk(r2|0| j1} | j1} | j1} || dkDrdnd }t2j4j6j9| | | ||j:r |j<nd | }|jddj1}|j|| |j>}|jA|}|d|fS) NaOlmo2Model is using Olmo2SdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rSrTrUrVrWr&rXr;r r[r^cudaTFr<) attn_mask dropout_pr)!rrr0rrdrPrerQrfrgrhrNrOrir"rjrrkrKrrlrqdevicetyperxrmrrrscaled_dot_product_attentionrcr.rrz)r4rSrTrUrVrWr&rXr{r|r}r~rrr]r\rrrrr6s r7rzOlmo2SdpaAttention.forwardWs    [ 7?+-)-"3#-# &**, UA{{4;;}#=> [[]!;< {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm ??<>S#7 jRUWZ#[ j  %#&snUL'5'<' !&&sE43C3CD kk+. D.00r8r) r=r>r?rmrrrrrrrrBrCs@r7rrUs 2637*."'59A1||A1!.A1u//0 A1 ! A1  A1A1!!1!12A1 u||Xell3XeELL>Q5RR SA1A1r8rc(eZdZdedeffd Z d dejdeejdeejdee dee d ee d eejd e ejee ejejfffd ZxZS)Olmo2DecoderLayerrJrKct|||t|j|j|_t|j|j|_|`y)NrMeps)r0r1rErr2post_attention_layernormpost_feedforward_layernorminput_layernormrRs r7r1zOlmo2DecoderLayer.__init__sQ 95(4V5G5GVM`M`(a%*6v7I7IvObOb*c'  r8rSrTrUrVrWr&rXrYc |} |jd|||||||d|\}} } |j|}| |z}|} |j|}|j|}| |z}|f} |r| | fz } |r| | fz } | S)Nrr/) self_attnrmlpr) r4rSrTrUrVrWr&rXr5residualself_attn_weightspresent_key_valueoutputss r7rzOlmo2DecoderLayer.forwards!?Mdnn ? ')%)/) ?  ? ; (*;55mD  =0 !/ 77 F  =0  "  )+ +G  )+ +Gr8r)r=r>r?rrr1rmrrrrrr FloatTensorrrBrCs@r7rrs!{!s!2637*.,1$)59&||&!.&u//0 & ! & $D> &D>&!!1!12& u  (51B1BEDUDU1U+V"WW X&r8rc eZdZy)Olmo2PreTrainedModelNrFr/r8r7rrrGr8rc$eZdZdeffd ZxZS) Olmo2ModelrJc t||tjt |j Dcgc]}t ||c}|_t|j|j|_ ycc}w)Nr) r0r1r ModuleListranger rlayersrErr2normrRs r7r1zOlmo2Model.__init__sc  mmCHIaIaCb ci vy 1 c  !!3!39L9LM  dsA=r=r>r?rr1rBrCs@r7rrsN{NNr8rc$eZdZdeffd ZxZS)Olmo2ForCausalLMrJcDt||t||_yr)r0r1rmodel)r4rJr6s r7r1zOlmo2ForCausalLM.__init__s  ' r8rrCs@r7rrs({((r8r)rrrr),rotypingrrrmr cache_utilsr pytorch_utilsrutilsr r r llama.modeling_llamar olmo.configuration_olmorolmo.modeling_olmorrrrrrrrrmodeling_flash_attention_utilsr get_loggerr=rrrEappendrIrrrrrr__all__r/r8r7rs "  1\\/0   J   H %~*~B < L) @9]@9Fe9.e9PC1*NC1R-(-` . NN((  r8