sgB&ddlZddlZddlmZmZmZddlZddlmcm Z ddl m Z m Z mZejeZe r6ddlmZmZmZddlmZmZdeej4ej6vZdej:d eej:ej:effd Zd ej:d ej:d ej:dej:def dZ dZ! d.dej:dej:dej:deejDfdZ#e dZ$ejJjMdddk(Z' d/dej:dej:dej:dej:dede(de)d eej:d!ee)d"eed#e(d$ee)d%e(d&eejTd'eejTd(eed)eedeejDf$d*Z+Gd+d,ed-Z,y)0N)OptionalTuple TypedDict)is_flash_attn_2_availableis_flash_attn_greater_or_equallogging)index_first_axis pad_input unpad_input)flash_attn_funcflash_attn_varlen_func window_sizeattention_maskreturncd|jdtj}tj|j dj }|j j }tjtj|dtjd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. )dimdtypeF)as_tupler)rr) sumtorchint32nonzeroflattenmaxitemFpadcumsum)rseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlenss ^/var/www/html/venv/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.py_get_unpad_datar&$s &))b )DmmN224uEMMOG*..0557u||$4!5;;OQWXJ  query_layer key_layer value_layer query_lengthct|\}}}|j\}} } } t|j|| z| | |}t|j|| z| | |}|| k(r't|j|| zd| |}|} |} |}nn|dk(rLd} t j |dztj |j} | dd}|jd}n|dd| df}t||\}}} } ||||| |f| |ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)rdeviceN) r&shaper reshaperarangerr-squeezer )r(r)r*rr+ indices_k cu_seqlens_kmax_seqlen_in_batch_k batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_qs r% _upad_inputr<?sUN6E^5T2I|2(9:FQR]_mFnC Y .C  |$  56  r'c|jd|jd|jd}|jjd|jd|jd}|jjd|jd|jd}|j}t j |jd|j tj}t j||dk(t j|j|j tjf}|jdz}||||||f||ffS)aK This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cummulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cummulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)r-rr) viewsize contiguousrrr0r-rcattensorr)querykeyvalue position_idsr; cu_seq_lens max_lengths r%prepare_fa2_from_position_idsrJs0@ JJr5::b>5::b> :E ..   CHHRL#((2, ?C     # #B 2 2 GE'')L \..q1,:M:MUZU`U`aI)) la' ( LL**,\5H5HPUP[P[ \ K!!#a'J 3y; *DzS]F^ __r'rDrErF target_dtypec||||fS|j}|tjk(rLtj d|d|j |}|j |}|j |}|||fS)aG PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! Args: query (`torch.Tensor`): Input query states to be passed to Flash Attention API key (`torch.Tensor`): Input key states to be passed to Flash Attention API value (`torch.Tensor`): Input value states to be passed to Flash Attention API target_dtype (`torch.dtype`, *optional*): The dtype to convert the attention tensors to. Conversion can be ignored by not providing the target dtype. zThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rrfloat32logger warning_onceto)rDrErFrK input_dtypes r%fa_peft_integration_checkrSs.c5  ++Kemm#~Q  &ff\"& #u r'z2.4.1FLASH_ATTENTION_DETERMINISTIC01F query_states key_states value_states is_causaldropoutrG softmax_scalesliding_windowuse_top_left_masksoftcap deterministic cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kc ,| s|}n |xr|dk7}txr| duxr|jd| kD}|rd| | fini}tr | t} | |d<| | |d<t ||||\}}}|U|jd}t |||||\}}}}}}|\}}|\}}t |||f|||||||d|}t||||}|S|C|/|dk7r;tj|d dk\js|jd}| | t||||\}}}}}}|\} }|\}}n|jd|jd |jd}|jd|jd |jd}|jd|jd |jd}t |||f| ||||||d|}|j|d|jd |jd}|St||||f||d |}|S) a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`float`): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_top_left_mask (`bool`, defaults to `False`): flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. rNrr`r_r)r9r3 max_seqlen_q max_seqlen_k dropout_pr\causalr)rr>)r\ri)_flash_supports_window_sizer. flash_241deterministic_grSr<rr rdiffallr@rJr/r?r )rWrXrYrr+rZr[rGr\r]r^r_r`rarbrcrdrKriuse_sliding_windows flash_kwargsr5r;rH max_seq_lensr9r3r:r4attn_output_unpad attn_outputs r%_flash_attention_forwardrtsZ 0|q0 $kd(BkzGWGWXYGZ]kGkI\MNN#CDacL  +M(5 _%") Y.Gj, .*L*l !!''* Wb *lNLX T j, ; &1" l7C442     &%..'     19j,W X O  ! \Q%6 <]_@`de@e?j?j?l!&&q)  M$9-lJ Vbc YL*lI{L,7 (M=)5 &L,(//L4E4Eb4Irs  -- UU   H %PPB"/48I8I8I/8Z8e8e3f"fELLU5<<WZ;Z5[6FF||FFLL F  FR/`l+/ & <<& & <<&5;;' &R +7 3 **..!@#F#M+/%)$(##0404"&"&*.%H,,H H,,HLL H  H  HH5<<(HE?HSMHHe_HHE,,-HE,,-H 3-!H"3-#H$5;;'%HV 9E r'