sgQXddlmZddlmZmZmZmZddlZddlm Z eGddZ ddeejd eejeefd ejd e d ee f d Z ddeejd eejeefd ejd e d ee f dZddejdej dee fdZddejdej dee fdZ dd eejeefdej dej&d e d ee deejf dZy)) dataclass)ListOptionalTupleUnionN)is_torchdynamo_compilingc^eZdZUdZeed<eed<ddedeefdZ ddededed e jd e e jd fd ee jf d Z dde jded e jdeed e jf dZe dde j"d e jd e jdedeef dZedde jd e jdeefdZede j(defdZe d dee jde jdedeeded ef dZy)!AttentionMaskConvertera9 A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length, key_value_length) that can be multiplied with attention scores Examples: ```python >>> import torch >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter >>> converter = AttentionMaskConverter(True) >>> converter.to_4d(torch.tensor([[0, 0, 0, 1, 1]]), 5, key_value_length=5, dtype=torch.float32) tensor([[[[-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, 0.0000e+00]]]]) ``` Parameters: is_causal (`bool`): Whether the attention mask should be a uni-directional (causal) or bi-directional mask. sliding_window (`int`, *optional*): Optionally, the sliding window masks can be created if `sliding_window` is defined to a positive integer. is_causalsliding_windowNc||_||_|j)|jdkrtd|jdyy)NrzaMake sure that when passing `sliding_window` that its value is a strictly positive integer, not ``)r r ValueError)selfr r s X/var/www/html/venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py__init__zAttentionMaskConverter.__init__9sh",    *t/B/Ba/GstxuHuHtIIJK 0H * batch_size query_lengthkey_value_lengthdtypedevicestrreturnc|jstd|jd||f}||z }d}|ddkDs |j |j |||||j}|S)z Creates a causal 4D mask of (bsz, head_dim=1, query_length, key_value_length) shape and adds large negative bias to upper right hand triangular matrix (causal mask). z"Please use `to_causal_4d` only if z has `is_causal` set to True.Nrrpast_key_values_lengthr )r r __class__r _make_causal_mask) rrrrrr input_shapercausal_4d_masks r to_causal_4dz#AttentionMaskConverter.to_causal_4dBs~~A$..AQQnop p"<0 !1L!@ r?Q $"5"5"A!33'=#22 4Nrattention_mask_2dc|jd|f}d}|ddkDs |jI|jr=| td||z }|j |||j ||j}n|j t d|j|||dj|j }|=|j|jtj|j}|} | S) a Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length, key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is causal, a causal mask will be added. rNrrzpThis attention mask converter is causal. Make sure to pass `key_value_length` to correctly create a causal mask.rz?Sliding window is currently only implemented for causal masking)tgt_len)shaper r rr!rNotImplementedError _expand_maskto masked_fillbooltorchfinfomin) rr%rrrr"r#rexpanded_attn_maskexpanded_4d_masks rto_4dzAttentionMaskConverter.to_4dcs))..q1<@  Oa 4#6#6#B' G&6 %D "!33(//'=#22 4N ,%&gh h"../@%Q\]_Q`.add  $ $   %!/!;!;  )y1}&:&:499R=!&LLaPwwu~ !A %99ekk'3IQV_efhlmsuvD  %->BH ::eood%**&MX`aL   lEKK,>,B,B CD$1$%,,S!WgH^>^__rrDr'c2|j\}}||n|}|ddddddfj|d||j|}d|z }|j|jtj t j |jS)zg Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)r;rBr+r,r.r-r/r0)rDrr'rCsrc_len expanded_mask inverted_masks rr*z#AttentionMaskConverter._expand_masks yy{ W$0'gQdA-.55c1gwORRSXY m+ (()9)9%**)Eu{{SXGYG]G]^^rrJ min_dtypec|jtjk(r td|j tj ||k(ddS)a Attend to all tokens in masked rows from the expanded attention mask, for example the relevant first rows when using left padding. This is required by F.scaled_dot_product_attention memory-efficient attention path. Details: https://github.com/pytorch/pytorch/issues/110213 `expanded_mask` is [bsz, num_masks, tgt_seq_len, src_seq_len] or [bsz, tgt_seq_len, src_seq_len]. `attention_mask` is [bsz, src_seq_len]. The dimension num_masks of `expanded_mask` is most often 1, but it can also be the number of heads in the case of alibi attention bias. For example, if `expanded_mask` is (e.g. here left-padding case) ``` [[[[0, 0, 0], [0, 0, 0], [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[0, 0, 0], [0, 1, 0], [0, 1, 1]]]] ``` then the modified `expanded_mask` will be ``` [[[[1, 1, 1], <-- modified [1, 1, 1], <-- modified [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[1, 1, 1], <-- modified [0, 1, 0], [0, 1, 1]]]] ``` z\AttentionMaskConverter._unmask_unattended expects a float `expanded_mask`, got a BoolTensor.rT)r7keepdim)rr.r-rmulall)rJrLs r_unmask_unattendedz)AttentionMaskConverter._unmask_unattendedsPT   %** ,n   %))MY,FBX\"]!]^^rattention_mask inputs_embeds is_trainingc|jd|jd}}||z}tjjxs0t |tj j xs t}d} ||s|s|dk(s||k(r |||krd} | S|||kr?t|jdk(ry|s$tj|dk(r |dk(s||k(rd} | S)a9 Detects whether the optional user-specified attention_mask & the automatically created causal mask can be ignored in case PyTorch's SDPA is used, rather relying on SDPA's `is_causal` argument. In case no token is masked in the `attention_mask` argument, if `query_length == 1` or `key_value_length == query_length`, we rather rely on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). rrFT) r(r.jit is_tracing isinstancefxProxyr lenrP) rRrSrr rT_rrrXignore_causal_masks r_ignore_causal_mask_sdpaz/AttentionMaskConverter._ignore_causal_mask_sdpas$(--a0-2E2Ea2H<'*@@YY))+vz-/Xv\t\v "  !J!Q&*:l*J#+/?./P%)""! #'7.'H>''(A-EIIn.A$B1$(8L(H)-&"!rN)cpurN)NF)__name__ __module__ __qualname____doc__r-__annotations__intrrr.rrrTensorr$r3 staticmethodSizer!r* FloatTensorfloatrQr_rrr r s<O$ .3   {{  ellE)*  %,, L+/ -  <<- - {{ - #3- -  - ^ '((, ``{{` `!$ ` ! ``: _5<< _ _hsm _ _._((._._._` )-! 7" .7"||7"!$7"! 7"  7"  7"7"rr rRr"rSrr ctd|}|d|z}|*>&?1&D,22 KO>NVcViVi3 ( #  #N,@,@(AQ(F%a.![^=MN %% &. 85eNtj| tj |jj" } | S) a Prepares the correct `attn_mask` argument to be used by `torch.nn.functional.scaled_dot_product_attention`. In case no token is masked in the `attention_mask` argument, we simply set it to `None` for the cases `query_length == 1` and `key_value_length == query_length`, and rely instead on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). Trpr)rRrSrr Nrr6rV)rrcuda)rL)r r.rWrXrYrZr[r r_r$rrr7r3typerQr/r0) rRr"rSrr rtrrXr^r2s r*_prepare_4d_causal_attention_mask_for_sdparz_sg14P^_"2)?? %%'r:mUXX^^+TrXpXrJ/HH%#5% I0 /  .;; NKO-=]EXEXanauau< , %    1 $- 288B#))!1 9  .55::fD5HH EKK 8K8K,L,P,P I   rrDrr'c2tj|||S) Creates a non-causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` dtype (`torch.dtype`): The torch dtype the created mask shall have. tgt_len (`int`): The target length or query length the created mask shall have. rDrr')r r*r}s r_prepare_4d_attention_maskr~s " . .Dw . WWrc2|j\}}||n|}tjjxs0t |tj j xs t}|stj|dk(rytj|||S)r|Nrr}) r(r.rWrXrYrZr[r rPr r*)rDrr'r]rrXs r#_prepare_4d_attention_mask_for_sdpars**A ,g2BG%%'i:dEHHNN+KiOgOiJ %))DAI.%22ESZ2[[rrrchtd|}||dz}|j|d|d|||}|S)a/ Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` Args: input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. dtype (`torch.dtype`): The torch dtype the created mask shall have. device (`int`): The torch device the created mask shall have. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Trprrr6)r r$)r"rrrr rtrrRs r _create_4d_causal_attention_maskrsR(14P^_- B?(55A B)9v6N rr`rb) dataclassesrtypingrrrrr.utils.import_utilsr r rirkrhrvrzrr~rrrrnrrrs"// 8 P"P" P"p%) 1U\\*1uzz5$./1<<1 1 SM 1t%) 7U\\*7uzz5$./7<<7 7 SM 7t XU\\ X%++ XPXY\P] X \ell\5;;\YabeYf\:#$$( uzz5$./ ;; LL  SM  ell r