sgfZ~ddlZddlmZmZddlZddlZddlmcmZ ddlmZm Z dgZ GddejZ y)N)OptionalTuple)nnTensorMultiheadAttentionceZdZejZ dgZ ddededede de de d e ed e ede d dffd Z d Z e dZej j"dZe dZ ddededede ede de ede de d eee effdZ ddededede ede de ede de d eee effdZxZS)r batch_firstN embed_dim num_headsdropoutbias add_bias_kv add_zero_attnkdimvdimreturnc | | d} t |||||||||| f i| tj|j|jfd|i| |_tj|j |jfd|i| |_tj|j|jfd|i| |_ tj|j|jfd|i| |_ tjjjj|_tjj j#|_tjj j#|_tjj j)|_tjj j)|_tjj j)|_y)N)devicedtyper )super__init__rLinearr linear_Qrlinear_Krlinear_Vout_projtorchao quantizedFloatFunctionalq_scaling_product quantization QuantStubquant_attn_outputquant_attn_output_weights DeQuantStub dequant_q dequant_k dequant_v)selfr r r r rrrrr rrfactory_kwargs __class__s ]/var/www/html/venv/lib/python3.12/site-packages/torch/ao/nn/quantizable/modules/activation.pyrzMultiheadAttention.__init__>s%+U;             NNDNN 15 9G   IIt~~ ,0 4B   IIt~~ ,0 4B   $..$..^t^~^ "'!6!6!F!F!H"'!6!6!@!@!B).)>)>)H)H)J&..::<..::<..::<cy)NQuantizableMultiheadAttention)r*s r- _get_namezMultiheadAttention._get_nameps.r.c  t||jk(sJt|dsJd||j|j|j |j du|jdu|j|j|j|j }|j|_|j|_ |j|_ |jj|j_|jj |j_|j"r|j }d}||jz}|j$||ddf}|-t&j(j+||||j,}t&j(j+||j,|j._||j._|j }|}||jz}|j$||ddf}|-t&j(j+||||j,}t&j(j+||j,|j0_||j0_|j }|}|j$|dddf}|-t&j(j+||d|j,}t&j(j+||j,|j2_||j2_nt)j*|j4|j._t)j*|j6|j0_t)j*|j8|j2_|j 4d|j._d|j0_d|j2_nt)j*|j d|j|j._t)j*|j |j|jdz|j0_t)j*|j |jdzd|j2_|j;t&j<j>jA|d}|S)Nqconfigz$The float module must have 'qconfig'rT)inplace)!type _FLOAT_MODULEhasattrr r r in_proj_biasbias_krrrr bias_vr4rweightr _qkv_same_embed_dimin_proj_weightrr Parameter requires_gradrrr q_proj_weight k_proj_weight v_proj_weightevalrr"prepare)clsotherobservedr _start_endr=s r- from_floatzMultiheadAttention.from_floatssE{c/////ui(P*PP( OO OO MM   t + \\ %    JJ JJ      ,,,, ==$)>>#8#8 !&!4!4  $ $%%DFEOO+D))&+q.9Fxx))$vd*;T=O=OP',xx'9'9&&BVBV'WH   $%)H   "%%DFEOO+D))&+q.9Fxx))$vd*;T=O=OP',xx'9'9&&BVBV'WH   $%)H   "%%DF))&'1*5Fxx))$vw-9K9KL',xx'9'9&&BVBV'WH   $%)H   "')||E4G4G'HH   $')||E4G4G'HH   $')||E4G4G'HH   $!!))-!!&)-!!&)-!!&)+&&q5??;*!!&*,&&u%//A:MO*!!&*,&&!(;'>?*!!&  88((0040Hr.c |j|j|j|j|jj ddu|j du|j|j|j|j }|j|jk(sJ|j 2tj|j j|_|j2tj|jj|_|j j \}}tj|j|j _|$tj||j _|jj \}}|j}|j&j \}}|j}|j(j \}} |j}|jrd} | |jz} ||j*| | ddf<|j,t/|dk(sJ||j,| | | } | |jz} ||j*| | ddf<|j,t/|dk(sJ||j,| | | } ||j*| dddf<|j,t/| dk(sJ| |j,| d|Stj||_tj||_tj||_|j,5d|j_d|j&_d|j(_|S||j,d|j||j,|j|jdz| |j,|jdzd|S)zUtility to convert the quantized MHA back to float. The motivation for this is that it is not trivial to conver the weights from the format that is used in the quantized version back to the float. Nrr5)r8r r r r _weight_biasr;rrrr r>rr@ dequantizer<rr=r rrr?r:allrBrCrD) r*fpwbwQbQwKbKwVbVrJrKs r-rPzMultiheadAttention.dequantizesC   NN NN LL ]] ' ' )! ,D 8 [[ $    II II     %%)A)AAAA ;; " T[[%;%;%=>BI ;; " T[[%;%;%=>BI }}))+1\\!,,.9  =!||ABKK ++-B ]]_++-B ]]_++-B ]]_ ! !FBLL(D02B  fTk1n -*27|#|/1t,FBLL(D02B  fTk1n -*27|#|/1t,F,.B  fgqj )*27|#|+-(  "||B/B !||B/B !||B/B &%) "%) "%) "  57BLL1EG  q0@B8:!1 45 r.ctd)NzdIt looks like you are trying to prepare an MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)rGrHs r- from_observedz MultiheadAttention.from_observed s " 0  r.querykeyvaluekey_padding_mask need_weights attn_maskaverage_attn_weights is_causalc 2|j||||||||S)aj Note:: Please, refer to :func:`~torch.nn.MultiheadAttention.forward` for more information Args: query, key, value: map a query and a set of key-value pairs to an output. See "Attention Is All You Need" for more details. key_padding_mask: if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. need_weights: output attn_output_weights. attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch. Shape: - Inputs: - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length. If a BoolTensor is provided, the positions with the value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged. - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length. 3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with ``True`` is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. - is_causal: If specified, applies a causal mask as attention mask. Mutually exclusive with providing attn_mask. Default: ``False``. - average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an effect when ``need_weights=True.``. Default: True (i.e. average weights across heads) - Outputs: - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - attn_output_weights: If ``average_attn_weights=True``, returns attention weights averaged across heads of shape :math:`(N, L, S)`, where N is the batch size, L is the target sequence length, S is the source sequence length. If ``average_attn_weights=False``, returns attention weights per head of shape :math:`(N, num_heads, L, S)`. ) _forward_impl) r*r^r_r`rarbrcrdres r-forwardzMultiheadAttention.forwards0r!!         r.c 2d} d} | |r td|r td|jrd|||fD\}}}|j\} } } |j| k(sJ|jd|jdk(r#|jd|jdk(sJ|j|jz}||jz|jk(sJdt |dz}|j |}|j|}|j|}|jj||}||jtjk(r6tjdd |j!tj"}|j%s1|jtj"k(sJd |j|j'd k(rY|j)d}t+|jd|jd|jdgk7rt-d |j'd k(rUt+|j| |jz|jd|jdgk7r(t-dt-d|j'd|S|jtjk(r6tjdd |j!tj"}|j.|j0| | |j.}|J|j0}|Jtj2||j5d| dg}tj2||j5d| dg}|t7j8|d}|Et7j8|d}n.| Jd| #Jd|j.J|j0J|j;j=| | |jz|j?dd}|>|j;j=d| |jz|j?dd}|>|j;j=d| |jz|j?dd}| ;| jd| |jzk(sJ| jd |k(sJ| }| ;| jd| |jzk(sJ| jd |k(sJ| }|jd}|,|jd| k(sJ|jd|k(sJ|j@ro|dz }tjB|jddf|jd dz}|jDr>tjF||jI|jK|j}tj2||gd}tjB|jddf|jd dz}|jDr>tjF||jI|jK|j}tj2||gd}|t7j8|d}|t7j8|d}|jM|}|jO|}|jQ|}tjR||j?dd }t+|j| |jz| |gk(sJ|>|jtj"k(r|jU|t dn||z }|w|j=| |j| |}|jW|j)dj)d t d}|j=| |jz| |}t7jX|d}t7jZ||jZ|j\}tjR||}t+|j| |jz| |gk(sJ|jr|j=| | |j}n;|j?ddj;j=| | |j}|j_|}|ja|}|jc|}|r6|j=| |j| |}|r|jed}||fS|dfS)Nz#Only allow causal mask or attn_maskz*causal mask not supported by AO MHA modulec3@K|]}|jddyw)rrNN) transpose).0xs r- z3MultiheadAttention._forward_impl..rs PqQ!2 PsrrNz(embed_dim must be divisible by num_headsgz^Byte tensor for `attn_mask` in `nn.MultiheadAttention` is deprecated. Use bool tensor instead.) stacklevelz;Only float and bool types are supported for attn_mask, not r5z,The size of the 2D attn_mask is not correct.z,The size of the 3D attn_mask is not correct.zattn_mask's dimension z is not supportedzeByte tensor for `key_padding_mask` in `nn.MultiheadAttention` is deprecated. Use bool tensor instead.)rrNz#bias cannot be added to static key.z%bias cannot be added to static value.)dimz-inf)ptraining)3AssertionErrorr sizer r floatrrrr! mul_scalarrruint8warningswarntoboolis_floating_pointrr unsqueezelist RuntimeErrorr;r<catrepeatFpad contiguousviewrkrzeros is_quantizedquantize_per_tensorq_scale q_zero_pointr'r(r)bmm masked_fill_ masked_fillsoftmaxr rtr$rr%mean)r*r^r_r`rarbrcrdrestatic_kstatic_vtgt_lenbszembed_dim_to_checkhead_dimscalingqkvr;r<src_lenk_zerosv_zerosattn_output_weights attn_outputs r-rgz MultiheadAttention._forward_implXs   Y !FG G  !MN N    PUC>T^^3 t~~ % 7 6 5 6 7/T) MM%  MM#  MM%   " " - -a 9  %++- /  &LL4 ++-EJJ1N _LY__L]^ _N}}!#%//2  ()aA -LL&'UVVA% ()$..(JJqMHHQK. ''UVV",Y]]_,==NO  ',<,B,Bekk,Q MM+  0225::>  ;; "t{{'>H$4))))))IIq&--3":;<IIq&--3":;<( !i 8I#/'(uu-=v'F$'N)NN''P)PP';;& &&;;& && LLN  t~~)=x H R RSTVW X = ##Bdnn(&>1&>&E# 33 3$ $r.) gTFFNNFNN)NTNTF)__name__ __module__ __qualname__rrr8 __constants__intrwr}rrr2 classmethodrLrjitunusedrPr]rrrhrg __classcell__)r,s@r-rrs))M*V#OM !#""!0=0=0= 0=  0=  0=0=sm0=sm0=0= 0=d/GGR YYIIV  .2!&*%)B B B  B #6* B  B F#B #B B  vx'' (B R.2!&*%)N%N%N% N% #6* N%  N%F#N%#N%N% vx'' (N%r.) rztypingrrr torch.jittorch.nn.functionalr functionalrr__all__rr1r.r-rs8"  X%..X%r.