sgG-8ddlmZddlZgdZGddejj ZGddejjZGdd ejjZGd d ejjZ Gd d ejjZ GddejjZ GddejjjjZGddejjZy))warnN)ReLU6 HardswishELU LeakyReLUSigmoidSoftmaxMultiheadAttentionPReLUcBeZdZdZdfd ZdZdZeddZxZ S)raApplies the element-wise function: :math:`\text{ReLU6}(x) = \min(\max(x_0, x), q(6))`, where :math:`x_0` is the zero_point, and :math:`q(6)` is the quantized representation of number 6. Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.quantized.ReLU6() >>> input = torch.randn(2) >>> # xdoctest: +SKIP >>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32) >>> output = m(input) c2t||||_yN)super__init__inplace)selfr __class__s [/var/www/html/venv/lib/python3.12/site-packages/torch/ao/nn/quantized/modules/activation.pyrzReLU6.__init__,s ! cjtjjj||jSr)torchops quantizedrelu6rrinputs rforwardz ReLU6.forward0s#yy""(( ==rcy)NQuantizedReLU6rs r _get_namezReLU6._get_name3rc,t|jSr)rr)moduse_precomputed_fake_quants r from_floatzReLU6.from_float6sS[[!!rF) __name__ __module__ __qualname____doc__rrr" staticmethodr' __classcell__rs@rrrs+0> ""rrcReZdZdZdfd ZdZdZeddZe dZ xZ S) rzThis is the quantized version of :class:`~torch.nn.Hardswish`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor c||d}t||jdtj|fi||jdtj|fi|yN)devicedtypescale zero_pointrrregister_bufferrtensor)rr5r6r3r4factory_kwargsrs rrzHardswish.__init__CsT$*U;  Well5&KN&KL \5<< +Un+UVrctjjj||j|j Sr)rrr hardswishr5r6rs rrzHardswish.forwardIs)yy"",,UDJJPPrcy)NQuantizedHardswishr r!s rr"zHardswish._get_nameL#rcx|jj\}}tt|t |Sr)activation_post_processcalculate_qparamsrfloatintr%r&r5r6s rr'zHardswish.from_floatOs077IIKzus:77rc8|t|t|Sr)rCrDclsr%r5r6s rfrom_referencezHardswish.from_referenceTs5<Z11r)NNr( r)r*r+r,rrr"r-r' classmethodrIr.r/s@rrr;sAW Q$8822rrcReZdZdZdfd ZdZdZeddZe dZ xZ S) rzThis is the quantized equivalent of :class:`~torch.nn.ELU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor alpha: the alpha constant c@t||||_||_yr)rrr5r6)rr5r6alphars rrz ELU.__init__bs  $rctjjjjj ||j |j|jSr) raonnr functionalelur5r6rNrs rrz ELU.forwardgs>xx{{$$//33 4::t   rcy)N QuantizedELUr r!s rr"z ELU._get_namelsrc|jj\}}tt|t ||j Sr)rArBrrCrDrNrEs rr'zELU.from_floatos677IIKz5<Z#))<77IIKz5<Z#2D2DckkRRrcd|t|t||j|jSr)rCrDrZrrGs rrIzLeakyReLU.from_references%5<Z#2D2DckkRRr)g{Gz?FNNr()r)r*r+r,rCrDboolrrr"rKr'rIr.r/s@rrrys!% W W W W  W  W $SSSSrrcDeZdZdZdedeffd ZdZeddZ xZ S)rzThis is the quantized equivalent of :class:`~torch.nn.Sigmoid`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor output_scaleoutput_zero_pointc>t|||_||_yr)rrrerf)rrerfrs rrzSigmoid.__init__s (!2rctjjj||j|j Sr)rrrsigmoidrerfrs rrzSigmoid.forwards2yy""** 4$$d&<&<  rcr|jj\}}|t|t|Sr)rArBrCrD)rHr%r&rerfs rr'zSigmoid.from_floats9  ' ' 9 9 ;  5&,=(>??rr() r)r*r+r,rCrDrrrKr'r.r/s@rrrs63U3s3  @@rrcReZdZdZdfd ZdZdZeddZe dZ xZ S) r a,This is the quantized version of :class:`~torch.nn.Softmax`. Args: dim: A dimension along which Softmax will be computed (so every slice along dim will sum to 1). scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor cLt|||_||_||_yr)rrdimr5r6)rrmr5r6rs rrzSoftmax.__init__s#  $rc|j}|;d}tjjj d|j|}tj j j|||j|jS)Nsoftmax) rmrrQrR_get_softmax_dimrrrpr5r6)rrrm stacklevels rrzSoftmax.forwardsihh ;J((%%66599; Cyy""**5#tzz4??SSrcy)NQuantizedSoftmaxr r!s rr"zSoftmax._get_names!rc|jj\}}t|jt |t |Sr)rArBr rmrCrDrEs rr'zSoftmax.from_floats677IIKzswwe c*o>>rcN||jt|t|Sr)rmrCrDrGs rrIzSoftmax.from_references377E%L#j/::r)NrXrr(rJr/s@rr r s@% T"??;;rr ceZdZejj j jZdZ e dZ e dZ y)r cy)NQuantizedMultiheadAttentionr r!s rr"zMultiheadAttention._get_names,rctd)NzpIt looks like you are trying to convert a non-observed MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)rHothers rr'zMultiheadAttention.from_floats" 0  rcHtjjj|dddd}||_|j h|j jd}tj|d\}}tj|||tj}t|d||jh|j jd}tjd\}}tj|||tj}t|d||` |`|S)NFT)mappingrremove_qconfigconvert_custom_config_dictbias_k) reduce_rangebias_v)rrP quantizationconvertrr _parameterspop_choose_qparams_per_tensorquantize_per_tensorquint8setattrrin_proj_weight in_proj_bias)rHr| convertedrsczprs r from_observedz MultiheadAttention.from_observedsHH))11 '+ 2 "     '**..x8F55f5QFB..vr2u||LF Ix 0    '**..x8F55UFB..vr2u||LF Ix 0  $  "rN) r)r*r+rrPrQ quantizabler _FLOAT_MODULEr"rKr'rr rrr r sHHHKK++>>M-  rr c eZdZdZ ddedededdffd Zdejddfd Z d ejdejfd Z d Z e dd Z e dZxZS)r a%This is the quantized equivalent of :class:`~torch.nn.PReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor num_parameters: number of parameters: 1, or the number of channels at input. Default: 1 rerfnum_parametersr[Nct|||_||_||_t j |t j}t j|ddt j}|j|y)N)r4rXr)r5r6r4) rrrr5r6rrandnrCrr set_weight)rrerfrwqwrs rrzPReLU.__init__ s` ,! + KKekk :  & &q V rrc||_yr)weight)rrs rrzPReLU.set_weight+s  rrctjjj||j|j |j Sr)rrrprelurr5r6rs rrz PReLU.forward.s4yy""(( 4;; DOO  rcy)NQuantizedPReLUr r!s rr"zPReLU._get_name3r#rc&|jj\}}|t|t||j}|j j}|j j }|||jtjk7rtd|j|j\}} tj|t|t| tj} |j| |SNz9PReLU's weight observer should have dtype quint8 but got ) rArBrCrDrrqconfigr4rrrrr) rHr%r&r5r6qprelufloat_wtobserverwt_scalewt_zpqweights rr'zPReLU.from_float6s77IIKzU5\3z?C4F4FG::##%;;%%' >>U\\ ) KHNNK[\ #446%++ eHos5z5<<  '" rc|t|t||j}|jj}|jj}|||j t jk7rtd|j |j\}}t j|t|t|t j} |j| |Sr) rCrDrrrr4rrrrBrr) rHr%r5r6rrrrrrs rrIzPReLU.from_referenceHsU5\3z?C4F4FG::##%;;%%' >>U\\ ) KHNNK[\ #446%++ eHos5z5<<  '" r)r()r)r*r+r,rCrDrrTensorrrr"rKr'rIr.r/s@rr r sRS ! 69 KN   ELLT U\\ ell  "rr )warningsrr__all__rQReLUrrrrrr rPrr Moduler r rrrs  %"EHHMM%"P2""2<=%((,,=@&S""&SR@ehh@6$;ehh$;N000CC0f@EHHOO@r