sgxuddlZddlZddlZddlmZddlmZddlmZeje Z GddejZ Gdd ejZGd d ejZGd d ejZGddejZGddejZddZddZddZGddeZGddeZGddeZddZGddeZy) N)nn)Function)loggingc>eZdZdZ dfd ZddZxZS)QuantEmbeddinga Quantized version of `torch.nn.Embedding`. Adds quantization-specific arguments on top of `torch.nn.Embedding`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. momentum (`float`, *optional*, defaults to `0.95`): Momentum for updating the activation quantization range. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. c t |||_||_||_||_||_||_||_tjtj||g|_ |jdtjd|jdtj|j| |_| |_| |_d|_t(j*|_y)Nweight_scaling_factorweight_integerF)super__init__num_dim padding_idxmax_norm norm_typescale_grad_by_freqsparser Parametertorchzerosweightregister_buffer zeros_like weight_bitmomentum quant_modepercentile_modeSymmetricQuantFunctionapplyweight_function) selfnum_embeddings embedding_dimrrrrr_weightrrr __class__s Z/var/www/html/venv/lib/python3.12/site-packages/transformers/models/ibert/quant_modules.pyrzQuantEmbedding.__init__,s "  &  ""4 ll5;; /N#OP  4ekk!nE -u/?/? /LM$  $$5;;c \|jsctjj||j|j |j |j|j|jdfS|j}|jj}|jjd}|jjd}t|j ||d|_|j%|j|j |j&|j"|_tjj||j(|j |j |j|j|j}||j"z|j"fS)Nr F)rr functional embeddingrrrrrrdatadetachminexpandmax$symmetric_linear_quantization_paramsrr r"rr ) r#x positionsincremental_statew w_transformw_minw_maxemb_ints r(forwardzQuantEmbedding.forwardMsT ''KK$$MMNN++KK   KKffmmo !((+!((+%I$//[`bgin%o""22 KK$*>*>@Z@Z --))       MM NN  # # KK 333T5O5OOOr)) NN@FFNffffff?FNN)__name__ __module__ __qualname____doc__rr; __classcell__r's@r(rrs1   ;per-channel mode is not currently supported for activation.)r ractivation_bitact_range_momentumr per_channel percentiler r! act_functionrrrrIrJNotImplementedError)r#rMrNrO channel_lenrr's r(rzQuantAct.__init__s ,"4$&288  %++a. 9  %++a. 9  !5u{{1~ F JJ$ J JJ$ J%&cd dr)c |jjd|jd|jd|jj dd|j j dd S)Nz(activation_bit=z, quant_mode: z , Act_min: z.2fz , Act_max: ))r'r@rMrrIitemrJ)r#s r(__repr__zQuantAct.__repr__si~~&&''78K8K7LM??+;tzz7H6MN )#.a 1 r)c||n||z}|jr|jrJd|jrJd|jj }|jj } | j jdk(r!|j jdk(sJd|jj dkDrF|jj dkr)|j|z|_|j| z|_ n|jdk(rKtj|j||_tj |j| |_ nb|j|jz|d|jz zz|_|j|jz| d|jz zz|_ |js|dfS| |jn|}| |jn|} t|j|| |j |_|3|j!||j|j|j} n.t"j%|||j|j||} |jj'd} | | z|jfS) Nz:percentile mode is not currently supported for activation.rLrz5NaN detected when computing min/max of the activationg&|g&|>r )rO)trainingrPrOr-r/r1isnansumrIrJrNrrr2rMrKrQ FixedPointMulr!view) r#r3pre_act_scaling_factoridentityidentity_scaling_factor specified_min specified_maxx_actrIrJ quant_act_intcorrect_output_scales r(r;zQuantAct.forwards_%8a< == d(d d&'' f)f f'JJNN$EJJNN$E !!#q(U[[]->->-@A-E GF GEzz~~')djjnn.>.G!ZZ%/ !ZZ%/ ((B."YYtzz59 "YYtzz59 !ZZ$*A*AAEQQUQhQhMhDii !ZZ$*A*AAEQQUQhQhMhDii $; +3 +3 "F   4;K;K#  " ) --a1D1DdooW[WnWnoM)//&##''' M $66;;B?33T5L5LLLr))r>FNF)NNNNNr@rArBrCrrWr;rDrEs@r(rGrGrs* e&  $ $<Mr)rGc8eZdZdZ dfd ZfdZddZxZS) QuantLineara8 Quantized version of `torch.nn.Linear`. Adds quantization-specific arguments on top of `torch.nn.Linear`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. bias_bit (`int`, *optional*, defaults to `32`): Bitwidth for the quantized bias. per_channel (`bool`, *optional*, defaults to `False`): Whether or not to use channel-wise quantization. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. ct|||_||_t j t j||g|_|jdt j|j|jdt j|j|r\t j t j||_ |jdt j|j||_ ||_ ||_||_||_ d|_t"j$|_y)Nr fc_scaling_factor bias_integerF)r r in_features out_featuresrrrrrrrbiasrrrObias_bitrr r!r") r#rmrnrorrprOrr's r(rzQuantLinear.__init__s &(ll5;; k/J#KL  -u/?/? /LM 0%++d>O>O2PQ  U[[%>?DI  1A1A$))1L M$$&  $$5;;r)cdt|}d|d|jd|jd}|S)N(z weight_bit=z , quant_mode=rU)r rWrr)r#sr's r(rWzQuantLinear.__repr__s9 G  s,t/}T__|jjd}|jjd}t|j|||j|_|j#|j|j|j$|j |_|j |z}|j -|j#|j |j(d||_|j-dd}||z } tjj| |j&|j*|z|fS)N)rro)r zInput activation to the QuantLinear layer should be globally (non-channel-wise) quantized. Please add a QuantAct layer with `per_channel = True` before this QuantAct layerr )routFrY)rrr+linearrroshaper-r.rOrr/r1r0r2rrkr"rr rprlr^) r#r3prev_act_scaling_factorr6r7r8_r9bias_scaling_factorx_ints r(r;zQuantLinear.forwards==''$++DII'NPTT T'27N7T7TX\7\  _ \ KKffmmo   yy!>HE1yy!>HE1OO%,,Q/EOO%,,Q/E!EdooW\^ceieueu!v"22 KK$*>*>@V@V #447NN 99 $ 4 4TYY uVi jD "9">">q""E++ MM t/B/BIZIZ [^q q   r))Tr= FFNrgrEs@r(riris ns<, # r)ric2eZdZdZdfd ZdZddZxZS)IntGELUa} Quantized version of `torch.nn.GELU`. Adds quantization-specific arguments on top of `torch.nn.GELU`. Args: quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "gelu" or "nonlinear" is given. c0t|||_|dvrtj dd|_|jst j |_d|_d|_ gd|_ |jdxx|jdzcc<y) N) nonlineargeluzForce dequantize geluFg -?)g]m{ҿgMr r) r rrloggerinforGELU activation_fnkconstcoeff)r#r force_dequantr's r(rzIntGELU.__init__7sv $ 1 1 KK/ 0#DO!#D  )  1 A& r)ctj|jd|z }tj|jd|dzz }tj|}tjtj || }|||zdz|zz}|dz|jdz}t j|d|jzz }|d|jzz}||fSNr rr) rfloorrsignr/abs floor_ster!r)r#r{scaling_factorb_intc_intrabs_inty_ints r(int_erfzIntGELU.int_erfGs DJJqMN:; DJJqMNA,==>zz% ))EIIe,uf55Q.67'*TZZ]:4:: 56'!TZZ-7n$$r)c|js|j|dfS||z }|j|||jz \}}d|z}|||zz}||zdz }||z|fS)N?r)rrrr)r#r3rr{ sigmoid_intsigmoid_scaling_factor shift_ints r(r;zIntGELU.forwardVs%%a($. .N".2ll5.SWSYSYBY.Z+ +11 y01'*@@1D~%~55r))Tnoner})r@rArBrCrrr;rDrEs@r(rr,s' % 6r)rc6eZdZdZdfd ZdZdZdZxZS) IntSoftmaxa Quantized version of `torch.nn.Softmax`. Adds quantization-specific arguments on top of `torch.nn.Softmax`. Args: output_bit (`int`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "softmax" or "nonlinear" is given. ct|||_d|_||_|dvrt j dd|_td|j|_d|_ d|_ gd |_ |jd xx|jd zcc<|jd xx|jd zcc<y) Nr|)rsoftmaxzForce dequantize softmaxFrgvq -)gN$?g'|:?rr rr) r r output_bitmax_bitrrrrGactx0rcoef)r#rrrr's r(rzIntSoftmax.__init__rs $ $ 4 4 KK2 3#DOB4??; 1  !  ! $  !  ! $ r)c6tj5tj|jd|z }tj|jd|dzz }ddd|z|zz}|jd|dzz}||fS#1swY-xYwr)rno_gradrr)r#r{rrrzs r(int_polynomialzIntSoftmax.int_polynomials ]]_ BKK ! ~ =>EKK ! ~q/@ @AE BU]e #e +1(99.   B Bs ABBctj5tj|j|z }dddtj||j z}t j||z }|||zz }|j||\}}tjt j|d|j |z zzd}|d|j zz }||fS#1swYxYw)Nrrr/) rrrrr1rrr!rclamp)r#r{rx0_intqrexp_intexp_scaling_factors r(int_expzIntSoftmax.int_exps ]]_ ;[[>!9:F ; %f!45 OOEFN + FQJ &*&9&9!^&L##++ioogdjj1n8M.MNTUV+am;&& ; ;s #C++C4c |js#tjj|ddfS||z }|j dd\}}||z }|j ||\}}|j ||\}}||z }|jdd} tjd|jz| z } tj|| zd|j|jz zz }dd|jzz }||z|fS)NrYrT)rkeepdimrr ) rrr+rr1rrr\rr!rr) r#r3rr{ x_int_maxryrrexp exp_int_sumfactors r(r;zIntSoftmax.forwards==(((3T9 9N"yyRy6 1 !&*ll5.&I###'((74F"G **kkb$k7 DLL;!>?//'F"2Q4<<$//;Y5Z"Z[Q//'77r))Fr) r@rArBrCrrrr;rDrEs@r(rres %"! '8r)rc8eZdZdZdfd ZdZdZddZxZS) IntLayerNorma Quantized version of `torch.nn.LayerNorm`. Adds quantization-specific arguments on top of `torch.nn.LayerNorm`. Args: output_bit (`int`, *optional*, defaults to `8`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "layernorm" or "nonlinear" is given. ct|||_||_t j t j||_t j t j||_ ||_ |dvrtjdd|_ |jdt jd||_d|_d|_t#|j|j|_y)N)r layernormzForce dequantize layernormFshiftr r|r)r rnormalized_shapeepsrrrrrrorrrrrrdim_sqrtrG activation)r#rrrrrr's r(rzIntLayerNorm.__init__s  0ll5;;/?#@A LL-=!>? $ 6 6 KK4 5#DO Wekk!n5$  "4??tOr)c tj5|dz}tj|dd}tjtj|d|j zz j j}|j}tj|j||_tjdt|dt|jdddy#1swYyxYw)NrTaxisrzDynamic shift adjustment: z -> ) rrr\log2sqrtrceilr1rrrint)r#ry_sq_intvar_intr shift_olds r( set_shiftzIntLayerNorm.set_shifts ]]_ \axHiiq$?GZZ 7Q _+D EFKKMRRTE I4::u5DJ KK4S^4DDTZZHYZ [  \ \ \s CC88Dc|j|tj|d|jzz }|dz}t j |dd}|S)z This fallback function is called when overflow is detected during training time, and adjusts the `self.shift` to avoid overflow in the subsequent runs. rTr)rrr!rrr\)r#r y_int_shiftedrrs r(overflow_fallbackzIntLayerNorm.overflow_fallbacksL u!4:: (=>  !#))H1d;r)c|jsx|jdd}||z }tj|dzdd}|tj|j|zz }||j z|j z}|dfS|jetj|jdtj}tj|j|j|_||z }tj|jdd}||z } tj| d|j zz } | dz} tj"| dd} |j$r[| j'd|j(zk\r;|j+| } | j'd|j(zdzksJdtjtj| d|j zz} tjd| z }tj| |zdz } |jdz }|j j,j/|j j,j/z }tj||z }| |z} ||j z}| |z}||fS) NrTr)dtypeg?zfError detected in overflow handling: `var_int` exceeds `self.max_bit` (the maximum possible bit width)li@)rmeanrrrrrortensorrwfloattodevice round_ster!rrr\rZr1rrr-r.)r#r3rryvarnr{mean_intrrrrstd_intrrobias_ints r(r;zIntLayerNorm.forwardsL66q$6/DDA**QT48CEJJtxx#~..ADKK$))+Ad7N ==  QWWQZu{{;A!JJqM,,QXX6DMN"??5::1d:#CD !4:: (=>  !#))H1d; =={{}4<</007{{}q$,,'<<X< //%**W"56DJJF1 23.yy~~$$&$++*:*:*A*A*CD??4.#89 '$++5 N ".  r))r=Frr}) r@rArBrCrrrr;rDrEs@r(rrs P&\ .!r)rcT|jd}t|d|dzz z}t||zdz}tj||j}|dk(r|dz}n#tj| |j }|s |j }|j }||fS)a Calculate the percentile max and min values in a given tensor Args: input (`torch.Tensor`): The target tensor to calculate percentile max and min. lower_percentile (`float`): If 0.1, means we return the value of the smallest 0.1% value in the tensor as percentile min. upper_percentile (`float`): If 99.9, means we return the value of the largest 0.1% value in the tensor as percentile max. output_tensor (`bool`, *optional*, defaults to `False`): If True, this function returns tensors, otherwise it returns values. Returns: `Tuple(torch.Tensor, torch.Tensor)`: Percentile min and max value of *input* rr g{Gz?)r)rwroundrkthvaluevaluesrV) inputlower_percentileupper_percentile output_tensor input_length lower_index upper_index upper_bound lower_bounds r(get_percentile_min_maxrs";;q>L ,?K..+6==K1!Ao ~~uf <CCC !&&( !&&(  ##r)ct|jdk(r)|jdddd}|jdddd}n_t|jdk(r%|jdd}|jdd}n"|jd}|jd}|r3|jd|z j |j |St jd|z |z|zS)a? Quantize single-precision input tensor to integers with the given scaling factor and zeropoint. Args: input (`torch.Tensor`): Single-precision input tensor to be quantized. scale (`torch.Tensor`): Scaling factor for quantization. zero_pint (`torch.Tensor`): Shift for quantization. inplace (`bool`, *optional*, defaults to `False`): Whether to compute inplace or not. Returns: `torch.Tensor`: Linearly quantized value of *input* according to *scale* and *zero_point*. rYr rr)lenrwr^mul_add_round_rr)rscale zero_pointinplaces r(linear_quantizer5s$ 5;;1 2q!Q'__RAq1 U[[ Q  2q!__R+  2__R(  3;$$Z0779 ;;sU{U*Z7 88r)ctj5d|dz zdz }|rhtjtj|j |j gdd\}}tj |d|z }nBt|j |j }tj |d|z }ddd|S#1swYSxYw)a/ Compute the scaling factor with the given quantization range for symmetric quantization. Args: saturation_min (`torch.Tensor`): Lower bound for quantization range. saturation_max (`torch.Tensor`): Upper bound for quantization range. per_channel (`bool`, *optional*, defaults to `False`): Whether to or not use channel-wise quantization. Returns: `torch.Tensor`: Scaling factor that linearly quantizes the given range between *saturation_min* and *saturation_max*. rr rg:0yE>rN)rrr1stackrr)num_bitssaturation_minsaturation_maxrOrrrys r(r2r2Xs$  5 (Q, ! # yyn.@.@.BNDVDVDX-Y_`!aghiHE1KK4014E**,n.@.@.BCEKK4014E 5 L 5 Ls B8CC!c0eZdZdZedZedZy)r zw Class to quantize the given floating-point values using symmetric quantization with given range and bitwidth. ctjdj|j}d|dz zdz }t |||d}tj || |dz }||_|S)a6 Args: x (`torch.Tensor`): Floating point tensor to be quantized. k (`int`): Quantization bitwidth. percentile_mode (`bool`): Whether or not to use percentile calibration. scale (`torch.Tensor`): Pre-calculated scaling factor for *x*. Note that the current implementation of SymmetricQuantFunction requires pre-calculated scaling factor. Returns: `torch.Tensor`: Symmetric-quantized value of *input*. grr F)r)rrrrrrr)ctxr3rrrrr new_quant_xs r(r;zSymmetricQuantFunction.forward}sh"\\#&))%,,7 !a%L1 %a EJ kk+r1q59  r)c|j}t|jdk(r|jdddd}nz'FixedPointMul.forward..sr)c(|jdddS)Nr rY)r^r2s r(r3z'FixedPointMul.forward..sq!R 0r)rr r<) rrwr`rrz_scaling_factorrtypedoublerr.r)rpre_actr_bit_numr5r`rareshaperz_int_A_B new_scaler,eoutputwx_intm1e1output1s r(r;zFixedPointMul.forwards %++ , 1!G0G 'A+  " ]]_! D%,-C%D "#*12I*J'#3C KK*@ @AE',,U\\:B"'' 4::5<<HBRI *Iy)DAqZZ -u||0DDF[[36!23F#X0G%GH,11%,,?&++EKK8>>u||LG #I. $Y/B ++ell3bggell6KK++gb&9: 6);;v{{5;;7!aCC! D! D! Ds H(I88Jcd}|j|j|jz }|j|jz dddd|dfSr})r`rr5)rr identity_grads r(rzFixedPointMul.backward/sU << #'--/#2F2FFM  "S%9%994tTS`bfffr)r?r r r)r(r]r]s<, $2D2Dhggr)r])F))r rrrrtorch.autogradrutilsr get_loggerr@rModulerrGrirrrrrr2r rrr.r]r r)r(rLs$ #   H %PPRYYPPfgMryygMTM "))M `66bii66rD8D8Nb!299b!J!$H 9F@*CX*CZ # # # #DQgHQgr)