sgΓ UddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m cm cmZddlm Z ddlm cmZddlmZddlmZmZmZddlmZmZddlmZddlm Z d d l!m"Z"m#Z#m$Z$gZ%eee&d <d e jNd e(de jNde jNfdZ)e jTe jVe jXe jZiZ.Gdde j^Z0Gdde j^Z1de0de1de jNfdZ2GddedddgZ3e0jie jje jXZ6e1jie jne jZZ8e3e6e8Z9de:fdZ;de:fdZd#e j^de:fd$Z?d%e d&ed'ee@e j^fde ee ee1ffd(ZAd)e d'ee@e j^fdee fd*ZBd!e d'ee@e j^fdee0fd+ZCd!e d'ee@e j^fdee jNfd,ZDd!e d'ee@e j^fddfd-ZEd!e d'ee@e j^fd.e jNd/ee jNddf d0ZFd)e d&ed'ee@e j^fd.e jNd/ee jNddf d1ZGd)e d'ee@e j^fddfd2ZHd&ed!e d3e fd4ZId&ed'ee@e j^fdee@e1ffd5ZJd&ed'ee@e j^fd6ee@e1fddfd7ZKd&efd8ZLd9e j^d:e j^d;e jNdee@eMffd<ZNd=ee@eMfd>e(defd?ZOy)@N) namedtuple)AnyDictListOptionalTuple)_get_observed_graph_module_attr) _with_args ObserverBasePerChannelMinMaxObserver) _parent_namecheck_min_max_valid) GraphModule)Node)get_new_attr_name_with_prefixmaybe_get_next_modulenode_arg_is_weightCUSTOM_MODULE_SUPP_LISTscaleaxisinputreturncldg|jz}|j|||<|j|S)zMReshapes the scale so that we can multiply it to the input by the given axis.r)ndimsizeview)rrr new_shapes U/var/www/html/venv/lib/python3.12/site-packages/torch/ao/quantization/fx/_equalize.py reshape_scaler s3ejj Ijj&IdO ::i  ceZdZdZej ej dddf dfd ZdZdZ dZ dZ e e ZxZS) _InputEqualizationObservera.Observer for tracking the running min/max values of input columns, and computing the quantization parameters for the overall min/max input values. Args: dtype: Quantized data type qscheme: Quantization scheme quant_min: Minimum quantization value. If unspecified, it will follow the 8-bit setup. quant_max: Maximum quantization value. If unspecified, it will follow the 8-bit setup. The running minimum/maximum :math:`x_\text{min/max}` are computed in the same way as :class:`~torch.ao.quantization.observer.PerChannelMinMaxObserver`, with the difference that the running min/max values are stored per column. This observer is intended to be used along with a WeightEqualizationObserver to calculate the equalization scale. Nct||tjtjhvr t d||_||_t|}td||||||_ tjd|_ g|_ y)Nz Input qscheme must be per-tensorrch_axisdtypeqscheme quant_min quant_maxfactory_kwargs)super__init__torchper_tensor_affineper_tensor_symmetric TypeErrorr'r((qsheme_mapping_per_tensor_to_per_channelr input_obstensorequalization_scaleequalization_shapeselfr'r(r)r*r+per_channel_qscheme __class__s rr-z#_InputEqualizationObserver.__init__?s  522E4N4NO O>? ?  FwO1')  #(,,q/-/r!c|jdk\r|jdks tddg|jz|_|jd|jd<|j |S)N>InputEqualizationObserver only supports Linear and Conv layersr)r ValueErrorr6rr3)r8x_origs rforwardz"_InputEqualizationObserver.forward\se q V[[A%5P  $%# "3%+[[^"~~f%%r!cZ|jj|jjfSN)r3min_valmax_valr8s rget_input_minmaxz+_InputEqualizationObserver.get_input_minmaxhs!&&(>(>??r!c|jdk(r|tjdk(rytj||j|_y)Nr)nelementr.r4reshaper6r5r8r5s rset_equalization_scalez1_InputEqualizationObserver.set_equalization_scaleksE  & & (A -2D UV2W "'--  7 7# r!c|jjdk(r8|jtjdk(rt j dy|j \}}t|jd|}tjtj||}tjtj||}||fS)z!Returns the scaled min/max inputsrz}Must call calculate_equalization_scale before calling calculate_scaled_minmax. Will not scale the next quantization observer.NNr) r5rIr.r4warningswarnrGr minmulmax)r8 min_inputs max_inputsequalization_scale_reshapedmin_input_scaledmax_input_scaleds rcalculate_scaled_minmaxz2_InputEqualizationObserver.calculate_scaled_minmaxts  # # , , .! 3''5<<?: MMC  $(#8#8#: Z&3  # #Q ' #!99UYYz;V%WX 99UYYz;V%WX!111r!rN)__name__ __module__ __qualname____doc__r.quint8r/r-rArGrLrY classmethodr with_args __classcell__r:s@rr#r#,sS(ll'' 0 0: &@ 20J'Ir!r#c|eZdZdZej ej dddf dfd ZdZdZ dZ e e Z xZS)_WeightEqualizationObserveraObserver for tracking the running min/max values of weight columns and rows, and computing the quantization parameters for the weight rows. Args: dtype: Quantized data type qscheme: Quantization scheme quant_min: Minimum quantization value. If unspecified, it will follow the 8-bit setup. quant_max: Maximum quantization value. If unspecified, it will follow the 8-bit setup. This observer is made up of 1 PerChannelMinMaxObserver `weight_col_obs` used to record the running minimum and maximum of columns of incoming weight tensors. This observer is intended to be used along with an InputEqualizationObserver to calculate the equalization scale. The running minimum/maximum :math:`w_\text{min/max}` are computed in the same way as :class:`~torch.ao.quantization.observer.PerChannelMinMaxObserver`. Nct|||_||_d|_|}|t j t jhvr t|}td||||||_ t jd|_ y)Nrr%) r,r-r'r(r&r.r/r0r2r weight_col_obsr4r5r7s rr-z$_WeightEqualizationObserver.__init__s    % u..0J0JK K"J7"S 6')  #(,,q/r!cv|jdk\r|jdks td|j|S)Nr<r=r>)rr?rg)r8w_origs rrAz#_WeightEqualizationObserver.forwards< q V[[A%5P ""6**r!cZ|jj|jjfSrC)rgrDrErFs rget_weight_col_minmaxz1_WeightEqualizationObserver.get_weight_col_minmaxs%##++T-@-@-H-HIIr!c||_yrC)r5rKs rrLz2_WeightEqualizationObserver.set_equalization_scales "4r!rZ)r[r\r]r^r.qint8r/r-rArkrLr`r rarbrcs@rreresN,kk'' 2 28+J5J'Ir!rer3 weight_obsc|j\}}|j\}}t||r t||s*tjdt j dS|j|jk(s)tdd|jd|jdzt j||z ||z z }d||dk(<t j|ddd}|S) aCalculates the equalization scale and sets the equalization_scale value in the observers. Args: input_obs: Observer that tracks the ranges for the input columns weight_obs: Observer that tracks the ranges for the weight columns ztMust run observer before calling calculate_equalization_scale. Returning default equalization scale torch.tensor(1).rz6Input and Weight must have the same column dimension. zFound z and z shapes instead.g)nanposinfneginf) rGrkrrOrPr.r4shaper?sqrt nan_to_num)r3rnrTrU min_weights max_weightsr5s rcalculate_equalization_scalerxs )99;Z!+!A!A!C[+ J 3  [ 9  F ||A    1 1 1 Dz''(k.?.?-@@PQ R   { "zJ'>?56)S01))* inp_obs1 -> eq_obs1 -> linear1 -> out_obs1 -> eq_obs2 -> linear2 -> out_obs2 If the node being passed in is the linear1 node, then we want to return eq_obs2, the following equalization observer for linear2. However, if there are no connecting layers: x -> inp_obs1 -> eq_obs1 -> linear1 -> out_obs1 -> add Then we want to return None. In the case of an unfused linear-relu layer with a connecting linear layer: linear1 -> relu -> out_obs1 -> eq_obs2 -> linear2 -> out_obs2 Since it is unfused, we want to skip over the relu layer and return eq_obs2, the following equalization observer for linear2. N)target_functional_type) rrr}ReLUrrelur r#rr|)rrmaybe_relu_nodemaybe_obs_nodemaybe_eq_obs_node maybe_eq_obss rmaybe_get_next_input_eq_obsrs( &dG 44 4,D'277CO/ '!&&   " dG\: "?G\ J -!; 3012L l$> ?? ? r!ct||}|rL|jjdk(r#|jtjdk(ry|jSy)aUIf the next next node is an InputEqualizationObserver then we want to return its equalization scale, else we return 1 This is used in the case where there are two connecting linear layers: linear1 -> LinearOutObs -> InputEqObs -> linear2 In this case, the node given is linear1 and we want to locate the InputEqObs. rN)rr5rIr.r4)rrnext_inp_eq_obss r!maybe_get_next_equalization_scalersR2$@O  . . 7 7 9Q >22ell1oE111 r!c8|t|j}t|tsJ|jd}t|t sJ|t|j}t|t sy|j\}}||y||_||_ y)zScales the following input quantization observer's min/max values by updating the values with the scaled min/max values calculated by the input equalization observer rN) rrr|r#rrr rYrDrE)rr input_eq_obsinput_quant_obs_nodeinput_quant_obsrWrXs rscale_input_observerrs 3t{{+,L l$> ?? ?99Q< *D 11 1c"6"="=>?O o| 4)5)M)M)O&&$4$<.O.Or!r5next_equalization_scalec$|yt|t|jr|t|jd}n|t|j}t|s t |sJ|j }t |tjsJt|d|}tj|tj|}|tj||_yt|d|}tj||}tj||_|j} | yt | tjsJt|d| }tj| |} tj| |_y)aScale the weights for input-weight equalization by multiplying the weight by 1/equalization_scale and next_equalization_scale Args: node: Current node whose weights we want to scale equalization_scale: Current node's calculated equalization scale next_equalization_scale: Next node's calculated equalization scale if the following node needs to be equalized, 1 otherwise Nrr)rrrrrrr|r.Tensorr rR reciprocalr} Parameterbias) rrr5r op_modulerrV scaled_weight next_equalization_scale_reshapedr scaled_biass rscale_weight_nodersd!)'#dkk2B*CDC ,-a0 C ,- *  ,Y 78 8   F fell ++ +#00BAv"NIIfe&6&67R&STM&<< 6 (55LaQW'X$IIm-MNM||M2I >>D | dELL )) )(55LaQU'V$))D"BCK\\+.INr!c"|yt||}|y|jd}|yt|tr&t|t |j t sJ|jd}|yt|tr|jdk(sJt|j \}} t||| } t|d| } tj| tj| } |t||| | yt|d| } tj| | } t||| | tj|j!t |j | sJd}|jD]5}t|ts|jdk(s$d|j"vs3|}n|yt|j \}}t|||}t|d|} tj|| }t||||y)z-Scales the weight value for functional layersNrget_attrrr)rrr|rrrr rr getattrr r.rRrsetattrallclose get_bufferr)rrrr5rweight_eq_obs_nodeweight_quant_obs_noderweight_parent_name weight_namerrVrr bias_noderbias_parent_name bias_namerrs rscale_weight_functionalr&s(!6gwG!/33A6$ +T 2z)0012L8  (,,Q/K k4 ([^^z-II I&2;3E3E&F# W/0+ >F #00BAv"NIIfe&6&67R&STM&*+[-H(5M($IIm-MNM G& 'mD >>%**3{/A/A+BC] SS SI  dD !dgg&;$))@SI   ".y/?/?"@i 7+,i 8D(55LaQU'V$))D"BCK G$ %y+>r!c t||}|y|jd}|yt|tsJ|t |j }t|t |j t sJ|jy)ztGiven the operation node, we want find the corresponding quantization observer and reset its min/max values Nr)rrr|rrrr reset_min_max_vals)rrrrweight_quant_obss rclear_weight_quant_obs_noderts6gwG!.33A6$ +T 22 2s#8#?#?@A gc"7">">?@, OO O'')r! prev_nodect|jj}|D]}|j|||jj |y)ziRemoves the given node from the model by replacing all of its users with the given previous node N)listrrreplace_input_withgraph erase_node)rrr orig_users user_nodes r remove_nodersL djjoo'(J6 $$T956 KK4 r!ci}|jjD]7}|jdk(st||jt s2||j}t|t sJt |||\}}||h|jdk(rt|t|jr;|t|jd}t|sJ||jn(||t|jjt||}|j||j||||j<:|S)aUpdate all of the observer's equalization scale. For each InputEqualizationObserver, we will find the location of the next WeightEqualizationObserver, create it, and calculate the equalization scale based on the two observers. We will then return a dictionary mapping operation node names to the corresponding WeightEqualizationObservers for that operation. rr)rnodesrr|rr#rrrrrrxrLr) rrweight_eq_obs_dictrrrrrr5s rupdate_obs_for_equalizationrsB !!= 77m # DKK "<) #4;;/Ll,FG GG%B4PW%X "G]-"7zz]*6gc'..>Q6RS$S%89!"E"EF">m"   / /0B C  0 01C D/< w|| ,9=< r!rc |jjD]}|jdk(rht||jt rJ|j d}|j d}t||sd|jvrt|||xt|||jj|5t|jdz}||}t||||jj|jjd|}ddd|jj!5||f} |jjdt"j$| } ddd|j'| t|||||j)|jd|j)|j} t| t*sJ| j} | j-dk(r| t#j.dk(rd} t1||} |jdk(rt3||| | +|jdk(rqt5|||| | t7||}|yt|t9|jt*sJt;|||j d}t|||t=d d |jd |jd zy#1swYxYw#1swYxYw) a Converts the equalization operations and updates the other nodes in the following way: - Removes the input equalization observers and inserts a mul operator along with an equalization scale node wherever applicable (we do not want to insert a mul operator between connecting linear layers). - Updates the input quantization observers with the scaled input min/max values. - Scales the weights by the current and next equalization scales. - Removes the weight equalization observer node if it exists. Before (after prepare): weight values | WeightQuantObs | WeightEqObs | x -> InpQuantObs -> InpEqObs -> linear -> OutQuantObs After this function: scaled weight values | equalization scale WeightQuantObs | | x -> mul -> InpQuantObs (scaled min/max) -> linear -> OutQuantObs After convert: equalization scale scaled weight values | | x -> mul -> quantize_per_tensor -> quantized::linear Note that although the equalization observer appeared after the quantization observer after prepare_fx, the mul node appears before the quantization node after convert_fx. This is because placing the equalization observer after the quantization observer in prepare_fx would allow us to keep the invariant that the graph before the current node inserts its observers is not modified. Having the equalization observer before the quantization observer would also cause some inconsistences between the ordering of the quantization and equalization observers. For example, a single linear layer would look like: x -> InpEqObs1 -> InpQuantObs1 -> linear1 -> OutQuantObs1 But between two connected linear layers, it would look like: linear1 -> OutQuantObs1 -> InpEqObs2 -> linear2 -> OutQuantObs2 rrr_equalization_scalerNrrz=Expected operation node to be 'call_module' or 'call_functionzInstead got node z as 'z'.)rrrr|rr#rrrrrinserting_beforerrr5 create_nodeinserting_afterr.rRrrrerIr4rrrrrrr?)rrrrinp_quant_obs_noderget_new_eq_scale_namer eq_scale_nodeinputsmul_noderr5maybe_next_equalization_scalers rconvert_eq_obsrsf !!] 77m # DKK "<) "&1 *//2I+9g>Y^^+E4);< !w /--.@A J(ENN%::)%-W5tWT[[%9%L%LM % 7 7 D I  J,,]; W#]3 ;;22?EIIvV W  1 1)X F t%7 8  # #DIIt 4 @.22499=Mm-HI II!.!A!A #++-2&%,,q/9%)",Mg- ) ww-'!'#57TO+'&1 &B$%P"%-!C 2 9 9:;=X ,D':/33A6 E#5yA S)$))E$''"EFu]: J J W Ws6A!K-:0K:-K7 :L ct|jd}t||}t|||t ||j S)zjReference function which applies changes needed for equalization, but does not quantize the nodes F)remove_duplicate)dict named_modulesrrrr)rrrs r_convert_equalization_refrTsI5&&&>?G5UGD5'#56 uekk **r!model_amodel_bxcddlmcmcm}ddlm}|}|dj tj|jd|d||j|\}}|||||j|||jd}|j|ddtjjjjjdi} |D],} || d ddd } || d dddd} | | | <.| S) aGRuns the Numeric Suite on model_a and model_b and returns a dictionary containing the SQNR between layers in model_a and model_b. Note: In order to support equalized models, this function has a hacky fix in which we do not match any torch.mul operators. This is because equalized models contain extra mul operators to scale the input by the equalization scale, but this edge case has not been resolved yet within the numeric suite code. Args: model_a: A float model model_b: A quantized model x: Inputs to use during calibration rN)get_unmatchable_types_mapfuns_unmatchablefp32int8)unmatchable_types_mapsqnr node_outputfqn)torch.ao.ns._numeric_suite_fxaons_numeric_suite_fxtorch.ao.ns.fx.mappingsr addr.rR add_loggers OutputLoggerextract_logger_info%extend_logger_results_with_comparisonfxutils compute_sqnr) rrrrr r  model_a_ns model_b_nsactivation_comparison_dictlayer_sqnr_dictkeylayerrs rget_layer_sqnr_dictr$gs+ /.A57,-11%))<^^ 3 ,J qMqM!#!7!7J",,"  )) O)&*3/ >vFqI%P)#.}=fEaHPQRS!%& r!r!num_layers_to_equalizect|jtjd}|d|}|Dcgc] }|dtf}}d|i}|Scc}w)aGiven the layer to SQNR dictionary, find the layers with the highest quantization errors, and return an equalization_qconfig_dict specifying to only equalize those top layers. Args: layer_sqnr_dict: Dictionary mapping layer names to SQNR values (found when comparing an equalized model against a float model) num_layers_to_equalize: Number of layers with the highest quantization errors to equalize r)r"Nr module_name)sorteditemsoperator itemgetterdefault_equalization_qconfig)r!r%layer_sqnr_sortedlayers_to_equalizeitemmodule_to_qconfig_listequalization_qconfig_dicts rget_equalization_qconfig_dictr2sv 446HrHs"33 ##Q J &(c'!!S!!%,,! U55  ; ;,( `(`(F?("))?(D$)$7R$ \\$N$'98&DE>9BB ,, : :C ;DD ++u:: E  309U  $HtH 343 T t ryyT))$/):>sBII~:N) 8D>8$?@ @A)X  bii0 d^$/ /c299n-/ ()/d c299n- ell*/t/d3 >.B/t/,8/ 8/ #ryy. !8/ 8/&ell3 8/  8/vK? K? K?#ryy. !K? K? &ell3 K?  K?\**S"))^8L*QU*$ !{ !$ !4 !* *!%c299n!5* #* *+*ZP P #ryy. !PS"==>P Pf +[ +&4 YY4!#4/4||4 #u*4n%#u*%%?B%%r!