sg ddlZddlZddlZddlmZmZmZmZmZm Z m Z ddl Z ddl m cmZddlmZddlmZddlmZmZmZmZddlmZmZmZddlmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'e rdd l(m)Z)gZ*e jVddd e jVddde jVde jVde jVde jVdfZ,e jVddd d e jVdddde jVde jVde jVde jVdfZ-d e.d e.de.de.dee/eff dZ0dedefdZ1dedefdZ2dedefdZ3dZ4d e.d e.de.dede.def dZ5d e.d e.de.dede.def dZ6dddedede.fdZ7dddedede.fdZ8d ede.fd!Z9d ede.fd"Z:d#edee/e eefffd$Z;d%eeefdeeeffd&Zd,ed-eeeffd.Z?d/edefd0Z@d/eded1e ed2fde.def d3ZAd/efd4ZBd/efd5ZCd'ed*efd6ZDd/edefd7ZEd/eded1e ed2fde.def d8ZFy)9N)AnyCallableDictListOptionalTuple TYPE_CHECKING)quantized_decomposed_lib)_WrapperModule)DerivedQuantizationSpec EdgeOrNodeQuantizationSpecBaseSharedQuantizationSpec)Graph GraphModuleNode)replace_pattern_with_filtersReplacedPatterns)_conv1d_bn_example_inputs_conv2d_bn_example_inputs"_get_aten_graph_module_for_pattern _is_bn_node_is_conv_or_conv_transpose_node_is_conv_transpose_fnfold_bn_weights_into_conv_node) InternalMatchis_per_channelhas_biasbias_is_quantizedis_cudareturnc"i}|rtjdgtj|d<tjdgtj|d<|rT|rRtjdgtj|d<tjdgtj|d<|rtjd|d<|rF|j D]3\}}t |tjs!|j||<5|S) zu Optional example inputs for quantized and folded conv-bn patterns used in convert, expressed as kwargs. rdtype weight_scalerweight_zero_point bias_scalebias_zero_point conv_bias) torchtensorfloatintrandnitems isinstanceTensorcuda)rr r!r"kwargskvs W/var/www/html/venv/lib/python3.12/site-packages/torch/ao/quantization/pt2e/qat_utils.py,_get_quantized_conv_bn_example_inputs_kwargsr9:sF!&qc!E~&+llA3eii&H"# )#(<<5;;#GF< (- aS (JF$ %#kk!n{LLN %DAq!U\\*FFHq  % Mconv_fnc&dtjdtjdtjdtjdtjdtjdtjdtjffd }t|S) Nx conv_weightr+ bn_weightbn_biasbn_running_meanbn_running_varr#cR|||}tj|||||d}|S)NT)training)F batch_norm)r=r>r+r?r@rArBr;s r8_conv_bn_patternz._get_conv_bn_pattern.._conv_bn_patternWs5 A{I . LL  7T r:r,r3r )r;rGs` r8_get_conv_bn_patternrIVs  << \\ << <<          * ++r:c&dtjdtjdtjdtjdtjdtjdtjdtjffd }t|S) Nr=r>r+r?r@rArBr#c d}tj||z}||z } dgt|jz} t rdnd} d| | <dgt|jz} d| d<|| j | z} tj ||j}|| |}|| j | z }||j | z}tj|||||d|}|S)z Approximated method to fuse conv and bn. It requires only one forward pass. conv_orig = conv / scale_factor where scale_factor = bn.weight / running_std. This is based on `nniqat.ConvBn2d._forward_approximate`. h㈵>rrr%TrDeps) r,sqrtlenshaperreshape zeros_liker&rErF)r=r>r+r?r@rArBbn_eps running_std scale_factor weight_shapeweight_in_channel_axis bias_shape scaled_weight zero_biasr;s r8_qat_conv_bn_patternz6_get_qat_conv_bn_pattern.._qat_conv_bn_patternksjj&!89  ;. sS!2!233 &;G&D!/1 +,S3{0011  1 #l&:&:<&HH $$Yagg> A}i 0 $$Z0 0 !!*- - LL     r:rH)r;r]s` r8_get_qat_conv_bn_patternr^js% <<%\\%<<%<< %  %  % % %N . //r:c&dtjdtjdtjdtjdtjdtjdtjdtjffd }t|S) Nr=r>r+r?r@rArBr#c xd}tj||z}||z } dgt|jz} t rdnd} d| | <dgt|jz} d| d<|| j | z} || d}|| j | z }t j|||||d|}|S)z] Same as `_get_qat_conv_bn_pattern`, but handles the case with no conv bias. rLrrrMNTrN)r,rPrQrRrrSrErF)r=r>r+r?r@rArBrUrVrWrXrYrZr[r;s r8!_qat_conv_bn_pattern_no_conv_biaszP_get_qat_conv_bn_pattern_no_conv_bias.._qat_conv_bn_pattern_no_conv_biassjj&!89  ;. sS!2!233 &;G&D!/1 +,S3{0011  1 #l&:&:<&HH A}d + $$Z0 0 LL     r:rH)r;ras` r8%_get_qat_conv_bn_pattern_no_conv_biasrbs" <<"\\"<< " << "  "" " "H ; <r?r@rArBr#c *tj|z}||z }dgt|jz} d| d<dgt|jz} d| d<||j | z} t | d|} r@tj |d|j} rt | d|} || | }n || d}||j | z }r||dj | z}tj||||| }|S) NrrMrFrlr5r+r%TrN) r,rPrQrRrSrurTr&rErF)r=r>r?r@rArBr5rVrWrXrZr[r\r!rUrvr;r rs r8_quantized_qat_conv_bn_patternzJ_get_quantized_qat_conv_bn_pattern.._quantized_qat_conv_bn_patterns@jj&!89  ;. sS!2!233  QS3{0011  1 #l&:&:<&HH #    (( )r?r@rArBr#c t| d|} r|d}rt| d|}nd} |||}tj||||| }|S)NFryr+TrN)rurErF)r=r>r?r@rArBr5biasr!rUrvr;r rs r8%_folded_quantized_qat_conv_bn_patternzX_get_folded_quantized_qat_conv_bn_pattern.._folded_quantized_qat_conv_bn_pattern*s"    +&D "" ! D A{D ) LL    # r:rH)rr r!r;rvrrUs````` @r8)_get_folded_quantized_qat_conv_bn_patternrsF$ <<$\\$<<$ $  $  $ $$L ? @@r:matchroriginal_graph pattern_graphc|jjD]:}t|st|jdkDxr|jdducSt d)zw Match filter for the subgraph rewriter that returns True if the conv node in the original graph has bias. Nz5Could not find conv node in matched conv + bn pattern) nodes_mapvaluesrrQargs ValueError)rrrns r8_has_conv_bias_filterrSsZ__ # # %= *1 -qvv;?._get_nodess ,<( 7L !Att&.q1 ((( 1~&xx8+++#+++  !$$$"""7L11r:rct|sJ|jd}t|tsJt |sJ|jd}t|tsJ|||fS)zC Return a 3-tuple of (orig_node, q_node, dq_node). r)rrr2rr)rq_node orig_nodes r8_get_q_dq_nodesz3_get_conv_bn_pattern_nodes.._get_q_dq_nodessga   &$'''F###KKN )T***61%%r:)convbn)rrr conv_weight_qconv_weight_dq conv_inputr>r conv_bias_q conv_bias_dqr+)rrrrlist_filter_nodes_maprr replacementskeysrr2rrQ)"rrroriginal_nodeso_convo_bn o_getitemr_convr_bn r_getitemmappingp_conv_ p_conv_input p_conv_weight r_conv_input r_conv_weight o_conv_input o_conv_weightp_conv_weight_qp_conv_weight_dqr_conv_weight_qr_conv_weight_dqo_conv_weight_qo_conv_weight_dq p_conv_bias r_conv_bias o_conv_bias p_conv_bias_qp_conv_bias_dq r_conv_bias_qr_conv_bias_dq o_conv_bias_qo_conv_bias_dqs" r8_get_conv_bn_pattern_nodesr~s%"2$t*2tT8D>/I)J2. &4 &E$d*:$; &+AKK8??ABN(8FD)(8FD)   V$, $$$$$$V$,!9-  Q[[%5%5%7 89NVQ(. %\=1(. %\=1 lD )) ) mT ** * lD )) ) mT ** *;;|,LKK .Mm$;J < 8 (84?N14E(--a01GV34Dr:nodeoriginal_to_replacement_nodecdtffd dtffd }d|jvry|jd}|jj D]\}}|||j|<||j |_y)ai Update the `SharedQuantizationSpec`s and `DerivedQuantizationSpec`s used in `node`'s quantization annotation after subgraph rewriting. The original annotation referred to the nodes in the original graph, so the nodes used in these special quantization specs will need to be updated to the corresponding nodes in the replacement graph. edge_or_nodec(t|tr|}j||St|trIt |dk(r;t d|Dr)|\}}j||j||fSt dt|)Nrc3<K|]}t|tywr)r2r).0r=s r8 zZ_update_special_qspecs_after_replacement.._get_new_edge_or_node..\s>AJq$'>sz"unexpected type for edge_or_node: )r2rgetrrQallrtype)r_nodesrcdestrs r8_get_new_edge_or_nodezG_update_special_qspecs_after_replacement.._get_new_edge_or_nodeUs lD ) E/33E5A A |U +L!Q&>>>$IC,00c:,00t<  A4 CUV Vr:qspecct|tr|j}t|St|tr6|jDcgc] }| }}t j ||S|Scc}w)N) derived_from)r2rrr r dataclassesreplace)rnew_edge_or_noder=new_derived_fromrs r8_get_new_qspecz@_update_special_qspecs_after_replacement.._get_new_qspecfsu e3 44U5G5GH )*:; ; 6 7BGBTBTUQ 5a 8U U&&u;KL LL Vs A7rN)r rrrr1 output_qspec)rrr annotation input_noderrs ` @r8(_update_special_qspecs_after_replacementrHsWJW"2! 145J'77==?G E1?1F "":.G,Z-D-DEJr:mctd|jjD}|s|Stjj rddgndg}|D]}t |tjt|}t |tjt|}t |tjt|}t |tjt|}|S)Nc32K|]}t|ywrrrrs r8rz$_fuse_conv_bn_qat..y7AQ7TFr")anygraphrr,r4 is_available_fuse_conv_bn_qat_helperrEconv1drconv2drconv_transpose1dconv_transpose2d)rhas_bnis_cuda_optionsr"s r8_fuse_conv_bn_qatrxs 77 7F ',zz'>'>'@tUmugO"   $ qxx2G  % qxx2G  % q!!#\}}|j|_ t|rt||t|||| |<@^|jj D]}t#|| |S)aU Given a graph of decomposed aten ops, replace the (conv + bn) pattern with the fused QAT subgraph equivalent. The input graph should already be annotated. The annotations in the original nodes will be preserved in the corresponding nodes in the new subgraph. Note: This also handles the (conv + bn + relu) pattern. T) match_filtersignore_literals)r eliminate_dead_code recompilerIrr^rrrbrrrrrrrrr)rr;rr"conv_bn_pattern match_patternqat_conv_bn_pattern"replacement_pattern_with_conv_biasreplacements_with_conv_bias qat_conv_bn_pattern_no_conv_bias replacement_pattern_no_conv_biasreplacements_no_conv_bias!all_original_to_replacement_nodesrrrrs r8r r sGG!KKM*73O6M37;)K*& #? *,- #KKM(MW'U$'I(($ != (+, !KKM()+% (+D D P/I!/L/S/S/U P +M+$1$6$6  !.}=,]!#3@P -m < P PWW]]W04UVW Hr:c btjjj}|jj D]}|j dk7s'|j|k7st|jdk(r:t|jD]j}|jj|5|jjd||j|j}ddd|j|l|jj!||j#y#1swYKxYw)a{ Helper function to duplicate all dequantize nodes in the graph if the node has more than one user. For example: Before: quantize -> dequantize -> a \--> b \--> c After: quantize -> dequantize_1 -> a \--> dequantize_2 -> b \--> dequantize_3 -> c This is useful for subgraph rewriting. E.g. if we wish to match the pattern [dequantize - a] above, subgraph matching would fail because the dequantize node has users outside the matched portion of the graph. Instead, we match [dequantize_1 - a], which is safe. rrN)r,rfrgrkr rrrrQusersrinserting_before create_noderr5replace_input_with erase_noder)rdq_opruserrs r8_duplicate_dequantize_noder*s( II * * @ @E WW]] 44? "ahh%&73qww<1;L M 1D))!, Y77..qvvqxxX Y  # #Ax 0 1 1KKM  Y Ys (3D%%D. cZtjjj}|jj D]}|j Dcgc]"}|jdk(r|j|k(r|$}}t|dkDsI|jj|d5|jjd||dji}ddd|D].}|j|jj|0|jycc}w#1swYTxYw)a Removes duplicate dequant nodes in the graph, for an operator that has multiple dequant nodes as a user, replace them with a single dequant node that can be shared across all the uses. This should be seen as the "reverse" of `_duplicate_dequantize_node`. rrrN)r,rfrgrkr rr#rrrQinserting_afterr%rreplace_all_uses_withr'r)rr(rr)dq_usersrdq_users r8_remove_extra_dequantizer0s  II * * @ @E WW]] , ww/)dkkU.B   x=1 ((!5 77..#UHQK,<,.GrrTFrrget_attrrsource_fn_stack)rr rr,r4r _fold_conv_bn_qat_helperrEr #_quantized_conv1d_bn_example_inputsr #_quantized_conv2d_bn_example_inputsrrrrfatenadd_r3rrnnmodules batchnorm BatchNorm2drr'rr)rrrr"rvals r8_fold_conv_bn_qatrCFs 77 7F ',zz'>'>'@tUmugO"   $ qxx>..55 5 ! :- ! !  **66"&)),=">?3A?@ GG  t $%GG!KKM H@s G8c |jj|jt|g}t j ddgddgddgddg}|D]r\}}}} |s|r t ||||} t||||| } t| ||fi| } t||||| } t| ||fi| } |jt|| | dt|jt||D]} t| }|jD]\}}|j|_t!|dt!|dd|vrd|vsJt!|dt!|dd}|d \}}|d \}}|d \}}d |vr|d \}}t#|||||t%| j&jD]}t)|st+|||jj|j|S) zn Replace the quantized (conv + bn) pattern with conv with bn weights folded into the weights of conv. TF)rrrrrNrrr>r+)r rrr* itertoolsproductr9r{rrextendrr0rrrr4rrrrr)rr;rr"rreplacement_optionsrr r!rvr5rreplacement_patternrnode_maprrr+rrrr>s r8r9r9js~GG!KKMq!L#++ u u u u     - = H&7 ; H&7. ; >7 6< H H&7. A   functionalrE$torch.ao.quantization.fx._decomposedr 'torch.ao.quantization.pt2e.export_utilsr torch.ao.quantization.quantizerr r rrtorch.fxrrrtorch.fx.subgraph_rewriterrrutilsrrrrrrr6torch.fx.passes.utils.matcher_with_name_node_map_utilsr__all__r0r:r;boolstrr9rIr^rbrur{rrrrrrrrrrrr r*r0r4rCr9r:r8rYsLLL IB .-UT  EKK1aEKK1aEKKNEKKNEKKNEKKN '#EKK1aEKK1aEKKNEKKNEKKNEKKN '#   #s(^ 8,(,x,((0h(08(0V%=8%=%=P 8A:A:A:A: A:  A:  A:H3A3A3A3A 3A  3A  3Al N  N N N N K  K K K KDTdt{"2{tCtTzAR