sgczddlZddlmZmZddlZddlmZddlmZm Z ddl m Z m Z e ddZ ejejej ej"gZej&ej(gZeDcic];}|ej,|j.ej,|j0f=c}Zej5eDcic]G}|eej8|j.eej8|j0fIc}dZe j=d e e d d d ej>d e dedededejBdej>fdZ"e e d dd ej>d e dedededejBdej>fdZ#e j=de e dd d ej>d ej>dej>dededejBdej>fdZ$e e ddd ej>d ej>dej>dededejBdej>fdZ%e j=de e dd d ej>d ej>dej>dej>dej>dejBdej>fdZ&e e ddd ej>d ej>dej>dej>dej>dejBdej>fdZ'e j=de e dd dd d ej>d e dedededejBd!eejBdej>fd"Z(e e dddd d ej>d ej>dej>dededejBd!eejBdej>fd#Z)e j=d$e e d%d dd d ej>d ej>dej>dededejBd!eejBdej>fd&Z*e e d%ddd d ej>d ej>dej>dededejBd!eejBdej>fd'Z+e j=d(e e d)d dd d ej>d ej>dej>dej>dej>dejBd!eejBdej>fd*Z,e e d)ddd d!eejBdej>fd+Z-e j=d,e e d-d d ej>d.ed/ed0e dejBdeej>ej>ff d1Z.e j=d2e e d3d d ej>d.ed/ed0e dejBdeej>ej>ff d4Z/e e d-dd ej>deded0e dejBdeej>ej>ff d5Z0e e d3dd ej>deded0e dejBdeej>ej>ff d6Z1d7Z2e j=d8e e d9d d ej>d:ej>d;ej>dfd=Z3e e d9dd ej>d:ej>d;ej>dfd>Z4e j=d?e e d@d dd d ej>d:ej>d;eej>dfdAZ5e e d@ddd d ej>d:ej>d;eej>dfdBZ6e j=dCe e dDd d ej>dejBdeej>ej>ffdEZ7e e dDdd ej>dejBdeej>ej>ffdFZ8e j=dGe e dHdId ej>dejBdeej>ej>ffdJZ9e j=dKe e dLd d ej>dejBdeej>ej>ffdMZ:e e dLdd ej>dejBdeej>ej>ffdNZ;dOZd:ej>d;ej>dededejBf dRZ=e e dQdd ej>d:ej>d;ej>dededejBf dSZ>e j=dTe e dUd ej~fd ej>d:ej>d;ej>dededejBdVejBfdWZ@e e dUdej~fd ej>d:ej>d;ej>dededejBdVejBfdXZAe j=dYe e dZd  djd ej>d:ej>d;ej>dededejBf d\ZBe e dZd djd ej>d:ej>d;ej>dededejBf d]ZCe j=d^e e d_d d[ej~fd`ej>d:ej>d;eej>dededejBdaedVejBfdbZDe j=dcGdddeejjZGe e dfdgd ej>d:ej>d;ej>dfdhZHe e dfdd ej>d:ej>d;ej>dfdiZIycc}wcc}w)kN)OptionalTuple)_unsqueeze_multiple)determine_qparamsvalidate_qmin_qmax)implLibraryquantized_decomposedDEFc|tvrtd|t|\}}||k\s Jd|d|||ks Jd|d|y)NzUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: )_DTYPE_TO_QVALUE_BOUNDS ValueError) quant_min quant_maxdtypequant_min_lower_boundquant_max_upper_bounds W/var/www/html/venv/lib/python3.12/site-packages/torch/ao/quantization/fx/_decomposed.py_quant_min_max_bounds_checkrs ++.ug6773J53Q00 - - ""7!8 YK Q - - - ""7!8 YK Q -zxquantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tensorCompositeExplicitAutogradinputscale zero_pointrrrreturnc|jtjtjfvr|j tj }|jtj k(sJd|jt |||d|z }tjtj||z|z||j |S)aAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead Tensorzquantize_per_tensor.tensorc|jdk(sJd|j|jdk(sJd|jt||j|j|||SzAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Same as `quantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values >Expecting zero_point tensor to be one element, but received : 9Expecting scale tensor to be one element, but received : numelritemr,s rquantize_per_tensor_tensorr6ns" a] G HXHXHZG[\]  S B5;;=/RS  uzz|Z__. 9e rc|jtjtjfvr|j tj }|j dk(sJd|j |j dk(sJd|j |jtj k(sJd|jtj||S)Nr0r1r2rr*)rr r!r"r#r$r4r+r,s rquantize_per_tensor_tensor_metar8s {{u}}enn55'a] G HXHXHZG[\]  S B5;;=/RS  u}}$T Eekk]ST$   E //rzquantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype) -> Tensorzquantize_per_tensor.tensor2c>|jdk(sJd|j|jdk(sJd|jt||j|j|j|j|Sr/r3r,s rquantize_per_tensor_tensor2r:s" a] G HXHXHZG[\]  S B5;;=/RS     rc"t||||||SN)r8r,s r quantize_per_tensor_tensor2_metar=s + uj)Y rzdequantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_tensor out_dtyper@c|j|k(sJd|d|j|tj}|tvr|j ||z |zSt d|)aAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per tensor quantized Tensor if combined with quantization parameters in the argument of this function (scale/zero_point) scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for input Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for input Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): dtype for input Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype: z , but got ,Unsupported dtype in dequantize_per_tensor: )rr r$r r#rrrrrrrr@s rr>r>szL  uH (z%++GHMM  ''#j0E99GwOPPrcT|tj}tj||SNr*)r r$r+rDs rdequantize_per_tensor_metarGs&MM   E 33rzdequantize_per_tensor.tensor(Tensor input, Tensor scale, Tensor zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensorc  |jdk(sJd|j|jdk(sJd|jt||j|j||||SzAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Same as `dequantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values r0r1r2r?r4r>r5rDs rdequantize_per_tensor_tensorrK(s* a] G HXHXHZG[\]  S B5;;=/RS     rc^|tj}|jdk(sJd|j|jdk(sJd|j|j|k(s Jd||tvrtj ||St d|)Nr0r1r2rBr*rC)r r$r4rr r+rrDs r!dequantize_per_tensor_tensor_metarMMsMM a] G HXHXHZG[\]  S B5;;=/RS ;;% J#B5'!JJ  ''Y77GwOPPrzdequantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensor2c B|jdk(sJd|j|jdk(sJd|jt||j|j|j|j||SrIrJrDs rdequantize_per_tensor_tensor2rOns* a] G HXHXHZG[\]  S B5;;=/RS     rc &t|||||||S)Nr?)rMrDs r"dequantize_per_tensor_tensor2_metarQs  - uj)Y rzrchoose_qparams.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams.tensorqminqmaxepsc |jtjtjtjfvsJd|j|t vsJdt j d|t||tj|\}}t|||||tj|gdS)[Given an input Tensor, derive the per tensor affine quantization parameter (scale and zero_point) for target quantized Tensor from the Tensor Args: input (torch.Tensor): floating point input Tensor quant_min (int): minimum quantized value for target quantized Tensor quant_max (int): maximum quantized value for target quantized Tensor dtype (torch.dtype): dtype for target quantized Tensor Returns: scale (float): quantization parameter for the target quantized Tensor zero_point (int): quantization parameter for the target quantized Tensor CExpecting input to have dtype torch.float32/16/b16, but got dtype: $Expecting target dtype to be one of , but got: F)has_customized_qrange) rr r$r!r"r keysraminmaxrTensorrrRrSrTrmin_valmax_vals rchoose_qparams_tensorras" ;;    [ MU[[MZ [  ((a -.E.J.J.L-M[Y^X_`a(tT"}}U+GW     cU# rz|choose_qparams_symmetric.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams_symmetric.tensorc |jtjtjtjfvsJd|j|t vsJdt j d|t||tj|\}}t|||||tj|gdtjS)rVrWrXrYF)rZqscheme) rr r$r!r"r r[rr\rr]per_tensor_symmetricr^s rchoose_qparams_symmetric_tensorres* ;;    [ MU[[MZ [  ((a -.E.J.J.L-M[Y^X_`a(tT"}}U+GW     cU#**  rc|jtjtjtjfvsJd|j||ks Jd|d|tj dtj |jtj dtj|jfS)NrWzKExpecting quant_min to be smaller than quant_max but received min: z max: r0rdevice) rr r$r!r"emptydoublerhint64rrrrTrs rchoose_qparams_tensor_metarms ;;    [ MU[[MZ [  I&  6)&& ;;q U\\ BEKK U\\E rctjdtj|jtjdtj|jfS)Nr0rg)r rirjrhrkrls r$choose_qparams_symmetric_tensor_metarosA ;;q U\\ BEKK U\\E rctt|j}d||<||d<|jt |}||fS)Nr)listrangedimpermutetuple)xaxis new_axis_listys r_permute_to_axis_zerorzsHquuw(MM$M! % &'A m rzquantize_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_channelscales zero_pointsrwc|jtjtjfvr|j tj }|jtj k(sJd|j||j ksJd|j t|||t||\}}dg|j z}|jd|d<|j|}|j|}tjtj|d|z z|z||} | jt|} | j |S)atAffine per channel quantization for the Tensor using the same quantization parameters for each channel/axis to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_point (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead rExpecting axis to be < r0rr)rr r!r"r#r$rsrrzshapeviewr%r&rtru) rr|r}rwrrrpermute_axis_list new_shaperesouts rr{r{-s26 {{u}}enn55' u}}$T Eekk]ST$ %))+ F!8 FF  9e<4UDAE eiik!I<<?IaL [[ #F""9-K ++ ES6\*+k99i C ++e-. /C 66%=rc|jtjtjfvr|j tj }|jtj k(sJd|j||j ksJd|j t|||tj||S)Nrrr*) rr r!r"r#r$rsrr+)rr|r}rwrrrs rquantize_per_channel_metar]s {{u}}enn55' u}}$T Eekk]ST$ %))+ F!8 FF  9e<   E //rzdequantize_per_channel(Tensor input, Tensor scales, Tensor? zero_points, int axis, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_channelc|j|k(sJd|d|j|tj}||jksJd|jt |||t ||\}}dg|jz} |j d| d<|j| }|||j| z |z} n||z} | j|} | jt|} | S)aAffine per channel dequantization for the Tensor using the same quantization parameters for each channel/axis to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per channel quantized Tensor if combined with quantization parameter in the argument of this function (scales/zero_points/axis) scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_points (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for output Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): requested dtype for output Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype , but got dtype: rr0r) rr r$rsrrzrrr#rtru) rr|r}rwrrrr@rrrrs rrr{s R  uN 'w.? }MNMM %))+ F!8 FF  9e<4UDAE eiik!I<<?IaL [[ #F{'' 22f<fn && C ++e-. /C Jrc|j|k(sJd|d|j|tj}||jksJd|jt |||tj ||S)Nrrrr*)rr r$rsrr+)rr|r}rwrrrr@s rdequantize_per_channel_metars  uN 'w.? }MNMM %))+ F!8 FF  9e<   E 33rzLchoose_qparams_per_token(Tensor input, ScalarType dtype) -> (Tensor, Tensor)choose_qparams_per_tokencp|jjdd}|jtjk(r|j }|tj k(rd}d|dz zdz }ntd||jd j|}tj|}||fS) Choose quantization parameters for per token quantization. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32/float16 Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor Returns: scales and zero_points, both float32 Tensors Trskeepdimr0z/unsupported dtype in choose_qparams_per_token: gh㈵>min) absamaxrr r!floatint8 Exceptionr%div zeros_like)rrr|n_bitsrr}s rrrs,YY[  "d  3F ||u}}$ LLN   &1*%) =eW E  \\d\ # ' ' 2F""6*K ; rcd|jdf}tj|tj|jtj|tj |jfSNr0rrgsizer rirjrhrkrrrs rchoose_qparams_per_token_metarS uzz"~ D ;;t5<< Eu{{ EKK H rz]_choose_qparams_per_token_asymmetric_impl(Tensor input, ScalarType dtype) -> (Tensor, Tensor))_choose_qparams_per_token_asymmetric_implCompositeImplicitAutogradcd\}}tj|dd}tj|dd}tj|tj|}tj |tj|}tj tjj}||z t||z z } | j|} || z } || z } || z} || z} tj| | zdkD|| z || z }tj|||j}| jtj|jtjfS)r)irTrrr)r aminrrrmaxfinfor$rTrr%wherer&r#)rrrRrSr_r` min_val_neg max_val_posrTr descaled_min descaled_maxzero_point_from_min_errorzero_point_from_max_errorrs rrrsC,JD$jjB5GjjB5G))GU%5%5g%>?K))GU%5%5g%>?K ++emm $ ( (C; &%t *< (Tensor, Tensor)#choose_qparams_per_token_asymmetricct||Sr<)r)rrs rrrFs 5UE BBrcd|jdf}tj|tj|jtj|tj |jfSrrrs r(choose_qparams_per_token_asymmetric_metarRrrctjt|jdd}||j k(sJd|d|j||j k(sJd|d|jy)Nrz num_tokens: z scales: z zero_points: )mathprodrqrr4)rr|r} num_tokenss r!_per_token_quant_qparam_dim_checkras4 -cr23Jflln$; j\6;;=/:;$ k''))E j\ 0@0@0B/CDE)rz}quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tokenct|||t||||jd|z j|j j ||j |}|S)aPer token quantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r)rrmuladdr&r%r#rr|r}rrrs rrrqs^6 9e<%eV[A #, [   y) $ E   LrcJt|||tj||SrFrr r+rs rquantize_per_token_metars# 9e<   E //rzdequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensordequantize_per_token output_dtypec8||z }|j||z}|S)aPer token dequantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): quantized Tensor (uint8, int8 etc.) scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` )r#rr|r}rrrrs rrrs&8 K E HH\ "V +E LrcJt|||tj||SrFrrs rdequantize_per_token_metars# 9e<   E 66rzquantize_per_channel_group(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size) -> Tensorquantize_per_channel_groupcL|dkDsJ||jdkDr!|jddk(r|jd}|jd|zdk(sJ|jdk(sJ|jd|}tj|j dk(sJ|jdd}|jdd}|j d|z j|jj||j|j|}|S)Nr0rrrr) rrsreshaper isnansumrrr&clamp_r# reshape_as) rr|r}rrr group_sizeto_quant input_int8s rrrs >>EKKO# R(8A(=[[_ ;;r?Z '1 ,, , 99;!  }}R,H ;;x $ $ &! ++ + ^^B "F%%b!,K  S6\" [    9 % E E   rc|dkDsJ||jdkDr!|jddk(r|jd}|jd|zdk(sJ|jdk(sJtj||S)aXGroupwise quantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r0rrrr*)rrsr r+)rr|r}rrrrs rquantize_per_channel_group_metar s8 >>EKKO# R(8A(=[[_ ;;r?Z '1 ,, , 99;!     E //rzdequantize_per_channel_group(Tensor input, Tensor scales, Tensor? zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size, ScalarType output_dtype) -> Tensordequantize_per_channel_groupw_int8rc |dkDsJ||jdkDr!|jddk(r|jd}|jd|zdk(sJ|jdk(sJ|jd|}|jdd}||jdd} n0tjgtj |j } |j| j|j|j|} | S)a!Groupwise dequantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): quantized Tensor (uint8/int8 etc.) scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` r0rrrrg) rrsrr zerosint32rhsubrrr#) rr|r}rrrrrw_int8_groupedzpw_dqs rrr5sD >>FLL$$b)9Q)>\\"% << j (A -- - ::<1  ^^B 3N ^^B "F  Q ' [[5;;v}} E   b ! % %f - 8 8 @ C CL QD Krzyfake_quant_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max) -> Tensorc,eZdZedZedZy)FakeQuantPerChannelc|jtjk7r|jtj}|jtjk7r|jtj}|jtjk(sJd|j||j ksJd|j t td|t t|dz|jz}t||}t||} tj|d|z z| z} tj| ||| z |z} tj| |k\| |k} |j| | S)Nrrrr0r)rr r$r#rrsrqrrndimrr&r% logical_andsave_for_backward) ctxrr|r}rwrrbroadcast_dimsunsqueeze_scalesunsqueeze_zero_pointstemprmasks rforwardzFakeQuantPerChannel.forwardosO <<5== (YYu}}-F    +%..5K KK5== ( X I%++ W X (eiik!J%rGrKrMrOrQrarermrorzr{rrrrrrrrrrrr$rrrrrautogradFunctionrrr)ks0rrsa " +M' ##95A;; EKKE""E$7$78 :I45A  AKEKKN..//DQRqQ[U[[^   #kekk!n&8&8"9::R  @  57RS! <<! !! !  ! ;; ! \\!T!H 5v> 0 << 0  0 0 0  0 ;; 0 \\ 0? 0 @ :G(,Q <<Q <<Q Q Q  Q ;; Q $Q \\QHQ4e #(, << << ||  ||  ;;  $ \\  @ ?H(,  $  \\ I 7  79TU$ <<$"$*-$49$BG++$ 5<< %&$V$N7 % $ <<$"$*-$49$BG++$ 5<< %&$  $N 7@ <<$'47>CLQKK 5<< %&A" A6J <<$'47>CLQKK 5<< %&K@  68ST, <<, LL,,  ,  ,  , ;;, \\,U,^ 6?0 <<0 LL00  0  0  0 ;;0 \\0@0._  8:UV(,; <<; LL;%,,';  ;  ;  ; ;;; $; \\;W;| 8&A(,4 <<4 LL4%,,'4  4  4  4 ;;4 $4 \\4B4*R   <<  ;;  5<< %&   F   << ;; 5<< %&  c / (A <<(A ;;(A 5<< %&(A  (AV] ) C <<C ;;C 5<< %&C  C)   << ;; 5<< %&  E@  46QR# <<# LL## #  # ;; #S#L 4f= 0 << 0 LL 0 0 0  0 ;; 0> 0Y  68ST!&  << LL    ;; ++U@ 6?!&  7 << 7 LL 7 7 7  7 ;; 7++ 7@ 7A: