sg6*ddlZdZdZdZdZdZy)Nc`tjd||dddfjd|}tjd||j|d}|jdk(rdnd}|jdk(rdnd}||z|z|d z|zz||zd zz}|dzdk(|dzdk(zj tj }|dzdk(|dzdk(zj tj } ||| z z }||| z z}d} || z} || z} | |z| z|| zz| zj d S) aS This is PyTorch implementation of main part of reorder_meta() function, from tools/util/include/cutlass/util/host_reorder.h file of CUTLASS source tree. Furthermore, CUTLASS template for sparse GEMM decides upon layout of this matrix, and at the moment for the sparse GEMM executed on tensor cores, this is layout described by ColumnMajorInterleaved<2> data structure, in include/cutlass/layout/matrix.h of CUTLASS source tree. The reordering of meta matrix into meta_reordered matrix calculated according to these segments of CUTLASS code is re-implemented here. Note that this calculation produces offsets for scattering metadata matrix elements into reordered metadata matrix elements (or, equivalently, for gathering reordered metadata matrix element back into metadata matrix elements). rdeviceN )torcharangerepeatitemsizetoint8view) m meta_ncols meta_dtyperdst_rowsdst_colsgroup interweavetopright bottomleft interleavecols_majcols_mins \/var/www/html/venv/lib/python3.12/site-packages/torch/sparse/_semi_structured_conversions.py*_calculate_meta_reordering_scatter_offsetsr!sd ||Aq0D9@@JOH||Az&9@@AFH%%*BE ))Q.AJEE! a<: % & e  ! " A"x!|q'89==ejjIHa<1$A):;?? KJ :%%H :%%H J:%H*$H qL: %:(= = H N Nr RRc|jdk7rtd|jd|j\}}|j}tj }|j tj k(rtj}ne|j tjtjtjfvrtj}ntd|j d|jdzdz}|dvr td |tjk(r|d zd k7r&td |d |dzd k7rtd |d|d|zzd k7rtd|dd|z|j tjk7r2d}|jd||z|}|d k7jd\}} } } n4d}|jd||z|} | d k7jdx\}} \} } |||zz} || z}|| z}|| z}|}|}||z| z}|| z}||jtj dzz}||jtj dzz}|j tjk7roj#d|j%d}|j#d|j%d}t j&||fdj||dz}n7 j#d|j%ddzj||dz}||dzz}|jd| |fj|}|dk(r=|ddddd f|dddddfdzz|dddddfdzz|dddddfdzz}n|dk(r||ddddd f|dddddfdzz|dddddfdzz|dddddfdzz|dddddfd zz|dddddfdzz|dddddfdzz|dddddfdzz}j)|| zf}t+|| ||}|j-d ||jd||j|| fS)z This function converts dense matrix into sparse semi-structured representation, producing "compressed" matrix, in the layout used by CUTLASS backend, and corresponding metadata matrix. rz)Expected 2-dimensional dense tensor, got -dimensional tensorInvalid datatype z of dense matrixr r )r r z6Invalid number of elements per meta element calculatedr rzNumber of rows of dense matrix z must be divisible by 16rz must be divisible by 32z"Number of columns of dense matrix z must be divisible by r r)dimN )r& RuntimeErrorshaperr rdtypeint32halfbfloat16floatint16rrunbindrint64gather unsqueezestack new_emptyr!scatter_)denserkrrquadbits_per_meta_elemksparsedense_4m0m1m2m3dense_2rexpr0expr1expr2bit0bit1bit2bit3idxs0idxs1sparse0sparse1sparsemeta_4meta_nmetameta_reordered meta_offsetss r )sparse_semi_structured_from_dense_cutlassrY/s  yy{a7 }DW X   ;;DAq \\FJ {{ejj [[ U^^U[[A A[[ .u{{m;KLMM'00149V+STTU[[ r6Q;1!4LM  r6Q;1!4LM  A& &'1,03I!NdJdIe f   {{ekk!**Rgw7!Q,..r2BB**Rgw7"a<//33BRw!778JH GE C"HE C2#IE D D 5=2 D B3;D DGGEKK(A- .E DGGEKK(A- .E {{ekk!..U__R%89..U__R%89gw/R8==aaHEOOB$71$<=BB1a1fM eqj !F [["j*@A B E Ej QF" 1a7OaAg!# %aAg!# %aAg"$ & 1 $ 1a7OaAg!# %aAg!# %aAg"$ &aAg"$  & aAg"$  & aAg"$  &aAg"$ & ^^Q^$56N= :z6LA|TYYr]; N'':6 77r"c| |jdk7rtd|jd|j\}}|j}|jdk7rtd|jd|j|k7rtd|d|jd|j}|t j t jfvrtd|d |jd zd z}|jt jk7rd }nd}|j\}} ||k7rtd |d || |z|zd|zk7rtd|d| |z|zdzdt|| ||} t j|jdd| j|| } t j|| d|zf||} |d k(r| dz| dddddf<| dz dz| dddddf<| d z dz| dddddf<| dz dz| dddddf<| d z dz| ddddd f<| dz dz| dddddf<| dz dz| dddddf<| dz dz| dddddf<n#|d k(r| dz| dddddf<| dz dz| dddddf<| d z dz| dddddf<| dz dz| dddddf<| d z dz| ddddd f<| dz dz| dddddf<| dz dz| dddddf<| dz dz| dddddf<| dz dz| ddddd f<| dz dz| dddddf<| d z dz| dddddf<| d!z dz| ddddd"f<| d#z dz| dddddf<| d$z dz| ddddd%f<| d&z dz| dddddf<| d'z dz| ddddd(f<| jdt jdd|z|z|z|)d zjddjddjdz} t j |dz|zf|j|}|jt jk7r#|j#d| |jdn\|jt j$j#d| |jt j$jd|j|d|zS)*z This function performs reverse of the function above - it reconstructs dense matrix from a pair of "compressed" matrix, given in the layout used by CUTLASS backend, and accompanying metadata matrix. rz*Expected 2-dimensional sparse tensor, got r$z(Expected 2-dimensional meta tensor, got zExpected meta matrix to be on z device, got matrix on z devicer%z of meta matrixr r zNumber of rows of meta matrix z4 must be equal to number of columns of spase matrix z#Number of columns of sparse matrix z different from the z<, expected according to the number of columns of meta matrixr rr1rr'Nrr+ r)r(r-r  r* r, r.r)r&r/r0rr1r r6r2rr5r!r9remptyrrzerosr=r3)rSrWrr?rrr@rA meta_nrowsrrXrVmeta_2 dense_offsetsr>s r 'sparse_semi_structured_to_dense_cutlassrksJzz|q8FY Z   < :z6L <<++B/L A F Fq* UD[[ J223F "+q!Qw19,q!Qw19,q!Qw19,q!Qw19,q!Qw2:-q!Qw2:-q!Qw2:-q!Qw 1 $+q!Qw19,q!Qw19,q!Qw19,q!Qw19,q!Qw2:-q!Qw2:-q!Qw2:-q!Qw2:-q!Qw2:-q!Qw BJ$.q!Rx BJ$.q!Rx BJ$.q!Rx BJ$.q!Rx BJ$.q!Rx BJ$.q!RxKKO QA W,V  ::aQ r"c|d}|jdddjdddD]}|D] }|| |S)a This function computes a 2:4 sparse tile by greedily taking the largest values. Since we take the largest values greedily, how the sorting algorithm handles duplicates affects the ultimate sparsity pattern. Note that this function does not have the same sorting semantics as our CUDA backend, which is exposed via `torch._sparse_semi_structured_tile` and thus returns a different pattern. cgd}gd}|jjddjD]>}|dz|dz}}||dkr#||dkr||xxdz cc<||xxdz cc<8d|||f<@y)N)rrrrT) descendingstabler rrr)flattensortindices)tile num_kept_row num_kept_colxrcs r greedy_prune_tilez7_sparse_semi_structured_tile..greedy_prune_tile(s# # $$T$BJJ A61q5qAA"|A':Q1$Q1$QT  r"rr r)unfold)r>rybatchrss r _sparse_semi_structured_tiler|sR aA&--aA6$ $D d # $$ Lr"c |jjtj}|j dddj ddd}|j dddj ddd}|j g|j ddddd}dtjdtjd z}|jtj|zjtj}|S) zH Calculates the compressed swizzled bitmask from a dense tensor rr rrr r'Ncudar[) boolrr uint8rzreshaper0rr5)r> int_bitmaskbitmask_8x8_chunksbitmask_4x4_chunksbitmask_binary_representation powers_of_twocompressed_swizzled_bitmasks r $_compute_compressed_swizzled_bitmaskr;s **,//%++.K%++Aq!4;;Aq!D,221a;BB1aK%?$6$>$>%  ! !"1 %%'(%*+%-.%! au{{6JJM &((5 Ebo  '&r")r r!rYrkr|rr"r rs' 'STB8Jf R<)'r"