sg] ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZddlmZmZddlmZGd d ZGd d ZGd deZ dZ dededeeddfdZ!deeddfdZ"deeddfdZ#deeddfdZ$deedeee%ffdZ&deedee%effdZ'deedeedeeeeefeee%feeffdZ(deedeefdZ)dZ*GddZ+y) N)deque)DictListSet NamedTupleTupleDeque)get_size_of_all_nodes) PartitionDevicePartitionerConfig get_partition_to_latency_mapping get_latency_of_partitioned_graph NodeLatencyget_extra_size_of PartitionMode) GraphModule)Nodemap_arg) split_modulec NeZdZdZdedeedeedeededdf d Zdefd Z y) DAGNodezDAGNode class maintains useful information for a partition (submodule), and its input submodules and output submodules. submodule_node input_nodes output_nodeslogical_device_ids size_bytesreturnNcJ||_||_||_||_||_yN)rrrrr)selfrrrrrs `/var/www/html/venv/lib/python3.12/site-packages/torch/fx/experimental/accelerator_partitioner.py__init__zDAGNode.__init__s+%3'2(4-?$c,t|jSr )strrr!s r"__str__zDAGNode.__str__*s4&&''r$) __name__ __module__ __qualname____doc__rrintr#r&r(r$r"rrs^ % %$Z %4j % !I %  %  %((r$rc JeZdZdZd dZdedeedeedeed eddf d Zy) DAGz$DAG class contains all the DAG nodesrNcg|_yr )nodesr's r"r#z DAG.__init__1s $& r$rrrlogical_devicesrcXt|||||}|jj|yr )rr2append)r!rrrr3rnodes r" create_nodezDAG.create_node4s- K   $r$rN) r)r*r+r,r#rrr-r7r.r$r"r0r0.sU.'    $Z  4j  c       r$r0c&eZdZUdZeed<eed<y)PartitionResultz4NameTuple used for returning DAG and a new fx moduledagmodule_with_submodulesN)r)r*r+r,r0__annotations__rr.r$r"r:r:Bs> H''r$r:c |D] }g|_ yr )r) partitions partitions r"reset_partition_devicerALs* ') $*r$ partition_0 partition_1r?rctt|}|jj|j|_|j |j ||j ||j |t|y)zGiven a list of partitions and its two partitions, combine these two partitions into a new one appending to the partitions and remove the previous two partitions from the list of partitions N)r lenr2unionrecalculate_mem_sizer5removereorganize_partitions)rBrCr?r@s r"combine_two_partitionsrJQsq#j/*I!''--k.?.?@IO ""$i k"k"*% r$cf|D] }t|_t|_"|D]}|jD]v}|j}|D]c}|D]\}||k7s ||jvs||jvs'|jj ||jj |^exy)zHGiven a list of partitions, mark parents and children for each partitionN)setchildrenparentsr2usersadd)r?r@r6rOnps r"set_parents_and_childrenrSbs  "  U E " 1 OO 1DJJE 1$1AI~!qww,4qww;N!**..q1 i01  1 1 1 r$cNt|D] \}}||_t|y)zmGiven a list of partitions, reorganize partition id, its parents and its children for each partition N) enumerate partition_idrS)r?ir@s r"rIrIzs/ "*-# 9!" #Z( r$ct}t}|D],}t|jdk(s|j|.t}d}|ru|j }||_|j||j }|D]}||vs|j||s|j}t}|dz }|ruy)zJGiven a list of partitions, mark the bfs level for each partition rN)rLrErNrPpop bfs_levelrMcopy)r? current_levelvisitedr@ next_levellevelrMchilds r"get_bfs_level_partitionrbs%(EM!eG) y  !Q &   i ()"%J E !%%' #  I%% &EJ&u% &&OO-MJ QJE  r$cXi}|D]"}|jD]}|j||<$|S)z;Given a list of partitions,return node to partition mapping)r2rV)r?node_to_partitionr@r6s r"get_node_to_partition_mappingresC)+= OO =D&/&<&< d # == r$devicesc6i}|D]}|||j<|S)z6Get a mapping from device logical ID to Device object.) logical_id)rflogical_id_to_deviceds r"get_logical_id_to_devicerks,.0 /-.Q\\*/ r$c6t|}i}i}|D]}g||<|j||<g}|D]d}|jgk7rB|jD]2}||} || j||| xx|jzcc<4T|j|f|||fS)zGiven a list of partitions and a list of devices, returns: 1. A mapping from device to partitions on it; 2. A mapping from device to its remaining memory size; 3. A list of partitions that do not have a device. )rkavailable_mem_bytesrr5used_mem_bytes) r?rfridevice_to_partitionsdevice_to_left_mem_bytesrjno_device_partitionsr@rhdevices r"get_device_partition_statsrss4G<:<24 <"$Q&'&;&; #< 3  ' '2 -':: M -j9$V,33I>(0I4L4LL0 M ! ' ' 23   r$c dtdttfddtffd }t||\}d}|D]F}tt j t jd||}|rE|S|S)z\Given a list of partitions and a list of devices, map each partition into a device. r@r?c t}|D]}|j|j}t|dk(r |jS|j|j}d}|jD]}|t ||z }|S)Nr)rLrFr2rErnr)r@r? all_nodesrRextra_size_neededr6s r"$calculate_extra_mem_bytes_needed_forzNget_device_to_partitions_mapping..calculate_extra_mem_bytes_needed_fors #u  1A!0I 1 y>Q ++ +OOIOO4 OO DD !24!C C  D  r$cD]^}||}||ks|j||jj|j|xx|zcc<yy)a3Given a partition, find a logical device for the partition The algorithm is to put the partition on the device that has just enough mem left for that partition. device_to_left_mem_bytes is a dictionary between device and its left mem size sorted by its left mem size TF)r5rrh)r@rjrwrxrpros r"find_device_forz9get_device_to_partitions_mapping..find_device_fors~* A D/2! !#;A#>>$Q'..y9,,33ALLA(+/@@+ r$TrYkey)r rrsdictsorteditemsoperator itemgetter) r?rfrzrq found_devicer@rxrpros @@@r" get_device_to_partitions_mappingrs ! !*.y/ !9, #:w7  L) #'/G/M/M/OU]UhUhijUk(l#m &y1     r$c|h}t|g}|rR|j}|jD]0}||k(ry||vs|j||j |2|rRy)z^Given a partition,check if there is a circular dependency on this partition using bfs TF)rpopleftrMrPr5)r@r^queuerRras r"check_dependencyrsm )kG#YK0E  MMOZZ (E !'KK&LL'  (  r$ceZdZdZddZdedejjde de fdZ dd e ddfd Z dd Zdd Zdefd ZdedefdZdefdZdZde ddfdZdedeeefddfdZdedeeefddfdZdZy) PartitioneraA fx module may not fit into one device. Partitioner class helps partition one fx module into submodules (partitions), so that the submodules can be executed crossing different accelerators. The main function of this class is self.partition_graph. It partitions the fx module based on the scheme specified in partition_config A DAG structure is returned along with a new fx module with submodule nodes. rNc.g|_i|_g|_yr )r?rdrfr's r"r#zPartitioner.__init__*s+-24%' r$ fx_module torch_modulepartitioner_configc ||_||_|j|_t|jdk(r t dt |j|jj j}td|Dr t dd}|D],}|jdk(rn||jjz }.t|jd}|jtjk(r(|j!|j"|j$na||j&kr|j)||j*n3|t-d |jDkDr t d |jtj.k(rT|jdj& t fd |jDs t d |j1 n|jtj2k(r'|j5|j6|j8nT|jtj:k(r'|j=|j6|j8n|j?|j@r|jA|jC}|jE|} tG| |} | S) zGiven the fx module, torch module and partitioner_config, find the partitions, do the partitions, and then return a DAG and a new fx module with submodule nodes (partitions) rz No devicesc38K|]}|jdvyw)>outputget_attr placeholderN)op).0r6s r" z.Partitioner.partition_graph..BsRDtww??Rsz.No Partition since no operations in the modulerc|jSr rmrjs r"z-Partitioner.partition_graph..Ks a>S>Sr$r{)logical_device_idc34K|]}|jywr r)rrjs r"rz.Partitioner.partition_graph..Ws&Sq'<'<&Ssz,Devices have no enough memory for the modulec3<K|]}|jk(ywr r)rrrrms r"rz.Partitioner.partition_graph..]s%..2EEsz'All devices must have same memory size!)$ graph_modulerrfrE RuntimeErrorr graphr2allrr total_sizemaxmoder aot_basedaot_based_partitionnode_to_partition_mapping#partition_to_logical_device_mappingrmfind_single_partitionrhsum sparse_nnsparse_nn_partition cost_awarecost_aware_partitiontransfer_rate_bytes_per_secnode_to_latency_mappingkl_basedkl_based_partitionsize_based_partition saturate_host do_partitiondump_dagr:) r!rrrr2total_size_of_graphr6device_with_max_memr<r;retrms @r"partition_graphzPartitioner.partition_graph/sx&()11 t||  !|, ,d//0!!''-- RER ROP P >Dww(" 4??#=#= =  > "$,,4ST  " "m&=&= =  $ $"<<"FF  !$7$K$K K  & &#7J7U7U ' !3&Sdll&S#S SMN N"&&-*A*AA&*ll1o&I&I#"&,,''PQQ(()<=#((M,D,DD))&BB&>> $((M,B,BB''&BB&>> ))+  + +    "&!2!2!4mm23c#9: r$rc|j}|jjjD]-}|jdk(r|jj |/||_|g|_t|j|_ y)z'Fit the whole fx module into one devicerN) create_partitionrrr2rrPrnrrer?rd)r!rrrBr6s r"rz!Partitioner.find_single_partitions++- %%++11 (Dww("    ! !$ '  ( &9 "*;)< &!>t!Or$c| dtf fd }i}g j}jjjD]}|j dvst jt jkrJt||j}|jdk(rN||} j||j||<|jj|jn|||krt jt jk(r"jdd}j|||}j}t||j}|j||<|jj|j|j!|||xx|zcc<j|t#jt%j_t)jj}|s t+dy)aThis method is to partition the fx module based on memory size. It uses greedy approach. The result may not be the best. The basic idea is: Step 1: Find a device which has enough memory to fit the current node, create a empty partition with the size of that device. Then keep adding the following nodes into the partition until the partition is full. Step 2: Repeat Step 1 until no device left Step 3: If some nodes are left, create a partition for each left node (single node partition). and then try to map those partitions into logical devices with enough mem left. rc t|t}tddd}jD]}|vs|j|k\s|}n|jdkrt t |dzj||S)ziGiven a node, this function is to find a logical device that could fit the node. rzis too large to fit any device)rrLr rfrmrr&r5)r6mem_size_neededrrrjoccupied_devicesr!s r"find_device_based_on_sizezCPartitioner.size_based_partition..find_device_based_on_sizes0ce call_method call_module call_functionrNz6Cannot Get a Valid Partition to Logical Device Mapping)r rrrr2rrEr?rfrrnr5rmrrhcreate_single_node_partitionadd_noderIrerdrr) r!rpartition_to_left_mem_bytesr@r6total_size_of_input_nodesrrnon_single_node_partitions!found_partition_to_device_mappingrs ` @r"rz Partitioner.size_based_partitions  v &=?#)+))+ %%++11. B__Q=O : $ A A$ G (&?t%DF(,(=(=(?I8I $ioo95 !' : :8 )&88??@Q@QR&&t,/ :>WW:55d;]. <^ doo.!>t!O,L OOT\\- )1WX Xr$ct|j|j\}}}t|dk(sJdt||jDcgc]}t||dkDs|}}i}t|dzt|zt|jkrd}|jDcgc] }||vr||vr|}}i} |D]f} |Dcgc]#}|j| j|| z k\r|%} }t| dk(rd}n&t | d} |j | | | | <h|snB|j| t|dzt|zt|jkr|jD]6\} }| j}||D]}|jj|8|jD]}t|jycc}wcc}wcc}w) aSaturate host by assigning replicates to unused devices with enough memory. It uses a greedy approach to find a next available set of devices to place all split partitions: For each used device, it searches for an idle device with minimal memory size that can hold all the partition located on that device; If the search is successful for all used devices, it then assigns the new devices' logical ID to the corresponding partition. rz2Expect no_device_partitions has 0 device, but get TFc|jSr rrs r"rz+Partitioner.saturate_host..$s !BWBWr$r{N) rsr?rfrErmminrHupdaterrhrr5print)r!rorprqrj used_devices replicated_device_to_used_devicesuccess idle_devicestemp_replicate_mapping used_deviceavailable_devices new_devicereplicate_deviceoriginal_devicerhr@rRs r"rzPartitioner.saturate_hosts_ 't E  $  $ % * \ ?DX@Y?Z [ \ *$(<<Ta37KA7N3ORS3ST TAC(,!#c*J&KKs LLP  GL(Q6V-VL &( " , A *%,,"66.{;<<%!%()Q.#G !28WX ##J/5@&z2 A  , 3 34J KC,!#c*J&KKs LLP  N. 3 3 5 @  )44J1/B @ ,,33J? @  @ (A !&& ' (_U%sG(+G(3G-(G2cPtjjfd}|S)z9Return a new fx module with submodule nodes (partitions).c"j|Sr )rd)r6r!s r"rz*Partitioner.do_partition..<s//5r$)rrr)r!r<s` r"rzPartitioner.do_partition7s+!-       5"  &%r$r<ct}|jjD]-}|jdk(r|S|jdvr%|jt j k(rCi}t|j|jt|j|jt|jdkDrt|j}n|g}t|jj!ddd}|j"|j$}|j"|j&}|j)|t||||0|S)z?Return the dag structure and the new fx module with submodules.r>rrrY_r)r0rr2rtargetr __getitem__rargs setdefaultkwargsrErOlistr-namersplitr?rrnr7) r!r<r;r6rrrV device_idsrs r"rzPartitioner.dump_dag@s e*0066 Dww(", +ww55{{h222,.K DII{55 6 DKK!7!7 8 4::"#DJJ/ $v tyy//Q7;.combine_partitions_based_on_size..s QEUEUr$r{N)r~rbr?)r?rmfind_combinationsorted_partitions'find_partition_to_combine_based_on_sizer!s r" combine_partitions_based_on_sizezIPartitioner.sparse_nn_partition..combine_partitions_based_on_size|sJ" $ "$*:;U$V!'8/V%':J0, * # r$c|jj|j}d}|D]}|t||z }|S)zuGiven two partitions, calculate how many mem bytes are needed if two partitions are combined r)r2rFr)p1p2r2mem_bytes_neededr6s r"calculate_mem_bytes_neededzCPartitioner.sparse_nn_partition..calculate_mem_bytes_neededsJHHNN288,E  C $5dE$BB  C# #r$rcpd}|jd}|dddD]}t|j|jz dks)||}||ks8t||j|j ||j ||j jdd}||fS||fS)z+step 1 in combine_partition_based_on_size()FrNrrYT)rZabsr[rJr?rHr5) rrmr?rsmallest_partitionrRrrr!s r"rzPPartitioner.sparse_nn_partition..find_partition_to_combine_based_on_sizes % !2!6!6q!9 &tt, )33akkABaG'A!EW'X$'+>>.q2DdooV"))*<="))!,"))$//"*=>+/(#Z/ / $Z/ /r$crj|nj||rj}|_|Sy)zyIf crossing the boundary between non-embedding nodes and embedding nodes, create a new partition N)r5rleft_mem_bytes)r@ new_partitionrmembedding_partitionsin_embedding_regionnon_embedding_partitionsr!s r"reset_partition_in_sparse_nnzEPartitioner.sparse_nn_partition..reset_partition_in_sparse_nnsF#$++I6(// : 113 +> (  r$r6c|jdk(rmj}t|jj dD]:}t ||st d|d|t||}dt|vs:yy)z$Check if a node is an embedding noder.zModule z has no attribute EmbeddingTF)rrr&rsplithasattrrgetattr)r6 submoduleatomr!s r"is_embedding_nodez:Partitioner.sparse_nn_partition..is_embedding_nodesww-' --  ,2237$D"9d3*%i[0B4&I!( 4 8I"c)n4#$r$F>rrrrz!is too large to fit into a device)rzNeed z devices, but only z provided partition_zN(embedding partition) and non embedding partitions can not fit into one device)T)rr r-rboolrrrrr2rrnrrrrrSr?rErfr&rUrVrhrr5rerd)r!rmrrr r@r6r&total_size_of_non_embedding_partitionsmsgrrWrrrrrs`` @@@@@r"rzPartitioner.sparse_nn_partitionjs $ Y >A   8 $ 0#I 0!$ 0Y 04i( )  0*   D T 1346 $)))+ %%++11 )DwwII$T*.AA!//14$@$K .A*A',=dIOO,T)- 0H0HH)*!=Y GI0A$ 0X-03FF* KK*MM""4(+ ), %YeD 1()ACVW()=?RS12.1 OI 2i6N6N N 2 O # $s4<<'8 8c./01'(c$,,'()  s# #%&:; DLAy79Q9QQ%&# )0012fg15 Q0J0J/K , '' Q(B(BC D 2 .try_combining_partitions/sH%BH%B R\\BLL01Q6"**$"++&&r2z:#JrN3 <'&z2?   $ <'/O 70,80/  < r$c t j|}t j||}t jdk(ryg}t t jdz D]`}t |dzt jD]9} || jdd}||kr||g}|}t j;bt|dk7r; j|d} j|d} t || jt jt jt j jt|dk7S)aGiven transfer rate between partitions and each node's latency, find two partitions to combine so the cost of the partitions can be reduced. The algorithm is : 1. Go through all the partition pairs and see if any pair of partitions can be combined. 2. Calculate the cost after the combination. 3. Select the minimum cost and combine its corresponding partition pair. rYFNr) rr?rrErangerIrJrbrArrf) rrrrpartition_pairrWjnew_costrrr!rs r"search_combinationz.search_combinationUsN,L!8, (4,+D 4??#q((*N3t/!34 ;q1uc$//&:;;A 81dooa>PQH4'*+Q')$//:; ;>"a'__^A%67__^A%67&r2t? #DOO 4 "4?? 3 ,T__dll K~&!+ +r$>rrrTN) rr rrr2rrrSr?rbrIrerd)r!rrrr6rrs``` @r"rz Partitioner.cost_aware_partitions*$ $ L( , ( ,T%%++11 8DwwCC11$7 8 !10 2+-D   doo.!>t!Or$c Bdfdfd}jtj|}tj|}g}g}g}jj j D]"} | jdvs|j| $|D]} j| } j| } tjD]7\} }| | k7s j| }|| | ||\}}||ks0|}|}| |g}9t|dk7s|d|d|d|dtjtjjtjtjjy)aThis function is a cost aware partition based on Kernighan-Lin algorithm. First, the graph is partitioned using size_based_partition. Then, each node is swapped with any other node in a different partition, and at the same time, the cost is estimated after the swapping. For example, we have nodes n0, n1, n2, n3 and n4. Using size_based_partition, n0 and n1 are in Partition p0. n2, n3 and n4 in Partition p1. The current cost is estimated. We first tried using n0 to swap with n2 from the other partition. Then we see that swapping n0 and n2 shows a lower cost than the current cost and it is the minimum among other pairs like (n0, None)(This means moving n0 to Partition without swapping other nodes), (n0, n3) and (n0, n4). We swap n0 and n2 and set the new cost as the current cost. Then We repeat this process for all the other nodes until all swapping pairs are tried. c|"|j||j||#|j||j|yyr ) remove_noder)n0n1rrs r" swap_nodesz2Partitioner.kl_based_partition..swap_nodessA~r" B~ Br"r$ctd} ||||t jt|s{t|spt jt j|}t j j}|s td}nt j| } ||||t jt jt j j|S)Nr) rrIr?rrArrrfr) r"r#rrrtransfer_rate_per_secrrrr!r$rs r"try_swap_nodesz6Partitioner.kl_based_partition..try_swap_nodessrr)rr2rr) r6rrrr&p1_nodesmin_cost node_pairr#rr's r"swap_node_to_partitionz>Partitioner.kl_based_partition..swap_node_to_partitions BHH~.HU|H$&I $>bee/J&J%"b"&=?T(?!%r I#H $? "r$>rrrrrYN)rrr?rrrr2rr5rdrUrErIrrf)r!rrr,rrr+rop_nodesrQr6rrrrrr new_node_pairr$r's`` @@r"rzPartitioner.kl_based_partitions0 # @ #. !!#'G OO4( $0 OO9;V !# *,""((.. #Att@@" # PD--d3H*B )9 2 !x'2B.D/3 /+Hm $'$1 *,b 2"9~"aL)A,q0A>RSCT&doo60$,,O9 P: doo.($,,Gr$ci}||_|jD]n}|j|}||vr6t|}|jj||||<|||_n||j|}|j |py)zqThis function helps to rebuild the partitions given the nodes and its corresponding partition id N)rdr r?r5rr)r!rr!partition_id_to_partition_mappingr6rVr@s r"rzPartitioner.aot_based_partition s CE)!:** %D11$7L#DD%l3 &&y1BK1,?/R 0 ,>**40    t $ %r$r8)r)r)r*r+r,r#rtorchnnModuler r:rr-rrrrr0rr rrrrrrrrrrr.r$r"rr s( PPhhooP. P  Pf=>69 "^@C(J&k&{s:) nsntn`w%*w"&dK&7!8w  wrK%*K"&dK&7!8K  KZ%r$r),r collectionsrtypingrrrrrr r1"torch.fx.passes.graph_manipulationr 'torch.fx.experimental.partitioner_utilsr r r rrrrrtorch.fx.graph_moduler torch.fx.noderrtorch.fx.passes.split_modulerrr0r:rArJrSrIrbr-rerkrsrrrr.r$r"r;s<< D   .'5((.  ((j(F*   )2 @DY   " i T 0 d9o $  Y D 8d9o$tSy/ d6l tCK7H "Y"*.v," 4Y' ($vs{*;T)_ LM"J4Y4*.v,4n$W %W %r$