sgSPddlZddlZddlZddlZddlmZddlmZmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZddlZddlmZddlmcmcmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/dd l0m1Z1m2Z2dd l3m4Z4m5Z5dd l6m7Z7m8Z8dd l9m:Z:m;Z;mZ>m?Z?ddl@mAZAerddlBmCZCejeEZFeGddZGde eHe fdeeeHe ffdZIeGddZJGddeZKGddeZLdeGde eHe fdeMdeMdeMdee eHe ff d ZNd!e deMfd"ZOdeGde eHe fdeJfd#ZPdeGd$eJdeMdee eHe ffd%ZQd&e(d'e eHe fd(eejde eHe ffd)ZSd&e(d$e d(eejde fd*ZTdeGd+eHd'e eHe fde eHe ffd,ZU dfd-e eHe fd.e!jd/eMd0eejjd1eMd(eejde eHe ffd2ZYdeGd3e eHe eHe ffd4eeHde eHe ffd5ZZd6eHd7eejd4eeHd8eejd9e2dejf d:Z]d6eHd;eejd4eeHdejfd<Z^d6eHd=ee d4eeHde fd>Z_ dgd?e eHe fd.e!jd0ejjd@eeee eHe fe e!jfdAeMdBeMde eHe ffdCZa dhd.e!jd@eeee eHe fe e!jfde ebe!jffdDZcd.ejBjde e1eHffdEZd did0ejjd.ee!jdBeMdFee e!jeeHfdGee e1eHfde eebeHfe!jff dHZe did0ejjd.ee!jdBeMdFee e!jeeHfdGee e1eHfde e!jeebeHfff dIZf dhd.e!jd@eeee eHe fe e!jfde e!jebffdJZgdKeeLdLe eLeeHebffdMe eeHebfe!jfd(eejddf dNZh dgd-e eHe fd(eejdMe eebeHfe!jfdFe e!jeeHfdOe eHeGfdPeMdeeeLe eLeebeHffffdQZidRe eHe fdMe eebeHfe!jfdFe e!jeeHfdee eHe ffdSZjd-e eHe fdeMfdTZkeGdUdVZld&e(dWe eHe fdee eHelffdXZmdeGdYee eHelfdWe eHe fdZe eHe eHe ffdeeeje eHeeejfff d[ZodeGdZe eHe eHe ffd6eHdeMdeMdeMddfd\ZpdeGdYee eHelfdWe eHe fdeMdeMdeMde eHe eHe fffd]ZqdeGdWe eHe fdeMdeMdeMde eHe ff d^Zr djd_eeLdLe eLeebeHffdOe eHeGfd-e eeHebfe fdeMdeMdeMde eHe ffd`Zs djd_eeLdLe eLeebeHffdOe eHeGfd-e eeHebfe fdeMdeMdeMde eHe ffdaZtej dkd.e!jd0ejjd-e eHe fd@eeee eHe fe e!jfd1eMdeMd(eejdAeMd/eMdeMde eHe ffdbZvd.e!jde eHeGffdcZwed&e(dde;ddfdeZxy)lN) ExitStack) dataclassfield)AnycastDictIterableIteratorList NamedTuple no_type_checkOptionalSequenceSetTuple TYPE_CHECKINGUnion)_gather_state_dict)_get_pg_default_device)_apply_to_modules _FSDPState._get_module_fsdp_state_if_fully_sharded_module_get_param_to_fqns_module_handle!_named_parameters_with_duplicatesclean_tensor_name)SimpleProfiler) FlatParameterFlatParamHandle)_ext_chunk_dtensor_ext_chunk_tensor) _lazy_init%_reset_flat_param_grad_info_if_needed)ShardingStrategyStateDictSettings StateDictType)DTensor Replicate) tree_map_only) ShardedTensorcFeZdZUeed<eed<eeefed<e e ed<y) FSDPParamInfostatehandle param_indicesparam_requires_gradN) __name__ __module__ __qualname__r__annotations__rrstrintr boolV/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/fsdp/_optim_utils.pyr,r,Bs&  S>!d#r9r, dictionaryreturnc#^Kt|j}|D] }|||f ywN)sortedkeys)r;r@ks r: sorted_itemsrBJs5 *//# $D As+-ceZdZUdZeeZeee jfe d<eeZ eee jfe d<eeZ eeefe d<y)_ConsolidatedOptimStateap This holds the consolidated optimizer state on the target rank. Positive- dimension tensor state is communicated across ranks, while zero-dimension tensor state and non-tensor state is taken directly from the target rank. PyTorch version 1.12 moved to using zero-dimension tensors for scalar values, but user implemented optimizers may still use float (i.e. a non-tensor). Thus, we support both and handle them identically. Attributes: tensor_state (Dict[str, torch.Tensor]): Mapping from positive-dimension tensor state name to the unsharded flat tensor representing the state. zero_dim_tensor_state (Dict[str, torch.Tensor]): Mapping from zero- dimension tensor state name to its value. non_tensor_state (Dict[str, Any]): Mapping from non-tensor state name to its value. )default_factory tensor_statezero_dim_tensor_statenon_tensor_stateN)r1r2r3__doc__rdictrFrr5torchTensorr4rGrHrr8r9r:rDrDPs\&-2$,GL$sELL()G5:45P4U\\ 12P',T'Bd38nBr9rDcNeZdZUdZej ed<ejed<y)_PosDimTensorInfoa Meatadata for positive-dimension tensors used internally for :meth:`scatter_full_optim_state_dict`. Attributes: shape (torch.Size): Sharded tensor shape (which is equal to the unsharded tensor shape if the tensor is optimizer state for a non-FSDP parameter and is hence not sharded). dtype (torch.dtype): Data type of the tensor. shapedtypeN)r1r2r3rIrKSizer4rPr8r9r:rNrNjs  :: ;;r9rNc0eZdZUdZeedfed<eed<y)_OptimStateKeyz This represents an optimizer state key that may be used commonly across ranks. It is based on the unflattened parameter names rather than parameter IDs to make it independent of each rank's own optimizer construction. .unflat_param_namesis_fsdp_managedN)r1r2r3rIrr5r4r7r8r9r:rSrSzs c3h'r9rSfsdp_param_infoflat_param_stateto_save shard_state cpu_offloadc |r |sJdt||}|rjt|||}|D]V}|st|jD]5}||} t | t j s#| j||<7X|SgS)aO Unflattens the optimizer state, consisting of the "state" part and the "param_groups" part. Unflattening the "state" part involves consolidating the state on the target rank and remapping from flattened to unflattened parameter IDs, and the "param_groups" part only involves remapping from flattened to unflattened parameter IDs. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): Entry for the flat parameter in the "state" part of the optimizer state dict. to_save (bool): Whether to save the state on this rank. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter if on the target rank or an empty :class:`list` otherwise. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. z7If ``shard_state`` is True, ``to_save`` has to be True.)_communicate_optim_state#_unflatten_communicated_optim_statelistr@ isinstancerKrLcpu) rVrWrXrYrZconsolidated_stateunflat_param_state optim_statekeyr-s r:_unflatten_optim_stateres< 7A@A"1@     . 3K 0 0 233C',E%eU\\: ',yy{K$ 3 3"! r9xcVtj|xr|jdk(SNr)rK is_tensordim)rfs r:_is_zero_dim_tensorrks ??1  .!%%'Q,.r9c|j}|jj}t}|j|j |j }}}t|D]\}} tj| rB| jdkDr.|jdk(s|jtjk(r| ||<c|jJd| j j"|jj"k7r| j%|j} |j&j)} | j*| } t-j.| | |j0|j2j5t7t8j:|j<j?} | d| ||<_tA| r#| jCjE||<| ||<|S)a Communicates the optimizer state for a flat parameter across ranks. All ranks will hold the entire non-sharded optimizer state on GPU. If ``N`` is the number of tensor optimizer states in the optimizer state dict, then the communication complexity is 0 if ``N = 0`` and ``N + 1`` otherwise (where the plus 1 comes from all-gathering the padding per rank). Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): The entry in the "state" part of the optimizer state dict corresponding to the flat parameter. Returns: ConsolidatedOptimState: Consolidated optimizer state for the target flat parameter. rNz'compute_device has not been initializedgroup)#r-r. flat_paramrDrFrGrHrBrKrirj world_sizesharding_strategyr$NO_SHARDcompute_devicedevicetypeto_full_param_paddedsize new_zerosdistall_gather_into_tensor process_group_device_handle synchronizernn Parameter_unpadded_unsharded_sizenumelrkdetachclone) rVrW fsdp_staterpr-rFrGrH state_namevalue buffer_size tensor_bufferunpadded_numels r:r\r\s,!&&J ''22J # %E  ## *:'L **:;#5 E ??5 !eiikAo %%*//3C3L3LL+0 Z())5 98 95||  J$=$=$B$BB!:!:;%77<<>K+EOO[9M  ' 'uJ,D,D   % % 1 1 3! jAAeg (5_n'EL $#5)49LLN4H4H4J%j1/4 ,G#5H Lr9r-c R|j}|j}|j}g}i}|j}|j|j |j } } } t|D]?} i} t| D]\}}||v}|s|j|}|||<n||}t|}|r|j}t|ddr;|jJt||j|j|j }n^|j"Jt%||j|j&|j(j+|j"|j }|| |<t| D] \}}|| |< t| D] \}}|| |< |j-| B|S)a Unflattens the communicated optimizer state (given by ``tensor_state``, ``non_tensor_state``, and ``zero_dim_tensor_state``) for a single flat parameter. This should only be called on the target rank. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. state (_ConsolidatedOptimState): Consolidated optimizer state. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. _use_dtensorF)r-r.rp _num_paramsrFrGrHrangerB_get_unflat_viewsnext_optim_state_dict_configgetattr _device_meshr rank_fsdp_extensionr}r!rqr~ device_countappend)rVr-rYrr.rprbflat_param_viewsnum_unflat_paramsrFrGrH_unflat_state_paramr flat_tensorviews_generatedviewsrc osd_configzero_dim_tensor non_tensors r:r]r]s,!&&J  # #F""J/1,."..  ## *:'L $ %'6'3L'A 9 #J (,<>>"4#""//"22 #K&33???"3#""--"11>>@"00"22 #K.9 z *9 9>,88M+N = 'J-< z * ='33C&D 8 "J -7 z * 8!!"45O'6P r9rrcrocdg}tj|dk(rttjd||d<tj |d|tj|dk(r|S|dS)Nrc|jdk(r|jSt|j|jSrh)rjr`rNrOrP)vs r:z,_broadcast_processed_state..\s,Aaeeg3DQWWagg3Vr9srcro)r{get_rankr)rKrLbroadcast_object_list)rrcroobjectss r:_broadcast_processed_staterSsl G }}Uq " LL V    wAU; }}Uq qzr9ctj|dk(rKt|tjr|j dk(r|S|j |j}n~t|tjr|j dk(sJd|St|ts|Stj|j|j|j}tj|d||S)NrzlFor non-zero ranks, a tensor state should have zero dimension, but got the state with shape {state.shape()}.rPrur) r{rr_rKrLrjrwrtrNzerosrOrP broadcast)rr-rotensors r:_broadcast_staterfs }}Uq %.%))+2BL*334 eU\\ *99;!# @ #LE#45L KKu{{:3L3L  NN6q. Mr9fqnc.|siS|j}|jj}|j|}|j|}t ||j |j}|jsiSi}|j}|j} |jD]s\} } tj| rT| jdkDrA|jt j"k7r$| j%|| dzj'} | || <u|S)z Shard the optimizer state for the original parameter with the name ``fqn``. This API should only be used when ``use_orig_params`` is True. pgrurrm)r-r.rpr/_shard_param_infosrr}rtin_shardintra_param_start_idxintra_param_end_idxitemsrKrirjrrr$rsflattenr) rVrrcrrp param_idxshard_param_infonew_optim_staterrrrs r:_shard_orig_param_stater}s   &&J ''22J--c2I!44Y?$ 009R9RK  $ $ &(O,BB*>>(..0, E OOE " a,,0@0I0IIMMO$9J8H"$4S$9*$EU9$S)*5 q'C ( (4S9O#++N,?,?,J,JK!8'(-"J2#$ !td3C&0s# INSJ4&PQRS"$!KKOOE48E(.2mmE.Bs+ *3%/? ?PPQR ##C5),, t9> !> te4C"&)), "4    0 0  . /'',,     K1   %   05578Kc"4"9"9";<< ##5"68   "" ""$J!: ': "/A.L z *RV V  -9 JqAM J J$%)Jz " GKKK2_& 6A 5??1#5#E!%%'A+ E  $7$: : 5??1#55 5O 6#88Qa88 u:? #7? @  f E_.AB  >c%jAoUV V  FLLE$9$ll^1UG5  e$J#&o7J"K  K  "  knnZ01 ]] KK!     (();V=R=RSK!::    0 0 {0012+, . 0 WE 0. sFFF!.AF&zero_dim_tensorsc |Dcgc]}|| }}|Dchc]}||jnd}}|Dchc]}| |jnd}}t|t|k7st|dk7st|dk7rtd|d|d|d|t t |}t t |}t j||t jdScc}wcc}wcc}w) a Flattens the zero-dimension tensor optimizer state given by the values ``zero_dim_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all tensors are the same and using that common value. NOTE: The requirement that the tensors are the same across all unflattened parameters comprising the flat parameter is needed to maintain the invariant that FSDP performs the same computation as its non-sharded equivalent. This means that none of the unflattened parameters can be missing this state since imposing a value may differ from having no value. For example, for Adam's "step", no value means maximum bias correction, while having some positive value means less bias correction. Args: state_name (str): Optimizer state name. zero_dim_tensors (List[torch.Tensor]): Zero-dimension optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: torch.Tensor: A zero-dimensional tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. NrmAll unflattened parameters comprising a single flat parameter must have scalar state with the same value and dtype but got values z and dtypes rrr`r) itemrPrrrrrKrru) rrrTrr values_setr rrPs r:rrsB$4Eaq}EE?OP!am!&&(5PJP:J KQaggD0 KF K %5!66 z?a  v;!  (\fX[l;!"  $   j! "E f E <<U5<<3F GG%FP KsC.C.C3C8 non_tensorsc|Dcgc]}|| }}t|}t|t|k7st|dk7rtd|d|d|tt |}|Scc}w)a Flattens the non-tensor optimizer state given by the values ``non_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all values are the same and using that common value. See the note in :func:`_flatten_zero_dim_tensor_optim_state`. Args: state_name (str): Optimizer state name. non_tensors (List[Any]): Non-tensor optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: Any: A non-tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. rmrrz" and unflattened parameter names )rrrrr)rrrTntnon_none_non_tensorsnon_tensor_setrs r:rrFs2*5G2BGG%N  C $44N8Kq8P ,-[ E++=*> @  d>*+J Hs A-A- sharded_osd optim_inputusing_optim_inputis_named_optimizerc t|}t|}tttj t ttff|r t||nt|||||}t|t|ksJi} i} |jD])\} } | |vr || } | | t| <| D]}| | |< +|d}i}|jD]F\}}t|tr|||<| j|j |j } ||| <Hd|vr[g}|dD]L}t#j$|}t'|dDchc]}| | c}}||d<|j)|N||dSd|iScc}w)a Rekeys the optimizer state dict from unflattened parameter names to flat parameter IDs according to the calling rank's ``optim``, which may be different across ranks. In particular, the unflattened parameter names are represented as :class:`_OptimStateKey` s. r-rparamsr)r_get_flat_param_to_fqnrrrrrr6r5'_get_param_to_param_id_from_optim_input_get_param_to_param_keyrrrr_rrTrrr?r)rrrrrrrflat_param_to_fqnparam_to_param_key$unflat_param_names_to_flat_param_key#unflat_param_name_to_flat_param_keyrrTflat_param_keyrsharded_osd_staterekeyed_osd_staterdrrekeyed_osd_param_groupsunflat_param_groupflat_param_groupflat_param_keyss r:_rekey_sharded_optim_state_dictr,ms &'u-M.u5>B R\\5c? *+! 4E; G(u0-AR ? ! "c-&8 88 8 ) (&3%8%8%:T!! * * +E2JX,U3E-FG!3 T ES /0A B T T$G,46-3358[ c3 %0 c " =AA  " "C$:$: -8.)8$9; "-n"= > #}}-?@ $.@-I)88IJO *9 X & $ + +,< = >+??KI-z%66 Zt,, - ySTTIf%&& 9,.8 ![0  & ~   X!678  +, --;  "m %  s*D D(D##D(c td}d}i}t|||t|Dcgc]\}}| c}}|Scc}}w)a Constructs a mapping from ``FlatParameter`` to a cleaned (devoid of prefixes from wrappers) fully qualified name (FQN). Note that this FQN is "non-canonical" because ``FlatParameter`` s do not come from the original module but are registered only after FSDP has been applied. This function returns the FSDP-given name for the ``FlatParameter`` (usually module._flat_param) as opposed to the canonical FQNs returned for ``FlatParameter`` s in ``_common_utils._get_param_to_fqns(...)``). Consequently, this function will only return a non-empty mapping if FSDP was applied with ``use_orig_params=False`` as, otherwise, the original parameters are used within the module and there would be no ``FlatParameter`` s in the module. cvt|dD])\}}t|tst||z}|||<+y)NF)recurse)rr_rr)moduleprefix tree_levelr! param_namerrs r: module_fnz)_get_flat_param_to_fqn..module_fnsI!B E"  + Je]3#FZ$78C'* e $  +r9c|Sr>r8)r!s r: return_fnz)_get_flat_param_to_fqn..return_fn   r9rr)rr@rBflat_param_to_fqn_retrrs r:rrsJ+!79   @FCi#"3' =< = =$E *F,,L 12%++fUKOOKu-A"'(;(;E(B"CDQDDD xUC /  $H- JD$^,d3D4y1}dV#J:>?3..?@B  9%%D@s  E":E'r merge_keysctj|}i}g}|jD]\} } | |dvr|| } t| t} | r*| d|vs#J| dt |j f| d|v} tt| | } |dk(s|r|j| | || <|rsttj|Dcgc]}g}}tj||||Dcgc] }|D]}| }}}tt|}||fS|dk(r|gndg}tj|d||dJ|d}t!||||||fScc}wcc}}w)a@ Construct the local mapping between the ``_OptimStateKey`` and parameter keys and all the ``_OptimStateKey`` across ranks. If ``merge_keys`` is False, rank0 must contain all the ``_OptimStateKey``, an exception will be raised otherwise. Note that ``merge_keys`` should equal to ``use_orig_params``. r-r)rTrUrnNr)r{rrr_rr^r@rSrrrrUrVr?rrr])rrorJrrr^rrQall_optim_state_keysrYrrrUoptim_state_keyrall_keys local_keysrdmerge_all_optim_state_keys key_obj_lists r:_map_param_key_to_optim_keysrfs == DJL 13.446B 5 ,W5 5 U#$UM: 744 Q+00237 4q'%;;($T{+  19 ' ' 88A$_5'B*d11%890 B0 0  x)=UK"*& j& /2C& & "&  &c*D&EF != =='+ai ! "dV  "":K U[[;O++J7;@++J7 @%4S!4 && 17s D rv output_statesc i}|jD]`\}}|Dcgc]}|| }}t|D chc]#} | jjD]} | %c} } } t } d} | D]%}g}t }t |D]\}}|j d|jj|d}|=|jj|d<| s |j} n| |jk(sJ|ddk(s{|j|| r| |k(sJ|} ||vr|jDcgc]}dc}||<||j|d}|%|j|jj}||||j|<(t |D]\}}|| vr |j jD]:\}}|j|d}|||k(sJd|d|d|dd|z|||<<|j"jD]K\}}|j|d}|-t%j&||sJd|d|d|dd|z|||<Mc |fScc}wcc} } wcc}w) a1 Given the ``gathered_state_info`` and ``input_states``, the API converted the StateInfo into the original state if the state is not a non-scalar tensor. For a multi-dimensional tensor, the local state will be stored in ``state_buffer`` in a correct order for later allgather purpose. NrzRank z has different values for z: rz Other ranks: )rr?rqr@rr.rrrOrrPaddr/rwr-rtrrrrKequal)rVrvrsry state_buffersrgathered_states state_infor-nall_tensor_states empty_ranksrPrnumels _empty_ranksr object_stateinfor local_statenamenon_tensor_valuecurr_non_tensor_valuescalar_tensor_valuecurr_scalar_tensor_values r:_convert_all_state_inforCs1>@M,224?;^&9:af: :"& E5 0B0B0D E1Q EQ E !$ '+, XJF%(UL&/ &; +"l a #++// DA#!%!1!1!3F2J $ $ 222":? $$T* +#k\&A AA&K."1"?"?-D- j)'s+// DAK&)nn_-B-B-Q-QR LWM* %o&C&CC&H I7 X@#,J"7 ; D,{"*6*B*B*H*H*J 8&&(6(:(:4(F%)1,0@@D6!;D6DTCUUVW&'<&=>? A (8t$ 8.:-H-H-N-N-P ;))+9+=+=dD+I(/75;;')A<D6!;D6DWCXXYZ&'?&@AB (;t$ ; ;S?;B - A; E0-s I;(J : Jc D|sy|jj}|j}|jD]O\}} | |} |j|} t | t r| jd} | tk7r| j} | jtf}t|j| }|| xx| jjdzcc<tj |}| j#|} n=| j#|j| } n| j#|j| } |r|j$}t'|ddr;|j(Jt+| |j,|j(|j.} n|j0Jt3| |j,|j4|j6j9|j0|j.} n>|s59!..:::*OO++.. "//;;;)OO))--::<,,.. ''0 / ,,. / ''(;(;(?(?@ $  $%*z"e2+X / / $ $sJ (J J J c |j}|jdk(r]tjtjj k(r.t jd|jj|jDcic]}|i}}t||||\} } t| dk(r|S|jjD cgc] \}} ||vrdnd} }} |jj } t#j$t&j(| |j*}|| j,}|jj/| jD]\}}g}|j| j0j3z}|| j0j3zdz }d\}}t5| j6| j8D]\}}| xr|j:| xr| | }|s|r|||zdz }}||cxkr|krnn||k\r||z dzn||z dz}n=||cxkr|krnn||kr||z dzn||z dz}n||cxkr |cxkr|krnn|}nd}|r|j=|||s6||,|j=t?t&j@|||dz }||z }| j0j3tCd |Dz }| jD|k(sFJd | jDd |d | j0j3d | j6d|d|d |dkDr|j=||t'jF|}|j3|jHz|j3k(sJd|jj/tKjLtJjNjP5tjR|||jT|jj/ddd|d| jVj3}|j} | jY|}!t|!t|jk(sJd|jjD]$\}} |j:| s||vs|!| |||<&t[||||||~|Scc}wcc} }w#1swYxYw)z Given the ``gathered_state_info`` and ``input_states``, the API allgathers all tensor states and restore non-tensor states from ``gathered_state_info``. rz@Memory Summary before calling to _allgather_orig_param_states %sTFrrm)rrNc3<K|]}|jywr>)r).0rs r: z/_allgather_orig_param_states..Ps1a 1szLManually calculated _sharded_numel_padded is incorrect. _shard_numel_padded=z, shard_numel_padded=z, _sharded_size.numel=z, _numels_with_padding=z, begin=z, end=,zThe size of local shard times the world size should equal to the gathered tensor size. The inconsistency may be from a bug of FlatParameter's metadata or the reconstruction logic in optimizer state dict.rnzThe number of parameters from FlatParameter is not consistent to the number of states used by optimizer state dict reconstruction logic.).r-rr{get_debug_level DebugLevelDETAILloggerrr~memory_summaryr@rrr/rr.rp functoolspartialrKemptyrt_padded_unsharded_sizer _sharded_sizerr_numels_with_padding_is_padding_maskr0rrrLsum_shard_numel_paddedcatrqrrr ALLGATHERr|r}r_get_unflat_views_alignedr)"rVrvrsrYrXrZrrryrPr~idxhas_state_paramsrp empty_funcgathered_tensorrbuffers local_buffersbeginend mem_offsetrr is_paddingfrozen_and_no_state padding_begin padding_end padding_lenshard_numel_padded local_shardunpadded_tensorflat_param_handle orig_statess" r:_allgather_orig_param_statesrso!&&J! 4 4 6$//:P:P P N  % % 4 4 6 DPCTCTCV/WCR/WM/W2,lME= =Q(55;;=$ C}$%/$$!''22J"" 5)B)BJ!!B!BCO))+,224r  G,. *":":"@"@"BBj..4466: $ I!$  + +Z-H-H" 8 E:'1.##77 BB4(33 0 .8e9Ka9O{  E8[8 +-$e+a/ 5[1_ #c8[8 !M1m+a/ 5[1_ ]?k?C?#(K"#K!((K)@A9%1!((ellGI!? @ii .   "Z%:%::o>S>S>UU   U !!--/  # #N$7$7$A$A B 4  ' 'J4L4L   % % 1 1 3  4**WJ,O,O,U,U,WX+22'AA/R ;3'D'D#EE   E (55;;= BHC22373-;O1 > ?   # #N$7$7$E$E F R"7 L"Q  R4          '55;;= HCm#"66s;%8/77<<>?@A8-,,./0 3   5 R R   s$,.E E'EE EE$r`c i}i}|D]P} |j| d} | | js%| jrZ| jd} |j| d} | U| in|| } t| |vri|t| <| |t| | <|st | jdk(sJ| jd}t j d5ttttf| } tj|| ||<|rBt||D]1\}}tj|s|j|||<3dddS|j!D]}t#t%|j'} || } t | j(dkDsJd| j*j-D]e\}}||vr | j(|st/|dt1| j*j'dt1|j'd|j3t5| |||||S#1swY[xYw)Nrrmnone_fsdp_managed_copyzgWith use_orig_params, FSDPParamInfo should have requires_grad information. However, the length is zero.zE is not in the optimizer state. The FSDPParamInfo has the param keys z( while the optimizer has the param keys r)rrUrTidrrrrrr5r6rrBrKrir`rrrr@r0r/rrr?updater)r`rQrrrXrYrZfsdp_osd_state all_statesrarYrrVr-rrr _all_statesrdrs r:_convert_state_with_orig_paramsrs&(N -/J0&T+G+K+K T,   _%D%D   * *!44Q7C488dCO&#+B1A)1LE/"*424 2o./38Jr/* +C 0 99:a? ?? / B B1 E ''(@A T sCx)< 48II$Y/501-9&'89.T) E %u5$HM '89*E T T T7&TT"((* 4 ((*+,05?667!;  8 ;(55;;= HCk!"66s;%8/77<<>?@A4+**,-.a 1    (   ' : Y T Ts A=II" cni}|D],}|j|d} | Jd|d| |jrp|jd} || } t| || |||} |sZt | t |jk(sJt |j| D] \} }||| < |st |jdk(sJ|jd} t j || || <|st|| D]1\}}tj|s|j|| |<3/|S)NzQIf use_orig_params is False, we must be able to find the corresponding param id. rrrm) rrUrTrerrrrBrKrir`)r`rQrrrXrYrZrrarYrrV unflat_staterrbrrs r:_convert_state_with_flat_paramsrs&(N0(P+G+K+K T, $  ''6&7q  E $  * *"44Q7C4S9O1 + L<(C0R0R,SSSS=@#66 >K9%'99KN#45 K 99:a? ?? / B B1 E 04 :J9:U0VN, -)5"#45*P%J!??51 DIIIKN#45jA PG(PT r9c tjt} | jtjtj j ttj|| xstj|dk(xs|} tjd5t|} t|} t|}tt t"t$t&ft(j*f|r t-||nt/|||| | }t1|}dddtjd5t3|| |\}}dddtjd5|rt4nt6}||d| || }ddd| siSdi}t9 j;}|dj=D]2\}}||vr ||vr|vrt?j@d|d |||<4d |vrtC| |d <| jEtjFd |S#1swYxYw#1swYxYw#1swYxYw) a Consolidates the optimizer state and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flat parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. Parameter keys are not well-defined. For a regular optimizer, the optimizer state_dict contains a mapping from parameter IDs to parameter states. Parameter IDs are the order of parameters in ``optim.param_groups()`` across all the groups. This API also allows user to pass ``optim_input`` for the mapping between parameters and parameter IDs. Using ``optim_input`` is being deprecated. If the optimizer is a ``NamedOptimizer``, the optimizer state_dict does not contain parameter IDs mapping but a mapping from parameter FQNs to parameter states. This API finds the mapping from FQNs to parameters if the optimizer is a ``NamedOptimizer``. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP knows how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- those are managed by other parallelisms and FSDP does not know how to handle/aggregate them. Args: model (nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) shard_state (bool): If ``True``, shard and distribute all non-zero-dimension states. Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=False``, then nonzero ranks return an empty :class:`dict`. r preprocessingNpreprocessing_with_comm)r^state_convertingr-zFound a optim state, aK, that FSDP cannot process. FSDP will directly copy everything to the returned state_dict. In most cases, this is a user-defined state that is not associated with any particular parameter. Another possible case is this state is managed by TorchRec. Otherwise, there may be a mismatched assumption of optim_state_dict of this mode.rz$FSDP _optim_state_dict() profiling: )$rrr enter_contextrrALLr#traversal_utils_get_fsdp_handlesr{rrrrnrrrr6r5rrr8rLrrfrrrrrrrrkcloser)rrrrrrYrorrrZcmrXrr!rrJrr`rQ convert_fnrfsdp_osdflat_param_fqnsrdrs r:_optim_state_dictrLsD B^++N,?,?,C,CDE)/*K*KE*RSnH e 4 9H[G    0D*51 25901AB! sCx",,. /%8{K,5"4mEV  "=U!CD"    9 :   )     "&   (      2 3  ,0  $ ( " W %      $  '8H+2245O&w/557$ U .  / !  $ $   #C5)L L $s%$())#: 0-$  HHJ!!"HI OeDD"     s%/A)rr"rrpr,r._fqnsrr.r/_paramsr0r requires_grad) r<r=r>fqn_to_param_inforr.rprVr local_fqnrs r:r@z._get_fqn_to_fsdp_param_info..module_fnsCFK   :v& F3 && ' FBC( (8(89 NC#FY$67C''(-44??:MRsRM%4 c "14O ) )# .!!-33::&&s+99 r9c|Sr>r8)rs r:rBz._get_fqn_to_fsdp_param_info..return_fn rCr9rD)rr@rBrrrs r:rrsL.!35  |j}|tjk(r t dddd|j _yy)Nrz'Found state_dict_type LOCAL_STATE_DICT.z3DeviceMesh is not compatible with LOCAL_STATE_DICT.zKPlease set state_dict_type to SHARDED_STATE_DICT to get DTensor state_dict.T)rstate_dict_typer&LOCAL_STATE_DICTroptim_state_dict_configr)rrrs r:_set_optim_use_dtensorrsWz>40-== m<< <9E]  HL  7 7 D1r9)FNFN)Fr>)NFNN)T)FT)yrrloggingr contextlibr dataclassesrrtypingrrrr r r r r rrrrrrrKtorch.distributed distributedr{'torch.distributed.fsdp._traversal_utilsfsdp_traversal_utilsrtorch.nnr#torch.distributed._state_dict_utilsr"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr#torch.distributed.fsdp._debug_utilsr"torch.distributed.fsdp._flat_paramrr'torch.distributed.fsdp._fsdp_extensionsr r!%torch.distributed.fsdp._runtime_utilsr"r#torch.distributed.fsdp.apir$r%r&torch.distributed.tensorr'r(torch.utils._pytreer)'torch.distributed._shard.sharded_tensorr* getLoggerr1rr,r5rBrDrNrSr7rerkr\r] ProcessGrouprrrModuler OptimizerrrrLrQrrrrr,r6r8rrLr rr]rfrkrnrprxrPrrrrrrno_gradrrrr8r9r:r s   (" AABE?M 8-E   8 $ $$ $T#s(^sCx0I  CC C2   Z4"438n44 4  4  $sCx. 4n/3/4/C"C38nCCLJ"J "JJ $sCx. JZc3h D%% & #s(^ &#&/78I8I/J. "   c3h  #s(^ L"-1)- N)38nN) 99N)N) EKK)) * N)  N) D%% & N) #s(^N)bG"G3S#X./GS G #s(^ GTRR%,,'RS R"%**- R  R  \\ Rj3H3H5<<(3HS 3H \\ 3Hl$$c$S $ $d %M,c3hM, 99M, ;; M, c3h R\\ " $  M,M,M, #s(^M,n H. 99H. c3h R\\ " $ H. #r|| H.V"%((//"d=#;M6N"N"&$=A<@ 0 ;; 0 BII 00DtCy!89: 0  ]C%7 89 0  %S/2<< '( 0j"&$=A<@ N ;; N BII NNDtCy!89: N  ]C%7 89 N  ",,c3h '( N2  N 99 N c3h R\\ " $  N ",,  N'&n-'&"&~uS#X'F"G'&U38_bll:;'& D%% & '&  '&` >>38n>> D%% &>>U38_bll:;>> d3i/0 >> !m!34 >>  >> 4 neCHo&E!F FG>>BS#XU38_bll:; d3i/0 $sCx. 0 $sCx. T &     ""sCx." $sI~ "JP "P d3 >23P sCx.P T#s(^+, P  8EKK $sD%,,1G,H'H"I IJ P fE+"E+T#s(^+,E+E+ E+  E+  E+ E+Pe"ed3 >23esCx.e e  e  e #tCH~ eP/"/sCx.// /  /  #s(^ /rX~.X"&~uS#X'F"GX!m!34X5c?C/0 X  X  XX #s(^XD6~.6"&~uS#X'F"G6!m!3465c?C/0 6  6  66 #s(^6r"Y 99Y ;; Y38nY c3h R\\ " $  YYY D%% &YYYY  #s(^!YYx.ryy.T#}:L5M.bLL*L LLr9