sgddlZddlZddlZddlZddlZddlmZmZddlm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZddlZddlmZddlmZddl mcm!Z"ddlm#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0dd l1m2Z2d d l3m4Z4m5Z5m6Z6gd Z7ejpe9Z: d Z;dZdZ?dZ@GddeZAeAjeAjfZDeAjeAjfZGGddeZHGddeZIGddeZJGddeZKGdde0ZLGdd ejeL!ZNGd"d#ZOd$ejd%eQd&ejd'dfd(ZRd$ejd%eQd)e#d'dfd*ZSd$ejd%eQd+ee#ejffd,ZTd-eeejFejfd'eejfd.ZUd/eeje#fd'e#fd0ZVd1ejfd2ZXejd3d4ZZd5e[d6ejd7e\d8ejfd9Z^ejd d:ejd;eQfd<Z`ejd d:ejd;eQfd=Zaejd d:ejd;eQfd>Zbd?Zcd@ejFdAe[fdBZdd)e#fdCZey)DN)autoEnum) accumulatechain)AnyCallablecastDict GeneratorIteratorList NamedTuple no_type_checkOptionalSequenceSetTupleUnion)Tensor)_FSDPDeviceHandle!_named_parameters_with_duplicates_no_dispatch_record_stream_set_fsdp_flattenedHandleTrainingState)_alloc_storage_data_ptr_allocated _free_storage _p_assert)_ParameterMeta)FakeProcessGroup)_ext_post_unflatten_transform_ext_pre_flatten_transformFSDPExtensions) FlatParameterFlatParamHandleFlatParamShardMetadata ParamInfoSharedParamInfoHandleShardingStrategyFSDP_USE_UNSAFE_SETATTRFSDP_SKIP_WRITEBACK_CHECKFSDP_USE_FULL_PREC_IN_EVAL*FSDP_USE_FAKE_ALL_GATHERFSDP_USE_FAKE_REDUCEcReZdZeZeZeZeZeZy)r*N) __name__ __module__ __qualname__r FULL_SHARD SHARD_GRAD_OPNO_SHARD HYBRID_SHARD_HYBRID_SHARD_ZERO2U/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/fsdp/_flat_param.pyr*r*vs&JFMvH6L&r;r*cDeZdZUdZeed<ejed<eed<y)r(z&Information for an original parameter. param_namemodule module_nameNr2r3r4__doc__str__annotations__nnModuler:r;r<r(r(s0O IIr;r(cveZdZUdZeed<ejed<eed<eed<ejed<eed<y) r)ai Additional information for a shared parameter. For each shared parameter, we designate one module and its parameter variable to be the primary owner, determined as the first one encountered in the parameter walk. These are prefixed with "prim". The primary module and parameter do not have their own :class:`SharedParamInfo` instance. r>r?r@prim_param_name prim_moduleprim_module_nameNrAr:r;r<r)r)s4O IIr;r)c\eZdZUdZeed<eeed<eeed<eeed<eeed<y)_ShardParamInfoz4Shard-related information for an original parameter.in_shardoffset_in_shardnumel_in_shardintra_param_start_idxintra_param_end_idxN)r2r3r4rBboolrDrintr:r;r<rLrLs6>Nc]"SM!$C=(!#&r;rLceZdZUdZeedfed<eejdfed<ee dfed<eee e fdfed<y)r'a This holds metadata specific to this rank's shard of the flat parameter. Attributes: param_names (Tuple[str, ...]): Prefixed parameter names of this rank's shard of the parameters; see :class:`FlatParameter`. param_shapes (Tuple[torch.Size, ...]): Parameter shapes of this rank's shard of the parameters; see :class:`FlatParameter`. param_numels (Tuple[int, ...]): Parameter numels of this rank's shard of the parameters; see :class:`FlatParameter`. param_offsets (Tuple[Tuple[int, int], ...]): [start, end] offsets (in units of numels) giving this rank's part of each flattened original parameter. . param_names param_shapes param_numels param_offsetsN) r2r3r4rBrrCrDtorchSizerSr:r;r<r'r'sP sCx  C((S/!sCx#-..r;r'ceZdZdZy)_FlatParameterMetacTt|tjxr t|ddS)N_is_flat_paramF) isinstancerYrgetattr)selfinstances r<__instancecheck__z$_FlatParameterMeta.__instancecheck__s)(ELL1 g &7  r;N)r2r3r4rcr:r;r<r\r\s r;r\ceZdZUdZej ed<ej ed<ej ed<eed<ee dfed<eej dfed<ee dfed <ee e dfed <eedfed <eedfed <ee dfed <eedfed<eej"ed<eed<eed<eed<eed<ee e fed<e ed<eed<eed<eed<e eej(ed<e eej(ed<e ee eed<e eeed<eeed<d+dZed ee d!eed"eej d#ee d$eed%ee e d&e eej(d'e eej(d(eed)dfd*Zy),r%a This is the flat parameter used by :class:`FullyShardedDataParallel`. It is comprised of one or more original parameters, which are flattened and concatenated to construct the flat parameter. Under the current design, this parameter logically represents both the unsharded and sharded flat parameter, and its data changes storages dynamically. - In the :class:`FullyShardedDataParallel` constructor, the parameter is initialized as unsharded and then sharded in-place. - At runtime, the parameter is lazily (re)-initialized. The sharded parameter data is saved in ``self._local_shard``, and a new ``Tensor`` ``self._full_param_padded`` is created, which is the all-gather destination and owns the unsharded parameter storage thereafter. (See :meth:`FlatParamHandle.init_flat_param_attributes`.) - Throughout runtime, the parameter data changes storages as needed, e.g. to the sharded flat parameter, low precision sharded flat parameter, or the unsharded flat parameter. NOTE: Since ``use_orig_params=True`` supports intra-``FlatParameter`` padding, we have two versions of the per-parameter numels, one that includes the padding (``_numels_with_padding``) and one that does not (``_numels``). The former may have length longer than the other data structures, while the latter has the same length as the number of actual original parameters like the other per-parameter data structures. NOTE: This is not a real class; instead, you will always get a Parameter back out if you try to create one of these. This is similar to the trick we implemented for Parameter to get it to work with subclasses; this is primarily so that FlatParameter supports combination with FakeTensor. Attributes: _unpadded_unsharded_size (torch.Size): Unsharded flat parameter's size without right-hand-side padding for divisibility by the world size. For ``use_orig_params=True``, this includes alignment padding. _padded_unsharded_size (torch.Size): Unsharded flat parameter's size with right-hand-side padding for divisibility by the world size. For ``use_orig_params=True``, this includes alignment padding. This is only set for sharded strategies since they require padding for the all-gather. _sharded_size (torch.Size): Sharded flat parameter's size with padding. This is also set for ``NO_SHARD``, in which case it is the same as the unsharded sizes. (We omit "padded" because there is no analogous unpadded one.) _num_params (int): Number of original parameters flattened into this flat parameter. This is the length of the per-parameter data structures. _param_infos (Tuple[ParamInfo, ...]): Each parameter's parameter info entry; see :class:`ParamInfo` for details. _shapes (Tuple[torch.Size, ...]): Each parameter's original shape. _fqns (Tuple[str, ...]): Each parameter's fully-qualified name (FQN) prefixed from the ``_fully_sharded_module``. The names are guaranteed to be unique in the subtree rooted at that module. _param_extensions (Tuple[Optional[Any], ...]): Each parameter's extension (i.e. some per-parameter state) used to customize pre-flatten and post-unflatten behavior or ``None``. This is experimental, and users should not depend on its existence in the future. _numels_with_padding (Tuple[int, ...]): Each parameter's numel including entries for the padding. This is used to construct views into the flat parameter via ``torch.split()``. This may have length longer than ``_num_params``. _numels (Tuple[int, ...]): Each parameter's numel excluding entries for padding. This has length equal to ``_num_params``. _shard_param_infos (Tuple[_ShardParamInfo, ...]): Each parameter's shard parameter info; see :class:`_ShardParamInfo` for details. _shared_param_infos (Tuple[SharedParamInfo, ...]): Shared parameter info entries; see :class:`SharedParamInfo` for details. _modules (Set[nn.Module]): Modules that contain some original parameter that is flattened into the flat parameter. _shard_numel_padded (int): Numel padded for this rank's sharded flat parameter. _local_shard (Tensor): Sharded flat parameter with padding if using a sharded strategy. If using ``NO_SHARD``, then this is the unpadded unsharded flat parameter, and there is no notion of a sharded flat parameter or padded unsharded flat parameter. _full_param_padded (Tensor): Unsharded flat parameter with padding. This is not defined for ``NO_SHARD``. When using mixed precision for parameters, this has the low precision. _full_prec_full_param_padded (Tensor): Full precision unsharded flat parameter with padding. This is used for unsharding outside of computation when using mixed precision for parameters. This is never defined for ``NO_SHARD``. _post_backward_hook_handle (RemovableHandle): Flat parameter's post-backward hook handle. (Compile only) _post_backward_hook_state (Tuple[AccumulateGrad, RemovableHandle]): Flat parameter's :class:`AccumulateGrad` object and post-backward hook handle. (Eager only) _mp_shard (Tensor): Low precision sharded flat parameter with padding. This is only defined when parameter mixed precision is enabled. For ``NO_SHARD``, this is used for computation. _cpu_grad (Tensor): Sharded gradient with padding stored on CPU. This is only defined when offloading parameters is enabled. _saved_grad_shard (Tensor): Sharded gradient with padding from previous iterations for gradient accumulation without :meth:`no_sync`. _params (Optional[List[nn.Parameter]]): If ``use_orig_params=True``, then each original parameter variable; otherwise, ``None``. This does not include any padding tensors. _shared_params (Optional[List[nn.Parameter]]): The original shared parameter variables if ``use_orig_params=True`` and ``None`` otherwise. _tensors (Optional[List[Optional[Tensor]]]): This saves the ``Tensor`` views created in the forward and tracked by autograd when ``use_orig_params=True`` and is ``None`` otherwise. This is to preserve those ``Tensor`` variables for the backward to ensure that the ``FlatParameter`` 's ``AccumulateGrad`` object does not change in which case the post-backward hook does not run. This is relevant for cases like reentrant activation checkpointing. _is_grad_none_mask (Optional[List[bool]]): If ``use_orig_params=True``, a mask over the original parameters' gradients indicating if it is logically ``None`` or not; otherwise, ``None``. This does not include entries for padding. This mask is needed because only some of the parameters may have ``None`` gradient, in which case the flat gradient must be non-``None`` and must use zeros to approximate those original ``None`` gradients. This mask informs FSDP to set the original parameter gradients to ``None`` (instead of zeros) as needed. _unpadded_unsharded_size_padded_unsharded_size _sharded_size _num_params. _param_infos_shapes_fqns_param_extensions_numels_with_padding_numels_shard_param_infos_shared_param_infos_modules_shard_numel_padded _local_shard_full_param_padded_full_prec_full_param_padded_post_backward_hook_state_post_backward_hook_handle _mp_shard _cpu_grad_saved_grad_shard_params_shared_params_tensors_is_grad_none_mask_is_padding_maskNc|tusJdtjjtj||}d|_|S)Nz&subclasses FlatParameter not supportedT)r%rE Parameter__new__r^)clsdata requires_gradrs r<rzFlatParameter.__new__ls?m#M%MM# LL t] Cr; param_infosnumelsshapesfqnsshared_param_infosparam_extensionsparams shared_paramsis_padding_maskreturnc 0t|t|k(sJt|t|k(sJt|t|k(sJt||_||_||_||_||_| |_g} t|| D]\} } | r | j| t| |_ t||_ t|j|jk(sJt||_ |jDchc]}|jc}j|jDchc]}|jc}|_|du| duk(sJ|| t| t|k(sJg|_t|| D]#\}} | r |j j|%| |_t%|j |j"D] }t'|t)|jDcgc]}dc}|_t)|jDcgc]}dc}|_nd|_d|_d|_d|_|j/|_t'|d|_ycc}wcc}wcc}wcc}w)a Initialize attributes holding metadata about the original parameters comprising the flat parameter. We expose this method separate from the constructor to keep the constructor only responsible for the flat parameter's tensor data. This method should only be called once per model, while the constructor may be called multiple times, e.g. when reloading from a checkpoint, in which case only the tensor data needs to be passed to the constructor. Since :meth:`load_state_dict` is implemented via :meth:`copy_`, the metadata is correctly assumed to be unchanged. Args: See the Attributes in the class docstring. NF)lenrhrirjrkrlrzipappendtuplernrmrpr?unionrqr{r|rrranger~r}sizere_post_backward_called)rrarrrrrrrrrnumels_without_paddingnumel is_paddingpispiparam_s r<_init_metadatazFlatParameter._init_metadatauss8;3v;...;3t9,,,;3'7#8888{+'  !1 /,.!$V_!= 5 E:&--e4 534 $)&M!4<< D$4$4444#();#< -1->->?r?EE#'#;#; dZ deded eddfd!Z!ded edee"d"ffd#Z#e$d$ed%ed&edeeeffd'Z%e$d$ed%ed&edeeeffd(Z&e$d$ed%ed&ede jNfd)Z(deeeeffd*Z)e*de+fd+Z,e*e j>did,Z-de fd-Z.d.Z/d/Z0de fd0Z1d1Z2de jfd2Z3d3edefd4Z4d3e jddfd5Z5d6Z6d7Z7e j>d8Z8d9Z9d:Z:d;Z;ed=e fd>Z?d?Z@d@ZAdidAZBe* djd$ee jdeCefdBZDe* djd$eedeefdCZEe*e jdDe ddfdEZGe*didFZHedidHZKe*e j>didIZLe*e j>de fdJZMdKeedLedMedNe jNdOedPe ddfdQZNdRZOdSZPdTZQdeRejfdUZSd$ede fdVZTdeCeeUeUffdWZVdeCeeUeUffdXZWeXdeeUfdYZYeXdeefdZZZdid[Z[d\Z\d$efd]Z]d$efd^Z^e$d$efd_Z_e$d$efd`Z`daZad$efdbZbd$efdcZceXde fddZdeXde fdeZeeXde fdfZfeXde fdgZgeXde fdhZhxZiS)kr&a A handle that manages a flat parameter (:class:`FlatParameter`). This includes sharding and view management. Args: params (Sequence[nn.Parameter]): The parameters to flatten into the flat parameter. fully_sharded_module (nn.Module): See [Note: Fully Sharded Module]. device (torch.device): The compute and communication device, which should be a non-CPU device. We refer to it as the compute device. sharding_strategy (ShardingStrategy): Sharding strategy to apply to this handle's ``FlatParameter``. offload_params (bool): Whether to offload the handle's ``FlatParameter`` to CPU. mp_param_dtype (Optional[torch.dtype]): Parameter mixed precision setting passed to the FSDP constructor. mp_reduce_dtype (Optional[torch.dtype]): Gradient reduction mixed precision setting passed to the FSDP constructor. keep_low_precision_grads (bool): Whether to keep gradients in low precision. use_orig_params (bool): If ``True``, then FSDP preserves the original parameter variables and returns them from ``named_parameters()`` (e.g. to support different optimizer hyperparameters within one :class:`FlatParameter`). If ``False``, then FSDP reconstructs the parameters every iteration and returns the :class:`FlatParameter` s from ``named_parameters()``. N)fsdp_extensionrfully_sharded_moduledevicesharding_strategyoffload_paramsmp_param_dtypemp_reduce_dtypekeep_low_precision_grads process_groupuse_orig_paramsrc tt |t|}t|dk(r#t d|j j d|jtjjtddk(|_ tjjtddk(|_tjjtddk(|_tjjt"ddk(|_|jrt't(dtd|j rt+t(dtd|j$rt-t(dt"d | } |j/| ||_t3j4|j0|_| |_|j s |j$r.t;| j=| j? |_ | j=|_| j?|_!||_"||_#| |_$||_%tLjN|_(tSjT|_+||_,d|_-d|_.d|_/d|_0d |_1d |_2d |_3|djh|_5|jm|||jnJ| rtq|jn nd|_9| |_:|jw|||jr| |jyd y)NrzCannot construct a z with an empty parameter list1zSince z=1, FSDP will not check for parameter or gradient writeback. Changing parameter or gradient storages may lead to silent correctness errors.z=1, FSDP will not execute all-gather ops. Your training will be incorrect, but can reveal how much time spent on all-gather ops.z=1, FSDP will not execute reduce-scatter ops. Your training will be incorrect, but can reveal how much time spent on reduce-scatter ops.)rank world_sizeF)unsharded_dtype as_params)=super__init__listr ValueError __class__r2_init_setattr_fnsosenvironget_FSDP_SKIP_WRITEBACK_CHECK_skip_writeback_check_FSDP_USE_FULL_PREC_IN_EVAL_use_full_prec_in_eval_FSDP_USE_FAKE_ALL_GATHER_use_fake_all_gather_FSDP_USE_FAKE_REDUCE_use_fake_reduce_warn_skip_writeback_checklogger_warn_use_fake_all_gather_warn_use_fake_reduce_init_get_unflat_views_fnrr from_device_device_handlerr rr_fake_process_groupr_sharding_strategy_offload_params_use_orig_params_keep_low_precision_gradsrIDLE_training_statedistget_debug_level _debug_level_fully_sharded_module'_unsharded_flat_param_for_skipped_views _handle_index_pre_forward_order_index_post_forward_index_needs_pre_forward_unshard_needs_pre_backward_unshard _prefetcheddtype_orig_param_dtype_init_param_reduce_dtypes_fwd_bwd_param_dtype_get_aligned_numel_aligned_numel_fsdp_extension_init_flat_param_and_metadata_use_unsharded_views)rarrrrrrrrrrralign_addressesrs r<rzFlatParamHandle.__init__s f v;! %dnn&=&=%>>[\   JJNN5r :c A " JJNN6 ;s B #%'JJNN3Lb$QUX$X! " /Db IS P  % % &345KK   $ $ %234DD   !./0HH * &&7 /;;DKKH*  $ $(=(='7"'')m6H6H6J(D $"&&( ',,."3- /)A&277 002%9" JN4-17;%26 */'+0( "( &&~G((444 t/H/H I  . ** ($*=*=  !!E!2r;ctjjtddk(}|||rt|_t |_yt|_t|_y)Nrr) rrr_FSDP_USE_UNSAFE_SETATTR_unsafe_setattr_tensor_setattr_tensor_unsafe_setattr_param_setattr_param_safe_setattr_tensor_or_param)rause_unsafe_setattrs r<rz!FlatParamHandle._init_setattr_fnsMsFZZ^^,DbISP  #9D "7D #@D "?D r;rcN|r|j|_y|j|_yN)_get_unflat_views_aligned_get_unflat_views_unaligned_get_unflat_views)rars r<rz)FlatParamHandle._init_get_unflat_views_fnXs-  * * 11 r;r? aligned_numelrc (t|dk(r td|dkrtd||j|\}}}t|}g} g} g} g} g} i}g}g}g}g}dx}}|j dD]\}}t |dD]\}}||vr ||vr:||\}}}|j || j t||||||J|dkDrX|||zz }|dkDrK||krFt||d|}|j ||j d| j |||z }t||j\}}ttj|}|j ||||f||<|j ||j d| j t|||| j |j| j |j |r|dz|zn|} | j | ||jz }||jz }t|dk(rtd |d ||j"dk(r%|dkDr ||k7rt$j'd ||z |||dkDr|j(||j(zz }|dkDrz||j(krk|j"dk(rt$j+d |t||d|}|j ||j d| j |||z }|j-|d| |_t0j3|j.| | | | | ||r t5|nd|rt5|| yd| y)a Initialize the ``FlatParameter`` and its metadata. NOTE: This should only be called once at construction time, after which the ``FlatParameter`` metadata is assumed to be static. NOTE: The elements of ``params`` should only be ``Tensor`` s when composing with ``DTensor`` -based tensor parallelism, in which case the elements may be ``DTensor`` local shards. rzExpects non-empty `params`-Expects non-negative `aligned_numel` but got F)remove_duplicate)recurseT.z2`params` were not found in `module`'s treeparams: z module: zLFSDP FlatParameter address alignment created %s numel of padding (%s vs. %s)zFFSDP FlatParameter world size divisibility created %s numel of padding)rrN)rr_validate_tensors_to_flattenset named_modulesrrr)_construct_padding_tensorr#rr rErr(rshaperrdebugrinfoflatten_tensors_into_flat_param flat_paramr%r_convert_to_params)!rarr?rrrflat_param_requires_gradr params_setrrrrrshared_param_memoparams_to_flattenrrr total_numeltotal_numel_without_paddingsubmodule_name submoduler>rrIrJrH numel_to_padpadding_tensor transform_t extensionfqns! r<rz-FlatParamHandle._init_flat_param_and_metadata_s6" v;! 9: : 1 ? O   - -f 5   $ [ (* #%46  @B;= &(&(455 1)/)=)=u)=)U3 A %NI%F5&2 A! E *--EVFBK!1?"((/&--'&%*+',  %q('4 m8S'T '!+ }0L-F ,eUF.N.44^D+2248"MM,7'<7K-G,,.*K!{;E$++I609>:/V%e,%,,U3#**51&&yY'WXMM%++-0MM%++.*',z9' KK$5;;=0K/5;;=@/e2 A3 Ah  !Q &!(*VH6  IIN!:: LL299+   1  ??kDOO.KLLaL4??$B99>KK.$ "; %""((8&&t, l+|+ )-)M)M 2*N*  $$ OO      5D 0 1$1@ } -  GK  r;tensorscd}d}d}|D]}t|tr td||js td|*|j|k7rtd|d|j|j s||j |k7r td|*|j|k7rtd|d|j|j}|xs |j }|j}|Jd|||fS) zCValidate the tensors to flatten and returns any necessary metadata.Nz Cannot flatten a `FlatParameter`z$Cannot flatten integer dtype tensorsz0Must flatten tensors with uniform dtype but got z and zNMust flatten tensors with uniform `requires_grad` when `use_orig_params=False`z5Must flatten tensors on the same device but got both z!Requires non-empty `tensors` list)r_r%ris_floating_pointrrrr)rarrrrtensors r<rz,FlatParamHandle._validate_tensors_to_flattens>(,37 )- #F&-0 !CDD}V%=%=%? !GHH V\\U%: FugN!<<.* )),8((,DD .!fmmv&= KheFMM?4LLE'?'W6CWCW $]]F5 #6(3X5XX3.66r;ct|dk(r td|dkrtd||j|\}}}g}|dkDrd}|D]x}|||zz } | dkDr)| |kr$t| |d|} |j | || z }|j t j t|||jz }z|j||jzz } | dkDr_| |jkrPt| |d|} |j | || z }n+|Dcgc] }t j t|"}}t j|dScc}w)a  Flatten ``tensors`` into a single flat tensor. The flattening optionally includes padding if ``aligned_numel`` is greater than 0, where ``aligned_numel`` gives the numel required to have address alignment. NOTE: The padding alignment algorithm must be kept in sync with :meth:`_init_flat_param_metadata`. We separate the two methods because the initialization happens once, whereas this method may be called multiple times throughout training (e.g. for checkpointing). rzExpects non-empty `tensors`rFdim) rrrrrrYflatten_detach_if_neededrrcat) rarrrrr flat_tensorsr rrrs r<flatten_tensorszFlatParamHandle.flatten_tensorss" w<1 :; ; 1 ? O  <$eUF&N!''7</K##EMM2CF2K$LMv||~-  . ??kDOO.KLLaL4??$B!: %"##N3|+ HO=C /78Lyy1--s!%Erc@|j||}t||S)Nr)r r%)rarrrflat_param_datas r<rz/FlatParamHandle.flatten_tensors_into_flat_paramNs# ..w F_MJJr;c|du|_|du|_|jr%|js||_|j|_n*|xs |j|_|xs |j|_|jJ|jJy)a] Initialize param and reduce dtypes. Precondition: ``self.flat_param`` is set. This ensures that this handle's parameters have a single dtype. Postcondition: This sets ``self._fwd_bwd_param_dtype`` and ``self._reduce_dtype``. If ``mp_param_dtype`` or ``mp_reduce_dtype`` is ``None``, then we assume the original parameter dtype. One special case is if ``mp_param_dtype`` is not ``None`` and ``mp_reduce_dtype`` is ``None``, in which case we assume the gradient reduction dtype matches the forward/backward parameter dtype. N)_low_prec_param_dtype_specified _low_prec_reduce_dtype_specifiedr _reduce_dtyper)rarrs r<rz)FlatParamHandle._init_param_reduce_dtypesWs(0>T/I,0?t0K-  0 099)7D %!%!:!:D (6(P$:P:PD %!0!JD4J4JD ((444!!---r;c|j}|js&|jdd|jdz nt |j dk(dt j||j|j\}}tjjjsB|jjdkD}|r|jj!d|j#||j|jz}|j|jdzzdz }|j||||j$r|j'yy)aM Shard the handle's ``FlatParameter``. This allocates new memory for the sharded flat parameter and frees the unsharded flat parameter's storage. Postcondition: ``self.flat_param`` is the sharded flat parameter. Shard metadata attributes are set for all sharding strategies. rr!z;The `FlatParameter` is not the sole occupant of its storageN)ruses_sharded_strategy_init_shard_metadatarrstorage_offsetr& _get_shardrrrY distributed_functional_collectivesis_torchdynamo_compiling_typed_storage_size_resize_set_r_use_sharded_views)rarsharded_flat_param numel_padded allocated start_idxend_idxs r<shardzFlatParamHandle.shard}s9__ ))  % %aJ,<,<,>,B C ))+q0M 0?/I/IDIIt0 ,  $$<<UUW&557==?!C --/88; OO. /*002TYY>I(..0DIIMBQFG  % %lIw G   # # % !r;r6unsharded_start_idxunsharded_end_idxcx|j}|j|_|j}t |dk\xr||kd|d|t ||kd|d||j ||}t ||jk(s Jd|jdt |||_||_ y) a Initialize shard-related metadata for this rank's shard of the flat parameter. This includes ``_sharded_size``, ``_shard_param_infos``, and ``_shard_numel_padded``. Args: numel_padded (int): Numel padded for this rank's sharded flat parameter. unsharded_start_idx (int): Start index in the unsharded flat parameter assigned to this rank. unsharded_end_idx (int): End index (inclusive) in the unsharded flat parameter assigned to this rank. Precondition: ``self.flat_param`` 's data is the sharded flat parameter. rzunsharded_start_idx: z unsharded_end_idx: znumel_padded: z sharded_flat_param_numel: zExpects length but got N) rrrgrr_get_shard_metadatarrhrorr)rar6r;r<rsharded_flat_param_numelshard_param_infoss r<r*z$FlatParamHandle._init_shard_metadatas,__ #-??#4  #-#3#3#5  1 $ Q)<@Q)Q#$7#88LM^L_ `   4 4\N+))A(B D !44 !2  ! "j&<&< < W Z334IcBS>T=U V W <(9 %)5 &r;.c~|j}t|t|jjk(s3Jdt|jjdt|g}||z dz}t t ||jj D]\}\\}}} | r||kxr||k\} | stddddd} nR||krd} ||z } n||z } d} | dk\r| |ksJd| d|d t|||z }|| z dz}td | || |} |j| t|S) z Compute the shard metadata based on ``unsharded_start_idx`` and ``unsharded_end_idx`` (inclusive). ``unsharded_start_idx`` and ``unsharded_end_idx`` give the interval of the unsharded flat parameter specifying the shard. z Expected r>r!FNrzInvalid `offset_in_shard` of z! for sharded flat parameter with z numelT) _get_flat_param_offsetsrrrm enumeraterrrLminrr)rar;r<flat_param_offsetsrAr@iunsharded_param_start_idxunsharded_param_end_idxrin_sharded_flat_paramshard_param_inforPrNrQrOs r<r?z#FlatParamHandle._get_shard_metadatas"99;%&# OO 0 0+   e s4????@A3OaKbJc d e 46#47J#JQ#N s-t/O/OP Q) 7 A @ &(? #'>>C%)BB ")#25$dD#Q &*CC-.)&?BU&UO,.GG*'(O#q(_?W-W4O3DE33K2LFTW /1BC/0$"57L!Lq!P#2#")' $   $ $%5 6S) 7T&''r;rrrctj|j|}t||dzkr|dj d}n||}|dj |j z }|dk\sJd||fS)a Return the unpadded shard of ``tensor`` for the given ``rank`` and ``world_size``. The returned value is a tuple of the shard of ``tensor`` without any padding and the numel to pad for that shard. If ``tensor`` is already flattened or may be viewed in the flattened shape (which is true in the expected usage), then this method does not allocate any new tensor memory. r!rz5Chunk's size should be at most the first chunk's size)rYrchunkr new_emptyr)rrrchunksrMrs r<_get_unpadded_shardz#FlatParamHandle._get_unpadded_shard s v&,,Z8 v;$( #1I''*E4LEay(5;;=8 A  C B C l""r;ctj|||\}}|j}|dkDrtj|d|g}||fS)a Return the shard of ``tensor`` with padding for the given ``rank`` and ``world_size`` and the numel padded for that shard. This method allocates new memory (via :meth:`clone`) since the unsharded ``tensor`` may be deallocated after this method returns. r)r&rPcloneFpad)rrrrMrr:s r<r,zFlatParamHandle._get_shard&sU.AA D* |  ! EE%!\!23El""r;ct|jdk(sJ|jtj|||\}}|j }t|dk(sJ|t j |d|zgS)z Return the shape of ``tensor`` after sharding including padding. This requires ``tensor`` to have 1D shape and ensures that the returned shape is 1D. r!r)rrr&rPrrYrZ)rrrunpadded_sharded_tensorrunpadded_sharded_sizes r<_get_sharded_sizez!FlatParamHandle._get_sharded_size:s6<< A%8&,,8%0?0S0S D*1 -!8 < < >()Q.J3H2IJ.zz03lBCDDr;ctt|jj}dg|ddz}|Dcgc]}|dz  }}tt ||}|Scc}w)z Return [start, end] offsets of each original parameter's flattened data in the unsharded flat parameter (without padding). NOTE: The returned list includes elements for alignment padding. rNr!)rrrrmr)racumulative_sumstartsendendsrXs r<rCz'FlatParamHandle._get_flat_param_offsetsJsc j)M)MNO~cr**#12Ca22S./ 3s Acg}g}g}g}t|jj|jj|jj|jj D]n\}}}}|j s|j||j||j||j|j|jfptt|t|t|t|S)z Return the shard-related metadata specific to this rank's shard of the flat parameter. NOTE: The returned tuple does not include elements for alignment padding but does account for the padding. ) rrrkrjrnrorMrrPrQr'r) ra fqns_list shapes_list numels_listshard_param_offsetsrrrrKs r<shard_metadatazFlatParamHandle.shard_metadataVs    36 OO ! ! OO # # OO # # OO . . 4   /C/ $,,   S !   u %   u %  & &$::$88  "& )  +  +  % &   r;c|j}|j|jk7rW|js|j|_|j s|js|j|_|j|_tjd}|jr't|j|k(d|jn|j|j|j|_ |jrk|jj|j|_ tj|j|j|j|_|j"rPtj$|j|j|j|_t)|j&|j*r|j"r |jn |j}|j-|j.z}tj0||j||_|j2j5|_t)|j2|j"rGtj0||j|j|_t)|j8yyy)a This initializes some attributes on the handle's ``FlatParameter``. This should be called during lazy initialization since it requires the parameter to be on the compute device if not offloading to CPU and we want to give users the chance to move the parameter appropriately after the FSDP constructor. For each tensor attribute on the ``FlatParameter``, see the unshard and reshard methods in this class for the allocation and free pattern. cpuzWExpects the `FlatParameter` to be on CPU when parameter CPU offloading is enabled, not rrrN)rrrr%rr&r'rYrrr_check_on_compute_devicerrs pin_memory zeros_likery_uses_param_mixed_precision empty_likerxrr)rremptyrtrrfru)rar cpu_deviceunsharded_param_dtypepadded_unsharded_numels r<init_flat_param_attributesz*FlatParamHandle.init_flat_param_attributes|sO__   t55 5 77,6,<,<)99<<%/%5%5"%/%5%5D "\\%(    !!Z/..8.?.?-@B   ) )$// :",//    &0&=&=&H&H{{'I'J # $)#3#3'' $j j,    + + $)#3#3''{{//$J *.. /  % % 33))%% " &0%5%5%7$//%I ",1KK&{{+-J ) 1;0M0M0R0R0TJ - *77 8//;@++*;;$**; 7 jEEF0# &r;cF|jtjk(r|jr|j d}|j r|j s|j}|jr|js|jsny|jr|js|jd}nN|jrB|jj|jk7r|j!|jdd}|j#|j|S)a" Return ``False`` if this is a no-op and ``True`` otherwise. Postcondition: ``self.flat_param`` 's data is on the device for communication and is what should be all-gathered. This means that it matches the dtype of the expected unsharded parameter. FT non_blocking)rrSUMMON_FULL_PARAMS_skipped_use_sharded_viewsr4rr_writeback_orig_paramsr)r needs_unshardrl_force_full_precision_use_low_precision_shardrr flat_param_tori)rarets r< pre_unshardzFlatParamHandle.pre_unshards  $7$J$J J//  # # %  )C)C--/C  & &((&&(   - -d6P6P  ) ) +C  ! !doo&<&< &K   t{{  >C %%doo6 r;c:|j|j}t|j|jj |jj |jj|jd|j|_ y)z\Allocate on the compute device and switch to using the low precision sharded flat parameter.TrtN) _check_low_precision_shardrrrxrsrcopy_torr)rars r<r{z(FlatParamHandle._use_low_precision_shards '')__   *"9"9">">"@  ""  # # & & $ '  %.. r;c|js:|jr|jn |j}|j |y|j }|j |}|j |y)a Run the unshard logic. This includes all-gathering the flat parameter and switching to using the unsharded flat parameter. If the handle does not need unsharding, then this only switches to using the unsharded flat parameter. For ``NO_SHARD``, this is a no-op. If FSDP is in :meth:`summon_full_params` and the handle uses parameter mixed precision, then the parameter is forced to full precision. N)ryr) _get_padded_unsharded_flat_paramr_use_unsharded_flat_param"_alloc_padded_unsharded_flat_param_all_gather_flat_param)raunsharded_flat_parampadded_unsharded_flat_params r<unshardzFlatParamHandle.unshard sy!!# --557__ !  * *+? @ #FFH&*&A&ABV&W# &&'BCr;cv|jsy|j}t||j}| S)z %$$r;c|j|j}|j}|j|t ||j |S)a( Allocate the *padded* unsharded flat parameter. The unpadded unsharded flat parameter is always a view into the padded one. This padded parameter is saved to a different attribute on the ``FlatParameter`` depending on if we force full precision. )_check_sharded_strategyrr_check_storage_freedrrfrarrs r<rz2FlatParamHandle._alloc_padded_unsharded_flat_param-sN $$&__ #DDF !!"67+Z-N-NO##r;c|j|j}|jr|jr~|j}t |j |jk7d|j|jjjdkDrt|j|S|j}|S)z Return a reference to the padded unsharded flat parameter depending on the calling context. This should only be called if using a sharded strategy. zExpects full precision but got r) rrrzrlrurrrrtuntyped_storagerrrs r<rz0FlatParamHandle._get_padded_unsharded_flat_param=s $$&__  % %$*J*J$.#J#J $**d.G.GG1$2K2K1LM ,,<<>CCEIj;;<$#$.#@#@ ##r;rctt|dxr t|dd|jj}|j |j z}t|j |k(d|d|j |j r |jn |j}|jrKttj|tj|}tj|||ntj ||||j"r$t%||j&j)|S)z All-gather the handle's flat parameter to the destination ``padded_unsharded_flat_param``. Then switch to use the all-gathered tensor. rrzEExpects a process group and world size to have been set via `shard()`zExpects z numel but got group)rhasattrrrrrrrris_cpurrYrMrget_world_size all_gatherall_gather_into_tensorrrrcurrent_stream)rarr5expected_numelpg tensor_lists r<rz&FlatParamHandle._all_gather_flat_paramZs2  D/ * Jwt\/J S "__11+113dooE ' - - /> A~&o6Q6W6W6Y5Z [ ((  $ $##   $ $ 79L9LR9PQK OOK);2 F  ' '+"     '"##224 +*r;c~|jj}|d|j}||j_|jt j k(}|jt jk(}|jr(|jr|ry|j| xr| y|r|jdyy)z Switch to use the *unpadded* unsharded flat parameter. This is a view into the *padded* unsharded flat parameter. NrF) rrerrrrFORWARD BACKWARD_PRErrwr)rarunsharded_sizeflat_param_part in_forwardin_pre_backwards r<rz)FlatParamHandle._use_unsharded_flat_paramsAA56N8L8L8NO.))-@-H-HH ..2E2R2RR  ..?  % %)>A/.A &   % % % 6r;c|jr|jr|j|j|jy)zo Run the post-unshard logic. This includes freeing the low precision shard if needed. N)rlr)!_free_low_precision_sharded_paramrirras r< post_unshardzFlatParamHandle.post_unshards3  + +0J0J  2 2 4 %%doo6r;c|jt|jj|jj t |jjy)z/Frees the low precision sharded flat parameter.N)rrrrxrrrrs r<rz1FlatParamHandle._free_low_precision_sharded_paramsJ '') # OO % %t':':'I'I'K  doo//0r;cZ|js|jy|j}|j|t j dtj |j}|jdu|d<tj||j|d|jk(rd|_ |jy|j}|jtjj k(r#t#j$d|j&dd|_ t j |j(|j}n8|j+|j|j|_ |j}t j,|j.|j|j0 }tj2|||j|jj4}|d|j7j9||_|jy) a Unshard the handle's ``FlatParameter``'s gradient. If all ranks have ``None`` gradient, then all original parameters will as well. This method performs an all-reduce and an all-gather. The additional all-reduce is tolerable since this method is not meant to be used on the computation critical path. Postcondition: ``_saved_grad_shard`` is defined and contains the value to set ``flat_param.grad`` after gradients are resharded. Nr!)rrrr[Rank z] Only some but not all ranks have a `None` `FlatParameter` gradient, so FSDP is using zeros to approximate those ranks' sharded gradients being `None`rgrh)r)_use_unsharded_grad_viewsr_check_unshardedrYzerosint32rgradr all_reducerrrzr DebugLevelINFOwarningswarnrrg_check_shardedrnrfrrrerview)rar num_grad_none sharded_gradpadded_unsharded_gradrs r< unshard_gradzFlatParamHandle.unshard_grads))  * * , __  j) AU[[M %??d2 a  T-?-?@  t .+/J (  * * ,  ?? "  DOO$8$88 TYYK(NN ,0J ( ;;z'?'? TL     0+5??J (%77L %  - -;;$$!  ## !<1C1C AA/0H.2F2F2HINN   &&(r;c|jr|j|jsy|jj|j_t |jdy)Nrz)r_use_sharded_grad_viewsr)rrzrdelattrrs r< reshard_gradzFlatParamHandle.reshard_gradsH   ( ( *)) #@@!45r;c|t|jtjtjfvd|j }|j |j j|jk7s$|j j|jk7r|j|j |j j|jk7}t| xs |jd|jd|j j|j j|jjk(}|r|s(|j j|_|j}n"tt|dd|j }|jj"}|j$rw|j"|k7rh|j'||_ nQ|j(}t|j j|k(d|d|j jd|_yyy) z Prepare the gradient for the backward computation. This is done by saving and clearing any existing sharded gradient in ``.grad`` to enable computing a new unsharded gradient. z:Expects to be in `BACKWARD_PRE` or `IDLE` (if prefetching)Nz&Expects the sharded gradient to be on r>ryz7`_cpu_grad` should be defined if the gradient is on CPUzFExpects `.grad` to be the unsharded gradient in `no_sync()` with size z but got size )rrrrrrrrrerrirrsrrzrryrrrrf)rargrad_offloadedprev_iter_synced_gradientsrlocal_shard_dtypepadded_unsharded_sizes r<prepare_gradient_for_backwardz-FlatParamHandle.prepare_gradient_for_backwards   #002E2J2JK L H __ ?? & OO "j&I&I I%%):)::  ) )$// :'__33t{{BN "":d&:&:8 F%??1124  $$&**//12 '* &3=??3G3GJ0#-#?#?L K8Q$.#7#7L%/$;$;$A$A!22$**.??(48I(JL%(2(I(I%OO((*.CC--B,CD$$.OO$8$8$:#;= #JOc; 'r;cDfd}j}t|dr<j|j||j|_||nt|dr{j|j ||jj |j|jrL|j|_|j /||n&tj xs |j dt|dr t|dyy)ziPrepare the gradient for optimizer computation by moving the sharded gradient to the ``.grad`` attribute.cTjsjrt|jdud|jjj k7rR|jj j |j_jrjyyyyy)NzUnexpected None grad!) rzrrrrrrrrr)rras r<"cast_grad_to_param_dtype_if_neededzVFlatParamHandle.prepare_gradient_for_optim..cast_grad_to_param_dtype_if_neededJs--$2P2P*//57NO??((D,E,EE+5??+=+=d>W>W+XJOO(,,446-F3Q-r;ryrzNzcAll sharded parameters that received a gradient in the post-backward should use `_saved_grad_shard`) rrr _check_on_cpuryrrirzrrr)r)rarrs` r<prepare_gradient_for_optimz*FlatParamHandle.prepare_gradient_for_optimGs 7__  :{ +    +   z *(22JO .z : Z!4 5    +  ) )* 5++7--j.J.JK//",">"> ??.6zB ...8!777?  :2 3 J 3 4 4r;c #$K|jt|jj|jjk(d|jjd|jj|j |jtt |j|jd|jtjd|j dt|jj|jjk(d|jjd|jj|j}|d|jjj|j|j|y#t|jj|jjk(d|jjd|jj|j}|d|jjj|j|j|wxYww)aS Move the unpadded unsharded flat parameter to CPU while in the context and moves it back to the previous device upon exit. For now, this assumes the ``FlatParameter`` is the unpadded unsharded flat parameter since (1) there is no reason to include the padding in the copy and (2) there is no use case for the sharded flat parameter. Precondition: ``self.flat_param`` 's data is the unpadded unsharded flat parameter on the compute device, and the handle uses a sharded strategy. Postcondition: Same as the precondition. z Expects size r>zEExpects the unpadded parameter to be a view into the padded parameterrfN)rrrrreri _same_storagerr|rYr_free_unsharded_flat_paramrrrr)rars r<to_cpuzFlatParamHandle.to_cpuss  $$& OO "doo&N&N NDOODDEYtOcOcOeNf g  %%doo6  $//4+P+P+R S S  5<<./ '') H  $$&$//*R*RR H HISWSbSbSgSgSiRjk +/*Q*Q*S ' '(A$//*?*?*A B H H   * *+F G $$&$//*R*RR H HISWSbSbSgSgSiRjk +/*Q*Q*S ' '(A$//*?*?*A B H H   * *+F Gs C;J>GCJCJ  Jfree_unsharded_flat_paramcJ|j|r|jyy)a~ Run the reshard logic. This includes freeing the unsharded flat parameter if ``free_unsharded_flat_param`` and switching to using the sharded flat parameter. Note that this also implicitly offloads the sharded flat parameter (if CPU offload is enabled) by pointing it to the ``_local_shard`` attribute which resides on CPU. N)_use_sharded_flat_paramr)rars r<reshardzFlatParamHandle.reshards$ $$& $  + + - %r;cr|jr+|js|js|jyyyy)aC Run the post-reshard logic. This includes freeing any memory that can now be freed given that the ``FlatParameter`` points to the full precision sharded flat parameter. Precondition: ``self.flat_param`` 's data points to the full precision sharded flat parameter. N)rlr)rzrrs r< post_reshardzFlatParamHandle.post_reshards:  , ,....  2 2 4// -r;c|j|j}|j|t||jj t |y)a1 Free the padded unsharded flat parameter. We allow this function to be called even when storage is not allocated The tensor to free depends on the calling context since the unshard may have forced full precision, in which case a different tensor is used. N)rrrirrrr)rars r<rz*FlatParamHandle._free_unsharded_flat_paramsU $$&#DDF %%&:;" $"5"5"D"D"F  *+r;c|j}|jrW|jtjk(}t j xr|xr|jtv}|r |j}|jr;|jj}t|t jdk(d||j|_ |jrr|_n|jrr|j se|j"duxr1|j$xr#|j"j&|j(k(}|r|j+y|j-yyyy)z-Switches to using the sharded flat parameter.rfz-Expects the local shard to be on CPU but got N)rrrrrrYis_grad_enabledr*NO_RESHARD_AFTER_FORWARD_HANDLE_STRATEGIESrrrsrrrr4rwrr)rrerr)rarrskip_use_sharded_viewsrraccumulated_grad_in_no_syncs r<rz'FlatParamHandle._use_sharded_flat_paramsQ__  --1D1L1LLJ%%'>>++=> #&'1$   ,,33F %,,u--?xH %11   %?S<'') 77 OO4/U22U"--1T1TT, /2240028  !r;cj}||}fdttj||jd|j |j D}|S)a- Return unflattened ``Tensor`` views into ``tensor``. If `tensor`` is ``None``, ``flat_param`` is used. The unflattening is based on ``flat_param`` 's metadata. Examples for ``tensor`` include ``flat_param.grad`` or unsharded tensor optimizer state. c3rK|].\}}}t|j||j0ywr)r"rr).0 subtensorrparam_extensionras r< z>FlatParamHandle._get_unflat_views_unaligned..s>  4E? *u%$$   s47rr)rrrYsplitrnrjrl)rarrviewss` r<rz+FlatParamHandle._get_unflat_views_unaligned s]__ >F  8; FJ$6$6A>"",,8   r;c \|j}||}tj||jd}d}g}t ||j D]\\}}|r |j t|j|j||j||j|dz }^|S)a= Return unflattened ``Tensor`` views into ``tensor`` with handling for padding. This method has the same contract as :meth:`_get_unflat_views_unaligned` except it checks for ``None`` placeholders representing padding for alignment, which may incur slightly more CPU overhead. rrr!) rrYrrmrrrr"rrjrlr)rarrsplitsidxrrrs r<rz)FlatParamHandle._get_unflat_views_aligned+s__ >F${{ J33  !$VZ-H-H!I  E: LL-JJz11#67005((  1HC  r;rc  |j}|j||j}ddlm}t t ||jD]\}\}\}}} |jrv|rtt||ur3|j||tj||j\|jj|} |j||| || _|r3|j||tj||j|} |jrv|j t"j$k(r||jj&|<n?|j t"j(k(r"|jj&|} || _| } |j+||| |jsj|j t"j$k(s| |j,|<t |jj.D]\}\}}} } }} t1|| }t3| xst5|tjd|dt||jr6|r4|jj6|}|j|||||_|r|j||||j+||||js|j t"j$k(s||j,|<y)a Unflatten the unsharded flat parameter by setting the original parameter variables to be views into it. Args: as_params (bool): If ``True``, then registers the original parameters as ``nn.Parameter`` s; if ``False``, then registers the original parameters only as ``Tensor`` s. ``False`` should be used during forward/backward computation and when hiding the original parameters from :meth:`nn.Module.named_parameters`. Note: when prefetching for next forward, current forward may be annotated with `@torch.no_grad()` `@torch.enable_grad()` ensures non-empty `view.grad_fn` otherwise `_post_backward_hook` will not get called rDTensorr"z as_params=z type(prim_param)=N)rrrtorch.distributed.tensorrrDrrirtyperrErrr{rrrrr}rr _parametersrpr`rr_r|)rarrrrrGrr>r?rr param_varrrHrI prim_param shared_params r<rz$FlatParamHandle._use_unsharded_viewsLs&__  j)&&(42; z.. /3 ) ? .A.-z61$$:(''" T9Q9QR //2##FJ>! ##LLZ5M5MN %) ((++/B/J/JJ6:003--1D1Q1QQ"&!9!9!!<&* $* $$VZC)),,0C0K0KK5>F&&z2S) ?bt:: ; @ A     6=_7J  EJ !EYK'9$z:J9KL $$#==a@ ##FJ E$. !##FJ C$$VZD)),,0C0K0KK5?F&&z27 @r;c|jjAt|jj|jjD] }d|_ y|j |jj|j |jj}tt||jjD]\}\}\}}}tt|||jj|dt||}|j|jk7s2|j|jk7s|j |j k7r8|jt#j$||_||j_||_t|jj(D]\}\}}}} } }tt|||r|dz|zn|dt||}t| | } |j| jjk7sF|j| jjk7s#|j | jj k7rB|jt#j$||_| j|j_| j|_y)z Unflatten the unsharded flat parameter's gradient. The original parameter variables' gradients are set to be views into the unsharded flat parameter's gradient. Nz is missingr)rrrr{r|rrrDrrirrrkr`rrrrYrmrrp) rarrrGrr>r?rr@rHrIrs r<rz)FlatParamHandle._use_unsharded_grad_viewssL ??   't668V8VW "!  "  doo223&&t';';<2; t33 43  " .A.-z61  +??((+,K8 FJ/E tzz);;$**,<<4;;.::%!&!1!1%!8EJ"& ! 7 "Ft:: ; - A       +5@;$z1jQQ\] FJ/E o>J z444;;*//"7"77<<:??#9#99::%!&!1!1%!8EJ",// '__ 3 -r;c#K|jd d|jdy#|jdwxYww)a` Unflatten the original parameters. The function assumes that the flat parameter is unsharded. When in the context, unflattens the original parameters as ``nn.Parameter`` views into the flat parameter, and after the context, restores the original parameters as ``Tensor`` views into the flat parameter. TrNF)rrs r<unflatten_as_paramsz#FlatParamHandle.unflatten_as_paramssC !!D!1 7   % % % 6D % % % 6sA-AAAcd|_|js|jdy|j}|j |t j d|jj|jjd}t|j|j|jD]V\}}\}}}|j||||js||_2|j }|j"} |||| z|_X|jj$Jt't|jj$|jj(D]5\} \}\}}}} } }|j|||t+| | } | |_7|j,t.j0k(rGt3t5|jj6D]} d|jj6| <yy)a, Set the original parameter variables' data to be flattened views into the sharded flat parameter. The views are kept as flattened to simplify the case where a parameter is sharded across ranks. Parameters whose data is not present in the sharded flat parameter have their data set to a size-0 empty tensor. We do not delete them to ensure to preserve expected behaviors like model printability. Parameters whose data is present must preserve their variables to be passable to an optimizer. NTrrF)rrr)rr)rrrrYrnrrrr{rorirrMrrNrOr|rDrpr`rr BACKWARD_POSTrrr})rarsize_0_empty_tensorrrKr>r?roffsetrOrGrHrIrs r<r4z"FlatParamHandle._use_sharded_viewss8<4))  % % % 5 __  J'#kk //''??))   AD    = =z?V?VA J J#EJ $   #6#D#D D3t7789 3.2((+ 3 Er;c|j}|j||j}|-t|j|j D] }d|_ y|j|t|j|j|jD]\}}}|jsd|_|j}|jr|s|j}|js|j|jk7rW|j t!j"||_||||zj%|j&|j _||||zj%|j&|_d|_|j Jt+t|j |j,D]N\}\}\} } } } } } t/| | } | r*|jrt1| | } | j |_Hd|_Py)a Set the original parameter variables' gradients to be flattened views into the sharded flat parameter's gradient. This is a no-op if there is no gradient. Parameters whose data is not present in the sharded flat parameter and parameters with ``requires_grad=False`` have their gradients set to ``None``. Since the gradient variables do not need to be preserved, this method does not manipulate existing ``Tensor`` data directly and creates new ``Tensor`` variables instead. N)rrrrr{r|rrror~rMrOrrNrrrYrmreshaperrrDrprr`)rarrrrK is_grad_nonerOrrGrrHrIrJrs r<rz'FlatParamHandle._use_sharded_grad_views7s__  J'   <z11:3L3LM "!  "  D!58     ) )  ) )6  & 1E#\ $,,! !1!@!@&&|-==F55 9R!::-).)9)9%)@EJ*."Vn%<+!'%++. &*&6N3J%K%S%S!KK& "&EJ5 &6((444FO  )):+I+I JG  " BABA1a+q%,K$I !$)<)<$[/B '__ !  "r;c N|jr(|j|js |jsy|j}d}|jr.|jr"|j}t t |dn|}|js |js |jn |j}tt|j|j|jD]\}\}\}}} } } \} } } |st| | s%|jr-|j |}t |dud|j"|t%| | |u} | xs t'|| }|jr| s|rt)d|j*| rt%| | }||j|<|r.t-j.| g}|j1|||||dd}|jr|jD|j8t-j.| g}|j1d|j|||dO|j]|js|jrw|duxst'|j| }|s|t-j2|}t-j.| g}|j1|j||||d||_|j}t|j4D]/\}\} } } }}} t%| | t%||us&t7d|S)as Write back any parameters that changed storage to the handle's ``FlatParameter``. Iterates over the original parameters and writes back any parameters that changed storages (due to a non-inplace operator) to the handle's ``FlatParameter``. This method preserves the ``FlatParameter` 's device even if an original parameter's device changes. Raises: RuntimeError: If an original parameter or gradient changes storages but no longer has the expected flattened shape. Returns: ``True`` if some writeback happened, and ``False`` otherwise. FzPIf skipped using sharded views, the unsharded flat parameter should be allocatedNz!Expects to have saved tensor for zOFSDP does not support changing the parameters between forward and backward for Tz/Changing shared parameters is not supported yet)r) is_shardedrrwrrrrrryrDrr{rorirr}rkr`rAssertionErrorrrYrZ_writeback_tensorrkrpNotImplementedError)rar wrotebackflat_param_tensorflat_param_gradrGrrMrNrOrr>r? param_changedneeds_param_writebackexpected_shapeneeds_grad_writebackrHrIs r<rxz&FlatParamHandle._writeback_orig_paramsssx"  & &OODOO433__   * *t/I/I !% L L  #$56&  !+ ))1E1E OO%%  ""--''  U 6 A  =X1 #Z6:..."++A.%7 8H8H8K7LM$FJ7uDM?$U,=>> "..!6$0040G0G/HJ 3(- ""1%$!&^,(mJJGC$(&.*/*:*::*F%*ZZ0@%AN** '&' '6JO&0ooOkU 6@z55 6  A     vz*'+2WW)E r; src_tensor dst_tensor tensor_indexrris_paramctt|dk(d||jtjj k(rt |dr |jntj}| |jnd}| |jnd} tjd|d|rdndd |jd |d |d |jd | |1|j|k7r"td|rdndd|d|j|&||||jzj!|y||||jzj#|j$j&Jd|j$j&|<y)a Write back ``src_tensor`` to ``dst_tensor`` at offset ``offset``, where ``src_tensor`` should have shape ``expected_shape``. ``is_param`` indicates if the tensor is the parameter (if ``True``) or gradient (if ``False``). If ``src_tensor`` is ``None``, then the effect is zeroing instead of copying. ``tensor_index`` gives the index of ``src_tensor`` in the metadata structures. Raises: RuntimeError: If the ``src_tensor`` does not have the expected shape. r!z$Expects a 1D expected shape but got rNrz] rGradientz needs writeback in z expected shape=z shape=z expected device=z device=zCannot writeback when the parametergradientz shape changes Expects r>T)rrrrrrrrget_rankrrrrr RuntimeErrorrrzero_rr~) rarrr rrr r src_shape src_devices r<rz!FlatParamHandle._writeback_tensor s*   1 $2>2B C     4 4 4 'f 54994==?D,6,B ((I.8.D**$J MMbj IJ $ 4 456""0!1 D##-#4#4"5Xj\K   !j&6&6.&H,H[*,UV**8)9:CSCSBTV   ! v)=)=)? ? @ F Fz R v)=)=)? ? @ F F H??55A AA?CDOO . .| r?rs r<_deregister_orig_paramsz'FlatParamHandle._deregister_orig_paramsW sw//66 ,J$. !Jvz* + ,/3oo.Q.Q , *J1avz* + ,r;c|jj|i||j_|jr?|j |jr|j y|j dyy)z;Wrap an in-place call to ``.to()`` for ``self.flat_param``.TrN)rrrrrr4r)raargskwargss r<r|zFlatParamHandle.flat_param_toc s_1t114B6B  t/'')))D)9 !r;c|jjDchc]}|jc}j|jjDchc]}|jc}Scc}wcc}w)zcReturn a :class:`set` of the modules whose parameters are included in this handle's flat parameter.)rrir?rrp)rarrs r< _get_moduleszFlatParamHandle._get_modulesm sP$(OO$@$@Ab AGG#'??#F#F GCSZZ G  A Gs A*A/ct|jdr |jsy|jj}|j |k(S)z Return whether ``tensor`` is *currently* sharded. For ``NO_SHARD``, we choose to have this always return ``False`` for clarity. rgF)rrr)rgr)rar sharded_sizes r<rzFlatParamHandle.is_shardeds s@9--44 {{} ,,r;c #K|jjDcgc]\}}}}}}t|||}}}}}t|jj|D]}|\}}}||fycc}}}}wwr)rrpr(rri)rar>r?r@rrrs r<param_module_namesz"FlatParamHandle.param_module_names s44    j&+ 6      < <>PQ ,J)3 &J;{+ + ,  sA8A0 ?A8c #K|jjDcgc]\}}}}}}t|||c}}}}D] \}}}||fycc}}}}wwr)rrpr()rar>r?r@rs r<shared_param_module_namesz)FlatParamHandle.shared_param_module_names so44 + +  j&+ 6 + , &J;{+ + , + sAA Acg}t|jj|jjD]#\}}|js|j |%|S)z?Return the FQNs of the parameters present in this rank's shard.)rrrkrorMr)ra fqns_in_shardrrKs r<_fqns_in_shardzFlatParamHandle._fqns_in_shard s]$& %( OO ! !4??#E#E&  * !C! (($$S)  * r;cH|j}t|dr|j}|St|dr|j}|St |j duxs;|j xs,|jtjtjfvd|j }|S)z%Return the handle's sharded gradient.ryrzNzZSharded strategies should use `_cpu_grad` or `_saved_grad_shard` unless in IDLE or FORWARD) rrryrzrrr)rrrr)rarrs r<rzFlatParamHandle.sharded_grad s__ :{ +''D& %Z!4 5//D  4'K111K'''//1D1I1IJK,  ??D r;c.|jsyt|jtjk(d|j }|j Jt|j D]/\}}|js|jJd|j|<1y)a2 Reset ``_is_grad_none_mask`` as needed. This method should only be called in the post-backward after gradient computation, in which case if a parameter requires gradient, then it will surely receive a gradient and we may reset its mask entry to ``False``. NzIExpects to only be called in the post-backward after gradient computationF) rrrrrrr{rDrr~)rarrGrs r<_reset_is_grad_nonez#FlatParamHandle._reset_is_grad_none s$$   $7$E$E E W __ !!---!*"4"45 9HAu""!44@@@38 --a0  9r;c0t|jdy)NzExpects sharded strategy)rr)rs r<rz'FlatParamHandle._check_sharded_strategy s$,,.HIr;c~t|j|jk(d|jd|jy)Nz+Expects tensor to be on the compute device z , was on )rrrars r<riz(FlatParamHandle._check_on_compute_device s5 MMT[[ (9$++iPVP]P] _ r;cvt|jtjdk(d|jy)Nrfz$Expects tensor to be on CPU but got )rrrYr-s r<rzFlatParamHandle._check_on_cpu s- MMU\\%0 026==/ B r;ctjjjst t |ddyy)Nrz9Expects storage to be freed but got storage with size > 0)rYr-r.r/rrrs r<rz$FlatParamHandle._check_storage_freed s7  88QQS "61-K Tr;c.tt|dy)NzExpects storage to be allocated)r_storage_size_allocatedr0s r<_check_storage_allocatedz(FlatParamHandle._check_storage_allocated s)&13TUr;ct|jdtt|jdddud|jjj }t||j k(d|j d|y)Nz&Not using low precision for parametersrxzExpects `_mp_shard` to existz)Expects the low precision shard to be on r>)rrlr`rrxr)rars r<rz*FlatParamHandle._check_low_precision_shard sw  , , 4   DOO[$ 7t C * **11 dkk !7 }IfX V r;c d}t|du|dz|jj}t|j|k(|d|d|jzy)NzExpects tensor to be unsharded but got `None` with size r>)rrrer)rar msg_prefixrs r<rz FlatParamHandle._check_unsharded s]6 &$j3C&CDAA KKM^ + :n%5Yv{{}oN N r;c d}t|du|dz|jj}t|j|k(|d|d|jzy)NzExpects tensor to be sharded r6r7r>)rrrgr)rarr8r s r<rzFlatParamHandle._check_sharded s\4 &$j3C&CD44  KKM\ ) :l^9V[[]OL L r;c<|jtjk7Sr)rr*r7rs r<r)z%FlatParamHandle.uses_sharded_strategy% s&&*@*I*IIIr;c4|j|jk7Sr)rrrs r<rlz+FlatParamHandle._uses_param_mixed_precision) s((D,B,BBBr;c4|j|jk7Sr)r'rrs r<_uses_reduce_mixed_precisionz,FlatParamHandle._uses_reduce_mixed_precision- s!!T%;%;;;r;c|jxs |jxrD|jtjk(xs%|j j xr |jSr)rlr=rrrvrtrainingrrs r<rzz%FlatParamHandle._force_full_precision1 s^  , , Q0Q0Q   $7$J$J J V++44 4 T9T9T  r;c|jduS)a? This property is used for sharding strategies that do not free after forward with ``use_orig_params=True``. This returns if this handle is currently in a state where it has skipped using sharded views, in which case it can restore view invariants via ``_use_sharded_views()``. N)rrs r<rwz*FlatParamHandle._skipped_use_sharded_views< s;;4GGr;)rNr)jr2r3r4rBrrrErrrFrYrr*rRrrr ProcessGroupr$rrrr rSrrrr r%rrno_gradr:r*rLr? staticmethodrPr,rZrXrCrr'rdrrr~r{rryrrrrrrrrrr contextlibcontextmanagerrrrrrr rr enable_gradrrr rr4rrxrrrr|rrrrCr"r$propertyr'rr*rrirrr3rrrr)rlr=rzrw __classcell__)rs@r<r&r&sZ48g3r||V345g3!iig3 g3 2 g3  g3!-g3"%++.g3#'g3((g3g3!0g3R @  T U62<</01T  T  T  T  T l%7E&",,"678%7 %7N0.f0.0.  0.dKfKK K  K!. -!."%++.!.  !.LU]]_&&B)6)6!)6 )6  )6V=( =(=( # $ =(~#### vs{  ##8#### vs{  ##& E& E E E E E eCHo)> # # # JU]]_SGSGp!T!F/ D4%t%$ $%,,$:/+%+/+ /+b7%*\\7 7>7 1U]]_7)7)r6@#D*5X)H)HV..$5*,$/3h*.& & <$(  f@U[@d[@t[@[@zC-C-J 7Y 7 7U]]_2323hU]]_8"8"tU]]_RRRh0DV$0D0D 0D  0D  0D0D 0Dd18,: c"))n - -D -,HU38_$=, ,8E#s(O+D ,S hv.<96J v  F VVVV   v  V JtJJCTCCrrc^||j|<ttj|||yr)rrrErF __setattr__)r?r>rs r<rrI s+&+Fz" "))V(U;r;rcx|jj|dttj|||yr)rpoprrErFrJ)r?r>rs r<rrR s/ :t, "))V(VrMs r<rrY s$vz" # FJ0r;rc|Dcgc]3}t|tjr|ntj|5c}Scc}wr)r_rEr)rts r<rrb s4LS SaAr||,A",,q/ A SS Ss8Aparam_or_tensorcZt|tjr|jS|Sr)r_rErdetach)rRs r<rrh s/ or|| 4  r;rc*d}t|}||z}|S)N)_get_dtype_size)r ALIGNMENTunsharded_dtype_sizers r<rrp s#I*?;!55M r;cLtjd|jS)Nr:r)rYrn element_sizer\s r<rWrWx s ;;r ' 4 4 66r; padding_numelrrrcDtj|f|||tzS)N)rrr)rYones_FLAT_PARAM_PADDING_VALUE)r^rrrs r<rr} s+   Ev  $ $r;logwarningc.tj|yrrrcrbrcs r<rr  NN7r;c.tj|yrrerfs r<rr rgr;c.tj|yrrerfs r<rr rgr;cddlm}t||r |j}t||r |j}|j j |j j k(S)Nrr)rrr_ _local_tensorrdata_ptr)abrs r<rr s\1!W OO!W OO    ' ' )Q->->-@-I-I-K KKr;rmrncf|jj|jz|k(Sr)rrr])rmrns r<rr s+    # # %)9 9Q >>r;cH|jj}|dkDS)Nr)rr)r storage_sizes r<r2r2 s$..0557L ! r;)frD functoolsloggingrrenumrr itertoolsrrtypingrrr r r r r rrrrrrrrYtorch.distributedr-rtorch.nnrEtorch.nn.functional functionalrSr$torch.distributed.fsdp._common_utilsrrrrrtorch.distributed.utilsrrrrtorch.nn.parameterr+torch.testing._internal.distributed.fake_pgr _fsdp_extensionsr"r#r$__all__ getLoggerr2rrrrrarrr*r5r8'RESHARD_AFTER_FORWARD_HANDLE_STRATEGIESr6r9rr(r)rLr'r\rr%r&rFrCrrrrrrr lru_cacherWrSrRrrLoggerrrrrrr2r:r;r<rsW '"  .H    8 $459; 7.!T!%%''+' ((...*  j$ 'j '/Z/,  o+BLL,>o+dB"HB"HLD< II<#&vr||?S9T1T % bll23 4T ",,T uR\\6-A'Bv Q77  ${{ ;? IN QGNNS Q7>>C Qw~~ L?%,,?3?Fr;