sgu$fddlZddlmZmZmZmZmZmZmZm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZmZdZGdd eZd ed ed efd ZGddZee dde j0deee jj0d e j0fdZde j0d efdZy)N) AnycastDictIterableListNoReturnOptionalSetTuple)_State)DistributedDataParallel) _get_registrycontractc eZdZdfd Zefdej deej deejde ddf dZ dd Z dej deej ddfd Z dd Z dd Zdej d eedfdee efdefdZdej deej(dej(dej(fdZxZS)_ReplicateStatereturnNct|tj|_d|_tj|_|j|_g|_d|_ d|_ i|_ g|_ y)NF) super__init__nn ParameterListmodulehas_initialized _param_list _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_args)self __class__s Z/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/_composable/replicate.pyrz_ReplicateState.__init__sk !#!1!1!3 %*-/-=-=-?!KK')# 59,.*,rignored_modulesignored_paramsprefixcbt|ry||vry|tk7r|dnt}|jdD]D\}}||vs |jj ||j j ||F|j D]\}} |j| |||| y)N.F)recurse)r))_is_fully_sharded_ROOT_MODULE_PREFIXnamed_parametersrappendrnamed_children_collect_params) r#rr'r(r)recurse_prefixnpname child_modules r%r2z_ReplicateState._collect_params s V $  _ $ #&99vhaL?R ++E+: ADAq&  ''*!!((N+;A3)?@ A #)"7"7"9  D,  ()$0 !  r&cPtjdfd}|y)NT) recursivecjJjjijjd_i_y)N)r initr!register_comm_hook)r#sr% _lazy_initz-_ReplicateState.lazy_init.._lazy_initAsK??. .. DIIt <$*;*; <  # # % DO "D r&)torch_disable_dynamo)r#r>s` r% lazy_initz_ReplicateState.lazy_init@s'    . # / #  r&c |jryd|_|jdd}||_|Dchc]}|jD]}|}}}|j D]-}t |s|j |j/ddlm} | |||j|||d|vrV|d;|d} t| tjr| jdk(rd|d<n | g|d<nd|d<|jdt|j fi||_t%j&|j"t(j+|j_ycc}}w) NT device_meshr)_localize_dtensor)r( device_idcpu device_ids)rgetr parametersmodulesr-update%torch.distributed.tensor.parallel.ddprDr2 isinstancer?devicetypepopr r_ddpweakrefref replicatestate _ddp_weakref) r#rr'kwargsrCmr5r( submodulerDrEs r%r<z_ReplicateState.initKsP    #jj5  %4MallnM!M!MM) >I +%%i&:&:&<= > L&@ V_nE & k".";/ i69>>U;R+/F<(-6;F<('+|$ JJ{ #+D,<,<GG 4;KK 4J  $1;NsE.c|jD]!\}}|jj|i|#|jjyN)r"rQr=clear)r# comm_args comm_kwargss r%r=z"_ReplicateState.register_comm_hookwsI&*&:&: D "I{ (DII ( () C{ C D ""$r&c ||_||_yr[)r r!r#argsrWs r%record_init_argsz _ReplicateState.record_init_args|s"r&ra.rWc|js |jr|j|j |j_|jj |i|Sr[)r r!rArrQrequire_backward_grad_sync _pre_forward)r#rrarWs r%forward_pre_hookz _ReplicateState.forward_pre_hooksL ??d// NN 37==/@ ,%tyy%%t6v66r&inputoutputc8|jj|Sr[)rQ _post_forward)r#rrgrhs r%forward_post_hookz!_ReplicateState.forward_post_hooks yy&&v..r&rN)__name__ __module__ __qualname__rr.rModuler Parameterstrr2rAr<r=rbr rrrfr?Tensorrk __classcell__)r$s@r%rrs -&*  RYYBLL)     @ *K *KRYY*K  *KX% #7ii7',S#X7@DS#X7 7/ /U\\"/ /  /r&rrarWrctd)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)rarWs r%unimplemented_deepcopyrws Q r&c*eZdZdZdeddfdZddZy)DDPcL|jd}|j|g|i|S)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsrarWorig_clss r%r}z DDP.__new__s.;;q>x:4:6::r&requires_gradient_syncrNc:| tj|_y)a Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rTrUr)r#rs r%set_requires_gradient_synczDDP.set_requires_gradient_syncs.D)C &r&cdtj|jj||fyr[)rTrUr"r0r`s r%r=zDDP.register_comm_hooks#--44dF^Dr&rl)rmrnror}boolrr=r;r&r%ryrys#; D D$ DEr&ry) state_clsrr'c tjjdd|vr=t|dttj fst dt|dt|r t d|i}n t|}tttj|}|j|jd|j!dd}|Fd d lm}|j'|}||k7r*d d lm}m}|j||j/||j/|j0|j2||fi||j4} d t6i} td | j8t:| f| } | |_|S)zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicaterEz6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`NT) with_kwargsrCr)_mesh_resources)rD_reconstruct_dtensor __deepcopy__ry)r?_C_log_api_usage_oncerMintrN RuntimeErrorrOr-setrrrTrUregister_forward_pre_hookrfrHtorch.distributed.device_meshr get_root_meshrLrDrregister_forward_hookrkrbr$rwrmry) rr'rWrUrCr root_meshrDrr~dctnew_clss r%rTrTs  HH  !>?f&-U\\/BC{ 3457    U  o. )//&"9 :E $$U%;%;$N**]D1KA#11+>   #   , ,-A B  ( (): ;   !8!89E6?=f=   C 1 2CS'#sS9GF Mr&c&t|}|yd|vS)z+Check if module is marked with fully_shard.F fully_shard)r)rregistrys r%r-r-s V$H H $$r&r[)rRtypingrrrrrrr r r r?torch.nnr#torch.distributed._composable_stater torch.nn.parallelr rrr.rrwryrprTrr-r;r&r%rsRRR 65-}/f}/@#EE4 O$<@E IIEhuxx78EYY E%EP%bii%D%r&