sg ddlmZddlmZmZddlZddlmcmcmZ ddl m Z ddl m Z ddlmZmZddlmZddlmZdgZd e j*d e d eeeeeffd e j*fd Zy)fnmatch)DictUnionN) DeviceMesh)is_rng_supported_meshTensorParallelRNGTracker)_validate_tp_mesh_dim) ParallelStyleparallelize_modulemodule device_meshparallelize_planreturncJ tjjdt|t |rrt t jtsTt|jt _t jj|ddt j_ t |tr|j||St |tr|jD]\}}|j!d}t#|dk(r t%d|s3|j'd t) fd|j+}|D]5\}}|r!dj-|} t/||| |i)t/|||7|rk|St1d t3|d ) a Apply Tensor Parallelism in PyTorch by parallelizing modules or sub-modules based on a user-specified plan. We parallelize module or sub_modules based on a parallelize_plan. The parallelize_plan contains :class:`ParallelStyle`, which indicates how user wants the module or sub_module to be parallelized. User can also specify different parallel style per module fully qualified name (FQN). Note that ``parallelize_module`` only accepts a 1-D :class:`DeviceMesh`, if you have a 2-D or N-D :class:`DeviceMesh`, slice the DeviceMesh to a 1-D sub DeviceMesh first then pass to this API(i.e. ``device_mesh["tp"]``) Args: module (:class:`nn.Module`): Module to be parallelized. device_mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. parallelize_plan (Union[:class:`ParallelStyle`, Dict[str, :class:`ParallelStyle`]]): The plan used to parallelize the module. It can be either a :class:`ParallelStyle` object which contains how we prepare input/output for Tensor Parallelism or it can be a dict of module FQN and its corresponding :class:`ParallelStyle` object. Return: A :class:`nn.Module` object parallelized. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch.distributed.tensor.parallel import parallelize_module, ColwiseParallel >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> # Define the module. >>> m = Model(...) >>> tp_mesh = init_device_mesh("cuda", (8,)) >>> m = parallelize_module(m, tp_mesh, {"w1": ColwiseParallel(), "w2": RowwiseParallel()}) >>> .. note:: For complex module architecture like Attention, MLP layers, we recommend composing different ParallelStyles together (i.e. ``ColwiseParallel`` and ``RowwiseParallel``) and pass as a parallelize_plan, to achieves the desired sharding computation. z4torch.distributed.tensor.parallel.parallelize_modulei) base_seedF.rz9Expect module path to be non-empty, but got empty string!c"t|dS)Nrr)tatoms X/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/tensor/parallel/api.pyz$parallelize_module..asgadD1zLExpect Union[ParallelStyle, Dict[str, ParallelStyle]] for parallelize_plan, z found!)torch_C_log_api_usage_oncer r isinstancerandom _rng_trackerr device_type _manual_seeddistribute_region_enabledr _applydictitemssplitlen ValueErrorpopfilternamed_childrenjoinr TypeErrortype) r rr module_pathparallelize_style path_splitsmatched_children_ submodule leaf_pathrs @rr r s\ HH  !WX+&[)*537{7N7NO(((E9>5"M2&&v{;; $d +.>.D.D.F V *K*%++C0K;1$ O"q)#)1))+$ %5 VLAy"$'HH'% +%{Y@Q4R +9kCTU V  V4  ""&'7"8!9 B  r)rtypingrrr torch.distributed.tensor._random distributedtensor_randomrtorch.nnnntorch.distributed.tensorrrr (torch.distributed.tensor.parallel._utilsr 'torch.distributed.tensor.parallel.styler __all__Modulestrr rrrDs~ 11/KA  ` II` ` M4]0B+CCD` YY ` r