sgL8ddlZddlZddlmZmZmZmZmZmZddl Z ddl m cm cm Zddlm cm cmZddlmZddlmZmZddlmZmZddlmZmZddlmZmZddlm Z m!Z!ddl"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*gd Z+e jXjZZ-Gd d e j\j^Z0Gd de j\j^Z1Gdde jdZ3 d)de jddeedeee(de3fdZ4 d*dejjdeedeee6ejjegdfdeeejjeegdfdeeejjeegdfdejjf dZ7 d)de jpdeedeee(de3fdZ9de jtdddddee jvd e jxd!e=deedeee(de3f d"Z>de jtdddddee jvd e jxd!e=deedeee(de3f d#Z?de jtdddddee jvd e jxd!e=deedeee(de3f d$Z@dde jtddd%d!e=dee jvd e jxdeedeee(de3f d&ZAdde jtddd%d!e=dee jvd e jxdeedeee(de3f d'ZBdde jtddd%d!e=dee jvd e jxdeedeee(de3f d(ZCy)+N)AnyCallablecastOptionalSequenceTuple)_mesh_resources DeviceMesh)check_tensor_metamesh_broadcast) DTensorSpec TensorMeta)is_rng_supported_meshOffsetBasedRNGTracker) Redistributeredistribute_local_tensor)compute_global_tensor_infocompute_local_shapenormalize_to_torch_size)Partial Placement ReplicateShard) DTensordistribute_tensordistribute_moduleonesemptyfullrandrandnzerosc\eZdZedddeeefdZedejfdZ y)_ToTorchTensorinputrgrad_placementscl|j|_||_|j}|j |SN)_spec dtensor_specr& _local_tensorview_as)ctxr%r& local_tensors P/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/tensor/_api.pyforwardz_ToTorchTensor.forwardHs6 !;;-** ##L11 grad_outputc \|j}|j}|j}|j}t |||j \}}t |}|xs |j }t||t|j||j}t|||jdfS)Nshapestridedtype tensor_meta requires_grad) r*meshr&r9r placementstupler rr5r7rr;) r-r2r*r<r& dtensor_meta_ tensor_stride grad_specs r/backwardz_ToTorchTensor.backwardWs''   --#// 5 |66 =m, )D\-D-D  ""(($"((  )77     r1N) __name__ __module__ __qualname__ staticmethodrrrr0torchTensorrCr1r/r$r$GsO 2 2"(9"56 2 2 5<<  r1r$ceZdZe d dej dedeedfde de ejde ee dfd d fd Z edd Zy)_FromTorchTensorNr% device_meshr=. run_checkr5r6returnrc f||_||_|r|r||}}nH|s3|s1t|||\} } tj| t | }}nt d|d|dd|j|jd|j}nY|rW| xr| } t|| t|D]4\} } | js|j}t||| 6t||t!|||j" }t%|j'|||j}|S) Nz Found shape:z , stride:.z3Please pass both shape and stride at the same time.rr:)check_shape_stride)mesh_dimr8)previous_placementprevious_device_meshrrHSizer> RuntimeErrorget_coordinate new_emptyr;r enumerate is_replicate contiguousr r rr7rr,)r-r%rMr=rNr5r6 tensor_shaperA global_shape global_striderRidx placement dist_spec dist_tensors r/r0z_FromTorchTensor.forwardxs^",#. V*/-Lv+E{J+ 'L-+0**\*BE-DX-LugYvha8E   % % ' /OOAU5H5HOIE &+!96z  e8J K#,J"7 EY))+",,.E"5+D  E  "   MM%  --  r1c|j}|j}|j|k7rQ|j}t |||jj }|j }t|||d}|dddddfS|jdddddfS)Nr8T) is_backward) rTrUr=r)r r9r+rto_local)r-r2rTrU current_spec target_specr.outputs r/rCz_FromTorchTensor.backwards 33"77  ! !%7 7&,,L%$"'--99K '44L.lKTF 4tT47 7##%tT4tCCr1NN)r2r)rDrErFrGrHrIr rrboolrrVintr0rCrJr1r/rLrLws'+,0@||@ @)S.) @  @  # @sCx)@ @@DDDr1rLceZdZUdZej ed<eed<ddgZe jZ e jed<e ejdej dededdfd Zd Zd Ze d Zd ZdZeejd(dZe d)dddddej deedeeededeej6deeedfddfdZdddeeedej fdZ d)dddeedeeededdfdZ dddeeedej fd Z!e"defd!Z#e"deedffd"Z$d#e%d$e&fd%Z'd&Z(d'Z)y)*ra ``DTensor`` (Distributed Tensor) is a subclass of ``torch.Tensor`` that provides single-device like abstraction to program with multi-device ``torch.Tensor``. It describes the distributed tensor sharding layout (DTensor Layout) through the :class:`DeviceMesh` and following types of :class:`Placement`: * :class:`Shard`: Tensor sharded on the tensor dimension ``dim`` on the devices of the ``DeviceMesh`` dimension * :class:`Replicate`: Tensor replicated on the devices of the ``DeviceMesh`` dimension * :class:`Partial`: Tensor is pending reduction on the devices of the ``DeviceMesh`` dimension When calling PyTorch operators, ``DTensor`` overrides the PyTorch operators to perform sharded computation and issue communications whenever necessary. Along with the operator computation, ``DTensor`` will transform or propagate the placements (DTensor Layout) properly (based on the operator semantic itself) and generate new ``DTensor`` outputs. To ensure numerical correctness of the ``DTensor`` sharded computation when calling PyTorch operators, ``DTensor`` requires every Tensor argument of the operator be DTensor. r+r)_op_dispatcherr.specr;rOc f|jr|stjd|jJdtj j ||jj|jj|j|j|j|}||_ ||_ |S)ag Construct a DTensor from a local tensor, device mesh, and placement and other tensor properties (i.e. shape, requires_grad, strides, etc). .. note:: This is not a public API and it's only supposed to be used by the operator implementations and internals. If you want to construct a DTensor from a local tensor, consider using ``DTensor.from_local``, if you want to construct a DTensor from a "global" tensor (where you already have tensor initialized and want to shard this tensor), consider using ``distribute_tensor``. zxTo construct DTensor from torch.Tensor, it's recommended to use local_tensor.detach() and make requires_grad consistent.zTensorMeta should not be None!)stridesr7devicelayoutr;)r;warningswarnr9rHrI_make_wrapper_subclassr5r6r7rrrsr)r+)clsr.ror;rs r/__new__zDTensor.__new__s(  % %m MMO +M-MM+ LL / /     " "$$++$$&&&&' 0 &r1c~d|jd|jjd|jjdS)NzDTensor(local_tensor=z, device_mesh=z , placements=))r+r)r<r=selfs r/__repr__zDTensor.__repr__s=&t'9'9&:.HYYfgkgqgqg|g|f}}~r1c8dg|j|jffS)ze protocol to inform how to flatten a DTensor to local tensor for PT2 tracing r+)r)r;r|s r/__tensor_flatten__zDTensor.__tensor_flatten__"s!   4::t/A/A"BBBr1c|Jd|d}|\}}t|||jj}t|j|j |}t |||S)NzEExpecting spec to be not None from `__tensor_flatten__` return value!r+r4r8r:)rr9r7r r<r=r) inner_tensors flatten_spec outer_size outer_strider.ror;unflatten_tensor_metaunflatten_specs r/__tensor_unflatten__zDTensor.__tensor_unflatten__)s  $ S R S $$_5 *m *""((!  % II OO-    '  r1ctd|jDs|S|jDcgc]}t|tr t n| }}|j |j |Scc}w)Nc3<K|]}t|tywr() isinstancer).0ps r/ z6DTensor.__coerce_tangent_metadata__..AsCa:a)CsrMr=)anyr=rrr redistributerM)r}rr=s r/__coerce_tangent_metadata__z#DTensor.__coerce_tangent_metadata__@sjC4??CCKBF// =>:a1IKq 8    T-=-=* UU s#A/cZ|\}}|j|j|jS)Nr)rrMr=)r}rror@s r/#__coerce_same_metadata_as_tangent__z+DTensor.__coerce_same_metadata_as_tangent__Hs3 q  ((!  r1NcLtjj|||xsiSr()rrndispatch)rwfunctypesargskwargss r/__torch_dispatch__zDTensor.__torch_dispatch__Os* %%..   Lb  r1F)rNr5r6rMr=rNr5r6.cH|xstj}|j}||jjk7r|j s|j |}|+t|jDcgc] }t}}ntt|}t|D][\}} | jstt| } | jdks7t| j|jz||<]t j#||t%||||Scc}w)a" Create a :class:`DTensor` from a local torch.Tensor on each rank according to the ``device_mesh`` and ``placements`` specified. Args: local_tensor (torch.Tensor): local torch.Tensor on each rank. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the local torch.Tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. Keyword args: run_check (bool, optional): at a cost of extra communications, perform sanity check across ranks to check each local tensor's meta information to ensure correctness. If have :class:`Replicate` in ``placements``, the data on first rank of the device mesh dimension will be broadcasted to other ranks. default: False shape (torch.Size, optional): A List of int which specifies the size of DTensor which build on top of `local_tensor`. Note this needs to be provided if the shape of ``local_tensor`` are different across the ranks. If not provided, ``shape`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None stride (tuple, optional): A List of int which specifies the stride of DTensor. If not provided, ``stride`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None Returns: A :class:`DTensor` object .. note:: When ``run_check=False``, it is the user's responsibility to ensure the local tensor passed in is correct across ranks (i.e. the tensor is sharded for the ``Shard(dim)`` placement or replicated for the ``Replicate()`` placement). If not, the behavior of the created DTensor is undefined. .. note:: ``from_local`` is differentiable, the `requires_grad` of the created `DTensor` object will depend on if `local_tensor` requires_grad or not. r)r get_current_mesh device_typerrtypeis_metatorangendimrlistrZis_shardrrdimrLapplyr>) r.rMr=rNr5r6rr@r`ras r/ from_localzDTensor.from_localZsl"G_%E%E%G !--  ,--22 2<;O;O'??;7L  /4[5E5E/FG!)+GJGj)J"+J"7 SY%%' $UI 6I }}q(*/ @Q@Q0Q*R 3  S %%   *       Hs4D)r&r&ctjs |jS|t|ts t |}t j ||S)a Get the local tensor of this DTensor on its current rank. For sharding it returns a local shard of the logical tensor view, for replication it returns the replica on its current rank. Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the Tensor returned from this function. `to_local` converts DTensor to local tensor and the returned local tensor might not be used as the original DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original DTensor layout. If not specified, we will assume the gradient layout remains the same as the original DTensor and use that for gradient computation. Returns: A :class:`torch.Tensor` or ``AsyncCollectiveTensor`` object. it represents the local tensor on its current rank. When an ``AsyncCollectiveTensor`` object is returned, it means the local tensor is not ready yet (i.e. communication is not finished). In this case, user needs to call ``wait`` to wait the local tensor to be ready. .. note:: ``to_local`` is differentiable, the ``requires_grad`` of the local tensor returned will depend on if the `DTensor` requires_grad or not. )rHis_grad_enabledr+rr>r$r)r}r&s r/rfzDTensor.to_localsM8$$&%% %  &z/5/Q#O4O## /  r1)async_oprc|xs |j}| tdt|}t|D]f\}}|j r tdt |t s2|jdksBt |j|jz||<ht|}tj||||S)as ``redistribute`` performs necessary collective operations that redistribute the current DTensor from its current placements to a new placements, or from is current DeviceMesh to a new DeviceMesh. i.e. we can turn a Sharded DTensor to a Replicated DTensor by specifying a Replicate placement for each dimension of the DeviceMesh. When redistributing from current to the new placements on one device mesh dimension, we will perform the following operations including communication collective or local operation: 1. ``Shard(dim)`` -> ``Replicate()``: ``all_gather`` 2. ``Shard(src_dim)`` -> ``Shard(dst_dim)``: ``all_to_all`` 3. ``Replicate()`` -> ``Shard(dim)``: local chunking (i.e. ``torch.chunk``) 4. ``Partial()`` -> ``Replicate()``: ``all_reduce`` 5. ``Partial()`` -> ``Shard(dim)``: ``reduce_scatter`` ``redistribute`` would correctly figure out the necessary redistribute steps for DTensors that are created either on 1-D or N-D DeviceMesh. Args: device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor. If not specified, it would use the current DTensor's DeviceMesh. default: None placements (List[:class:`Placement`], optional): the new placements that describes how to place the DTensor into the DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. default: replicate on all mesh dimensions Keyword args: async_op (bool, optional): whether to perform the DTensor redistribute operation asynchronously or not. Default: False Returns: A :class:`DTensor` object .. note:: ``redistribute`` is differentiable, which means user do not need to worry about the backward formula of the redistribute operation. .. note:: ``redistribute`` currently only supports redistributing DTensor on the same DeviceMesh, Please file an issue if you need to redistribute DTensor to different DeviceMesh. z&placements is needed for redistribute!zTCan not redistribute to Partial, redistributing to Partial is for internal use only!r) rMrWrrZ is_partialrrrrr>rr)r}rMr=riras r/rzDTensor.redistributesj"5T%5%5  GH H*% %j1 ALAy##%"jIu-)--!2C %immdii&? @ 1  A:& !!$ ZJJr1c|jtg|jjzd}tj ||S)a Return the full tensor of this DTensor. It will perform necessary collectives to gather the local tensors from other ranks in its DeviceMesh and concatenate them together. It's a syntatic sugar of the following code: ``dtensor.redistribute(placements=[Replicate()] * mesh.ndim).to_local()`` Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the full Tensor returned from this function. `full_tensor` converts DTensor to a full torch.Tensor and the returned torch.tensor might not be used as the original replicated DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original replicated DTensor layout. If not specified, we will assume the gradient layout of the full tensor be replicated. Returns: A :class:`torch.Tensor` object that represents the full tensor of this DTensor. .. note:: ``full_tensor`` is differentiable. F)r=r)rrrMrr$r)r}r& redist_ress r/ full_tensorzDTensor.full_tensorsH4&&! }t'7'7'<'< ?%%;;= = **ELL 9.t45 59: :r1ct|jdr|jj|St|jtj r|j Std)N__get_tensor_shard__r)rr+rrrHrIrfrW)r}indexs r/rzDTensor.__get_tensor_shard__fsV 4%%'= >%%::5A A **ELL 9==? "9: :r1)rJNrj)*rDrErF__doc__rHrI__annotations__r __slots__ op_dispatch OpDispatcherrnrG_disable_dynamorkryr~rrrr classmethodrrr rrrVrrlrrfrrpropertyrMr=strrrrrrJr1r/rrs$<<  '*I0H{/G/G/INK,,I 'll''  '  ''V@C  ,V    -148R  &*,0R llR j)R Xi01R  R  # R sCx)R  R R jCG# "*8I+>"?# # N-148FK  FKj)FKXi01FK  FK  FKRCGA"*8I+>"?A A>Z%E)S.1%% ;# ;s ; ;;r1rtensorrMr=rOc ltjjd|xstj}|j }|dk(r ddlm}||||Stjst|rt|t_ |js td||jj k7r|j"s|j%|}|*t'|j(Dcgc] }t+}}t-||j(k7r%t/dt-|d |j(d t1|t2ra|j4|k7rt/d |j4d |d |j6t9|k7rt/d |j6d|d|S|j;}t=|}t?|D]\} } | jArZtCtD| } | jFdkr'tE| jF|j(z} | || <| jI||| }p| jKr$tCt*| } | jM||| }td| d| dt9|}|JdtO||tQ|jS|jU|jV} t3|jY|jZ| |jZS#t$r}d}t||d}~wwxYwcc}w)av Distribute a leaf ``torch.Tensor`` (i.e. nn.Parameter/buffers) to the ``device_mesh`` according to the ``placements`` specified. The rank of ``device_mesh`` and ``placements`` must be the same. The ``tensor`` to distribute is the logical or "global" tensor, and the API would use the ``tensor`` from first rank of the DeviceMesh dimension as the source of truth to perserve the single-device semantic. If you want to construct a DTensor in the middle of the Autograd computation, please use :meth:`DTensor.from_local` instead. Args: tensor (torch.Tensor): torch.Tensor to be distributed. Note that if you want to shard a tensor on a dimension that is not evenly divisible by the number of devices in that mesh dimension, we use ``torch.chunk`` semantic to shard the tensor and scatter the shards. The uneven sharding behavior is experimental and subject to change. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to distribute the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. If not specified, we will by default replicate the tensor across the ``device_mesh`` from the first rank of each dimension of the `device_mesh`. Returns: A :class:`DTensor` or ``XLAShardedTensor`` object. .. note:: When initialize the DeviceMesh with the ``xla`` device_type, ``distribute_tensor`` return `XLAShardedTensor` instead. see `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_tensorxlar)xla_distribute_tensorDTo use DTensor API with xla, you must install the torch_xla package!NzY`distribute_tensor` should be used to distribute leaf tensors! but found non-leaf tensor!zW`placements` must have the same length as `device_mesh.ndim`! Found placements length: z, and device_mesh.ndim: rQz-Cannot distribute a DTensor with device mesh z to a different device mesh z,Cannot distribute a DTensor with placements z to a different placements z-. do you want to call `redistribute` instead?z8Trying to distribute tensor with unsupported placements z on device mesh dimension !z(distributing a tensor should not be Noner4)r<r=r9r:).rH_C_log_api_usage_oncer rrtorch_xla.distributed.spmdr ImportErrorrandom _rng_trackerrris_leafrWrrrrrrrrlen ValueErrorrrrMr=r>detachrrZrrrr _shard_tensorr[_replicate_tensorr rsizer6r7requires_grad_r;) rrMr=rremsgr@r.r`raros r/rrosCJ HH  !BCC!A!A!CK))Ke * ) Z    #8#E3K@ >> g  fmm(((;'+01A1A+BCaikC C :+***((+J'88PQ\QaQaPbbc e  &'"    ,?@R@R?ST..9]!=    j 1 1>v?P?P>QR--7L9*+   ==?Lj!J#J/Y    UI.I}}q !)--&++"=> "+ 3$22<cRL  # # %Y 2I$66|[RUVLJ9+Uopsottuv z"J  #O%OO #  ++-==?,,  D ##F$8$89 ** c *XCc" ) *,Ds L/L1 L.L))L.module partition_fninput_fn output_fnctjjdxstjj }|dk(r ddlm}|||Sdtjdtd dfd } |"|jD]\} } | | n+|jD]\} } || | | | ttjj } | d k(r2t#j$d t&d |j)fdn*| dk(r|j)fdnt+d| dttjj } | d k(r3t#j$dt&d |j-fd|S| dk(r|j-fd|St+d| d|S#t$r}d}t||d}~wwxYw)ai This function expose three functions to control the parameters/inputs/outputs of the module: 1. To perform sharding on the module before runtime execution by specifying the ``partition_fn`` (i.e. allow user to convert Module parameters to :class:`DTensor` parameters according to the `partition_fn` specified). 2. To control the inputs or outputs of the module during runtime execution by specifying the ``input_fn`` and ``output_fn``. (i.e. convert the input to :class:`DTensor`, convert the output back to ``torch.Tensor``) Args: module (:class:`nn.Module`): user module to be partitioned. device_mesh (:class:`DeviceMesh`): the device mesh to place the module. partition_fn (Callable): the function to partition parameters (i.e. shard certain parameters across the ``device_mesh``). If ``partition_fn`` is not specified, by default we replicate all module parameters of ``module`` across the mesh. input_fn (Callable): specify the input distribution, i.e. could control how the input of the module is sharded. ``input_fn`` will be installed as a module ``forward_pre_hook`` (pre forward hook). output_fn (Callable): specify the output distribution, i.e. could control how the output is sharded, or convert it back to torch.Tensor. ``output_fn`` will be installed as a module ``forward_hook`` (post forward hook). Returns: A module that contains parameters/buffers that are all ``DTensor`` s. .. note:: When initialize the DeviceMesh with the ``xla`` device_type, ``distribute_module`` return nn.Module with PyTorch/XLA SPMD annotated parameters. See `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_modulerr)xla_distribute_modulerNmr<rOc tg|jz}|jjD]S\}}| t |t r|j |tjt|j||U|jjD]3\}}| t |t rt||||j|<5yr() rr _parametersitemsrrregister_parameternn Parameterrdata_buffers)rr<full_replicatekeyparambuffers r/replicate_module_params_buffersz:distribute_module..replicate_module_params_buffers9s$+2----/ JC E7)C$$LL!25::t^!TU  ::++- RKC!*VW*E"3FD."Q 3 Rr1zDeprecating input_fn that takes two arguments (inputs, device_mesh), please use input_fn that takes in (module, inputs, device_mesh) instead!) stacklevelc|Sr(rJ)r@inputsrMrs r/z#distribute_module..csxP[?\r1c||Sr(rJ)modrrMrs r/rz#distribute_module..gsHS&+$Fr1z-input_fn should take in 3 arguments, but got z arguments!zDeprecating output_fn that takes two arguments (inputs, device_mesh), please use output_fn that takes in (module, inputs, device_mesh) instead!c|Sr(rJrroutputsrMrs r/rz#distribute_module..ysYw -Lr1c||Sr(rJrs r/rz#distribute_module..}sYsG[-Qr1z.output_fn should take in 3 arguments, but got )rHrrr rrrrrrModuler named_modulesrinspect signature parametersrtru FutureWarningregister_forward_pre_hookrregister_forward_hook) rrMrrrrrrrrnamesubmodnum_argss ` `` r/rrs7R HH  !BCC!A!A!CK))Ke * ) \8Y R299RJR4R&#002 ALD& +FK @ A#002 ALD& v{ 3 +FK @ A w((2==> q= MM[    , ,-\ ] ]  , ,F ?zU w((3>>? q= MM\    ( (L  M]  ( (Q  M @ +V  M_ *XCc" ) *s G$$ H-G;;Hrc ^|xstj}|j|d<|xs%tdt |j D}|j t |k(sJd|dtjk(sJdtjj|}t|||}|tjk(r|jdd}|||fi|}n|tjk(s|tjk(r|j!dtj"} t%|d | } t'|t|| } t)j*|r-t(j,st)j.t(_t(j,Jt(j,j1| 5||fi|}dddn ||fi|}t'|t|t%||j2 } t5|| |d  S#1swYFxYw) Nrrc30K|]}tywr()r)rr@s r/rz'_dtensor_init_helper..s$RQY[$Rsz6mesh dimension does not match the length of placementsrszlayout value not supported! fill_valuerr7)rr8r;r:)r rrr>rrrrHstrided _prims_commonmake_contiguous_strides_forrrpopr r!getget_default_dtyperr rrrr_distribute_regionr7r) init_oprrMr=r torch_stride local_shaperr.r7r9ros r/_dtensor_init_helperrs C!A!A!CK"..F8Ru$R% @P@P:Q$RRJ   s  @?@  ( u}} ,K.KK ,&&BB4HL&dKDK%**ZZ a0 {JA&A EJJ 'U[["8 7E$;$;$=> tU3 ;j(9{S  ' ' 4V=P=P"(">">"@F ""...  3 3D 9 :";9&9L : :{5f5  j       D  _-  : :s H##H,Fr7rsr;rMr=r7rsr;c Xt|}ttj||||||S)aS Returns a :class:`DTensor` filled with the scalar value 1, with the shape defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrHrr7rsr;rMr=r torch_sizes r/rrs4<).J  # r1c Xt|}ttj||||||S)at Returns a :class:`DTensor` filled with uninitialized data. The shape of the :class:`DTensor` is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: empty(1,2,3..) or empty([1,2,3..]) or empty((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrHrrs r/rrs4<).J  # r1c Zt|}ttj|||||||S)a Returns a :class:`DTensor` filled with ``fill_value`` according to ``device_mesh`` and ``placements``, with the shape defined by the argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) fill_value(Scalar): the value to fill the output tensor with. Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank )rr7rsr;rMr=)rrrHr)rrr7rsr;rMr=rs r/rr"s8B).J  #  r1)r;r7rsrMr=c Xt|}ttj||||||S)a Returns a :class:`DTensor` filled with random numbers from a uniform distribution on the interval ``[0, 1)``. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrHr r;r7rsrMr=rrs r/r r Qs4>).J  # r1c Xt|}ttj||||||S)a Returns a :class:`DTensor` filled with random numbers from a normal distribution with mean 0 and variance 1. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrHr!r!s r/r!r!}s4>).J  # r1c Xt|}ttj||||||S)a  Returns a :class:`DTensor` filled with the scalar value 0. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: zeros(1,2,3..) or zeros([1,2,3..]) or zeros((1,2,3..)) Keyword args: requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r)rrrHr"r!s r/r"r"s48).J  # r1rj)NNNN)DrrttypingrrrrrrrH"torch.distributed.tensor._dispatch distributedr _dispatchr torch.distributed.tensor._random_randomrtorch.nnrtorch.distributed.device_meshr r *torch.distributed.tensor._collective_utilsr r &torch.distributed.tensor._dtensor_specr rrr&torch.distributed.tensor._redistributerrtorch.distributed.tensor._utilsrrr(torch.distributed.tensor.placement_typesrrrr__all__opsatenautogradFunctionr$rLrIrrrrrrVrrr7rsrkrrrr r!r"rJr1r/r6sAA 8811EXJ   yy~~.- U^^,,- `]Du~~..]D@U;ellU;t )-04J LLJ*%J),-J J^)-KOGKHL H IIH*%H8S"))Z$@$$FGHHxC