sg vddlZddlZddlmZmZmZmZddlmZddl Z ddl m Z m Z ddl mZmZmZgdZdedeefd Zded eedeefd Zd edefd Ze j.fd edededefdZd edefdZ dd edeedefdZeded ededefdZd edefdZd ede j<defdZy)N)AnyListOptionalSet) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturnc*t|tSN)_get_all_graph_pipes_helperset)rs R/var/www/html/venv/lib/python3.12/site-packages/torch/utils/data/graph_settings.pyrrs &uce 44id_cachecg}|jD]J\}\}}||vr|j||j||jt ||L|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphs rrrsh!G(- I$$) H   Ux 29hGH I Nrr"ct|txs-t|dxrtj|j S)Nr) isinstancerhasattrinspectismethodrr"s r_is_sharding_datapiper*)s; h 5 6 *+ 6   X44 5rnum_of_instances instance_idcFt|}dfd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. cP|jD]\}}d}t|rs|td|d|tj|j }t |jdkr|j n|j  |}||}||y)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr* RuntimeErrorr' signaturerlen parameters) r prev_applieddpr#appliedsig_helperr,r+r0s rr:zapply_sharding.._helper=s"\\^ (MB G$R(+&77CnD^_a^bd ''(9(9:s~~&*%%&6 D%%(+n&& Iw '' (rr)r )r"r+r,r0rr:s ``` @rrr0s'  "E((, EN Orct|dxrNt|dxr@tj|jxrtj|jS)N set_shuffleset_seed)r&r'r(r<r=r)s r_is_shuffle_datapiper>XsT-( 0 Hj ) 0   X11 2 0   X.. / rshufflec||St|}t|}|Dcgc]}t|s|}}|s*|r(tjd|j }|g}|D]}|j ||Scc}w)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)r rr>warningswarnr?r<)r"r?r all_pipespipe shufflersshufflers rrras  "E#E*I"+J$/CD/IJIJ   [ ##%  &W%& OKs A<A<z`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngct||Sr)r )r"rHs rrrs Xs ++rc\t|dxrtj|jS)Nr=)r&r'r(r=r)s r_is_random_datapiperKs% 8Z ( PW-=-=h>O>O-PPrct|}t|}t}g}|D]G}t||vrt |s|j ||j t|I|D]_}ttjdtjj|j}|j|a|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds )dtype) generator)r rridrKrrinttorchemptyint64random_itemr=)r"rHrrCcacherandom_datapipesrD random_seeds rr r s  "E#E*I EE  d8u   t $  # #D ) IIbh   !# KK%++ . 6 6 6 E J J L  k" # Orr) r'rAtypingrrrrtyping_extensionsrrR(torch.utils.data.datapipes.iter.shardingrr torch.utils.data.graphr r r __all__rrQrboolr*DEFAULTrr>r FutureWarningrrK Generatorr rMrrrcsT++( IH 5}5h5   $'H  (^ H'.. %%%%  %P837  !)$   F .  ,,,,  ,Q(QtQu8r