sg1ddlZddlZddlZddlmZmZddlmZmZm Z ddl m Z ddl Z ddl mZddlmZddlmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)gdZ*e de+ d$dede#deejXde-de.deedefdZ/ede'ddddddede e0ejbdfdee#deedeejXdef d Z2edddddddede e0ejbdfdee#deedeejXdef d!Z3dedefd"Z4 d$dede#deejXde-de.deedefd#Z5y)%N)FutureThreadPoolExecutor)castOptionalUnion) deprecated)_offload_state_dict_to_cpu)_storage_setup)DefaultSavePlanner)_dcp_method_logger)MetadataSTATE_DICT_TYPE)SavePlan SavePlanner) AsyncStager)Stateful) StorageWriter)_get_default_group) _api_bc_check _DistWrapper_profile)save_state_dictsave async_saveza`save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.)category state_dictstorage_writer process_groupcoordinator_rankno_distplannerreturnc |jt5t||||||cdddS#1swYyxYw)z3This method is deprecated. Please switch to 'save'.N)resetr_save_state_dict)rrrr r!r"s `/var/www/html/venv/lib/python3.12/site-packages/torch/distributed/checkpoint/state_dict_saver.pyrrsF             s5>T)log_exceptions checkpoint_idrr"rr*c ntjjdtjxrtj }|rt jdt5ttt||d}tt|||||cdddS#1swYyxYw)a Save a distributed model in SPMD style. This function is different from ``torch.save()`` as it handles ``ShardedTensor`` , and ``DTensor`` by having each rank only save their local shards. For each ``Stateful`` object (having both a ``state_dict`` and a ``load_state_dict``), save will call ``state_dict`` before serialization. .. warning:: There is no guarantees of Backwards Compatibility across PyTorch versions for saved state_dicts. .. warning:: If using the `process_group` argument, make sure that only its ranks call `save_state_dict` and that all data in state_dict belong to it. .. note:: When saving checkpoint for FSDP's `ShardingStrategy.HYBRID_SHARD`, only one of the shard_group should be calling `save_state_dict` and the corresponding process group needs to be passed in. .. note:: If no process group is available, this function assumes the intention is to save the state_dict in the local process. .. note: Rank 0 is assumed to be the coordinator rank. Args: state_dict (Dict[str, Any]): The state_dict to save. checkpoint_id (Union[str, os.PathLike, None]): The ID of this checkpoint instance. The meaning of the checkpoint_id depends on the storage. It can be a path to a folder or to a file. It can also be a key if the storage is a key-value store. (Default: ``None``) storage_writer (Optional[StorageWriter]): Instance of StorageWriter used to perform writes. If this is not specified, DCP will automatically infer the writer based on the checkpoint_id. If checkpoint_id is also None, an exception will be raised. (Default: ``None``) planner (Optional[SavePlanner]): Instance of SavePlanner. If this is not specificed, the default planner will be used. (Default: ``None``) process_group (Optional[ProcessGroup]): ProcessGroup to be used for cross-rank synchronization. (Default: ``None``) Returns: Metadata: Metadata object for the saved checkpoint. Example: >>> # xdoctest: +SKIP >>> my_model = MyModule() >>> state_dict = {"model": my_model} >>> fs_storage_writer = torch.distributed.checkpoint.FileSystemWriter("/checkpoint/1") >>> torch.distributed.checkpoint.save( >>> state_dict=state_dict, >>> storage_writer=fs_storage_writer, >>> ) .. note:: save_state_dict uses collectives to coordinate writes across ranks. For NCCL-based process groups, internal tensor representations of objects must be moved to the GPU device before communication takes place. In this case, the device used is given by ``torch.cuda.current_device()`` and it is the user's responsibility to ensure that this is set so that each rank has an individual GPU, via ``torch.cuda.set_device()``. z!torch.distributed.checkpoint.savezftorch.distributed is unavailable or uninitialized, assuming the intent is to save in a single process.Freader)rrrr!r"N)torch_C_log_api_usage_oncedist is_availableis_initializedwarningswarnrrrr r&_stateful_to_state_dict)rr*rr"rr!s r'rr9sd HH  !DE$$&@4+>+>+@AG t     >.-PUV  .z:)'     s ,5B++B4c|tjjdtjrJtj r6|xs t }tjd|jvsJdttt||d}t|}t|tr|j|}n t!|d}t#dj%t&||||| }|j)fd t|tr|j*r|j-|S) aaAsynchronous version of ``save``. This code first de-stages the state_dict on to the staging storage (defaults to CPU memory), and then calls the `save` in a separate thread. .. warning:: This feature is experimental and subject to change. Args: state_dict (Dict[str, Any]): The state_dict to save. checkpoint_id (Union[str, os.PathLike, None]): The ID of this checkpoint instance. The meaning of the checkpoint_id depends on the storage. It can be a path to a folder or to a file. It can also be a key if the storage is a key-value store. (Default: ``None``) storage_writer (Optional[StorageWriter]): Instance of StorageWriter used to perform 'stage' and 'save'. If this is not specified, DCP will automatically infer the writer based on the checkpoint_id. If checkpoint_id is also None, an exception will be raised. (Default: ``None``) planner (Optional[SavePlanner]): Instance of SavePlanner. If this is not specificed, the default planner will be used. (Default: ``None``) process_group (Optional[ProcessGroup]): ProcessGroup to be used for cross-rank synchronization. (Default: ``None``) Returns: Future: A future holding the resultant Metadata object from `save`. Example: >>> # xdoctest: +SKIP >>> my_model = MyModule() >>> state_dict = {"model": my_model} >>> fs_storage_writer = torch.distributed.checkpoint.FileSystemWriter("/checkpoint/1") >>> checkpoint_future = torch.distributed.checkpoint.async_save( >>> state_dict=state_dict, >>> storage_writer=fs_storage_writer, >>> ) >>> >>> # ... do some work ... >>> >>> checkpoint_future.result() z'torch.distributed.checkpoint.async_savecpuzfA CPU backend must be enabled for async save; try initializing process group with 'cpu:gloo,cuda:nccl'Fr,) type_checkr) max_workersr)c(jdS)NF)wait)shutdown)fexecutors r'zasync_save..s("3"3"3"?)r.r/r0r1r2r3rdevice _device_typesrrr r6 isinstancerstager rsubmitradd_done_callback should_synchronize_after_executesynchronize_staging) rr*rr"rpgstaged_state_dictr>r?s @r'rrsl HH  !JK t224  202 LL 2#3#3 3 t s t 3~nmERN)4J.+.*00<6zeT!a0H #%#  A?@ >;/  ; ;**, HrAci}|jD]*\}}t|tr|jn|||<,|S)z]Creates a shallow copy of `state_dict` where `state_dict` is called for each Stateful object.)itemsrDrr)rstateful_state_dictkeyelems r'r6r6sK%%' T!+D(!;DOO  C   rAc tjjdt|| |  t Jdi}t ddx}||d<t d i| fd}t d i|fd} jd||  t d i| fd} t d i|fd} jd| | S) Nz,torch.distributed.checkpoint.save_state_dictr*cJj}dtjjjvr2t j djjnj|jjjj}j|}|S)N storage_metazThe function definition for SavePlanner.set_up_planner has been updated to include the storage_meta argument. Please update your implementation to include this parameter.)rrSis_coordinator) rSinspect signatureset_up_planner parametersr4r5rTset_up_storage_writercreate_local_planprepare_local_plan)rS local_plandistWr"rrs r' local_stepz$_save_state_dict..local_steps"""%224 !2!273I3I!J!U!U U MM.   " ":u/C/C D  " "%)$33 #  ,,U-A-AB..0 #66zB rAcZJj|\}j|}|SN)create_global_planprepare_global_plan)all_local_plansglobal_metadatar"rs r' global_stepz%_save_state_dict..global_step3s<"""+2+E+Eo+V((<<_MrAplancJj}j|}|j|jSr`) finish_plan write_datar<value)final_local_plan all_writes central_planr"rs r'riz$_save_state_dict..write_data>sL""""..|<#../?I !!rAc6Jj|S)N)metadataresults)finish) all_resultsrdrs r'finish_checkpointz+_save_state_dict..finish_checkpointGs(*** LrAwrite) r.r/r0rr getattrr reduce_scatter all_reduce)rrrr r!r" ckpt_kwargsckpt_idr^rerirsrmr]rds`` ` @@@r'r&r&s HH  !OP G 5E FE$&   OK>?DAAN'. O$&+&',&+&'#11&*kRL&+&"'"&+&'   GZ1B CCrA)NrFN)6rUosr4concurrent.futuresrrtypingrrrtyping_extensionsrr.torch.distributed distributedr1#torch.distributed._state_dict_utilsr +torch.distributed.checkpoint._storage_utilsr ,torch.distributed.checkpoint.default_plannerr #torch.distributed.checkpoint.loggerr %torch.distributed.checkpoint.metadatar r$torch.distributed.checkpoint.plannerrr$torch.distributed.checkpoint.stagingr%torch.distributed.checkpoint.statefulr$torch.distributed.checkpoint.storager"torch.distributed.distributed_c10drutilsrrr__all__ FutureWarning ProcessGroupintboolrstrPathLikerrr6r&rurAr'rs| 9((( JFKBKF<:>A88 4 ! 26%)   ! D--.     k "     .4(48.2%)15 c c bkk4/0c ]+ c k " c D--. c c )c L4(48.2%)15 X X bkk4/0X ]+ X k " X D--. X  X )X vO26%) FDFD!FDD--.FD FD  FD k " FDFDrA