sgp ddlZddlZddlZddlmZmZddlmZddlm Z ddl m cm Z ddl m Z ddlmZmZe j"s&edej&ej(dedzZ dd ed efd Zdd Zddeded efdZdded dfdZd eeeffdZ ddedededed df dZdeded dfdZded efdZ dZ!y)N)partialwraps)Tuple_rref_context_get_debug_info) FILE_SCHEMATEST_WITH_TSANz"c10d not available, skipping tests)filez {file_name} setup_rpcclean_shutdowncjttStfd}|S)ak We use this decorator for setting up and tearing down state since MultiProcessTestCase runs each `test*` method in a separate process and each process just runs the `test*` method without actually calling 'setUp' and 'tearDown' methods of unittest. Note: pass the string representation of MessageTypes that should be used with the faulty agent's send function. By default, all retriable messages ("RREF_FORK_REQUEST", "RREF_CHILD_ACCEPT", "RREF_USER_DELETE", "CLEANUP_AUTOGRAD_CONTEXT_REQ") will use the faulty send (this default is set from faulty_rpc_agent_test_fixture.py). )r r faulty_messagesmessages_to_delaycddlmcmcm}d|_|j |_|j|j} rtr7tjjdz|_ dtj_ tjd|j z|j|j |j | |g|i|} rtj"|S)NrF<zworker%d)namebackendrank world_sizerpc_backend_options)graceful)torch.distributed.rpc.api distributedrpcapi_ignore_rref_leakr worker_idsetup_fault_injectionrr constantsDEFAULT_RPC_TIMEOUT_SEC rpc_timeoutDEFAULT_SHUTDOWN_TIMEOUTinit_rpc rpc_backendrshutdown) selfargkwargsrr return_valuer rrold_test_methodr s U/var/www/html/venv/lib/python3.12/site-packages/torch/testing/_internal/dist_utils.pynew_test_methodz"dist_init..new_test_method8s 0/ % ""?4EF"66 25--2W2WZ[2[#/9; 6 LL$))+((YY??$7  't r.returncy)Nr2r.r,noopr3[sr.rexpected_error_regexc tjd|tdtjd6#t $r:}t j|t|rt|cYd}~SYd}~?d}~wwxYw)aC Loops until an RPC to the given rank fails. This is used to indicate that the node has failed in unit tests. Args: rank (int): Rank of the node expected to fail expected_error_regex (optional, str): Regex of exception message expected. Useful to ensure a specific failure occurs, not just any. workerr2)args皙?)patternstringN) rrpc_syncr3timesleep Exceptionresearchstr)rr4es r,wait_until_node_failurerC_sf   LL6$$R 8 JJsO  yy!5c!fE1v F s48 A;*A6+A;6A;timeoutctj} t}t|d}t|d}|dk(r|dk(rytjdtj|z |kDrt d|d|ds) a2 The RRef protocol holds forkIds of rrefs in a map until those forks are confirmed by the owner. The message confirming the fork may arrive after our tests check whether this map is empty, which leads to failures and flaky tests. to_here also does not guarantee that we have finished processind the owner's confirmation message for the RRef. This function loops until the map is empty, which means the messages have been received as processed. Call this function before asserting the map returned by _get_debug_info is empty. num_pending_futuresnum_pending_usersrr8z:Timed out waiting to flush pending futures and users, had z pending futures and z pending usersN)r<rintr= ValueError)rDstart debug_inforFrGs r,,wait_until_pending_futures_and_users_flushedrLqs IIKE 13 !*-B"CD +> ?@ ! #(9Q(>  3 99;  (*++@AR@SSac  r.c2t}|d}|d}||fS)zf Retrieves number of OwnerRRefs and forks on this node from _rref_context_get_debug_info. num_owner_rrefs num_forksr) rref_dbg_info num_ownersrOs r,get_num_owners_and_forksrRs, 12M01Jk*I y  r.rQrOc Vtj} tjt|tdd\}}t |}t |}||k(r||k(rytj dtj|z |kDrtd|d|d|d |d|d ) z Waits until timeout for num_forks and num_owners to exist on the rank. Used to ensure proper deletion of RRefs in tests. r2r)r7rDNzTimed out waiting z sec for z owners and z forks on rank, had z forks)r<rr; worker_namerRrHr=rI)rQrOrrDrJnum_owners_on_ranknum_forks_on_ranks r,#wait_until_owners_and_forks_on_rankrXs IIKE 03  7b!1 --!!34 12  +0AY0N  1 99;  ($WIYzl,ykZ*+<8I7J&R  r.rc`tjstjd|||yy)Ngloo)r init_methodrr)distis_initializedinit_process_group)r[rrs r, initialize_pgr_s-     #!   !r.c d|S)Nr6r2)rs r,rUrUs D6?r.cT|Dcgc]}||jvs|c}d}|Scc}w)aQ Returns the first event that matches partial_event_name in the provided function_events. These function_events should be the output of torch.autograd.profiler.function_events(). Args: function_events: function_events returned by the profiler. event_name (str): partial key that the event was profiled with. r)r)function_eventspartial_event_nameevents r,get_function_eventres1!0 Tu3E3SU TUV WE L Us%%)NTTNN)r0N)z.*))"r?sysr< functoolsrrtypingrtorch.distributedrr\torch.distributed.rpcrr$torch.testing._internal.common_utilsrr is_availableprintstderrexitINIT_METHOD_TEMPLATEboolr/r3rHrArCrLrRrXr_rUrer2r.r,rss6 $ ##>Lt .SZZ@ CHHQK#]2 BBBJ #SC$#t4!%S/!@B #+.9< 0 S c d cc r.