sg@)`ddlmZmZmZmZmZmZmZddlZddl m Z ddlm Z ddl m Z e d ddddd d d eeee feeee ffd eeefd eeeefdedef dZe ddee j"deeeefeeefffdZdeee dfee fdede fdZy))AnyDictListOptionalSequenceTupleUnionN)Tensor) exposed_inz torch.funcTF tie_weightsstrictmoduleztorch.nn.Moduleparameter_and_buffer_dictsargskwargsr rct|tr|}nt|trtd|Ds t d|Dcgc]}|j D]}|} }}i} | D]}| j |d} | dz| |<| jD cgc] \} } | dkDs | }} } t|dkDrt |d|D cic]}|jD]\}} ||  }}}} nt dt|tjjj||||||Scc}}wcc} } wcc} }}w)aPerforms a functional call on the module by replacing the module parameters and buffers with the provided ones. .. note:: If the module has active parametrizations, passing a value in the :attr:`parameter_and_buffer_dicts` argument with the name set to the regular parameter name will completely disable the parametrization. If you want to apply the parametrization function to the value passed please set the key as ``{submodule_name}.parametrizations.{parameter_name}.original``. .. note:: If the module performs in-place operations on parameters/buffers, these will be reflected in the ``parameter_and_buffer_dicts`` input. Example:: >>> a = {'foo': torch.zeros(())} >>> # xdoctest: +SKIP >>> mod = Foo() # does self.foo = self.foo + 1 >>> print(mod.foo) # tensor(0.) >>> functional_call(mod, a, torch.ones(())) >>> print(mod.foo) # tensor(0.) >>> print(a['foo']) # tensor(1.) .. note:: If the module has tied weights, whether or not functional_call respects the tying is determined by the tie_weights flag. Example:: >>> a = {'foo': torch.zeros(())} >>> # xdoctest: +SKIP >>> mod = Foo() # has both self.foo and self.foo_tied which are tied. Returns x + self.foo + self.foo_tied >>> print(mod.foo) # tensor(1.) >>> mod(torch.zeros(())) # tensor(2.) >>> functional_call(mod, a, torch.zeros(())) # tensor(0.) since it will change self.foo_tied too >>> functional_call(mod, a, torch.zeros(()), tie_weights=False) # tensor(1.)--self.foo_tied is not updated >>> new_a = {'foo': torch.zeros(()), 'foo_tied': torch.zeros(())} >>> functional_call(mod, new_a, torch.zeros()) # tensor(0.) An example of passing multiple dictionaries .. code-block:: python a = ({'weight': torch.ones(1, 1)}, {'buffer': torch.zeros(1)}) # two separate dictionaries mod = nn.Bar(1, 1) # return self.weight @ x + self.buffer print(mod.weight) # tensor(...) print(mod.buffer) # tensor(...) x = torch.randn((1, 1)) print(x) functional_call(mod, a, x) # same as x print(mod.weight) # same as before functional_call And here is an example of applying the grad transform over the parameters of a model. .. code-block:: python import torch import torch.nn as nn from torch.func import functional_call, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) def compute_loss(params, x, t): y = functional_call(model, params, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(dict(model.named_parameters()), x, t) .. note:: If the user does not need grad tracking outside of grad transforms, they can detach all of the parameters for better performance and memory usage Example:: >>> detached_params = {k: v.detach() for k, v in model.named_parameters()} >>> grad_weights = grad(compute_loss)(detached_params, x, t) >>> grad_weights.grad_fn # None--it's not tracking gradients outside of grad This means that the user cannot call ``grad_weight.backward()``. However, if they don't need autograd tracking outside of the transforms, this will result in less memory usage and faster speeds. Args: module (torch.nn.Module): the module to call parameters_and_buffer_dicts (Dict[str, Tensor] or tuple of Dict[str, Tensor]): the parameters that will be used in the module call. If given a tuple of dictionaries, they must have distinct keys so that all dictionaries can be used together args (Any or tuple): arguments to be passed to the module call. If not a tuple, considered a single argument. kwargs (dict): keyword arguments to be passed to the module call tie_weights (bool, optional): If True, then parameters and buffers tied in the original model will be treated as tied in the reparameterized version. Therefore, if True and different values are passed for the tied parameters and buffers, it will error. If False, it will not respect the originally tied parameters and buffers unless the values passed for both weights are the same. Default: True. strict (bool, optional): If True, then the parameters and buffers passed in must match the parameters and buffers in the original module. Therefore, if True and there are any missing or unexpected keys, it will error. Default: False. Returns: Any: the result of calling ``module``. c3<K|]}t|tywN) isinstancedict).0ds S/var/www/html/venv/lib/python3.12/site-packages/torch/_functorch/functional_call.py z"functional_call..}sK1:a&KszFExpected all elements of parameter_and_buffer_dicts to be dictionariesrzL appeared in multiple dictionaries; behavior of functional call is ambiguouszTExpected parameter_and_buffer_dicts to be a dict, or a list/tuple of dicts, but got r )rrrall ValueErrorkeysgetitemslentypennutils stateless_functional_call)rrrrr rparameters_and_buffersrkall_keysall_keys_countervkeyn repeated_keyss rfunctional_callr0 s^,d3!; . 9K0JKKX  :L!1668LaALALL+- (A $$Q*A"#a% Q  (,<+A+A+CMaq1uM M }  ! /!mn 5" " " 9=AAqD" " " 678 :  88   . .  / 'M N " sE E+E"E modelsreturnc xt|dk(r tdtd|Dstd|Ds tdt|dtfd|Ds td|Dcgc]}t |j }}|dDcic]!t tfd|D#}}|Dcgc]}t |j}}|dDcic]!t tfd |D#}}||fScc}wcc}wcc}wcc}w) a0stack_module_state(models) -> params, buffers Prepares a list of torch.nn.Modules for ensembling with :func:`vmap`. Given a list of ``M`` ``nn.Modules`` of the same class, returns two dictionaries that stack all of their parameters and buffers together, indexed by name. The stacked parameters are optimizable (i.e. they are new leaf nodes in the autograd history that are unrelated to the original parameters and can be passed directly to an optimizer). Here's an example of how to ensemble over a very simple model: .. code-block:: python num_models = 5 batch_size = 64 in_features, out_features = 3, 3 models = [torch.nn.Linear(in_features, out_features) for i in range(num_models)] data = torch.randn(batch_size, 3) def wrapper(params, buffers, data): return torch.func.functional_call(models[0], (params, buffers), data) params, buffers = stack_module_state(models) output = vmap(wrapper, (0, 0, None))(params, buffers, data) assert output.shape == (num_models, batch_size, out_features) When there's submodules, this follows state dict naming conventions .. code-block:: python import torch.nn as nn class Foo(nn.Module): def __init__(self, in_features, out_features): super().__init__() hidden = 4 self.l1 = nn.Linear(in_features, hidden) self.l2 = nn.Linear(hidden, out_features) def forward(self, x): return self.l2(self.l1(x)) num_models = 5 in_features, out_features = 3, 3 models = [Foo(in_features, out_features) for i in range(num_models)] params, buffers = stack_module_state(models) print(list(params.keys())) # "l1.weight", "l1.bias", "l2.weight", "l2.bias" .. warning:: All of the modules being stacked together must be the same (except for the values of their parameters/buffers). For example, they should be in the same mode (training vs eval). rz7stack_module_state: Expected at least one model, got 0.c34K|]}|jywrtrainingrms rrz%stack_module_state..s+q +c36K|]}|j ywrr5r7s rrz%stack_module_state..s2Raqzz>2RzLstack_module_state: Expected all models to have the same training/eval mode.c3:K|]}t|k(ywr)r#)rr8 model0_typs rrz%stack_module_state..s5tAw*$5sz@stack_module_state: Expected all models to be of the same class.c3(K|] }| ywr)rparamsr)s rrz%stack_module_state..s'Kfq 'Kc3(K|] }| ywrr?)rbuffersr)s rrz%stack_module_state..s'Nw 'NrA) r" RuntimeErrorrr#rnamed_parametersconstruct_stacked_leaftuple named_buffers)r1model all_paramsr)r@ all_buffersrCr=s ` @rstack_module_staterLsWt 6{aTUU +F+ +s2R62R/R Z  fQiJ 5f5 5 N  ?EEU$u--/0EJEA  !%'K 'K"KQ OOF=CC54++-.CKCQ  !%'N+'N"NPQ RRG 7?FDs< D(&&D- D2<&D7tensors.namectd|D}td|D}|s|std|dtj|}|r|j j }|S)Nc34K|]}|jywr requires_gradrts rrz)construct_stacked_leaf..s=AOO=r9c36K|]}|j ywrrQrSs rrz)construct_stacked_leaf..sBQ0Br;z Expected z0 from each model to have the same .requires_grad)rrDtorchstackdetachrequires_grad_)rMrNall_requires_gradnone_requires_gradresults rrFrFsr=W==B'BB %7vM N  [[ !F//1 Mr)typingrrrrrrr rVtorch.nnr$r torch._functorch.utilsr strboolr0ModulerLrFr?r]rrdsEEDD - L (, O O O %d3;&7$sF{BS9T&T UO U O T#s(^ $ O  O OOd LO OO 4S>4S> )*OOd  5%tF|3 4