sg jUdZddlZddlmZmZmZddlZgZeee d<dejde fdZ dejd e dejfd Zd e d eedefd ZgdZdee ej$fde fdZ ddejdejdejdeejfdZy)zCDefines utilities for interacting with scaled_dot_product_attentionN)ListOptionalUnion__all__tensorsreturnc&td|DS)z0Returns True if any of the tensors requires gradc34K|]}|jyw)N) requires_grad).0ts L/var/www/html/venv/lib/python3.12/site-packages/torch/nn/attention/_utils.py z'_input_requires_grad..s01q0s)any)rs r_input_requires_gradr s 00 00 inpt_tensorog_sizec@|jd|k7r |dd|fS|S)z'Handles the unpad of the last dimension.N)size)rrs r_postprocess_flash_outputrs.w&3=)) r head_dim_sizescalec:||Sdtj|z S)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. g?)mathsqrt)rrs r_calculate_scalers#   =) ))r)  @iinc|tvS)z:Returns true if the head dim is supported by FlexAttention)_SUPPORTED_HEAD_DIMS)r's r_supported_head_dimr*%s $ $$rquerykeyvalue attn_maskc |j|jk7s|j|jk7r3td|jd|jd|jd|j|jk7s|j|jk7r3td|jd|jd|jd|jdks&|jdks|jdkr?td |jd |jd |jdy) NzLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: z instead.zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: rzUExpected query, key, and value to all be at least 2 dimensional, but got query.dim: z , key.dim: z and value.dim: )dtype ValueErrordevicedim)r+r,r-r. dropout_p is_causalrs r_validate_sdpa_inputr6*s" {{cii5;;%++#=$$)KK= cii[I % }I 7   ||szz!U\\U\\%A%%*\\N. M!!&i 9   yy{Q#'')a-599;?cyy{m;swwyk1A%))+i Y  ,;r)NgFN)__doc__rtypingrrrtorchrstr__annotations__Tensorboolrintrfloatrr)SymIntr*r6rrrBsI (( c15<<1D1 5<<#%,,*C**E*B%5ell!23%%)-  <<   <<  %  r