sg8dZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZmZmZmZmZd d lmZej@e!Z"dZ#dZ$Gdde jJZ&Gdde jJZ'Gdde jJZ(Gdde(Z)Gdde jJZ*Gdde jJZ+Gdde+Z,Gdde jJZ-Gd d!e jJZ.e+e,d"Z/Gd#d$e jJZ0Gd%d&e jJZ1Gd'd(e jJZ2Gd)d*eZ3d+Z4d,Z5ed-e4Gd.d/e3Z6ed0e4Gd1d2e3Z7y)3zPyTorch ViViT model.N)OptionalSetTupleUnion)nn)CrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) VivitConfigzgoogle/vivit-b-16x2-kinetics400rc0eZdZdZfdZddefdZxZS)VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). ct||j|_|j|_|j|_|j|j dz|j|j dzz|j|j dzz|_|j|_tj|j|j|j|j|_ y)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfconfig __class__s [/var/www/html/venv/lib/python3.12/site-packages/transformers/models/vivit/modeling_vivit.pyr zVivitTubeletEmbeddings.__init__8s  ++ ++ -- __ 2 2$//!"44 6$//!"44 6   ++))   !3!3ATAT]c]p]p interpolate_pos_encodingc \|j\}}}}}|sP||jk7s||jk7r2td|d|d|jdd|jdd |jddddd }|j |}|j dj dd}|S) NzImage image size (*z) doesn't match model (rrz).rr )shaper" ValueErrorpermuter*flatten transpose) r, pixel_valuesr1 batch_sizer!r)heightwidthxs r/forwardzVivitTubeletEmbeddings.forwardHs>J>P>P; J fe'Vt-F%SWSbSbJb$VHAeW5__Q'($//!*<)=RA  $++Aq!Q: OOL ) IIaL " "1a (r0F)__name__ __module__ __qualname____doc__r boolr? __classcell__r.s@r/rr-s dr0rcpeZdZdZfdZdej dededej fdZd de fd Z xZ S) VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. ct|tjt j dd|j |_t||_ tjt j d|jjdz|j |_ tj|j|_|jdd|_||_y)Nr)rr r Parametertorchzerosr& cls_tokenrpatch_embeddingsr%position_embeddingsDropouthidden_dropout_probdropoutr#r$r-r+s r/r zVivitEmbeddings.__init__as ekk!Q8J8J&KL 6v >#%<< KK400<J>P>P; J fe**&>>J\\*- r0r@) rArBrCrDr rLTensorintr1rEr?rFrGs@r/rIrIZsL &D5<<&D&DUX&D]b]i]i&DPdr0rIc eZdZdeddffd Zdej dej fdZ d deej de de e ej ej fe ej ffd Z xZ S) VivitSelfAttentionr-rUNct||j|jzdk7r3t |ds't d|jfd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)rr r&num_attention_headshasattrr6rpattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluerQattention_probs_dropout_probrSr+s r/r zVivitSelfAttention.__init__s1     : : :a ?PVXhHi"6#5#5#6"78334A7  $*#=#= #&v'9'9Fc|jdd|j|jfz}|j|}|j ddddS)NrWrrrr )rYrwryrcr7)r,r> new_x_shapes r/transpose_for_scoresz'VivitSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r0 head_maskoutput_attentionsc|j|}|j|j|}|j|j|}|j|}t j ||j dd}|tj|jz }tjj|d} |j| } || |z} t j | |} | jddddj} | j!dd|j"fz} | j%| } |r| | f} | S| f} | S)NrWr\rrrr )r}rr~rrLmatmulr9mathsqrtryrrasoftmaxrSr7 contiguousrYrzrc) r, hidden_statesrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r/r?zVivitSelfAttention.forwardsT!JJ}5--dhh}.EF // =0IJ //0AB !<< Y5H5HR5PQ+dii8P8P.QQ--//0@b/I,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**+BC 6G=/2O\M]r0NF)rArBrCrr rLrorrrErrr?rFrGs@r/rrrrsG{GtG$%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!r0rrc eZdZdeddffd Z ddeejdede e ejejfe ejfffd Z xZ S) VivitSdpaSelfAttentionr-rUNcFt|||j|_yN)rr rr+s r/r zVivitSdpaSelfAttention.__init__s  ,2,O,O)r0rrc \|s|&tjdt | |||S|j |}|j |j |}|j |j|}|j |}tjjj|||||jr |jnddd}|jddddj}|j!dd |j"fz} |j%| }|dfS) NaVivitSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.F) is_causalscalerrrr r)logger warning_oncerr?r}rr~rrLrrascaled_dot_product_attentiontrainingrr7rrYrzrc) r,rrrrrrrrrr.s r/r?zVivitSdpaSelfAttention.forwards5  5   d  7?!  !JJ}5--dhh}.EF // =0IJ //0AB ++HH    15D - -CI &--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**+BC d""r0r) rArBrCrr rrLrorErrr?rFrGs@r/rrspP{PtP bg$#(0(>$#Z^$# uU\\5<</0% 2EE F$#$#r0rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r-rUNct|tj|j|j|_tj |j|_yr) rr rr{r&denserQrRrSr+s r/r zVivitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r0r input_tensorcJ|j|}|j|}|SrrrSr,rrs r/r?zVivitSelfOutput.forwards$ =1  ]3 r0) rArBrCrDrr rLror?rFrGs@r/rrsD >{>t> U\\RWR^R^r0rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) VivitAttentionr-rUNct|t||_t ||_t |_yr)rr rr attentionroutputset pruned_headsr+s r/r zVivitAttention.__init__(s0 +F3%f- Er0headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr\)lenrrrwryrrr}r~rrrrzunion)r,rindexs r/ prune_headszVivitAttention.prune_heads.s u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r0rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r,rrr self_outputsattention_outputrs r/r?zVivitAttention.forward@sE ~~mY@QR ;;|A F#% QR(88r0r)rArBrCrr rrprrLrorrErrr?rFrGs@r/rr's"{"t" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r0rc(eZdZdeddffd ZxZS)VivitSdpaAttentionr-rUNcDt||t||_yr)rr rrr+s r/r zVivitSdpaAttention.__init__Ps  /7r0)rArBrCrr rFrGs@r/rrOs8{8t88r0rc$eZdZfdZdZxZS)VivitIntermediatecPt|tj|j|j |_tj|j|_ t|jtrt|j|_y|j|_yr)rr rr{r&intermediate_sizerrQrRrS isinstance hidden_actstrr intermediate_act_fnr+s r/r zVivitIntermediate.__init__Vss YYv1163K3KL zz&"<"<= f'' -'-f.?.?'@D $'-'8'8D $r0cl|j|}|j|}|j|}|Sr)rrrS)r,rs r/r?zVivitIntermediate.forward_s4 =1 00?  ]3 r0rArBrCr r?rFrGs@r/rrUs 9r0rc$eZdZfdZdZxZS) VivitOutputct|tj|j|j |_tj|j|_ yr) rr rr{rr&rrQrRrSr+s r/r zVivitOutput.__init__hsB YYv779K9KL zz&"<"<= r0cT|j|}|j|}||z}|Srrrs r/r?zVivitOutput.forwardms. =1  ]3 % 4 r0rrGs@r/rrgs > r0r)eagersdpac*eZdZdZfdZddZxZS) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.ct||j|_d|_t |j ||_t||_t||_ tj|j|j|_tj|j|j|_y)Nreps)rr chunk_size_feed_forward seq_len_dimVIVIT_ATTENTION_CLASSES_attn_implementationrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterr+s r/r zVivitLayer.__init__s '-'E'E$01L1LMfU-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr0c|j|j|||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|S)N)rrr)rrrrr)r,rrrself_attention_outputsrr layer_outputs r/r?zVivitLayer.forwards!%  ! !- 0 / "0"  2!4(,)=8 ++M: ((6 {{<? /G+r0r)rArBrCrDr r?rFrGs@r/rr}sX[r0rc.eZdZfdZ ddZxZS) VivitEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rr r-r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r,r-_r.s r/r zVivitEncoder.__init__sN  ]]fF^F^@_#`1Jv$6#`a &+#$asA#ct|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr+|jr|j | j || |} n | || |} | d}|s`|| dfz}j|r||fz}|st d|||fDSt|||S)Nrrc3&K|] }|| ywrr).0vs r/ z'VivitEncoder.forward..smq_`_lms)last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler ) r,rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r/r?zVivitEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]OM^ _ )!,M &9]1=M fryy"))4 5 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  - KK  Sdkk.K.K  L.r0N) rArBrCrDr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdparrr0r/rrs0 L$O&*#NMr0raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`VivitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`VivitImageProcessor`]. See [`VivitImageProcessor.preprocess`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare ViViT Transformer model outputting raw hidden-states without any specific head on top.ceZdZd fd ZdZdZeeee e  dde e jde e jde ede ed ed e ed eee je ffd ZxZS) VivitModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jy)Nr)rr r-rIrTrencoderrrr&r layernormrpooler post_init)r,r-add_pooling_layerr.s r/r zVivitModel.__init__/si   )&1#F+ f&8&8f>S>ST->k&)D  r0c.|jjSr)rTrO)r,s r/get_input_embeddingszVivitModel.get_input_embeddings<s///r0c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrr)r,heads_to_prunerrs r/ _prune_headszVivitModel._prune_heads?sE+002 CLE5 LL  u % / / ; ;E B Cr0 output_typerr:rrrr1rrUc||n|jj}||n|jj}||n|jj}| t d|j ||jj }|j||}|j|||||}|d} |j| } |j|j| nd} |s | | f|ddzSt| | |j|jS)a Returns: Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrl)rrrrrr)r pooler_outputrr)r-rruse_return_dictr6 get_head_maskrrTrrr r rr) r,r:rrrr1rembedding_outputencoder_outputssequence_outputrs r/r?zVivitModel.forwardJs*n2C1N-TXT_T_TqTq$8$D $++JjJj &1% u 'tn u #' u d^u  uU&&')CC Du eCu r0ra ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c eZdZfdZeeeee d de e jde e jde e jde e de e de d e e d eee jeffd ZxZS) VivitForVideoClassificationc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jy)NF)r"r) rr num_labelsrrrr{r&Identity classifierr!r+s r/r z$VivitForVideoClassification.__init__ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r0r)r:rlabelsrrr1rrUc||n|jj}|j||||||}|d} |j| dddddf} d} |}|jdk(r2t } | | j d|j d} n 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```N)rrrr1rrrrWr)losslogitsrr) r-r-rr:r8r rcrrrr)r,r:rr;rrr1rrr1r>r=loss_fctrs r/r?z#VivitForVideoClassification.forwards@&1% @ 'tn @ #'@ d^@  uU&&')>> ?@ `C@ r0r6)8rDrtypingrrrrrLtorch.utils.checkpointrtorch.nnrr activationsr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrutilsrrrrrconfiguration_vivitr get_loggerrAr_CHECKPOINT_FOR_DOCr3ModulerrIrrrrrrrrrrrrrVIVIT_START_DOCSTRINGr2rr6rr0r/rNs .. .!bb-Q-   H %7*RYY*ZLbiiL`99z)#/)#Zbii&$RYY$P88  $ ")) "  $$N0 2990 f ")) M?M@ 2eS %S S l O "6O O r0