sg3 dZddlZddlZddlmZmZmZddlZddl Z ddl Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&e!rddl'm(Z(e$jRe*Z+dZ,dZ-dZ.gdZ/dZ0dZ1 dQdee2e2fde3de2dee jhde2dejjf dZ6Gdde jnZ8Gdd e jnZ9Gd!d"e jnZ:Gd#d$e jnZ;Gd%d&e jnZ<Gd'd(e jnZ=Gd)d*e jnZ>Gd+d,e>Z?Gd-d.e>Z@e>e@e?d/ZAGd0d1e jnZBGd2d3e jnZCGd4d5e jnZDGd6d7e jnZEGd8d9e jnZFGd:d;eZGdeHGd?d@eGZJedAeHGdBdCeGZKedDeHGdEdFeGZLedGeHGdHdIeGZMGdJdKe jnZNGdLdMe jnZOedNeHGdOdPeGZPy)RzPyTorch Data2VecAudio model.N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10is_peft_availablelogging)Data2VecAudioConfig)_flash_attention_forwardrz!facebook/data2vec-audio-base-960h)ri$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'g̼P@shape mask_prob mask_lengthattention_mask min_masksreturnc|\}dkr tdkDrtdddtjjdj fd}|-|j dj jnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr rr"sequence_lengths g/var/www/html/venv/lib/python3.12/site-packages/transformers/models/data2vec/modeling_data2vec_audio.pycompute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spannsoi,6DwNOoy9 [ (? :-!#Car/cV|jdkDr|ddddd|j f}|SNr)rrrs r-rtzData2VecAudioPadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr/rvr{s@r-r}r}sKr/r}c$eZdZfdZdZxZS) Data2VecAudioPositionalConvLayerczt|tj|j|j|j |j dz|j |_t|j |_ t|j|_ tj|jd|_y)Nr)rWpaddinggroupsFrZ)r\r]rra hidden_sizeconv_pos_kernel_sizenum_conv_pos_embedding_groupsrer}rr rhrirfrgrkrlrns r-r]z)Data2VecAudioPositionalConvLayer.__init__s II      33//1477   -V-H-HI  !?!?@,,v'9'9eTr/c|j|}|j|}|jdd}|j|}|jdd}|j |}|SNrr)rerrqrgrirrs r-rtz(Data2VecAudioPositionalConvLayer.forwardsd -0  ]3 %//15  6 %//15  6 r/rvr{s@r-rrs Ur/rc$eZdZfdZdZxZS)$Data2VecAudioPositionalConvEmbeddingct|tjt |j Dcgc] }t |c}|_ycc}wN)r\r]r ModuleListr<rrlayersrkrlrKrns r-r]z-Data2VecAudioPositionalConvEmbedding.__init__s@ mm?DVEcEc?d e! -f 5 e  esAc|jdd}|jD] }||} |jdd}|Sr)rqr)rkrslayers r-rtz,Data2VecAudioPositionalConvEmbedding.forwardsI%//15 [[ 1E!-0M 1%//15 r/rvr{s@r-rrs  r/rc.eZdZdZfdZdZdZxZS)Data2VecAudioFeatureEncoderz.Construct the features from raw audio waveformc t|tjt |j Dcgc]}t ||c}|_d|_d|_ ycc}w)N)rmFT) r\r]rrr<num_feat_extract_layersrU conv_layersgradient_checkpointing_requires_grad)rkrlirns r-r]z$Data2VecAudioFeatureEncoder.__init__sX ==AFvGeGeAf gA #FQ 7 g ',#" hsA%cJ|jD] }d|_ d|_yNF) parameters requires_gradrrkparams r-_freeze_parametersz.Data2VecAudioFeatureEncoder._freeze_parameterss(__& (E"'E  (#r/c |dddf}|jr|jrd|_|jD]K}|jr5|jr)|jr|j |j |}D||}M|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)rk input_valuesrs conv_layers r-rtz#Data2VecAudioFeatureEncoder.forward"s$QW-    4==*.M '** :J""t'B'Bt}} $ A A''!! !+= 9  :r/)rwrxry__doc__r]rrtrzr{s@r-rrs8#$ r/rc$eZdZfdZdZxZS)Data2VecAudioFeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)Nr0eps) r\r]rrfr^layer_norm_epsrgLinearr projectionDropoutfeat_proj_dropoutdropoutrs r-r]z'Data2VecAudioFeatureProjection.__init__7sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r/cp|j|}|j|}|j|}||fSr)rgrr)rkrsnorm_hidden_statess r-rtz&Data2VecAudioFeatureProjection.forward=s:!__];(:;  ]3 000r/rvr{s@r-rr6s < 1r/rceZdZdZ ddededededededeeffd Z d e jd ed efd Z dde jdee jdee e jdee jdee jdede e jee jee e jffdZxZS)Data2VecAudioAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderrY is_causalrlc t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)rY)r\r]rrrhead_dimrlr4scalingrrrrk_projv_projq_projout_proj) rkrrrrrYrrlrns r-r]zData2VecAudioAttention.__init__Is "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r/tensorseq_lenbszc|j|||j|jjddj Sr)viewrrrq contiguousrkrrrs r-_shapezData2VecAudioAttention._shapehs7{{3GQQRSUVWbbddr/rskey_value_statespast_key_valuer!layer_head_maskoutput_attentionsr#c |du}|j\}} } |j||jz} |r0|.|djd|jdk(r |d} |d} n |rE|j |j |d|} |j |j |d|} n|}|j |j |d|} |j |j |d|} tj|d| gd} tj|d| gd} nD|j |j |d|} |j |j |d|} |jr| | f}||jzd|jf}|j | | |j|} | j|} | j|} | jd}tj| | jdd}|j||jz| |fk7r/t!d||jz| |fd|j|{|j|d| |fk7r#t!d |d| |fd|j|j||j| ||z}|j||jz| |}t"j$j'|d}||j|jfk7r*t!d |jfd|j|jdddd|j||j| |z}|j||jz| |}|r?|j||j| |}|j||jz| |}nd}t"j$j)||j(|j* }tj|| }|j||jz| |jfk7r9t!d ||jz| |jfd|j|j||j| |j}|jdd}|j|| |j,}|j/|}|||fS) #Input shape: Batch x Time x ChannelNrrrr0dimz$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )pr `attn_output` should be of size )sizerrrrrrtorchcatrrrrrHbmmrqr4r functionalsoftmaxrrrr)rkrsrrr!rris_cross_attentionrtgt_lenrK query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r-rtzData2VecAudioAttention.forwardks.T9',,.Wa{{=1DLL@ *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > Ct{{<#>CCZP 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<AAr/)FTFNNNNNF)rwrxryrr'floatr>rrr]rTensorrrrtrzr{s@r-rrFsNG  04CCC C  C  CC,-C>eU\\eCece 488<1526"'vB||vB#5<<0vB!u||!45 vB !. vB "%,,/ vB vB u||Xell3XeELL>Q5RR SvBr/rcVeZdZdZfdZdej dedefdZ ddej de ej d e e ej d e ej d e ej d e d e ej e ej e e ej ffdZ xZ S)Data2VecAudioFlashAttention2aV Data2VecAudio flash attention module. This module inherits from `Data2VecAudioAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cDt||i|t |_yr)r\r]r_flash_attn_uses_top_left_mask)rkargskwargsrns r-r]z%Data2VecAudioFlashAttention2.__init__s& $)&) 3V2W.W+r/rrrcR|j|||j|jSr)rrrrs r-_reshapez%Data2VecAudioFlashAttention2._reshapes{{3GGr/rsrrr!rrr#c |r td|du}|j\}} } |j|j|d|} |rP|N|djd|jdk(r,|dj dd} |dj dd} n*|rE|j|j |d|} |j|j|d|} n||j|j |d|} |j|j|d|} tj|dj dd| gd} tj|dj dd| gd} nD|j|j |d|} |j|j|d|} |jr$| j dd| j ddf}| jd}|||djdz }| j}|tjk(rtjrtj}nMt|j dr|j j"}n |jj$j}t&j)d |d | j+|} | j+|} | j+|} t-| | | || |j.r |j0nd |j2|j4 }|j7|| d}|j9|}|sd}||fS) NzIData2VecAudioFlashAttention2 attention does not support output_attentionsr0rrrrrp_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .r)rruse_top_left_mask)r4rrrrrqrrrrrr2float32is_autocast_enabledget_autocast_gpu_dtypehasattrrlrweightlogger warning_oncetorrrrrrHr)rkrsrrr!rrrrq_lenrKrrr kv_seq_len input_dtype target_dtyperrs r-rtz$Data2VecAudioFlashAttention2.forwards0 hi i.T9%**, UA}}T[[%?SI *q!''*.>.D.DQ.GG(*44Q:J)!,66q!$  (??<8L#|4J'??<8L.     $(MMDLLsnn"AA  "))#ub9 mmK0  LL.88r/r)rwrxryrr]rrr'rrrr>rtrzr{s@r-rrsXHu||HcHH 488<1526"'i9||i9#5<<0i9!u||!45 i9 !. i9 "%,,/ i9 i9 u||Xell3XeELL>Q5RR Si9r/rc$eZdZ d dejdeejdeeejdeejdeejdedeejeejeeejfffd ZxZ S) Data2VecAudioSdpaAttentionrsrrr!rrr#cz|s|*tjdt| ||||||S|du}|j \}} } |j |} |r0|.|dj d|j dk(r |d} |d} n |rE|j|j|d|} |j|j|d|} n|}|j|j|d|} |j|j|d|} tj|d| gd} tj|d| gd} nD|j|j|d|} |j|j|d|} |jr| | f}|j| | |} |jr || dkDrd nd }tjjj!| | | ||j"r |j$nd | }|j ||j&| |j(fk7r7t+d ||j&| |j(fd|j |j-dd}|j/|| |j0}|j3|}|d|fS)rNaData2VecAudioModel is using Data2VecAudioSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr!rrrrrr0rTFr) attn_mask dropout_prrr)rrr\rtrrrrrrrrrrrrscaled_dot_product_attentionrrrrr4rqrHrr)rkrsrrr!rrrrrrKrrrrrrns r-rtz"Data2VecAudioSdpaAttention.forwardfs  ;   l 7?!1-- /"3 # .T9',,.Wa{{=1 *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7N{{<#> !NN~/E'TU+D[` hh))FF   $&*mmdll G     #t~~w !N N2CRVR_R_3`2ab$$&')  "++Aq1 "))#wG mmK0 D.00r/r) rwrxryrrrrr>rtrzr{s@r-rrds 488<1526"'f1||f1#5<<0f1!u||!45 f1 !. f1 "%,,/ f1 f1 u||Xell3XeELL>Q5RR Sf1f1r/r)eagersdpaflash_attention_2c$eZdZfdZdZxZS)Data2VecAudioFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr)r\r]rractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r-r]z!Data2VecAudioFeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r/c|j|}|j|}|j|}|j|}|j |}|Sr)rrrrrrrs r-rtz Data2VecAudioFeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r/rvr{s@r-rrs @r/rc&eZdZfdZddZxZS)Data2VecAudioEncoderLayerct|t|j|j|j |j d|_tj|j|_ tj|j|j|_t||_tj|j|j|_y)NF)rrrrr)r\r] DATA2VEC2AUDIO_ATTENTION_CLASSES_attn_implementationrnum_attention_headsattention_dropout attentionrrrrrfrrgr feed_forwardfinal_layer_normrs r-r]z"Data2VecAudioEncoderLayer.__init__s 9&:U:UV((00,,  zz&"7"78 ,,v'9'9v?T?TU4V< " V-?-?VEZEZ [r/c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|S)Nr!r)r&rrgr'r()rkrsr!r attn_residualrrKoutputss r-rtz!Data2VecAudioEncoderLayer.forwards% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr/rrvr{s@r-r r s \r/r c reZdZfdZ ddej deejdededef dZ xZ S) Data2VecAudioEncoderct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_|j&dk(|_ycc}w)NrFr)r\r]rlrpos_conv_embedrrfrrrgrrrrr<num_hidden_layersr rrr#_use_flash_attention_2rs r-r]zData2VecAudioEncoder.__init__s  B6J,,v'9'9v?T?TUzz&"7"78 mmPUV\VnVnPo$p1%>v%F$pq &+#&,&A&AEX&X#%qs!Crsr!routput_hidden_states return_dictc4|rdnd}|rdnd}||jdjdd|jd}d||<|jr |d|vr|nd}nd|ddddddfj |j z }|t j|j jz}|j|jdd|jd|jd}|j|} || z}|j|}|j|}txs t|} |jD]} |r||fz}t j g} |j"r| |j$j&krdnd } | r| rG|j(r+|j"r|j+| j,|||}n | ||| }|d}| rd }|s|dfz}|r||fz}|st/d |||fDSt1||| S)Nr0rrr?r1TFr*NNc3&K|] }|| ywrr6).0vs r- z/Data2VecAudioEncoder.forward..asmq_`_lms)last_hidden_staters attentions) unsqueezerepeatrr2rr2rfinfominexpandr0rgrr r rr7rrl layerdroprrrtupler )rkrsr!rr3r4all_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r-rtzData2VecAudioEncoder.forwards[#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"#11-@%(;;  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. %)]]8KdkkNcNc8cTjoN![..4==$($E$E%&) %M%*%nXi%M!.a 0  ,  &9]1=Mrtrzr{s@r-r.r.s_Y26"'%* G ||G !.G  G # G  G r/r.c$eZdZfdZdZxZS)Data2VecAudioAdapterct|jjk7rTt j jj|_t jj|_ndx|_|_t jfdtjD|_ j|_ y)Nc34K|]}tywr)Data2VecAudioAdapterLayer)r:rKrls r-r<z0Data2VecAudioAdapter.__init__..us#p!$=f$E#ps)r\r]output_hidden_sizerrrprojrfproj_layer_normrr<num_adapter_layersrrDrs `r-r]zData2VecAudioAdapter.__init__ks   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#puU[UnUnOo#pp ))r/ch|j.|j"|j|}|j|}|jdd}|jD]D}tj j }|j r||jkDs=||}F|jdd}|Sr)rTrUrqrr5r6rrD)rkrsrlayerdrop_probs r-rtzData2VecAudioAdapter.forwardxs 99 T%9%9%E IIm4M 00?M%//15 [[ 5EYY--/N==^dnn%D %m 4  5 &//15 r/rvr{s@r-rOrOjs  *r/rOc$eZdZfdZdZxZS)rRct|tj|jd|jz|j |j d|_y)Nrr)rXr)r\r]rrarSadapter_kernel_sizeadapter_striderers r-r]z"Data2VecAudioAdapterLayer.__init__sJ II  % % )) )  & &((   r/cj|j|}tjj|d}|S)Nrr)rerrglurrs r-rtz!Data2VecAudioAdapterLayer.forwards/ -0  ))-Q)? r/rvr{s@r-rRrRs  r/rRceZdZdZeZdZdZdZdZ dZ dZ d de e jefdeefd Z d d ed e jfd Zy)Data2VecAudioPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. data2vec_audiorTct|trtjd|jj z }t jj|jj| |t jj|jj| |yt|tr5t jj|jjdyt|t jrm|jjj!d|j"j$|j%|jjj'yyt|t j(t j*frc|j$|jjj'|j&|jjj-dyyt|t j.rt jj1|j|jjtj|j2|j4|j6dzz }t jj|j| |yyy)zInitialize the weightsr)abrr)meanstdNr7)rrmathsqrtr in_featuresrinituniform_rrYr constant_rerdatanormal_rlinitializer_rangezero_rf GroupNormfill_rakaiming_normal_r in_channelsrW)rkmoduleks r- _init_weightsz*Data2VecAudioPreTrainedModel._init_weightss f< = !f//;;;tj||z |ddzS)Nfloor) rounding_moder)rdivr)rWrXs r-_conv_out_lengthzWData2VecAudioPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [r/r)rlrxziprbrcr<rVr\)rkrLrxrrWrXrKs r- _get_feat_extract_output_lengthsz=Data2VecAudioPreTrainedModel._get_feat_extract_output_lengthss2=1Ddkk--+  \ $'t{{'>'> @W@W#X Q K,]KPM Q 4;;99: _ 04;;C]C] ^  _r/feature_vector_lengthr!c|jddddf}|j||}|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)Nr0rrxr)r2devicer)r) cumsumrrrlongrr=r2rr@flipr>)rkrr!rxnon_padded_lengthsoutput_lengthsrJs r-"_get_feature_vector_attention_maskz?Data2VecAudioPreTrainedModel._get_feature_vector_attention_masks ,22r2:1b5A>>?Q_j>k'**5::6#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr/r)rwrxryrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparwrr LongTensorr'rr>rrr6r/r-r`r`s 'L($O&*#!N96Z^"5#3#3S#89HPQU2Y]%(:?:J:Jr/r`a Data2VecAudio was proposed in [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://arxiv.org/pdf/2202.03555) by Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu and Michael Auli. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Data2VecAudioConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aJ Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type *List[float]* or a *numpy.ndarray*, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into *input_values*, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type *torch.FloatTensor*. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should be passed if the corresponding processor has `config.return_attention_mask == True`, which is the case for all pre-trained Data2Vec Audio models. Be aware that that even with `attention_mask`, zero-padded inputs will have slightly different outputs compared to non-padded inputs because there are more than one convolutional layer in the positional encodings. For a more detailed explanation, see [here](https://github.com/huggingface/transformers/issues/25621#issuecomment-1713759349). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zgThe bare Data2VecAudio Model transformer outputting raw hidden-states without any specific head on top.c\eZdZdeffd ZdZ ddejdeejdeejfdZ e e e eeede  dd eej$deej$deejd eed eed eedeeeffdZxZS)Data2VecAudioModelrlct||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_t!||_|j$r t'|nd|_|j+y)Nr)r\r]rlrfeature_extractorrfeature_projectionmask_time_probmask_feature_probr Parameterrrrrkmasked_spec_embedr.encoderrxrOadapter post_initrs r-r]zData2VecAudioModel.__init__-s   !s 113r/rsmask_time_indicesr!ct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)rr r!r")rr2)rr r"r0)getattrrlrrrr2rrrSmask_time_lengthmask_time_min_masksrrrr>rmask_feature_lengthmask_feature_min_masksrC)rkrsrr!rJr,rmask_feature_indicess r-_mask_hidden_statesz&Data2VecAudioModel._mask_hidden_statesEst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r/audio) checkpoint output_typermodalityexpected_outputrrr3r4r#cH||n|jj}||n|jj}||n|jj}|j |}|j dd}|!|j |jd|d}|j|\}}|j|||}|j|||||} | d}|j|j|}|s ||f| ddzSt||| j| jS) NrrFr)rr!r!rr3r4r)r=extract_featuresrsr>)rlrr3use_return_dictrrqrrrrrrrrsr>) rkrr!rrr3r4rrsencoder_outputss r-rtzData2VecAudioModel.forwardssb"2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' '00 ->~1 ,, )/!5# ' (* << # LL7M!#34qr7JJ J&+-)77&11   r/r8NNNNN)rwrxryrr]rr FloatTensorrrrrDATA2VEC_AUDIO_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr>rrrtrzr{s@r-rr(s 2"4:>59 ,((,$E$5$56,!!1!12 ,\++JK&+$. 269=,0/3&*2 u||,2 !.2 $E$5$56 2 $D> 2 'tn 2 d^2  u-- .2 L2 r/rzkData2VecAudio Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).ceZdZfdZdZdZeeee e e e e  d deej deej deedeed eed eej d eee ffd ZxZS)Data2VecAudioForCTCct||t||_t j |j |_|jtd|jdt|dr|jr |jn |j}t j||j|_|j#y)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Data2VecAudioForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.rx)r\r]rrarr final_dropoutr vocab_sizer4rnrrxrSrrlm_headr)rkrlrSrns r-r]zData2VecAudioForCTC.__init__s  08zz&"6"67    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r/cXtjdt|jyrThe method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningrrs r-freeze_feature_extractorz,Data2VecAudioForCTC.freeze_feature_extractor'   Q  ##%r/cL|jjjyrrarrrs r-rz*Data2VecAudioForCTC.freeze_feature_encoder --@@Br/)rrrr expected_lossrr!rr3r4labelsr#c ||n|jj}|I|j|jjk\r"t d|jj|j |||||}|d}|j |}|j|} d} |b||n$tj|tj}|j|jdjtj} |dk\} | jd} |j| }tj j#| dtj$j'dd}tj(j*j-d 5tj j/||| | |jj0|jj2|jj4 } ddd|s| f|t6dz}| | f|zS|St9| | |j:|j< S#1swYExYw) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrr1r0)rr2rF)enabled)blank reduction zero_infinitylosslogitsrsr>)rlrr(rr4rarrr ones_likerrr9r masked_selectrr log_softmaxrrqbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONr rsr>)rkrr!rr3r4rr,rsrrrL labels_masktarget_lengthsflattened_targets log_probsoutputs r-rtzData2VecAudioForCTC.forwards)2&1%rrrtrzr{s@r-rrs * &C++JK&"$,( 26,0/3&*)-D u||,D !.D $D> D 'tn D d^ D &D  un$ %D LD r/rz Data2VecAudio Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZeee e e e d dde ejde ejd e ed e ed e ed e ejd eee ffdZxZS)&Data2VecAudioForSequenceClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)NrxzdSequence classification does not support the use of Data2VecAudio adapters (config.add_adapter=True)r)r\r]rrxr4rrar1use_weighted_layer_sumrrrrC layer_weightsrrclassifier_proj_size projector num_labels classifierrrkrl num_layersrns r-r]z/Data2VecAudioForSequenceClassification.__init__5s  6= )f.@.@v 18--1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r/cXtjdt|jy)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. rNrrs r-rz?Data2VecAudioForSequenceClassification.freeze_feature_extractorFrr/cL|jjjyrrrs r-rz=Data2VecAudioForSequenceClassification.freeze_feature_encoderRrr/cP|jjD] }d|_ yz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNrarrrs r-freeze_base_modelz8Data2VecAudioForSequenceClassification.freeze_base_modelY* ((335 (E"'E  (r/rrrrrrr!rr3r4rr#c||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}||jd} nZ|j|j d|} d|| <|jd| jdjddz } |j#| } d} |Ft%}|| jd|jj&|jd} |s| f|tdz}| | f|zS|St)| | |j*|j, S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrr0rrr)rlrrrarrstackrrrrrr9rrerrrrrrrsr>)rkrr!rr3r4rr,rs norm_weights pooled_output padding_maskrrloss_fctrs r-rtz.Data2VecAudioForSequenceClassification.forwardas0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL+.M<- ()--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r/r)rwrxryr]rrrrrrrrrrrrr>rrrtrzr{s@r-rr-s" &C(++JK&,$ 26,0/3&*)-; u||,; !.; $D> ; 'tn ; d^ ; &;  u.. /; L; r/rzi Data2VecAudio Model with a frame classification head on top for tasks like Speaker Diarization. ceZdZfdZdZdZdZeee e e e d dde ejde ejd e ejd e ed e ed e ed eee ffdZxZS)(Data2VecAudioForAudioFrameClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_|j |_|j%y)NrxzgAudio frame classification does not support the use of Data2VecAudio adapters (config.add_adapter=True)r)r\r]rrxr4rrar1rrrrrCrrrrr init_weightsrs r-r]z1Data2VecAudioForAudioFrameClassification.__init__s  6= )f.@.@- 18--1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r/cXtjdt|jyrrrs r-rzAData2VecAudioForAudioFrameClassification.freeze_feature_extractorrr/cL|jjjyrrrs r-rz?Data2VecAudioForAudioFrameClassification.freeze_feature_encoderrr/cP|jjD] }d|_ yrrrs r-rz:Data2VecAudioForAudioFrameClassification.freeze_base_modelrr/rrrr!rrr3r4r#c ||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|} d} |\t} | | jd|jt j |jd|jd} |s| f|tdz} | St#| | |j$|j& S) rNTrrrr0r)axisr)rlrrrarrrrrrrrr9rrrargmaxrrsr>)rkrr!rrr3r4r,rsrrrrrs r-rtz0Data2VecAudioForAudioFrameClassification.forwardsj0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r/r)rwrxryr]rrrrrrrrrrrrr>rrrtrzr{s@r-rrs" &C(++JK&)$ 26)-,0/3&*3 u||,3 !.3 & 3 $D> 3 'tn 3 d^3  u++ ,3 L3 r/rc&eZdZdfd ZdZxZS) AMSoftmaxLossctt| ||_||_||_t jtj||d|_ t j|_ y)NT)r) r\rr]scalemarginrrrrrandnrrr)rk input_dimrrrrns r-r]zAMSoftmaxLoss.__init__sS mT+-  $ll5;;y*#EUYZ '') r/c|j}tjj|jd}tjj|d}t j ||}||jz }tjj||j}|jt j|j||z}|j||}|S)Nrrr)flattenrr normalizerrmmrone_hotrrwherer>r) rkrsrr cos_thetapsionehotrrs r-rtzAMSoftmaxLoss.forward"s!((!(< // 1/E HH]F3 $++%&&vt?ekk&++-iHHyy( r/)g>@g?rvr{s@r-rrs * r/rcXeZdZdfd Zdej dej fdZxZS) TDNNLayerct||dkDr|j|dz n|j||_|j||_|j ||_|j||_tj|j|j z|j|_ tj|_ y)Nrr)r\r]tdnn_dimr_r` tdnn_kernelrW tdnn_dilationdilationrrkernelReLUrirjs r-r]zTDNNLayer.__init__2s +BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST Rs>FcXtjdt|jyrrrs r-rz0Data2VecAudioForXVector.freeze_feature_extractorjrr/cL|jjjyrrrs r-rz.Data2VecAudioForXVector.freeze_feature_encodervrr/cP|jjD] }d|_ yrrrs r-rz)Data2VecAudioForXVector.freeze_base_model}rr/rLcVd}|jjD] }|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nrr6r~s r-rzJData2VecAudioForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r/r)rlr$)rkrLrrWs r-_get_tdnn_output_lengthsz0Data2VecAudioForXVector._get_tdnn_output_lengthss:  >  ;;22 LK,]KKM Lr/rrrr!rr3r4rr#c||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}|jD] } | |} |%|jd} |j!d} n|j#|jd} |j%| }g} g} t'|D]U\}}| j)||d|fjd| j)||d|fj!dWt j | } t j | } t j*| | gd}|j-|}|j/|}d}||j1||}|s||f|tdz}||f|zS|St3||||j4|j6S) rNTrrrr0r)rr embeddingsrsr>)rlrrrarrrrrrrrr9rr0rerfrr9 enumeraterErrrr2rrsr>)rkrr!rr3r4rr,rsr tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrrrs r-rtzData2VecAudioForXVector.forwards0&1%->r1a-HHMMRSMTM#AJM}5 )) 6J&}5M 6  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; J 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI J"KK 6M ;;|4L!II}l&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r/r)rwrxryr]rrrrrrr'r9rrrrrrrrr>rrtrzr{s@r-r.r.Ps& &C( eE I 'tn I d^ I &I  um# $I LI r/r.r)Qrrgrtypingrrrnumpyr5rtorch.utils.checkpointrtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_outputsr r rrrrmodeling_utilsrutilsrrrrrrrconfiguration_data2vec_audiormodeling_flash_attention_utilsr get_loggerrwrrrrrrrr'rrndarrayrSModulerUr}rrrrrrrr"rr r.rOrRr`DATA2VEC_AUDIO_START_DOCSTRINGrrrrrrr!r.r6r/r-rUs# )) %!@7.>J   H %!"(:&u26 t c?tttU--. t  t ZZ tnRYY8BIIryy6 299 ""))"L 1RYY 1 [BRYY[B~|9#9|9~h1!7h1X$ &5$ ryy2  HR 299R l299@ $Q?Qh"&"#Jm"A 5A A Hu"v 6v v r# p -Ip p f#  h /Kh  h XBII0 >#  O :O  O r/