sgIdZddlZddlmZmZmZmZmZddlZ ddl Z ddl m Z m Z mZddlmZddlmZddlmZmZej,eZGd d eZy) z!Feature extractor class for CLAP.N)AnyDictListOptionalUnion)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature) TensorTypeloggingceZdZdZddgZ ddededededef fd Zd e ee ffd Z dd e jd ee jd e jfdZdZd e jd e jfdZ ddee jeeee jeeefdedeedeedeedeeeefd efdZxZS)ClapFeatureExtractora Constructs a CLAP feature extractor. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the *Short Time Fourier Transform* (STFT) which should match pytorch's `torch.stft` equivalent. Args: feature_size (`int`, *optional*, defaults to 64): The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters (`n_mels`). sampling_rate (`int`, *optional*, defaults to 48000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves to warn users if the audio fed to the feature extractor does not have the same sampling rate. hop_length (`int`,*optional*, defaults to 480): Length of the overlaping windows for the STFT used to obtain the Mel Spectrogram. The audio will be split in smaller `frames` with a step of `hop_length` between each frame. max_length_s (`int`, *optional*, defaults to 10): The maximum input length of the model in seconds. This is used to pad the audio. fft_window_size (`int`, *optional*, defaults to 1024): Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency resolution of the spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples. padding_value (`float`, *optional*, defaults to 0.0): Padding value used to pad the audio. Should correspond to silences. return_attention_mask (`bool`, *optional*, defaults to `False`): Whether or not the model should return the attention masks coresponding to the input. frequency_min (`float`, *optional*, defaults to 0): The lowest frequency of interest. The STFT will not be computed for values below this. frequency_max (`float`, *optional*, defaults to 14000): The highest frequency of interest. The STFT will not be computed for values above this. top_db (`float`, *optional*): The highest decibel value used to convert the mel spectrogram to the log scale. For more details see the `audio_utils.power_to_db` function truncation (`str`, *optional*, defaults to `"fusion"`): Truncation pattern for long audio inputs. Two patterns are available: - `fusion` will use `_random_mel_fusion`, which stacks 3 random crops from the mel spectrogram and a downsampled version of the entire mel spectrogram. If `config.fusion` is set to True, shorter audios also need to to return 4 mels, which will just be a copy of the original mel obtained from the padded audio. - `rand_trunc` will select a random crop of the mel spectrogram. padding (`str`, *optional*, defaults to `"repeatpad"`): Padding pattern for shorter audio inputs. Three patterns were originally implemented: - `repeatpad`: the audio is repeated, and then padded to fit the `max_length`. - `repeat`: the audio is repeated and then cut to fit the `max_length` - `pad`: the audio is padded. input_features is_longer frequency_min frequency_maxtop_db truncationpaddingc ^t|d||||d| | |_| |_| |_||_|dz dz|_||_||_||z|_ ||_ ||_ | |_ t|j ||| |dd|_t|j ||| |dd|_y)N) feature_size sampling_rate padding_valuereturn_attention_maskhtk)num_frequency_binsnum_mel_filters min_frequency max_frequencyrnorm mel_scaleslaney)super__init__rrrfft_window_sizenb_frequency_bins hop_length max_length_snb_max_samplesrrrr mel_filtersmel_filters_slaney)selfrrr,r-r*rrrrrrrkwargs __class__s c/var/www/html/venv/lib/python3.12/site-packages/transformers/models/clap/feature_extraction_clap.pyr)zClapFeatureExtractor.__init__Ts  %''"7     $ ."1Q"6!!;$(*]:****#55(''' #2#55('''# returnctj|j}|jj|d<d|vr|d=d|vr|d=|S)a) Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance, excpet for the mel filter banks, which do not need to be saved or printed as they are too long. feature_extractor_typer/r0)copydeepcopy__dict__r3__name__)r1outputs r4to_dictzClapFeatureExtractor.to_dictsQt}}-+/>>+B+B'( F "}% 6 )+, r5waveformr/c t|t|jd|j|jd|d}|jS)a Compute the log-mel spectrogram of the provided `waveform` using the Hann window. In CLAP, two different filter banks are used depending on the truncation pattern: - `self.mel_filters`: they correspond to the default parameters of `torchaudio` which can be obtained from calling `torchaudio.transforms.MelSpectrogram().mel_scale.fb`. These filters are used when `truncation` is set to `"fusion"`. - `self.mel_filteres_slaney` : they correspond to the default parameters of `librosa` which used `librosa.filters.mel` when computing the mel spectrogram. These filters were only used in the original implementation when the truncation mode is not `"fusion"`. hanng@dB) frame_lengthr,powerr/log_mel)r r r*r,T)r1r?r/log_mel_spectrograms r4_np_extract_fbank_featuresz/ClapFeatureExtractor._np_extract_fbank_featuressI*  D00& 9--# #$$$r5c tjttd||z dzd}t |ddk(rdg|d<t |ddk(rdg|d<tj j |d}tj j |d}tj j |d}||||zddf}||||zddf} ||||zddf} tj|ddddf}tjjj||dgdd} | ddj} tj| || | gd } | S) Nrrr@bilinearF)sizemode align_cornersaxis)np array_splitlistrangelenrandomchoicetorchtensornn functional interpolatenumpystack) r1mel total_frames chunk_framesranges idx_front idx_middleidx_backmel_chunk_frontmel_chunk_middlemel_chunk_back mel_shrink mel_fusions r4_random_mel_fusionz'ClapFeatureExtractor._random_mel_fusionssU1l\.IA.M%N OQRS vay>Q F1I vay>Q F1III$$VAY/ YY%%fQi0 99##F1I.i)l*BBAEFzJ,EEqHIX<(??BCll3tT1}-.XX((44 |R(z5  ]1%++- XXz?>! z )\)x=:5ii''8a<8#C# *:; ;;HdF]F]^_cef_fg D&  Cx'55h@P@PQ)T__H!wwxA>{ KHk)":H #=>H!wwx:H66(Q X^^A=N0N,OV`rstX% ;;HdFVFVW HHiIy%QXYZ &  !;;HdF]F]^_cef_fg &  r5 raw_speechrzrreturn_tensorsc ||n |j}|r|n |j}|O||jk7rUtd|jj d|jd|jd|d t jdt|tjxrt|jdkD}|r&t|jdkDrtd ||xs@t|ttfxr(t|d tjttf} | r3|D cgc]'} tj| tj )}} n| s@t|tjs&tj|tj }nht|tjrN|j"tj"tj ur|j%tj }| stj|g}|D cgc]$} |j'| |r|n |j(||&} } g} g}| D]'\}}| j+||j+|)|d k(rXJXJz2::$FJbjjAJ  BJJ /J4D4DQSQ[Q[H\4\#**2::6J**Z01J '     *H[H[]gip q    ( %KC   S !   V $ %  !c)n&9yy((C N;H"&Ih  ilD )NWX7G2::>XIX-66&fX6 6,5IN%n5  %+>>~NNOY "Y7s-,L8")L=,M M) rKii igFri6Nrorq)N)NNNNN)r< __module__ __qualname____doc__model_input_namesfloatrwstrr)rrr>rRarrayrrrHrlrrrrr r __classcell__)r3s@r4rr s/b*;7# %""3 3 3 3 3 3 jc3h %288%(SUS[S[J\%hjhrhr%,28!rxx8!UWU]U]8!z!%$('+;?h"**d5k4 3CT$u+EVVWhh# h SM h  } h!sJ!78h hr5r)rr9typingrrrrrr^rRrY audio_utilsr r r !feature_extraction_sequence_utilsr feature_extraction_utilsr utilsrr get_loggerr<rrr'r5r4rsK( 33 HHI4(   H %J3Jr5