sgcvdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%er e!rddl&Z&e$jNe(Z)e dZ*Gd d eZ+Gd d eZ,ee,jZe,_-e,jZj8e,jZjj]d dde,jZ_yy)zH Feature extraction saving/loading class for common feature extractors. N)UserDict) TYPE_CHECKINGAnyDictOptionalTupleUnion)custom_object_save)FEATURE_EXTRACTOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_func download_urlis_flax_available is_jax_tensoris_numpy_arrayis_offline_mode is_remote_urlis_tf_availableis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsSequenceFeatureExtractorceZdZdZddeeeefdedee fffd Z dedeefdZ defd Z d Z d Zd Zd ZdZddeeee ffdZddeeee ffdZddZxZS) BatchFeatureaU Holds the output of the [`~SequenceFeatureExtractor.pad`] and feature extractor specific `__call__` methods. This class is derived from a python dictionary and can be used as a dictionary. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. Ndata tensor_typecHt|||j|y)N)r#)super__init__convert_to_tensors)selfr"r# __class__s X/var/www/html/venv/lib/python3.12/site-packages/transformers/feature_extraction_utils.pyr&zBatchFeature.__init__Ms!  K8itemreturncVt|tr|j|Std)z If the key is a string, returns the value of the dict associated to `key` ('input_values', 'attention_mask', etc.). zRIndexing with integers is not available when using Python based feature extractors) isinstancestrr"KeyErrorr(r,s r* __getitem__zBatchFeature.__getitem__Qs( dC 99T? "op pr+cH |j|S#t$rtwxYwN)r"r1AttributeErrorr2s r* __getattr__zBatchFeature.__getattr__[s* !99T? " !  !s!cd|jiSNr"r"r(s r* __getstate__zBatchFeature.__getstate__as ""r+c"d|vr |d|_yyr9r:)r(states r* __setstate__zBatchFeature.__setstate__ds U?f DI r+c6|jjSr5)r"keysr;s r*rAzBatchFeature.keysisyy~~r+c6|jjSr5)r"valuesr;s r*rCzBatchFeature.valuesmsyy!!r+c6|jjSr5)r"itemsr;s r*rEzBatchFeature.itemsqsyy  r+c|yt|ts t|}|tjk(r5ts t dddl}|j |j}|fS|tjk(r.ts t dddl fdj}|fS|tjk(r1ts t dddl m}|jt }|fSdfd t"}|fS) NNNzSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.ct|ttfrt|dkDrt|dtj rt j |}n_t|dttfrFt|ddkDr5t|ddtj rt j |}t|tj rj|Sj|S)Nr) r/listtuplelennpndarrayarray from_numpytensor)valuetorchs r* as_tensorz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]3E Q!%(BJJ7 ""58dE];aMA-&uQx{BJJ? "eRZZ0+5++E22'5<<..r+zEUnable to convert output to JAX tensors format, JAX is not installed.ct|ttfrt|dtttjfr`|Dcgc] }t |}}t t |dkDr1|/|Dcgc]}tj|c}t}tj||Scc}wcc}w)Nrr )dtype) r/rIrJrLrMrKsetasarrayobject)rQrUval value_lensrSs r*rSz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]3 58dTY[][e[eMf8g6;!c$i!mq'Cc"C%)=CQTAU!#McRUhZWo!pqqJJL DAq!U\\*/Fu/F/Fq/I"addD3F3 Au||,1Cdd&d1     r+rGr5)r-r!)__name__ __module__ __qualname____doc__rrr0rr rr&r3r7r<r?rArCrErfr'rq __classcell__)r)s@r*r!r!>s 9Xd38n595QUWZ\fQfKg9qqc q!! #&  "!9$%Z:P1Q9$vhuS*_7M.N>*r+r!ceZdZdZdZdZdefdZe dde ee jfde e ee jfde d e d e e ee fd ef d Zdd e ee jfde fdZede ee jfdeeeefeeefffdZedeeefdefdZdeeeffdZede ee jfdefdZdefdZde ee jffdZdZeddZy)FeatureExtractionMixinz This is a feature extraction mixin used to provide saving/loading functionality for sequential and image feature extractors. Nc |jdd|_|jD]\}} t|||y#t$r%}t j d|d|d||d}~wwxYw)z'Set elements of `kwargs` as attributes.processor_classNz Can't set z with value z for )pop_processor_classrEsetattrr6loggererror)r(rsrjrQerrs r*r&zFeatureExtractionMixin.__init__sx!' + To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]. Examples: ```python # We can't instantiate directly the base class *FeatureExtractionMixin* nor *SequenceFeatureExtractor* so let's show the examples on a # derived class: *Wav2Vec2FeatureExtractor* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h" ) # Download feature_extraction_config from huggingface.co and cache. feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "./test/saved_model/" ) # E.g. feature_extractor (or model) was saved using *save_pretrained('./test/saved_model/')* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("./test/saved_model/preprocessor_config.json") feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False ) assert feature_extractor.return_attention_mask is False feature_extractor, unused_kwargs = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False, return_unused_kwargs=True ) assert feature_extractor.return_attention_mask is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)rwarningswarn FutureWarningriget_feature_extractor_dict from_dict) clsrrrrrrrsrfeature_extractor_dicts r*from_pretrainedz&FeatureExtractionMixin.from_pretrainedsp({#1 %5!"%z$4d;  % MME   l#E  #F7O)G)G)GHe)pio)p&s}}3>v>>r+save_directory push_to_hubc 6|jdd}|>. ) ?>2BBe!fg g NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     ' t^D A)+ ^E[(\% 78 12O1PQR   ' ' -jj) ( ...r+r-c |jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d} |jd d} |jd d} | )tjd t| t d | }|jdd} |jdd} d| d}| | |d<t r| st jdd} t|}tjj|}tjj|r$tjj|t}tjj|r|}d}n6t|r|}t!|}nt} t#||||||| ||||  } t)|dd5}|j+}dddt-j.}|rt jd|nt jdd ||s,d!|vrt3|d!||d!<d"|vrt5|d"||d"<||fS#t$$rt&$rt%d|d|dtdwxYw#1swYxYw#t,j0$rt%d|dwxYw)#a6 From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the feature extractor object. rNrFresume_downloadproxies subfolderrrrrrr_from_pipeline _from_autofeature extractor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rrrrrrr user_agentrz"Can't load feature extractor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at auto_mapcustom_pipelines)rrrrrirrrr0rrisdirrr rrrrEnvironmentError ExceptionopenreadjsonloadsJSONDecodeErrorrr)rrrsrrrrrrrrr from_pipelinerris_localfeature_extractor_fileresolved_feature_extractor_filereadertextrs r*rz1FeatureExtractionMixin.get_feature_extractor_dictsDJJ{D1 $4e< **%6=**Y-JJ{D1  7D)$4d;!::&8%@::j$/  % MME   l#E #3T:  **\59#6?[  $+8J' (  %5 KKE F# (+,I(J%77==!>? 77==6 7%'WW\\2OQg%h " 77>>7 8.K +H 8 9%B ".:;X.Y +%; " 2=1*'#1#$3%5')% 3/4 5sWM %QW{{} %%)ZZ%5 "  KK56U5VW X KK-.D-E_UtTuv 335O*:68U6&z2"%;;=_*+=>@]>&'9:&v--S$  &89V8WX99V8WX//E.FeM  % %## "45T4UUpq  s0J )K7J<K ,J9<KK#K+rc |jdd}g}|jD] \}}||vs |||<|j|"|D]}|j|d|di|}tj d||r||fS|S)a  Instantiates a type of [`~feature_extraction_utils.FeatureExtractionMixin`] from a Python dictionary of parameters. Args: feature_extractor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the feature extractor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~feature_extraction_utils.FeatureExtractionMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the feature extractor object. Returns: [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature extractor object instantiated from those parameters. return_unused_kwargsFNzFeature extractor )rrEappendrr)rrrsr to_removerjrQfeature_extractors r*rz FeatureExtractionMixin.from_dict5s$ &zz*@%H  ,,. &JC,,.3&s+  % & "C JJsD ! " 9"89 ():(;<= $f, ,$ $r+ctj|j}|jj|d<d|vr|d=d|vr|d=|S)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. feature_extractor_type mel_filterswindow)copydeepcopy__dict__r)rx)r(outputs r*to_dictzFeatureExtractionMixin.to_dictZsP t}}-+/>>+B+B'( F "}% v x  r+ json_filect|dd5}|j}dddtj}|di|S#1swY&xYw)a Instantiates a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] from the path to a JSON file of parameters. Args: json_file (`str` or `os.PathLike`): Path to the JSON file containing the parameters. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature_extractor object instantiated from that JSON file. rrrNr)rrrr)rrrrrs r*from_json_filez%FeatureExtractionMixin.from_json_filegsP)S7 3 !v;;=D !!%D!1,+,, ! !s AAc|j}|jD]3\}}t|tjs!|j ||<5|j dd}|||d<tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. rNrT)indent sort_keys ) rrEr/rLrMtolistrrdumps)r( dictionaryrjrQrs r*to_json_stringz%FeatureExtractionMixin.to_json_stringzs\\^ $**, 1JC%,"',,. 3 1 &>>* This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoFeatureExtractor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r/r0rxtransformers.models.automodelsautohasattrrir)r auto_class auto_modules r*register_for_auto_classz.FeatureExtractionMixin.register_for_auto_classsC *c*#,,J66{J/ |+FGH H$r+)NFFNmain)F)AutoFeatureExtractor)rxryrzr{rr&r0r classmethodr rPathLikerboolrrrrrrPreTrainedFeatureExtractorrrrrrrrrr+r*r~r~s K 0C08<$!&,0m?',S"++-='>m?E#r{{"234m? m?  m? c4i() m?m?m?^;/eC4D.E;/TX;/zr.,1#r{{2B,Cr. tCH~tCH~- .r.r.h"%tCH~"%Lf"%"%H c3h -uS"++-='>-C]--$GG* 05bkk1A+B 0D%%r+r~rrzfeature extractor file)rX object_class object_files)/r{rrrr collectionsrtypingrrrrrr rcrLdynamic_module_utilsr utilsr r rrrrrrrrrrrrrrrrrrR get_loggerrxrrr!r~rformatrr+r*r s  CC4.   H %"#=>z8zzz%^z%z &//E/Q/Q%R"%%--91G1S1S1[1[1b1b"1GVn2c2&&.:r+