sg HdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm Z m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZmZerddlmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-e-j\e/Z0e)e e1jdZ3d d d dZ4ejjdk\r ejlZ6n ejlZ6GddedZ7GddedZ8GddedZ9GddedZ:GddedZ;Gdde7e8e9e:e;dZ<Gdde#Z=d Z>e(e=j~e=_?e=j~j8e=j~jjd!d"d#$e=j~_yy)%z8 Processing saving/loading class for common processors. N)Path)AnyDictListOptionalTuple TypedDictUnion)custom_object_save)ChannelDimensionis_valid_imageis_vision_available)PILImageResampling)PaddingStrategyPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategy) PROCESSOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_funcdirect_transformers_import download_urlis_offline_mode is_remote_urlloggingrFeatureExtractionMixinImageProcessingMixin) AutoTokenizerAutoFeatureExtractorAutoImageProcessor) ceZdZUdZeeeeeeeefe d<eeeeeeefe d<eeeeeeeefe d<ee e d<ee e e fe d<ee e e fe d<eee d<eee d <ee e d <eee d <ee e d <ee e d <ee e d<ee e d<ee e d<ee e d<ee e d<ee e d<y) TextKwargsa Keyword arguments for text processing. For extended documentation, check out tokenization_utils_base methods and docstrings associated. Attributes: add_special_tokens (`bool`, *optional*) Whether or not to add special tokens when encoding the sequences. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*) Activates and controls padding. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*): Activates and controls truncation. max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. stride (`int`, *optional*): If set, the overflowing tokens will contain some tokens from the end of the truncated sequence. is_split_into_words (`bool`, *optional*): Whether or not the input is already pre-tokenized. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. return_overflowing_tokens (`bool`, *optional*): Whether or not to return overflowing token sequences. return_special_tokens_mask (`bool`, *optional*): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*): Whether or not to return `(char_start, char_end)` for each token. return_length (`bool`, *optional*): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*): Whether or not to print more information and warnings. padding_side (`str`, *optional*): The side on which padding will be applied. text_pair text_targettext_pair_targetadd_special_tokenspadding truncation max_lengthstrideis_split_into_wordspad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbose padding_sideN)__name__ __module__ __qualname____doc__rr rrr__annotations__boolstrrrintP/var/www/html/venv/lib/python3.12/site-packages/transformers/processing_utils.pyr*r*Ps#Ji):DOTRcMddeffy"3T)_dK\F]]^^uY0A4 ?TXYjTk%klmm & 4o- ..dC!3344  SM!$'  %#D>)#D>)'~- (.$TN*D>! d^3-rFr*F)totalcTeZdZUdZeeed<eeee fed<ee ed<eeee fed<ee de fed<eeed<ee ed <eeed <ee e e e fed <ee e e e fed <eeed <eeee fed<eeed<ee ed<ee ee fed<y) ImagesKwargsaW Keyword arguments for image processing. For extended documentation, check the appropriate ImageProcessor class methods and docstrings. Attributes: do_resize (`bool`, *optional*): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. crop_size (`Dict[str, int]`, *optional*): Desired output size when applying center-cropping. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. pad_size (`Dict[str, int]`, *optional*): The size `{"height": int, "width" int}` to pad the images to. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. do_resizesize size_divisor crop_sizerresample do_rescalerescale_factor do_normalize image_mean image_stddo_padpad_sizedo_center_crop data_formatinput_data_formatN)r=r>r?r@rrBrArrCrDr floatrr rErFrGrJrJs#J~ 4S> ""3-S#X''u136788UO#4. ud5k1233eT%[0122 TNtCH~&&TN"*++c+;&; <==rFrJceZdZUdZeeed<eeee fed<ee ed<eded<eeed<ee ed<eeed <ee e e e fed <ee e e e fed <eeed <eeed <ee ed<ee ee fed<y) VideosKwargsa Keyword arguments for video processing. Attributes: do_resize (`bool`): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. rKrLrMrrOrPrQrRrSrTrUrWrXrYN)r=r>r?r@rrBrArrCrDrZr rr rErFrGr\r\s@~ 4S> ""3-+,,UO#4. ud5k1233eT%[0122 TNTN"*++c+;&; <==rFr\ceZdZUdZeeed<eedee edeee fed<eee e e fed<eeed<ee ed<eeed<ee ed <y ) AudioKwargsa Keyword arguments for audio processing. Attributes: sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_attention_mask (`bool`, *optional*): Whether or not [`~ASTFeatureExtractor.__call__`] should return `attention_mask`. sampling_ratez np.ndarray raw_speechr/r1r0r4r6N) r=r>r?r@rrDrAr rrZrBrCrrErFrGr^r^s:C= |T%[$|:LdSWX]S^N__`aa eD#67 88   %#D>)rFr^c(eZdZUeeeefed<y) CommonKwargsreturn_tensorsN)r=r>r?rr rCrrArErFrGrbrbsU3 ?344rFrbceZdZUdZiej Zeed<iej Zeed<ie j Z e ed<ie j Z e ed<ie j Ze ed<y)ProcessingKwargsa' Base class for kwargs passing to processors. A model should have its own `ModelProcessorKwargs` class that inherits from `ProcessingKwargs` to provide: 1) Additional typed keys and that this model requires to process inputs. 2) Default values for existing keys under a `_defaults` attribute. New keys have to be defined as follows to ensure type hinting is done correctly. ```python # adding a new image kwarg for this model class ModelImagesKwargs(ImagesKwargs, total=False): new_image_kwarg: Optional[bool] class ModelProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: ModelImagesKwargs _defaults = { "images_kwargs: { "new_image_kwarg": False, } "text_kwargs": { "padding": "max_length", }, } ``` For Python 3.8 compatibility, when inheriting from this class and overriding one of the kwargs, you need to manually update the __annotations__ dictionary. This can be done as follows: ```python class CustomProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: CustomImagesKwargs CustomProcessorKwargs.__annotations__["images_kwargs"] = CustomImagesKwargs # python 3.8 compatibility ```python common_kwargs text_kwargs images_kwargs videos_kwargs audio_kwargsN)r=r>r?r@rbrArfr*rgrJrhr\rir^rjrErFrGreres#J#  & &#M<  $ $K#  & &#M<#  & &#M<!  % %!L+rFrecheZdZUdZddgZdgZgZeee d<dZ dZ dZ gZ eee d<dZd eeeffd Zd efd Zd eeej*ffd ZdZd'defdZedeeej*fd eeeefeeefffdZedeeeffdZ d(dedeed eeeffdZ e d)deeej*fdeeeej*fdededeeeefdef dZ!ed*dZ"edZ#e$d Z%e&d!Z'd"Z( d+d#eeeeefdeed$ed efd%Z)d&Z*y),ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizer chat_templateoptional_call_argsN valid_kwargsc L|jD]}t|||j|d!|D]}||jvst d|dt ||jD]\}}||vrt d|d|||<t |t |jk7rJtdt |jddj|jdt |d|jD]\}}t||d }tj||}t|trtd |D}ntt|}t||s(t d t!|j"d |d |dt|||y)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: , z. Got z arguments instead._classc3BK|]}|tt|ywNgetattrtransformers_module.0ns rG z*ProcessorMixin.__init__..}s$j\]\iW-@!%D$jsz Received a z for argument z, but a z was expected.)optional_attributessetattrpop attributes TypeErrorziplen ValueErrorjoinitemsryAUTO_TO_BASE_CLASS_MAPPINGget isinstancetuplerztyper=) selfargskwargsoptional_attributekeyargattribute_name class_name proper_classs rG__init__zProcessorMixin.__init__bs#'":": T  D,fjj9KT.R S T GC$//)">se1 EFF G$'tT__#= - C'"CNCSST UVV),~&  - v;#doo. .*3t+?*@ TYYW[WfWfMgLhhnt9+02  $*<<> / NC .)9'@AJ377 JOJ*e,$$jj$jj &':JG c<0!$s)"4"4!5^NCSS[\f[gguv D.# . /rFreturncjtj|j}tj|j }|j }|Dcgc]}||jjvs|}}|dgz }|jDcic] \}}||vs ||}}}|jj|d<d|vr|d=d|vr|d=d|vr|d=d|vr|d=|jDcic]1\}}t|ts|jjdk(s||3}}}|Scc}wcc}}wcc}}w)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this processor instance. auto_mapprocessor_classrnimage_processorrmroBeamSearchDecoderCTC) copydeepcopy__dict__inspect signaturer parameters __class__rrr=rr)routputsig attrs_to_savexkvs rGto_dictzProcessorMixin.to_dictsEt}}- . $1XqQdnn>W>W5WX X*% #)<<>H41aQ-5G!Q$HH$(NN$;$; ! & {#  &() & (*+ f $'   1q.1Q[[5I5IMc5c qD   1YI sD$-D$ D)D))6D/cX|j}tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. Tindent sort_keys )rjsondumps)r dictionarys rGto_json_stringzProcessorMixin.to_json_strings'\\^ zz*Q$?$FFrFjson_file_pathct|dd5}|j|jdddy#1swYyxYw)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this processor instance's parameters will be saved. wutf-8encodingN)openwriter)rrwriters rG to_json_filezProcessorMixin.to_json_files<.# 8 0F LL,,. / 0 0 0s 8Ac |jDcgc]}d|dtt||}}dj|}|jj d|d|j Scc}w)Nz- z: rz: z )rreprryrrr=r)rnameattributes_reprs rG__repr__zProcessorMixin.__repr__svPTP_P_`RvRWT4-@(A'BC``))O4..))*#o->d4CVCVCXBYZZas"A0 push_to_hubc |jdd}| This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_id)config_set_processor_classrchat_template.jinjachat_template.jsonsave_raw_chat_templateFrrrzchat template saved in rorrrrzprocessor saved in )rr)(rwarningswarn FutureWarningrrosmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classrryrr init_kwargsappendr hasattrrrr=save_pretrainedrrrrorrloggerinforrsetkeysr_upload_modified_files)rsave_directoryrrrrrfiles_timestampsrattrsaconfigs attributeoutput_processor_fileoutput_raw_chat_template_fileoutput_chat_template_fileprocessor_dictrchat_template_json_strings rGrzProcessorMixin.save_pretraineds0 $4d;  % MME zz'4(4 l-F7O NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     'IMY~WT>2YEYafg\]A7N)O UVVgGg NN4 t^G D"oo 6Nn5Iy"89..t~~/F/FG  % %n 5  6    '"&// :#D.9 i)@A!--j9 :!# ^^ L(* ^EZ([%$&GGLLAU$V!    )zz2E:7wO5SYLL!3!345 56S5TUVJJ1C1CDQZ^_bff*3S7K|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d} |jd d } |jd d} |jd d} d| d} | | | d<tr|stjdd}t |}t j j|}t j j|r$t j j|t}t j j|r |}d}d}d}nmt|r|}t|}d}d}nPt}d}d} t||||||||| | | d }t||||||||| | | d }t||||||||| | | d }|-t!|dd5}|j#}ddd|d<nF|Dt!|dd5}|j#}dddt%j&d}||d<|i|fS t!|dd5}|j#}dddt%j&}|rtjd |ntjd d!|d|vr|dtj+d"|s,d#|vrt-|d#||d#<d$|vrt/|d$||d$<||fS#t$rt$rtd|d|dtdwxYw#1swYNxYw#1swY,xYw#1swYxYw#t$j($rtd|dwxYw)%a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a processor of type [`~processing_utils.ProcessingMixin`] using `from_args_and_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the processor object. cache_dirNforce_downloadFresume_downloadproxiesrlocal_files_onlyrevision subfolder_from_pipeline _from_auto processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueTrr) rrrrrr user_agentrr%_raise_exceptions_for_missing_entrieszCan't load processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerrrroz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at zChat templates should be in a 'chat_template.jinja' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.rcustom_pipelines)rrrrrCrrisdirrrisfiler rrEnvironmentError ExceptionrreadrloadsJSONDecodeError warning_oncerr)clsrrrrrrrrrr from_pipelinerris_localprocessor_fileresolved_processor_fileresolved_chat_template_fileresolved_raw_chat_template_filechat_template_fileraw_chat_template_filereaderrotextrs rGget_processor_dictz!ProcessorMixin.get_processor_dict7sA$JJ{D1 $4e< **%6=**Y- 7D)!::&8%@::j$/JJ{B/  #3T:  **\59#.?S  $+8J' (  %5 KKE F# (+,I(J%77==!>? 77==6 7WW\\*GXN 77>>7 8&C #*. '.2 +H 8 9:N&23P&Q #*. '.2 ++N!5 %: "< *51"'#1#$3%5)%':? +'$/:1&'#1#$3%5)%':? /+3>1*'#1#$3%5)%':? 3/8 + 65sWM .QW &  .&3F? # ( 413I %V{{} % JJt,_=M&3F? # # *v:  -sWE %{{} %!ZZ-N  KK56M5NO P KK5n5E_UlTmn o n ,1P1\   ^  ^+-G":.0M.z*"^35W"#568U612v%%$  &01N0OP99V8WX//=.>eE  . . % % % %## "45L4MMhi  sI!AL$6M%M "M90M-M9$,MM M*-M62M99#Nrc |j}|jdd}|jdd}d|vr|d=d|vr|d=|j||j}||i|}| t |d|t |j D]+}t||st |||j|-|j|tjd||r||fS|S) a Instantiates a type of [`~processing_utils.ProcessingMixin`] from a Python dictionary of parameters. Args: processor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~processing_utils.ProcessingMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the processor object. Returns: [`~processing_utils.ProcessingMixin`]: The processor object instantiated from those parameters. return_unused_kwargsFroNrr)processor_configrqz Processor ) rrvalidate_init_kwargsrqrrrrupdaterr) rrrrrro unused_kwargsrrs rGfrom_args_and_dictz!ProcessorMixin.from_args_and_dicts"(,,.%zz*@%H ?D9   .01  'z*00._b_o_o0p 00  $ I >v{{}% 9Cy#& 3 38 9  m$ j ,- f$ $ rFModelProcessorKwargstokenizer_init_kwargsc iiiiid}iiiiidt}D]}|jj|ij|<|j|jj D]@}||vst |j|rt|j|n||}|||<B|jt|t|z } |D]}|j|jj D]n}||vr0||j|d} | dk7r/|| vr+td|d|d||vr|j|d} nd} | dk7sV| |||<|j|ptfd|DrT|jD]@\}} |vs | jD]#\} } | |vs | ||| <|j| %BnZ|D]U}||vs||jdjj vr |||d|<=tj!d|d W|D]}||j|d|S) a Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance. The order of operations is as follows: 1) kwargs passed as before have highest priority to preserve BC. ```python high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"} processor(..., **high_priority_kwargs) ``` 2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API. ```python processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}}) ``` 3) kwargs passed during instantiation of a modality processor have fourth priority. ```python tokenizer = tokenizer_class(..., {"padding": "max_length"}) image_processor = image_processor_class(...) processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call ``` 4) defaults kwargs specified at processor level have lowest priority. ```python class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False): _defaults = { "text_kwargs": { "padding": "max_length", "max_length": 64, }, } ``` Args: ModelProcessorKwargs (`ProcessingKwargs`): Typed dictionary of kwargs specifically required by the model passed. tokenizer_init_kwargs (`Dict`, *optional*): Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults. Returns: output_kwargs (`Dict`): Dictionary of per-modality kwargs to be passed to each modality-specific processor. )rgrhrjrirf __empty__zKeyword argument z+ was passed two times: in a dictionary for z and as a **kwarg.c3&K|]}|v ywrwrE)r|rdefault_kwargss rGr~z/ProcessorMixin._merge_kwargs..{s7sn$7srfzKeyword argument `zA` is not a valid argument for this processor and will be ignored.)r _defaultsrrrArrrnryrrraddanyrrr)rr"r#r output_kwargs used_keysmodality modality_keyvaluenon_modality_kwargs kwarg_valuesubdictsubkeysubvaluerr's @rG _merge_kwargszProcessorMixin._merge_kwargss^    E ' CH';'E'E'I'I(TV'W'\'\'^N8 $ 4 D DX N ^ ^ c c e C #88#4>><@  =2<@ >CN8,\: C C ^,"&kC ,>>% 0H 4 D DX N ^ ^ c c e 0 v%"("2"6"6|["QK"k1lFY6Y(/ ~>33;*FM(3F;%MM&12 2 i'2BB?Scchhjj>DSk o6s;++05vw  & KH ( # * *=+I J KrFrrrrrc ,||d<||d<||d<||d<|jdd}|)tjdt| t d|}|||d <|j |fi|} |j |fi|\} }|j| | fi|S) a[ Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rrrrrNrrr)rrrrr_get_arguments_from_pretrainedrr!) rrrrrrrrrrrs rGfrom_pretrainedzProcessorMixin.from_pretrainedsN({#1 %5!"%z$4d;  % MME   l#E  #F7O1s112OZSYZ!7!7!78U!`Y_!`%s%%dNEfEErFct|ts |j}ddlmcm}t ||st|d||_y)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) rrCr=transformers.models.automodelsautorrr)r auto_class auto_modules rGregister_for_auto_classz&ProcessorMixin.register_for_auto_classsC *c*#,,J66{J/ |+FGH H$rFc <g}|jD]}t||d}t|tr7td|D}|j dd}|r |d|d}n|d}ntt |}|j |j|fi||S)Nruc3DK|]}|tt|ndywrwrxr{s rGr~z@ProcessorMixin._get_arguments_from_pretrained..s#o_`1=(;Q ?VZ Zos use_fastTr r)rryrrrrzrr8) rrrrrrclassesrBattribute_classs rGr7z-ProcessorMixin._get_arguments_from_pretraineds!nn bN (8&?@J*e,odnoo!::j$7 6&-ajO&-ajO")*=z"J KK7778U`Y_` a b rFcNt||jd}t|ddS)Nrmodel_input_names)ryr)rfirst_attributes rGrFz ProcessorMixin.model_input_namess'!$(:;(;TBBrFc|j}i}t|t|z }|r=dj|}tj d|d|Dcic]}||| }}|Scc}w)NrtzISome kwargs in processor config are unused and will not have any effect: z. )rrrrwarning)rrqkwargs_from_configr  unused_keysunused_key_strrs rGrz#ProcessorMixin.validate_init_kwargss-224 ,-L0AA !YY{3N NN[\j[kkmn >IIQ 0 33IMIJs A+c xt|rtjdt|t|jkDrJt dt|jddj |jdt|dt ||jDcic]\}}|| c}}Scc}}w)a Matches optional positional arguments to their corresponding names in `optional_call_args` in the processor class in the order they are passed to the processor call. Note that this should only be used in the `__call__` method of the processors with special arguments. Special arguments are arguments that aren't `text`, `images`, `audio`, nor `videos` but also aren't passed to the tokenizer, image processor, etc. Examples of such processors are: - `CLIPSegProcessor` - `LayoutLMv2Processor` - `OwlViTProcessor` Also note that passing by position to the processor call is now deprecated and will be disallowed in future versions. We only have this for backward compatibility. Example: Suppose that the processor class has `optional_call_args = ["arg_name_1", "arg_name_2"]`. And we define the call method as: ```python def __call__( self, text: str, images: Optional[ImageInput] = None, *arg, audio=None, videos=None, ) ``` Then, if we call the processor as: ```python images = [...] processor("What is common in these images?", images, arg_value_1, arg_value_2) ``` Then, this method will return: ```python { "arg_name_1": arg_value_1, "arg_name_2": arg_value_2, } ``` which we could then pass as kwargs to `self._merge_kwargs` zPassing positional arguments to the processor call is now deprecated and will be disallowed in v4.47. Please pass all arguments as keyword arguments.zExpected *at most* zK optional positional arguments in processor callwhich will be matched with  z+ in the order they are passed.However, got z positional arguments instead.Please pass all arguments as keyword arguments instead (e.g. `processor(arg_name_1=..., arg_name_2=...))`.)rrrrprrr)rr arg_valuearg_names rG'prepare_and_validate_optional_call_argsz6ProcessorMixin.prepare_and_validate_optional_call_argssX t9 MMB  t9s4223 3%c$*A*A&B%CD..1hht7N7N.O-PQ #D {+}}  @C4I`I`?ab(; 8)#bbbs% B6 conversationtokenizec |$|j |j}n td|jj|f||d|S)a Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input conversations to turn them into a single tokenizable string. Args: conversation (`List[Dict, str, str]`): The conversation to format. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's chat template is used. tokenize (`bool`, *optional*, defaults to `False`): Whether to tokenize the output or not. **kwargs: Additional keyword arguments zNo chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.)rorS)rorrnapply_chat_template)rrRrorSrs rGrUz"ProcessorMixin.apply_chat_templateKsd.  !!- $ 2 2  m 2t~~11  (5 LR  rFc<|jj|dS)a Post-process the output of a vlm to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. Returns: `List[str]`: The decoded text. T)skip_special_tokens)rn batch_decode)rgenerated_outputss rGpost_process_image_text_to_textz.ProcessorMixin.post_process_image_text_to_textos ~~**+r?r@rrrprrCrAfeature_extractor_classtokenizer_classrrqrrrrrr rPathLikerrrBr classmethodrrr!rerr5r8r?r7propertyrF staticmethodrrQrUrZrErFrGrlrlSs&{3J*+$&S &"OK L$s) $/L&c3h&P G G 05bkk1A+B 0[ j'4j'Xj&,1#r{{2B,Cj& tCH~tCH~- .j&j&X+d38n++`15. (~ c4i B8<$!&,0=F',S"++-='>=FE#r{{"234=F =F  =F c4i() =F=F=F~%%2"CC  8cz(, " Dc3h01"  }"  " " H XrFrlc dtfd  fdfdd}||}|}||}|}|r|r||fS||s||s|r|rtjd||fStd)a For backward compatibility: reverse the order of `images` and `text` inputs if they are swapped. This method should only be called for processors where `images` and `text` have been swapped for uniformization purposes. Note that this method assumes that two `None` inputs are valid inputs. If this is not the case, it should be handled in the processor's `__call__` method before calling this method. rcHt|txr|jdS)Nhttp)rrC startswith)vals rGis_urlz1_validate_images_text_input_order..is_urls#s#>v(>>rFc~t|ttfr|D] }|r yyt|s |syy)NFT)rlistrr)imgsimg$_is_valid_images_input_for_processorrhs rGrmzO_validate_images_text_input_order.._is_valid_images_input_for_processorsF dT5M * !;C@  ! !&&,rFct|tryt|ttfrt |dk(ry|D] }|cSy)NTrF)rrCrjrr)tt_s"_is_valid_text_input_for_processors rGrqzM_validate_images_text_input_order.._is_valid_text_input_for_processorsI a  D%= )1v{ ?9#>> ?rFc||xs|duSrwrE)input validators rG _is_validz4_validate_images_text_input_order.._is_valids05D=0rFzYou may have used the wrong order for inputs. `images` should be passed before `text`. The `images` and `text` inputs will be swapped. This behavior will be deprecated in transformers v4.47.zGInvalid input type. Check that `images` and/or `text` are valid inputs.)rBrrr) imagesrruimages_is_validimages_is_text text_is_validtext_is_imagesrmrqrhs @@@rG!_validate_images_text_input_orderr{~s?t?  1 (LMO7?Nd$FGM9$?N=t| >t|Tbgu v V| ^ __rFrr\zprocessor files)object object_class object_files)Ar@rrrrsystypingrpathlibrrrrrrr r numpynptyping_extensionsdynamic_module_utilsr image_utilsr rrrtokenization_utils_baserrrrrutilsrrrrrrrrrrr r! get_loggerr=r__file__parentrzr version_infoUnpackr*rJr\r^rbrerlr{rformatrErFrGrs  EEE4NN/       H %1h1F1FG/40 w ]]F  % %F7 %7 t4>9E4>n->9E->`$*)5$*N59E54z<{L`e4nh X^h XV7`t'~'A'AB%%1)7)C)C)K)K)R)RGX*S*N&2rF