sgafdZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGddeZy ) z Processor class for Bros. )ListOptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypec" eZdZdZdgZdZdfd Z ddeee e ee e fde dee e e fdee e efd eed ed eed ee d ee de de de de de deee efdef dZdZdZedZxZS) BrosProcessora Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`, *optional*): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. tokenizer) BertTokenizerBertTokenizerFastc >| tdt| |y)Nz"You need to specify a `tokenizer`.) ValueErrorsuper__init__)selfrkwargs __class__s [/var/www/html/venv/lib/python3.12/site-packages/transformers/models/bros/processing_bros.pyrzBrosProcessor.__init__)s"  AB B #textadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturnc J|jd||||||||| | | | | ||d|}|S)z This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrr r!r"r#r$r%r&r'r(r)r*)r)rrrrrr r!r"r#r$r%r&r'r(r)r*rencodings r__call__zBrosProcessor.__call__/sW0"4>> 1!!1"7"7&?'A#9') ! &rc:|jj|i|S)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrs rr1zBrosProcessor.batch_decode\s +t~~**D;F;;rc:|jj|i|S)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder2s rr5zBrosProcessor.decodecs %t~~$$d5f55rcj|jj}ttj |SN)rmodel_input_nameslistdictfromkeys)rtokenizer_input_namess rr8zBrosProcessor.model_input_namesjs' $ @ @DMM"7899rr7)NTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributestokenizer_classrrr r rboolstrr r rintr rr/r1r5propertyr8 __classcell__)rs@rrrs^ JrLs,)(.wwS:NS:r