sg%dZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z mZddlmZmZdd lmZGd d e d Zej*eZGdde Zy)z Processor class for Donut. N)contextmanager)ListOptionalUnion) ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingceZdZiZy)DonutProcessorKwargsN)__name__ __module__ __qualname__ _defaults]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/donut/processing_donut.pyrrsIrrF)totalc eZdZdZddgZdZdZdfd Z ddede e e e e e efd eefd Zd Zd Zed ZddZedZedZxZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`], *optional*): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerAutoImageProcessor AutoTokenizerc d}d|vr+tjdt|jd}||n|}| t d| t dt ||||j|_d|_ y)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr __class__s rr'zDonutProcessor.__init__9s  & ( MM  !' +> ? -<-H/N_  "HI I  AB B )4!%!5!5*/'rimagestextr+c |jdd}|rtjd|jr|j||fi|S| | t d|j tfd|jji|}||j|fi|d}|/|s||djdd |j|fi|d} |S| S d d <| d |d <|S) a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the doctsring of the above two methods for more information. legacyTa_Legacy behavior is being used. The current behavior will be deprecated in version 5.0.0. In the new behavior, if both images and text are provided, the default value of `add_special_tokens` will be changed to `False` when calling the tokenizer if `add_special_tokens` is unset. To test the new behavior, set `legacy=False`as a processor call argument.zBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs images_kwargs text_kwargsadd_special_tokensF input_idslabels) r$logger warning_oncer)r(r% _merge_kwargsrr init_kwargsr setdefault) r*r-r.audiovideosr+r0 output_kwargsinputs encodingss r__call__zDonutProcessor.__call__Ms0Hd+    \   * *)4))&$A&A A >dlab b***  "&.."<"<   )T))&SM/4RSF  f0m,778LeT&tL}]/KLI <M ^ (5F8 "+K"8F; Mrc:|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoder*argsr+s rrCzDonutProcessor.batch_decodes +t~~**D;F;;rc:|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoderDs rrGzDonutProcessor.decodes %t~~$$d5f55rc#Ktjdd|_|j|_d|j |_d|_yw)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r!r"r)rr(rr*s ras_target_processorz"DonutProcessor.as_target_processorsH   9 +/'!% !%!5!5*/'sA A c||jj}i}|r4tjd|tj}|n |j d}tj |}tjd|d|tj}|j }||j|d}n|j }tj |} tj |} tj| d| |tjtjz} | | j dj} d| vr3d| vr/|j| d | } | rt| dk(r| d } | ||<ntg||<| jd D]?} | j} | |vr| d d k(r | dddk(r| dd} ||j| At||dk(r ||d ||<||j|t|zdj}|ddd k(r|g|j|ddd | zS|r4t|r|r|gS|S|rgSd|iS)zS Convert a (generated) token sequence into an ordered JSON format. Nz zz(.*?)z text_sequence)rget_added_vocabresearch IGNORECASEgroupescapereplaceDOTALLstrip token2jsonlensplitappendfind)r*tokensrOrPoutput start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs rr^zDonutProcessor.token2jsons{  ..88:K))L&"--HK"##A&C))C.K T+a"8&"--PI%++-K  R8%OO- &(ii &<#$&IIi$8!))*+51B0CDfbmm^`^g^gNg&%mmA.446G(W-? $Ze f "5zQ(-a*/F3K&(s $+MM)$<5D#'::rs[ %((%HHC+5   H %$^$r