sg``dZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl mZddlmZd d lmZmZmZmZerdd lmZerddlZej2eZe d z Z d diZdedefdZdZddefdZ ddede!defdZ"dZ#dZ$defdZ%ee GddeZ&y)z" Fast tokenizer class for Nougat. N)partial)Pool)ListUnion)INIT_TOKENIZER_DOCSTRING)PreTrainedTokenizerFast)add_end_docstrings)is_levenshtein_availableis_nltk_availableloggingrequires_backends)ratiou tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. tokenizer_fileztokenizer.jsontextreturnctjdd|tj}tjdd|tj}tjdd|tj}|jdd }|jd d jd d }tjd d|}tjdd|}tjdd|tj}|S)a1 Make text compatible with Markdown formatting. This function makes various text formatting adjustments to make it compatible with Markdown. Args: text (`str`): The input text to be made Markdown-compatible. Returns: `str`: The Markdown-compatible text. z%^\(([\d.]+[a-zA-Z]?)\) \\\[(.+?)\\\]$z\[\2 \\tag{\1}\]flagsz%^\\\[(.+?)\\\] \(([\d.]+[a-zA-Z]?)\)$z\[\1 \\tag{\2}\]z3^\\\[(.+?)\\\] \(([\d.]+[a-zA-Z]?)\) (\\\[.+?\\\])$z\[\1 \\tag{\2}\] \3z\. z. z\bm{z\mathbf{z{\\bm z\\mbox{ ?\\boldmath\$(.*?)\$}z \\mathbf{\1}z^((?:http|ftp|https):\/\/(?:[\w_-]+(?:(?:\.[\w_-]+)+))(?:[\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-]))z[\1](\1)z```\s*(.+?)\s*```z ```\n\1\n```)resubMreplaceS)rs f/var/www/html/venv/lib/python3.12/site-packages/transformers/models/nougat/tokenization_nougat_fast.pymarkdown_compatibler8s 66: dd  D << %D << - 5 5i MD 662OT JD 66i  D 66&RTT JD Kc 4d}tttj||tjtj zD]H}|j \}}|jddz}|jdj|}d}|jd|dd}|jddz}nd}{|d|||d} } t|D]\} } d} | jjd\}}}|s.tjd |tjtj zr|jd } || dkDrd ndd | zz| dkDs|dk(r|n|z| jzz }| dk(rd } | |z| z}K|S) a Normalize lines in the given text that resemble list items. The function looks for lines that start optionally with '-' or '*', possibly followed by Roman numerals or digits indicating nesting levels. The function reformats such lines to make them more structured. Args: generation (str): The input text containing lines that need to be normalized. Returns: str: The input text with the list-like lines normalized. Note: The function uses regular expressions to identify and reformat the list-like lines. The patterns capture optional bullet points, nesting levels indicated by numerals, and the actual list item content. The normalization adjusts the bullet point style and nesting levels based on the captured patterns. zX(?:^)(-|\*)?(?!-|\*) ?((?:\d|[ixv])+ )?.+? (-|\*) (((?:\d|[ixv])+)\.(\d|[ixv]) )?.*(?:$)rr  rNz^[\dixv]+((?:\.[\dixv])?)+$.  )reversedlistrfinditerIrspangroupsplit enumeratestrip partitionmatchcount) generationpatternr/startstopdelimsplits replacementdelim1prepostiitemlevelpotential_numeral_rests rnormalize_list_like_linesrAbs,jG$r{{7JbddRTTkRST.jjl t A$Q%%e, ;;q> %ABZF[[^c)FF v& 45(9T ( GAtE)-)?)?)D & q$xx68IQSQUQUXZX\X\Q\])//4 QB4%<8QUeWXjE^dehlhrhrhtt K  2:D;&- ?.B rcNt|t|D] }||dvs |cSy)z Find the index of the next punctuation mark. Args: text (`str`): String to examine start_idx (`int`, *optional*) Index where to start )r"?!r#N)rangelen)r start_idxr;s rfind_next_punctuationrHs59c$i ( 7+ +H rmin_lencB|j}t|}|d|zkr|Sd}t|t|dz D]8}d}td|D] }|||z |z dz |||z dz k7sd}n|s7|}:||S|| d}|} |} | j |r| d| } | d| } | j |r|t| d} | } t |t| } t |dddt| }| r|r||| }|d| dz} || vrnnH|dt| }|S)a Attempt to truncate repeating segments in the input string. This function looks for the longest repeating substring at the end of the input string and truncates it to appear only once. To be considered for removal, repetitions need to be continuous. Args: text (`str`): The input raw prediction to be truncated. min_len (int): The minimum length of the repeating segment. Returns: `str`: The input string with repeated segments truncated. NTrr!F)lowerrFrEintendswithrH)rrI text_lower text_lengthmax_repetition_lengthrepetition_lengthsamer;lcssubstituted_textsubstituted_text_lowerrepeating_tailsubstituted_text_lower_out sentence_endsentence_startsentencetext_outs rtruncate_repetitionsr^s Jj/KQ[  !"7C a,@A 6q+, A+(99A=ABjQ\_`Q`cdQdFee  $5 ! 6$ ++, -C' ) )# .+,C.C-CD!78O:O9O!P ! ) )# .  $: ; =>N"8 ,Z=W9XY .z$B$/?E_A`a N!.>H)34FlQ6F)G &>)  5c456H Orc|d}t|tr||Sg}|D]}|j|||S)NcLtjdd|jS)Nz(?:[\d_]|\*\*)r )rrr-)ss r_cleanzremove_numbers.._cleansvv'Q/5577r) isinstancestrappend)linesrboutls rremove_numbersrisF8%e} C  6!9 Jrc\tjt|}tt|dz D]}|dz}||s-|t|dz kr|dz }||s|t|dz krt||dksLt||dkDs^t||dkspt||dkDs||j dr||||k(st ||||dkDsd|||tj |d}g}t|dk(r|Sd}ttj|dkDD])\}}|s |j||||dzf|dz}+|j|||ddzf|D cgc]} | d| dz d kDs| c} Scc} w) a5 Get slices of text based on specific criteria within the lines. This function identifies and returns slices of text from the input lines based on certain conditions. These conditions were chosen by the Nougat authors: - The slice is less than 200 characters long. - The slice is more than 3 characters long. - The slice does not start with "[MISSING_PAGE". - The slice is either the same as the next slice or the ratio of the two in terms of Levensthein distance is greater than 0.9. Args: lines (`List[str]`): The list of lines containing the text. clean_lines (`List[str]`): A cleaned version of the text (without numbers). Returns: `List[tuple]`: A list of tuples representing the start and end indices of text slices. r!r z [MISSING_PAGE?rrKrL) npzerosrFrE startswithrwherer,diffre) rf clean_linesindicesr;jidsslicesj0xslis r get_slicesr{s,hhs5z"G 3u:> "  Ea.QUa%7 FAa.QUa%7  A # %KN#a'KN#c)KN#a'N--o>Q;q>1U;q>;WX>5Z]`5`GAaL  ((7 A C F 3x1} B"''#,*+1 MM3r7CFQJ/ 0QB MM3r7CGaK()! :CSVc!f_r%9C :: :s F)"F)c L||d}t|}d}ttd|ddz td|ddz dD]}||s ||dk(r||d<nt|t ||dks4|dz|d<t |td|dz j dd}t |d t |zk\rt||dkr||d<d }nttt ||dtt ||ddzD]$}t|t ||dks||d<nt ||dkrt |dz |d<d j||d|ddz}t||ddz t||d} } t| \} } | jrt| \} } | jrt| \} }|jrt| \} }|jr| |k7rn |rd|vrd|j ddz} t ||d z dz }|dkDr|d | }|jS#t$rY[wxYw#t$rY|jSwxYw) a Remove a slice of text from the lines based on specific criteria. This function identifies a slice of text within the lines and removes it based on certain conditions. Args: lines (list of str): The list of lines containing the text. clean_text (list of str): A cleaned version of the text (without numbers). slice (tuple): A tuple representing the start and end indices of the slice to be removed. Returns: str: The removed slice of text as a single string. rFr!rLz ## Referencesrlz* [g?Tr#N)r&rEmaxrrir.rFminjoinr,next isnumeric StopIterationUnboundLocalErrorr-)rf clean_textslicebasesectioncheck_start_flagline_idx potential_ref to_deleteiteraiterbiaaibbdeltas rremove_slice_from_linesr1s eAh D5kG#aqA.AuQx!|0DbI X  ?o -!GAJ  4h8 9C ?!AGAJ*5Q1 1E+F+P+PQV+WXZ+[\M=!TCI%55%m:TWZ:Z% #   #c%j%(3SUU1XPQ\5RS ~eHo6 7# =!GAJ  5zWQZZ!^  % WQZ!^<=IU71:>23YuWQZ?P5Q5E  5kGR++-u+Q++-5kGR++-u+Q++-Av EY.I//6r::  E'!*%&+a/ 19!'E6*I ??       ??  s0'"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskNc Bt |d|||||||d|||_y)N) vocab_filerclean_up_tokenization_spaces unk_token bos_token eos_token pad_token)super__init__r) selfrrrrrrrkwargs __class__s rrzNougatTokenizerFast.__init__s?  !))E  %rrrc0|jd}t|dk(ryt|}t||}g}|D]}|j t ||| t |D]}|j|d}tjdd|}|S)av Remove hallucinated or missing references from the text. This function identifies and removes references that are marked as missing or hallucinated from the input text. Args: text (`str`): The input text containing references. Returns: `str`: The text with hallucinated references removed. r#rr z [MISSING_PAGE_POST] z,## References\n+\[MISSING_PAGE_POST(:\d+)?\]z [MISSING_PAGE_POST\1]) r+rFrir{rerr%rrr)rrrfrsrwrrs rremove_hallucinated_referencesz2NougatTokenizerFast.remove_hallucinated_referencess 4  u:?$U+ E;/  QE   4UKO P Q!), JI<< +HID Jvv ; (    rr1c|jdD]Q}|jddkDs)|jddkDs|jddkDs@|j|d}S|jd d }|jd d }|jd d}tjdd|tj }|jdd}|jdd}|S)a Takes a generated string and fixes tables/tabulars to make them match the markdown format needed. Args: generation (str): The generated text to be postprocessed. Returns: str: The postprocessed text. Example: ```python correct_tables("\begin{table} \begin{tabular}{l l} & \ \end{tabular} \end{table}") "\begin{table} \begin{tabular}{l l} & \ \end{tabular} \end{table}" ``` r#z\begin{tabular}rmz \multicolumn<&ir z\begin{table} \begin{tabular}z\begin{table} \begin{tabular}z\end{tabular} \end{table}z\end{tabular} \end{table}z\end{table} Tabz\end{table} Tabz(^.+)\\begin{tabz\1\n\\begin{tabrz(\begin{tabular}{l l} & \\ \end{tabular}z \begin{tabular}{} \end{tabular})r+r0rrrr)rr1rhs rcorrect_tablesz"NougatTokenizerFast.correct_tabless$!!$' 7Aww)*R/177?3Kb3PTUT[T[\_T`cfTf'//26  7  ''(IKmn ''(EGef ''(:E VVFJ^`^b^bc   & 9   $r)9)9#)>)I)I#)NSVW\S]`aSa KKLuUWyX Y5":.J)*5 88D VVBB Z\Z^Z^_ VV5w RTRVRVW VV0"jA VV G   VVC]T^_ VV J   VVCRT .z:   z * & J 88OZ 0 # J  " "#9 :*,J   d #B ' 2 23K L#f,J "&,,S1"5  1 1 7 7 99#%J ((4 ''(;=NO VV N   VV;RL VV/ZrttL VVJJbddC VVIvz: &z2 2 3 "c!  "sAM MM num_workerscdt|ddgt|tro|Lt|trzzz)T)TN)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesslow_tokenizer_classrrdrrboolrrrrNr __classcell__)rs@rrrrsB*$&67%*%.33rs  IH1\\!   H %&'78'c'c'T9x$AsASA#AH .;b>>B,-S1S.Sr