sgdZddlZddlmZmZddlmZmZmZer ddl Z ddl m Z eje Z GddeZy) z' Feature extractor class for MarkupLM. N) BatchFeatureFeatureExtractionMixin)is_bs4_availableloggingrequires_backends) BeautifulSoupc@eZdZdZfdZdZdZdZdefdZ xZ S)MarkupLMFeatureExtractorao Constructs a MarkupLM feature extractor. This can be used to get a list of nodes and corresponding xpaths from HTML strings. This feature extractor inherits from [`~feature_extraction_utils.PreTrainedFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. c >t|dgt|di|y)Nbs4)rsuper__init__)selfkwargs __class__s k/var/www/html/venv/lib/python3.12/site-packages/transformers/models/markuplm/feature_extraction_markuplm.pyrz!MarkupLMFeatureExtractor.__init__+s$( "6"c g}g}|jr|n |jjD]y}|jjd}|j j|j dt |k(rdnt fdt|dD|{|j|j||fS)NF) recursiverc32K|]\}}|us |yw)Nr).0ischilds r z6MarkupLMFeatureExtractor.xpath_soup..7s1e1Z[_dZd!1es ) nameparentparentsfind_allappendlennext enumeratereverse)relement xpath_tagsxpath_subscriptsr siblingsrs @r xpath_soupz#MarkupLMFeatureExtractor.xpath_soup/s "<<W^^mm FuzzUCH   ejj )  # ##h-'T1e (TU@V1e-e E     "+++rc^t|d}g}g}g}|jD]}t|tjj s(t |jtjjurXtj|j}|s~|j||j|\}} |j||j| t|t|k7r tdt|t|k7r td|||fS)Nz html.parserz3Number of doc strings and xtags does not correspondz3Number of doc strings and xsubs does not correspond)r descendants isinstancer r(NavigableStringtyper Taghtmlunescapestripr#r,r$ ValueError) r html_string html_codeall_doc_stringsstring2xtag_seqstring2xsubs_seqr(text_in_this_tagr)r*s rget_three_from_singlez.MarkupLMFeatureExtractor.get_three_from_single>s!+}=  ,, :G'3;;#>#>?'s{{>#'==#9#?#?#A '&&'78/3w/G, ,&&z2 ''(89 :  3#7 7RS S  3'7#8 8RS S1AAArc`d}t||D]\}}|d|z }|dk7s|d|dz }|S)N/r[])zip)rr)r*xpathtagnamesubss rconstruct_xpathz(MarkupLMFeatureExtractor.construct_xpath[sR -=> %MGT q ] "Eqy1TF!$ % rreturncfd}t|trd}n9t|ttfr#t |dk(st|dtrd}|st dt |dtt|ttfxrt|dt}|s|g}g}g}|D]t}|j|\}}} |j|g} t||| D])\} } } |j| | }| j|+|j| v||d}t|d}|S) a\ Main method to prepare for the model one or several HTML strings. Args: html_strings (`str`, `List[str]`): The HTML string or batch of HTML strings from which to extract nodes and corresponding xpaths. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **nodes** -- Nodes. - **xpaths** -- Corresponding xpaths. Examples: ```python >>> from transformers import MarkupLMFeatureExtractor >>> page_name_1 = "page1.html" >>> page_name_2 = "page2.html" >>> page_name_3 = "page3.html" >>> with open(page_name_1) as f: ... single_html_string = f.read() >>> feature_extractor = MarkupLMFeatureExtractor() >>> # single example >>> encoding = feature_extractor(single_html_string) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) >>> # batched example >>> multi_html_strings = [] >>> with open(page_name_2) as f: ... multi_html_strings.append(f.read()) >>> with open(page_name_3) as f: ... multi_html_strings.append(f.read()) >>> encoding = feature_extractor(multi_html_strings) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) ```FTrzQHTML strings must of type `str`, `List[str]` (batch of examples), but is of type .)nodesxpathsN)data tensor_type) r/strlisttupler$r6r1boolr=r#rCrGr)r html_strings valid_strings is_batchedrKrLr7r9r:r; xpath_stringsnodetag_listsub_list xpath_stringrMencoded_inputss r__call__z!MarkupLMFeatureExtractor.__call__csU`  lC ( M  tUm 4< A%LOS)I $ ""&|"4!5Q8  *\D%=AhzR^_`RacfGgi (>L' )KAEA[A[\gAh >O_.> LL )M,/Rb,c 3(h#33HhG $$\2 3 MM- ( )&1%4TBr) __name__ __module__ __qualname____doc__rr,r=rGrr\ __classcell__)rs@rr r !s+# ,B:T Trr )r`r3feature_extraction_utilsrrutilsrrrr r get_loggerr]loggerr rrrrfsI LAA!   H %V5Vr