Ë ©sg\ ãó<—dZddlmZddlmZmZmZGd„d«Zy)z Tokenization utils for RoFormer.é)ÚList)ÚNormalizedStringÚPreTokenizedStringÚ normalizerscó:—eZdZd d„Zdededeefd„Zdefd„Z y) ÚJiebaPreTokenizerÚreturnNcó–—||_tjdddd¬«|_ ddl}||_y#t$r t d«‚wxYw)NFT)Ú clean_textÚhandle_chinese_charsÚ strip_accentsÚ lowercaserzkYou need to install rjieba to use RoFormerTokenizer. See https://pypi.org/project/rjieba/ for installation.)ÚvocabrÚBertNormalizerÚrjiebaÚ ImportErrorÚjieba)Úselfrrs úb/var/www/html/venv/lib/python3.12/site-packages/transformers/models/roformer/tokenization_utils.pyÚ__init__zJiebaPreTokenizer.__init__s]€ØˆŒ Ü&×5Ñ5ØØ!%ØØô  ˆÔð  Û ð ˆ øô ò ÜðIóð ð ús §3³AÚiÚnormalized_stringcó\—g}|jjt|«d¬«D]€\}}}||jvr|j |||«Œ*|j j |«j«}|D])}|sŒ|t|«z}|j |||«|}Œ+Œ‚|S)NF)Úhmm) rÚtokenizeÚstrrÚappendrÚ normalize_strÚsplitÚlen)rrrÚsplitsÚtokenÚstartÚendÚ token_lists rÚ jieba_splitzJiebaPreTokenizer.jieba_split(s¹€Øˆð"&§¡×!4Ñ!4´SÐ9JÓ5KÐQVÐ!4Ó!Wò $Ñ ˆE5˜#ؘŸ ™ Ñ"Ø— ‘ Ð/°°cÐ:Õ;à!×-Ñ-×;Ñ;¸EÓB×HÑHÓJ Ø'ò$EÚØ#¤c¨%£jÑ0˜ØŸ ™ Ð&7¸¸cÐ&BÔCØ #™ñ $ð $ð*ˆ óÚpretokcó:—|j|j«y)N)rr&)rr(s rÚ pre_tokenizezJiebaPreTokenizer.pre_tokenizeCs€Ø ‰ T×%Ñ%Õ&r')r N) Ú__name__Ú __module__Ú __qualname__rÚintrrr&rr*©r'rrrs5„óð"˜SðÐ5EðÈ$ÐO_ÑJ`óð6'Ð#5ô'r'rN)Ú__doc__ÚtypingrÚ tokenizersrrrrr/r'rúr3sðñ'åçHÑH÷.'ò.'r'