+sg2VddlmZddlZddlmZddlmZmZddl m Z GddeZ y)) annotationsN)Dataset)NLTK_IMPORT_ERRORis_nltk_available) InputExamplec<eZdZdZdfddZdZdZed dZy) DenoisingAutoEncoderDataseta The DenoisingAutoEncoderDataset returns InputExamples in the format: texts=[noise_fn(sentence), sentence] It is used in combination with the DenoisingAutoEncoderLoss: Here, a decoder tries to re-construct the sentence without noise. Args: sentences: A list of sentences noise_fn: A noise function: Given a string, it returns a string with noise, e.g. deleted words c,tj|SN)r delete)ss m/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/datasets/DenoisingAutoEncoderDataset.pyz$DenoisingAutoEncoderDataset.s@[@b@bcd@ects2ttj|jj ||_||_yr )r ImportErrorrformat __class____name__ sentencesnoise_fn)selfrrs r__init__z$DenoisingAutoEncoderDataset.__init__s7 "/66t~~7N7NOP P"  rcZ|j|}t|j||gS)N)texts)rrr)ritemsents r __getitem__z'DenoisingAutoEncoderDataset.__getitem__s)~~d#4==#6"=>>rc,t|jSr )lenr)rs r__len__z#DenoisingAutoEncoderDataset.__len__!s4>>""rcNddlm}ddlm}||}t |}|dk(r|St j j||kD}t|dk(r"d|t j j|<|jt j||}|S)Nr) word_tokenize)TreebankWordDetokenizerT) nltkr#nltk.tokenize.treebankr$r nprandomrandsumchoice detokenizearray)text del_ratior#r$wordsn keep_or_notwords_processeds rr z"DenoisingAutoEncoderDataset.delete%s&Bd# J 6KiinnQ')3 { q /3K ((+ ,13>>rxx{?[\rN)rz list[str])g333333?) r __module__ __qualname____doc__rrr! staticmethodr rrr r s0 7f!?#  rr ) __future__rnumpyr'torch.utils.datartransformers.utils.import_utilsrr*sentence_transformers.readers.InputExamplerr r8rrr>s "$PC)')r