+sgddlmZddlmZddlmZddlZddlmZmZddl m Z ddl m Z Gdd ejZy) ) annotations)Iterable)AnyN)Tensornn)util)SentenceTransformerc\eZdZdejfdfd ZddZddZed dZ xZ S) MultipleNegativesRankingLossg4@c~t|||_||_||_t j |_y)ad This loss expects as input a batch consisting of sentence pairs ``(a_1, p_1), (a_2, p_2)..., (a_n, p_n)`` where we assume that ``(a_i, p_i)`` are a positive pair and ``(a_i, p_j)`` for ``i != j`` a negative pair. For each ``a_i``, it uses all other ``p_j`` as negative samples, i.e., for ``a_i``, we have 1 positive example (``p_i``) and ``n-1`` negative examples (``p_j``). It then minimizes the negative log-likehood for softmax normalized scores. This loss function works great to train embeddings for retrieval setups where you have positive pairs (e.g. (query, relevant_doc)) as it will sample in each batch ``n-1`` negative docs randomly. The performance usually increases with increasing batch sizes. You can also provide one or multiple hard negatives per anchor-positive pair by structuring the data like this: ``(a_1, p_1, n_1), (a_2, p_2, n_2)``. Then, ``n_1`` is a hard negative for ``(a_1, p_1)``. The loss will use for the pair ``(a_i, p_i)`` all ``p_j`` for ``j != i`` and all ``n_j`` as negatives. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - `Training Examples > Natural Language Inference <../../examples/training/nli/README.html>`_ - `Training Examples > Paraphrase Data <../../examples/training/paraphrases/README.html>`_ - `Training Examples > Quora Duplicate Questions <../../examples/training/quora_duplicate_questions/README.html>`_ - `Training Examples > MS MARCO <../../examples/training/ms_marco/README.html>`_ - `Unsupervised Learning > SimCSE <../../examples/unsupervised_learning/SimCSE/README.html>`_ - `Unsupervised Learning > GenQ <../../examples/unsupervised_learning/query_generation/README.html>`_ Requirements: 1. (anchor, positive) pairs or (anchor, positive, negative) triplets Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - :class:`CachedMultipleNegativesRankingLoss` is equivalent to this loss, but it uses caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. However, it is slightly slower. - :class:`MultipleNegativesSymmetricRankingLoss` is equivalent to this loss, but with an additional loss term. - :class:`GISTEmbedLoss` is equivalent to this loss, but uses a guide model to guide the in-batch negative sample selection. `GISTEmbedLoss` yields a stronger training signal at the cost of some training overhead. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() N)super__init__modelscalesimilarity_fctrCrossEntropyLosscross_entropy_loss)selfrrr __class__s l/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/losses/MultipleNegativesRankingLoss.pyrz%MultipleNegativesRankingLoss.__init__s8b   ,"$"5"5"7cL|Dcgc]}|j|d}}|d}tj|dd}|j|||jz}tj d|j d|j}|j||Scc}w)Nsentence_embeddingr)device) rtorchcatrrarangesizerr) rsentence_featureslabelssentence_featurereps embeddings_a embeddings_bscores range_labelss rforwardz$MultipleNegativesRankingLoss.forwardes[lmGW +,-ABmmAw yyab* $$\<@4::M||Av{{1~fmmL &&v|<<nsB!cH|j|jjdS)N)rr)rr__name__rs rget_config_dictz,MultipleNegativesRankingLoss.get_config_dictost7J7J7S7STTrcy)Nat @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } r+s rcitationz%MultipleNegativesRankingLoss.citationrs r)rr rfloatreturnNone)r zIterable[dict[str, Tensor]]r!rr1r)r1zdict[str, Any])r1str) r* __module__ __qualname__rcos_simrr(r,propertyr/ __classcell__)rs@rr r s5BFW[WcWcU8n=U  rr ) __future__rcollections.abcrtypingrrrrsentence_transformersr)sentence_transformers.SentenceTransformerr Moduler r.rrr?s,"$ &Ip299pr