+sgn0ddlmZddlmZmZddlmZddlmZddl m Z ddl Z ddl Z ddl m Z mZddlmZmZdd lmZdd lmZ dd ZGd d ej.Zy)) annotations)IterableIterator) nullcontext)partial)AnyN)Tensornn)SentenceTransformerutil) RandContext)StaticEmbedding+CachedMultipleNegativesSymmetricRankingLossc |jJ|jJtj5t ||j|jD]u\}}}t |j |dd||D]M\\}}}tj |j|j|z} | jOw dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cachertorch enable_gradzipembed_minibatch_iterdotflattenbackward) grad_outputsentence_featuresloss_objrgradrreps_mb_grad_mb surrogates {/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/losses/CachedMultipleNegativesSymmetricRankingLoss.py_backward_hookr&s >> %% %  ! ! -- -     %589JHNN\d\r\r5s % 1 dM),--%5"&+"/ . * %% !g"IIgoo&79JKkY ""$ % % % % %s BCCceZdZdejddf d fd Z d d dZ d ddZddZddZ dd Z dd Z xZ S)rg4@ Fct|t|dtr t d||_||_||_tj|_ ||_ d|_ d|_ ||_y)a Boosted version of :class:`MultipleNegativesSymmetricRankingLoss` (MNSRL) by GradCache (https://arxiv.org/pdf/2101.06983.pdf). Given a list of (anchor, positive) pairs, MNSRL sums the following two losses: 1. Forward loss: Given an anchor, find the sample with the highest similarity out of all positives in the batch. 2. Backward loss: Given a positive, find the sample with the highest similarity out of all anchors in the batch. For example with question-answer pairs, the forward loss finds the answer for a given question and the backward loss finds the question for a given answer. This loss is common in symmetric tasks, such as semantic textual similarity. The caching modification allows for large batch sizes (which give a better training signal) with constant memory usage, allowing you to reach optimal training signal with regular hardware. Note: If you pass triplets, the negative entry will be ignored. An anchor is just searched for the positive. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. show_progress_bar: If True, shows progress bar during processing Requirements: 1. (anchor, positive) pairs 2. Should be used with large batch sizes for superior performance, but has slower training time than non-cached versions Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | (anchor, positive) pairs | none | +---------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Like :class:`MultipleNegativesRankingLoss`, but with an additional symmetric loss term and caching mechanism. - Inspired by :class:`CachedMultipleNegativesRankingLoss`, adapted for symmetric loss calculation. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedMultipleNegativesSymmetricRankingLoss(model, mini_batch_size=32) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf rzCachedMultipleNegativesSymmetricRankingLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using MultipleNegativesSymmetricRankingLoss instead.N)super__init__ isinstancer ValueErrormodelscalesimilarity_fctr CrossEntropyLosscross_entropy_lossmini_batch_sizerrshow_progress_bar)selfr.r/r0r3r4 __class__s r%r+z4CachedMultipleNegativesSymmetricRankingLoss.__init__)szZ  eAh 0P    ,"$"5"5"7.04 =A!2cv|rtntj}| tn|}|jD cic] \} } | | || } } } |5|5|rt | j nd}|j | d} dddddd |fScc} } w#1swYxYw#1swY |fSxYw)z Embed a mini-batch of sentences.Nsentence_embedding)rrno_graditemsr valuesr.) r5rbeginendrr random_state grad_contextrandom_state_contextkvsentence_feature_minibatchrepss r%embed_minibatchz;CachedMultipleNegativesSymmetricRankingLoss.embed_minibatchs'0{U]] 0<0D{},BRBXBXBZ%[$!Qa5o%["%[ ! T TTe{,F,M,M,OPko zz"<=>RS T T\!! &\ T T T\!!s)BB,/B B, B) %B,,B8c # K|d}|j\}}ttjd||jd|j D];\}} | |jz} |j || | |||dn||\} } | | f=yw)z5Iterate over mini-batches of sentences for embedding. input_idsrzEmbed mini-batchesdescdisableN)rr=r>rrr?)shape enumeratetqdmtranger3r4rF) r5rrrrrHbszr"iberEr?s r%rz@CachedMultipleNegativesSymmetricRankingLoss.embed_minibatch_iters-[9 Q KK$$) 222    %DAqD(((A!%!5!5!1#"3%2%:T a@P "6" D, $ $% %sB Bc tj|d}tj|ddDcgc]}tj|c}}t|}tj||j}g}t j d||jd|j D]}t||jz|} |j||| ||jz} |j| ||| } | dd|| f} |j| j|dt| } | | zdz }|j|j|j!t#|t|z }|j%}|Dcgc]}|Dcgc]}|j&c}c}}|_|Scc}wcc}wcc}}w)z1Calculate the symmetric loss and cache gradients.rNdevicezPreparing cachesrI)rcatlenarangerWrNrOr3r4minr0r/r2trappenddetachsumrequires_grad_r r)r5rE embeddings_ar embeddings_b batch_sizelabelslossesrRrSscores forward_losspositive_scores backward_loss loss_mbatchlossrss r%"calculate_loss_and_cache_gradientszNCachedMultipleNegativesSymmetricRankingLoss.calculate_loss_and_cache_gradientssyya) yyQR!A1%))A,!AB & j1D1DE%'   #...   0AA,,,j9A!00a1BLQTXT^T^^F)-)@)@PQRS)UL$Q!VnO*.*A*A/BSBSBUW]^t`cds`tWu*vM'-71r|rr}rr}r?zRandContext | Noner~z!tuple[Tensor, RandContext | None]) rrrr}rr}rzlist[RandContext] | Noner~z+Iterator[tuple[Tensor, RandContext | None]])rEzlist[list[Tensor]]r~r )rIterable[dict[str, Tensor]]rfr r~r )r~zdict[str, Any]) ry __module__ __qualname__r cos_simr+rFrrorqrwrz __classcell__)r6s@r%rr(s=A\\!"' [3"[3[3; [3  [3  [3 [3H,0"+"" "  "  ")" +"037 %+%% % 0 % 5 %< D:2 r7)rr rrrrr~r) __future__rcollections.abcrr contextlibr functoolsrtypingrrrNr r sentence_transformersr r ?sentence_transformers.losses.CachedMultipleNegativesRankingLossr sentence_transformers.modelsrr&Modulerr7r%rsa"." ;W8%%2%:% %.m "))m r7