+sg_%ddlmZddlZddlZddlmZddlZddlZddl m Z ddl m Z ddlmZddlmZddlmZdd lmZGd d ej*Zy) ) annotationsN)Path) load_file) save_file) Tokenizer)nn)PreTrainedTokenizerFast)get_device_nameceZdZ d d fd Zd dZddZddZeddZddZ dddZ ddZ e d dd Z e dd ZxZS)StaticEmbeddingc t|t|tr |j}nt|t s t d|Vt|tjrtj|}tjj|d|_n7|*tj|j||_n t d|jj |_|jj"|_||_|j$j'|j)dd|_y)u Initializes the StaticEmbedding model given a tokenizer. The model is a simple embedding bag model that takes the mean of trained per-token embeddings to compute text embeddings. Args: tokenizer (Tokenizer | PreTrainedTokenizerFast): The tokenizer to be used. Must be a fast tokenizer from ``transformers`` or ``tokenizers``. embedding_weights (np.array | torch.Tensor | None, optional): Pre-trained embedding weights. Defaults to None. embedding_dim (int | None, optional): Dimension of the embeddings. Required if embedding_weights is not provided. Defaults to None. Example:: from sentence_transformers import SentenceTransformer from sentence_transformers.models import StaticEmbedding from tokenizers import Tokenizer # Pre-distilled embeddings: static_embedding = StaticEmbedding.from_model2vec("minishlab/M2V_base_output") # or distill your own embeddings: static_embedding = StaticEmbedding.from_distillation("BAAI/bge-base-en-v1.5", device="cuda") # or start with randomized embeddings: tokenizer = Tokenizer.from_pretrained("FacebookAI/xlm-roberta-base") static_embedding = StaticEmbedding(tokenizer, embedding_dim=512) model = SentenceTransformer(modules=[static_embedding]) embeddings = model.encode(["What are Pandas?", "The giant panda (Ailuropoda melanoleuca; Chinese: 大熊猫; pinyin: dàxióngmāo), also known as the panda bear or simply the panda, is a bear native to south central China."]) similarity = model.similarity(embeddings[0], embeddings[1]) # tensor([[0.9177]]) (If you use the distilled bge-base) Raises: ValueError: If the tokenizer is not a fast tokenizer. ValueError: If neither `embedding_weights` nor `embedding_dim` is provided. zThe tokenizer must be fast (i.e. Rust-backed) to use this class. Use Tokenizer.from_pretrained() from `tokenizers` to load a fast tokenizer.NF)freezez?Either `embedding_weights` or `embedding_dim` must be provided. base_model)super__init__ isinstancer _tokenizerr ValueErrornpndarraytorch from_numpyr EmbeddingBagfrom_pretrained embeddingget_vocab_sizenum_embeddings embedding_dim tokenizer no_paddinggetr)selfrembedding_weightsrkwargs __class__s _/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/models/StaticEmbedding.pyrzStaticEmbedding.__init__sV  i!8 9!,,IIy1^   (+RZZ8$)$4$45F$G!__<<=NW\<]DN  &__Y-E-E-GWDN^_ _"nn;;!^^99$- !!#!**\48c |jj|d}|Dcgc]}|j}}tjt j dg|ddDcgc] }t|c}z}tj|Dcgc] }|D]}| c}}tj} | |dScc}wcc}wcc}}w)NF)add_special_tokensr)dtype) input_idsoffsets) r encode_batchidsrrrcumsumlentensorlong) r"textsr$ encodingsencoding encodings_ids token_idsr-token_idr,s r&tokenizezStaticEmbedding.tokenize[sNN//%/P 6?@(@ @""299aSTabeceTf3gyC N3g-g#hiLLM!dyZc!dh(!d(!dlqlvlvw &7;; A3g!dsB>CC c <|j|d|d|d<|S)Nr,r-sentence_embedding)r)r"featuresr$s r&forwardzStaticEmbedding.forwardcs()-8MxXaOb)c%&r'ciSNr"s r&get_config_dictzStaticEmbedding.get_config_dictgs r'c"tjSr@)mathinfrBs r&max_seq_lengthzStaticEmbedding.max_seq_lengthjs xxr'c|jSr@)rrBs r& get_sentence_embedding_dimensionz0StaticEmbedding.get_sentence_embedding_dimensionns!!!r'c ^|r9t|jtjj |dnBt j |jtjj |d|jj tt|dz y)Nmodel.safetensorspytorch_model.bintokenizer.json) save_safetensors_file state_dictospathjoinrsaverstrr)r"save_dirsafe_serializationr$s r&rSzStaticEmbedding.saveqsl  !$//"3RWW\\(L_5` a JJt("'',,xAT*U V CX1A ABCr'c tjtt|dz }tj j tj j|dr*ttj j|d}nItjtj j|dtjdd}|d}t||S) NrMrKrLcpuT) map_location weights_onlyzembedding.weight)r#) r from_filerTrrPrQexistsrRload_safetensors_filerloaddevicer )load_dirr$rweightss r&r^zStaticEmbedding.loadxs''DN=M,M(NO 77>>"'',,x1DE F+BGGLLCV,WXGjj X':;%,,W\J]lpG,-yGDDr'cR ddlm}t}|||||||}t |j t jr tj|j } n|j j} |j} || | |S#t$r tdwxYw)a Creates a StaticEmbedding instance from a distillation process using the `model2vec` package. Args: model_name (str): The name of the model to distill. vocabulary (list[str] | None, optional): A list of vocabulary words to use. Defaults to None. device (str): The device to run the distillation on (e.g., 'cpu', 'cuda'). If not specified, the strongest device is automatically detected. Defaults to None. pca_dims (int | None, optional): The number of dimensions for PCA reduction. Defaults to 256. apply_zipf (bool): Whether to apply Zipf's law during distillation. Defaults to True. use_subword (bool): Whether to use subword tokenization. Defaults to True. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the distilled model's tokenizer and embedding weights. Raises: ImportError: If the `model2vec` package is not installed. r)distillz\To use this method, please install the `model2vec` package: `pip install model2vec[distill]`) vocabularyr_pca_dims apply_zipf use_subwordr#r) model2vec.distillrc ImportErrorr rrrrrrweightr) cls model_namerdr_rerfrgrc static_modelr#rs r&from_distillationz!StaticEmbedding.from_distillations<  1 !" !!#   l,,bjj 9 % 0 01G1G H  , 6 6 = = +55 90AjYY) n  s BB&cD ddlm}|j|}t |j t jr tj|j }n|j j}|j}||||S#t$r tdwxYw)aH Create a StaticEmbedding instance from a model2vec model. This method loads a pre-trained model2vec model and extracts the embedding weights and tokenizer to create a StaticEmbedding instance. Args: model_id_or_path (str): The identifier or path to the pre-trained model2vec model. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the tokenizer and embedding weights the model2vec model. Raises: ImportError: If the `model2vec` package is not installed. r) StaticModelzSTo use this method, please install the `model2vec` package: `pip install model2vec`rh) model2vecrqrjrrrrrrrrkr)rlmodel_id_or_pathrqrnr#rs r&from_model2veczStaticEmbedding.from_model2vecs" u -#223CD l,,bjj 9 % 0 01G1G H  , 6 6 = = +55 90AN^__ ust t us B B)NN)rz#Tokenizer | PreTrainedTokenizerFastr#znp.array | torch.Tensor | Noner int | NonereturnNone)r4z list[str]rvdict[str, torch.Tensor])r=rxrvrx)rvzdict[str, float])rvint)T)rUrTrVboolrvrw)r`rTrvr )NNTT)rmrTrdzlist[str] | Noner_z str | Nonererurfrzrgrzrvr )rsrTrvr )__name__ __module__ __qualname__rr:r>rCpropertyrGrIrSr^ classmethodrort __classcell__)r%s@r&r r s=A$( F96F9:F9" F9  F9P<"D E(,!" 3Z3Z%3Z 3Z  3Z  3Z3Z 3Z3Zj``r'r ) __future__rrErPpathlibrnumpyrrsafetensors.torchrr]rrN tokenizersrr transformersr sentence_transformers.utilr Moduler rAr'r&rs="  @@ 06D`biiD`r'