+sgFddlmZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl m Z mZejeZerddlmZ ddlmZ ddlmZmZ d dd Z d dd Z d dd Z d dd Zy#e$rYrwxYw#e$rYtwxYw)) annotationsN)Path) TYPE_CHECKINGCallableLiteral)disable_datasets_cachingis_datasets_availableSentenceTransformer)OVQuantizationConfig)OptimizationConfigQuantizationConfigc  ddlm}ddlm} ddlm}m} ddlm} t||r3t|r(t|d|rt|dj|s td|dj} | j|  ttr0| j vr tdxst#| d t% fd d |||d y #t$r tdwxYw)a Export an optimized ONNX model from a SentenceTransformer model. The O1-O4 optimization levels are defined by Optimum and are documented here: https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/optimization The optimization levels are: - O1: basic general optimizations. - O2: basic and extended general optimizations, transformers-specific fusions. - O3: same as O2 with GELU approximation. - O4: same as O3 with mixed precision (fp16, GPU-only) See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be optimized. Must be loaded with `backend="onnx"`. optimization_config (OptimizationConfig | Literal["O1", "O2", "O3", "O4"]): The optimization configuration or level. model_name_or_path (str): The path or Hugging Face Hub repository name where the optimized model will be saved. push_to_hub (bool, optional): Whether to push the optimized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the optimized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided optimization_config is not valid. Returns: None rr  Transformer)ORTModelForFeatureExtraction ORTOptimizer)AutoOptimizationConfigPlease install Optimum and ONNX Runtime to use this function. You can install them with pip: `pip install optimum[onnxruntime]` or `pip install optimum[onnxruntime-gpu]`]The model must be a Transformer-based SentenceTransformer model loaded with `backend="onnx"`.z\optimization_config must be an OptimizationConfig instance or one of 'O1', 'O2', 'O3', 'O4'.N optimizedc,j|SN) file_suffix)optimize)save_dirroptimization_config optimizers P/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/backend.pyz-export_optimized_onnx_model..j););QRT! s:"- A   8   s C&&C;c  ddlm}ddlm} ddlm}m} ddlm} t||r3t|r(t|d|rt|dj|s td|dj} | j|  ttrTdvr td d d } t!| d xs)j"j$j'd | 'j"j$j'dt) fdd|||dy #t$r tdwxYw)a Export a quantized ONNX model from a SentenceTransformer model. This function applies dynamic quantization, i.e. without a calibration dataset. Each of the default quantization configurations quantize the model to int8, allowing for faster inference on CPUs, but are likely slower on GPUs. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="onnx"`. quantization_config (QuantizationConfig): The quantization configuration. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the quantized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)r ORTQuantizer)AutoQuantizationConfigrr)arm64avx2avx512 avx512_vnnizqquantization_config must be an QuantizationConfig instance or one of 'arm64', 'avx2', 'avx512', or 'avx512_vnni'.NF) is_static_ _quantizedc,j|Srquantize)rrquantization_config quantizers rr z5export_dynamic_quantized_onnx_model..r!r"#export_dynamic_quantized_onnx_modelr$r%)r-r r.rr/rr>r0r?r1r2r3r4r5r6r7r9 weights_dtypenamelowerr:)r;rJr)r*r+rr rrr>r?r<quantization_config_namerKs ` ` @rrLrLuseB:D RL u1 25z%(K0%(--/KL k  /4Ah.A.AI,,Y7I%s+ &P PD $7q#9 Rg&<>QR]bc!s(;(I(I(N(N(T(T(V'WWXYqXr%s ,::??EEGH S sB"- C   8   s D//Ec ddlm} ddlm}  ddlm} m} m}m}ts tdt| r3tr(td| rtdj| s td||}dj}| | |j|t!d |||fDr!t#d |||fDs td fd ||nd}||nd}||nd}ndt%5j'||fd| |j(nd|dddt+fdd||||| dy#t$r tdwxYw#1swY:xYw)a Export a quantized OpenVINO model from a SentenceTransformer model. This function applies Post-Training Static Quantization (PTQ) using a calibration dataset, which calibrates quantization constants without requiring model retraining. Each default quantization configuration converts the model to int8 precision, enabling faster inference while maintaining accuracy. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="openvino"`. quantization_config (OVQuantizationConfig | dict | None): The quantization configuration. If None, default values are used. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. dataset_name(str, optional): The name of the dataset to load for calibration. If not specified, the `sst2` subset of the `glue` dataset will be used by default. dataset_config_name (str, optional): The specific configuration of the dataset to load. dataset_split (str, optional): The split of the dataset to load (e.g., 'train', 'test'). Defaults to None. column_name (str, optional): The column name in the dataset to use for calibration. Defaults to None. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str, optional): The suffix to add to the quantized model file name. Defaults to `qint8_quantized`. Raises: ImportError: If the required packages `optimum` and `openvino` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="openvino"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)OVConfigOVModelForFeatureExtractionr OVQuantizerzPlease install datasets, optimum-intel and openvino to use this function. You can install them with pip: `pip install datasets optimum[openvino]`zaPlease install datasets to use this function. You can install it with pip: `pip install datasets`zaThe model must be a Transformer-based SentenceTransformer model loaded with `backend="openvino"`.N)rJc3$K|]}|du ywN.0params r z9export_static_quantized_openvino_model..s j5  jc3$K|]}|du ywrVrWrXs rr[z9export_static_quantized_openvino_model..sw#Twr\zEither specify all of `dataset_name`, `dataset_config_name`, `dataset_split`, and `column_name`, or leave them all unspecified.c.j|dddS)N max_lengthiT)paddingr_ truncation) tokenizer)examplesr;s rpreprocess_functionzCexport_static_quantized_openvino_model..preprocess_functionsx#Z^__r"gluesst2trainsentencec|SrVrW)rc column_namerds rr z8export_static_quantized_openvino_model..%s1DXkEZ1[r"i,) dataset_namedataset_config_namerd num_samples dataset_splitc,j|S)N)save_directory ov_configrH)rcalibration_datasetrqrKs rr z8export_static_quantized_openvino_model..+s);); I*<* r"&export_static_quantized_openvino_modelopenvinor%)r-r r.r optimum.intelrRrSr rTr1r r2r3r4r5r6anyallrget_calibration_datasetrmr:)r;rJr)rkrlrnrjr*r+rr rrRrSr rTov_modelrrrqrdrKs` ` @@@@rrsrssT:D jj ! " o  u1 25z%(K0%(--/JK o  "24,1!H,?,?H-@AI++H5I j<9Lm]h*i jjsvw(46I=Ze'fwt N  `$0#;<L1D1P-V\%2%>MGM!,!8+jK ! # '??% 3 [;N;Z+77`c' @   F"- e   V   R  s E)E1E.1E:cV|dk(rd|d}n |dk(rd|d}tj5} || |dk(ret| |z } tj| dz | z tj| dz | |z j d | j } |dk(rZt| |z } | jd d t| z } | |z } tj| | | j } |rwd } |rPt|jd djddjdd}d|d|d|d|d|dd } tj| ||dd|d| |nt||z } | jd d t| z } | |z } tj| | |dk(rPt| |z j d }t| |z j d }tj||dddy#1swYyxYw)Nr$model_z.onnxrtopenvino_model_z.xmlzopenvino_model.xmlzopenvino_model.binz.binT)parentsexist_ok(z( z, z, )z )zGHello! *This pull request has been automatically generated from the [`zT`](https://sbert.net/docs/package_reference/util.html#sentence_transformers.backend.zI) function from the Sentence Transformers library.* ## Config ```python a ``` ## Tip: Consider testing this pull request before merging by loading the model from this PR with the `revision` argument: ```python from sentence_transformers import SentenceTransformer # TODO: Fill in the PR number pr_number = 2 model = SentenceTransformer( "z5", revision=f"refs/pr/{pr_number}", backend="z#", model_kwargs={"file_name": "a "}, ) # Verify that everything works as expected embeddings = model.encode(["The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium."]) print(embeddings.shape) similarities = model.similarity(embeddings, embeddings) print(similarities) ``` r;z Add exported z model ) folder_path path_in_reporepo_id repo_typecommit_messagecommit_descriptionr+) tempfileTemporaryDirectoryrshutilmove with_suffixas_posixmkdirreprreplacehuggingface_hub upload_foldercopy)r&r'r(r)r*r+rr, file_namerdst_dirsource destinationropt_config_string bin_sourcebin_destinations rr:r:8s&[M/ J %k]$7  $ $ &Q9(! j H~/H KK#77I9M N KK#77(Y:N9[9[\b9c d((*H f 8nw.G MM$M 6(^i/F!I-K KK ,'')H !# $(L$8$8g$F$N$NtU\$]$e$efikp$q!*@AU?UVjkj@@ Y""+ -+&"B  ) )$$*!!.wiwymL#5# -.8G MM$M 6(^i/F!I-K KK ,*$"8ny8EEfM #'=9#<"I"I&"Q J8cQ9Q9Q9s G*HH()FFN)r;r rz4OptimizationConfig | Literal['O1', 'O2', 'O3', 'O4']r)r7r*boolr+rr str | NonereturnNone)r;r rJzFQuantizationConfig | Literal['arm64', 'avx2', 'avx512', 'avx512_vnni']r)r7r*rr+rrrrr)NNNNFFqint8_quantized)r;r rJz"OVQuantizationConfig | dict | Noner)r7rkrrlrrnrrjrr*rr+rrr7rr)FFNr$)r&rr'r7r)r7r*rr+rrrr,r7) __future__rloggingrrpathlibrtypingrrrrsentence_transformers.utilrr getLogger__name__logger)sentence_transformers.SentenceTransformerr rur r1r0r rr#rLrsr:rWr"rrs" 33V   8 $M 6 \" V VMVV V  V  V Vz" Q Q_QQ Q  Q  Q Qp $&* $"(l l;ll l $ l  lllll lh"`9`9`9 `9  `9  `9`9`9I        s$ B;C;CCC C