sgAddlZddlmZmZmZddlZddlmZddlm Z m Z m Z m Z ddl mZddlmZddlmZerddlZeeZd d d d d ddde e de dide didZeedj1ZGddeZGddZGddeZGddeZGddeZGdd eZGd!d"eZ Gd#d$eZ!eeeeee e!d%Z"d&Z#d(d'Z$y))N)Dict NamedTupleOptional)tqdm)GGUF_CONFIG_MAPPINGGGUF_TENSOR_MAPPINGGGUF_TOKENIZER_MAPPING_gguf_parse_value)is_torch_available)is_gguf_available) get_loggerversion tensor_countkv_count)rrr file_typequantization_version)rr)GGUFgeneral tokenizertokenizer_config)ignoreconfigtensorsrrrc@eZdZUejed<eed<eed<y) GGUFTensorweightsnamemetadataN)__name__ __module__ __qualname__npndarray__annotations__strdict[/var/www/html/venv/lib/python3.12/site-packages/transformers/modeling_gguf_pytorch_utils.pyrr:s ZZ INr)rceZdZddZdZy)TensorProcessorNc|xsi|_yNr)selfrs r*__init__zTensorProcessor.__init__As l r)c t||iSr.)rr0rrkwargss r*processzTensorProcessor.processDs'4,,r)r.)r r!r"r1r5r(r)r*r,r,@s #-r)r,c peZdZdfd ZdZ ddej dedeedej fdZ xZ S) LlamaTensorProcessorc&t||yNr/superr1r0r __class__s r*r1zLlamaTensorProcessor.__init__I 'r)c d|vsd|vrx|jjd}|jjd}d||fvr t||iSd|vr|j|||}nd|vr|j|||}t||iS)Nz.attn_k.z.attn_q.num_attention_headsnum_key_value_heads)rgetr_reverse_permute_weights)r0rrr4 num_heads num_kv_headss r*r5zLlamaTensorProcessor.processLs  t!3 (=>I;;??+@AL <00!'444T!77IVt#77LY'4,,r)rn_headrEreturnc|||k7r|}|jd|zdz}|j||dg|jdd}|jddj|jS)Nrr)shapereshapeswapaxes)r0rrFrEdimws r*rCz-LlamaTensorProcessor._reverse_permute_weightsYsr  #,(>!FmmA&(A- GOOFC ?W]]12-> ?zz!Q'' 66r)r.) r r!r"r1r5r#r$intrrC __classcell__r=s@r*r7r7HsE( -OS 7zz 7+. 7>Fsm 7  7r)r7c\eZdZdfd ZdZdej deeefdede fdZ xZ S) Qwen2MoeTensorProcessorc&t||yr9r:r<s r*r1z Qwen2MoeTensorProcessor.__init__gr>r)c d|vrE|jd}|jd}|r!|j||||t|diSd|vrtj|d}t||iS)N_exptensor_key_mappingparsed_parametersffn_gate_inp_shexpraxis)rB_split_moe_expert_tensorrr# expand_dims)r0rrr4rWrXs r*r5zQwen2MoeTensorProcessor.processjsz T>!',@!A  & +> ? !--g7H$Pbc!'444 4 'nnW15G'4,,r)rrXrrWc d}d|vrd}nd|vrd}nd|vrd}ntd|d |D]}||vs|j|||}|jjd d }t d |D]N}|jd d|d|d } ||} t j tj| |d| <Py)N ffn_gate_exps gate_proj ffn_down_exps down_proj ffn_up_expsup_projzCannot map expert tensor z in Qwen2Moe architecture. num_experts<rz.weight.r) ValueErrorreplacerrBrangetorch from_numpyr#copy) r0rrXrrWexp_name tensor_name w_counteri temp_name exp_weights r*r\z0Qwen2MoeTensorProcessor._split_moe_expert_tensorws  d ""H  $"H d " H8>XYZ Z- RKd"||K1CK1PQ RKKOOM26 q)$ \A Y!A3az0IJI J6;6F6FrwwzGZ6[ i ( 3 \r)r.) r r!r"r1r5r#r$rr&r'r\rPrQs@r*rSrSfsC( -\zz\6:39o\MP\fj\r)rScveZdZdfd ZdZdej dedefdZdej dedefdZ xZ S) BloomTensorProcessorc&t||yr9r:r<s r*r1zBloomTensorProcessor.__init__r>r)c d|vrI|jd}|jd}d|vr|j|||}n|j|||}t||iS)Nattn_qkvrF hidden_sizeweight)r_reverse_reshape_weights_reverse_reshape_biasr)r0rrr4rDn_embeds r*r5zBloomTensorProcessor.processse   H-Ikk-0G477GT44WiQ'4,,r)rrFr~c(tj|dd\}}}|j|||z|}|j|||z|}|j|||z|}tj|||gd}|j|dz||zz|S)NrrZr)r# array_splitrKstack)r0rrFr~qkv qkv_weightss r*r|z-BloomTensorProcessor._reverse_reshape_weightss..!!41a IIfg/ 9 IIfg/ 9 IIfg/ 9hh1ayq1 ""6A:F1B#CWMMr)ctj|d\}}}|j|||z}|j|||z}|j|||z}tj|||gdj }|S)NrrrZ)r#rrKrflatten)r0rrFr~q_biask_biasv_biasqkv_biass r*r}z*BloomTensorProcessor._reverse_reshape_biass"$!;6(9:6(9:6(9:88VVV41=EEGr)r.) r r!r"r1r5r#r$rOr|r}rPrQs@r*rvrvsN(- N NC NRU N RZZ  s r)rvc&eZdZdfd ZdZxZS)T5TensorProcessorc&t||yr9r:r<s r*r1zT5TensorProcessor.__init__r>r)c d}|jdD]}|jst|}nt||d|iS)Nrhbid)splitisdigitrOr)r0rrr4rchunks r*r5zT5TensorProcessor.processsHZZ_ E}}%j '4%66r)r.r r!r"r1r5rPrQs@r*rrs (7r)rc&eZdZdfd ZdZxZS)GPT2TensorProcessorc&t||yr9r:r<s r*r1zGPT2TensorProcessor.__init__r>r)c d|vs d|vsd|vsd|vr |j}|dk(rDd}|jdi}tjt j ||d|<d}t ||iS) Nzattn_qkv.weightzffn_down.weightz ffn_up.weightzattn_output.weight output.weightzlm_head.weightrXr)TrBrlrmr#rnr)r0rrr4rXs r*r5zGPT2TensorProcessor.processs  % D($&#t+iiG ? "$D & +> C 161A1A"'''BR1S i ( .D'4,,r)r.rrQs@r*rrs (-r)rc&eZdZdfd ZdZxZS)MambaTensorProcessorc&t||yr9r:r<s r*r1zMambaTensorProcessor.__init__r>r)c d|vrd|vrd|vr|jdd}d|vrtj|d}d|vrtj| }t ||iS) Nssm_dbiasr{zmixer.Dzssm_conv1d.weightrrZssm_a)rjr#r]logrr3s r*r5zMambaTensorProcessor.processsk d?vT1hd6J<<3D $ &nnW15G d?ffgX&G'4,,r)r.rrQs@r*rrs ( -r)r)llamaqwen2moebloomt5 t5encodergpt2mambac|j|}|jDcgc]%}t|j||j'c}Scc}wr.)fieldsdatar partstypes)readerfieldvalue _data_indexs r* read_fieldrs? MM% EX]XbXb c ekk+6 D cc cs*A c &'trtr ddlm}m}n t j dtd||}|j}t|j}tDcic]}|i}}t|dd} t|d} d| vrd| vrd} nd | vsd | vr d |d d <d } n| } d| vrd} d| vrShd&d't&fd|jD} t'fd|jD} | |d d<| |d d<d}d| vrh|jddj!}t#j$d|}|t'd| d|j)j+d}| |zt,vrt'd | |zd!t/d"|jD|d d#<|jj1D]%\}}|j3| | }|jd$}|d}d$j5|d%d}|j6Dcgc]%}t9|j:||j<'}}t?|d%k(r|d}tA|tBr| |vr|j3| | }tD]F}t|}||vs|||vs|||}|dk(r'|||||<||vs6|jE|H||vs t jGd&|d'|(d(|d vr3|d)}d*|vrt?|d*|d d(<nt jId+|rStd,| |z}|jKd i}tLjK| tN}||-}tQ|jd./D]} | jR}!|| j6| jT}"|jW|"|!||0}#|#jX}"|#jR}!|#jZjKd1}$|!w|D]L}%|%j]|$2|!vs|!j3|%j]|$2||%j]|$2}!Nt_j`tcjd|"|d,|!<t?|dkDrt jGd3||Scc}wcc}w)4a Load a GGUF file and return a dictionary of parsed parameters containing tensors, the parsed tokenizer and config attributes. Args: gguf_checkpoint_path (`str`): The path the to GGUF file to load return_tensors (`bool`, defaults to `True`): Whether to read the tensors from the file and return them. Not doing so is faster and only loads the metadata in memory. r) GGUFReader dequantizezLoading a GGUF checkpoint in PyTorch, requires both PyTorch and GGUF>=0.10.0 to be installed. Please see https://pytorch.org/ and https://github.com/ggerganov/llama.cpp/tree/master/gguf-py for installation instructions.zKPlease install torch and gguf>=0.10.0 to load a GGUF checkpoint in PyTorch.zgeneral.architecturez general.namermistralrrTr is_gated_actr qwen2_moestablelm> attn_k.bias attn_q.bias attn_v.biasffn_normc3HK|]}D]}||jvywr.r).0tensor bias_nameattn_bias_names r* z'load_gguf_checkpoint..4s)mF^lmQZyFKK/m/ms"c3:K|]}|jvywr.r)rr ffn_norm_names r*rz'load_gguf_checkpoint..5s#^VMV[[$@#^sruse_parallel_residualr_falcon/z-\d+b-Nz>From file name, cannot determine the number of parameters for z architecture-z Architecture z not supportedc3:K|]}d|jk7yw)rNr)rrs r*rz'load_gguf_checkpoint..Js=+16;;&=stie_word_embeddingsrhrz1Some keys were not parsed and added into account z | vocab_sizertokenszCan't find a way to retrieve missing config vocab_size from tokenizer parameters. This will use default value from model config class and cause unexpected behavior.rr/z,Converting and de-quantizing GGUF tensors...)desc)rrrWrXr)rz0Some keys of the GGUF file were not considered: )3r r ggufrrloggererror ImportErrorrlistkeysGGUF_TO_TRANSFORMERS_MAPPINGranyrrlowerresearchrigroupstripGGUF_SUPPORTED_ARCHITECTURESallitemsrjjoinrr rrlen isinstancer&removeinfowarningrBTENSOR_PROCESSORSr,rr tensor_typer5rrformatrlrmr#rn)(gguf_checkpoint_pathreturn_tensorsrrrr reader_keysrrX architecture model_nameupdated_architecturerr model_sizegguf_file_namemgguf_keyrrprefix config_keyrr parameterparameter_renamesrenamed_config_keytokenizer_parametersrWrProcessorClass processorrrrresultrrprrs( @@r*load_gguf_checkpointrsO13//  A ghh , -F ]]Fv{{}%K(DE1BEEf&<=a@LFN3J,9 #:(   !<6:(#N3#+\!* \!F" mfnnmm ##^v~~#^ ^2:(#J/CX?X(#$;<J<-33C8<BBD IIi 0 9PQ]P^^kl WWY__S) j (DD= )B(C>RSS:==5;^^=:h 56 "==..0b%##L2FGs#qXXeABi( ]b]g]ghk"5;;{#;U[[Ihh u:?!HE eS !le&;MM,0DEE5 1I z%J"%+%1GL%i01CD{*&&x0 1 { " KKKH:UXY^X_` a9b@,X660= + +8;>0^_ TF;;D f.@.@AG&&#5"3 'FnnG;;D//%%e,C|1 v %%#%.$6<< (:(:s(:(CEWXcEdEkEkpsEkEtuD v 271A1A"'''BR1S i ( ./ T2 ;! F{mTU EFvis 5 S1*S)F)%rtypingrrrnumpyr#r integrationsrr r r utilsr utils.import_utilsr utils.loggingrrlr rrrrrrr,r7rSrvrrrrrrr(r)r*rs6" -- &1% H  !*" "-F\] ""5kBC$&<=O&PQ  $$@$K$P$P$RS --7?7<&\o&\R$?$N 7 7-/-4-?-*"' ! "  !d [r)