sgdZddlZddlZddlmZmZmZmZddlZ ddl m Z ddl m Z mZmZmZmZddlmZmZmZmZmZerddlZej2eZdd iZd Zd ZGd d eZy)z!Tokenization class for Pop2Piano.N)ListOptionalTupleUnion) BatchFeature) AddedToken BatchEncodingPaddingStrategyPreTrainedTokenizerTruncationStrategy) TensorTypeis_pretty_midi_availableloggingrequires_backendsto_numpyvocabz vocab.jsonc,||z }| t||}|SN)minnumbercutoff_time_idx current_idxs g/var/www/html/venv/lib/python3.12/site-packages/transformers/models/pop2piano/tokenization_pop2piano.pytoken_time_to_noter's$6K"+7 c|||1||}||kr%|}|j||||g|dk(rdn|}|||<|S|||<|S)Nr)append) rcurrent_velocitydefault_velocitynote_onsets_readyrnotes onset_idx offset_idx onsets_readys rtoken_note_to_noter'/sl ,%f- { "$J LL)Z9IJ K#3q#84kL(4 f % L%0&! LrceZdZdZddgZeZ d(fd ZedZ dZ de de fd Z d)de fd Zd ej d e d e de fdZ d*d ej dej d e d e de f dZd+d ej dedefdZd,dej dej de fdZd+dedeedeefdZ d-deej eej:fdeedee defdZ d-deej eej:fdeedee defdZ! d.deej eej:eeej:fdee"ee#fdee"eefdee d ee d!ee"d"eeee$fd#e"defd$Z% d/d%e&d&e"fd'Z'xZ(S)0Pop2PianoTokenizera Constructs a Pop2Piano tokenizer. This tokenizer does not require training. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab (`str`): Path to the vocab file which contains the vocabulary. default_velocity (`int`, *optional*, defaults to 77): Determines the default velocity to be used while creating midi Notes. num_bars (`int`, *optional*, defaults to 2): Determines cutoff_time_idx in for each token. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"-1"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 1): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 0): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to 2): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. token_idsattention_maskc 2t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}||_||_t |d5} t j| |_ddd|jjD cic]\} } | |  c} } |_ t |0d||||d|y#1swYQxYwcc} } w)NF)lstriprstriprb) unk_token eos_token pad_token bos_token) isinstancestrr r!num_barsopenjsonloadencoderitemsdecodersuper__init__) selfrr!r7r0r1r2r3kwargsfilekv __class__s rr?zPop2PianoTokenizer.__init__[s JTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir 0  %  +$99T?DL +*.););)=>A1>        + +?sD DDc,t|jS)z-Returns the vocabulary size of the tokenizer.)lenr;r@s r vocab_sizezPop2PianoTokenizer.vocab_size}s4<<  rcBt|jfi|jS)z(Returns the vocabulary of the tokenizer.)dictr;added_tokens_encoderrHs r get_vocabzPop2PianoTokenizer.get_vocabsDLL>D$=$=>>rtoken_idreturnc|jj||jd}|jd}dj |ddt |d}}||gS)a? Decodes the token ids generated by the transformer into notes. Args: token_id (`int`): This denotes the ids generated by the transformers to be converted to Midi tokens. Returns: `List`: A list consists of token_type (`str`) and value (`int`). _TOKEN_TIME_Nr)r=getr0splitjoinint)r@rNtoken_type_value token_typevalues r_convert_id_to_tokenz'Pop2PianoTokenizer._convert_id_to_tokensi <<++H8H 6TU+11#6HH%5ab%9:C@PQR@S+AA+EEJ.; 66$ 07F 6{avQ? @" =I rbeatstepcj|dn|}|j||||}|j||||}|S)al Converts tokens to Midi. This method calls `relative_batch_tokens_ids_to_notes` method to convert batch tokens to notes then uses `notes_to_midi` method to convert them to Midi. Args: tokens (`numpy.ndarray`): Denotes tokens which alongside beatstep will be converted to Midi. beatstep (`np.ndarray`): We get beatstep from feature extractor which is also used to get Midi. beat_offset_idx (`int`, *optional*, defaults to 0): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`, *optional*, defaults to 2): A parameter to control the Midi output generation. cutoff_time_idx (`int`, *optional*, defaults to 12): Denotes the cutoff time index for each note in generated Midi. r)r_r`rar) offset_sec)ro notes_to_midi)r@r_rpr`rarr#midis r!relative_batch_tokens_ids_to_midiz4Pop2PianoTokenizer.relative_batch_tokens_ids_to_midisT0 /6!O77+)+ 8  !!%h>W!X rrdc &|Dcgc]}|j|}}|}d}tt|jj Dcgc]}|j dc}dzD cgc]} d} } g} |D]Y\} } | dk(r| dk(snI| dk(rt | ||}'| dk(r| }/| d k(rt| ||j| || } Ptd t| D]J\}}| ||dz}nt||dz}t||}| j||||jgLt| dk(rgStj| } | dddfd z| dddfz}| |j!} | Scc}wcc}wcc} w) a Converts relative tokens to notes which will then be used to create Pretty Midi objects. Args: tokens (`numpy.ndarray`): Relative Tokens which will be converted to notes. start_idx (`float`): A parameter which denotes the starting index. cutoff_time_idx (`float`, *optional*): A parameter used while converting tokens to notes. rNOTErSN TOKEN_SPECIAL TOKEN_TIMErTOKEN_VELOCITY TOKEN_NOTE)rr r!r"rr#zToken type not understood!)r[rfsumr;keysendswithrr'r! ValueError enumeratemaxrrGrharrayargsort)r@r_rdrr]wordsrr rCir"r#rYrpitch note_onsetcutoffr% note_orders rrgz/Pop2PianoTokenizer.relative_tokens_ids_to_notess@FFe**51FF +0RVR^R^RcRcRe5fQajj6H5f1gjk1k+lmaTmm"' ? J_,Q;|+0!?P[ //#) |+*!%5%)%:%:&7 + !!=>>+ ?."++= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~file_utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. Returns: `BatchEncoding` containing the token_ids. rr)rrrrrT)r#rr)rrrrrrr4) r5rhndarrayndimlist"_get_padding_truncation_strategiesrrpad)r@r#rrrrrrrrA is_batchedpadding_strategyrr*s r__call__zPop2PianoTokenizer.__call__s d)35"**(EUZZ1_:V[\]V^`dKe ElDDkDkE !!1 E  E A-z6 ,A,IDOd !...$7% I)(($7% IHH $!1"7) rfeature_extractor_output return_midic Ntt|dxrt|dxr t|d}|s |djddkDr td|rt |ddddfdk(|djdk7s%|djd|d jdk7rAtd |jdd |djdd |d jd|djd|jdk7rtd |djdd|jd|djddk7s|d jddk7r2td|djdd|d jdd|r&t j |ddddfdk(d}n|jdg}g}g}d}t|D]\} } ||| } | dddt jt j | t|jk(ddzf} |d| } |d | } |rx|d| }|d| }| dt jt j |dk(ddz} | dt jt j |dk(ddz} t| } t| } t| } |j| | |j|jdzdz}|jdjD]C}|xj | dz c_|xj"| dz c_|j%|E|j%||| dzz }|rt'||dSt'd|iS)aF This is the `batch_decode` method for `Pop2PianoTokenizer`. It converts the token_ids generated by the transformer to midi_notes and returns them. Args: token_ids (`Union[np.ndarray, torch.Tensor, tf.Tensor]`): Output token_ids of `Pop2PianoConditionalGeneration` model. feature_extractor_output (`BatchFeature`): Denotes the output of `Pop2PianoFeatureExtractor.__call__`. It must contain `"beatstep"` and `"extrapolated_beatstep"`. Also `"attention_mask_beatsteps"` and `"attention_mask_extrapolated_beatstep"` should be present if they were returned by the feature extractor. return_midi (`bool`, *optional*, defaults to `True`): Whether to return midi object or not. Returns: If `return_midi` is True: - `BatchEncoding` containing both `notes` and `pretty_midi.pretty_midi.PrettyMIDI` objects. If `return_midi` is False: - `BatchEncoding` containing `notes`. r+attention_mask_beatsteps$attention_mask_extrapolated_beatstep beatstepsrrSzattention_mask, attention_mask_beatsteps and attention_mask_extrapolated_beatstep must be present for batched inputs! But one of them were not present.Nextrapolated_beatstepzbLength mistamtch between token_ids, beatsteps and extrapolated_beatstep! Found token_ids length - z, beatsteps shape - z$ and extrapolated_beatsteps shape - z!Found attention_mask of length - z but token_ids of length - zLength mistamtch of beatsteps and extrapolated_beatstep! Since attention_mask is not present the number of examples must be 1, But found beatsteps length - z", extrapolated_beatsteps length - .rc)r_rprar)r#pretty_midi_objectsr#)boolhasattrshaperr}rhwhererrrWr1rrur7rr#rrrr )r@r*rrattention_masks_present batch_idx notes_listpretty_midi_objects_listrdrjend_idxeach_tokens_idsrrrrpretty_midi_objectnotes r batch_decodezPop2PianoTokenizer.batch_decodeUs8#' ,.> ? Z02LM Z02XY# '+CK+P+V+VWX+Y\]+]H  #,-=>q!tDIJ+K8>>qAB+K8>>qA+,CDJJ1MN!**3//!*<)==QRjkvRwR}R}~SARBB::RSj:k:q:qrs:t9uw ((89??BiooVWFXX 78PQa8b8h8hij8k7lmHIRIXIXYZI[H\] )5;;A>!C+,CDJJ1MQRR 44L[4Y4_4_`a4b3cdFG_`wGxG~G~@GAFBBCD #!9:J!KAqD!QUV!VWXYZI"+,I #%  ' 2# %NE7' ':O-a1r266"((?VYZ^ZhZhViCi:jkl:m3nqr3r1r.rsO0=eDI$<=T$UV[$\ !'+CD^+_`e+f(7O:884&&^rxx8PTU8U/VWX/Y(Z]^(^_ (=XbffRXX&Ja&OPQRSTWXX)%'7O +I$,-B$C !!%!G!G&.#}}!%!2a 7 "H" +66q9?? ( il* IaL(!!$' ( % + +,> ? 1 $IG# %J  :Nf!gh hgz233r)Mrz-1102)ry)rr r)g)NN)FNNNNNT)T))__name__ __module__ __qualname____doc__model_input_namesrvocab_files_namesr?propertyrIrMrWrr[r^rhrrorufloatrgrsr6rrrrrrrr r rrrr rrrr __classcell__)rEs@rr)r)>s2%&67)   D!!?#S#T#$NcN * ** *  *` !!   **      H:2:::%:bg:x2::QT@!c!HSM!]bcf]g!4=A$( E4RZZk&6&6!778E4&&89E4SM E4  E4T=A$( %ERZZk&6&6!778%E&&89%ESM %E  %E\6;;?$(,004;?z JJ !! " k&&' ( * ztS/12z$%778zSMz%SMz (~z!sJ!78zz z@! w4#/w4 w4rr)) rr9rtypingrrrrnumpyrhfeature_extraction_utilsrtokenization_utilsr r r r r utilsrrrrrr get_loggerrrrrr'r)r4rrrsm( //4uu__   H % \  N 4,N 4r