sgkB\dZddlZddlmZmZmZmZddlZddl Z ddl m Z ddl m Z ddlmZddlmZmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"d dl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.e rddl/m0Z0e"jbe2Z3dZ4dZ5Gdde jlZ7Gdde7Z8e7e8dZ9Gdde jlZ:Gdde jlZ;Gdde jlZ<Gd d!eZ=d"Z>d#Z?d$Z@d%ZAGd&d'e=eZBed(e>jd)*Gd+d,eBZDed-e>jd.*Gd/d0eBZEed1e>jd2*Gd3d4e=ZFed5e?Gd6d7e=ZGy)8zPyTorch BARK model.N)DictOptionalTupleUnion)nn) functional)GenerationMixin)#AlternatingCodebooksLogitsProcessor!BarkEosPrioritizerLogitsProcessorSuppressTokensLogitsProcessor)_prepare_4d_attention_mask)CausalLMOutputWithPastMaskedLMOutput)PreTrainedModelget_parameter_device)add_start_docstrings%add_start_docstrings_to_model_forwardis_accelerate_availableis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10logging) AutoModel)BarkCoarseConfig BarkConfigBarkFineConfigBarkSemanticConfigBarkSubModelConfig)BarkCoarseGenerationConfigBarkFineGenerationConfigBarkSemanticGenerationConfig)_flash_attention_forwardzsuno/bark-smallrcFeZdZdfd ZdZdZddZ ddZxZS) BarkSelfAttentionct||j|_tj|j|_tj|j|_|j|_|j|_ |j|jz|_ |j|jzdk7r&td|jd|jdtj|jd|jz|j|_tj|j|j|j|_||_|ra|j"}t%j&t%j(||ft*j-dd||}|j/d |yy) Nrz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r biasdtyperr))super__init__dropoutrDropout attn_dropout resid_dropout hidden_size embed_dim num_headshead_dim ValueErrorLinearr)att_projout_proj is_causal block_sizetorchtrilonesboolviewregister_buffer)selfconfigr:r;r) __class__s Y/var/www/html/venv/lib/python3.12/site-packages/transformers/models/bark/modeling_bark.pyr-zBarkSelfAttention.__init__Hsk ~~ JJv~~6ZZ7++))$..8    0 0 0A 5MdnnM]^NN#2'   &"4"4a&:L:L6LSYS^S^_  &"4"4f6H6Hv{{[ " **J::ejj*j)ANOTTUVXY[egqrD   . c||jdd||fz}|j|}|jddddS)J Splits hidden_size dim into attn_head_size and num_heads Nrrrr )sizer@permuterBtensorr4attn_head_size new_shapes rE _split_headszBarkSelfAttention._split_headsfsCKKM#2&)^)DD Y'~~aAq))rFc|jddj}|j|jdd||zfz}|S)S Merges attn_head_size dim and num_attn_heads dim into hidden_size rrN) transpose contiguousr@rJrBrMr4rNs rE _merge_headszBarkSelfAttention._merge_headsnsL!!!Q'224V[[]3B/9~3M2OOP rFctj||jdddtj|j z z}|j rz|jd|jd}}|j|jdddd||z |d|fdk(tj|jj}|||z}tjj|d}|j!|j}|j#|}|||z}tj||} | |fS)NrIrS?rdim)r<matmulrTmathsqrtr5r:rJ masked_fillr)finfor+minrrsoftmaxtor0) rBquerykeyvalueattention_mask head_mask attn_weights query_length key_length attn_outputs rE_attnzBarkSelfAttention._attnzs&||E3==R+@AS499UYUbUbKcEcd >>',zz"~sxx|*L(33 !Q \ 9J F  STXYY L../33L  %'.8L}},,\r,B #u{{3 ((6   ')3Lll<7 L((rFc|j|j|jd\}}} |j||j|j }|j||j|j }|j| |j|j } |<|d} |d} t j| |fd}t j| | fd} |dur|| f} nd} |j||| ||\} }|j| |j|j } |j| } |j| } | | f}|r||fz }|S)NrrZrrrST) r8splitr3rPr4r5r<catrmrWr9r1)rB hidden_statesrgpast_key_valuesrh use_cacheoutput_attentionsrdrerfpast_key past_valuepresentrlrioutputss rEforwardzBarkSelfAttention.forwardsO!MM-8>>t~~ST>UsE!!%GT^^T]]C!!%G  &&q)H(+J))XsO4CIIz51r:E  ElGG$(JJuc5.R[$\! \'' T^^T]]S mmK0 ((5 (    &GrFFNNNNNFF) __name__ __module__ __qualname__r-rPrWrmry __classcell__rDs@rEr&r&Ds./<* )D%rFr&c@eZdZdZfdZdZdZ ddZxZS)BarkSelfFlashAttention2aH Bark flash attention module. This module inherits from `BarkSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cDt||i|t |_yN)r,r-r_flash_attn_uses_top_left_mask)rBargskwargsrDs rEr-z BarkSelfFlashAttention2.__init__s& $)&) 3V2W.W+rFcX|jdd||fz}|j|}|S)rHNrI)rJr@rLs rErPz$BarkSelfFlashAttention2._split_headss5KKM#2&)^)DD Y' rFcX|j|jdd||zfz}|S)rRNrS)r@rJrVs rErWz$BarkSelfFlashAttention2._merge_headss1 V[[]3B/9~3M2OOP rFc |j\}}} |j|j|jd\} } } |j | |j |j } |j | |j |j } |j | |j |j } |\|djdd} |djdd}tj| | fd} tj|| fd} |dur%| jdd| jddf}nd}t| | | |||jr |jnd|j|j}|j||j |j }|j!|}|j#|}||f}|rd}||fz }|S)NrrZrrT)r.use_top_left_maskr:)rJr8ror3rPr4r5rTr<rpr$trainingr.rr:rWr9r1)rBrqrgrrrhrsrt batch_size query_len_rdrerfrurvrwrlrxris rEryzBarkSelfFlashAttention2.forwards$1#5#5#7 Iq!MM-8>>t~~ST>UsE!!%GT^^T]]C!!%G  &&q)33Aq9H(+55a;J))XsO3CIIz51q9E  }}Q*EOOAq,ABGG.     $(MMDLLs"AAnn  '' T^^T]]S mmK0 ((5 ( L   &GrFr|) r}r~r__doc__r-rPrWryrrs@rErrs.X4rFr)eagerflash_attention_2c*eZdZdZdfd ZdZxZS) BarkLayerNormzOLayerNorm but with an optional bias. PyTorch doesn't support simply bias=False.ct|tjt j ||_|r.tjt j||_yd|_yr) r,r-r Parameterr<r>weightzerosr))rBr2r)rDs rEr-zBarkLayerNorm.__init__$sH ll5::k#:; >BBLL[!9:  rFctj||jj|j|jdS)Ngh㈵>)eps)F layer_normrshaper))rBinputs rEryzBarkLayerNorm.forward)s,||E4;;#4#4dkk499RVWWrF)T)r}r~rrr-ryrrs@rErr!sYM XrFrc$eZdZfdZdZxZS)BarkMLPct|tj|jd|jz|j |_tjd|jz|j|j |_tj|j|_ tj|_ y)Nr() r,r-rr7r2r)in_projr9r/r.GELUgelurBrCrDs rEr-zBarkMLP.__init__.s yy!3!3Q9K9K5KRXR]R]^  !f&8&8"8&:L:LSYS^S^_ zz&..1 GGI rFc|j|}|j|}|j|}|j|}|Sr)rrr9r.)rBrqs rEryzBarkMLP.forward5s@ ]3  -0  m4  ]3 rFr}r~rr-ryrrs@rErr-s rFrc2eZdZdfd Z ddZxZS) BarkBlockct||rMt|j|j|_t|j|j|_nHtj|j|_tj|j|_t|j|||_ t||_ y)Nr(r:)r,r-rr2r) layernorm_1 layernorm_2r LayerNormBARK_ATTENTION_CLASSES_attn_implementationattnrmlp)rBrCr:rDs rEr-zBarkBlock.__init__>s   -V-?-?fkkRD ,V-?-?fkkRD !||F,>,>?D !||F,>,>?D *6+F+FGZcd 6?rFc|j|}|j||||||}|d} |dd} || z}||j|j|z}|r|f| z} | S|f| ddz} | S)Nrrrgrhrsrtrr)rrrr) rBrqrrrgrhrsrtintermediary_hidden_states attn_outputsrlrxs rEryzBarkBlock.forwardOs&*%5%5m%D"yy &+)/ ! #1o qr"%2[%@"%?$((   7 8C & " 13g=G23gabkAGrFrzr|rrs@rErr=s#(!rFrc^eZdZdZeZdZdZdZfdZ e de jfdZ xZ S)BarkPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. FTct|tjfrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weights.r)meanstdNrY) isinstancerr7rdatanormal_rCinitializer_ranger)zero_ Embedding padding_idxrfill_rBmodules rE _init_weightsz!BarkPreTrainedModel._init_weights}s fryyl + MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rFc$t||i|yr)r,r-)rBinputsrrDs rEr-zBarkPreTrainedModel.__init__s &+F+rFreturnc:t|ds t|S|jD]g}t|dst|jds'|jj>t j |jjcSt|S) `torch.device`: The device on which the module is (assuming that all the module parameters are on the same device). _hf_hookexecution_device)hasattrrmodulesrrr<devicers rErzBarkPreTrainedModel.devicestZ('- -lln FF +FOO-?@OO44@||FOO$D$DEE  F$D))rF)r}r~rrr config_classsupports_gradient_checkpointing_supports_flash_attn_2rr-propertyr<rrrs@rErrssD L&+#!* ,* **rFraG This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`{config}`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aI This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BarkConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: codebook_idx (`int`): Index of the codebook that will be predicted. input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, number_of_codebooks)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Initially, indices of the first two codebooks are obtained from the `coarse` sub-model. The rest is predicted recursively by attending the previously predicted channels. The model predicts on windows of length 1024. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): NOT IMPLEMENTED YET. input_embeds (`torch.FloatTensor` of shape `(batch_size, input_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `input_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. input_embeds (`torch.FloatTensor` of shape `(batch_size, input_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. Here, due to `Bark` particularities, if `past_key_values` is used, `input_embeds` will be ignored and you have to use `input_ids`. If `past_key_values` is not used and `use_cache` is set to `True`, `input_embeds` is used in priority instead of `input_ids`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZeZfdZdZdZddZe e  dde e jde ee jde e jde e jd e e jd e e j d e e jd e ed e ede ede edeee jeffdZedeee jde jdeee jfdZxZS)BarkCausalModelc t||||_tj|j |j |_tj|j|j |_ tj|j|_ tjt|jDcgc]}t!|dc}|_|j$dk(|_t)|j |j*|_tj.|j |j0d|_d|_|j7ycc}w)NTrrr(F)r,r-rCrrinput_vocab_sizer2input_embeds_layerr;position_embeds_layerr/r.drop ModuleListrange num_layersrlayersr_use_flash_attention_2rr)layernorm_finalr7output_vocab_sizelm_headgradient_checkpointing post_initrBrCrrDs rEr-zBarkCausalModel.__init__)s   #%,,v/F/FHZHZ"[%'\\&2C2CVEWEW%X"JJv~~. mmPUV\VgVgPh$i1Yv%F$ij &,&A&AEX&X#,V-?-?fkkRyy!3!3V5M5MTYZ &+# %jsE#c|jSrrrBs rEget_input_embeddingsz$BarkCausalModel.get_input_embeddings>s&&&rFc||_yrrrBnew_embeddingss rEset_input_embeddingsz$BarkCausalModel.set_input_embeddingsAs "0rFc |jdd}|jdd}|jdd}|Y|jd}|ddjd}|jd|kDr|} n|jddz } |dd| df}d}n2|!|jdr|jd}n|jd}| |ddd|f}| |ddd|f}|U|S|jjddz }|j |dk(d|r|dd|jd df}nd}|)|jdrd|||jd||d S|||jd||d S) N input_embedsrg position_idsrrrrsrI) input_idsrrrrsrrg)rrrrsrrg)getrlongcumsum masked_fill_) rBrrrrrrgrseq_len past_lengthremove_prefix_lengths rEprepare_inputs_for_generationz-BarkCausalModel.prepare_inputs_for_generationDszz.$7 $4d;zz.$7  &ooa(G)!,Q/55a8Kq!K/'2$(1q'9A'=$!!%9%:":;I L'FJJ{,C&,,Q/#//!,  %+AxxK8N  #'8G8 4L  %,*>)..077;a?L  % %n&91 =+A 0B/B/D,DE L  # ;(?! ,#2#ZZ 4 ,"0  #.K0(,   rFrrrrgrrhlabelsrrsrtoutput_hidden_states return_dictrc | | n|jj} | | n|jj} ||n|jj}| | n|jj} d} | t d| | t d||n"||j|}n|n t d|jdd} |jd}| d}| |jn |j}|%d}tdgt|jz}n|ddjd}|;tj|||ztj |}|j#d}|j%|}|O|dkr t d|j&r d|vr|nd}n*|j)|d}t+||j,d }|j/||jj0}|j3||z}| |jdfz}|j4r%|j6r|rt8j;d d }|rd nd}| rd nd}| rd nd}t=t?|j|D]~\}\}}| r||fz}|j4r1|j6r%|jA|jB|d||||| }n|||||||| }|d}|r ||d fz}| sr|||rdnd fz}|jE|}|j)|}| r||fz}|jG|}| stdd||||fDStI| ||||S)NzXTraining is not implemented yet for Bark - ensure you do not pass `labels` to the model.CYou cannot specify both input_ids and input_embeds at the same time4You have to specify either input_ids or input_embedsrIrrSr+r$batch_size has to be defined and > 0rtgt_lenzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrc3&K|] }|| ywrr.0vs rE z*BarkCausalModel.forward.. sijiv)losslogitsrrrq attentions)%rCrtrrsuse_return_dictNotImplementedErrorr6rrJrrtuplelenrr<aranger unsqueezerrr@rr+ get_head_maskrrrrlogger warning_once enumeratezip_gradient_checkpointing_func__call__rrr)rBrrrrgrrhrrrsrtrrr input_shaper seq_lengthrrposition_embedsrq output_shapepresent_key_valuesall_self_attentionsall_hidden_statesiblockpast_layer_key_valuesrxrs rEryzBarkCausalModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKO)!,Q/44R8K   << Z+5MUZU_U_hnoL'11!4L44\B  %Q !GHH**343FD!/!4!4Z!D">  & &4==##p" #,R$$5b4"6BD1:3t{{O;\1] ^ -A-,#$58H$H!**t}};;NN!"aL% !$9#1'l'&7 $AJM%771:-%G" &9W)QYZ=[<]&]#= ^@,,]; %**<8  1]4D D m, &*<>OQde &.+*   rFbeam_idxc,tfd|DS)a This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. c3FK|]}tfd|Dyw)c3tK|]/}|jdj|j1yw)rN) index_selectrcr)r  past_stater)s rEr z;BarkCausalModel._reorder_cache...!s.jQ[*))!X[[9J9J-KLjs58Nr)r  layer_pastr)s rEr z1BarkCausalModel._reorder_cache.. s%  j_ij j s!r/)rrr)s `rE_reorder_cachezBarkCausalModel._reorder_caches -   rFr) NNNNNNNNNNN)r}r~rr rr-rrrr"BARK_CAUSAL_MODEL_INPUTS_DOCSTRINGrr<Tensorr FloatTensor LongTensorr?rrry staticmethodr1rrs@rErr&s%L*'1< |++MN-1>B15/3,0-1/3$(,0/3&*Q ELL)Q "%(9(9":;Q !. Q u||, Q ELL) Q ))*Q u||,Q D>Q $D>Q 'tnQ d^Q  uU\\"$:: ;Q OQ f  uU\\23  ?D||  uU\\" #    rFrzBark semantic (or text) model. It shares the same architecture as the coarse model. It is a GPT-2 like autoregressive model with a language modeling head on top.r)rCc eZdZdZeZ ddejdede e e ejfde ejdejf fd Z xZS) BarkSemanticModelsemanticrsemantic_generation_confighistory_promptrgrc | td|jd}|j}||jz}|-|j d|z j |j }|E|d| d}tjj|d|t|z f|jd}nLtj|jg|ztjj|j }tj"|d|d }tj|j$gg|ztjj|j } tj&|j)|ddd|f|j)|ddd|dzfz|j)| gd } t+t-|j.|j} | j1t+t-|jdz|j2j4t7| |j } |j9d |j:} t=|j>| |j }tA|tjD||dzftjj|j f| | |g|d |}|dd|dzdf}|S)a Generates text semantic tokens from an input prompt and an additional optional `Bark` speaker prompt. Args: input_ids (`Optional[torch.Tensor]` of shape (batch_size, seq_len), *optional*): Input ids, i.e tokenized input sentences. Will be truncated up to semantic_generation_config.max_input_semantic_length tokens. Note that the output audios will be as long as the longest generation among the batch. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. attention_mask (`Optional[torch.Tensor]`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Returns: torch.LongTensor: Output semantic tokens. N/`semantic_generation_config` has to be providedrrsemantic_promptconstant)rfmoder*rZ)r min_eos_p) eos_token_idrAr)rlogits_processorgeneration_config)#r6rmax_input_semantic_lengthtext_encoding_offsetr_r?text_pad_tokenrrpadrsemantic_pad_tokenr<rMintrcrrepeat_interleavesemantic_infer_tokenrprlistrsemantic_vocab_sizeextendrCrr rrAr rBr,generater>)rBrr:r;rgrrrEsemantic_history infer_arrayrtokens_to_suppress suppress_tokens_logits_processorrAearly_stopping_logits_processorsemantic_outputrDs rErPzBarkSemanticModel.generate/s< & -NO O__Q' $>$X$X! : O OO  %!--q>/A.G.G.IKeKtKtuI  %-.?@B[A[A\] !}}00 -4D0EEF0CC 1   %||+>>?B[[chclcl bo !223CD3I:[\]ll(== > ?* LTYT]T] "T[[/ yy'' !5O6O5O2O(PQ))*:1>]@Y\]@]>];]*^_`'' 4    " ,@@B\BoBo p  !! 1DDqH$++JgJgh i ,II[dmdtdt+u(JJ{,F,P,PQ *K3@@I^g^n^n+ '  '* JJ $=$AB%)) T W WX\XcXc d %>@_`8    *!-F-J-L*LMrFNNN)r}r~rbase_model_prefixrrr<r3r#rrstrr5rPrrs@rEr8r8&s #%L DH<@15 a<<a%Aa!c5<<&7!89 a !. a   aarFr8zBark coarse acoustics model. It shares the same architecture as the semantic (or text) model. It is a GPT-2 like autoregressive model with a language modeling head on top.rc&eZdZdZeZ ddedededededeee e jff dZ dd e jde d ededeee e jfd eed ee j"ee j"e j"ffffd ZxZS)BarkCoarseModelcoarse_acousticsmax_coarse_historysemantic_to_coarse_ratiorr: codebook_sizer;c ,|tj|dd|d}|dj}|2td|jdD]} || ddfxx|| zz cc<tj |ddj d}||jz}tj|d|d}ttj||z } t| |jd|jddzz ttj|jd|z g} tt| |z} |dd| dfj}|dd| dfj}|dddd f}||fStjgg|ztj j|j}tjgg|ztj j|j}||fS) a Preprocess the optional `Bark` speaker prompts before `self.generate`. Args: max_coarse_history (`int`): Maximum size of coarse tokens used. semantic_to_coarse_ratio (`int`): Ratio of semantic to coarse frequency batch_size (`int`): Batch size, i.e the number of samples. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. codebook_size (`int`): Codebook channel size, i.e. the size of the output vocabulary per codebook channel. history_prompt (`Optional[Dict[str,torch.Tensor]]`): Optional `Bark` speaker prompt. Returns: Returns: `tuple(torch.FloatTensor)`: - **x_semantic_history** (`torch.FloatTensor` -- Processed semantic speaker prompt. - **x_coarse_history** (`torch.FloatTensor`) -- Processed coarse speaker prompt. Nr>rrZ coarse_promptrrIrrSr*)r<rKclonerrrTreshaperNrJnpfloorraroundrMrcr) rBr]r^rr:r_r;x_semantic_historyx_coarse_historynmax_semantic_historyn_semantic_hist_providedn_coarse_hist_provideds rEpreprocess_historiesz$BarkCoarseModel.preprocess_historiess1<  %!&!8!8HY9Z[_9`blrs!t -o>DDF (q"2"8"8";<@A$QT*ma.??*@ %/?AFNNrR /2L2`2`` $667G7Mz_`a $'rxx0BE]0]'^#_ '*(&,,Q/2D2J2J12MPQ2QQ!1!7!7!:=U!UVW( $&)/GJb/b)c%d "!3A8P7P7Q4Q!R!V!V!X /4J3J3K0KLPPR /3B37 "#333"'rdZ.?uyy!Q!T!TUYU`U`!a $||RD:,=UYYORRSWS^S^_ !#333rFrVcoarse_generation_configreturn_output_lengthsrc | td| td|j}|j} |j} |j ||j k(|j |j|jz |jz} ttj| | z } ||j k7jd} tj| | z|jz } tj| |jzj} tj | j#}|j$d}|j'|| | |||\}}|j$d}tj(||g}ttj*|| z }d}|j$d}t-|D]d}|tt|| z z}|ddtj d|| z gdf}|ddd|f}t/j0|d||j$dz fd|j }tj(|tj2|j4gg|zj7|j8|dd| dfg}t;|j$d|j<|}t?||f|gtC| ||z |d |}|j$d}tj(||dd|dfg}|j$d|z }~g|dd|df}|r|| fS|S) aW Generates coarse acoustics tokens from input text semantic tokens and an additional optional `Bark` speaker prompt. Args: semantic_output (`torch.Tensor` of shape (batch_size, seq_len), *optional*): Input text semantic ids, i.e the output of `BarkSemanticModel.generate`. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. coarse_generation_config (`BarkCoarseGenerationConfig`): Generation config indicating how to generate the coarse tokens. codebook_size (`int`, *optional*, defaults to 1024): Codebook channel size, i.e. the size of the output vocabulary per codebook channel. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. return_output_lengths (`bool`, *optional*): Whether or not to return the output lengths. Useful when batching. Returns: By default: torch.LongTensor: Output coarse acoustics tokens. If `return_output_lengths=True`: `Tuple(torch.Tensor, torch.Tensor): The output coarse acoustics tokens, and the length of each sample of the batch. Nr=-`coarse_generation_config` has to be providedrr)r;r]r^rr:r_rIr?)rCmax_new_tokensrD)"r6max_coarse_input_lengthr]sliding_window_lenrrIcoarse_semantic_pad_tokencoarse_rate_hzsemantic_rate_hzn_coarse_codebooksrJrdresumr<rfmaxitemrrmhstackceilrrrHrMcoarse_infer_tokenrcrr rNr,rPra)rBrVr:rnr_r;rorrsr]rtr^rjoutput_lengthsmax_generated_lenrrgx_coarsebase_semantic_idxn_window_stepstotal_generated_lenlen_coarse_historyr semantic_idx input_coarsealternatingLogitsProcessor output_coarseinput_coarse_len coarse_outputrDs rErPzBarkCoarseModel.generatesF & -NO O # +LM M":"R"R5HH5HH $$ 9LL L $ > > % 3 3(99 :&99 : ! #288,>AY,Y#Z[)-E-_-__ddefg 5 58P8c8c c ^6N6a6a%abffh!IIn5::<$**1- '+'@'@)1%=!'A' (A( $H/44Q7,,(:O'LMRWW%69K%KLM%^^A.~&( A,s59LOg9g3h/iiL+1bffaH\9\5].^.`+`aL'+C,C+C(CDL55+l.@.@.DDE(BB L!<< LL#;#N#N"O!PS]!]^aabfbmbmnQ!3 3 445L*M""1%*>>* & "G,",>? v33V5I5IJ  &,,f.F.FUS   ',##11 + m %k  s1G%4G*=3G/c|jSrrrs rErz"BarkFineModel.get_input_embeddingss'''rFc||_yrrrs rErz"BarkFineModel.set_input_embeddingss #1 rFc|jSrrrs rEget_output_embeddingsz#BarkFineModel.get_output_embeddingss }}rFc||_yrr)rBnew_output_embeddingss rEset_output_embeddingsz#BarkFineModel.set_output_embeddingss - rFc  |j}tj|Dcgc]}|j|||c}}|j ||dj j d}|jj|jjsT|j}tj|Dcgc]}|j||c}}|j||jScc}wcc}w)Nr) rrr_get_resized_embeddingsrrrrrCtie_word_embeddings_get_resized_lm_headr) rBnew_num_tokenspad_to_multiple_ofold_embeddings_listold_embeddingsnew_embeddings_listold_lm_head_list old_lm_headnew_lm_head_lists rE_resize_token_embeddingsz&BarkFineModel._resize_token_embeddingss"779 mm': ",,^^M_`   !!"56,Q/66<= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc Return: `torch.nn.Embedding`: Pointer to the input tokens Embeddings Module of the model. r)rrrrCr vocab_size tie_weights)rBrr model_embedss rEresize_token_embeddingsz%BarkFineModel.resize_token_embeddingss044^EWX  !&8&@ )5Q(>(>(D(DQ(G %!-a!7!7!=!=a!@ !-a!7!7!=!=a!@&q/0066q9 rFclt|jddrg|_|j}|j }t |jj |jjz D]<}|j||||dz|jjd|d>yy)NrTr lm_heads..weight) getattrrC_tied_weights_keysrrrrr_tie_or_clone_weightsappend)rBoutput_embeddingsinput_embeddingsr&s rE _tie_weightszBarkFineModel._tie_weightss 4;; 5t <&(D # $ : : < #88: 4;;44t{{7P7PPQ G**+|jD]}t|ds|j!y)z Tie the weights between the input embeddings list and the output embeddings list. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. rTrrrrN) rrCrrrrrrrrrrr)rBrrr&rs rErzBarkFineModel.tie_weightss 4;; 5t <&(D # $ : : < #88: 4;;44t{{7P7PPQ G**+.kslq^_^klr)rrrqr)rCrtrrrr6rrrr<rpryrJrrrrrrrr+rrrrrr@rrrr)rBrrrgrrhrrrtrrrr&rrrr rr!rqr"r$r%r'rxrs rEryzBarkFineModel.forwardsa2C1N-TXT_T_TqTq$8$D $++JjJj &1%> $5b4"6BD!$++. JHAu#$58H$H!-#A,"3 G$AJM &9WQZM&I# J ,,]; %**<8  1]4D D H|dkk.G.GGHWlT63DFY$Zll l+*   Es&+K7rr:rnfine_generation_configr_r;c | td| td| td|jd|j}|j} |j} |j |j dd|j}tj||jz |}|j d} |)tj|djd| d } nd} |j} tj|d|j| z fd |}| Ctj | dd| dddf|gd }| dd| dddfj d }nd}d}|j d | kr/| |j d z }tj|ddd|fd | }|j d | |z z | z }t#t%j&|}t)d|d z}t+|D]}t-|| z|j d | z g}t-||| zz|j d | z g}||z }|dd||| zddf}t+| |jD]}|j/||j0}||d k(r%|dd|dd|f}tj2|d}nk|ddddd|f|z }tj4|d dd|| f}|j7d|f}tj8|d j | d}|j;tj<}||dd|d|f<~~t+| |jD]}|dd|d|f|dd||| |z z|f< ~|j?d ddddd|df}|dkDr|ddddd| f}|j d|j dk7r td|S)ap Generates fine acoustics tokens from input coarse acoustics tokens and an additional optional `Bark` speaker prompt. Args: coarse_output (`torch.Tensor` of shape (batch_size, seq_len)): Input coarse acoustics ids, i.e the output of `BarkCoarseModel.generate`. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. coarse_generation_config (`BarkCoarseGenerationConfig`): Generation config indicating how to generate the coarse tokens. fine_generation_config (`BarkFineGenerationConfig`): Generation config indicating how to generate the fine tokens. codebook_size (`int`, *optional*, defaults to 1024): Codebook channel size, i.e. the size of the output vocabulary per codebook channel. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. Returns: torch.LongTensor: Output fine acoustics tokens. Nr=rqz+`fine_generation_config` has to be provided temperaturerrI fine_promptrZr?r)r@rfrY) num_samplesrrSz-input and output should have the same seq_len) r6rrmax_fine_history_lengthmax_fine_input_lengthr@rrxr< remainderrNrKTrrHn_fine_codebooksrprJrdr}rzrraryrargmaxrbrc multinomialrcint32rT)rBrr:rnrr_r;rrrrrx_fine_historyn_coarse fine_input n_historyn_remove_from_endn_loopsn_outer start_idxstart_fill_idxrel_start_fill_idx input_buffern_innerrrelevant_logitscodebook_predsprobss rErPzBarkFineModel.generatets< & -NO O # +LM M ! )JK K jj0F0R0RS "8"P"P 6 L L&**=+>+>q+A2G_GrGrs  8R8f8f(fhuv "((+  %"44^M5R5T5TUY5Z\flmnN"N+>>UU  &77(B C      %N17N6N6OQR3R$SU_#`fghJ'q+B*B*CQ'FGMMaPII   A !6 6 5 8H8H8K K zAq!5F+Gj`mnJ!&&q)-BY-NOSjjbggg&'a/A%W~ GW'>> @P@PQR@SVk@klmI W'>>> @P@PQR@SVm@mnN"0)!; %aYAV5V)VXY&YZL +A+R+RS +g|<CC&+*<&,Q0B0C^m^-S&TO%*\\/2%FN&,Q>M>-A&B[&POIIo2>qBTUjBj?jkE!MM2}*=>E%*%6%6u!%L%Q%QR\^`%aN!/!2!25;;!?@N Q 2 3W<=N +$!+A+R+RS B!$6$7!@A~:ORd:d(eegnn B= @ ))!Q/1ij0@A q #Aq*=,=+=*=$=>J   B =#6#6r#: :LM MrFrr{) NNNNNNNNN)NNNrN)%r}r~rrXrrmain_input_namer-rrrrrrrJrrrrrrBARK_FINE_INPUTS_DOCSTRINGr<r3r5r?rrrryr#r!r"rrYrPrrs@rErrps )!L$O>(2.+,Y]%&sm%HPQT % %N G&*++EF-115/3,0-1/3,0/3&*l l ELL)l !. l u||, l ELL) l ))*l u||,l $D>l 'tnl d^l  uU\\"N2 3l Gl bDH?C;?!<@F||F%AF#= F !9 F  F!c5<<&7!89F   FrFra6 The full Bark model, a text-to-speech model composed of 4 sub-models: - [`BarkSemanticModel`] (also referred to as the 'text' model): a causal auto-regressive transformer model that takes as input tokenized text, and predicts semantic text tokens that capture the meaning of the text. - [`BarkCoarseModel`] (also refered to as the 'coarse acoustics' model), also a causal autoregressive transformer, that takes into input the results of the last model. It aims at regressing the first two audio codebooks necessary to `encodec`. - [`BarkFineModel`] (the 'fine acoustics' model), this time a non-causal autoencoder transformer, which iteratively predicts the last codebooks based on the sum of the previous codebooks embeddings. - having predicted all the codebook channels from the [`EncodecModel`], Bark uses it to decode the output audio array. It should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice. c leZdZeZfdZedejfdZdde e fdZ ddZ ej dde ejde eeejfd e edej$fd Ze dd e ej*d e eeeee ffd edeffd ZxZS) BarkModelct||t|j|_t |j |_t|j|_ tj|j|_||_yr)r,r-r8semantic_configr9r[coarse_acoustics_configr\rfine_acoustics_configrr from_config codec_config codec_modelrCrs rEr-zBarkModel.__init__sh  )&*@*@A /0N0N O+F,H,HI$001D1DE rFrcNt|jds t|S|jjD]g}t|dst|jds'|jj >t j|jj cSy)rrrN)rr9rrrrr<rrs rErzBarkModel.devices}t}}j1'- -mm++- FF +FOO-?@OO44@||FOO$D$DEE  FrFgpu_idctrddlm}n tdt j d|}|j j dk7r/|jdtjj||jj|\|j_ }d}|j|j|jfD]}||||\}}||_||j||\}}||_y)a Offloads all sub-models to CPU using accelerate, reducing memory usage with a low impact on performance. This method moves one whole sub-model at a time to the GPU when it is used, and the sub-model remains in GPU until the next sub-model runs. Args: gpu_id (`int`, *optional*, defaults to 0): GPU id on which the sub-models will be loaded and offloaded. r)cpu_offload_with_hookz1`enable_model_cpu_offload` requires `accelerate`.zcuda:cpuN)prev_module_hook)r accelerater ImportErrorr<rtyperccuda empty_cacher9rr\rfine_acoustics_hookrcodec_model_hook)rBrrrrhookcpu_offloaded_models rEenable_cpu_offloadzBarkModel.enable_cpu_offload0s # $ 8QR RfX./ ;;  u $ GGEN JJ " " $/DDMMDdDdfl.m+ (! MM  ! !   $  `  ,,?Z^_GAt  `$( '(8(8&SWX4!%rFc|jdd}|jjj|}|nt ||Dcgc]\}}|ddd|fj d!}}}|Dcgc]+}|jj |j-}}|S|jj |}|jd}|Scc}}wcc}w)z:Turn quantized audio codes into audio array using encodec.rrN)rTr quantizerdecoderrdecodersqueeze)rB fine_outputrembsamplelout audio_arrs rE codec_decodezBarkModel.codec_decodeWs"++Aq1 ((// <  %BES.AYZ+616!RaR%=**1-ZCZRUV))11&9AACVIV ""**3/C AI [Vs $C40Crr;roc tdi|jj}tdi|jj}t di|jj }|jdd|jddd}i} i} |jD]\} } | jdr| tdd} | || <+| jdr| tdd} | | | <P| jdr| tdd} | | | <u| |vr| || <| | vr| | | <| | vs| | | <d|vr|jd|jj|f||d |} d| vr| jd|jj| f||||jj|d | }d}|r|\}}||jz}d| vr| jd|j j|f|||||jjd | }t#|d dD|j$j'|j(j+|j,|_|j/||}t#|d d|j0j'|rH|Dcgc] }t|}}t2j4j6j9|dd}||fS|Scc}w)a^ Generates audio from an input prompt and an additional optional `Bark` speaker prompt. Args: input_ids (`Optional[torch.Tensor]` of shape (batch_size, seq_len), *optional*): Input ids. Will be truncated up to 256 tokens. Note that the output audios will be as long as the longest generation among the batch. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. Note that for now, this model takes only one speaker prompt per batch. kwargs (*optional*): Remaining dictionary of keyword arguments. Keyword arguments are of two types: - Without a prefix, they will be entered as `**kwargs` for the `generate` method of each sub-model. - With a *semantic_*, *coarse_*, *fine_* prefix, they will be input for the `generate` method of the semantic, coarse and fine respectively. It has the priority over the keywords without a prefix. This means you can, for example, specify a generation strategy for all sub-models except one. return_output_lengths (`bool`, *optional*): Whether or not to return the waveform lengths. Useful when batching. Returns: By default: - **audio_waveform** (`torch.Tensor` of shape (batch_size, seq_len)): Generated audio waveform. When `return_output_lengths=True`: Returns a tuple made of: - **audio_waveform** (`torch.Tensor` of shape (batch_size, seq_len)): Generated audio waveform. - **output_lengths** (`torch.Tensor` of shape (batch_size)): The length of each waveform in the batch Example: ```python >>> from transformers import AutoProcessor, BarkModel >>> processor = AutoProcessor.from_pretrained("suno/bark-small") >>> model = BarkModel.from_pretrained("suno/bark-small") >>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)` >>> voice_preset = "v2/en_speaker_6" >>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset) >>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100) >>> audio_array = audio_array.cpu().numpy().squeeze() ``` rgNrA)rgrA semantic_coarse_fine_rD)r;r:)r;r:rnr_ro)r;r:rnrr_rrTr) batch_first padding_valuer)r#rDrr!rr"rpopitems startswithrr9rPr\r_rxrrroffloadrrcrrrrutilsrnn pad_sequence)rBrr;rorr:rnrkwargs_semantic kwargs_coarse kwargs_finererfrVrroutputaudiors rErPzBarkModel.generateis:h&B%kDDZDZDjDj%k"#=#o@V@V@n@n#o !9!iD>"7    ,9 )M>+/G/Z/ZZN + - OO/ 0-$$--  )'A%=#900>>    4. 5 A  $ $ , , .#//224;;?D !!&.9 4+T 2 >  ! ! ) ) + 8=>fc&k>N>HHLL--eUV-WE.( ( ?sK% torch_dtype device_maphard_check_onlycheck_device_mapct||||||}|j|j_|j|j_|j|j _|S)a( `_check_and_enable_flash_attn_2` originally don't expand flash attention enabling to the model sub-configurations. We override the original method to make sure that Bark sub-models are using Flash Attention if necessary. If you don't know about Flash Attention, check out the official repository of flash attention: https://github.com/Dao-AILab/flash-attention For using Flash Attention 1.0 you can do it directly via the `BetterTransformer` API, have a look at this specific section of the documentation to learn more about it: https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#decoder-models The method checks if the current setup is compatible with Flash Attention as it requires the model to be in half precision and not ran on CPU. If all checks pass and `hard_check_only` is False, the method will set the config attribute `_attn_implementation` to "flash_attention_2" so that the model can initialize the correct attention module )rr)r,_check_and_enable_flash_attn_2rrrr)clsrCrrrrrDs rErz(BarkModel._check_and_enable_flash_attn_2sh67 K__o8 7=6Q6Q3>D>Y>Y&&;r/s // $) CFCJ   H %(z zzW/Wv0 XBII X bii  3 3l/*//*d""(T1&"j} )?} @U%%-A%B ee  eP&%%-?%@  ToT  Tn7%%-=%> E'E  EP  #&M#M'&MrF