sgZdZddlmZddlZddlmZddlmZmZm Z m Z ddl Z ddl ZddlmZddlmZdd lmZmZmZmZmZmZdd lmZmZmZmZd d lm Z d d l!m"Z"ejFe$Z%dZ&eGddeZ'eGddeZ(GddeZ)dZ*dZ+ee*Gdde)Z,ede*Gdde)eZ-ede*Gdde)eZ.y)zTFRAG model implementation.) annotationsN) dataclass)ListOptionalTupleUnion)PretrainedConfig)TFLogitsProcessorList)TFCausalLanguageModelingLossTFModelInputTypeTFPreTrainedModelkeras shape_list unpack_inputs) ModelOutput%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) RagConfig) RagRetrieverrceZdZUdZdZded<dZded<dZded<dZded <dZ ded <dZ ded <dZ ded <dZ ded <dZ ded<dZded<dZded<dZded<dZded<dZded<dZded<dZded<y)TFRetrievAugLMMarginOutputa Base class for retriever augmented marginalized models outputs. Args: loss (`tf.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. past_key_values (`List[tf.Tensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `tf.Tensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. doc_scores (`tf.Tensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. retrieved_doc_embeds (`tf.Tensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`tf.Tensor` (int32) of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`tf.Tensor`(int32) of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`tf.Tensor` (int32) of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. Ntf.Tensor | Noneloss tf.TensorlogitsList[tf.Tensor] | Nonepast_key_values doc_scoresretrieved_doc_embedsretrieved_doc_idscontext_input_idscontext_attention_mask"question_encoder_last_hidden_stateTuple[tf.Tensor, ...] | Nonequestion_enc_hidden_statesquestion_enc_attentionsgenerator_enc_last_hidden_stategenerator_enc_hidden_statesgenerator_enc_attentionsgenerator_dec_hidden_statesgenerator_dec_attentions)__name__ __module__ __qualname____doc__r__annotations__rr r!r"r#r$r%r&r(r)r*r+r,r-r.Z/var/www/html/venv/lib/python3.12/site-packages/transformers/models/rag/modeling_tf_rag.pyrr/sBH"D !FI.2O+2#'J '-1*1*.'.*.'./3,3;?&(8??C <C<@9@8<#%5<@D!=D=A:A@D!=D=A:Ar5rceZdZUdZdZded<dZded<dZded<dZded <dZ ded <dZ ded <dZ ded <dZ ded <dZ ded<dZded<dZded<dZded<dZded<dZded<dZded<y)TFRetrievAugLMOutputa Args: logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. past_key_values (`List[tf.Tensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `tf.Tensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. doc_scores (`tf.Tensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. retrieved_doc_embeds (`tf.Tensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`tf.Tensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. Nrrrr rr!r"r#r$r%r&r'r(r)r*r+r,r-r.)r/r0r1r2rr3r r!r"r#r$r%r&r(r)r*r+r,r-r.r4r5r6r8r8s>@FI.2O+2#'J '-1*1*.'.*.'./3,3;?&(8??C <C<@9@8<#%5<@D!=D=A:A@D!=D=A:Ar5r8cFeZdZdZeZdZdgZe d ddZ y)TFRagPreTrainedModela RAG models were released with the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. RAG is a retriever augmented model and encapsulate three components: a question encoder, a dataset retriever and a generator, the encoder and generator are trainable while the retriever is just an indexed dataset. rag position_idsNc|jDcic]%\}}|jdr|tdd|'}}}|jDcic]%\}}|jdr|tdd|'} }}|jD]} |d| z= | jD]} |d| z= |j dd} | R|Jdddlm} d|vrdd lm} | j|}||d<| j|g|d |jd |} | j d d}|O|Jd ddlm }d| vrdd lm} | j|}|| d<|j|fd |jd | }|jdd}|+tj| j|jfi|}|| |||Scc}}wcc}}w)a Instantiates an question encoder and a generator from one or two base classes of the library from pretrained model checkpoints. Params: question_encoder_pretrained_model_name_or_path (`str`, *optional*): Information necessary to initiate the question encoder. Can be either: - A string with the *shortcut name* of a pretrained model to load from cache or download, e.g., `google-bert/bert-base-uncased`. - A string with the *identifier name* of a pretrained model that was user-uploaded to our S3, e.g., `dbmdz/bert-base-german-cased`. - A path to a *directory* containing model weights saved using [`~TFPreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *pytorch index checkpoint file* (e.g, `./pt_model/`). In this case, `question_encoder_from_pt` should be set to `True`. generator_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the generator. Can be either: - A string with the *shortcut name* of a pretrained model to load from cache or download, e.g., `google-t5/t5-small`. - A string with the *identifier name* of a pretrained model that was user-uploaded to our S3, e.g., `facebook/bart-base`. - A path to a *directory* containing model weights saved using [`~TFPreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *pytorch checkpoint file* (e.g, `./pt_model/`). In this case, `generator_from_pt` should be set to `True`. model_args (remaining positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. retriever ([`RagRetriever`], *optional*): The retriever to use. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). - To update the question_encoder configuration, use the prefix *question_encoder_* for each configuration parameter. - To update the generator configuration, use the prefix *generator_* for each configuration parameter. - To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a `config` is provided or automatically loaded. Example: ```python >>> from transformers import RagRetriever, TFRagModel >>> # initialize a RAG from two pretrained models. >>> model = TFRagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", "google-t5/t5-small" ... ) >>> # alternatively, initialize from pytorch pretrained models can also be done >>> model = TFRagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", ... "facebook/bart-base", ... generator_from_pt=True, ... question_encoder_from_pt=True, ... ) >>> # saving model after fine-tuning >>> model.save_pretrained("./rag") >>> # load retriever >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # load fine-tuned model with retriever >>> model = TFRagModel.from_pretrained("./rag", retriever=retriever) ```question_encoder_N generator_modelznIf `model` is not defined as an argument, a `question_encoder_pretrained_model_name_or_path` has to be defined TFAutoModelconfig) AutoConfigquestion_encodernameload_weight_prefix generatorzqIf `generator_model` is not defined as an argument, a `generator_pretrained_model_name_or_path` has to be definedTFAutoModelForSeq2SeqLM)rFrJrD retriever)items startswithlenkeyspopauto.modeling_tf_autorCauto.configuration_autorEfrom_pretrainedrIrLgetr'from_question_encoder_generator_configsrD)cls.question_encoder_pretrained_model_name_or_path'generator_pretrained_model_name_or_pathrM model_argskwargsargumentvaluekwargs_question_encoderkwargs_generatorkeyrFrCrEquestion_encoder_configrJrLgenerator_configrDs r6*from_pretrained_question_encoder_generatorz?TFRagPreTrainedModel.from_pretrained_question_encoder_generatorsuf$*<<># %""#67 S,-/ 0% 7# # $*<<> %""<0 S&( )5 0  +//1 2C*S01 2#((* +C|c)* + 366wE  #AM  M <66@*4*D*DEs*t'4K'1:{::>   '# #9  *   %((d;  :F ! F H//@#-#=#=>e#f -= *?/??7 # #9# IHd+ >FF '')9)9=CF$4 RXdmnnS#  s *G *G)NNN)rYstrrZrerMrreturnr) r/r0r1r2r config_classbase_model_prefix_keys_to_ignore_on_load_missing classmethodrdr4r5r6r:r:smL'6&7#?C7;"& Yo8;Yo25Yo Yo YoYor5r:a RAG is a sequence-to-sequence model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. The question encoder can be any *autoencoding* model, preferably [`TFDPRQuestionEncoder`], and the generator can be any *seq2seq* model, preferably [`TFBartForConditionalGeneration`]. The model can be initialized with a [`RagRetriever`] for end-to-end generation or used in combination with the outputs of a retriever in multiple steps---see examples for more details. The model is compatible any *autoencoding* model as the `question_encoder` and any *seq2seq* model with language model head as the `generator`. It has been tested with [`TFDPRQuestionEncoder`] as the `question_encoder` and [`TFBartForConditionalGeneration`] as the `generator`. This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a Tensorflow [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. The model is in a developing state as it is now fully supports in eager-mode only, and may not be exported in SavedModel format. Args: config ([`RagConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. question_encoder ([`TFPreTrainedModel`]): An encoder model compatible with the faiss index encapsulated by the `retriever`. generator ([`TFPreTrainedModel`]): A seq2seq model used as the generator in the RAG architecture. retriever ([`RagRetriever`]): A retriever class encapsulating a faiss index queried to obtain context documents for current inputs. a Args: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_outputs (`tuple(tuple(tf.Tensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`TFRagModel`]) model during decoding. decoder_input_ids (`tf.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. past_key_values (`tuple(tuple(tf.Tensor))`): Tuple consists of two elements: `encoder_outputs` of the RAG model (see `encoder_outputs`) and `past_key_values` of the underlying generator. Can be used to speed up decoding. `past_key_values` are used in the ([`RagTokenForGeneration`]) model during decoding. doc_scores (`tf.Tensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. context_input_ids (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. output_retrieved(`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`TFRetrievAugLMOutput`] instead of a plain tuple. n_docs (`int`, *optional*, defaults to `config.n_docs``) Number of documents to retrieve and/or number of documents for which to generate an answer. ceZdZdZ d dfd Zd dZeeee e e  d d dZ d dZ xZS) TFRagModeltf_rag_model_1c ^| ||Jd|,tj|j|jfi|}n-t||jsJd|d|jt ||fi||#ddlm}|j|jd}|7ddlm }||n |j}|j|jd |d z }||_|j5t|tsJd t!|jd ||_||_ ||_ y)NzQEither a configuration or an question_encoder and a generator has to be provided.zconfig: z has to be of type rArBrF)rHrKrJz /generatorrGz`self.retriever` is of type z&, but should be of type `RagRetriever`)rrWrD isinstancergsuper__init__rSrC from_configrFrLrIrJrMrtype) selfrDrFrJrMrIr\rCrL __class__s r6rqzTFRagModel.__init__sl!  (Y-B _ ^ _  >FF '')9)9=CFfd&7&78 sHVHL_`d`q`q_r:s s8 *6*  # ;*66v7N7NUg6h    G7I7U!3[_[r[r /;;  {GY\hGh<I# >> %< k-d4>>.B-CCij k'DN 0"r5c||_yN)rMrtrMs r6 set_retrieverzTFRagModel.set_retriever s "r5 output_typergc d|vsJd||n|jj}|jduxr|duxs | duxs|duxr|du}|d|rF|j||d|}|d}|j||j |j jj |d}|d |d |d |d f\}} }}tj|tj}tj| tj} tj|tj}tj|tj}tjtjtj|d |dd }n|Jd| Jd|Jd|Jd|jd |zdk(sJd|d|jdd|tj||d}|tj||d}|j || ||||| d| }|s d}d}d}d}d}nj }|j"}|r| sd}d} d}d}t%|j&||j(|| |||j*|j,|j.|j0|j2S)aa Returns: Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, TFRagModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-base") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = TFRagModel.from_pretrained("facebook/rag-token-base", retriever=retriever, from_pt=True) >>> input_dict = tokenizer.prepare_seq2seq_batch( ... "How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="tf" ... ) >>> input_ids = input_dict["input_ids"] >>> outputs = model(input_ids) ```decoder_cached_states8Please use past_key_values to cache intermediate outputsNT)attention_mask return_dicttrainingrtfprefixn_docsreturn_tensorsr$r%r"doc_idsraxis transpose_bzMake sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `context_attention_mask` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.z^Make sure that `doc_scores` are passed when passing `encoder_outputs` to the forward function.M The first dimension of `context_input_ids` should be a multiple of `n_docs`= , but is .)rencoder_outputsdecoder_input_idsdecoder_attention_maskr use_cacherrrw)rr!r r$r%r"r#r&r(r)r*r+r,r-r.)rDrrMrFnumpyrJrrcastint32float32squeezematmul expand_dimsshaperepeat hidden_states attentionsr8rr encoder_last_hidden_stateencoder_hidden_statesencoder_attentionsdecoder_hidden_statesdecoder_attentions)rt input_idsrrrrr r!r$r%routput_attentionsoutput_hidden_statesoutput_retrievedrrrr\has_to_retrievequestion_enc_outputsr&retriever_outputsr"r# gen_outputsr(r)s r6callzTFRagModel.call#s\ $6 1 F E F 1"-4;;3E3E NN$ & ("d*b.D.LbPZ^bPb (4'   "'+'<'<n$Ya(=($6J62%)NN6<<>>>0077!#' %3%!&&9:%&>?%&<=%i0 fb!#9;OQb%'GG,=rxx$H!)+1G)R&')ww/CRZZ'P$$&GG,=rxx$H! ZZII'IPQR,$(   )4P4.9T9"-J-  " l k l "  #f,2 [\b[cd!''*+1 . 2  ( " *;V! L  ! -%'YY/EvTU%V "nn 1+/#9+%  15 .)- &&* ##' $ )=)K)K &&:&E&E #&6 ' %) "#' $ #%%!'77/#9!5//Q'A$;,7,Q,Q(3(I(I%0%C%C(3(I(I%0%C%C  r5c|jryd|_tj|jj5|jj ddddtj|j j5|j j ddddy#1swYWxYw#1swYyxYw)NT)builtr name_scoperJrHbuildrFrt input_shapes r6rzTFRagModel.builds ::  ]]4>>.. / ' NN  & ' ]]40055 6 .  ! ! ' ' - . . ' ' . .sB0 B<0B9<C)NNNNN) rDOptional[PretrainedConfig]rFOptional[TFPreTrainedModel]rJrrMOptional[RagRetriever]rIz Optional[str]rMr)NNNNNNNNNNNNNNNF)"rTFModelInputType | Nonernp.ndarray | tf.Tensor | Nonerrrrrrr 1Tuple[Tuple[Union[np.ndarray, tf.Tensor]]] | Noner!rr$rr%rr bool | Nonerrrrrrr int | Nonerrrboolrfr8rw)r/r0r1rIrqryrrRAG_FORWARD_INPUTS_DOCSTRINGrr8_CONFIG_FOR_DOCrr __classcell__rus@r6rlrls}).28<15,0,0 *#**#6*#/ *# * *# * *#X#*+GH+?o^.28<9=;?@DMQ48;?@D!%)-,0(,!#'#k *k 6k 7 k 9 k !> k Kk 2k 9k !>k k 'k *k &k k !!k "#k & 'k _Ik Z.r5rlzr A TF RAG-token model implementation. It performs RAG-token specific marginalization in the forward pass. c eZdZdZ d dfd ZddZ ddZedZedZ edZ e dd Z dd Z eeeeee  d dd Zdddddddef dd ZdZdZddZddZddZddZxZS)TFRagTokenForGenerationz!tf_rag_token_for_generation_1/ragNc | ||Jd|+tj|j|jfi|}t||t |||||j d|_yNzHEither a configuration or an encoder and a generator has to be provided.r;)rDrFrJrMrIrHrrWrDrprqrlrIr;rtrDrFrJrMr\rus r6rqz TFRagTokenForGeneration.__init__!  (Y-B V U V  >FF '')9)9=CF  -#66  r5c&||j_yrwr;rMrxs r6ryz%TFRagTokenForGeneration.set_retriever&r5c 4| |ddddf}d||||||d|d S)NT) rrr!r%rr rdo_marginalizerr4) rtrr rrrr!rr\s r6prepare_inputs_for_generationz5TFRagTokenForGeneration.prepare_inputs_for_generationsB  & 1!RS& 9 .$&4!2.""  r5c.|jjSrwrrts r6rMz!TFRagTokenForGeneration.retriever"xx!!!r5c.|jjSrwr;rJrs r6rJz!TFRagTokenForGeneration.generator&rr5c.|jjSrwr;rFrs r6rFz(TFRagTokenForGeneration.question_encoder*xx(((r5cNfd}tjj||S)a, RAG-specific `_gather_beams`: gathers the beam slices indexed by beam_indices into new beam array. If the nested tensor has a shape mismatch with the beam indices, then it means it is the cache. In that case, isolates and takes care of the extra dimension for ndocs. c|jdjdk7}|rW|jdjdz}jd}tj||d|g|jdd}tj|dd}|r+tj|zdg|jdd}|S)NrrrAr)paramsindicesr batch_dimsr )rrreshapegather)tensor is_rag_cacher batch_sizegathered_tensor beam_indicess r6 gather_fnz8TFRagTokenForGeneration._gather_beams..gather_fn6s!<<?l.@.@.CCLaL,>,>q,AA)//2 FZV,WfllSTSUFV,WX iiv|RS`abO"$**_zF?RTV>sYhYnYnopoqYr>s"t" "r5)rnest map_structure)nestedr batch_axisrs ` r6 _gather_beamsz%TFRagTokenForGeneration._gather_beams.s! # ww$$Y77r5c||n|jj}tjj |d}tj ||j d|z|d|j dg}tjj |d}tj|d}tj|d}||z}tj|dS)Nrrrr) rDrrnn log_softmaxrrrreduce_logsumexp)rt seq_logitsr!r seq_logprobs doc_logprobs log_prob_sums r6 marginalizez#TFRagTokenForGeneration.marginalizeHs!-4;;3E3Euu(("(= zz,1A1A!1D1NPVXZ\f\l\lmo\p0qr uu((!(< ~~l< ~~l< #l2 ""d|vsJd|r|n|jj}|r|n|jj}|||}d} |j||||||| ||| | | | ||}d}|j}|C|J|j |j|j |||jj|}|r|j||j |}tdid|d|d |jd |j d |jd |jd |jd|jd|jd|j d|j"d|j$d|j&d|j(d|j*d|j,S)a do_marginalize (`bool`, *optional*): If `True`, the logits are marginalized over all documents by making use of `torch.nn.functional.log_softmax`. labels (`tf.Tensor` or `np.ndarray` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the cross entropy classification loss according to Rag-Token model formulation See https://arxiv.org/pdf/2005.11401.pdf Section 2.1 for details about Rag-Token formulation. Indices should be in `[0, ..., config.vocab_size - 1]`. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `tf.Tensor.sum` operation. kwargs (`Dict[str, any]`, *optional*, defaults to `{}`): Legacy dictionary, which is required so that model can use *generate()* function. Returns: Example: ```python >>> import tensorflow as tf >>> from transformers import AutoTokenizer, RagRetriever, TFRagTokenForGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = TFRagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever, from_pt=True) >>> input_dict = tokenizer.prepare_seq2seq_batch( ... "How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="tf" ... ) >>> outputs = model(input_dict, output_retrieved=True) >>> # or use retriever separately >>> # 1. Encode >>> input_ids = input_dict["input_ids"] >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.numpy(), return_tensors="tf") >>> doc_scores = tf.squeeze( ... tf.matmul( ... tf.expand_dims(question_hidden_states, axis=1), docs_dict["retrieved_doc_embeds"], transpose_b=True ... ), ... axis=1, ... ) >>> # 3. Forward to generator >>> outputs = model( ... inputs=None, ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=input_dict["labels"], ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```r}r~NFrrrrr$r%r!r rrrrrr reduce_lossepsilonrrrr r!r$r%r"r#r&r(r)r*r+r,r-r.r4)rDrrr;rget_nllr!label_smoothingrrr r$r%r"r#r&r(r)r*r+r,r-r.)rtrrrrrr r!r$r%rrrrrrlabelsrrrr\outputsrrs r6rzTFRagTokenForGeneration.callTsv $6 1 F E F 1,:t{{?Y?Y%0kdkk6M6M   ($*!I(( )+/#9/#9!+/!5- $  $0 00<<""' 33  D %%fg.@.@&IF)   $33 ))  &77  $+#A#A  ")!=!= &77 07/Y/Y (/'I'I %,$C$C -4,S,S )0(K(K &-%E%E )0(K(K &-%E%E!  r5c  |jtjjdi| } n|jj |j 7|4|j||d} |j || jjtj|jjjd} | d| d| d} }}tj|tj }tj|tj }tj| tj} tj"tj$| d | d }tj&|d }|j(dzdk(sJd d |j(dd|j(dz|j*jj-}|||j.j0d }tj2j4zdftjj6tj }|d}d fd }||j4}||j4|d<tj8|j4d }|| d<|| d<|| d<| d<|j;tj(|d|}j4dk(rb|j<d|j>j@jB|j.j0jDjFd | Sj4dkDrЉj4jHkr&tKdj4djHdfd}||}|| d| d<|| dd| dd<|jLd|j>j@jB|j.j0jDjFd | StKdj4)!a| Implements TFRAG token decoding. Args: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`tf.Tensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. doc_scores (`tf.Tensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which had the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`TFLogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and a model's config. If a logit processor is passed that is already created with the arguments or a model's config an error is thrown. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Return: `tf.Tensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. rrrrr$r%r"rrTrrrr)rrrrrlast_hidden_statec|jdd}df|z}tj||}|f|z}tj||}|zzf|z}tj||S)z Broadcast tensor with `num_beams` replica, with correct order Input: tensor of shape (batch_size*n_docs , d) Output: tensor of shape (batch_size*num_beams*n_docs , d) rN)rrr broadcast_to)r num_beams d_shape_list new_shaperrs r6extend_enc_outputz;TFRagTokenForGeneration.generate..extend_enc_outputjs"<<+L$Q/,>IZZ 2F$Y7,FI__VY7F$i/&8:\II::fi0 0r5)rr!rrrr)generation_configinput_ids_seq_lengthlogits_processor) r max_length pad_token_id eos_token_idrrr output_scoresreturn_dict_in_generatezwBeam search decoding cannot return more sequences than it has beams. Please set num_beams >= num_return_sequences, got z and z (respectivelly)cjt|}tj|djg|ddzS)zFUnflattens the first, flat batch*beam dimension of a non-scalar array.rrN)rrrr)rrrs r6unflatten_beam_dimz.unflatten_beam_dims7"6*zz&2/@/J/J*KeTUTVi*WXXr5uH`num_beams` has to be an integer strictly superior to 0 (≥ 1), but is r4rw)'rcopydeepcopyupdaterDrrMrFrastypenprrJrrrrrrrrr; get_encoderrrfillrdecoder_start_token_idr_get_logits_processor greedy_searchrrrr r num_return_sequences ValueError beam_search)rtrrr$r%r!rrrr\ model_kwargsquestion_hidden_statesoutr"encoderrrrr pre_processorr rs `` @r6generatez TFRagTokenForGeneration.generatesF  $ $ 6 6  MM*;</(//9&9 "-4;;3E3E >> %*;*C%)%:%:9Uc%:%def%g "..&,,.55bjjA~~,,33# !C'(,-*+8L5  !#(9288 D %'WW-CRXX%N "#%77+?#L 5A>@TbfJJQ7J!''*V39 [\b[cd!''*+1 . 9 ',,Q/69 (($$002!'1/AA!2!G!G  GG +55 5q 9 GG%<-S-SS 22C2M2M1NO)>>??OQ  Y !33D E -? M]@^-_L) *CU./0CDDL* +,? @$4## +,77.;;.;;!."3"E"E%6%K%K/==(9(Q(Q   Z[l[v[vZwx r5cJ|jjjSrw)r;rJget_input_embeddingsrs r6r!z,TFRagTokenForGeneration.get_input_embeddingssxx!!6688r5cJ|jjjSrw)r;rJget_output_embeddingsrs r6r#z-TFRagTokenForGeneration.get_output_embeddingssxx!!7799r5c |)|jjj}|Jd|jjj}|Jdt j t |ddft j||j}t j||ddddfgd}t j|dk(t j t |t j||j|}tjj|t jd|j}t j|g5t j|}ddd|S#1swY|SxYw)zCShift input ids one token to the right, and pad with start_token_idNzself.generator.config.decoder_start_token_id has to be defined. In Rag we commonly use Bart as generator, see Bart docs for more informationz1self.model.config.pad_token_id has to be defined.rrri)rJrDrrrrrrdtypeconcatwhere debuggingassert_greater_equalcontrol_dependenciesidentity)rtrstart_token_idr start_tokensshifted_input_ids assert_gte0s r6shift_tokens_rightz*TFRagTokenForGeneration.shift_tokens_rightsb  !!^^22IIN!- A - ~~,,99 '\)\\'ww 9 5a 8!fg II|Yq#2#v5F&GLHH  % GGJ01277<3Y Z  ll778I277STVgVmVmKno  $ $k] 3 ? " ,= >  ?!  ?! s !FF c ||n|jj}tj|ddddftj|j ddgtj |jjj|jgd}|j|||}|j||d|}|S)NrrrT) from_logitsr) rDrrr&rrrrJrr%rhf_compute_loss) rtrr!targetrrr rag_logprobsrs r6rzTFRagTokenForGeneration.get_nlls!-4;;3E3E AqrE]BGGV\\!_a$8"''$++BWBWBdBdflfrfr:st u '' JG ##FLdXc#d r5ctjjdtjjj}|durr)clip_value_minclip_value_max)rrrAr?)rlossesSparseCategoricalCrossentropy ReductionSUMr clip_by_valuemathlogr not_equalrDrJr boolean_maskr reduce_sum)rtry_predsmooth_epsilonr2rloss_fnepsr melted_labels active_lossreduced_logitsnll_loss smooth_losseps_irs r6r3z'TFRagTokenForGeneration.hf_compute_losss=,,<<ll,,00=  % C%%fSQRUXQXYFWW[[(F 651 ll=$++2G2G2T2TU FRa2}}^"== mmK0 !5!5b!99n$05;3FF r5c|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYwNTr;rgetattrrrr;rHrrs r6rzTFRagTokenForGeneration.builde ::  4 % 1txx}}- %t$ % % 2 % % A11A:NNNNrDrrFrrJrrMrr)NNNNNN)rrwNNNNNNNNNNNNNNNNNNF)(rrrrrrrrrrr rr!rr$rr%rrrrrrrrrrrrrrrrrrrrrrfrrrrr)FN)rYTF)r/r0r1rIrqryrpropertyrMrJrF staticmethodrrrrrrrrrr rr!r#r0rr3rrrs@r6rrs\=.28<15,0  * 6 /  *  :' 6""""))882 9*+GH+ETcd.28<;?@D9=MQ48;?@D!%)-,0(,!&*04#'#')V *V 6V 9 V !> V 7 V KV 2V 9V !>V V 'V *V &V V $!V ".#V $!%V &!'V ()V , $-V eIV t.2+/#.0S*S)Sj9:!B <%r5rzx A TF RAG-sequence model implementation. It performs RAG-sequence specific marginalization in the forward pass. cpeZdZdZ d dfd ZddZedZedZedZ e e e e ee d ddZ dd Z d dd Zed Zdd ZxZS)TFRagSequenceForGenerationz$tf_rag_sequence_for_generation_1/ragc | ||Jd|+tj|j|jfi|}t||t |||||j d|_yrrrs r6rqz#TFRagSequenceForGeneration.__init__*rr5c&||j_yrwrrxs r6ryz(TFRagSequenceForGeneration.set_retrieverGrr5c.|jjSrwrrs r6rMz$TFRagSequenceForGeneration.retrieverJrr5c.|jjSrwrrs r6rJz$TFRagSequenceForGeneration.generatorNrr5c.|jjSrwrrs r6rFz+TFRagSequenceForGeneration.question_encoderRrr5rzc d|vsJd|r|n|jj}|r|n|jj}|||}d} |j||||||| ||| | | | ||}d}|?|j |j |j |||jj|}tdid|d|j d |j d |jd |jd |jd |jd|jd|jd|jd|j d|j"d|j$d|j&d|j(d|j*S)a exclude_bos_score (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the score of the BOS token is disregarded when computing the loss. labels (`tf.Tensor` or `np.ndarray` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the cross entropy classification loss according to Rag-Sequence model formulation See https://arxiv.org/pdf/2005.11401.pdf Section 2.1 for details about Rag-Sequence formulation. Indices should be in `[0, ..., config.vocab_size - 1]`. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `tf.Tensor.sum` operation. kwargs (`Dict[str, any]`, *optional*, defaults to `{}`): Legacy dictionary, which is required so that model can use *generate()* function. Returns: Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, TFRagSequenceForGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-sequence-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = TFRagSequenceForGeneration.from_pretrained( ... "facebook/rag-sequence-nq", retriever=retriever, from_pt=True ... ) >>> input_dict = tokenizer.prepare_seq2seq_batch( ... "How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="tf" ... ) >>> outputs = model(input_dict, output_retrieved=True) >>> # or use retriever separately >>> # 1. Encode >>> input_ids = input_dict["input_ids"] >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.numpy(), return_tensors="tf") >>> doc_scores = tf.squeeze( ... tf.matmul( ... tf.expand_dims(question_hidden_states, axis=1), docs_dict["retrieved_doc_embeds"], transpose_b=True ... ), ... axis=1, ... ) >>> # 3. Forward to generator >>> outputs = model( ... inputs=None, ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=input_dict["labels"], ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```r}r~NFrrrrr!r r$r%r"r#r&r(r)r*r+r,r-r.r4)rDexclude_bos_scorerr;rrr!rrr r$r%r"r#r&r(r)r*r+r,r-r.)rtrrrrrr r!r$r%rrrrrrdrrrrr\rrs r6rzTFRagSequenceForGeneration.callVsx $6 1 F E F 12C- HeHe%0kdkk6M6M   ($*!I(( )+/#9/#9!+/!5- $  <<""' 33  D*  >> )) $33  &77  $+#A#A  ")!=!= &77 07/Y/Y (/'I'I %,$C$C -4,S,S )0(K(K &-%E%E )0(K(K &-%E%E!  r5c 6tjddddftjjddgtjj j jjgdj jxs j j j}||nj j}tjtjdddf|} |duxr| } fd} tjj|d} tj| |jd|z|d|jdf} tjj|d} tj | d} tj | d} | ddddddddf}| ddddddddf}| ddddddddf}tj||| z|gd}tj dtj dtj"|dt%jt%|jk(sJd}||}tj&|dd }| ||\}}|r&| r$tj&|ddddddfd}ntj&|d}tj&|d}tj(j+|d}tj(j+|d}| }| }|r*tj&|}tj&|}||jdz }d |z |z||zz}|S) Nrrrctjtjjjj j }tj|r.tj|d|}tj|d|}tj|dtj|dfS)NrYrr) requalrrDrJrr% reduce_anyr'r)ll smooth_objpad_maskrtr4s r6 _mask_padsz6TFRagSequenceForGeneration.get_nll.._mask_padssxx 0E0E0R0RTZT`T`(abH}}X&XXhR0XXhZ@ ::br*BJJz,KK Kr5rrAcd}tj|d|jdf}|j}tj|d}|||}tj||S)Nctjtjtj|d|j|dddfgd}tj ||}tj |dS)Nr)r%rr)rstackrangerr% gather_ndr)r4 id_tensoridxresults r6gather2dzJTFRagSequenceForGeneration.get_nll..torch_gather..gather2dschh)).torch_gathers\ 7 ZZEKKO'<=F$??L 9g6Ifi0F::fl3 3r5)rrT)rkeepdimsr:)rr&rrrrDrJrr% bos_token_idr reduce_allrgrrrrrrPrDr@r)rtrr!r4rrrdrrzequal_bos_token_id_alluse_bosrlrrfirst_token_scoressecond_token_scores remainderr5rxrirjrLrMrNrs` ` r6rz"TFRagSequenceForGeneration.get_nlls' AqrE]BGGV\\!_a$8"''$++BWBWBdBdflfrfr:st u  {{//U4;;3H3H3U3U !-4;;3E3E!#rxxq!t l/S!Td*E/E Luu(("(= zz :++A.&8&"jFVFVWYFZ[ uu((!(< ~~l< ~~l< *!QA+6*1a1a<8 Aqr1- yy"46IL6XZc!dklm Q/R06626<< C (:(:$;;;; 4,& 9]]> %*;*C%)%:%:9Uc%:%def%g " $&,,.~~,,33# !/! " !# $- [!/8 +,)- %&+4+@Y__Q'FWF]F]^_F`djFj :&4 EE"3EFNeaiSYEY"Z 6t~~66#    #%88DVf1gQR#aggi6F6F6H2I12L1g1n1n1p,q#r -33N $ " %%!)(D~WXFY Z }5EY]^-9T9"-J- (*ww'.!)<($-C56>UZ]^U^bhTh,i),.GG4MP^`aOb,c)(25EAI3F3I(J%(*0EXYGZ([%"&:+D4+&* GGMMGFO+;@XMYZ[\M LL#3]C Di4 El  T[[5J5J5W5W XXY2hs%/K-c$t|Dcgc]}|jdc}t|Dcgc]}|jdc}f}tj||}tj |}d}|D]K}||||jdzd|jdfj |||jdz }Mtj|}tj||dddjScc}wcc}w)Nrr) sumrmaxrrVariableassignconvert_to_tensorrr%)tensorsrtroutputinds r6rz'TFRagSequenceForGeneration._cat_and_pads W556QX R 7 R NR 2R 9R !>R "R *R -R )R R *!R ".#R $$%R &$'R ()R , =-R eIR josRl.2+/#!LY*LY)LY\77(%r5r])/r2 __future__rr  dataclassesrtypingrrrrrr tensorflowrconfiguration_utilsr generationr modeling_tf_utilsr r rrrrutilsrrrrconfiguration_ragr retrieval_ragr get_loggerr/loggerrrr8r:RAG_START_DOCSTRINGrrlrr]r4r5r6rsH "" !//3/lk('   H % TBTB TBn OB;OB OBdho,hoV&R? D'':;i.%i.<i.X'  {%24P{%  {%|'  C%!57SC%  C%r5