sg3`dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd dlm Z ejBe"Z#dZ$dZ%Gdde jLZ'Gdde jLZ(Gdde jLZ)de(iZ*Gdde jLZ+Gdde jLZ,Gdde jLZ-Gdde jLZ.Gd d!e jLZ/Gd"d#eZ0d$Z1d%Z2ed&e1Gd'd(e0Z3Gd)d*e jLZ4Gd+d,e jLZ5ed-e1Gd.d/e0Z6eGd0d1eZ7ed2e1Gd3d4e0Z8y)5zPyTorch Splinter model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions ModelOutputQuestionAnsweringModelOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)SplinterConfigztau/splinter-baserc eZdZdZfdZ d deejdeejdeejdeejdee de f d Z xZ S) SplinterEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)ddt+|dd|_y) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr"selfconfig __class__s a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/splinter/modeling_splinter.pyr%zSplinterEmbeddings.__init__+s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ input_idstoken_type_idsr inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|d}||jdd|||zf}|:tj|tj|jj }||j |}|j|}||z} |jdk(r|j|} | | z } |j| } |j| } | S)Nr rdtypedevicer#) sizerr5zeroslongrGr*r.r"r,r/r3) r:r?r@rrArB input_shape seq_lengthr. embeddingsr,s r=forwardzSplinterEmbeddings.forward<s  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r>)NNNNr) __name__ __module__ __qualname____doc__r%rr5 LongTensor FloatTensorintrrN __classcell__r<s@r=rr(sQ^&1559375901 E,,-!!1!12u//0    1 12  !)  r>rcPeZdZd fd Zdej dej fdZ ddej deejdeejdeejd eejd ee e ejd ee de ej fd Z xZ S)SplinterSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|xs t#|dd|_|j$dk(s|j$d k(rF|j&|_tj(d |j&zd z |j|_|j,|_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r"r# relative_keyrelative_key_queryr)r$r%r(num_attention_headshasattr ValueErrorrUattention_head_size all_head_sizerLinearquerykeyvaluer1attention_probs_dropout_probr3r8r"r+r&distance_embedding is_decoderr:r;r"r<s r=r%zSplinterSelfAttention.__init__`s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++r>xrCc|jdd|j|jfz}|j|}|j ddddS)Nr rr_rr )rHr`rcviewpermute)r:rm new_x_shapes r=transpose_for_scoresz*SplinterSelfAttention.transpose_for_scoreszsLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r> hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc$|j|}|du} | r||d} |d} |}n| rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} |du} |j r| | f}t j| | jdd}|jdk(s|jdk(r| jd| jd}}| rDt j|dz tj|j jdd}n@t j|tj|j jdd}t j|tj|j jdd}||z }|j!||j"zdz }|j%| j& }|jdk(rt j(d | |}||z}nE|jdk(r6t j(d | |}t j(d | |}||z|z}|t+j,|j.z }|||z}t0j2j5|d}|j7|}|||z}t j|| }|j9dddd j;}|j=dd|j>fz}|j|}|r||fn|f}|j r||fz}|S)Nrrr_dimr r]r^rE)rFzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rfrrrgrhr5catrkmatmul transposer"shapetensorrJrGror6rjr+torFeinsummathsqrtrcr functionalsoftmaxr3rp contiguousrHrd)r:rsrtrurvrwrxrymixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r=rNzSplinterSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11Gr>NNNNNNF) rOrPrQr%r5TensorrrrrTrboolrNrVrWs@r=rYrY_s,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| cr>rYcneZdZfdZdej dej dej fdZxZS)SplinterSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r$r%rrer(denser/r0r1r2r3r9s r=r%zSplinterSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r>rs input_tensorrCcr|j|}|j|}|j||z}|Srrr3r/r:rsrs r=rNzSplinterSelfOutput.forward7 =1  ]3 }|'CD r>rOrPrQr%r5rrNrVrWs@r=rr1> U\\RWR^R^r>reagerceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S)SplinterAttentionct|t|j|||_t ||_t|_y)Nr") r$r%SPLINTER_SELF_ATTENTION_CLASSES_attn_implementationr:routputset pruned_headsrls r=r%zSplinterAttention.__init__sC 3F4O4OP ,C  )0 Er>c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr{)lenrr:r`rcrrrfrgrhrrrdunion)r:headsindexs r= prune_headszSplinterAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r>rsrtrurvrwrxryrCc p|j|||||||}|j|d|} | f|ddz} | S)Nrr)r:r) r:rsrtrurvrwrxry self_outputsattention_outputrs r=rNzSplinterAttention.forwardsWyy    ! "    ;;|A F#% QR(88r>rr) rOrPrQr%rr5rrrTrrrNrVrWs@r=rrs";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| r>rcVeZdZfdZdej dej fdZxZS)SplinterIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r$r%rrer(intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr9s r=r%zSplinterIntermediate.__init__/s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r>rsrCcJ|j|}|j|}|Sr)rr)r:rss r=rNzSplinterIntermediate.forward7s& =1 00? r>rrWs@r=rr.s#9U\\ellr>rcneZdZfdZdej dej dej fdZxZS)SplinterOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r$r%rrerr(rr/r0r1r2r3r9s r=r%zSplinterOutput.__init__?s` YYv779K9KL f&8&8f>S>STzz&"<"<= r>rsrrCcr|j|}|j|}|j||z}|Srrrs r=rNzSplinterOutput.forwardErr>rrWs@r=rr>rr>rceZdZfdZ d dej deejdeejdeejdeejdeeeejdee d eej fd Z d Z xZ S) SplinterLayercft||j|_d|_t ||_|j |_|j|_|jr,|j st|dt |d|_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is addedr#r)r$r%chunk_size_feed_forward seq_len_dimr attentionrkadd_cross_attentionrbcrossattentionr intermediaterrr9s r=r%zSplinterLayer.__init__Ns '-'E'E$*62 ++#)#=#=  # #?? D6)g!hii"3FT^"_D 08$V, r>rsrtrurvrwrxryrCc ||ddnd}|j|||||} | d} |jr | dd} | d} n| dd} d} |jrT|Rt|dstd|d||d dnd}|j | ||||||}|d} | |ddz} |d}  | z} t |j |j|j| }|f| z} |jr|  fz} | S) Nr_)ryrxrrr rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r}) rrkrarbrrfeed_forward_chunkrr)r:rsrtrurvrwrxryself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r=rNzSplinterLayer.forward\s}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!12 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9  7" ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44Gr>cL|j|}|j||}|Sr)rr)r:rintermediate_outputrs r=rz SplinterLayer.feed_forward_chunks,"//0@A{{#68HI r>r) rOrPrQr%r5rrrTrrrNrrVrWs@r=rrMs -"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?Br>rcDeZdZfdZ ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d e eej e ffd Z xZ S)SplinterEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r$r%r;r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r:r;_r<s r=r%zSplinterEncoder.__init__sN  ]]5IaIaCb#caM&$9#cd &+#$dsA#rsrtrurvrwpast_key_valuesrryoutput_hidden_states return_dictrCc | rdnd} |rdnd} |r|jjrdnd} |jr%|jr|rtj dd}|rdnd}t |jD]\}}| r| |fz} |||nd}|||nd}|jr/|jr#|j|j|||||||}n||||||||}|d}|r ||dfz }|s|| |dfz} |jjs| |dfz} | r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr rr_c3$K|]}|| ywrr).0vs r= z*SplinterEncoder.forward..s = slast_hidden_staterrs attentionscross_attentions) r;rrtraininglogger warning_once enumerater_gradient_checkpointing_func__call__tupler )r:rsrtrurvrwrrryrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskrx layer_outputss r=rNzSplinterEncoder.forwards#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4# VOA|#$58H$H!.7.CilO3B3N_Q/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::" &9]1=M) NNNNNNFFT)rOrPrQr%r5rrrTrrrr rNrVrWs@r=rrs ,7;15=A>BEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS r>rc"eZdZdZeZdZdZdZy)SplinterPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. splinterTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNg?)rrreweightdatanormal_r;initializer_rangebiaszero_r&rr/fill_)r:modules r= _init_weightsz%SplinterPreTrainedModel._init_weights s fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r>N) rOrPrQrRr config_classbase_model_prefixsupports_gradient_checkpointingrrr>r=rrs "L"&*#*r>raK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SplinterConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a/ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `{0}`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `{0}`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `{0}`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Splinter Model transformer outputting raw hidden-states without any specific head on top.c!eZdZdZfdZdZdZdZee jde e e e ddeej"d eej"d eej"d eej"d eej"d eej"deej"deej"deeej&deedeedeedeedeee ffdZxZS) SplinterModela* The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. ct||||_t||_t ||_|jyr)r$r%r;rrMrencoder post_initr9s r=r%zSplinterModel.__init__es;   ,V4&v.  r>c.|jjSrrMr*)r:s r=get_input_embeddingsz"SplinterModel.get_input_embeddingsos...r>c&||j_yrr)r:rhs r=set_input_embeddingsz"SplinterModel.set_input_embeddingsrs*/'r>c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r:heads_to_prunerrs r= _prune_headszSplinterModel._prune_headsusE +002 CLE5 LL  u % / / ; ;E B Cr>batch_size, sequence_length checkpoint output_typerr?rtr@rrurArvrwrrryrrrCc | | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}| | ddjdnd}|tj|||zf|}|&tj|tj| }|j||}|jjrE|C|j\}}}||f}|tj||}|j!|}nd}|j#||jj$}|j'||||| }|j)|||||| | | | | }|d}| s |f|d dzSt+||j,|j.|j0|j2 S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embedsrr_)rGrE)r?rr@rArB) rtrurvrwrrryrrrr)r;ryruse_return_dictrkrrb%warn_if_padding_and_no_attention_maskrHrGrr5onesrIrJget_extended_attention_maskinvert_attention_mask get_head_maskrrMrr rrsrr)r:r?rtr@rrurArvrwrrryrrrK batch_sizerLrGrBextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputs r=rNzSplinterModel.forward}sR2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!) NNNNNNNNNNNNN)rOrPrQrRr%r r"r&rSPLINTER_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr5rrrTrrrrNrVrWs@r=rrZs  /0C++D+K+KLi+jk&=$-11515/3,0048<9==A$(,0/3&*w ELL)w !.w !. w u||, w ELL) w   -w  ( 5w !) 6w "$u'8'8"9:w D>w $D>w 'tnw d^w  u?? @w l w r>rcXeZdZdfd Zdej dej fdZxZS)SplinterFullyConnectedLayerct|||_||_t j |j|j|_t||_t j|j|_ yr) r$r% input_dim output_dimrrerr act_fnr/)r:rBrCrr<s r=r%z$SplinterFullyConnectedLayer.__init__sV "$YYt~~t? Z( doo6r>inputsrCcl|j|}|j|}|j|}|Sr)rrDr/)r:rErss r=rNz#SplinterFullyConnectedLayer.forwards2 6*  M2 }5 r>)gelurrWs@r=r@r@s#7ellu||r>r@c(eZdZdZfdZdZxZS)QuestionAwareSpanSelectionHeadzf Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: ct|t|j|j|_t|j|j|_t|j|j|_t|j|j|_tj|j|jd|_ tj|j|jd|_ y)NF)r) r$r%r@r(query_start_transformquery_end_transformstart_transform end_transformrrestart_classifierend_classifierr9s r=r%z'QuestionAwareSpanSelectionHead.__init__s %@ASASU[UgUg%h"#>v?Q?QSYSeSe#f :6;M;MvOaOab89K9KVM_M_` " &*<*P>PW\ ] ii(:(:Fc|j\}}}|jdjdd|}tj|d|}|j |}|j |}|j|} |j|} |j|} | jddd} tj| | } |j|} | jddd} tj| | } | | fS)Nr r)r|rrr_) rH unsqueezerepeatr5gatherrKrLrMrNrOrprrP)r:rE positionsrr|r gathered_repsquery_start_repsquery_end_reps start_repsend_repsrs start_logits end_logitss r=rNz&QuestionAwareSpanSelectionHead.forward sKKM 1c##B'..q!S9 V%@ 55mD11-@))&1 %%f---.>? ''1a0 ||M:> ++N; ##Aq!,\\-: Z''r>)rOrPrQrRr%rNrVrWs@r=rIrIs \(r>rIz Splinter Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede ede e jdeee ffdZxZS)SplinterForQuestionAnsweringct||t||_t ||_|j |_|jyrr$r%rr rI splinter_qassquestion_token_idrr9s r=r%z%SplinterForQuestionAnswering.__init__=C  %f- ;FC!'!9!9 r>r'r(r?rtr@rrurAstart_positions end_positionsryrrquestion_positionsrCc | | n|jj} d} | |Dtjtj||j j d}nJtj|jdtj|j|j}|jd} d} |j||||||| | |  }|d}|j|| \}}| r"|jd |jd }}|d|d |z tj |j"j$zz}|d |z tj |j"j$zz}d}||t'|jd kDr|jd}t'|jd kDr|jd}|jd }|j)d||j)d|t+| }|||}|||}||zd z }| s||f|d dz}||f|zS|St-||||j.|j0 S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NFr r{r)rFlayoutrGTrtr@rrurAryrrr ignore_indexr_lossr[r\rsr)r;r,r5argmaxeqrbrUrIrHrJrhrGrRr rasqueezefinforFminrclamp_r rrsr)r:r?rtr@rrurArdreryrrrfquestion_positions_were_none"question_position_for_each_examplerr:r[r\ total_loss ignored_indexloss_fct start_lossend_lossrs r=rNz$SplinterForQuestionAnswering.forwardGsH&1% @R@RST@U*L  %'1~+=\M_M_A`AdAd*ddL#q>'9U[[IYIY=Z=^=^&^^J  &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M  " "1m 4  M 2']CH!,@J M:H$x/14J"J/'!"+=F/9/EZMF* Q6 Q+%!!//))   r> NNNNNNNNNNNN)rOrPrQr%rr;r<rr=rr>rr5rrSrrrrNrVrWs@r=r^r^5sh++D+K+KLi+jk&0$-11515/3,0046:48,0/3&*9=^ ELL)^ !.^ !. ^ u||, ^ ELL) ^   -^ "%"2"23^   0 01^ $D>^ 'tn^ d^^ %U%5%56^  u22 3^ l ^ r>r^ceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<y)SplinterForPreTrainingOutputa Class for outputs of Splinter as a span selection model. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrmr[r\rsr)rOrPrQrRrmrr5rT__annotations__r[r\rsrrrr>r=r}r}sr.)-D(5$$ %,&*L%##*$(J!!(8r}z Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. ceZdZfdZeej d ddeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee deejde e effdZdejdejfdZxZS)SplinterForPreTrainingct||t||_t ||_|j |_|jyrr`r9s r=r%zSplinterForPreTraining.__init__rcr>z*batch_size, num_questions, sequence_lengthr?rtr@rrurArdreryrrrfrCc b| | n|jj} | | | td| | td| |j|} |j ||||||| | |  } | d}|j \}}}|j || \}}| j d}||jdj|||}|d|z tj|jjzz}|d|z tj|jjzz}d}|||jdtd|dz |jdtd|dz t|jj }||j#||z||j#||z}||j#||z||j#||z}||zdz }| s||f| ddz}||f|zS|St%|||| j&| j( S) a start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz>question_positions must be specified when input_embeds is usedrirrrjr_rl)r;r, TypeError_prepare_question_positionsr rHrarRr7r5rqrFrrrsmaxr r)ror}rsr)r:r?rtr@rrurArdreryrrrfrr:r2sequence_lengthr|r[r\ num_questions attention_mask_for_each_questionrvrxryrzrs r=rNzSplinterForPreTraining.forwardsB&1%$? @ (T[[5M5MNH!!!*}"H%x/14J"J/'!"+=F/9/EZMF* Q6 Q+%!!//))   r>ctj||jjk(\}}tj|}tj |j d|jf|jjtj|j}tj|Dcgc]}tj|c}}||||f<|Scc}w)NrrE) r5wherer;rbbincountfullrHrr)rJrGr~r6)r:r?rowsflat_positionsrrUncolss r=rz2SplinterForPreTraining._prepare_question_positionsHs${{9 8U8U+UVnt, JJ ^^A  1 1 3 4 KK $ $**##  yy=Aa%,,q/AB . $*Bs<C(r{)rOrPrQr%rr;r<rr5rrSrrrr}rNrrVrWs@r=rrso+!(()UV -11515/3,0046:48,0/3&*9=b ELL)b !.b !. b u||, b ELL) b   -b "%"2"23b   0 01b $D>b 'tnb d^b %U%5%56b  u22 3b b H U\\ ell r>r)9rRr dataclassesrtypingrrrrr5torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_splinterr get_loggerrOrr=r>ModulerrYrrrrrrrrSPLINTER_START_DOCSTRINGr;rr@rIr^r}rrr>r=rs !// %!tt-lluu2   H %)"33nCBIICN   "# 0 0h 299  RYY SBIISnZ biiZ z*o*8 /dh\ +\ \ ~"))$#(RYY#(L o #:o o d :;: :>  }4}}r>