sg d Z ddlmZ ddlZddlZddlZddlmZ ddlm Z m
Z
m Z m Z ddl
ZddlZddlmZ ddlmZmZmZmZmZ dd lmZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z% dd
l&m'Z'm(Z(m)Z) dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0 d d
l1m2Z2 e0jf e4 Z5dZ6dZ7dZ8dZ9d1dZ: G d de Z; e.de8 G d de; Z< G d de#jz j| Z? G d de#jz j| Z@e$ G d d e#jz j| ZA G d d e#jz j| ZBe G d d!e, ZC e.d"e8 G d# d$e; ZD e.d%e8 G d& d'e;e ZE e.d(e8 G d) d*e;e ZF e.d+e8 G d, d-e;e! ZG e.d.e8 G d/ d0e;e ZHy)2z
TF 2.0 Flaubert model.
) annotationsN) dataclass)DictOptionalTupleUnion )get_tf_activation)TFBaseModelOutputTFMultipleChoiceModelOutput TFQuestionAnsweringModelOutputTFSequenceClassifierOutputTFTokenClassifierOutput) TFModelInputTypeTFMultipleChoiceLossTFPreTrainedModelTFQuestionAnsweringLoss TFSequenceClassificationLossTFSequenceSummaryTFSharedEmbeddingsTFTokenClassificationLossget_initializerkeraskeras_serializable
unpack_inputs) check_embeddings_within_bounds
shape_liststable_softmax) MULTIPLE_CHOICE_DUMMY_INPUTS ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging )FlaubertConfigz flaubert/flaubert_base_casedr& a~
This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior.
TensorFlow models and layers in `transformers` accept two formats as input:
- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.
The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like `model.fit()` things should "just work" for you - just
pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second
format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with
the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument:
- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
`model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring:
`model({"input_ids": input_ids, "token_type_ids": token_type_ids})`
Note that when creating models and layers with
[subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) then you don't need to worry
about any of this, as you can just pass inputs like you would to any other Python function!
Parameters:
config ([`FlaubertConfig`]): Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
a<
Args:
input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`):
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and
[`PreTrainedTokenizer.encode`] for details.
[What are input IDs?](../glossary#input-ids)
attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- `1` for tokens that are **not masked**,
- `0` for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
langs (`tf.Tensor` or `Numpy array` of shape `(batch_size, sequence_length)`, *optional*):
A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are
languages ids which can be obtained from the language names by using two conversion mappings provided in
the configuration of the model (only provided for multilingual models). More precisely, the *language name
to language id* mapping is in `model.config.lang2id` (which is a dictionary string to int) and the
*language id to language name* mapping is in `model.config.id2lang` (dictionary int to string).
See usage examples detailed in the [multilingual documentation](../multilingual).
token_type_ids (`tf.Tensor` or `Numpy array` of shape `(batch_size, sequence_length)`, *optional*):
Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,
1]`:
- `0` corresponds to a *sentence A* token,
- `1` corresponds to a *sentence B* token.
[What are token type IDs?](../glossary#token-type-ids)
position_ids (`tf.Tensor` or `Numpy array` of shape `(batch_size, sequence_length)`, *optional*):
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
config.max_position_embeddings - 1]`.
[What are position IDs?](../glossary#position-ids)
lengths (`tf.Tensor` or `Numpy array` of shape `(batch_size,)`, *optional*):
Length of each sentence that can be used to avoid performing attention on padding token indices. You can
also use *attention_mask* for the same result (see above), kept here for compatibility Indices selected in
`[0, ..., input_ids.size(-1)]`:
cache (`Dict[str, tf.Tensor]`, *optional*):
Dictionary string to `tf.FloatTensor` that contains precomputed hidden states (key and values in the
attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential
decoding.
The dictionary object will be modified in-place during the forward pass to add newly computed
hidden-states.
head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*):
Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
- `1` indicates the head is **not masked**,
- `0` indicates the head is **masked**.
inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
model's internal embedding lookup matrix.
output_attentions (`bool`, *optional*):
Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.
output_hidden_states (`bool`, *optional*):
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.
return_dict (`bool`, *optional*):
Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.
training (`bool`, *optional*, defaults to `False`):
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).
c
t | d }||}n;t j | |j }|t j |d k }|r\t j
t j t j dd| f || df t j |d| df }n|}t j j t | || g |r,t j j t | || | g ||fS )zH
Generate hidden states mask, and optionally an attention mask.
r dtyper% axis)
r tfranger) expand_dims
less_equaltilereshape debugging assert_equal)slenlengthscausal padding_maskbsmaskalen attn_masks d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/flaubert/modeling_tf_flaubert.py get_masksr= s
G Q B xxGMM2bnnW155 MM GGBJJtaD\2RqM BBJJtVWY]_`UaDb
LL j .T
;
!!*Y"7"dD9IJ ? c ( e Zd ZdZeZdZed Zy)TFFlaubertPreTrainedModelz
An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
models.
transformerc t j g dg dg dgt j }t j g dg dg dgt j }| j j rK| j j
dkD r2||t j g dg dg dgt j d S ||d
S ) N) r r r% )r% r r r )r r r r( )r% r% r r r% )r% r% r% r r )r% r r r% r% r% ) input_idsattention_masklangs)rH rI )r, constantint32config use_lang_embn_langs)self inputs_list
attns_lists r< dummy_inputsz&TFFlaubertPreTrainedModel.dummy_inputs s kk?O_"U]_]e]ef [[/?O!T\^\d\de
;; # # (;(;a(? (",o%X`b`h`hi
"-
K Kr> N) __name__
__module__ __qualname____doc__r& config_classbase_model_prefixpropertyrS r> r< r@ r@ s(
"L%
L Lr> r@ zbThe bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.c e Zd Z fdZe ee eee e
d dd Z ddZ xZ
S )TFFlaubertModelc P t | |g|i | t |d | _ y )NrA name)super__init__TFFlaubertMainLayerrA rP rM inputskwargs __class__s r< rb zTFFlaubertModel.__init__ s)
3&3F3.vMJr>
checkpoint output_typerX c B | j ||||||||| |
| | |
}|S )N
rH rI rJ token_type_ids position_idsr5 cache head_mask
inputs_embedsoutput_attentionsoutput_hidden_states return_dicttraining)rA )rP rH rI rJ rm rn r5 ro rp rq rr rs rt ru outputss r< callzTFFlaubertModel.call sG . "" ))% ' /!5# #
r> c | j ry d| _ t | dd Nt j | j j
5 | j j
d d d d y y # 1 sw Y y xY w)NTrA )builtgetattrr,
name_scoperA r` buildrP input_shapes r< r| zTFFlaubertModel.build" sm ::
4 - 9 t / / 4 45
- &&t,
-
- :
-
-s A11A:
NNNNNNNNNNNNF rH np.ndarray | tf.Tensor | NonerI r rJ r rm r rn r r5 r ro Optional[Dict[str, tf.Tensor]]rp r rq tf.Tensor | Nonerr Optional[bool]rs r rt r ru r returnz Union[Tuple, TFBaseModelOutput]N)rT rU rV rb r r# FLAUBERT_INPUTS_DOCSTRINGr! _CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrw r|
__classcell__rg s @r< r] r] s
K *+DE &%$ 488 r] c V e Zd Z ej Z fdZd ZddZddZ xZ
S ) TFFlaubertMultiHeadAttentionc 2 t | di | t t j | _ || _ || _ |j | _ | j | j z dk( sJ t j j |t |j d | _ t j j |t |j d | _ t j j |t |j d | _ t j j |t |j d | _ t j j% |j& | _ t+ | _ || _ y )Nr q_linkernel_initializerr` k_linv_linout_linr[ )ra rb nextr NEW_IDlayer_iddimn_headsrr r layersDenser init_stdr r r r Dropoutattention_dropoutdropoutset pruned_heads)rP r r rM rf rg s r< rb z%TFFlaubertMultiHeadAttention.__init__/ s2
"6" 9 @ @A
!'!9!9 xx$,,&!+++\\''PVP_P_@`gn'o
\\''PVP_P_@`gn'o
\\''PVP_P_@`gn'o
||))#/RXRaRaBbir)s ||++F,D,DE Er> c t r )NotImplementedError)rP headss r< prune_headsz(TFFlaubertMultiHeadAttention.prune_heads? s !!r> c \ t | \ }} |
||n|d |z }
nt | d }
j j z t t | dk( rd||
fndd|
f} fd} fd}
| j | }|/ | j | } | j
| }nB| j |vr2|x}} | j | } | j
| }|x j |v rY|E| j \ }}t j |gd }t j |gd }n| j \ }}f| j < t j |j }t j |t j j | }t j |j }t j ||d
}t j || }t j ||j }|d d |z
z z
}t# |d
} j% || }|||z }t j | } |
| } j' | f}|r||fz }|S )zd
Self-attention (if kv is None) or attention over source sentence (provided by kv).
r4 r% r c v t j t j | dj f d S )
projectionr rE r% r perm)r, transposer1 r xr8 dim_per_headrP s r< shapez0TFFlaubertMultiHeadAttention.call..shapeS s- <<
1r2t||\.R SZfg gr> c z t j t j | d dj z f S )zcompute contextr r r )r, r1 r r r s r< unshapez2TFFlaubertMultiHeadAttention.call..unshapeW s0 ::bll1< @2r4< rE r* r( T) transpose_bgꌠ9Y>)Fg ?r ru )r r r lenr r r r r, concatcastr) multiplymathrsqrtmatmulr1 r r r )rP inputr9 kvro rp rr ru qlenr klen mask_reshaper r qkvk_v_f_dim_per_headscoresweightscontextrv r8 r s ` @@r< rw z!TFFlaubertMultiHeadAttention.callB sx #5)
D#
: =4eFmd.BD b>!$D xx4<</ .1*T2B.Cq.HAtT*rSTVWY]N^ h i
$**U# $
:djj'(Adjj'(A
]dmm5 8JAdjjm$Adjjm$A }}%: "4== 1FB 2q'2A 2q'2A /DAq$%q6E$-- QWW= KK277== 8 9 GGAQWW %1aT2zz$ -wwt6<<0$#*-- b1,,w,: )G))GQ''"<<(*
*Gr> c | j ry d| _ t | dd Zt j | j j
5 | j j
d d | j g d d d t | dd Zt j | j j
5 | j j
d d | j g d d d t | dd Zt j | j j
5 | j j
d d | j g d d d t | dd [t j | j j
5 | j j
d d | j g d d d y y # 1 sw Y AxY w# 1 sw Y xY w# 1 sw Y xY w# 1 sw Y y xY w)NTr r r r ) ry rz r, r{ r r` r| r r r r r} s r< r| z"TFFlaubertMultiHeadAttention.build ss ::
4$ ' 3 tzz/
9
$dhh!78
9 4$ ' 3 tzz/
9
$dhh!78
9 4$ ' 3 tzz/
9
$dhh!78
9 4D ) 5 t|| 0 01
; ""D$#9:
;
; 6
9
9
9
9
9
9
;
;s0 )F32)G )G )G3F= G GG!Fr ) rT rU rV itertoolscountr rb r rw r| r r s @r< r r , s'
Y__
F "DL;r> r c . e Zd Z fdZddZddZ xZS )TFFlaubertTransformerFFNc t | di | t j j |t |j d | _ t j j |t |j d | _ |j r t d n
t d | _ t j j |j | _
|| _ || _ y )Nlin1r lin2gelurelur[ )ra rb r r r r r r r gelu_activationr
actr r in_dim
dim_hidden)rP r r out_dimrM rf rg s r< rb z!TFFlaubertTransformerFFN.__init__ s
"6"LL&&zoV\VeVeFfms&t LL&&w?SYSbSbCcjp&q 060F0F$V,L]^dLe ||++FNN; $r> c | j | }| j | }| j | }| j || }|S )Nr )r r r r )rP r ru r s r< rw z TFFlaubertTransformerFFN.call sA IIe HHQK IIaL LLXL .r> c | j ry d| _ t | dd Zt j | j j
5 | j j
d d | j g d d d t | dd [t j | j j
5 | j j
d d | j g d d d y y # 1 sw Y rxY w# 1 sw Y y xY w)NTr r )
ry rz r, r{ r r` r| r r r r} s r< r| z TFFlaubertTransformerFFN.build s ::
4 & 2 tyy~~.
; tT[[ 9:
; 4 & 2 tyy~~.
? tT__ =>
?
? 3
;
;
?
?s )C%2)C1%C.1C:r r )rT rU rV rb rw r| r r s @r< r r s % ?r> r c e Zd ZeZ fdZddZd Zd Ze d dd Z
xZ S ) rc c t | di | || _ |j | _ |j | _ |j
| _ | j dz | _ |j | _ |j | _ |j | _
|j | _ |j | _ t |dd | _ t |dd | _ |j | _ |j" | _ |j$ | _ |j( | _ |j* | _ t, j. j1 |j2 | _ t5 | j | j |j* d | _ t, j. j9 |j: d | _ g | _ g | _ g | _! g | _" tG | j D ] }| j> jI tK | j | j |d
| | j@ jI t, j. j9 |j: d | | jB jI tM | j | j | j |d
| | jD jI t, j. j9 |j: d| y )NrF layerdropg pre_normF
embeddingsinitializer_ranger` layer_norm_emb)epsilonr` z
attentions_._)rM r` zlayer_norm1_._zffns_._zlayer_norm2_._r[ )'ra rb rM r rO emb_dimr
hidden_dimn_words pad_indexr6 n_layersrN rz r r rr rs use_return_dictrt max_position_embeddingsembed_init_stdr r r r r r LayerNormalizationlayer_norm_epsr
attentions layer_norm1ffns layer_norm2r- appendr r )rP rM rf irg s r< rb z TFFlaubertMainLayer.__init__ sl
"6" ~~ ~~ >> ((Q, ~~ )) mm
" / / c:
E:
!'!9!9 $*$?$?!!11'-'E'E$ $ 3 3 ||++FNN; , LL$((f6K6KR^
$ll = =fF[F[br = s t}}% A OO " ",T\\488F[hijhkYlm
# # //8M8MVdefdgTh/i
II (4??DHHU[dklmknbop
# # //8M8MVdefdgTh/i
r> c t j d 5 | j d| j | j gt | j | _ d d d | j dkD rl| j r`t j d 5 | j d| j | j gt | j | _
d d d | j ry d| _ t | dd Mt j | j j 5 | j j d d d d t | dd Zt j | j j 5 | j j d d | j g d d d | j" D ]; }t j |j 5 |j d d d d = | j$ D ]H }t j |j 5 |j d d | j g d d d J | j& D ]; }t j |j 5 |j d d d d = | j( D ]H }t j |j 5 |j d d | j g d d d J y # 1 sw Y xY w# 1 sw Y /xY w# 1 sw Y xY w# 1 sw Y txY w# 1 sw Y uxY w# 1 sw Y 8xY w# 1 sw Y xY w# 1 sw Y xY w)Nposition_embeddingsr )r` r initializerr% lang_embeddingsTr )r, r{
add_weightr r r r r rO rN r ry rz r r` r| r r r r r )rP r~ layers r< r| zTFFlaubertMainLayer.build s
]]0
1 '+!33TXX> +D,?,? @ (7 (D $ <L LL%(L2 5L? M M c | j S r )r rP s r< get_input_embeddingsz(TFFlaubertMainLayer.get_input_embeddings r> c ` || j _ t | d | j _ y Nr )r weightr
vocab_sizerP values r< set_input_embeddingsz(TFFlaubertMainLayer.set_input_embeddings s$ !& %/%6q%9"r> c
4
|
| t d |t | \ }}n | t | d d \ }}n t d |p|Ut j t j t j
|| j |j d }nt j |g|z }t j j t | d | dt | d d | d
f t ||| j | \ }}|Bt j t j | d }t j ||df }t j j t | ||g d t | d
||g d
f |?t j j t | ||g dt | d
||g d
f |t d g| j" z }|H|F||d z
}|d d | d f }|d d | d f }| |d d | d f }|d d | d f }|d d | d f }| 1t% || j& j( | j' | } | t j* | j, | z }|/| j. r#|t j* | j0 | z }||| j' | z }| j3 | }| j5 ||
}t j ||j }|t j |d z }| rdnd }|
rdnd }t | j" D ]y }t7 j8 dd }|
r|| j: k r,| r||fz }| j< s] | j> | ||d ||| |
|
}|d }|
r ||d fz }| j5 ||
}||z } | j@ | | }n\ | j@ | | } | j> | ||d ||| |
|
}|d }|
r ||d fz }| j5 ||
}||z }| j< s.| | jB | | z } | jD | | }n- | jD | | }| | jB | | z }|t j |d z }| | r||fz }| |dxx |jG d z
cc< | stI d |||fD S tK ||| S )NzDYou cannot specify both input_ids and inputs_embeds at the same timerE z5You have to specify either input_ids or inputs_embedsr( r% r* r zExpected batch size z and received batch size z mismatched)r7 zPosition id shape z and input shape z Lang shape r4 r r r[ c 3 & K | ] }|| y wr r[ ).0r s r< z+TFFlaubertMainLayer.call.. s Yq1=Ys )last_hidden_state
hidden_statesr )&
ValueErrorr r,
reduce_sumr not_equalr r) convert_to_tensorr2 r3 r= r6 r. r- r0 r r r r r gatherr rN r r r randomuniformr r r r r r sizetupler )rP rH rI rJ rm rn r5 ro rp rq rr rs rt ru r8 r4 r9 r; _slentensorr r r dropout_probability attn_outputsattntensor_normalizeds r< rw zTFFlaubertMainLayer.call s| & ]%> cd d
"!),HB
&!-0!4HB TU U ?$ --GGBLLDNN C9??[bc ..v{;
LL % %j&9!&>"((4.qAL 77<"a9L
LL % %j &>T
K L!9 ::KRQUJ> A# - 9
== 1tq 1 aL% % $A$!+|A.@!@J||D8|< $ ,))!,V4$7D$4$4Q$7$?! 1tq 1% aL% % $A$!+|A.@!@J||D8|< $ == ,$))A,v"66,))!,V4$7D$4$4Q$7$?! ,$))A,/@"AAbnnT ;;F? r r r ) rT rU rV r& rX rb r| r r r rw r r s @r< rc rc s !L-^$4L : 488 rc c J e Zd ZdZ fdZ fdZd Zd Zd Zd Z d Z
xZ S ) TFFlaubertPredLayerz?
Prediction layer (cross_entropy or adaptive_softmax).
c t | di | |j | _ |j | _ |j | _ |j du r|| _ y t )NFr[ )ra rb asmr r input_embeddingsr )rP rM r rf rg s r< rb z TFFlaubertPredLayer.__init__ sP
"6":: ~~ )) :: $4D !% %r> c n | j | j fddd | _ t | | y )NzerosTbias)r r trainabler` )r r r# ra r| )rP r~ rg s r< r| zTFFlaubertPredLayer.build s/ OO4< c | j S r )r r s r< get_output_embeddingsz)TFFlaubertPredLayer.get_output_embeddings s $$$r> c ` || j _ t | d | j _ y r )r r r r r s r< set_output_embeddingsz)TFFlaubertPredLayer.set_output_embeddings s( ', $+5e+ c d| j iS )Nr# )r# r s r< get_biasz TFFlaubertPredLayer.get_bias s ""r> c D |d | _ t |d d | _ y )Nr# r )r# r r r s r< set_biasz TFFlaubertPredLayer.set_bias s" &M $U6]3A6r> c J | j |d }|| j z }|S )Nlinear)mode)r r# )rP r s r< rw zTFFlaubertPredLayer.call s+ --m(-K
% 1
r> ) rT rU rV rW rb r| r& r( r* r, rw r r s @r< r r s,
&&# %@#7 r> r c < e Zd ZU dZdZded< dZded< dZded< y) TFFlaubertWithLMHeadModelOutputac
Base class for [`TFFlaubertWithLMHeadModel`] outputs.
Args:
logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape
`(batch_size, sequence_length, hidden_size)`.
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
Nz tf.TensorlogitszTuple[tf.Tensor] | Noner r )rT rU rV rW r2 __annotations__r r r[ r> r< r1 r1 s) & FI -1M*1*.J'.r> r1 z
The Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input
embeddings).
c e Zd Z fdZd Zd Zd Ze ee e
e e e
d d d Zd
dZ xZS ) TFFlaubertWithLMHeadModelc t | |g|i | t |d | _ t || j j
d | _ d| _ y )NrA r_ zpred_layer_._projF)ra rb rc rA r r
pred_layersupports_xla_generationrd s r< rb z"TFFlaubertWithLMHeadModel.__init__8 sN
3&3F3.vMJ-fd6F6F6Q6QXkl',$r> c | j S r )r7 r s r< get_lm_headz%TFFlaubertWithLMHeadModel.get_lm_head? r r> c t j dt | j dz | j j z S )NzMThe method get_prefix_bias_name is deprecated. Please use `get_bias` instead./)warningswarn
FutureWarningr` r7 r s r< get_prefix_bias_namez.TFFlaubertWithLMHeadModel.get_prefix_bias_nameB s/
egtuyy3 !5!555r> c $ | j j }| j j }|j d }t j
|dfd |z }t j ||gd }|t j | |z }nd }||dS )Nr r% r* rH rJ )rM
mask_token_idlang_idr r, fillr ones_like)rP re rf rC rD effective_batch_size
mask_tokenrJ s r< prepare_inputs_for_generationz7TFFlaubertWithLMHeadModel.prepare_inputs_for_generationF s 11
++%% %||A WW 2A 6:]J
FJ/a8 LL(72EE #e44r> rh c | j ||||||||| |
| | |
}|d }| j | }| s |f|dd z S t ||j |j S )Nrl r r% )r2 r r )rA r7 r1 r r )rP rH rI rJ rm rn r5 ro rp rq rr rs rt ru transformer_outputsoutputrv s r< rw z TFFlaubertWithLMHeadModel.callT s . # . . ))% ' /!5# /
%Q'//&): 3AB 77 7.*=*K*KXkXvXv
r> c | j ry d| _ t | dd Mt j | j j
5 | j j
d d d d t | dd Nt j | j j
5 | j j
d d d d y y # 1 sw Y exY w# 1 sw Y y xY w)NTrA r7 )ry rz r, r{ rA r` r| r7 r} s r< r| z TFFlaubertWithLMHeadModel.build s ::
4 - 9 t / / 4 45
- &&t,
- 4 t , 8 t 3 34
,%%d+
,
, 9
-
-
,
, C % C CC r ) rH r rI r rJ r rm r rn r r5 r ro r rp r rq r rr r rs r rt r ru r r z-Union[Tuple, TFFlaubertWithLMHeadModelOutput]r )rT rU rV rb r: r@ rI r r# r r! r r1 r rw r| r r s @r< r5 r5 0 s - 6 5 *+DE &3$ 488 r5 z
Flaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output)
e.g. for GLUE tasks.
c e Zd Z fdZe eej d ee e
e d dd Z ddZ
xZS ) #TFFlaubertForSequenceClassificationc t | |g|i | |j | _ t |d | _ t ||j d | _ y )NrA r_ sequence_summaryr )ra rb
num_labelsrc rA r r rR rd s r< rb z,TFFlaubertForSequenceClassification.__init__ sL
3&3F3 ++.vMJ 1&FOObt u r> batch_size, sequence_lengthrh c | j ||||||||| |
| | |
}|d }| j | }|
dn| j |
| }| s|f|dd z }||f|z S |S t |||j |j
S )a
labels (`tf.Tensor` of shape `(batch_size,)`, *optional*):
Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
rl r Nr% lossr2 r r )rA rR hf_compute_lossr r r )rP rH rI rJ rm rn r5 ro rp rq rr rs rt labelsru rK rL r2 rW s r< rw z(TFFlaubertForSequenceClassification.call s < # . . ))% ' /!5# /
%Q'&&v. ~t4+?+?+O Y!4QR!88F)-)9TGf$ Ev E)-;;*55
r> c | j ry d| _ t | dd Mt j | j j
5 | j j
d d d d t | dd Nt j | j j
5 | j j
d d d d y y # 1 sw Y exY w# 1 sw Y y xY w)NTrA rR )ry rz r, r{ rA r` r| rR r} s r< r| z)TFFlaubertForSequenceClassification.build s ::
4 - 9 t / / 4 45
- &&t,
- 4+T 2 > t 4 4 9 9:
2%%++D1
2
2 ?
-
-
2
2rN NNNNNNNNNNNNNF) rH TFModelInputType | NonerI r rJ r rm r rn r r5 r ro r rp r rq r rr r rs r rt r rY r ru boolr z3Union[TFSequenceClassifierOutput, Tuple[tf.Tensor]]r )rT rU rV rb r r# r formatr! r r r rw r| r r s @r< rP rP s v *+D+K+KLi+jk &.$ .28 rP z
Flaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layer on top of the hidden-states output to compute `span start logits` and `span end logits`).
c e Zd Z fdZe eej d ee e
e d dd Z ddZ
xZS ) $TFFlaubertForQuestionAnsweringSimplec t | |g|i | t |d | _ t j
j
|j t |j d | _
|| _ y )NrA r_
qa_outputsr ) ra rb rc rA r r r rS r r rb rM rd s r< rb z-TFFlaubertForQuestionAnsweringSimple.__init__ sd
3&3F3.vMJ ,,,, /&//2RYe -
r> rT rh c | j ||||||||| |
| | |
}|d }| j | }t j |dd \ }}t j |d }t j |d }d}|
| d|
i}||d< | j |||f }| s||f|d d z }||f|z S |S t
||||j |j
S ) a
start_positions (`tf.Tensor` of shape `(batch_size,)`, *optional*):
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
are not taken into account for computing the loss.
end_positions (`tf.Tensor` of shape `(batch_size,)`, *optional*):
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
are not taken into account for computing the loss.
rl r rE r r* Nstart_position end_positionr% )rW start_logits
end_logitsr r ) rA rb r, splitsqueezerX r
r r )rP rH rI rJ rm rn r5 ro rp rq rr rs rt start_positions
end_positionsru rK sequence_outputr2 rf rg rW rY rL s r< rw z)TFFlaubertForQuestionAnsweringSimple.call s' F # . . ))% ' /!5# /
.a0 1#%88FAB#? jzz,R8 ZZ
4
&=+D&8F%2F> "'' z0JKD"J/2Eab2IIF)-)9TGf$ Ev E-%!-;;*55
r> c | j ry d| _ t | dd Mt j | j j
5 | j j
d d d d t | dd et j | j j
5 | j j
d d | j j g d d d y y # 1 sw Y |xY w# 1 sw Y y xY w)NTrA rb )
ry rz r, r{ rA r` r| rb rM hidden_sizer} s r< r| z*TFFlaubertForQuestionAnsweringSimple.buildF ::
4 - 9 t / / 4 45
- &&t,
- 4 t , 8 t 3 34
M%%tT4;;3J3J&KL
M
M 9
-
-
M
M C"%3C."C+.C7)NNNNNNNNNNNNNNF) rH r\ rI r rJ r rm r rn r r5 r ro r rp r rq r rr r rs r rt r rj r rk r ru r] r z7Union[TFQuestionAnsweringModelOutput, Tuple[tf.Tensor]]r )rT rU rV rb r r# r r^ r! r r
r rw r| r r s @r< r` r` s4 *+D+K+KLi+jk &2$ .28 r` z
Flaubert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for
Named-Entity-Recognition (NER) tasks.
c e Zd Z fdZe eej d ee e
e d dd Z ddZ
xZS ) TFFlaubertForTokenClassificationc d t | |g|i | |j | _ t |d | _ t
j j |j | _ t
j j |j t |j d | _ || _
y )NrA r_
classifierr )ra rb rS rc rA r r r r r r r rt rM rd s r< rb z)TFFlaubertForTokenClassification.__init__[ s
3&3F3 ++.vMJ ||++FNN; ,,,, /&//2RYe -
r> rT rh c , | j ||||||||| |
| | |
}|d }| j || }| j | }|
dn| j |
| }| s|f|dd z }||f|z S |S t |||j
|j S )z
labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
rl r r Nr% rV )rA r rt rX r r r )rP rH rI rJ rm rn r5 ro rp rq rr rs rt rY ru rK rl r2 rW rL s r< rw z%TFFlaubertForTokenClassification.callf s 8 # . . ))% ' /!5# /
.a0 ,,,J 1 ~t4+?+?+O Y!4QR!88F)-)9TGf$ Ev E&-;;*55
r> c | j ry d| _ t | dd Mt j | j j
5 | j j
d d d d t | dd et j | j j
5 | j j
d d | j j g d d d y y # 1 sw Y |xY w# 1 sw Y y xY w)NTrA rt )
ry rz r, r{ rA r` r| rt rM rn r} s r< r| z&TFFlaubertForTokenClassification.build ro rp r[ ) rH r\ rI r rJ r rm r rn r r5 r ro r rp r rq r rr r rs r rt r rY r ru r] r z0Union[TFTokenClassifierOutput, Tuple[tf.Tensor]]r )rT rU rV rb r r# r r^ r! r r r rw r| r r s @r< rr rr R s *+D+K+KLi+jk &+$ .28 rr z
Flaubert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks.
c e Zd Z fdZed Ze eej d e
e e e
d dd Zd dZ xZS )
TFFlaubertForMultipleChoicec t | |g|i | t |d | _ t ||j
d | _ t j j dt |j d | _ || _
y )NrA r_ rR r r% logits_projr )ra rb rc rA r r rR r r r r r rz rM rd s r< rb z$TFFlaubertForMultipleChoice.__init__ sw
3&3F3.vMJ 1&FOObt u <<--
/&2J2J"KR_ .
r> c \ | j j rl| j j dkD rSt j t
t j t j t
t j dS dt j t
t j iS )zn
Dummy inputs to build the network.
Returns:
tf.Tensor with dummy inputs
r% r( rB rH )rM rN rO r, rK r rL r s r< rS z(TFFlaubertForMultipleChoice.dummy_inputs ss ;; # # (;(;a(? [[)ERXX V%AR
R[[)ERXX V
r> z(batch_size, num_choices, sequence_lengthrh c j | t | d }t | d }n t | d }t | d }|t j |d|f nd }|t j |d|f nd }|t j |d|f nd }|t j |d|f nd }|t j |d|f nd }| %t j | d|t | d f nd }|t j d d }| j ||||||||||
| | |
}|d }| j
| }| j | }t j |d|f }|
d n| j |
| }| s|f|dd z }||f|z S |S t |||j |j S ) Nr% rE r r zwThe `lengths` parameter cannot be used with the Flaubert multiple choice models. Please use the attention mask instead.)rt ru r rV ) r r, r1 loggerwarningrA rR rz rX r r r ) rP rH rI rJ rm rn r5 ro rp rq rr rs rt rY ru num_choices
seq_lengthflat_input_idsflat_attention_maskflat_token_type_idsflat_position_ids
flat_langsflat_inputs_embedsrK rL r2 reshaped_logitsrW s r< rw z TFFlaubertForMultipleChoice.call s 4 $Y/2K#I.q1J$]3A6K#M215JDMDYIJ/?@_cN\Nhbjj "j9I JnrN\Nhbjj "j9I JnrJVJbBJJ|b*5E Fhl? ~t4+?+?+X%'*=ab*AAF)-)9TGf$ Ev E*"-;;*55
r> c | j ry d| _ t | dd Mt j | j j
5 | j j
d d d d t | dd Mt j | j j
5 | j j
d d d d t | dd et j | j j
5 | j j
d d | j j g d d d y y # 1 sw Y xY w# 1 sw Y xY w# 1 sw Y y xY w)NTrA rR rz ) ry rz r, r{ rA r` r| rR rz rM rS r} s r< r| z!TFFlaubertForMultipleChoice.build) s$ ::
4 - 9 t / / 4 45
- &&t,
- 4+T 2 > t 4 4 9 9:
2%%++D1
2 4 - 9 t / / 4 45
M &&dDKK4J4J'KL
M
M :
-
-
2
2
M
Ms$ D<% E?3E<EEE r[ ) rH r\ rI r rJ r rm r rn r r5 r ro r rp r rq r rr r rs r rt r rY r ru r] r z4Union[TFMultipleChoiceModelOutput, Tuple[tf.Tensor]]r )rT rU rV rb rZ rS r r# r r^ r! r r r rw r| r r s @r< rx rx s< $ *!(()ST &/$ .28!I
I
V Mr> rx r )IrW
__future__r r r r= dataclassesr typingr r r r numpynp
tensorflowr, activations_tfr
modeling_tf_outputsr r r
r r modeling_tf_utilsr r r r r r r r r r r r tf_utilsr r r utilsr r r! r" r# r$ configuration_flaubertr&
get_loggerrT r} r r FLAUBERT_START_DOCSTRINGr r= r@ r] r Layerr r rc r r1 r5 rP r` rr rx r[ r> r< r sZ #
! / / /
S R 3
H %4 "( TH V:L 1 L0 h 4-/ 4- 4-pk;5<<#5#5 k;^ ?u|| 1 1 ?> do%,,,, do doP 0 %,,,, 0 f /k / /2 V, 9 V,V,r O2*CEa O2O2d ]M+DF] ]M]M@ RM'@B[ RMRMj ~M";=Q ~M~Mr>