sgǩdZddlmZddlZddlmZddlmZm Z m Z ddl Z ddl mZddlmZmZmZmZmZmZmZdd lmZmZdd lmZmZmZmZmZd d l m!Z!ejDe#Z$d Z%eGddeZ&GddejNjPZ)GddejNjPZ*GddejNjPZ+GddejNjPZ,GddejNjPZ-GddejNjPZ.GddejNjPZ/GddejNjPZ0Gd d!ejNjPZ1Gd"d#ejNjPZ2Gd$d%ejNjPZ3Gd&d'ejNjPZ4Gd(d)ejNjPZ5Gd*d+ejNjPZ6eGd,d-ejNjPZ7Gd.d/eZ8d0Z9d1Z:ed2e9Gd3d4e8Z;ed5e9Gd6d7e8eZ>S H& !Q!DC , normalizationepsilonrMr%)r+r,rrO ZeroPadding2DrL isinstance collectionsabcIterablerHConv2Dr initializer_rangerlLayerNormalizationrsrIrJ) r.rRrHrIrJrKrLr/r0s r'r,zTFCvtConvEmbeddings.__init__s "6"||11'1B (2:{?W?W(X*_iku^v,,--"'.v/G/GH. #\\<>T^^3zz,z;PTP^P^`h6ij ||L|D r&c|jrtj|d||zgd\}}t|\}}}tj|||||f}|j ||} |j ||} |j||} |jrKtj| fd} tj|| fd} tj|| fd} |j|jz} |j|j| } |j|j| } |j|j| } tj| | d|j z} t#| d}|j%||}tj|| }t|\}}}}tj&|d }tj||||j| zf}|S) NrrrVaxisT) transpose_b)logitsrrr)rr4splitrrrrrconcatrJrrrrrmatmulrrrQr)r.rXrrr< cls_tokenrrrIkeyqueryvaluerattention_scoreattention_probscontextrs r'r?zTFCvtSelfAttention.callgs   &(hh|a%=PRS&T #I|1;<0H- Kzz,z65R^6_` --lX-N11,1R11,1R   IIy%0q9E))Y,15CIIy%0q9E>>T^^3778M8Me8TU55d6I6I#6NO778M8Me8TU))E3DADJJN(bI,,,J))OU3)'21k1,,w\:**Wz;QY@Y&Z[r&cL|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddZtj|jj 5|jj dd|jgdddt|ddZtj|jj 5|jj dd|jgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYxYw#1swYxYw#1swY[xYw#1swYxYw#1swYxYw#1swYyxYw)NTrrrrrr) rZr[r4r\rrMr]rrrrJrrr^s r'r]zTFCvtSelfAttention.builds1 ::  47 > Jt@@EEF >1177= > 45t < Ht>>CCD <//55d; < 47 > Jt@@EEF >1177= > 4+T 2 >t4499: J%%++T4,HI J 4)4 0 <t22778 H##))4t~~*FG H 4+T 2 >t4499: J%%++T4,HI J J ? > > < < > > J J H H J JsH I%I'?I4)J)J')JI$'I14I>J JJ#T)rRrrrbrJrbrnrbrrbrrbrrbrrbrrrrdrr@rrdrrc rXrrrbrrbr<rdrerrA) r r!r"r#r,rr?r]rBrCs@r'rrs$ $GAGAGA GA  GA  GAGAGAGA #GAGA#GAGAR DJr&rc6eZdZdZdfd ZdddZddZxZS) TFCvtSelfOutputzOutput of the Attention layer .c t|di|tjj |t |j d|_tjj||_ ||_ yNdense)rrqrMr%) r+r,rrOrr r|rrPrQrJ)r.rRrJ drop_rater/r0s r'r,zTFCvtSelfOutput.__init__s` "6"\\''@X@X0Y`g(  ||++I6 "r&cP|j|}|j||}|SN)inputs)rr<rrQrs r'r?zTFCvtSelfOutput.calls*zzz6 ||<(|K r&c|jryd|_t|dd[tj|jj 5|jj dd|jgdddyy#1swYyxYwNTrrZr[r4r\rrMr]rJr^s r'r]zTFCvtSelfOutput.buildr ::  4$ ' 3tzz/ ?   $dnn!=> ? ? 4 ? ? )A>>B)rRrrJrbrr@rcrrArfrCs@r'rrs)# ?r&rcreZdZdZ d dfd ZdZdd dZd dZxZS) TFCvtAttentionzDAttention layer. First chunk of the convolutional transformer block.c t|di|t||||||||| | | | d |_t ||| d|_y)N attentionroutputr%)r+r,rrr dense_output)r.rRrrJrnrrrrrrrrrr/r0s r'r,zTFCvtAttention.__init__s]" "6"+         !    ,FIyxXr&ctrA)NotImplementedError)r.headss r' prune_headszTFCvtAttention.prune_headss!!r&cV|j||||}|j||}|SrU)rr)r.rXrrr< self_outputattention_outputs r'r?zTFCvtAttention.calls4nn\658nT ,,[8,Lr&c|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTrr)rZr[r4r\rrMr]rr^s r'r]zTFCvtAttention.builds ::  4d + 7t~~223 +$$T* + 4 . :t00556 .!!''- . . ; + + . .s C %C CC r)rRrrrbrJrbrnrbrrbrrbrrbrrbrrrrdrr@rr@rrdrc)rXrrrbrrbr<rdrA) r r!r"r#r,rr?r]rBrCs@r'rrsN $!Y!Y!Y !Y  !Y  !Y!Y!Y!Y #!Y!Y#!Y!Y!YF" .r&rc4eZdZdZdfd ZddZddZxZS)TFCvtIntermediatezNIntermediate dense layer. Second chunk of the convolutional transformer block.c t|di|tjj t ||zt |jdd|_||_ y)Ngelur)rrq activationrMr%) r+r,rrOrrbr r|rrJ)r.rRrJ mlp_ratior/r0s r'r,zTFCvtIntermediate.__init__sX "6"\\''i)+,.v/G/GH (  #r&c(|j|}|SrA)r)r.rXs r'r?zTFCvtIntermediate.callszz,/ r&c|jryd|_t|dd[tj|jj 5|jj dd|jgdddyy#1swYyxYwrrr^s r'r]zTFCvtIntermediate.buildrr)rRrrJrbrrbrrArfrCs@r'rrsX#?r&rc6eZdZdZdfd ZdddZddZxZS) TFCvtOutputzu Output of the Convolutional Transformer Block (last chunk). It consists of a MLP and a residual connection. c t|di|tjj |t |j d|_tjj||_ ||_ ||_ yr) r+r,rrOrr r|rrPrQrJr)r.rRrJrrr/r0s r'r,zTFCvtOutput.__init__sg "6"\\''@X@X0Y`g(  ||++I6 ""r&cZ|j|}|j||}||z}|Srr)r.rX input_tensorr<s r'r?zTFCvtOutput.calls4zzz6 ||<(|K #l2 r&c @|jryd|_t|ddqtj|jj 5|jj ddt|j|jzgdddyy#1swYyxYwr) rZr[r4r\rrMr]rbrJrr^s r'r]zTFCvtOutput.build"s ::  4$ ' 3tzz/ U   $c$..4>>2Q.R!ST U U 4 U Us ?BB)rRrrJrbrrbrrbrc)rXrrrr<rdrerrArfrCs@r'rrs# Ur&rcteZdZdZ d dfd ZdddZd dZxZS) TFCvtLayera& Convolutional Transformer Block composed by attention layers, normalization and multi-layer perceptrons (mlps). It consists of 3 chunks : an attention layer, an intermediate dense layer and an output layer. This corresponds to the `Block` class in the original implementation. c t|d i|t||||||||| | | | |d|_t ||| d|_t ||| | d|_|dkDr t|dn tjjdd|_ tjjdd |_tjjdd |_||_y) Nrr intermediaterr2 drop_pathrrrlayernorm_beforertlayernorm_afterr%)r+r,rrrrrrr)rrO Activationrr}rr rJ)r.rRrrJrnrrrrrrrrrdrop_path_raterr/r0s r'r,zTFCvtLayer.__init__2s& "6"'         !      .fiQ_`' 9iV^_# .{ ;(( (D  !& ? ?Se ? f$||>>tRc>d"r&c |j|j||||}|j||}||z}|j|}|j |}|j ||}|j||}|SrU)rrrr rr)r.rXrrr<r layer_outputs r'r?zTFCvtLayer.callcs>>$*?*? *MvW\go>p>>*:X>N(,6 ++L9 ((6 ((|D ~~lX~F r&c2|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddZtj|jj 5|jj dd|jgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYxYw#1swYxYw#1swYNxYw#1swYxYw#1swYxYw#1swYyxYw)NTrrrrrr ) rZr[r4r\rrMr]rrrrrJr r^s r'r]zTFCvtLayer.buildts ::  4d + 7t~~223 +$$T* + 4 . :t00556 .!!''- . 4 . :t00556 .!!''- . 4d + 7t~~223 +$$T* + 4+T 2 >t4499: J%%++T4,HI J 4*D 1 =t33889 I$$**D$+GH I I > + + . . . . + + J J I IsH I %I?I'I43)J)J  II$'I14I>J  Jr)rRrrrbrJrbrnrbrrbrrbrrbrrbrrrrdrr@rr@rr@r r@rrdrcrrArfrCs@r'rr+s, $!/#/#/# /#  /#  /#/#/#/# #/#/##/#/#/#/# !/#b"Ir&rc6eZdZdZdfd ZdddZddZxZS) TFCvtStageaK Cvt stage (encoder block). Each stage has 2 parts : - (1) A Convolutional Token Embedding layer - (2) A Convolutional Transformer Block (layer). The classification token is added only in the last stage. Args: config ([`CvtConfig`]): Model configuration class. stage (`int`): Stage number. c t|d i|||_||_|jj|jrQ|j dd|jj dft|jjdd|_t|j|j|j|jdk(r |jn|j |jdz |j|j|j |j|j|j|j|jd|_t!j"d |j$|j|j&|}|Dcgc] }|j)j+"}}t-|j&|jDcgc]X}t/|f|j0|j|j |j|j2|j|j4|j|j6|j|j8|j|j:|j|j<|j|j>|j|j@|j|j|j|jB|j||j|j|jd |d [c}|_"ycc}wcc}w) NrrTzcvt.encoder.stages.2.cls_token)r5 initializer trainablerMr embedding)rHrIrKrJrLrSrMr2zlayers.)rrJrnrrrrrrrrrr rrMr%)#r+r,rRstager add_weightrJr r|rE patch_sizesrI patch_stride patch_paddingrrr4linspacer depthnumpyitemrangerr kernel_qkvrrrrrrrrrO)r.rRrr/drop_path_ratesr;jr0s r'r,zTFCvtStage.__init__s "6"  ;;  ,!__!T[[22267+DKK,I,IJ5 -DN) KK))$**504 a,,VEUEUVZV`V`cdVdEe&&tzz2&&tzz2((4))$**5  ++c6+@+@+Lfll[`Nab5DE1779>>+EE(6<< 34' &%   **4::6 **4::6"--djj94 **4::6 **4::6!,,TZZ8&,&B&B4::&N4$*$>$>tzz$J **4::6 **4::6.tzz:%// ;qc]!   F s %M'EMcDd}|j||}t|\}}}}||z}tj||||f}|jj |j r;tj|j |d}tj||fd}|jD]} | ||||} | }|jj |j rtj|d||zgd\}}tj|||||f}||fS)Nrr)repeatsrrrrV) rrr4rrRrrrepeatrrOr) r.rXr<rrrrrIrlayer layer_outputss r'r?zTFCvtStage.calls ~~lH= 3=\2J/ FE<un zz,z;P\6]^ ;;  , $..*1MI99i%>QGL[[ )E!,QM(L ) ;;  ,&(hh|a%=PRS&T #I|zz,z65R^6_` Y&&r&c|jryd|_t|ddMtj|jj 5|jj ddddt|ddK|jD];}tj|j 5|j dddd=yy#1swYbxYw#1swYUxYw)NTrrO)rZr[r4r\rrMr]rOr.r_r%s r'r]zTFCvtStage.builds ::  4d + 7t~~223 +$$T* + 44 ( 4 &]]5::.&KK%&& & 5 + +&&s C*CCC )rRrrrbrc)rXrr<rdrArfrCs@r'rrs - ^'0 &r&rcReZdZdZeZdfd Z d ddZddZxZ S) TFCvtEncoderz Convolutional Vision Transformer encoder. CVT has 3 stages of encoder blocks with their respective number of layers (depth) being 1, 2 and 10. Args: config ([`CvtConfig`]): Model configuration class. c t|di|||_tt |j Dcgc]}t ||d|c}|_ycc}w)Nzstages.rr%)r+r,rRrr6rrstages)r.rRr/ stage_idxr0s r'r,zTFCvtEncoder.__init__sX "6" W\]`agamam]nWo JSJvy /D E   sAc |rdnd}|}tj|d}d}t|jD]\}} | ||\}}|s||fz}tj|d}|r.t |D cgc]} tj| dc} }|st d|||fDSt |||Scc} w)Nr%)rrrrrrV)rrrrc3&K|] }|| ywrAr%).0vs r' z$TFCvtEncoder.call..sbqTUTabsrrr)r4r enumerater,tupler) r.rWoutput_hidden_states return_dictr<all_hidden_statesrXrr stage_modulehss r'r?zTFCvtEncoder.calls#7BD# ||L|D  !*4;;!7 H A &2<(&S #L)#$5$G! H ||L|D  %Uf&grr||B\'J&g h b\9>O$Pbb b,*%+  'hs7Cc|jryd|_t|ddK|jD];}tj|j 5|j dddd=yy#1swYIxYw)NTr,)rZr[r,r4r\rMr]r(s r'r]zTFCvtEncoder.buildsp ::  44 ( 4 &]]5::.&KK%&& & 5&&s A..A7 rRr)FTF) rWr r6Optional[bool]r7r=r<r=re6Union[TFBaseModelOutputWithCLSToken, Tuple[tf.Tensor]]rA) r r!r"r#r config_classr,r?r]rBrCs@r'r*r*s[L 05&*#(  & - $  !  @  B&r&r*c^eZdZdZeZdfd Ze d ddZddZ xZ S) TFCvtMainLayerzConstruct the Cvt model.c Vt|di|||_t|d|_y)Nencoderrr%)r+r,rRr*rC)r.rRr/r0s r'r,zTFCvtMainLayer.__init__/s( "6" #F; r&c| td|j||||}|d}|s |f|ddzSt||j|jS)N You have to specify pixel_valuesr6r7r<rrr3) ValueErrorrCrrr)r.rWr6r7r<encoder_outputssequence_outputs r'r?zTFCvtMainLayer.call4s{  ?@ @,, !5# ' *!,#%(;; ;,-+;;)77  r&c|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYw)NTrC)rZr[r4r\rCrMr]r^s r'r]zTFCvtMainLayer.buildQsi ::  4D ) 5t||001 ) ""4( ) ) 6 ) )rar<NNNF) rWzTFModelInputType | Noner6r=r7r=r<r=rer>rA) r r!r"r#rr?r,rr?r]rBrCs@r'rArA)sh"L< 15/3&*#(  - - $  !  @   8)r&rAceZdZdZeZdZdZy)TFCvtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. cvtrWN)r r!r"r#rr?base_model_prefixmain_input_namer%r&r'rMrMZs L$Or&rMa This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using [`keras.Model.fit`] method which currently requires having all the tensors in the first argument of the model call function: `model(inputs)`. Args: config ([`CvtConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. al Args: pixel_values (`np.ndarray`, `tf.Tensor`, `List[tf.Tensor]` ``Dict[str, tf.Tensor]` or `Dict[str, np.ndarray]` and each example must have the shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CvtImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True. training (`bool`, *optional*, defaults to `False``): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). z]The bare Cvt Model transformer outputting raw hidden-states without any specific head on top.ceZdZdfd Zeeeeee  d ddZ ddZ xZ S) TFCvtModelcPt||g|i|t|d|_y)NrNr)r+r,rArNr.rRrr/r0s r'r,zTFCvtModel.__init__s( 3&3F3!&u5r& output_typer?c| td|j||||}|s |df|ddzSt|j|j|j S)a Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFCvtModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13") >>> model = TFCvtModel.from_pretrained("microsoft/cvt-13") >>> inputs = image_processor(images=image, return_tensors="tf") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NrE)rWr6r7r<rrr3)rGrNrrrr)r.rWr6r7r<outputss r'r?zTFCvtModel.callsy>  ?@ @((%!5#  AJ=712;. .,%77#33!//  r&c|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYw)NTrN)rZr[r4r\rNrMr]r^s r'r]zTFCvtModel.buildse ::  4 % 1txx}}- %t$ % % 2 % %rar<rK) rWtf.Tensor | Noner6r=r7r=r<r=rer>rA) r r!r"r,rrTFCVT_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCr?r]rBrCs@r'rRrRs 6 *+AB+HWfg*./3&*#( - &- -- $ - ! - @ - hC- ^%r&rRz Cvt Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ceZdZdfd Zeeeeee  d ddZ ddZ xZ S) TFCvtForImageClassificationcXt||g|i||j|_t|d|_t j jdd|_t j j|jt|jddd |_ ||_ y) NrNrrr layernormrtTr classifierr)r+r, num_labelsrArNrrOr}r`rr r|rarRrTs r'r,z$TFCvtForImageClassification.__init__s 3&3F3 ++!&u588K8X ,,,,##.v/G/GH$ -  r&rUc|j||||}|d}|d}|jjdr|j|}nUt |\} } } } t j || | | | zf}t j|d}|j|}t j|d} |j| }|d n|j|| }|s|f|d d z}||f|zS|St|||j S) a+ labels (`tf.Tensor` or `np.ndarray` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFCvtForImageClassification >>> import tensorflow as tf >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13") >>> model = TFCvtForImageClassification.from_pretrained("microsoft/cvt-13") >>> inputs = image_processor(images=image, return_tensors="tf") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = tf.math.argmax(logits, axis=-1)[0] >>> print("Predicted class:", model.config.id2label[int(predicted_class_idx)]) ```rFrrrr)rrrrrN)labelsrr)lossrr) rNrRrr`rr4rr reduce_meanrahf_compute_lossr r)r.rWrdr6r7r<rXrIrrrIrrsequence_output_meanrrers r'r?z TFCvtForImageClassification.calls(R(( !5#  "!*AJ ;;  $"nnY7O7A6Q 3J fe jj\[adi[i@jkO ll?KO"nn_=O!~~oAF!56~t4+?+?vV\+?+]Y,F)-)9TGf$ Ev E54^e^s^sttr&c*|jryd|_t|ddMtj|jj 5|jj ddddt|ddgtj|jj 5|jj dd|jjdgdddt|ddt|jdrhtj|jj 5|jj dd|jjdgdddyyy#1swY xYw#1swYxYw#1swYyxYw)NTrNr`rrarM) rZr[r4r\rNrMr]r`rRrJhasattrrar^s r'r]z!TFCvtForImageClassification.build8sE ::  4 % 1txx}}- %t$ % 4d + 7t~~223 N$$dD$++2G2G2K%LM N 4t , 8t/]]4??#7#78SOO))4t{{7L7LR7P*QRSS0 9  % % N NSSs$ E0%6E=/6F 0E:=F Fr<)NNNNF) rWrZrdrZr6r=r7r=r<r=rez?Union[TFImageClassifierOutputWithNoAttention, Tuple[tf.Tensor]]rA) r r!r"r,rrr[rr r\r?r]rBrCs@r'r^r^s$*+AB+Q`op*.#'/3&*#( @u&@u!@u- @u $ @u ! @u I@uqC@uD Sr&r^)=r# __future__rcollections.abcrx dataclassesrtypingrrr tensorflowr4modeling_tf_outputsr modeling_tf_utilsr r r r rrrtf_utilsrrutilsrrrrrconfiguration_cvtr get_loggerr loggerr\rrOLayerr)rErNrrrrrrrrrrr*rArMTFCVT_START_DOCSTRINGr[rRr^r%r&r'rysd"!))I3)   H % 7K7 7(/ELL&&/(%8ell((%8P7G%,,,,7Gt"Mu||'9'9"MJ););85<<#5#58DMJ++MJ`?ell((?27.U\\''7.t? **?4U%,,$$U:^I##^IB]&##]&@:&5<<%%:&z-)U\\''-)-)`%,%8&c>%%>% >%B eS"68TeSeSr&