sgt}XddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m Z mZmZddlmZddlmZmZmZddlmZddlmZmZdd lmZmZmZmZm Z d d l!m"Z"ejFe$Z%d Z&d Z'Gdde jPZ)Gdde jPZ*GddeZ+Gdde jPZ,Gdde,Z-Gdde jPZ.Gdde jPZ/Gdde/Z0Gdde jPZ1Gd d!e jPZ2e/e0d"Z3Gd#d$e jPZ4Gd%d&e jPZ5Gd'd(e jPZ6d)Z7gd*Z8d+Z9ed,e9Gd-d.e+Z:d/Z;d0Z<ed1e9Gd2d3e+Z=gd4Z>y)5N)DictListOptionalSetTupleUnion)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging torch_int) IJepaConfigzfacebook/ijepa_vith14_1krc`eZdZdZfdZddej dedej fdZxZ S)IJepaPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)super__init__ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesnnConv2d projection)selfconfigr"r#r$r%r* __class__s [/var/www/html/venv/lib/python3.12/site-packages/transformers/models/ijepa/modeling_ijepa.pyr!zIJepaPatchEmbeddings.__init__-s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hi pixel_valuesinterpolate_pos_encodingreturnc |j\}}}}||jk7rtd|jd|d|sV||jdk7s||jdk7r2td|d|d|jdd|jdd |j |j d j dd }|S) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).)shaper$ ValueErrorr"r-flatten transpose)r.r3r4 batch_sizer$heightwidth embeddingss r1forwardzIJepaPatchEmbeddings.forward<s2>2D2D/ L&% 4,, ,!../yaI (++u8J/J (%9+,Adooa.@-AE__\2::1=GG1M r2F) __name__ __module__ __qualname____doc__r!torchTensorboolrB __classcell__r0s@r1rr&s3 jELLD]b]i]ir2rc eZdZdZddededdffd Zdejde d e dejfd Z dd ejd e ejd edejfdZ xZS)IJepaEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. r/use_mask_tokenr5Nct||r4tjt j dd|j nd|_t||_ |jj}tjt jd||j |_ tj|j|_|j |_||_y)Nr)r r!r+ ParameterrHzerosr% mask_tokenrpatch_embeddingsr*randnposition_embeddingsDropouthidden_dropout_probdropoutr#r/)r.r/rOr*r0s r1r!zIJepaEmbeddings.__init__Rs Q_",,u{{1a9K9K'LMei 4V <++77 #%<< A{FL^L^0_#` zz&"<"<=  ++ r2rAr?r@c0|jd}|jjd}tjj s||k(r||k(r |jS|j}|jd}||j z}||j z} t |dz} |jd| | |}|jdddd}tjj||| fdd }|jddddjdd|}|S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rg?rr r9bicubicF)sizemode align_corners) r:rVrHjit is_tracingr#rreshapepermuter+ functional interpolateview) r.rAr?r@r* num_positionspatch_pos_embeddim new_height new_widthsqrt_num_positionss r1r4z(IJepaEmbeddings.interpolate_pos_encoding\s#!&&q) 0066q9 yy##%+*F6UZ?++ +22r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nr2r3bool_masked_posr4cx|j\}}}}|j||}|Z|jd} |jj|| d} |j dj | } |d| z z| | zz}|r||j |||z}n||jz}|j|}|S)N)r4rr[?) r:rTrSexpand unsqueezetype_asr4rVrY) r.r3rmr4r>_r?r@rA seq_length mask_tokensmasks r1rBzIJepaEmbeddings.forwards (4'9'9$ Avu**&>>J\\*- r2rCNF)rDrErFrGrrJr!rHrIintr4r BoolTensorrBrKrLs@r1rNrNMs{DT%5<<%%UX%]b]i]i%T7;). ll"%"2"23#'   r2rNceZdZdZeZdZdZdZddgZ dZ de e je je jfdd fd Zy ) IJepaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ijepar3TrN IJepaLayermoduler5Nc t|tjtjfrtjj |j jjtjd|jjj|j j|j _|j%|jjjyyt|tj rJ|jjj|j jj#dyt|t$rtjj |j&jjtjd|jjj|j&j|j&_yy)zInitialize the weights)meanstdNro)r&r+Linearr,init trunc_normal_weightdatatorHfloat32r/initializer_rangedtypebiaszero_ LayerNormfill_rNrV)r.r~s r1 _init_weightsz"IJepaPreTrainedModel._init_weightssZ fryy"))4 5"$!6!6 ""%%emm43DKKDaDa"7"b$$% MM {{&   &&('  - KK   " " $ MM   $ $S )  0.0gg.C.C**//225==AKK11/D/b++112  & & +1r2)rDrErFrGr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdparr+rr,rrr2r1r{r{s[ L$O&*#*L9N3E"))RYY *L$M3RV3r2r{c eZdZdeddffd Zdej dej fdZ d deej de de e ej ej fe ej ffd Z xZ S) IJepaSelfAttentionr/r5Nct||j|jzdk7r3t |ds't d|jfd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads r7)r)r r!r%num_attention_headshasattrr;rxattention_head_size all_head_sizer+rqkv_biasquerykeyvaluerWattention_probs_dropout_probrYr.r/r0s r1r!zIJepaSelfAttention.__init__s1     : : :a ?PVXhHi"6#5#5#6"78334A7  $*#=#= #&v'9'9FS"S%**+BC 6G=/2O\M]r2rw)rDrErFrr!rHrIrrrJrrrBrKrLs@r1rrsG{GtG$%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!r2rc eZdZdeddffd Z d dej deejde de e ejejfe ejfffd Z xZ S) IJepaSdpaSelfAttentionr/r5NcFt|||j|_yN)r r!rrs r1r!zIJepaSdpaSelfAttention.__init__s  ,2,O,O)r2rrrc ^|s|'tjdt | |||S|j |}|j |j |}|j |j|}|j |}tjjj|||||jr |jnddd}|jdddd j}|j!dd |j"fz} |j%| }|dfS) Na`IJepaSdpaAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrrrF) is_causalscalerr9rr r)logger warning_oncer rBrrrrrHr+rdscaled_dot_product_attentiontrainingrrcrr]rrf) r.rrrrrrrrrr0s r1rBzIJepaSdpaSelfAttention.forwards:  5   w  7?+#"3#  !JJ}5--dhh}.EF // =0IJ //0AB ++HH    15D - -CI &--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**+BC d""r2rw)rDrErFrr!rH FloatTensorrrIrJrrrBrKrLs@r1rrsP{PtP-1"' '#(('#ELL)'# '# uU\\5<</0% 2EE F '#'#r2rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) IJepaSelfOutputz The residual connection is defined in IJepaLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r/r5Nct|tj|j|j|_tj |j|_yr) r r!r+rr%denserWrXrYrs r1r!zIJepaSelfOutput.__init__1sB YYv1163E3EF zz&"<"<= r2r input_tensorcJ|j|}|j|}|SrrrYr.rrs r1rBzIJepaSelfOutput.forward6s$ =1  ]3 r2) rDrErFrGrr!rHrIrBrKrLs@r1rr+sD >{>t> U\\RWR^R^r2rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) IJepaAttentionr/r5Nct|t||_t ||_t |_yr)r r!r attentionroutputset pruned_headsrs r1r!zIJepaAttention.__init__>s0 +F3%f- Er2headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrrrrrrrrrrrunion)r.rindexs r1 prune_headszIJepaAttention.prune_headsDs u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r2rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r.rrr self_outputsattention_outputrs r1rBzIJepaAttention.forwardVsE ~~mY@QR ;;|A F#% QR(88r2rw)rDrErFrr!rrxrrHrIrrJrrrBrKrLs@r1rr=s"{"t" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r2rc(eZdZdeddffd ZxZS)IJepaSdpaAttentionr/r5NcDt||t||_yr)r r!rrrs r1r!zIJepaSdpaAttention.__init__es  /7r2)rDrErFrr!rKrLs@r1rrds8{8t88r2rc`eZdZdeddffd Zdej dej fdZxZS)IJepaIntermediater/r5Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r r!r+rr%intermediate_sizerr& hidden_actstrr intermediate_act_fnrs r1r!zIJepaIntermediate.__init__ks] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r2rcJ|j|}|j|}|Sr)rr)r.rs r1rBzIJepaIntermediate.forwardss& =1 00? r2 rDrErFrr!rHrIrBrKrLs@r1rrjs19{9t9U\\ellr2rcxeZdZdeddffd Zdej dej dej fdZxZS) IJepaOutputr/r5Nct|tj|j|j |_tj|j|_ yr) r r!r+rrr%rrWrXrYrs r1r!zIJepaOutput.__init__{sB YYv779K9KL zz&"<"<= r2rrcT|j|}|j|}||z}|Srrrs r1rBzIJepaOutput.forwards. =1  ]3 % 4 r2rrLs@r1rrzs?>{>t> U\\RWR^R^r2r)eagersdpac eZdZdZdeddffd Z d dejdeejde de e ejejfe ejffd Z xZ S) r}z?This corresponds to the Block class in the timm implementation.r/r5Nct||j|_d|_t |j ||_t||_t||_ tj|j|j|_tj|j|j|_y)Nreps)r r!chunk_size_feed_forward seq_len_dimIJEPA_ATTENTION_CLASSES_attn_implementationrr intermediaterrr+rr%layer_norm_epslayernorm_beforelayernorm_afterrs r1r!zIJepaLayer.__init__s '-'E'E$01L1LMfU-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr2rrrc|j|j|||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|S)N)rrr)rrrrr)r.rrrself_attention_outputsrr layer_outputs r1rBzIJepaLayer.forwards "&  ! !- 0 /"0"  2!4(,)=8 ++M: ((6 {{<? /G+r2rw)rDrErFrGrr!rHrIrrJrrrBrKrLs@r1r}r}sI[{[t[-1"' ||ELL)  uU\\5<</0% 2EE F r2r}ceZdZdeddffd Z d dej deej deded ede e e ff d Z xZ S) IJepaEncoderr/r5Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wrw) r r!r/r+ ModuleListrangenum_hidden_layersr}layergradient_checkpointing)r.r/rsr0s r1r!zIJepaEncoder.__init__sN  ]]fF^F^@_#`1Jv$6#`a &+#$asA#rrroutput_hidden_states return_dictct|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr+|jr|j | j || |} n | || |} | d}|s`|| dfz}j|r||fz}|st d|||fDSt|||S)Nrrrc3&K|] }|| ywrr).0vs r1 z'IJepaEncoder.forward..smq_`_lms)last_hidden_stater attentions) enumerater r r_gradient_checkpointing_func__call__tupler) r.rrrr r all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r1rBzIJepaEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]OM^ _ )!,M &9]1=M