sg'0ddlmZmZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZddlmZmZd d lmZmZmZd ZGd d eZGddeZgdZdZedeGddeeZdZdZedeGddeeZgdZy))OptionalUnionN)BCEWithLogitsLossCrossEntropyLossMSELoss) IJepaConfig)ImageClassifierOutput)PreTrainedModel)add_start_docstrings torch_int) ViTEmbeddingsViTForImageClassificationViTModelzfacebook/ijepa_vith14_1kc eZdZddededdffd Zdejdededejfd Z dd ejd e ejd edejfd Z xZ S)IJepaEmbeddingsconfiguse_mask_tokenreturnNct||||`|jj}t j tjd||j|_ y)N) super__init__ cls_tokenpatch_embeddings num_patchesnn Parametertorchrandn hidden_sizeposition_embeddings)selfrrr __class__s Z/var/www/html/venv/lib/python3.12/site-packages/transformers/models/ijepa/modular_ijepa.pyrzIJepaEmbeddings.__init__sL 0 N++77 #%<< A{FL^L^0_#` embeddingsheightwidthc0|jd}|jjd}tjj s||k(r||k(r |jS|j}|jd}||j z}||j z} t |dz} |jd| | |}|jdddd}tjj||| fdd }|jddddjdd|}|S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rg?rr rbicubicF)sizemode align_corners) shaper#r jit is_tracing patch_sizer reshapepermuter functional interpolateview) r$r(r)r*r num_positionspatch_pos_embeddim new_height new_widthsqrt_num_positionss r&interpolate_pos_encodingz(IJepaEmbeddings.interpolate_pos_encoding!s#!&&q) 0066q9 yy##%+*F6UZ?++ +22r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nr' pixel_valuesbool_masked_posr@cx|j\}}}}|j||}|Z|jd} |jj|| d} |j dj | } |d| z z| | zz}|r||j |||z}n||jz}|j|}|S)N)r@rr,?) r1r mask_tokenexpand unsqueezetype_asr@r#dropout) r$rArBr@ batch_size_r)r*r( seq_length mask_tokensmasks r&forwardzIJepaEmbeddings.forwardHs (4'9'9$ Avu**&>>J\\*- r')F)NF)__name__ __module__ __qualname__rboolrr Tensorintr@r BoolTensorrO __classcell__r%s@r&rrsa{aDaTa%5<<%%UX%]b]i]i%T7;). ll"%"2"23#'   r'rceZdZdZeZdZdZdZddgZ dZ de e je je jfdd fd Zy ) IJepaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ijeparATr IJepaLayermodulerNc t|tjtjfrtjj |j jjtjd|jjj|j j|j _|j%|jjjyyt|tj rJ|jjj|j jj#dyt|t$rtjj |j&jjtjd|jjj|j&j|j&_yy)zInitialize the weightsg)meanstdNrD) isinstancerLinearConv2dinit trunc_normal_weightdatator float32rinitializer_rangedtypebiaszero_ LayerNormfill_rr#)r$r]s r& _init_weightsz"IJepaPreTrainedModel._init_weightspsZ fryy"))4 5"$!6!6 ""%%emm43DKKDaDa"7"b$$% MM {{&   &&('  - KK   " " $ MM   $ $S )  0.0gg.C.C**//225==AKK11/D/b++112  & & +1r')rPrQrR__doc__r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdparrrbrcrnrpr'r&rZrZcs[ L$O&*#*L9N3E"))RYY *L$M3RV3r'rZ)riaG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`IJepaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. z_The bare IJepa Model transformer outputting raw hidden-states without any specific head on top.c.eZdZddededeffd ZxZS) IJepaModelradd_pooling_layerrcVt||||_t|||_y)N)r)rrrrr()r$rr|rr%s r&rzIJepaModel.__init__s%   )&Pr')FF)rPrQrRrrSrrWrXs@r&r{r{s( Q{QtQ]aQQr'r{zjmtzt/ijepa_vith14_1kz Egyptian cata IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states) e.g. for ImageNet. Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ceZdZdeffd Z d deejdeejdeejdeedeedeed eed e e e ffd Z xZ S) IJepaForImageClassificationrcht||t|d|_|j y)NF)r|)rrr{r[ post_init)r$rr%s r&rz$IJepaForImageClassification.__init__s(  %@  r'rA head_masklabelsoutput_attentionsoutput_hidden_statesr@ return_dictrcn||n|jj}|j||||||}|d} |j| j d} d} ||j | j }|jj|jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jjdk(rIt} |jdk(r&| | j|j} n| | |} n|jjdk(r=t} | | j!d |j|j!d } n,|jjdk(rt#} | | |} |s| f|ddz} | | f| zS| St%| | |j&|j( S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrr@rrr)r< regressionsingle_label_classificationmulti_label_classificationr,)losslogits hidden_states attentions)ruse_return_dictr[ classifierr_rhdevice problem_type num_labelsrkr longrUrsqueezerr9rr rr)r$rArrrrr@routputssequence_outputrrloss_fctoutputs r&rOz#IJepaForImageClassification.forwards &1% A 'tn A #+4.A d^A  u++ ,A r'r)rZr{r) typingrrr torch.nnrrrr-transformers.models.ijepa.configuration_ijeparmodeling_outputsr modeling_utilsr utilsr r vit.modeling_vitrrr_CHECKPOINT_FOR_DOCrrZ_EXPECTED_OUTPUT_SHAPEIJEPA_START_DOCSTRINGr{_IMAGE_CLASS_CHECKPOINT_IMAGE_CLASS_EXPECTED_OUTPUTr__all__rxr'r&rs" AAE5-1GmGT3?3D( eQ%xQ Q2- G "68QG G T r'