sgز dZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZmZmZd d lmZej<eZ d Z!dZ"gdZ#eGddeZ$eGddeZ%Gdde jLZ'Gdde jLZ(Gdde jLZ)Gdde jLZ*d?de jVde,de-de jVfd Z.Gd!d"e jLZ/Gd#d$e jLZ0Gd%d&e jLZ1Gd'd(e jLZ2Gd)d*e jLZ3Gd+d,e jLZ4Gd-d.eZ5d/Z6d0Z7ed1e6Gd2d3e5Z8d4e jVd5e9de jVfd6Z:d4e jVd7e9d8e9de jVfd9Z;Gd:d;e jLZ<ede5Z=y)@zPyTorch SegGpt model.N) dataclass)DictListOptionalTupleUnion)nn) functional)ACT2FN)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) SegGptConfigrzBAAI/seggpt-vit-large)r iiceZdZUdZej ed<dZee ej ed<dZ ee ej ed<dZ ee ej ed<y)SegGptEncoderOutputa Output type of [`SegGptEncoderOutput`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, patch_height, patch_width, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`Tuple[torch.FloatTensor]`, `optional`, returned when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, patch_height, patch_width, hidden_size)`. attentions (`Tuple[torch.FloatTensor]`, `optional`, returned when `config.output_attentions=True`): Tuple of *torch.FloatTensor* (one for each layer) of shape `(batch_size, num_heads, seq_len, seq_len)`. intermediate_hidden_states (`Tuple[torch.FloatTensor]`, *optional*, returned when `config.intermediate_hidden_state_indices` is set): Tuple of `torch.FloatTensor` of shape `(batch_size, patch_height, patch_width, hidden_size)`. Each element in the Tuple corresponds to the output of the layer specified in `config.intermediate_hidden_state_indices`. Additionaly, each feature passes through a LayerNorm. last_hidden_stateN hidden_states attentionsintermediate_hidden_states) __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrrrr]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/seggpt/modeling_seggpt.pyrr1sd"(((82D2D/ L&% 4,, ,w  T__Q' '5DOOA4F+F$VHAeW4KDOO\]L^K__`aeapapqras`ttvw __\2::1aAF r$)rrrrr0rK __classcell__r>s@r%r+r+cs j r$r+c eZdZdZdeddffd ZdededejfdZ dd ejd ejd e ejd e e dejf d Z xZS)SegGptEmbeddingszX Construct the embeddings from patch, position embeddings for input and prompt. r=returnNc~t|tjt j ddd|j |_tjt j ddd|j |_tjt j ddd|j |_ tjt j ddd|j |_ tjt j ddd|j |_ t||_ |j|jzdzdz}tjt j d||j |_tj$|j&|_y)NrrB)r/r0r Parameterr zerosr4 mask_tokensegment_token_inputsegment_token_prompttype_token_semantictype_token_instancer+patch_embeddingspretrain_image_sizer2randnposition_embeddingsDropouthidden_dropout_probdropout)r<r= num_positionsr>s r%r0zSegGptEmbeddings.__init__s3 ,,u{{1aF@SUWX``abdeghjkle_# O#**1aA6 6"**1feR@ @r$rFprompt_pixel_valuesbool_masked_posembedding_typecR|j|}|j|}|j\}}} } |jj||| d} |j dj | j d|| d} |d| z z| | zz}||nd}|j|| } ||jz}||jz}|| z}|| z}|dk(r |j}n |dk(r |j}ntd|||z}||z}tj||fd}|S)NrcrinstancesemanticzBEmbedding type should be either 'semantic' or 'instance', but got rdim)rYrCrTexpand unsqueezetype_asrlrorUrVrWrXrDr cat)r<rFrprqrrinput_embeddingsprompt_embeddingsrG patch_height patch_width_rTw pos_embedtype_embeddingrJs r%rKzSegGptEmbeddings.forwardsh 00> 112EF3C3I3I0 L+q__++J kSUV  % %b ) 1 1* = E Eb,Xcef g-Q7*q.H+9+E:11, L ,d.F.FF-0I0II,i7- 9 Z '!55N z )!55Nabpaqrs s+n<->YY 02CD!L r$)NN)rrrrrr0intr Tensorror BoolTensorstrrKrLrMs@r%rOrOs>|>> AsA3A5<<A,7;(, +ll+#\\+"%"2"23 + ! +  +r$rOc8eZdZdZfdZdededejdejfdZdejd ejd ejd ejde eefde eefdejfd Z dd ejdejfdZ xZ S)SegGptAttentionz=Multi-head Attention block with relative position embeddings.ct||j|j}}t |t j jr|n||f}t |t j jr|n||f}|d|jz|d|jzf}|j|jz}|j|_ |dz|_ tj|j|jdz|j|_tj|j|j|_|j |_|j r|| t#dtj$t'j(d|dzdz ||_tj$t'j(d|dzdz ||_yy)Nrrgr biaszBInput size must be provided if using relative positional encoding.rB)r/r0r1r2r5r6r7r8r4num_attention_headsscaler Linearqkv_biasqkvproj use_relative_position_embeddingsrDrRr rS rel_pos_h rel_pos_w)r<r=r1r2 input_sizehead_dimr>s r%r0zSegGptAttention.__init__s !'!2!2F4E4EJ #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq  mv'8'88*Q-6K\K\:\] %%)C)CC#)#=#= t^ 99V//1C1Ca1Gfoo^IIf00&2D2DE 060W0W-  0 0! !eff \\%++a*Q-6G!6KX*VWDN\\%++a*Q-6G!6KX*VWDN 1r$q_sizek_sizerel_posrPctdt||zdz }tj|j d|j ddj ddd|d}|j d|j dd}tj|dddft||z dz}tj|dddft||z dz}||z |dz t||z dzz}||jS) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of the query. k_size (int): size of key k. rel_pos (`torch.Tensor`): relative position embeddings (L, channel). Returns: Extracted positional embeddings according to relative positions. rBrrrclinear)rerfN?) rmaxrjrkrlrCrEr arangelong) r<rrr max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r% get_rel_poszSegGptAttention.get_rel_poss 1s6622Q67 -- OOAw}}Q/ 4 < bhwkzbhwc,wkc->bhwkN)rrCrlr einsum)r<rrrrrr query_height query_width key_height key_widthrelative_position_heightrelative_position_widthrGrrwreshaped_queryrel_hrel_ws r%add_decomposed_rel_posz&SegGptAttention.add_decomposed_rel_poss>%+! k & I#'#3#3L*i#X "&"2"2; 9"U"[[ Asz<cR -~?WX -~?VW||J k:yYeAq!Q,--aAtQ6F0GG||J {(BJQZDZ[ r$rc |j\}}}}|j|j|||zd|jdj ddddd}|jd||jz||zdj d\}} } ||j z| jddz} |jr.|j| ||j|j||f||f} tjjj| tj dj#|j$} |rE| j'||j||zd} | j'||jz||zd} nd} | | zj||j||d} | j dddddj|||d} |j)| } | | fS) Nr rcrBrr)dtyperw)rCrrlrrEunbindr transposerrrrr r r softmaxfloat32torviewr)r<routput_attentionsrGrHrIrrrkeyvalue attn_weightsattn_weights_reshaped attn_outputs r%rKzSegGptAttention.forwardCs'4':':$ FE1 HH] # WZ%D4L4Lb Q WQ1a #  KK:8P8P+PRX[`R`bdellmnosE *cmmB.CC  0 066eT^^T^^fe_W]_dVeLxx**22s r%r0zSegGptMlp.__init__ksX IIf00&..A IIfnnf.@.@A &++,r$rrPcl|j|}|j|}|j|}|Sr)rrrr<rs r%rKzSegGptMlp.forwardqs2 -0 /  -0 r$)rrrr0r rrKrLrMs@r%rrjs#- U\\ellr$rinput drop_probtrainingrPc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rr)r)rdevice)rCndimr randrrfloor_div)rrr keep_probrC random_tensoroutputs r% drop_pathrysCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr$cxeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) SegGptDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrPc0t|||_yr)r/r0r)r<rr>s r%r0zSegGptDropPath.__init__s "r$rcDt||j|jSr)rrrrs r%rKzSegGptDropPath.forwards FFr$c8dj|jS)Nzp={})formatrr<s r% extra_reprzSegGptDropPath.extra_reprs}}T^^,,r$r) rrrrrfloatr0r rrKrrrLrMs@r%rrsGb#(5/#T#GU\\GellG-C-r$rceZdZdededdffd Z d dejdede d e de e ejejfe ejff d Z xZ S) SegGptLayerr=drop_path_raterPNctt|t||_t ||_|dkDr t |ntj|_ tj|j|j|_ tj|j|j|_y)Nreps)r/r0r attentionrmlprr Identityr LayerNormr4layer_norm_epslayernorm_beforelayernorm_after)r<r=rr>s r%r0zSegGptLayer.__init__s (0V$;IC;O7UWU`U`Ub " V-?-?VEZEZ [!||F,>,>FDYDYZr$r ensemble_condfeature_ensemblerc|j|j||}|d}|dd}|r|jddz|k\r|j|jddzd\}} |dk(ra|jddz} | j d| d} | j ddj | } | j|j} n"| j ddj | } tj|| gd}|j||z}|} |j|}|j|}| |j|z}|f|z}|S) N)rrrrBrvrcT)rwkeepdim) rrrCsplitrlmean expand_asr r{rrr) r<rrrrself_attention_outputsattention_outputoutputspromptinputs num_promptsresiduals r%rKzSegGptLayer.forwardsz"&  ! !- 0/"0" 2!4(,  0 6 6q 9Q >- O-334D4J4J14MQR4RXY3ZNFF!.44Q71< ;;D9CCFK'6D9CCFK$yy&&)9qA '78=H  ,,]; /  4>>-#@@  "W,r$)FF)rrrrrr0r rrboolrrrKrLrMs@r%rrs[|[U[t["'"' #||## #  # uU\\5<</0% 2EE F #r$rcpeZdZdeddffd Z d dej dededed edee e ff d Z xZ S) SegGptEncoderr=rPNc t|||_tjd|j |j Dcgc]}|j}}tjt|j Dcgc]}t|||c}|_ tj|j|j|_d|_ycc}wcc}w)NrrF)r/r0r=r linspacernum_hidden_layersitemr ModuleListrangerlayersrr4r layernormgradient_checkpointing)r<r=xdprir>s r%r0zSegGptEncoder.__init__s  !&63H3H&JbJb!cdAqvvxddmm%PVPhPhJi$jQ[Q%@$jk f&8&8f>S>ST&+#e$js CC"rrroutput_hidden_states return_dictc|rdnd}|rdnd}g}t|jD]\} } |r||fz}|jj| kDrdnd} |jr,|j r |j | j|| ||} n | || ||} | d}| |jjk(r.|d|jddz||jddzdzdz}| |jjvr |j|j||s|| dfz}|r||fz}|std||||fDSt||||S)Nr#rBrrrbc3$K|]}|| ywrr#).0vs r% z(SegGptEncoder.forward..s=s)rrrr) enumerater r= merge_indexrr_gradient_checkpointing_func__call__rC!intermediate_hidden_state_indicesappendr tupler) r<rrrrrall_hidden_statesall_self_attentionsrr layer_moduler layer_outputss r%rKzSegGptEncoder.forwards#7BD$5b4%'"(5 POA|#$58H$H!"&!8!81! 1]4D D '):s r%r0zSegGptLayerNorm.__init__s ll5::.>#?@ LL-=!>? &   #F F%(A$BRBRAS&TU U!1 3r$rrPcd|jdk(rWtjjj ||j |j |j|j}|S|jdk(r|j}|j}|jdd}||z jdjdd}||z tj||jzz }|j|}|j ddddf|z|jddddfz}|S)Nr'r(rT)rrBr)r+r r r layer_normr-r*rrrrrpowsqrtr)r<r input_dtypeuss r%rKzSegGptLayerNorm.forward!s     .##..q$2G2GVZV_V_aeaiaijA  !1 1''K Aq$'AQ A##At#4AQ%**Q\22A;'A AtTM*Q.1dD=1IIAr$)gư>r') rrrrr0r rrKrLrMs@r%r%r%s( 4  %,, r$r%c>eZdZfdZdej fdZxZS)SegGptDecoderHeadcTt|tj|j|jdd|_t |j|jd|_t|j|_ tj|jddd|_ y)Nr r)r-paddingr()r-rr+T)r-r) r/r0r r:decoder_hidden_sizeconvr%rr r ract_fctheadrs r%r0zSegGptDecoderHead.__init__0s II  & &  & &   )#77V=R=R`p f//0 IIf88!QUV r$rc|j|}|j|}|j|}|j|}|Sr)r;r r<r=rs r%rKzSegGptDecoderHead.forward>s@ -0 }5  ]3  -0 r$)rrrr0r r!rKrLrMs@r%r7r7/s WU%6%6r$r7cveZdZfdZdej dej fdZdej fdZxZS) SegGptDecodercBt|tj|jt |j z|jdz|jzd|_ t||_ |j|_|j|_||_ y)NrBTr) r/r0r rr4lenrr2r: decoder_embedr7 decoder_predr=rs r%r0zSegGptDecoder.__init__Hs YY   V%M%M!N N   q 6#=#= =  .f5 ++#)#=#=  r$rrPc|j\}}}}|j||||j|j|j}|j dddddd}|j|d||jz||jzf}|S) Nrrr rBrrcrC)rCrlr2r:rE)r<rrGr~rrs r%_reshape_hidden_statesz$SegGptDecoder._reshape_hidden_statesTs3@3F3F0 L+q%--  k4??DOOUYUmUm &--aAq!Q? %--r<$//#A;QUQ`Q`C`a. r$cl|j|}|j|}|j|}|Sr)rCrHrDrs r%rKzSegGptDecoder.forward`s8**=9 33MB ))-8 r$) rrrr0r r!rHrKrLrMs@r%r@r@Gs9  E4E4E %J[J[ U%6%6r$r@ceZdZdZeZdZdZdZddgZ de e je je jfdd fd Zy ) SegGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. modelrFTrOrmodulerPNc2|jj}t|tjtj frtj j|jjjtjd|j|jj|j_ |j%|jjjyyt|tj rJ|jjj|jjj#dyt|t$r tj j|j&jjtjd|j|j&j|j&_ tj j|j(jjtjd|j|j(j|j(_ yt|t*rtj j|j,jjtjd|j|j,j|j,_ tjj j/|j0|tjj j/|j2|tjj j/|j4|tjj j/|j6|tjj j/|j8|yy)zInitialize the weightsr)rstdNr)rO)r=initializer_ranger5r rr:init trunc_normal_r*datarr rrrzero_rfill_rrrrOr\normal_rTrUrVrWrX)r<rMrOs r% _init_weightsz#SegGptPreTrainedModel._init_weightstskk++ fryy"))4 5"$!6!6v}}7I7I7L7LU]]7[bekn!6!o!r!r ##"FMM {{&   &&('  - KK   " " $ MM   $ $S )  0$&GG$9$9  %%((7%:%b!!''(    ! %'GG$9$9  %%((7%:%b!!''(    !  0 1.0gg.C.C**//225==A/D/b++112  & & + HHMM ! !&"3"3 ! = HHMM ! !&"<"<# ! F HHMM ! !&"="=3 ! G HHMM ! !&"<"<# ! F HHMM ! !&"<"<# ! F2r$)rrrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrr rr:rrWr#r$r%rKrKhsY  L$O&*#+];&GE"))RYY *L$M&GRV&Gr$rKaH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SegGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a  Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt pixel values. Prompt pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt mask. Prompt mask can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). feature_ensemble (`bool`, *optional*): Boolean indicating whether to use feature ensemble or not. If `True`, the model will use feature ensemble if we have at least two prompts. If `False`, the model will not use feature ensemble. This argument should be considered when doing few-shot inference on an input image i.e. more than one prompt for the same image. embedding_type (`str`, *optional*): Embedding type. Indicates whether the prompt is a semantic or instance embedding. Can be either instance or semantic. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare SegGpt Model transformer outputting raw hidden-states without any specific head on top.c\eZdZdeffd ZdefdZdeee efddfdZ e e e ee dd ej"d ej"d ej"d eej&d eedeedeej,deedeedeedeeeffdZxZS) SegGptModelr=ct||||_t||_t ||_|jyr)r/r0r=rOrJrencoder post_initrs r%r0zSegGptModel.__init__s;   *62$V,  r$rPc.|jjSr)rJrYrs r%get_input_embeddingsz SegGptModel.get_input_embeddingss///r$heads_to_pruneNc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr`layerr prune_heads)r<rdrgheadss r% _prune_headszSegGptModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr$ output_typerXrFrp prompt_masksrqrrrlabelsrrrc n||n|jj}| | n|jj} | | n|jj} ||nd}|jj j jj} |j| }|j| }tj||fd}|tj||fdntj||fd}||tjd|w|jj j} tj| tj j|j"}d|| dzd|j%d}|j |||| } |j'| ||| | }|S) a labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Returns: Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptModel >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptModel.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> list(outputs.last_hidden_state.shape) [1, 56, 28, 1024] ``` NFrBrvzLabels were provided, but bool_masked_pos were not. It will be set to default value. If you're training the model, make sure to provide a bool_masked_pos.r/rr)rrrq)rrrr)r=rruse_return_dictrJrYr;r*rrr r{logger warning_oncer9rSrrryr`)r<rFrprmrqrrrrnrrrexpected_dtyper9embedding_outputencoder_outputss r%rKzSegGptModel.forwards^2C1N-TXT_T_TqTq$8$D $++JjJj &1%_] 7;+/(,.2,0/3&*[ll[#\\[ll [ "%"2"23 [ #4. [! [**+[$D>['tn[d^[ u)) *[^D[r$r^tensorr2c|j\}}}}||z}||z}|j||||||f}|jdddddd}|j|||z|dzdzf}|S)NrGrrBrr rFr)rCrlrE)ryr2rGr3rHrIr~rs r%patchifyr{Gs.4ll+J feZ'L:%K ^^:|\:Wbdn"o^ pF ^^Aq!Q1 -F ^^:|k/I:WX=[\K\"]^ ^F Mr$r~rc b|jd}t|jddz dz}||z|jdk7r"td|jdd|d|d |j|||||df }|j dd ddd d }|j|d||z||zf }|S)Nrrcr rbrzNumber of patches z does not match patch height (z ) and width (rArGrFrBr)rCrrDrlrE)ryr~rrGr2s r% unpatchifyr}SsaJfll2&*s23Jk!V\\!_4 a 11OP\~]jkvjwwy z  ^^:|[*V`bc"d^ eF ^^Aq!Q1 -F ^^:q,2K[[eMe"f^ gF Mr$ceZdZfdZdej dej dej dej fdZxZS) SegGptLosscft||j|_|j|_yr)r/r0betar2rs r%r0zSegGptLoss.__init__cs& KK  ++r$rmr)rnrqctj||fd}|dddddfjdd|jdzdz}t ||j d|jz|j d|jz}t j||d|j}||zj|jz }|S)aNComputes the L1 loss between the predicted masks and the ground truth masks. Args: prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values from mask prompt. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, 2*height, width)`): Predicted masks. labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Ground truth mask for input images. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: `torch.FloatTensor`: The mean L1 loss between the predicted masks and the ground truth masks. rBrvNrr none) reductionr) r r{repeatr2r}rCrjsmooth_l1_lossrsum)r<rmr)rnrq ground_truthmaskr(s r%rKzSegGptLoss.forwardhs2yy,!7Q? q!Tz*11!Q8JQ8NO$ 2 21 5 H,J\J\]^J_cgcrcrJrs LFQUQZQZ[t   "TXXZ/ r$) rrrr0r r!rrKrLrMs@r%rrbsK, !''!%%!!! ! )) !r$rzCSegGpt model with a decoder on top for one-shot image segmentation.c0eZdZdeffd Zeeeee  dde jde jde jde e jde ed e ed e e j d e ed e ed e edeeeffdZxZS)SegGptForImageSegmentationr=ct||||_t||_t ||_|jyr)r/r0r=r^rLr@decoderrars r%r0z#SegGptForImageSegmentation.__init__s;    ( $V,  r$rkrFrprmrqrrrrnrrrrPc 4||n|jj}| | n|jj} | | n|jj} ||jj j j} tj| tjj|j}d|| dzd|jd}|j ||||||||| |  } | r | jn| d} tj| d} |j!| }d}| t#|j}|||||}| s)|f}| r || dfz}|r| rdnd}|| |fz}||f|z}|St%||| j&| j( S) a^ labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Returns: Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptForImageSegmentation >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptForImageSegmentation.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> result = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[(image_input.height, image_input.width)])[0] >>> print(list(result.shape)) [170, 297] ``` Nr/rrBr) rFrprmrqrrrrnrrrrcrv)r(r)rr)r=rrrprLrJrYr9r rSrrrryrr{rrr'rr)r<rFrprmrqrrrrnrrrr9rrr)r(loss_fnridxs r%rKz"SegGptForImageSegmentation.forwards^2C1N-TXT_T_TqTq$8$D $++JjJj &1%a 'tna d^a  u33 4a hDa r$r)rF)>rcollections.abcr6 dataclassesrtypingrrrrrr torch.utils.checkpointr torch.nnr rj activationsr modeling_utilsr utilsrrrrrrconfiguration_seggptr get_loggerrrqrx_CHECKPOINT_FOR_DOC_EXPECTED_OUTPUT_SHAPErr'Moduler+rOrrrrrrrrrr%r7r@rKSEGGPT_START_DOCSTRINGrwr^rr{r}rrr#r$r%rsB!55 $!-/   H %!.& J+J J0 :K: :0 BII FRryyRjK4biiK4^  U\\eTV[VbVb* -RYY -,")),^B BIIB Lbii< 0BIIB2GO2Gj "Jfs's sl U\\ s u||  u|| 3 S U\\ ''TIn !6n n r$