sgXLdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z m Z ddl mZddlmZdd lmZmZmZdd lmZdd lmZmZmZmZmZmZmZdd lm Z d dl!m"Z"erddl#m$Z$ejJe&Z'dZ(dZ)eGddeZ*eGddeZ+eGddeZ,eGddeZ-Gdde j\Z/dZ0Gdde j\Z1Gdd e j\Z2Gd!d"e j\Z3Gd#d$e j\Z4d%Z5Gd&d'e j\Z6Gd(d)e j\Z7Gd*d+e j\Z8Gd,d-eZ9d.Z:d/Z;Gd0d1e9Z<Gd2d3e9Z=ed4e:Gd5d6e9Z>Gd7d8e j\Z?ed9e:Gd:d;e9Z@ede9ZAd?eBfd@ZCGdAdBe j\ZDGdCdDe j\ZEgdEZFy)FzPyTorch DETR model.N) dataclass)DictListOptionalTupleUnion)Tensornn)ACT2FN)_prepare_4d_attention_mask)BaseModelOutput"BaseModelOutputWithCrossAttentionsSeq2SeqModelOutput)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardis_timm_availableloggingreplace_return_docstringsrequires_backends) load_backbone) DetrConfig) create_modelrzfacebook/detr-resnet-50c:eZdZUdZdZeejed<y)DetrDecoderOutputa Base class for outputs of the DETR decoder. This class adds one attribute to BaseModelOutputWithCrossAttentions, namely an optional stack of intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. This is useful when training the model with auxiliary decoding losses. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` and `config.add_cross_attention=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. intermediate_hidden_states (`torch.FloatTensor` of shape `(config.decoder_layers, batch_size, num_queries, hidden_size)`, *optional*, returned when `config.auxiliary_loss=True`): Intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. Nintermediate_hidden_states __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__Y/var/www/html/venv/lib/python3.12/site-packages/transformers/models/detr/modeling_detr.pyrr3s 2?C):): ;Br)rc:eZdZUdZdZeejed<y)DetrModelOutputa) Base class for outputs of the DETR encoder-decoder model. This class adds one attribute to Seq2SeqModelOutput, namely an optional stack of intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. This is useful when training the model with auxiliary decoding losses. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the decoder of the model. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. intermediate_hidden_states (`torch.FloatTensor` of shape `(config.decoder_layers, batch_size, sequence_length, hidden_size)`, *optional*, returned when `config.auxiliary_loss=True`): Intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a layernorm. Nrr r(r)r*r,r,Qs!!F?C):): ;Br)r,ceZdZUdZdZeejed<dZ ee ed<dZ ejed<dZ ejed<dZ eee ed<dZeejed<dZeeejed <dZeeejed <dZeeejed <dZeejed <dZeeejed <dZeeejed<y)DetrObjectDetectionOutputa Output type of [`DetrForObjectDetection`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` are provided)): Total loss as a linear combination of a negative log-likehood (cross-entropy) for class prediction and a bounding box loss. The latter is defined as a linear combination of the L1 loss and the generalized scale-invariant IoU loss. loss_dict (`Dict`, *optional*): A dictionary containing the individual losses. Useful for logging. logits (`torch.FloatTensor` of shape `(batch_size, num_queries, num_classes + 1)`): Classification logits (including no-object) for all queries. pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_queries, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual image in the batch (disregarding possible padding). You can use [`~DetrImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. auxiliary_outputs (`list[Dict]`, *optional*): Optional, only returned when auxilary losses are activated (i.e. `config.auxiliary_loss` is set to `True`) and labels are provided. It is a list of dictionaries containing the two above keys (`logits` and `pred_boxes`) for each decoder layer. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the decoder of the model. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. Nloss loss_dictlogits pred_boxesauxiliary_outputslast_hidden_statedecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_stateencoder_hidden_statesencoder_attentions)r!r"r#r$r/rr%r&r'r0rr1r2r3rr4r5rr6r7r8r9r:r(r)r*r.r.ys-^)-D(5$$ %, $Ix~$ $FE  $$(J!!(.2xT +259x 1 129@D8E%*;*;$<=D=Au'8'8!9:A;?huU%6%678?=Ax(9(9:A@D8E%*;*;$<=D=Au'8'8!9:Ar)r.ceZdZUdZdZeejed<dZ ee ed<dZ ejed<dZ ejed<dZ ejed<dZeee ed<dZeejed <dZeeejed <dZeeejed <dZeeejed <dZeejed <dZeeejed<dZeeejed<y)DetrSegmentationOutputaP Output type of [`DetrForSegmentation`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` are provided)): Total loss as a linear combination of a negative log-likehood (cross-entropy) for class prediction and a bounding box loss. The latter is defined as a linear combination of the L1 loss and the generalized scale-invariant IoU loss. loss_dict (`Dict`, *optional*): A dictionary containing the individual losses. Useful for logging. logits (`torch.FloatTensor` of shape `(batch_size, num_queries, num_classes + 1)`): Classification logits (including no-object) for all queries. pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_queries, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual image in the batch (disregarding possible padding). You can use [`~DetrImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_queries, height/4, width/4)`): Segmentation masks logits for all queries. See also [`~DetrImageProcessor.post_process_semantic_segmentation`] or [`~DetrImageProcessor.post_process_instance_segmentation`] [`~DetrImageProcessor.post_process_panoptic_segmentation`] to evaluate semantic, instance and panoptic segmentation masks respectively. auxiliary_outputs (`list[Dict]`, *optional*): Optional, only returned when auxiliary losses are activated (i.e. `config.auxiliary_loss` is set to `True`) and labels are provided. It is a list of dictionaries containing the two above keys (`logits` and `pred_boxes`) for each decoder layer. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the decoder of the model. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. Nr/r0r1r2 pred_masksr3r4r5r6r7r8r9r:)r!r"r#r$r/rr%r&r'r0rr1r2r=r3rr4r5rr6r7r8r9r:r(r)r*r<r<s$3j)-D(5$$ %, $Ix~$ $FE  $$(J!!($(J!!(.2xT +259x 1 129@D8E%*;*;$<=D=Au'8'8!9:A;?huU%6%678?=Ax(9(9:A@D8E%*;*;$<=D=Au'8'8!9:Ar)r<c2eZdZdZfdZfdZdZxZS)DetrFrozenBatchNorm2dz BatchNorm2d where the batch statistics and the affine parameters are fixed. Copy-paste from torchvision.misc.ops with added eps before rqsrt, without which any other models than torchvision.models.resnet[18,34,50,101] produce nans. cJt||jdtj||jdtj ||jdtj ||jdtj|y)Nweightbias running_mean running_var)super__init__register_bufferr%oneszeros)selfn __class__s r*rFzDetrFrozenBatchNorm2d.__init__sn  Xuzz!}5 VU[[^4 ^U[[^< ]EJJqM:r)c H|dz}||vr||=t ||||||||y)Nnum_batches_tracked)rE_load_from_state_dict) rJ state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsnum_batches_tracked_keyrLs r*rOz+DetrFrozenBatchNorm2d._load_from_state_dicts?#)+@"@ "j 023 %  oWa r)cB|jjdddd}|jjdddd}|jjdddd}|jjdddd}d}|||zj z}|||zz }||z|zS)Nrgh㈵>)rAreshaperBrDrCrsqrt)rJxrArBrDrCepsilonscales r*forwardzDetrFrozenBatchNorm2d.forwards$$QAq1yy  B1-&&..q"a; ((00B1= +/6688lU**5y4r))r!r"r#r$rFrOr_ __classcell__rLs@r*r?r?s;    r)r?c|jD]_\}}t|tjr t |j }|j jtjdk(s|j jj|j |jjj|j|jjj|j|jjj|j||j|<tt!|j#dkDsUt%|by)z Recursively replace all `torch.nn.BatchNorm2d` with `DetrFrozenBatchNorm2d`. Args: model (torch.nn.Module): input model metarN)named_children isinstancer BatchNorm2dr? num_featuresrAdevicer%datacopy_rBrCrD_moduleslenlistchildrenreplace_batch_norm)modelnamemodule new_modules r*roro's,,. ' f fbnn -.v/B/BCJ==''5<<+??!!&&,,V]];$$**6;;7'',,2263F3FG&&++11&2D2DE#-ENN4 tFOO%& '! + v & 'r)cZeZdZdZfdZdej dej fdZxZS)DetrConvEncoderz Convolutional backbone, using either the AutoBackbone API or one from the timm library. nn.BatchNorm2d layers are replaced by DetrFrozenBatchNorm2d as defined above. c,t |||_|jrt |dgt |di}|in|j }|jdd}|jd|j}|jr|jdd|d<t|jf|jd||d |}n t|}tj 5t#|ddd||_|jr$|j$j&j)n|j$j(|_d}|j |j}n.|j,|j,j.}n t1d d |vrp|j$j3D]R\}}|jr!d |vsd |vsd|vs!|j5d3d|vs8d|vs=d|vsB|j5dTyy#1swYxYw)Ntimmbackbone_kwargs out_indices)rr in_chans output_strideT) pretrained features_onlyryr|zGEither `backbone` or `backbone_config` should be provided in the configresnetlayer2layer3layer4Fzstage.1zstage.2zstage.3)rErFconfiguse_timm_backbonergetattrcopypop num_channelsdilationgetrbackboneuse_pretrained_backbonerr%no_gradrorp feature_infochannelsintermediate_channel_sizesbackbone_config model_type ValueErrornamed_parametersrequires_grad_) rJrkwargsryrrbackbone_model_typerq parameterrLs r*rFzDetrConvEncoder.__init__Gs    # # dVH -V%6;F!>Rv{{}F **]LAK!::j&2E2EFL*0**_b*I'#!99"'%   H%V,H]]_ ) x ( ) 282J2JDJJ # # , , .PTPZPZPcPc '# ?? &"(//   # # /"("8"8"C"C fg g * *#'::#>#>#@ 8i++t+0DY]I]!007 ,$1F9\`K`!007  8 + ) )s % H  H pixel_values pixel_maskc|jjr|j|n|j|j}g}|D]t}tj j |dj|jddjtjd}|j||fv|S)N)sizer) rrrp feature_mapsr functional interpolatefloatshapetor%boolappend)rJrrfeaturesout feature_mapmasks r*r_zDetrConvEncoder.forward{s/3{{/L/L4::l+RVR\R\]iRjRwRw# ,K==,,Z-=-C-C-EKL]L]^`^aLb,cffglgqgqrstuD JJ T* + , r)) r!r"r#r$rFr%r r_r`ras@r*ruru?s)28h ELL ell r)ruc(eZdZdZfdZdZxZS) DetrConvModelzp This module adds 2D position embeddings to all intermediate feature maps of the convolutional encoder. c>t|||_||_yN)rErF conv_encoderposition_embedding)rJrrrLs r*rFzDetrConvModel.__init__s ("4r)c|j||}g}|D]?\}}|j|j||j|jA||fSr)rrrrdtype)rJrrrposrrs r*r_zDetrConvModel.forwardsd j9!$ Y K JJt..{DADD[EVEVW X YCxr)r!r"r#r$rFr_r`ras@r*rrs5 r)rc*eZdZdZdfd ZdZxZS)DetrSinePositionEmbeddingz This is a more standard version of the position embedding, very similar to the one used by the Attention is all you need paper, generalized to work on images. ct|||_||_||_||dur t d|dt jz}||_y)NFz+normalize should be True if scale is passedrz) rErF embedding_dim temperature normalizermathpir^)rJrrrr^rLs r*rFz"DetrSinePositionEmbedding.__init__sW *&"  e!3JK K =KE r)c h| td|jdtj}|jdtj}|jrB||ddddddfdzz |j z}||ddddddfdzz |j z}tj |jtj|jj}|jdtj|dd z|jz z}|dddddddf|z }|dddddddf|z }tj|ddddddd ddfj|dddddddddfjfd j!d }tj|ddddddd ddfj|dddddddddfjfd j!d }tj"||fd j%d d dd}|S)NzNo pixel mask providedr)rrzrYgư>)rrhfloor) rounding_moderr{dimr )rcumsumr%float32rr^arangerint64rhrrdivstacksincosflattencatpermute) rJrry_embedx_embeddim_tpos_xpos_yrs r*r_z!DetrSinePositionEmbedding.forwards  56 6##AU]]#;##AU]]#; >>BC!3d!:;djjHGArs!3d!:;djjHG T//u{{&*H*H)IJKKr)ceZdZdZ ddedededeffd Zdejded efd Z dejd e e fd Z dd ejde ejd e ejde ejde ejdede eje eje e ejffdZxZS) DetrAttentionz Multi-headed attention from 'Attention Is All You Need' paper. Here, we add position embeddings to the queries and keys (as explained in the DETR paper). embed_dim num_headsdropoutrBct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_tj||||_ tj||||_ tj||||_ tj||||_ y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).࿩rB)rErFrrrhead_dimrscalingr Lineark_projv_projq_projout_proj)rJrrrrBrLs r*rFzDetrAttention.__init__s "" !Y. ==9 $ 6MdnnM]^;b" }}d* ii 94@ ii 94@ ii 94@  )YTB r)tensorseq_len batch_sizec|j|||j|jjddj S)Nrrz)viewrr transpose contiguous)rJrrrs r*_shapezDetrAttention._shapes7{{:w NXXYZ\]^iikkr)object_queriesc||S||zSrr()rJrrs r*with_pos_embedzDetrAttention.with_pos_embeds'/vLVn5LLr) hidden_statesattention_maskkey_value_statesspatial_position_embeddingsoutput_attentionsreturnc|du}|j\}} } ||} |j||}||} |j||}|j||jz} |rE|j |j |d|}|j |j  d|}nD|j |j |d|}|j |j  d|}||jzd|jf}|j | | |j|} |j|}|j|}|jd}tj| |jdd}|j||jz| |fk7r/td||jz| |fd|j|{|j|d| |fk7r#td|d| |fd|j|j||j| ||z}|j||jz| |}tjj!|d}|r?|j||j| |}|j||jz| |}nd}tjj#||j"|j$ }tj||}|j||jz| |jfk7r7td ||j| |jfd|j|j||j| |j}|jdd}|j'|| | }|j)|}||fS) z#Input shape: Batch x Time x ChannelNrYrrzz$Attention weights should be of size z , but is z!Attention mask should be of size rptrainingz `attn_output` should be of size )rrrrrrrrrrr%bmmrrr rsoftmaxrrrZr)rJrrrr r r is_cross_attentionr target_lenrhidden_states_originalkey_value_states_original query_states key_states value_states proj_shape source_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r*r_zDetrAttention.forward s.T9,9,>,>,@) J   %%2 " // ~NM ' 2(8 %#223CE`a {{=1DLL@ T[[1A%BB SJ;;t{{3L'MrS]^LT[[%?ZPJ;;t{{3I'JBPZ 4>>12t}}E Mt{{<ZHMMzZ $Z__j1 (|((*5 __Q' yyz/C/CAq/IJ    :#> J"W W6 T^^8SU_ak7l6mn %%'(*   %""$Q J(OO 7Q T^8_7`a&++-.0(,,ZU_`cqqL',,Z$..-H*V`aL}},,\r,B  %1$5$5j$..R\^h$i !055j4>>6QS]_ijL$( !]]**<4<>:t}}] !++Aq1 !))*j)L mmK0 111r))T)NNNNF)r!r"r#r$intrrrFr%r rrrrr_r`ras@r*rrs2 CCC C  C0lU\\lClSlMU\\M8FCSM 261537>B"'Y2||Y2!.Y2!. Y2 #5<<0 Y2 &.ell%; Y2 Y2 u||Xell3XeELL>Q5RR SY2r)rc ~eZdZdeffd Z ddej dej dej defdZxZ S) DetrEncoderLayerrcft||j|_t |j|j |j |_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)Nrrr)rErFrrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrJrrLs r*rFzDetrEncoderLayer.__init__gs &nn44,,  %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r)rrrr c|}|j||||\}}tjj||j|j}||z}|j |}|}|j |j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|jrtj|js#tj|jrEtj|j j"dz }tj$|| |}|f}|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. object_queries (`torch.FloatTensor`, *optional*): Object queries (also called content embeddings), to be added to the hidden states. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr ri)minmax)r(r rrrr*r,r/r-r0r1r%isinfanyisnanfinforr5clamp) rJrrrr residualr clamp_valueoutputss r*r_zDetrEncoderLayer.forwardws&!&*nn'))/ '5' # | --mt||VZVcVc-d  =0 11-@  **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0 --m< =={{=)--/5;;}3M3Q3Q3S#kk-*=*=>BBTI % M |Q\ ]  "    &Gr)NF) r!r"r#rrFr%r rr_r`ras@r*r#r#fsN=z=((,"' 3||3 3 3  3r)r#ceZdZdeffd Z d dej deej deej deej deej deej d eefd Z xZ S) DetrDecoderLayerrct||j|_t |j|j |j |_|j|_t|j|_ |j|_ tj|j|_t |j|j |j |_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)Nr%)r)rErFrrrdecoder_attention_headsr'r(rr r+r,r-r r)r* encoder_attnencoder_attn_layer_normrdecoder_ffn_dimr/r0r1r2s r*rFzDetrDecoderLayer.__init__s  &nn44,,  ~~ #F$>$>?"(";";$&LL$@!) NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r)rrrquery_position_embeddingsr9encoder_attention_maskr c,|}|j||||\}} tjj||j|j}||z}|j |}d} |h|}|j ||||||\}} tjj||j|j}||z}|j|}|}|j|j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|f} |r| | | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. object_queries (`torch.FloatTensor`, *optional*): object_queries that are added to the hidden states in the cross-attention layer. query_position_embeddings (`torch.FloatTensor`, *optional*): position embeddings that are added to the queries and keys in the self-attention layer. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, target_len, source_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr rN)rrr rr r ) r(r rrrr*rCrDr,r/r-r0r1) rJrrrrFr9rGr r;self_attn_weightscross_attn_weightsr=s r*r_zDetrDecoderLayer.forwards>!,0>>'4)/ ,:, ( ( --mt||VZVcVc-d  =0 11-@ " ,$H040A0A+8!65,:"3 1B1 -M-MM11-4< >Gr))NNNNNF) r!r"r#rrFr%r rrr_r`ras@r*r@r@s=z=62615<@8<9=,1M||M!.M!. M $,ELL#9 M ( 5 M!) 6M$D>Mr)r@c&eZdZeZdZdZgdZdZy)DetrPreTrainedModelrpr)rur#r@c.|jj}|jj}t|trt j j|jjt j j|jjt j j|jj|t j j|jj|nvt|trft j j|jjt j j|j jt|t j"t j$t j&frY|jj(j+d||j%|jj(j-yyt|t j.rf|jj(j+d||j02|jj(|j0j-yyy)N)gainr )meanstd)rinit_stdinit_xavier_stdreDetrMHAttentionMapr initzeros_k_linearrBq_linearxavier_uniform_rAruniform_rrrConv2drfrinormal_zero_r padding_idx)rJrrrP xavier_stds r* _init_weightsz!DetrPreTrainedModel._init_weightsskk""[[00 f0 1 GGNN6??// 0 GGNN6??// 0 GG # #FOO$:$: # L GG # #FOO$:$: # L  < = GG  V2299 : GG  V55<< = fryy"))R^^D E MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r)N) r!r"r#r config_classbase_model_prefixmain_input_name_no_split_modulesr_r(r)r*rLrLsL$OV?r)rLaI This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DetrConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a` Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DetrImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, num_queries)`, *optional*): Not used by default. Can be used to mask object queries. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing the flattened feature map (output of the backbone + projection layer), you can choose to directly pass a flattened representation of an image. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`, *optional*): Optionally, instead of initializing the queries with a tensor of zeros, you can choose to directly pass an embedded representation. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c<eZdZdZdeffd Z ddZxZS) DetrEncoderaU Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`DetrEncoderLayer`]. The encoder updates the flattened feature map through multiple self-attention layers. Small tweak for DETR: - object_queries are added to the forward pass. Args: config: DetrConfig rct|||j|_|j|_t j t|jDcgc] }t|c}|_ |jycc}wr) rErFrencoder_layerdrop layerdropr ModuleListrangeencoder_layersr#layers post_initrJr_rLs r*rFzDetrEncoder.__init__ysf  ~~ 11mmuVMbMbGc$d!%5f%=$de  %esBc||n|jj}||n|jj}||n|jj}|}tj j ||j |j}|t||j}|rdnd}|rdnd} t|jD]c\} } |r||fz}d} |jr&tjg} | |jkrd} | rd}n| ||||}|d}|s[| |d fz} e|r||fz}|std ||| fDSt!||| S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Flattened feature map (output of the backbone + projection layer) that is passed to the encoder. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding pixel features. Mask values selected in `[0, 1]`: - 1 for pixel features that are real (i.e. **not masked**), - 0 for pixel features that are padding (i.e. **masked**). [What are attention masks?](../glossary#attention-mask) object_queries (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Object queries that are added to the queries in each self-attention layer. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrr(FTNN)rr rrc3&K|] }|| ywrr(.0vs r* z&DetrEncoder.forward..seqWXWdesr4r attentions)rr output_hidden_statesuse_return_dictr rrrr r enumeraterlr%randrhtupler)rJ inputs_embedsrrr ry return_dictrencoder_statesall_attentionsi encoder_layerto_dropdropout_probability layer_outputss r*r_zDetrEncoder.forwardsD2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r))NNNNNNr!r"r#r$rrFr_r`ras@r*rerejs.  z !S r)recBeZdZdZdeffd Z ddZxZS) DetrDecodera Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`DetrDecoderLayer`]. The decoder updates the query embeddings through multiple self-attention and cross-attention layers. Some small tweaks for DETR: - object_queries and query_position_embeddings are added to the forward pass. - if self.config.auxiliary_loss is set to True, also returns a stack of activations from all decoding layers. Args: config: DetrConfig rcpt|||j|_|j|_t j t|jDcgc] }t|c}|_ t j|j|_ d|_|jycc}wr>)rErFrdecoder_layerdroprhr rirjdecoder_layersr@rlr)r layernormgradient_checkpointingrmrns r*rFzDetrDecoder.__init__s  ~~ 11mmuVMbMbGc$d!%5f%=$de fnn5&+#  %esB3c X||n|jj}||n|jj}| | n|jj} ||} |j dd} d} | | | t ||j  dz} ||t ||j  d}|jjrdnd} |rdnd}|rdnd}|r|dnd}t|jD]\}}|r| fz }|jr%tjg}||jkr?|jr-|jr!|j|j  | ||d}n| | |||||}|d} |jjr|j#| } | | fz } |s||dfz }|||dfz }|j# } |r|| fz }|jjrtj$| } | st'd | |||| fDSt)| |||| S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): The query embeddings that are passed into the decoder. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on certain queries. Mask values selected in `[0, 1]`: - 1 for queries that are **not masked**, - 0 for queries that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding pixel_values of the encoder. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). object_queries (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Object queries that are added to the queries and keys in each cross-attention layer. query_position_embeddings (`torch.FloatTensor` of shape `(batch_size, num_queries, hidden_size)`): , *optional*): Position embeddings that are added to the values and keys in each self-attention layer. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrY)tgt_lenr()rrrFr9rGr rrrzc3$K|]}|| ywrr(rss r*rvz&DetrDecoder.forward..~s=s)r4rrxr7r)rr ryrzrr rauxiliary_lossr{rlrr%r|rhr_gradient_checkpointing_func__call__rrr}r)rJr~rr9rGrrFr ryrr input_shapecombined_attention_mask intermediateall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerrrs r*r_zDetrDecoder.forwards`2C1N-TXT_T_TqTq$8$D $++JjJj &1%rrrprrrJrqparams r*freeze_backbonezDetrModel.freeze_backbones<==55;;LLN (KD%   ' (r)c|jjjjD]\}}|j dy)NTrrs r*unfreeze_backbonezDetrModel.unfreeze_backbones<==55;;LLN 'KD%   & 'r) output_typer`rrdecoder_attention_maskencoder_outputsr~decoder_inputs_embedsr ryrr c ||n|jj}||n|jj}| | n|jj} |j\} } } } |j }|t j| | | f|}|j||\}}|d\}}| td|j|}|jdjddd}|djdjddd}|jd}||j|||||| }nI| rGt|ts7t|dt!|dkDr|dndt!|dkDr|dnd }|j"j$j'dj)| dd}t j*|}|j-|d|||d|||| }| s||zSt/|j0|j2|j4|j6|j0|j2|j4|j8 S) a Returns: Examples: ```python >>> from transformers import AutoImageProcessor, DetrModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50") >>> model = DetrModel.from_pretrained("facebook/detr-resnet-50") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> # the last hidden states are the final query embeddings of the Transformer decoder >>> # these are of shape (batch_size, num_queries, hidden_size) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 100, 256] ```NrrYz/Backbone does not return downsampled pixel maskrzrrr~rrr ryrrw r~rrrFr9rGr ryr)r4r5r6r7r8r9r:r)rr ryrzrrhr%rHrrrrrrrerrlrFrArr zeros_likerr,r4rrxr7r)rJrrrrr~rr ryrrrrrrhrobject_queries_listrrprojected_feature_mapflattened_featuresrflattened_maskrFqueriesdecoder_outputss r*r_zDetrModel.forwardsT2C1N-TXT_T_TqTq$8$D $++JjJj &1%2D2D/ L&%$$  j&%%@&QJ )- lJ(O%%%RL T <NO O!% 5 5k B3::1=EEaAN,R088;CCAq!La  ""ll0--"3%9' +OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO%)$B$B$I$I$S$STU$V$]$]^hjkmn$o!""#<=,,!)&?"1!"4#1/!5#'  "_4 4-??"1"?"?.99,==&5&G&G"1"?"?.99'6'Q'Q  r))NNNNNNNN)r!r"r#rrFrrrrrDETR_INPUTS_DOCSTRINGrr,_CONFIG_FOR_DOCr%r&r LongTensorrrrr_r`ras@r*rrs4z&('++@A?Y26>B7;59=A,0/3&*z ''z U--.z !)):): ; z "%"3"34 z   1 12 z  ((9(9:z $D>z 'tnz d^z  uU&&'8 9z ZBz r)rc(eZdZdZfdZdZxZS)DetrMLPPredictionHeada Very simple multi-layer perceptron (MLP, also called FFN), used to predict the normalized center coordinates, height and width of a bounding box w.r.t. an image. Copied from https://github.com/facebookresearch/detr/blob/master/models/detr.py ct|||_|g|dz z}tjdt |g|z||gzD|_y)Nrc3NK|]\}}tj||ywr)r r)rtrKks r*rvz1DetrMLPPredictionHead.__init__..Bs#g1BIIaO#gs#%)rErF num_layersr riziprl)rJ input_dim hidden_dim output_dimrhrLs r*rFzDetrMLPPredictionHead.__init__>sS $ LJN +mm#gYKRSOUVZdYeUe@f#gg r)ct|jD]D\}}||jdz kr%tjj ||n||}F|S)Nr)r{rlrr rrelu)rJr\rlayers r*r_zDetrMLPPredictionHead.forwardDsT!$++. VHAu01DOOa4G0G ""58,USTXA Vr)rras@r*rr5sh r)rz DETR Model (consisting of a backbone and encoder-decoder Transformer) with object detection heads on top, for tasks such as COCO detection. cteZdZdeffd Zeeeee  dde jde e jde e jde e jde e jd e e jd e eed e ed e ed e edeee jeffdZxZS)DetrForObjectDetectionrct||t||_t j |j |jdz|_t|j |j dd|_ |jy)Nrr{r )rrrr) rErFrrpr rr num_labelsclass_labels_classifierrbbox_predictorrmr2s r*rFzDetrForObjectDetection.__init__Rsr  v& (*yy NNF--1( $4nnAZ[  r)rrrrrr~rlabelsr ryrr c | | n|jj} |j|||||||| |  } | d} |j| } |j | j }d\}}}|d\}}|jj rC| r | jn| d}|j|}|j |j }|j| ||j||j||\}}}| s| | |f|z| z}n| |f| z}|||f|zS|St||| ||| j| j| j| j| j| j | j" S)a labels (`List[Dict]` of len `(batch_size,)`, *optional*): Labels for computing the bipartite matching loss. List of dicts, each dictionary containing at least the following 2 keys: 'class_labels' and 'boxes' (the class labels and bounding boxes of an image in the batch respectively). The class labels themselves should be a `torch.LongTensor` of len `(number of bounding boxes in the image,)` and the boxes a `torch.FloatTensor` of shape `(number of bounding boxes in the image, 4)`. Returns: Examples: ```python >>> from transformers import AutoImageProcessor, DetrForObjectDetection >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50") >>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> target_sizes = torch.tensor([image.size[::-1]]) >>> results = image_processor.post_process_object_detection(outputs, threshold=0.9, target_sizes=target_sizes)[ ... 0 ... ] >>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]): ... box = [round(i, 2) for i in box.tolist()] ... print( ... f"Detected {model.config.id2label[label.item()]} with confidence " ... f"{round(score.item(), 3)} at location {box}" ... ) Detected remote with confidence 0.998 at location [40.16, 70.81, 175.55, 117.98] Detected remote with confidence 0.996 at location [333.24, 72.55, 368.33, 187.66] Detected couch with confidence 0.995 at location [-0.02, 1.15, 639.73, 473.76] Detected cat with confidence 0.999 at location [13.24, 52.05, 314.02, 470.93] Detected cat with confidence 0.999 at location [345.4, 23.85, 640.37, 368.72] ```)rrrr~rr ryrrNNNrqr{) r/r0r1r2r3r4r5r6r7r8r9r:)rrzrprrsigmoidrr loss_functionrhr.r4r5r6r7r8r9r:)rJrrrrr~rrr ryrr=sequence_outputr1r2r/r0r3 outputs_class outputs_coordroutputs r*r_zDetrForObjectDetection.forwardcsv&1%((9AAC -=*i*  +5 (M={{))EPwAAV]^_V` $ < <\ J $ 3 3L A I I K 151C1C ZmUb2 .D). , *-0AAGK *-7373CT9%. O O(!/%77")"?"?&99$55&-&G&G")"?"?&99  r)r)r!r"r#rrFrrrr.rr%r&rrrdictrrrr_r`ras@r*rrJs:z"++@A+DSbc26>B7;59=A'+,0/3&*m ''m U--.m !)):): ; m "%"3"34 m   1 12 m  ((9(9:m d$m $D>m 'tnm d^m  uU&&')BB Cm dBm r)rz DETR Model (consisting of a backbone and encoder-decoder Transformer) with a segmentation head on top, for tasks such as COCO panoptic. cteZdZdeffd Zeeeee  dde jde e jde e jde e jde e jd e e jd e eed e ed e ed e edeee jeffdZxZS)DetrForSegmentationrcvt||t||_|j|j }}|jj jjj}t||z|ddddd||_ t|||d|j|_|jy)NrYr )rrP)rErFrdetrrr&rprrrDetrMaskHeadSmallConv mask_headrSrRbbox_attentionrm)rJr hidden_sizenumber_of_headsrrLs r*rFzDetrForSegmentation.__init__s  +62 (.~~v7U7U_ %)YY__%=%=%J%J%e%e". / )+Edd+KBC+PR] 1 osH^H^  r)rrrrrr~rrr ryrr c | | n|jj} |j\} } } }|j}|t j | | |f|}|j jj||\}}|d\}}|j\} } } }|j jj|}|jdjddd}|djdjddd}|jd}|,|j jj||||| | }nI| rGt|ts7t|dt|dkDr|dndt|dkDr|dnd }|j jj j"j%dj'| dd}t j(|}|j jj+|d|||d||| | }|d}|j j-|}|j j/|j1}|djdddj3| |jj4| |}|j3| | |}|j7||| }|j9|||dd|dd|ddg} | j3| |j jj:| jd | jd}!d \}"}#}$|d\}%}&|jj<rW| r |j>n|d}'|j j-|'}%|j j/|'j1}&|jA|||||!|j|%|&\}"}#}$| s'|$|||!f|$z|z|z}(n |||!f|z|z}(|"|"|#f|(zS|(StC|"|#|||!|$|jD|jF|jH|jJ|jD|jF|jH S)a labels (`List[Dict]` of len `(batch_size,)`, *optional*): Labels for computing the bipartite matching loss, DICE/F-1 loss and Focal loss. List of dicts, each dictionary containing at least the following 3 keys: 'class_labels', 'boxes' and 'masks' (the class labels, bounding boxes and segmentation masks of an image in the batch respectively). The class labels themselves should be a `torch.LongTensor` of len `(number of bounding boxes in the image,)`, the boxes a `torch.FloatTensor` of shape `(number of bounding boxes in the image, 4)` and the masks a `torch.FloatTensor` of shape `(number of bounding boxes in the image, height, width)`. Returns: Examples: ```python >>> import io >>> import requests >>> from PIL import Image >>> import torch >>> import numpy >>> from transformers import AutoImageProcessor, DetrForSegmentation >>> from transformers.image_transforms import rgb_to_id >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50-panoptic") >>> model = DetrForSegmentation.from_pretrained("facebook/detr-resnet-50-panoptic") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> # Use the `post_process_panoptic_segmentation` method of the `image_processor` to retrieve post-processed panoptic segmentation maps >>> # Segmentation results are returned as a list of dictionaries >>> result = image_processor.post_process_panoptic_segmentation(outputs, target_sizes=[(300, 500)]) >>> # A tensor of shape (height, width) where each value denotes a segment id, filled with -1 if no segment is found >>> panoptic_seg = result[0]["segmentation"] >>> # Get prediction score and segment_id to class_id mapping of each segment >>> panoptic_segments_info = result[0]["segments_info"] ```Nr)rrYrzrrrrwr)rrrrq) r/r0r1r2r=r3r4r5r6r7r8r9r:)&rrzrrhr%rHrrprrrrrrerrlrFrArrrrrrrrrrrrrrrr<r4rrxr7))rJrrrrr~rrr ryrrrrrrhrrrrrrrrrFrrrr1r2memory bbox_mask seg_masksr=r/r0r3rrrrs) r*r_zDetrForSegmentation.forwardsx&1%2D2D/ L&%$$  Z$?OJ)- (@(@Zd(@(e%%%RL T2=2C2C/ L&% $  @ @ M3::1=EEaAN,R088;CCAq!La  ""iioo550--"3%9' 6OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO%)IIOO$M$M$T$T$^$^_`$a$h$h 1% !""#<=))//11!)&?"1!"4#1/!5#2  *!,22?CYY--o>FFH  #++Aq!499*dkkFYFY[achi"":vu= ''te'L NN#8)hqkRSnV^_`VabcVdfnopfqrsftEuv ^^J 0@0@0L0Lioo^`Naclcrcrsucvw -=*i*  +5 (M={{))MXII^mnp^q $ A A, O $ 8 8 F N N P 151C1C J ]\i2 .D). , *j9B7;59=A'+,0/3&*j ''j U--.j !)):): ; j "%"3"34 j   1 12 j  ((9(9:j d$j $D>j 'tnj d^j  uU&&')?? @j aBj r)rlengthc||jdjdt|dddjddS)Nrr)rrr!r)rrs r*_expandrs7   A  % %aVaA > F Fq! LLr)c<eZdZdZfdZdededeefdZxZS)rz^ Simple convolutional head, using group norm. Upsampling is done using a FPN approach ct||dzdk7r td||dz|dz|dz|dz|dzg}tj||dd |_tj d||_tj||d dd |_tj td|d |d |_ tj|d |ddd |_ tj td|d|d|_ tj|d|ddd |_ tj td|d|d|_tj|d|ddd |_tj td|d|d|_tj|dd dd |_||_tj|d|d d |_tj|d |dd |_tj|d|dd |_|j-D]r}t/|tjstj0j3|j4d tj0j7|j8dty) NrzsThe hidden_size + number of attention heads must be divisible by 8 as the number of groups in GroupNorm is set to 8rzr{r~rr r)padding)a)rErFrr rZlay1 GroupNormgn1lay2r4gn2lay3gn3lay4gn4lay5gn5out_layradapter1adapter2adapter3modulesrerTkaiming_uniform_rA constant_rB)rJrfpn_dims context_dim inter_dimsmrLs r*rFzDetrMaskHeadSmallConv.__init__sG  7a<)  ;!+[A-={a?OQ\`bQbdosuduv IIc315 <<3'IIc:a=!Q? <<Az!} 5z!}EIIjmZ]AqI <<Az!} 5z!}EIIjmZ]AqI <<Az!} 5z!}EIIjmZ]AqI <<Az!} 5z!}EyyA1a@  (1+z!}a@  (1+z!}a@  (1+z!}a@  -A!RYY'((Q(7!!!&&!, -r)r\rfpnsctjt||jd|j ddgd}|j |}|j |}tjj|}|j|}|j|}tjj|}|j|d}|jd|jdk7r-t||jd|jdz}|tjj||jdddz}|j|}|j!|}tjj|}|j#|d}|jd|jdk7r-t||jd|jdz}|tjj||jdddz}|j%|}|j'|}tjj|}|j)|d}|jd|jdk7r-t||jd|jdz}|tjj||jdddz}|j+|}|j-|}tjj|}|j/|}|S)Nrrrnearest)rmoderz)r%rrrrrrr rrrrrrrrrrrrrrrr)rJr\rrcur_fpns r*r_zDetrMaskHeadSmallConv.forwards IIwq)//!"45y7H7HA7NOQR S IIaL HHQK MM  q ! IIaL HHQK MM  q !--Q( <<?affQi 'gqvvayGLLO'CDG bmm// bc8JQZ/[ [ IIaL HHQK MM  q !--Q( <<?affQi 'gqvvayGLLO'CDG bmm// bc8JQZ/[ [ IIaL HHQK MM  q !--Q( <<?affQi 'gqvvayGLLO'CDG bmm// bc8JQZ/[ [ IIaL HHQK MM  q ! LLOr)) r!r"r#r$rFr rr_r`ras@r*rrs, -D&&F&$v,&r)rc8eZdZdZdfd ZddeefdZxZS)rSzdThis is a 2D attention module, which only returns the attention softmax (no multiplication by value)c&t|||_||_t j ||_t j||||_t j||||_ t||jz dz|_ y)Nrr) rErFrrr DropoutrrrWrVrnormalize_fact)rJ query_dimrrrrBrPrLs r*rFzDetrMHAttentionMap.__init__so "$zz'*  )ZdC  )ZdC #J$?@DHr)rc|j|}tjj||jj j dj d|jj}|j|jd|jd|j|j|jz}|j|jd|j|j|jz|jd|jd}tjd||jz|}|W|j|j dj dtj |j"j$tjj'|j)ddj|j+}|j-|}|S)NrYrrrzbqnc,bnchw->bqnhwrzr)rWr rconv2drVrArrBrrrrr%einsumr masked_fill_r9rr4rrrr)rJqrrqueries_per_head keys_per_headweightss r*r_zDetrMHAttentionMap.forwards{ MM!  MM DMM$8$8$B$B2$F$P$PQS$TVZVcVcVhVh i66!''!*aggaj$..$//]a]k]kJklqwwqz4>>4??dnn;\^_^e^efh^iklkrkrsukvw ,,24DtGZGZ4Z\ij    !2!r0s !55 !Bgg-2*!   H %/ C:C C: $C($C $CN ;B ;B ;B| BB[BB BBN$ BII$ N'0EbiiEPBII*" "J299. ~2BII~2BDryyDNfryyfR?/?< #Lo %o dm %m ` ^ #^ ^ DBII* A 0A A H  A -A A HMCM MBIIM`8 r)