sg'.dZddlmZmZddlZddlmZddlm Z Gdd ejjjZ Gd d ejjjZGd d ejjjZy)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) shape_list) IdeficsConfigc~eZdZdededededededdffd Zfd Zd ejdejfd Z xZ S) TFIdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc t |d i|||||f\|_|_|_|_|j j|_t|jds|jdzn|jjdz|_ g|_ t|D]s}|jjt|j|j|j|jd|dt!|j|d|dgut"j$j&j)dd |_y ) ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r rrrperceiver_configqk_layer_norms_perceiverqk_layer_normshasattr vision_configintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r r rrrkwargsi __class__s [/var/www/html/venv/lib/python3.12/site-packages/transformers/models/idefics/perceiver_tf.pyrz$TFIdeficsPerceiverResampler.__init__1s+( "6"FOQXZbdmFmC dmT^$55NN6//= NNQ %%//!3   u A KK  / dmmTEXEXahijhkkm_n!!6!6wqcQS_U   ((//<>>4>>2[_fo'    k"r1contextctj|jd}tj|tj|dddg}|j D]\}}||||z}|||z}|j |S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r( expand_dimsr4tiler5r#r)r,r;r4attnffs r0callz TFIdeficsPerceiverResampler.callcs..A6'''BHHW$5a$8!Q#?@  ,HD"7G,w6GkG+G ,w''r1) __name__ __module__ __qualname__rintrr9r(TensorrC __classcell__r/s@r0r r 0si)^#)^03)^ let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projuse_biasrk_projv_proj output_projNr)rrr rrrr(r)r*r+rMrNrOrPqk_scaleDenserQrTrUrV)r,r rrrr-r/s r0rz$TFIdeficsPerceiverAttention.__init__ps "6"6?(3 dm,"$((//"D"DTXl"D"m"$((//"D"DTXl"D"m    " B B4Vd B eD  " B B4Vd B eD  t+ hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h 88??00UQ^0_r1r;r4c |j|}|j|}t|\}}}|j|}|j t j ||gd}|jt j ||gd}|||fD cgc]T} t jt j| || jd|j|jfgdVc} \}}}|jr"|j|}|j|}t j d||j"z|} | t j$| dd z } t j&j)| d} t j d | |} |j+t jt j| gd|d|j|jzfScc} w) a= Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r=r)rrr)permz... i d, ... j d -> ... i jT)r>keepdimsz... i j, ... j d -> ... i d)rMrNrrQrTr(concatrU transposereshaper5rrrrOrPeinsumrW reduce_maxnnsoftmaxrV)r,r;r4 batch_size seq_lengthr qkvxscoresstabilized_scoresrA resampleds r0rCz TFIdeficsPerceiverAttention.calls))'2))'2,6w,?) J  KK  KK 7G"42> ? KK 7G"42> ? AY  LLA AGGAJ dmm'\]dp q 1a   !!!$A!!!$A8!dmm:KQO"R]]6T%RRuu}}.R}8II;T1E  JJr||ILAJPRTXT`T`cgcpcpTpCq r   sAG5) rDrErFrGboolrr(rHrCrIrJs@r0r&r&osY`#``s`TX`gk`*+ BII+  + bii+ r1r&cheZdZdeffd ZdeeejdejfdZ xZ S)r'r c t|d i||jj|_tj j jdd|_tj j j|dd|_ tj j jd|_ tj j j|jdd |_ y ) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcrRactrc_projNr)rrr!r r(r)r*r+rrrXrsReLUrtru)r,intermediate_sizer r-r/s r0rzTFIdeficsMLP.__init__s "6"--77((//44T4M((//''(9EPT'U88??''U'3hhoo++DNNUQY+Z r1 hidden_statesrc|j|}|j|}|j|}|j|}|S)N)rrrsrtru)r,rxs r0rCzTFIdeficsMLP.calls@ .  . /  M2 r1) rDrErFrrrrr(rHrCrIrJs@r0r'r's6[-[(5+;"<r1r')__doc__typingrr tensorflowr(modeling_tf_utilsrconfiguration_ideficsrr)r*Layerr r&r'rr1r0rsj4 #+0<("((//"7"7<(~A "((//"7"7A H288??((r1