sg1ddlZddlZddlmZmZddlmZGddeZGddeZ Gd d e Z Gd d e Z Gd deZ GddeZ y)N)DatasetIterableDataset) ModelOutputceZdZdZdZdZy)PipelineDatasetc.||_||_||_yNdatasetprocessparams)selfr r rs R/var/www/html/venv/lib/python3.12/site-packages/transformers/pipelines/pt_utils.py__init__zPipelineDataset.__init__ s   c,t|jSr lenr rs r__len__zPipelineDataset.__len__4<<  rc^|j|}|j|fi|j}|Sr r )riitem processeds r __getitem__zPipelineDataset.__getitem__s.||A DLL55 rN)__name__ __module__ __qualname__rrrrrrrs !rrc,eZdZddZdZdZdZdZy)PipelineIteratorNcf||_||_||_|dk(rd}||_d|_d|_y)a Roughly equivalent to ``` for item in loader: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```N)loaderinferrloader_batch_size_loader_batch_index_loader_batch_data)rr&r'rr(s rrzPipelineIterator.__init__sA6    ! $ !2$( "&rc,t|jSr )rr&rs rrzPipelineIterator.__len__?s4;;rc:t|j|_|Sr iterr&iteratorrs r__iter__zPipelineIterator.__iter__BT[[)  rctjtjr*jjj d}ni}jj D]\}}t|trz|j}t|dtjrtfd|D||<n4t|dtjrtfd|D||<|dvr{t|trkt|dtjrtfd|D||<n4t|dtjrtfd|D||<|d||<t|jtjr#|jj d||<ct|jtjr(tj|jd||<|j||<jj|}xjdz c_|S) ze Return item located at `loader_batch_index` within the current `loader_batch_data`. rc3ZK|]"}|jjd$ywrNr) unsqueeze.0elrs r z5PipelineIterator.loader_batch_item..U)1n`b"T5M5M2N2X2XYZ2[1n(+c3dK|]'}tj|jd)ywr4np expand_dimsr)r7s rr:z5PipelineIterator.loader_batch_item..W)1tfh"..DD\D\A]_`2a1t-0> attentions hidden_statespast_key_valuesc3ZK|]"}|jjd$ywr4r5r7s rr:z5PipelineIterator.loader_batch_item..\r;r<c3dK|]'}tj|jd)ywr4r>r7s rr:z5PipelineIterator.loader_batch_item..^rArBNr%) isinstancer*torchTensorr)r6itemsrto_tupletupler?ndarrayr@ __class__)rresultloader_batchedkelements` rloader_batch_itemz"PipelineIterator.loader_batch_itemFs d--u|| <,,T-E-EFPPQRSF N"55;;= J 7g{3%..0G!'!*ell;,11nfm1n,nq)#GAJ ;,11tls1t,tq)JJzZachOi!'!*ell;,11nfm1n,nq)#GAJ ;,11tls1t,tq)?(,N1%(@(@ A5<<P)00H0H(I(S(STU(VN1%(@(@ A2::N)+wt?W?W7XZ[(\N1%)00H0H(IN1%= JB,,66~FF   A%  rc|j)|j|jkr|jSt|j}|j |fi|j }|jt|tjr|}n7t|tr|d}n!t|jd}||}t|tr t|}n|jd}d|cxkr|jkr nn||_t|tr|dn||_d|_|jS|S)Nr)r)r(rTnextr/r'rrHrIrJrMlistkeysrshaper*)rrr first_tensorkeyobserved_batch_sizes r__next__zPipelineIterator.__next__us'  # # /D4L4LtOeOe4e))+ +DMM"DJJt3t{{3  ! ! -)U\\2( Iu-(| 9>>+,Q/(~ ,-&),&7#&2&8&8&;#&?)?)??*=&6@E6RilXaD #'(D $))+ + rr )rrr rrr0rTr]r!rrr#r#s%'N -^"rr#c,eZdZdfd ZdZdZxZS)PipelineChunkIteratorc(t||||y)aI Roughly equivalent to ``` for iterator in loader: for item in iterator: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item N)superr)rr&r'rr(rOs rrzPipelineChunkIterator.__init__s$ /rcHt|j|_d|_|Sr )r.r&r/ subiteratorrs rr0zPipelineChunkIterator.__iter__sT[[)  rcj|j6 |jt|jfi|j|_ t|j}|S#t $rN|jt|jfi|j|_t|j}Y|SwxYwr )rcr'rVr/r StopIteration)rrs rr]zPipelineChunkIterator.__next__s    # \)tzz$t}}*=MMD  /T--.I / *tzz$t}}*=MMD T--.I /sAAB21B2r )rrr rr0r] __classcell__)rOs@rr_r_s0( rr_ceZdZdZdZdZy)PipelinePackIteratora Roughly equivalent to ``` packed = [] for item in loader: packed.append(item) if item["is_last"]: yield packed packed = [] ``` but it also handles cases where `item` are batched (meaning it's a dict of Tensor with first dimension > 1. In that case it does ``` packed = [] for batch in loader: # item is batched for item in batch: packed.append(item) if item["is_last"]: yield packed packed = [] ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```c:t|j|_|Sr r-rs rr0zPipelinePackIterator.__iter__r1rcd}g}|j|j|jkri|j|jkrP|j}|jd}|j ||r|S|j|jkrP|se|j t |jfi|j}|jt|tjr|}n!t|jd}||}t|tr t|}n|jd}d|cxkr|jkr nn||_||_d|_|j|jkru|j}|jd}|j ||r|S|j|jkrPn$|}|jd}|j ||se|S)NFis_lastr)r)r(rTpopappendr'rVr/rrHrIrJrWrXrrYr*)rrk accumulatorrrrZr[r\s rr]zPipelinePackIterator.__next__s  # # /D4L4LtOeOe4e**T-C-CC--/((9-""4(&& **T-C-CC" 4 #6F$++FI%%1i6#,Ly~~/03C#,S>LlD1*-l*;'*6*<*rsd 5' g @@F*,*Za+aH ) ) ]W ]r