sg{ v d Z ddlZddlZddlmZ ddlmZmZmZmZ ddl Z
ddl m Z ddl
mZmZmZ ddlmZ dd lmZ ej* e Ze G d
d Z G d d
Ze G d de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z y) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)DictListOptionalUnion)tqdm )IntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZe ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZeed< dZeed< dZeed< dZ eed< dZ eee eee ef f ed< dZ ed ed< d Z d efd Z!e"d efd Z#y) TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`List[Dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`List[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevent callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ y t | j t ry i }| j D ] }t |t st dt
| |j j }||v r?t || t s || g||< || j |j |j ||< || _ y )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r$
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr$ callbacknames P/var/www/html/venv/lib/python3.12/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__q s # !D " " *&(D #
//
6 "$ 3 3
@!(_>#himnviwhxy ))22-- &&8&>E4Ft4L3M*40&t,33HNN4DE/7~~/?&t,
@ '9D # json_pathc t j t j | dd dz }t |dd 5 }|j | ddd y# 1 sw Y yxY w) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r0 r6 json_stringfs r3 save_to_jsonzTrainerState.save_to_json sT jj!3!3D!9!tTW[[
)S7
3 !q
GGK ! ! !s AA!c t |dd 5 }|j }ddd | di t j S # 1 sw Y $xY w)z3Create an instance from the content of `json_path`.rr= r> N )rD readr@ loads)clsr6 rG texts r3 load_from_jsonzTrainerState.load_from_json sI )S7
3 q668D &TZZ%&& s AA )$r,
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r r strr r r boolr r r! r" r r$ r4 rH classmethodrP rK r5 r3 r r # s4 6p "E8E?!KIs M3 JJ c c !"3"J*.Kd3:&'.#'K% '+/8C=/"&4&"&4&"'4'J<@L$sE#uc4"7 889@26 ./696!c ! 's ' 'r5 r c , e Zd ZdZdefdZed Zy)r( aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )NzInternal class that just calls the list of callbacks in order.c g | _ |D ] }| j | || _ || _ || _ || _ d | _ d | _ t d | j D s#t j d| j z y y )Nc 3 < K | ] }t |t y wN)r& DefaultFlowCallback.0cbs r3 z+CallbackHandler.__init__.. s P2:b"56Ps zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r0 r r r r r r s r3 __init__zCallbackHandler.__init__ s "B b ! "
0 " ( $ # P PP NN$ $$%
Qr5 c P t |t r | n|}t |t r|n |j }|| j D cg c] }|j c}v r)t j d| ddz | j z | j j | y c c}w )NzYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r& r* r+ r r r r r. )r0 r1 r cb_classcs r3 r z CallbackHandler.add_callback s %h5XZ8 )(D 98x?Q?Q T^^< < < NN#H:-uv+,$$%
b!
=s B#c t |t r=| j D ]- }t || s| j j | |c S y | j D ]& }||k( s | j j | |c S y r r& r* r remover0 r1 r s r3 pop_callbackz CallbackHandler.pop_callback ss h %nn
b(+NN))"- I
nn
>NN))"- I
r5 c t |t r;| j D ]+ }t || s| j j | y y | j j | y r r r s r3 remove_callbackz CallbackHandler.remove_callback sR h %nn
b(+NN))"-
NN ! !( +r5 c F dj d | j D S )Nr; c 3 H K | ] }|j j y wr )r+ r, r s r3 r z0CallbackHandler.callback_list.. s H2 ..Hs ")joinr r^ s r3 r z CallbackHandler.callback_list s yyH HHHr5 r` r/ ry c * | j d||| S )Nr}
call_eventr0 r` r/ ry s r3 r} zCallbackHandler.on_init_end }dE7CCr5 c 8 d|_ | j d||| S )NFr )rj r r s r3 r z CallbackHandler.on_train_begin s ',$ /ugFFr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_train_end ~tUGDDr5 c 8 d|_ | j d||| S )NFr )rk r r s r3 r z CallbackHandler.on_epoch_begin s $)! /ugFFr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_epoch_end r r5 c T d|_ d|_ d|_ | j d||| S )NFr )rn rm rl r r s r3 r z CallbackHandler.on_step_begin s/ ""' #eWEEr5 c * | j d||| S )Nr r r s r3 r z%CallbackHandler.on_pre_optimizer_step s 6eWMMr5 c * | j d||| S )Nr r r s r3 r z!CallbackHandler.on_optimizer_step s 2D%IIr5 c * | j d||| S )Nr r r s r3 r z CallbackHandler.on_substep_end s /ugFFr5 c * | j d||| S )Nr r r s r3 r zCallbackHandler.on_step_end r r5 c < d|_ | j d|||| S )NFr r )rm r r0 r` r/ ry r s r3 r zCallbackHandler.on_evaluate s# "' }dE7GTTr5 c . | j d|||| S )Nr r r r s r3 r zCallbackHandler.on_predict s |T5'7SSr5 c 8 d|_ | j d||| S )NFr )rl r r s r3 r zCallbackHandler.on_save s #y$w??r5 c < d|_ | j d|||| S )NFr )logs)rn r )r0 r` r/ ry r s r3 r zCallbackHandler.on_log s# "xugDIIr5 c * | j d||| S )Nr r r s r3 r z"CallbackHandler.on_prediction_step s 3T5'JJr5 c | j D ]_ } t || |||f| j | j | j | j
| j | j d|}|^|}a |S )N)r r r r r r )r getattrr r r r r r )r0 eventr` r/ ry r| r1 results r3 r zCallbackHandler.call_event s !H-WXu- jj!%!6!6 .. ! . .!%!6!6 $ 4 4 F ! ! r5 N) r, rQ rR rS r r r r propertyr r
r ri r} r r r r r r r r r r r r r r r rK r5 r3 r r s H& "
, I ID 1 D, DQ_ DG#4 G\ GTb GE!2 E< ER` EG#4 G\ GTb GE!2 E< ER` EF"3 FL FSa F N*; NL N[i NJ&7 J JWe JG#4 G\ GTb GD 1 D, DQ_ DU 1 U, UQ_ UT 0 T TP^ T@ - @l @^ @J, J\ JN JK'8 K KXf Kr5 r c 8 e Zd ZdZdededefdZdededefdZy)r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
r` r/ ry c |j dk( r|j rd|_ |j t j
k( r#|j |j z dk( rd|_ |j t j
k( r<|j |j z dk( r |j |j k rd|_
|j t j
k( r2|j dkD r#|j |j z dk( rd|_ |j |j k\ r:d|_ |j t j" t j$ fvrd|_ |S )Nr Tr )r logging_first_steprn logging_strategyr
STEPSr
eval_strategyr
eval_delayrm
save_strategyr r rl r rj NOBESTr{ s r3 r z DefaultFlowCallback.on_step_end s, !d&=&=!%G $4$:$: :u?P?PSXSfSf?fjk?k!%G
"2"8"8 8!!E$4$4495#4#44&*G #
,"4"4 4 1$!!E$4$449"&G /+/G (!!,//<;L;L)MM&*#r5 c |j t j k( rd|_ |j t j k( r |j
|j k rd|_ |j t j k( rd|_
|S )NT) r r
EPOCHrn r r r rm r r rl r{ s r3 r z DefaultFlowCallback.on_epoch_end> sp $4$:$: :!%G !1!7!7 7DOOu{{ z/. Consider increasing `max_str_len` if needed.]r r r8 )
r r rb r& rW r r poproundrE )
r0 r` r/ ry r r| shallow_logsre rf _s
r3 r zProgressCallback.on_log s & &4+<+<+H L
(1 a%#a&43C3C*C?Axs4K[K[J\ ]H H !O
'(LO
( t4A,&(-l7.CQ(G W% # #C $5 6! ,I &r5 c ` |j r"| j j d | _ y y r )r r r r{ s r3 r z ProgressCallback.on_train_end s* & & # # % $D 'r5 )d r )
r, rQ rR rS rV r r r r r r r r rK r5 r3 r r N s6
'C '
2
*' ' 7&%r5 r c e Zd ZdZddZy)PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc X |j dd }|j r t | y y )Nr )r r print)r0 r` r/ ry r r| r s r3 r zPrinterCallback.on_log s' HH\4 ( & & $K 'r5 r )r, rQ rR rS r rK r5 r3 r r s r5 r c F e Zd ZdZd dedee fdZd Zd Z d Z
de fd Z y
) EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
early_stopping_patienceearly_stopping_thresholdc . || _ || _ d| _ y )Nr r
r early_stopping_patience_counter)r0 r
r s r3 r z EarlyStoppingCallback.__init__ s '>$(@%/0,r5 c |j rt j nt j }|j 8 |||j r-t ||j z
| j kD rd| _ y | xj dz
c_ y )Nr r )greater_is_betternpgreaterlessr absr r )r0 r` r/ ry metric_valueoperators r3 check_metric_valuez(EarlyStoppingCallback.check_metric_value sl !%!7!72::RWW $ \5#4#4 5L5#4#4458U8UU34D 0 0 0A 5 0r5 c |j sJ d |j J d |j t j k7 sJ d y )Nzr( s ! . . E E ,
H % s' s' s'l) )X :
_ :
:
zO
O
d@o @F2/ 2jG% G%To E
O_ E
r5