sg"PdZddlZGddZGddeZGddeZy) z'Stochastic optimization methods for MLPNc*eZdZdZddZdZdZdZy) BaseOptimizera9Base (Stochastic) gradient descent optimizer Parameters ---------- learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights Attributes ---------- learning_rate : float the current learning rate c2||_t||_yN)learning_rate_initfloat learning_rate)selfrs `/var/www/html/venv/lib/python3.12/site-packages/sklearn/neural_network/_stochastic_optimizers.py__init__zBaseOptimizer.__init__s"4"#56cf|j|}td|D|D] \}}||z } y)aUpdate parameters with given gradients Parameters ---------- params : list of length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params grads : list of length = len(params) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params c3 K|]}|ywr).0ps r z.BaseOptimizer.update_params..*s!4!!4s N) _get_updateszip)r paramsgradsupdatesparamupdates r update_paramszBaseOptimizer.update_paramss=##E* !4V!4g> ME6 VOE r cy)zhPerform update to learning rate and potentially other states at the end of an iteration Nrr time_steps r iteration_endszBaseOptimizer.iteration_ends-s r c$|rt|dzy)aHDecides whether it is time to stop training Parameters ---------- msg : str Message passed in for verbose output verbose : bool Print message to stdin if True Returns ------- is_stopping : bool True if training needs to stop Stopping.T)printr msgverboses r trigger_stoppingzBaseOptimizer.trigger_stopping3s  # $ %r N)皙?)__name__ __module__ __qualname____doc__r rrr&rr r rr s 7" r rc@eZdZdZ dfd ZdZdZdZxZS) SGDOptimizeraStochastic gradient descent optimizer with momentum Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights lr_schedule : {'constant', 'adaptive', 'invscaling'}, default='constant' Learning rate schedule for weight updates. -'constant', is a constant learning rate given by 'learning_rate_init'. -'invscaling' gradually decreases the learning rate 'learning_rate_' at each time step 't' using an inverse scaling exponent of 'power_t'. learning_rate_ = learning_rate_init / pow(t, power_t) -'adaptive', keeps the learning rate constant to 'learning_rate_init' as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if 'early_stopping' is on, the current learning rate is divided by 5. momentum : float, default=0.9 Value of momentum used, must be larger than or equal to 0 nesterov : bool, default=True Whether to use nesterov's momentum or not. Use nesterov's if True power_t : float, default=0.5 Power of time step 't' in inverse scaling. See `lr_schedule` for more details. Attributes ---------- learning_rate : float the current learning rate velocities : list, length = len(params) velocities that are used to update params ct||||_||_||_||_|Dcgc]}t j|c}|_ycc}wr) superr lr_schedulemomentumnesterovpower_tnp zeros_like velocities) r rrr0r1r2r3r __class__s r r zSGDOptimizer.__init__xsN +,&     =CDE2==/DDsAc~|jdk(r.t|j|dz|jzz |_yy)aPerform updates to learning rate and potential other states at the end of an iteration Parameters ---------- time_step : int number of training samples trained on so far, used to update learning rate for 'invscaling' invscalingN)r0rrr3r rs r rzSGDOptimizer.iteration_endss?   | +d--.)a-DLL1PP   ,r c|jdk7r|rt|dzy|jdkr|rt|dzy|xjdzc_|rt|d|jzzy) Nadaptiver!Tgư>z# Learning rate too small. Stopping.g@z Setting learning rate to %fF)r0r"r r#s r r&zSGDOptimizer.trigger_stoppingsu   z )cL()    %cAAB c!  #69K9KKK Lr cXt|j|Dcgc]$\}}|j|z|j|zz &}}}||_|jrEt|j|Dcgc]$\}}|j|z|j|zz &}}}|Scc}}wcc}}w)Get the values used to update params with given gradients Parameters ---------- grads : list, length = len(coefs_) + len(intercepts_) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params Returns ------- updates : list, length = len(grads) The values to add to params )rr6r1r r2)r rvelocitygradrs r rzSGDOptimizer._get_updatess #&doou"= $ MMH $t'9'9D'@ @  " =='*$//5&A"Hd (4+=+=+DDG  s )B 2)B&)r'constant?Tg?) r(r)r*r+r rr&r __classcell__r7s@r r-r-Hs/-dE"  r r-c,eZdZdZ dfd ZdZxZS) AdamOptimizeraStochastic gradient descent optimizer with Adam Note: All default values are from the original Adam paper Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.001 The initial learning rate used. It controls the step-size in updating the weights beta_1 : float, default=0.9 Exponential decay rate for estimates of first moment vector, should be in [0, 1) beta_2 : float, default=0.999 Exponential decay rate for estimates of second moment vector, should be in [0, 1) epsilon : float, default=1e-8 Value for numerical stability Attributes ---------- learning_rate : float The current learning rate t : int Timestep ms : list, length = len(params) First moment vectors vs : list, length = len(params) Second moment vectors References ---------- :arxiv:`Kingma, Diederik, and Jimmy Ba (2014) "Adam: A method for stochastic optimization." <1412.6980> ct||||_||_||_d|_|Dcgc]}t j|c}|_|Dcgc]}t j|c}|_ ycc}wcc}w)Nr) r/r beta_1beta_2epsilontr4r5msvs)r rrrHrIrJrr7s r r zAdamOptimizer.__init__sm +,   5;Bc|xjdz c_t|j|Dcgc]'\}}|j|zd|jz |zz)c}}|_t|j|Dcgc]*\}}|j |zd|j z |dzzz,c}}|_|j tjd|j |jzz zd|j|jzz z |_ t|j|jDcgc]8\}}|j |ztj||jzz :}}}|Scc}}wcc}}wcc}}w)r>r:) rKrrLrHrMrIrr4sqrtr rJ)r rmr@vrs r rzAdamOptimizer._get_updatess[ ! tww. 4 KK!Oq4;;$6 6  tww. 4 KK!Oq4;;47; ;   # #gga$++tvv--. /4;;&& ( DGGTWW- 1   ! #rwwqzDLL'@ A  #   s,E8=/E>6=F)gMbP?rBg+?g:0yE>)r(r)r*r+r rrCrDs@r rFrFs+\SW = r rF)r+numpyr4rr-rFrr r rTs5- <<~z=zzZMZr