a jn @sdZddlmZddlmZddlmZddlZ gdZ Gdd d e Z d d Z d.ddZd/ddZeZd0ddZd1ddZddZddZdd Zd2d!d"Zd3d#d$Zd4d%d&Zd5d)d*Zd6d,d-ZdS)7z Functions --------- .. autosummary:: :toctree: generated/ line_search_armijo line_search_wolfe1 line_search_wolfe2 scalar_search_wolfe1 scalar_search_wolfe2 )warn) _minpack2)DCSRCHN)LineSearchWarningline_search_wolfe1line_search_wolfe2scalar_search_wolfe1scalar_search_wolfe2line_search_armijoc@s eZdZdS)rN)__name__ __module__ __qualname__rr@E:\soccer\ml_env\Lib\site-packages\scipy/optimize/_linesearch.pyrsrcCs,d|kr|krdks(ntddS)Nrrz.'c1' and 'c2' do not satisfy'0 < c1 < c2 < 1'.) ValueError)c1c2rrr _check_c1_c2s rr-C6??2:0yE>+=c  s|durgR}|gdgdgfdd} fdd}t|}t| |||||| | | | d \}}}|dd||dfS)a1 As `scalar_search_wolfe1` but do a line search to direction `pk` Parameters ---------- f : callable Function `f(x)` fprime : callable Gradient of `f` xk : array_like Current point pk : array_like Search direction gfk : array_like, optional Gradient of `f` at point `xk` old_fval : float, optional Value of `f` at point `xk` old_old_fval : float, optional Value of `f` at point preceding `xk` The rest of the parameters are the same as for `scalar_search_wolfe1`. Returns ------- stp, f_count, g_count, fval, old_fval As in `line_search_wolfe1` gval : array Gradient of `f` at the final point Notes ----- Parameters `c1` and `c2` must satisfy ``0 < c1 < c2 < 1``. Nrcs(dd7<|gRSNrrrsargsffcpkxkrrphiSszline_search_wolfe1..phics<|gRd<dd7<tdSrnpdotr)rfprimegcgvalr!r"rrderphiWsz"line_search_wolfe1..derphi)rramaxaminxtol)r%r&r )rr'r"r!gfkold_fval old_old_fvalrrrr+r,r-r#r*derphi0stpZfvalr)rrr r'r(r)r!r"rr&s&   rc Cst|||dur|d}|dur*|d}|dur^|dkr^tdd|||} | dkrbd} nd} d} t||||| ||} | | ||| d\} }}}| ||fS)a Scalar function search for alpha that satisfies strong Wolfe conditions alpha > 0 is assumed to be a descent direction. Parameters ---------- phi : callable phi(alpha) Function at point `alpha` derphi : callable phi'(alpha) Objective function derivative. Returns a scalar. phi0 : float, optional Value of phi at 0 old_phi0 : float, optional Value of phi at previous point derphi0 : float, optional Value derphi at 0 c1 : float, optional Parameter for Armijo condition rule. c2 : float, optional Parameter for curvature condition rule. amax, amin : float, optional Maximum and minimum step size xtol : float, optional Relative tolerance for an acceptable step. Returns ------- alpha : float Step size, or None if no suitable step was found phi : float Value of `phi` at the new point `alpha` phi0 : float Value of `phi` at `alpha=0` Notes ----- Uses routine DCSRCH from MINPACK. Parameters `c1` and `c2` must satisfy ``0 < c1 < c2 < 1`` as described in [1]_. References ---------- .. [1] Nocedal, J., & Wright, S. J. (2006). Numerical optimization. In Springer Series in Operations Research and Financial Engineering. (Springer Series in Operations Research and Financial Engineering). Springer Nature. Nr?)\(@d)phi0r1maxiter)rminr)r#r*r7old_phi0r1rrr+r,r-alpha1r8Zdcsrchr2phi1taskrrrr es 5 r c  sdgdgdgdg fdd} | fdd|dur` gR}t| }dur fdd}nd}t| ||||| | || d \}}}}|durtd td d nd}|dd|||fS) a Find alpha that satisfies strong Wolfe conditions. Parameters ---------- f : callable f(x,*args) Objective function. myfprime : callable f'(x,*args) Objective function gradient. xk : ndarray Starting point. pk : ndarray Search direction. The search direction must be a descent direction for the algorithm to converge. gfk : ndarray, optional Gradient value for x=xk (xk being the current parameter estimate). Will be recomputed if omitted. old_fval : float, optional Function value for x=xk. Will be recomputed if omitted. old_old_fval : float, optional Function value for the point preceding x=xk. args : tuple, optional Additional arguments passed to objective function. c1 : float, optional Parameter for Armijo condition rule. c2 : float, optional Parameter for curvature condition rule. amax : float, optional Maximum step size extra_condition : callable, optional A callable of the form ``extra_condition(alpha, x, f, g)`` returning a boolean. Arguments are the proposed step ``alpha`` and the corresponding ``x``, ``f`` and ``g`` values. The line search accepts the value of ``alpha`` only if this callable returns ``True``. If the callable returns ``False`` for the step length, the algorithm will continue with new iterates. The callable is only called for iterates satisfying the strong Wolfe conditions. maxiter : int, optional Maximum number of iterations to perform. Returns ------- alpha : float or None Alpha for which ``x_new = x0 + alpha * pk``, or None if the line search algorithm did not converge. fc : int Number of function evaluations made. gc : int Number of gradient evaluations made. new_fval : float or None New function value ``f(x_new)=f(x0+alpha*pk)``, or None if the line search algorithm did not converge. old_fval : float Old function value ``f(x0)``. new_slope : float or None The local slope along the search direction at the new value ````, or None if the line search algorithm did not converge. Notes ----- Uses the line search algorithm to enforce strong Wolfe conditions. See Wright and Nocedal, 'Numerical Optimization', 1999, pp. 59-61. The search direction `pk` must be a descent direction (e.g. ``-myfprime(xk)``) to find a step length that satisfies the strong Wolfe conditions. If the search direction is not a descent direction (e.g. ``myfprime(xk)``), then `alpha`, `new_fval`, and `new_slope` will be None. Examples -------- >>> import numpy as np >>> from scipy.optimize import line_search A objective function and its gradient are defined. >>> def obj_func(x): ... return (x[0])**2+(x[1])**2 >>> def obj_grad(x): ... return [2*x[0], 2*x[1]] We can find alpha that satisfies strong Wolfe conditions. >>> start_point = np.array([1.8, 1.7]) >>> search_gradient = np.array([-1.0, -1.0]) >>> line_search(obj_func, obj_grad, start_point, search_gradient) (1.0, 2, 1, 1.1300000000000001, 6.13, [1.6, 1.4]) rNcs(dd7<|gRSrralpharrrr#szline_search_wolfe2..phicsDdd7<|gRd<|d<tdSrr$r?)rr'r(r) gval_alphar!r"rrr*$sz"line_search_wolfe2..derphics2d|kr||}|||dS)Nrr)r@r#x)r*extra_conditionr)rAr!r"rrextra_condition21s  z,line_search_wolfe2..extra_condition2)r8*The line search algorithm did not converge stacklevel)r%r&r rr)rZmyfprimer"r!r.r/r0rrrr+rCr8r#r1rD alpha_starphi_star derphi_starr) rr*rCrr r'r(r)rAr!r"rrs.^ rc Cst|||dur|d}|dur*|d}d} |durV|dkrVtdd|||} nd} | dkrfd} |durxt| |} || } |} |}|durdd}t| D]T}| dks|dur| |krd}|}|}d}| dkrd}n d d |}t|td d q|dk}| ||| |ks*| | krT|rTt| | | | |||||||| \}}}q|| }t|| |kr|| | r| }| }|}q|dkrt| | | | |||||||| \}}}qd | }|durt||}| } |} | } || } |}q| }| }d}td td d ||||fS)aFind alpha that satisfies strong Wolfe conditions. alpha > 0 is assumed to be a descent direction. Parameters ---------- phi : callable phi(alpha) Objective scalar function. derphi : callable phi'(alpha) Objective function derivative. Returns a scalar. phi0 : float, optional Value of phi at 0. old_phi0 : float, optional Value of phi at previous point. derphi0 : float, optional Value of derphi at 0 c1 : float, optional Parameter for Armijo condition rule. c2 : float, optional Parameter for curvature condition rule. amax : float, optional Maximum step size. extra_condition : callable, optional A callable of the form ``extra_condition(alpha, phi_value)`` returning a boolean. The line search accepts the value of ``alpha`` only if this callable returns ``True``. If the callable returns ``False`` for the step length, the algorithm will continue with new iterates. The callable is only called for iterates satisfying the strong Wolfe conditions. maxiter : int, optional Maximum number of iterations to perform. Returns ------- alpha_star : float or None Best alpha, or None if the line search algorithm did not converge. phi_star : float phi at alpha_star. phi0 : float phi at 0. derphi_star : float or None derphi at alpha_star, or None if the line search algorithm did not converge. Notes ----- Uses the line search algorithm to enforce strong Wolfe conditions. See Wright and Nocedal, 'Numerical Optimization', 1999, pp. 59-61. Nr3rr4r5cSsdS)NTr)r@r#rrrrCsz-scalar_search_wolfe2..extra_conditionz7Rounding errors prevent the line search from convergingz4The line search algorithm could not find a solution zless than or equal to amax: %srFrGrE)rr9rangerr_zoomabs)r#r*r7r:r1rrr+rCr8alpha0r;phi_a1phi_a0Z derphi_a0irIrJrKmsgZnot_first_iterationZ derphi_a1alpha2rrrr Js8        r c CsLtjddddz|}||}||} || d|| } td} | d| d<|d | d<| d | d<|d| d <t| t|||||||| g\} } | | } | | } | | d| |}|| t|d| }Wn"tyYWd d S0Wd n1s.0Yt|sHd S|S) z Finds the minimizer for a cubic polynomial that goes through the points (a,fa), (b,fb), and (c,fc) with derivative at a of fpa. If no minimizer can be found, return None. raisedivideZoverinvalidrF)rFrF)rr)rr)rr)rrN) r%errstateemptyr&ZasarrayflattensqrtArithmeticErrorisfinite)afafpabfbcr CdbZdcdenomZd1ABradicalxminrrr _cubicmins.     4 rmc Cstjddddrz@|}|}||d}||||||}||d|} Wn tyrYWddS0Wdn1s0Yt| sdS| S)z Finds the minimizer for a quadratic polynomial that goes through the points (a,fa), (b,fb) with derivative at a of fpa. rUrVr4@N)r%rZr^r_) r`rarbrcrdDrfrgrjrlrrr_quadmins  2 rpc Csd} d} d}d}|}d}||}|dkr4||}}n ||}}| dkrb||}t|||||||}| dks|dus|||ks|||kr||}t|||||}|dus|||ks|||kr|d|}||}||| ||ks||kr|}|}|}|}np||}t|| |kr>| ||r>|}|}|}q|||dkrb|}|}|}|}n|}|}|}|}|}| d7} | | krd}d}d}qq|||fS)a Zoom stage of approximate linesearch satisfying strong Wolfe conditions. Part of the optimization algorithm in `scalar_search_wolfe2`. Notes ----- Implements Algorithm 3.6 (zoom) in Wright and Nocedal, 'Numerical Optimization', 1999, pp. 61. r>rg?皙?N?r)rmrprN)Za_loZa_hiZphi_loZphi_hiZ derphi_lor#r*r7r1rrrCr8rRZdelta1Zdelta2Zphi_recZa_recZdalphar`rcZcchkZa_jZqchkZphi_ajZ derphi_ajZa_starZval_starZ valprime_starrrrrMsb     (   rMc sjtdgfdd}|dur6|d} n|} t|} t|| | ||d\} } | d| fS)aMinimize over alpha, the function ``f(xk+alpha pk)``. Parameters ---------- f : callable Function to be minimized. xk : array_like Current point. pk : array_like Search direction. gfk : array_like Gradient of `f` at point `xk`. old_fval : float Value of `f` at point `xk`. args : tuple, optional Optional arguments. c1 : float, optional Value to control stopping criterion. alpha0 : scalar, optional Value of `alpha` at start of the optimization. Returns ------- alpha f_count f_val_at_alpha Notes ----- Uses the interpolation algorithm (Armijo backtracking) as suggested by Wright and Nocedal in 'Numerical Optimization', 1999, pp. 56-57 rcs(dd7<|gRSrr)r;rrrr#szline_search_armijo..phiNr3)rrO)r%Z atleast_1dr&scalar_search_armijo) rr"r!r.r/rrrOr#r7r1r@r<rrrr ps"     r c Cs0t||||||||d}|d|dd|dfS)z8 Compatibility wrapper for `line_search_armijo` )rrrOrrrF)r ) rr"r!r.r/rrrOrrrrline_search_BFGSsrucCs||}|||||kr$||fS| |dd||||}||}|||||krj||fS||kr|d|d||} |d|||||d||||} | | } |d |||||d||||} | | } | tt| dd| |d| } || } | ||| |krP| | fS|| |dkstd| |dkr||d} |}| }|}| }qjd|fS)a(Minimize over alpha, the function ``phi(alpha)``. Uses the interpolation algorithm (Armijo backtracking) as suggested by Wright and Nocedal in 'Numerical Optimization', 1999, pp. 56-57 alpha > 0 is assumed to be a descent direction. Returns ------- alpha phi1 rFrnrYg@rgQ?N)r%r]rN)r#r7r1rrOr,rQr;rPfactorr`rcrTZphi_a2rrrrss8" ,$rsrqrrcCs|d}t|} d} d} d} || |} || \}}|| ||| d|krX| } q| d||d| d|}|| |} || \}}|| ||| d|kr| } q| d||d| d|}t||| || } t||| || } q| | ||fS)a@ Nonmonotone backtracking line search as described in [1]_ Parameters ---------- f : callable Function returning a tuple ``(f, F)`` where ``f`` is the value of a merit function and ``F`` the residual. x_k : ndarray Initial position. d : ndarray Search direction. prev_fs : float List of previous merit function values. Should have ``len(prev_fs) <= M`` where ``M`` is the nonmonotonicity window parameter. eta : float Allowed merit function increase, see [1]_ gamma, tau_min, tau_max : float, optional Search parameters, see [1]_ Returns ------- alpha : float Step length xp : ndarray Next position fp : float Merit function value at next position Fp : ndarray Residual at next position References ---------- [1] "Spectral residual method without gradient information for solving large-scale nonlinear systems of equations." W. La Cruz, J.M. Martinez, M. Raydan. Math. Comp. **75**, 1429 (2006). rrF)maxr%clip)rx_kdZprev_fsetagammatau_mintau_maxf_kZf_baralpha_palpha_mr@xpfpFpalpha_tpalpha_tmrrr_nonmonotone_line_search_cruzs((      r333333?c Cs(d} d} d} || |}||\}}||||| d|krF| } q| d||d| d|}|| |}||\}}||||| d|kr| } q| d||d| d|}t||| | | } t||| | | } q | |d}| |||||}|}| |||||fS)a Nonmonotone line search from [1] Parameters ---------- f : callable Function returning a tuple ``(f, F)`` where ``f`` is the value of a merit function and ``F`` the residual. x_k : ndarray Initial position. d : ndarray Search direction. f_k : float Initial merit function value. C, Q : float Control parameters. On the first iteration, give values Q=1.0, C=f_k eta : float Allowed merit function increase, see [1]_ nu, gamma, tau_min, tau_max : float, optional Search parameters, see [1]_ Returns ------- alpha : float Step length xp : ndarray Next position fp : float Merit function value at next position Fp : ndarray Residual at next position C : float New value for the control parameter C Q : float New value for the control parameter Q References ---------- .. [1] W. Cheng & D.-H. Li, ''A derivative-free nonmonotone line search and its application to the spectral residual method'', IMA J. Numer. Anal. 29, 814 (2009). rrF)r%ry)rrzr{rrfQr|r}r~rnurrr@rrrrrZQ_nextrrr_nonmonotone_line_search_cheng3s*/       r) NNNrrrrrr)NNNrrrrr) NNNrrrNNr>)NNNrrNNr>)rrr)rrr)rrr)rrqrr)rrqrrr)__doc__warningsrZscipy.optimizerZminpack2Z_dcsrchrnumpyr%__all__RuntimeWarningrrrr Z line_searchrr rmrprMr rursrrrrrrsN     ? M   "[ 4 ? I