Verified formula topic
Optimization Formulas
Browse 0 reviewed optimization formulas with concrete inputs, outputs, calculations, applicability checks, and LaTeX.
Browse 8 supporting references
Optimization
Gradient Descent Update
x_{k+1}=x_k-\eta\nabla f(x_k)Updates parameters in the direction opposite the objective gradient.
Open formulaOptimization
Gradient Descent Update
\theta_{t+1}=\theta_t-\eta\nabla J(\theta_t)Updates parameters in the negative gradient direction.
Open formulaOptimization
Jensen’s Inequality
f(\mathbb E[X])\leq\mathbb E[f(X)]Relates a convex function of an expectation to the expectation of the convex function.
Open formulaOptimization
KKT Stationarity
\nabla f(x)+\sum_i\lambda_i\nabla g_i(x)+\sum_j\nu_j\nabla h_j(x)=0Gives the stationarity part of the Karush–Kuhn–Tucker conditions.
Open formulaOptimization
Lagrange Multiplier Condition
\nabla f(x)=\lambda\nabla g(x)States the first-order condition for an equality-constrained extremum with one regular constraint.
Open formulaOptimization
Linear Program Standard Form
\min_x c^Tx\quad\text{s.t.}\quad Ax=b,\;x\geq0Expresses a linear objective with equality constraints and nonnegative variables.
Open formulaOptimization
Newton Optimization Step
x_{k+1}=x_k-[\nabla^2f(x_k)]^{-1}\nabla f(x_k)Uses gradient and Hessian information to form a local quadratic step.
Open formulaOptimization
Single-Constraint Lagrange Condition
\nabla f(x)=\lambda\nabla g(x)Gives a first-order condition for a constrained extremum with one equality constraint.
Open formula