Gradient descent

SSU | [GD]

Optimization algorithm solving solving ${\text{argmin}}_{\theta }\mathcal{L}(\theta )$ by moving through the parameter space steps given by the gradient.

Each step is given by${\theta }_{k+1}={\theta }_k-{\alpha }_k\nabla \mathcal{L}({\theta }_k),$ where ${\alpha }_k$ is called the learning rate or step size at iteration $k$. The learning rate exist in order to prevent algorithm get stuck at local minimum.

Backpropagation is the standart way to compute gradient $\nabla \mathcal{L}({\theta }_k)$.

Properties