Concept
Optimization
The algorithms that actually move the weights: adaptive step sizes, momentum, and the tricks that make deep training converge.
Papers
-
Adam: A Method for Stochastic Optimization
Adam combined momentum with per-parameter adaptive learning rates into an optimizer that just works across a huge range of problems. It is the default optimizer behind most of the papers in this dossier.
arxiv.org ↗