Zhenyu Liao, University of Paris - Saclay

Abstract

Modern neural networks are commonly trained with gradient-based methods. The understanding of the dynamics of gradient descent algorithm is one of the key issues for the theoretical comprehension of why deep neural nets work so well today. In this work, we introduce a random matrix-based framework to analyze the dynamics of a simple toy network model trained by gradient descent. This preliminary result opens the door for future studies of more elaborate structures and models appearing in today’s neural networks.

Video