ABSTRACT:

The information in the weights of deep networks plays a key role in understanding their behavior. I will describe several results connecting the optimization dynamics of a deep neural network (DNN) with acquisition and storage of information inside its weights. In particular, I will show that a deep network encounters two distinct learning phases: an initial irreversible “critical” learning period followed by reversible learning process closer to convergence. During the initial critical period the network acquires information and organizes itself to optimally process the data, and small changes in this phase may permanently affect the network. This creates challenges in the development of large-scale transfer learning systems and in the development of differential privacy and machine unlearning algorithms. I will present several methods by which we can efficiently address these challenges for real-world applications.

BIO:
Alessandro Achille is a Senior Applied Scientist at Amazon AI in Pasadena. His research interests include representation learning and information theory applied to deep neural networks. Prior to joining Amazon, Alessandro obtained a PhD in Computer Science from UCLA in 2019 and a Master’s degree in Pure Math at the Scuola Normale Superiore in Pisa in 2015.

Hosted by Professor Stefano Soatto

Via Zoom Webinar