Deep Learning: Alchemy or Science? Agenda

February 22, 2019

Agenda:

9:45 am Institute Welcome, Robbert Dijkgraaf, Director, Institute for Advanced Study VIDEO

10:00 am Brief introduction to deep learning and the "Alchemy" controversy, Sanjeev Arora, Princeton University; Visiting Professor, School of Mathematics VIDEO

10:45 am Troubling Trends in ML Scholarship, Zachary Lipton, Carnegie Mellon University VIDEO

Abstract: The machine learning community is struggling to deal with several well-documented crises in scholarship: (i) a blurring of fact and fancy (ii) experiments divorced from falsifiability (iii) math that cannot, should not, and often isn’t meant to be followed, and (iv) exposition that sows confusion and distorts the public discourse. However, in other ways, the field is healthier than ever: (a) vibrant economy supports careers in machine learning, (b) mature tooling makes algorithms easier to run and experiments easier to reproduce, and (c) the field is far more welcoming and accessible to new talent. While, at an individual level, clear steps can be improve the quality of research and the resulting papers, what steps can be taken at a community-level is a far more challenging question. What levers can influence community practices? Who should pull them? And which interventions can curb flawed scholarship without undermining the community’s strengths? This talk will aim to present a balanced picture, both of the status quo, the ecosystem that supports it, and the difficulty of improving upon it.

11:25 am Coffee Break in Lobby of Wolfensohn Hall

11:40 am The Epistemology of Deep Learning, Yann LeCun, Facebook AI Research/New York University VIDEO

Abstract: Clearly, Deep Learning research would greatly benefit from better theoretical understanding. DL is partly engineering science in which we create new artifacts through theoretical insight, intuition, biological inspiration, and empirical exploration. But understanding DL is a kind of "physical science" in which the general properties of this artifact is to be understood. The history of science and technology is replete with examples where the technological artifact preceded (not followed) the theoretical understanding: the theory of optics followed the invention of the lens, thermodynamics followed the steam engine, aerodynamics largely followed the airplane, information theory followed radio communication, and computer science followed the programmable calculator. My two main points are that (1) empiricism is a perfectly legitimate method of investigation, albeit an inefficient one, and (2) our challenge is to develop the equivalent of thermodynamics for learning and intelligence. While a theoretical underpinning, even if only conceptual, would greatly accelerate progress, one must be conscious of the limited practical implications of general theories.

12:45 pm Lunch in Simons Hall

2:00 pm Reproducible, Reusable, and Robust Reinforcement Learning, Joelle Pineau, Facebook/McGill University VIDEO

Abstract: We have seen significant achievements with deep reinforcement learning in recent years. Yet reproducing results for state-of-the-art deep RL methods is seldom straightforward. High variance of some methods can make learning particularly difficult when environments or rewards are strongly stochastic. Furthermore, results can be brittle to even minor perturbations in the domain or experimental procedure. In this talk, I will review challenges that arise in experimental techniques and reporting procedures in deep RL. I will also describe several recent results and guidelines designed to make future results more reproducible, reusable and robust.

2:55 pm Afternoon Tea in Lobby of Wolfensohn Hall

3:30 pm Surrogates, Shai Shalev-Shwartz, Hebrew University of Jerusalem VIDEO

Abstract: Through examples from practical and synthetic problems, I will argue that our understanding of fundamental properties of deep learning is lacking. One way to make progress is to study surrogate models and surrogate distributions, and I’ll present some useful examples. Based on joint work with Jonathan Fiat and Eran Malach.

4:25 pm Successes and Challenges in Neural Models for Speech and Language, Michael Collins, Google Research/Columbia University VIDEO

Abstract: In recent years there has been dramatic progress in key problems in speech and natural language processing, largely driven by neural methods. In this talk I'll describe a sequence of NLP/speech problems and neural architectures of increasing complexity. I'll describe the successes of these approaches and also the (many) questions that they raise.

5:30 pm Public programming: Panel discussion VIDEO

6:25 pm Close of program Sanjeev Arora