Minae Kwon, Stanford University

Abstract

The way we learn from humans is changing as models become more capable. Previously, we’d rely heavily on human demonstrations, handcrafted rewards, and preference labels to train models. However, human feedback is not very scalable for several reasons, including the fact that it simply requires a lot of human effort to specify objectives and preferences. I explore two ways in which we can physically scale the amount of human feedback by reducing human specification burden using foundation models.

[Video]