Workshop Details

Despite impressive recent advances in multi-agent learning, developing analogues of the foundation models encountered in single-agent learning settings remains elusive. The challenge is multi-faceted, and this talk will focus on one of the many facets: the lack of a clear training target. This derives from the fact that multi-agent settings involving agents who use DNNs to model their strategies or utilities quickly escape the scope of classical Game Theory—due to non-convexities, Nash equilibrium existence breaks and standard equilibrium analysis is inapplicable. If this is the case, what is even the goal of training a model in such a setting? I will discuss this challenge and how to overcome it from a combined game-theoretic, complexity-theoretic and learning-theoretic perspective.

[Video]

Loss functions are extremely useful in supervised machine learning to objectively evaluate, compare, and train models. On the other hand, it is not clear how to utilize loss functions for generative models such as GANs and Large Language Models, which use different learning paradigms and present unique challenges for evaluation. This talk will discuss these motivations and challenges and present results from a recent ICML paper, where we define and design loss functions for discrete black-box generative models. Based on joint work with Dhamma Kimpara and Rafael Frongillo, accessible at https://arxiv.org/abs/2211.03761.

[Video]

Game Theory is the study of mathematical models of strategic interactions among rational agents. As humans, language is a key medium of interaction. As such, assuming we can build a suitable model of the players, strategies, and payoffs associated with these interactions (i.e., a binding to the conventional symbolic language of game theory), we can then ask existing algorithms to provide us with optimal, rational solutions (e.g., equilibria), thereby giving us, for example, predictors of stable, rational conversational strategies in dialogue. Large language models have arguably reached a point where grounding to semantics and concepts is strong and accurate enough to rely on as a simulacra of human (+AI) dialogue. These foundation models allow us to begin to think of game theoretic algorithms in the space of natural language. They can also help us quickly prototype new game models via the expressivity of natural language, potentially leading to an explosion in the current repository of games one could study and test our solution concepts on. I’ll present one such binding from dialogue to game theory that we are considering as well as flavors of generalizing existing algorithms to this setting.

[Video]

Machine learning methods have long used majority vote among annotators for ground truth labels, but annotator disagreement often reflects real differences in opinion, not noise. This issue is particularly key for training large language models, which perform a wide range of often sensitive tasks for a diverse population. For example, a crucial problem in hate speech detection is whether a statement is offensive to the demographic that it targets, which may constitute a small fraction of the annotator pool. In this talk, I’ll present a model that predicts individual annotators’ ratings on potentially offensive text and combines this information with the predicted group targeted by the text to model the opinions of relevant stakeholders. I’ll also discuss ongoing challenges and opportunities of designing large language models that incorporate human feedback from multiple perspectives.

[Video]

Large language modeling has changed the nature of natural language processing, with striking success in a large variety of tasks. Despite this success, failures exist, raising questions about their readiness for real use. In this talk I explore two problems for large language models: hallucination and bias. I discuss hallucination in the context of summarization and I present bias in the context of understanding African American Language.

[Video]

While current language models have shown impressive performance in tasks involving generation, factual assertions and verification, they still exhibit notable limitations such as struggling with domains requiring strategic reasoning. On the other hand, AI has demonstrated superhuman gameplay in strategic domains like chess, Go, and poker. However, these successes have been largely confined to two-player or competitive games without language use.

In this talk, I will delve into our work on utilizing language models and game theory for building strategic AI agents for Diplomacy - a 7-player board game featuring both cooperative and competitive elements and where natural language communication between players plays a vital role. Finally, I will briefly highlight some of our recent and upcoming work leveraging these insights for building better models of human behavior, human language use and for improving language models.

[Video]

Insights from human language, for example, how compositional operators work over reusable concepts or how people communicate with one another to solve tasks, provide a rich source of information as to how intelligent behaviour arises in human beings. Can this help us endow agents with the ability to intelligently function in multi-agent populations in the same way that language-speaking humans do? In this talk we’ll go over several pieces of work in which language has been used to guide multi-agent decision making, as well as how insights from game theory can be used to build better language models. These recent works provide a preview of the future of research at the intersection of game theory and natural language processing, and the role that foundation models can play in this area.

[Video]

Evaluating deep multi-agent training algorithms (MATAs) can be quite complicated due to stochasticity in training and sensitivity of agent performance to the behavior of other agents. We propose a meta-game evaluation framework where each MATA is cast as a meta-strategy, and repeatedly sampling normal-form empirical games over combinations of meta-strategies resulting from various random seeds. These empirical games provide the basis for constructing a sampling distribution, using bootstrapping, over a variety of game analysis statistics. We suggest that this framework provides a principled approach for analyzing advanced AI methods that interact with humans or other AI actors.

[Video]

List of Posters:

• Generative Adversarial Equilibrium Solvers, Denizalp Goktas
• Stackelberg Games with Side Information, Keegan Harris
• Self-Confirming Transformer for Locally Consistent Online Adaptation in Multi-Agent Reinforcement Learning, Tao Li
• LLM-augmented Preference Learning from Natural Language, Farhad Mohsin
• Strategic Knowledge Transfer, Max Smith
• Game-theoretic Algorithms for Reinforcement Learning from Human Feedback, Gokul Swamy
• On Learning Latent Models with Multi-Instance Weak Supervision, Kaifu Wang
• Inverse Game Theory for Stackelberg Games: the Blessing of Bounded Rationality, Jibang Wu
• Generative Social Choice, Manuel Wüthrich
• Mechanism Design with Large Language Models, Haifeng Xu
• Causal Reasoning with LLMs: Problems, explorations, intuitions, and beyond, Jiayao Zhang

It is immensely empowering to delegate information processing and automation work to machines and have them carry out difficult tasks on our behalf. But programming computers is hard. The traditional approach to this problem is to try to fix people: They should work harder to learn to code. In this talk, I argue that a promising alternative is to meet people partway. Specifically, powerful new approaches to machine learning provide ways to infer intent from disparate signals and, with your help, could help make it easier for everyone to get computational help with their vexing problems.

[Video]

[Video]

The way we learn from humans is changing as models become more capable. Previously, we’d rely heavily on human demonstrations, handcrafted rewards, and preference labels to train models. However, human feedback is not very scalable for several reasons, including the fact that it simply requires a lot of human effort to specify objectives and preferences. I explore two ways in which we can physically scale the amount of human feedback by reducing human specification burden using foundation models.

[Video]

Major advances in Machine Learning (ML) and Artificial Intelli- gence (AI) increasingly take the form of developing and releasing general-purpose models. These models are designed to be adapted by other businesses and agencies to perform a particular, domain- specific function. This process has become known as adaptation or fine-tuning. This paper offers a model of the fine-tuning process where a Generalist brings the technological product (here an ML model) to a certain level of performance, and one or more Domain- specialist(s) adapts it for use in a particular domain. Both entities are profit-seeking and incur costs when they invest in the technology, and they must reach a bargaining agreement on how to share the revenue for the technology to reach the market. For a relatively general class of cost and revenue functions, we characterize the conditions under which the fine-tuning game yields a profit-sharing solution. We observe that any potential domain-specialization will either contribute, free-ride, or abstain in their uptake of the technology, and we provide conditions yielding these different strategies. We show how methods based on bargaining solutions and sub-game perfect equilibria provide insights into the strategic behavior of firms in these types of interactions, and we find that profit-sharing can still arise even when one firm has significantly higher costs than another. We also provide methods for identifying Pareto-optimal bargaining arrangements for a general set of utility functions.

An old idea is to design AI systems that improve themselves. We show how to take first steps towards this goal by designing recursively self-improving systems for generating code. That is, starting with a short seed program that improves code at some given task using a language model, the code is used to generate a better improver. We discuss safety concerns and discuss how it may be framed as a game.

[Video]