### DIMACS Monitoring Message Streams Seminar Series

Title: Large Margin Generative Models

Speaker: **Tony Jebara**, Columbia University

Date: May 7, 2004 10:00 am

Location: DIMACS Center, CoRE Bldg, Room 433, Rutgers University, Busch Campus, Piscataway, NJ

Abstract:

Generative models such as Bayesian networks, exponential family
distributions and mixtures are elegant formalisms to setup and specify
prior knowledge about a learning problem. However, the standard
estimation methods they rely on, including maximum likelihood and
Bayesian integration do not focus the modeling resources on a
particular input-output task. In applied settings when models are
imperfectly matched to the real data, discriminative learning is
crucial for improving performance with such models. We consider
classifiers built from the log-likelihood ratio of generative models
and find parameters for these models such that the resulting
discrimination boundary has a large margin. Through maximum entropy
discrimination, we show how all exponential family models can be
estimated with large margin using convex programming. Furthermore, we
consider interesting latent models such as mixture models and hidden
Markov models where the additional presence of latent variables makes
large margin estimation difficult. We propose a variant of the maximum
entropy discrimination method that uses variational bounding on
classification constraints to make computations tractable in the
latent case. The method finds large margin settings reliably by
iteratively interleaving standard expectation steps with large margin
maximization information projection steps. Interestingly, the method
gives rise to Lagrange multipliers that behave like posteriors over
hidden variables. Preliminary experiments are shown.

See Webpage: http://www.stat.rutgers.edu/~madigan/mms/spring04.html