Skip to content Skip to navigation
Computer Science Department Colloquium
4/17/2014 03:20 pm
CBIM Multipurpose Room ( Room 22 )

Lessons Learned from a Large-scale Empirical Comparison of Learning Algorithms

Dr. Alexandru Niculescu-Mizil, NEC Research

Faculty Host: Tina Eliassi-Rad

Abstract

In this talk, I will share some of the insights we gained from a large-scale comparison of binary classification algorithms. In our study, we evaluated some of the most widely used learning algorithms using a variety of metrics that emphasize different performance aspects: accuracy at a set threshold, ability to rank positive cases higher than negative ones, or ability to predict well-calibrated probabilities.  Besides addressing the obvious questions that arise with such a comparison (is there a "best" learning algorithm? are SVMs better than neural networks? do newly developed algorithms like boosting and SVM really provide an improvement in practice?), I will discuss a few unexpected and, I would say, more exciting findings from the study. I will look in depth at the ability of learning algorithms to produce well-calibrated probabilistic predictions, I will address the question of what performance metric should be used for model selection, and, time permitting, I will show how performance can be improved by using Ensemble Selection to combine predictions from models trained by the different learning algorithms.

This is joint work with Rich Caruana, Art Munson, David Skalak,Tom Fawcett, Geoff Crew, Alex Ksikes and Cristi Bucila.

Bio

Alexandru Niculescu-Mizil is researcher at NEC Laboratories America. Before joining NEC, he was a Herman Goldstine postdoctoral fellow at IBM T.J. Watson Research Center. He received his Ph.D. from Cornell University in 2008 under the supervision of Rich Caruana, a Masters of Science degree in Computer Science from Cornell University and a Magna Cum Laude Bachelors degree in Mathematics and Computer Science from University of Bucharest. His research interests are in machine learning and data mining, particularly in inductive transfer, graphical model structure learning, probability estimation, empirical evaluations, ensemble methods and on-line learning. He received an ICML Distinguished Student Paper Award in 2005 for his work on probability estimation, and a COLT Best Student in 2008 paper award for his work on on-line learning. In 2009, he led the IBM Research team that won the KDDCUP competition.