Theoretical Understanding of Classic Learning Algorithms

Introduction

Machine learning has evolved from being a relatively isolated discipline to have a disruptive influence on all areas of science, industry, and society. Learning algorithms are typically classified into either deep learning or classic learning, where deep learning excels when data and computing resources are abundant, whereas classic algorithms shine when data is scarce.

Project Overview

In the TUCLA project, we expand our theoretical understanding of classic machine learning, with a particular emphasis on two of the most important such algorithms, namely Bagging and Boosting. As a result of this study, we shall provide faster learning algorithms that require less training data to make accurate predictions. The project accomplishes this by pursuing several objectives:

1. Establishing a Novel Framework
   We will establish a novel learning theoretic framework for proving generalization bounds for learning algorithms. Using the framework, we will design new Boosting algorithms and prove that they make accurate predictions using less training data than what was previously possible. Moreover, we complement these algorithms by generalization lower bounds, proving that no other algorithm can make better use of data.

2. Designing Parallel Versions of Boosting Algorithms
   We will design parallel versions of Boosting algorithms, thereby allowing them to be used in combination with more computationally expensive base learning algorithms. We conjecture that success in this direction may lead to Boosting playing a more central role also in deep learning.

3. Exploring Applications of Bagging
   We will explore applications of the classic Bagging heuristic. Until recently, Bagging was not known to have significant theoretical benefits. However, recent pioneering work by the PI shows that Bagging is an optimal learning algorithm in an important learning setup. Using these recent insights, we will explore theoretical applications of Bagging in other important settings.