Daniela Witten
University of Washington
The many flavors of penalized linear discriminant analysis
Linear discriminant analysis (LDA) is a well-known
statistical method for classification in the traditional setting where
the number of observations exceeds the number of features. The LDA
classification rule results from three distinct arguments: (1) a
geometric argument (Fisher's discriminant problem), (2) direct
application of Bayes' rule to a simple normal model for the data
(normal model), and (3) a direct extension of linear regression to the
classification problem (optimal scoring). In the high-dimensional
setting when the number of variables exceeds the number of
observations, LDA cannot be directly applied, and so a number of
authors have sought to extend LDA to this setting using penalization
approaches taken from the regression literature. Interestingly, though
Fisher's discriminant problem, the normal model, and optimal scoring
yield the same classification rule in the absence of penalties, they
yield different results when penalized. I will present various
approaches, new and old, for regularizing these three viewpoints of
LDA, and will discuss their similarities and differences.