In Lecture 5 of Stanford’s CS229 Machine Learning course, the focus is on Gaussian Discriminant Analysis (GDA), marking the first introduction to generative models in the course. Unlike discriminative models that directly model the conditional probability of labels given data, generative models aim to learn the joint distribution of data and labels by modeling how data is generated for each class. The lecture uses a simplified example of classifying cats and elephants based on features like height and weight, illustrating how generative models learn separate Gaussian distributions for each class and then classify new data points by comparing their likelihood under each class model using Bayes’ rule.
The lecture then delves into the mathematical foundations of GDA, starting with the one-dimensional Gaussian distribution and extending to multi-dimensional Gaussians characterized by mean vectors and covariance matrices. The covariance matrix captures the shape, size, and orientation of the data distribution, with properties such as symmetry and positive definiteness ensuring well-defined probability densities. Visualizations of Gaussian contours in two dimensions help clarify these concepts, including the effects of diagonal and full covariance matrices on the shape of the distribution.
A key assumption in the basic GDA model is that different classes share the same covariance matrix but have different means. This assumption simplifies the model and leads to a linear decision boundary between classes. The lecture explains how maximum likelihood estimation can be used to find closed-form solutions for the class priors, means, and shared covariance matrix from training data, avoiding the need for iterative gradient descent. The resulting decision boundary is shown to be linear due to the shared covariance assumption, and the lecture contrasts this with Quadratic Discriminant Analysis (QDA), where different covariances per class lead to quadratic decision boundaries.
The relationship between GDA and logistic regression is explored, revealing that under the shared covariance assumption, GDA’s posterior probabilities correspond to a logistic function, making the decision boundary equivalent to that of logistic regression. However, logistic regression is a discriminative model that makes fewer assumptions about data distribution, often requiring more data but being more robust. The lecture highlights that while logistic regression has been traditionally favored, recent advances in AI have renewed interest in generative models due to their ability to learn rich data representations, as seen in models like GPT and diffusion models.
Finally, the lecture introduces Naive Bayes as a simple generative model for discrete features, such as words in spam filtering. By assuming conditional independence of features given the class, Naive Bayes reduces the complexity of modeling the joint distribution, making it computationally efficient and surprisingly effective. The lecture concludes by summarizing the key points: the introduction of generative models, the derivation and estimation of GDA parameters, the comparison with logistic regression, and the application of generative modeling to discrete data with Naive Bayes, setting the stage for further exploration of datasets and evaluation methods in subsequent lectures.
