13.5 Case Study: Training a Neural Network

Original Pytorch walkthrough

1Learning Objectives¶

In the preceding lecture, we were introduced to stochastic gradient descent as a computationally feasible alternative to a full backpropagation/gradient descent step. We then went through a whirlwind tour of how these principles can be applied all the way from traditional data science to modern deep learning. In this case study, we will focus on elucidating the main principles specifically behind training a neural network. By the end of this page, you should understand:

• How to load and divide up a large dataset
• The intuition behind various considerations for designing neural networks
• How to set-up a training loop for the neural network.

2The CIFAR-10 Dataset¶

In previous lectures and case studies, we have seen classifying handwritten digits as a recurring setting. This is largely due to historical importance and subsequent popularity of the MNIST dataset, which was one of the key domino pieces leading to the current deep learning boom. The MNIST dataset has some nice properties: the data is rather clean and unambiguous for humans, leaving designing performant models as the main challenge to get computers to perform comparably (if not better). For this same reason, a dataset of comparable importance to MNIST in deep learning is the CIFAR-10/100 (Canadian Institute For Advanced Research) dataset.

In particular, CIFAR-10 like MNIST is comprised of small, labelled images belonging to one of 10 classes. Whereas in MNIST these 10 classes were the 10 digits, the 10 classes of CIFAR-10 are various real-world things: ‘plane’, ‘car’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, and ‘truck’. The classification task is therefore harder than in MNIST, since the photos are generally more “in the wild”, more diverse within each class (think of all the breeds of dogs!), and are representations of 3-D objects, rather than 2-D writing.

Let’s start by loading the data using the convenient torchvision package. Recall that we want to set aside a test set that is disjoint to the training set in order to evaluate the performance of our final model. For CIFAR-10, the training set contains 50000 images divided equally across tasks, while the test set contains 10000 images.

To get an idea of the task at hand, let’s take a look at some sample images from the dataset.

As one can see, the images in CIFAR-10 are quite grainy (32x32 pixels), so one can imagine labeling some of these images can be hard even for humans! A statistic to keep in mind is that the average human performance on CIFAR-10 is estimated to be around 94%.

3An Appropriate Neural Network Architecture¶

Having loaded and perused the data, we now need to determine the model we’ll use for our classifier. In class, we discussed the Multi-Layer Perception (MLP) as a backbone model in deep learning. As a quick review, an $L$-layer MLP is a composition of $L$ affine transformations interleaved with a fixed, non-linear transformation $\sigma(\cdot)$:

We recall the “learnable” parameters in an MLP are the parameters of the affine tranformation $(W_1, \bold b_1), \cdots (W_{L+1}, \bold b_{L+1})$.

While incredibly versatile, naively applying MLPs on their own turn out to be suboptimal for classifying images for various reasons, with the following as a primary reason. To pass an image through an MLP, it is flattened (a.k.a. vectorized) from a 2-D image (with 3 color channels) into a long vector. In the case of CIFAR-10, this turns into a $32\times 32 \times 3 = 3072$-dimensional input. A fully-connected MLP would then define a weight matrix mapping this 3072-dimensional vector into the next hidden layer. However, as humans we understand images often have local structure, e.g. most pixels are similar to the ones around them. This local structure is completely ignored by an MLP, which connects every input element to every neuron in the next layer. This causes MLPs on image recognition to suffer various issues, including:

• Parameter inefficiency: for most in-the-wild images, most pixels that are far away from each other often have little relevance to each other. Therefore, a fully-connected MLP may introduce many more weights than necessary.
• Sensitivity to translation: as humans, moving an object within the image frame does not affect our prediction. This is known as “translation invariance”. However, shifting an object makes it a completely different image to an MLP, and an MLP does not naturally encode this invariance. Therefore, an MLP can take a long time to reach satisfactory performance, and often remains brittle to otherwise very benign changes to an image.

This motivates a new type of neural network that addresses these issues unique to images. Enter: the Convolutional Neural Network (CNN). The CNN was designed precisely to address the above issues with standard MLPs. At the highest level, a CNN contains the following three types of layers:

1. Convolutional layer: filters patches of nearby pixels in a sliding window (where “convolution” comes from) and maps them to intermediate features.
2. Pooling layer: downsamples patches of pixels/features, e.g. sending a 2x2 patch to a single value. The value taken can either be the maximum of the downsampled patch (“max pooling”) or the average (“average pooling”).
3. Fully-connected layer (read: standard MLP). Once the features are of satisfactorily low-dimension after enough CNN and pooling layers, the features are flattened and MLP layers can be applied to produce the final output.

A standard CNN usually comprises of interleaved convolutional and pooling layers, followed by a sequence of fully-connected layers to produce the final output. The learnable parameters of the CNN are contained in the convolutional layers’ filter parameters as well as the MLP layers.

3.1What exactly is a convolution?¶

We introduce the mathematical basis for convolutional layers. We won’t sweat too much on the practical details, which are more appropriate for a deep learning course like ESE 5460. From the graphics above, one can gather that a convolution layer passes a filter (read: matrix) over sliding patches of the input and produces a (usually) smaller output.

A convolutional filter involves a few key parameters:

1. Filter size: the filter size determines how large a patch is considered at once, e.g. $3\times 3$ in the above graphic.
2. Stride: stride determines the step-size with which the filter moves across the input. A stride of 1 means the filter moves across the image one pixel at time, while 2 means the filter skips every other pixel. The stride of the above gif is 1, for example.
3. Padding: padding determines how the filter behaves at the edges of the image. For example, usually the corner pixels of the image can’t fit a filter, since imposing filter exceeds the bounds of the image. Valid padding (or “no padding”) means any pixel that can’t fit the filter is skipped like in the above gif, whereas same padding pads any out-of-bounds pixels of the filter with a dummy value (usually 0). Since valid padding means the filter can’t be applied to edge pixels (more severe the larger the filter), the output is guaranteed to be smaller, whereas same padding can ensure the output stays the same size as the input.

Now onto the math. Let’s consider a 2D image that is $n \times n$ pixels (no third color dimension!), and a convolutional filter of size $k \times k$, stride = 1, padding = 'valid':

Then the elements of the linear convolution (denoted by $\ast$) between $X$ and $K$ is:

Intuitively, each pixel in the output is the dot-product between the filter and the corresponding patch of the input image. To complete the convolutional layer and make it a deep learning object, we simply add a bias $B$ and entrywise nonlinearity $\sigma(\cdot)$:

The learnable parameters of a convolutional layer are therefore $(K,B)$.

We can also demonstrate the parameter efficiency of convolutional layers. The dimension of the conv layer parameters $(K,B)$ are:

If we had used a fully-connected MLP layer to map between the $n \times n$ input and $(n-2k)\times (n-2k)$ output, we come to a parameter count of

which is orders of magnitude more parameters than the conv layer. Even for the modest $32\times 32$ pixel CIFAR-10 images, this computes to $1024\times$ more parameters (not even considering the RGB depth dimension).

4Let it loose!¶

Now that we understand the fundamental mechanisms and benefits of a convolutional layer for vision/image-related tasks, let’s try it out! Fortunately for us, smart people have done the numerical heavylifting and made it very easy to build a CNN. Let’s construct a CNN composed of two iterations of convolution and pooling, followed by a two-layer MLP to bring us to the final output.

We now designate what loss and optimization algorithm to use. We will use the cross-entropy loss, which is a standard choice for multi-class classification problems, and stochastic gradient descent for the optimization algorithm. SGD takes two user-set hyperparameters: the learning rate (i.e. the step size) and momentum. Momentum determines to what extent we commit to the stochastic gradient direction. For example, if $\theta^{(k)}$ denotes our parameters at iteration $k$ and $d^{(k)}$ denotes the stochastic gradient direction, no momentum would yield

i.e. fully following the stochastic gradient. A momentum parameter of $\alpha \in (0,1)$ instead yields

in other words the previous iterate carries some “momentum”. Momentum is essential in machine learning contexts where the small batches relative to the dataset can yield highly stochastic iterates, preventing a model from committing too hard to a bad draw of SGD.

We won’t worry too much about the details behind cross-entropy loss, beyond the fact that penalizes predicting the wrong label.

Having defined our loss criterion and optimizer, we are almost ready to train our model. We now describe the essential ingredients of the training loop. The first moving part is the number of epochs, which is the number of times we want to iterate over the full training set when training the model.

Within each epoch, we perform the following sequence of operations iteratively:

1. Sample a new mini-batch of (images, labels)
2. Set the gradient from last iteration to zero
3. Pass the inputs through our current model
4. Compute the loss between the labels and the model outputs
5. Accumulate the stochastic gradient via backpropagation
6. Perform a model update via SGD (with momentum)

We are ready to run our first training loop.

Let’s plot the training loss of our model over the iterations of SGD.

Whoa Nelly! This becomes a familiar sight when doing deep learning. If you were wondering how optimizing over a model that is so explicitly non-convex and stochastic could be possible, it doesn’t come without its own hazards--as seen here, the loss can be very sensitive to small changes in the parameters. However, there is a clear overall trend down.

More importantly, did it learn anything useful? Let’s compute the accuracy attained on the test set.

Though a far cry from the human-tier of accuracy at 94%, our basic CNN does markedly better than the 10% for guessing at random! We also note that we used a small number of epochs for this demonstration, and the performance would likely improve if we had let our model train for longer.

Let’s take a peek at our model’s performance across the different classes.

And now for further visualization through our favorite confusion matrix.

Somewhat predictably, our CNN encounters the highest confusion when distinguishing between the various mammals.