<!DOCTYPE html>
Capstone Project¶
Image classifier for the SVHN dataset¶
Instructions¶
In this notebook, you will create a neural network that classifies real-world images digits. You will use concepts from throughout this course in building, training, testing, validating and saving your Tensorflow classifier model.
This project is peer-assessed. Within this notebook you will find instructions in each section for how to complete the project. Pay close attention to the instructions as the peer review will be carried out according to a grading rubric that checks key parts of the project instructions. Feel free to add extra cells into the notebook as required.
How to submit¶
When you have completed the Capstone project notebook, you will submit a pdf of the notebook for peer review. First ensure that the notebook has been fully executed from beginning to end, and all of the cell outputs are visible. This is important, as the grading rubric depends on the reviewer being able to view the outputs of your notebook. Save the notebook as a pdf (File -> Download as -> PDF via LaTeX). You should then submit this pdf for review.
Let's get started!¶
We'll start by running some imports, and loading the dataset. For this project you are free to make further imports throughout the notebook as you wish.
In [27]:
import tensorflow as tf
from scipy.io import loadmat
import matplotlib.pyplot as plt
import numpy as np
For the capstone project, you will use the SVHN dataset. This is an image dataset of over 600,000 digit images in all, and is a harder dataset than MNIST as the numbers appear in the context of natural scene images. SVHN is obtained from house numbers in Google Street View images.
- Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu and A. Y. Ng. "Reading Digits in Natural Images with Unsupervised Feature Learning". NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Your goal is to develop an end-to-end workflow for building, training, validating, evaluating and saving a neural network that classifies a real-world image into one of ten classes.
In [28]:
# Run this cell to load the dataset
train = loadmat('data/train_32x32.mat')
test = loadmat('data/test_32x32.mat')
Both train and test are dictionaries with keys X and y for the input images and labels respectively.
In [29]:
train["X"][0].shape
Out[29]:
In [30]:
test["y"].shape
Out[30]:
In [31]:
a = np.where(test["y"]<10, test["y"], 0)
a = np.unique(a, return_counts=True)
a
Out[31]:
1. Inspect and preprocess the dataset¶
- Extract the training and testing images and labels separately from the train and test dictionaries loaded for you.
- Select a random sample of images and corresponding labels from the dataset (at least 10), and display them in a figure.
- Convert the training and test images to grayscale by taking the average across all colour channels for each pixel. Hint: retain the channel dimension, which will now have size 1.
- Select a random sample of the grayscale images and corresponding labels from the dataset (at least 10), and display them in a figure.
In [32]:
train_data = np.transpose(train["X"], (3,0,1,2))
train_targets = np.where(train["y"]<10, train["y"], 0)
test_data = np.transpose(test["X"], (3,0,1,2))
test_targets = np.where(test["y"]<10, test["y"], 0)
In [33]:
fig=plt.figure(figsize=(32, 32))
columns = 10
rows = 1
for i in range(1, columns*rows +1):
fig.add_subplot(rows, columns, i)
plt.imshow(train_data[i])
plt.axis("off")
plt.show()
In [34]:
train_data_gray = np.mean(train_data, axis=-1)
test_data_gray = np.mean(test_data, axis=-1)
In [35]:
fig=plt.figure(figsize=(32, 32))
columns = 10
rows = 1
for i in range(1, columns*rows +1):
fig.add_subplot(rows, columns, i)
plt.imshow(train_data_gray[i])
plt.axis("off")
plt.show()
2. MLP neural network classifier¶
- Build an MLP classifier model using the Sequential API. Your model should use only Flatten and Dense layers, with the final layer having a 10-way softmax output.
- You should design and build the model yourself. Feel free to experiment with different MLP architectures. Hint: to achieve a reasonable accuracy you won't need to use more than 4 or 5 layers.
- Print out the model summary (using the summary() method)
- Compile and train the model (we recommend a maximum of 30 epochs), making use of both training and validation sets during the training run.
- Your model should track at least one appropriate metric, and use at least two callbacks during training, one of which should be a ModelCheckpoint callback.
- As a guide, you should aim to achieve a final categorical cross entropy training loss of less than 1.0 (the validation loss might be higher).
- Plot the learning curves for loss vs epoch and accuracy vs epoch for both training and validation sets.
- Compute and display the loss and accuracy of the trained model on the test set.
In [36]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense
In [37]:
def get_new_model(input_shape):
model = Sequential([
Flatten(input_shape=input_shape),
Dense(units=128, activation="relu"),
Dense(units=128, activation="relu"),
Dense(units=128, activation="relu"),
Dense(units=64, activation="relu"),
Dense(units=64, activation="relu"),
Dense(units=64, activation="relu"),
Dense(units=10,use_bias=True, activation="softmax",bias_initializer='ones')
])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",metrics=["acc"])
return model
In [38]:
mlp_model = get_new_model(train_data_gray[0].shape)
mlp_model.summary()
In [39]:
checkpoint_best_path = "model_checkpoint_best/checkpoint"
checkpoint_best = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_best_path,
save_weights_only=True,
save_best_only=True,
save_freq="epoch",
verbose=2
)
earlystoping = tf.keras.callbacks.EarlyStopping(min_delta=0.001,patience=5)
class training_callback(tf.keras.callbacks.Callback):
def on_train_begin(self, logs=None):
print("Start training....")
def on_train_end(self, logs=None):
print("Finished training!")
In [40]:
history = mlp_model.fit(train_data_gray, train_targets, validation_split=0.2, epochs=30, batch_size=512, verbose=2,
callbacks=[checkpoint_best,earlystoping, training_callback()])
In [41]:
import pandas as pd
In [42]:
df = pd.DataFrame(history.history)
df.plot(y=["loss","val_loss"])
df.plot(y=["acc","val_acc"])
Out[42]:
In [43]:
mlp_model.evaluate(test_data_gray, test_targets, verbose=0)
Out[43]:
In [44]:
plt.imshow(test_data_gray[1024])
Out[44]:
In [45]:
np.argmax(mlp_model.predict(test_data_gray[1024][np.newaxis,...]))
Out[45]:
3. CNN neural network classifier¶
- Build a CNN classifier model using the Sequential API. Your model should use the Conv2D, MaxPool2D, BatchNormalization, Flatten, Dense and Dropout layers. The final layer should again have a 10-way softmax output.
- You should design and build the model yourself. Feel free to experiment with different CNN architectures. Hint: to achieve a reasonable accuracy you won't need to use more than 2 or 3 convolutional layers and 2 fully connected layers.)
- The CNN model should use fewer trainable parameters than your MLP model.
- Compile and train the model (we recommend a maximum of 30 epochs), making use of both training and validation sets during the training run.
- Your model should track at least one appropriate metric, and use at least two callbacks during training, one of which should be a ModelCheckpoint callback.
- You should aim to beat the MLP model performance with fewer parameters!
- Plot the learning curves for loss vs epoch and accuracy vs epoch for both training and validation sets.
- Compute and display the loss and accuracy of the trained model on the test set.
In [46]:
from tensorflow.keras.layers import Conv2D, MaxPool2D, BatchNormalization, Flatten, Dense, Dropout
In [47]:
def get_cnn_model(input_shape,wd,reg):
model = Sequential([
Conv2D(filters=16, kernel_size=(3,3),activation="relu",input_shape=input_shape,kernel_regularizer=tf.keras.regularizers.l2(reg)),
Conv2D(filters=8, kernel_size=(3,3), activation="relu"),
MaxPool2D(pool_size=(3,3)),
Dropout(wd),
Dense(units=32, activation="relu"),
BatchNormalization(),
Dense(units=32, activation="relu"),
Flatten(),
Dense(units=10, activation="softmax")
])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",metrics=["acc"])
return model
In [48]:
cnn_model = get_cnn_model(train_data_gray[0][...,np.newaxis].shape, 0.5, 0.01)
cnn_model.summary()
In [49]:
checkpoint_best_cnn_path = "model_checkpoint_best_cnn/checkpoint"
checkpoint_best = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_best_cnn_path,
save_weights_only=True,
save_best_only=True,
save_freq="epoch",
verbose=2
)
earlystoping = tf.keras.callbacks.EarlyStopping(min_delta=0.001,patience=5)
class training_callback(tf.keras.callbacks.Callback):
def on_train_begin(self, logs=None):
print("Start training....")
def on_train_end(self, logs=None):
print("Finished training!")
In [50]:
history = cnn_model.fit(train_data_gray[...,np.newaxis], train_targets, validation_split=0.2, epochs=30, batch_size=512, verbose=1,
callbacks=[checkpoint_best, earlystoping, training_callback()])
In [51]:
df = pd.DataFrame(history.history)
df.plot(y=["loss","val_loss"])
df.plot(y=["acc","val_acc"])
Out[51]:
In [ ]:
In [52]:
del mlp_model, cnn_model
4. Get model predictions¶
- Load the best weights for the MLP and CNN models that you saved during the training run.
- Randomly select 5 images and corresponding labels from the test set and display the images with their labels.
- Alongside the image and label, show each model’s predictive distribution as a bar chart, and the final model prediction given by the label with maximum probability.
In [53]:
mlp_model = get_new_model(train_data_gray[0].shape)
cnn_model = get_cnn_model(train_data_gray[0][...,np.newaxis].shape, 0.5, 0.001)
In [54]:
mlp_model.load_weights("model_checkpoint_best/checkpoint")
mlp_model.summary()
In [55]:
cnn_model.load_weights("model_checkpoint_best_cnn/checkpoint")
cnn_model.summary()
In [59]:
random_idx = np.random.choice(test_data.shape[0], 5)
random_test_data_gray = test_data_gray[random_idx, ...]
random_test_data = test_data_gray[random_idx, ...]
random_test_targets = test_targets[random_idx,...]
In [60]:
mlp_predictions = mlp_model.predict(random_test_data_gray)
fig, axes = plt.subplots(5, 2, figsize=(16, 12))
fig.subplots_adjust(hspace=0.4,wspace=-0.2)
for i, (prediction, image, label) in enumerate(zip(mlp_predictions, random_test_data, random_test_targets)):
axes[i, 0].imshow(np.squeeze(image))
axes[i, 0].get_xaxis().set_visible(False)
axes[i, 0].get_yaxis().set_visible(False)
axes[i, 0].text(10, -1.5, f"Digit {label}")
axes[i, 1].bar(np.arange(0, 10), prediction)
axes[i, 1].set_xticks(np.arange(0, 10))
axes[i, 1].set_title("Categorical distribution. Model prediction")
plt.show()
In [61]:
cnn_predictions = cnn_model.predict(random_test_data_gray[...,np.newaxis])
fig, axes = plt.subplots(5, 2, figsize=(16, 12))
fig.subplots_adjust(hspace=0.4,wspace=-0.2)
for i, (prediction, image, label) in enumerate(zip(cnn_predictions, random_test_data, random_test_targets)):
axes[i, 0].imshow(np.squeeze(image))
axes[i, 0].get_xaxis().set_visible(False)
axes[i, 0].get_yaxis().set_visible(False)
axes[i, 0].text(10, -1.5, f"Digit {label}")
axes[i, 1].bar(np.arange(0, 10), prediction)
axes[i, 1].set_xticks(np.arange(0, 10))
axes[i, 1].set_title("Categorical distribution. Model prediction")
plt.show()
In [ ]: