<!DOCTYPE html>
Using the logs dictionary¶
In this reading, we will learn how to take advantage of the logs dictionary in Keras to define our own callbacks and check the progress of a model.
import tensorflow as tf
print(tf.__version__)
The logs dictionary stores the loss value, along with all of the metrics we are using at the end of a batch or epoch.
We can incorporate information from the logs dictionary into our own custom callbacks.
Let's see this in action in the context of a model we will construct and fit to the sklearn diabetes dataset that we have been using in this module.
Let's first import the dataset, and split it into the training and test sets.
# Load the diabetes dataset
from sklearn.datasets import load_diabetes
diabetes_dataset = load_diabetes()
# Save the input and target variables
from sklearn.model_selection import train_test_split
data = diabetes_dataset['data']
targets = diabetes_dataset['target']
# Split the data set into training and test sets
train_data, test_data, train_targets, test_targets = train_test_split(data, targets, test_size=0.1)
Now we construct our model.
# Build the model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = tf.keras.Sequential([
Dense(128, activation='relu', input_shape=(train_data.shape[1],)),
Dense(64,activation='relu'),
tf.keras.layers.BatchNormalization(),
Dense(64, activation='relu'),
Dense(64, activation='relu'),
Dense(1)
])
We now compile the model, with
- Mean squared error as the loss function,
- the Adam optimizer, and
- Mean absolute error (
mae) as a metric.
# Compile the model
model.compile(loss='mse', optimizer="adam", metrics=['mae'])
Defining a custom callback¶
Now we define our custom callback using the logs dictionary to access the loss and metric values.
# Create the custom callback
class LossAndMetricCallback(tf.keras.callbacks.Callback):
# Print the loss after every second batch in the training set
def on_train_batch_end(self, batch, logs=None):
if batch %2 ==0:
print('\n After batch {}, the loss is {:7.2f}.'.format(batch, logs['loss']))
# Print the loss after each batch in the test set
def on_test_batch_end(self, batch, logs=None):
print('\n After batch {}, the loss is {:7.2f}.'.format(batch, logs['loss']))
# Print the loss and mean absolute error after each epoch
def on_epoch_end(self, epoch, logs=None):
print('Epoch {}: Average loss is {:7.2f}, mean absolute error is {:7.2f}.'.format(epoch, logs['loss'], logs['mae']))
# Notify the user when prediction has finished on each batch
def on_predict_batch_end(self,batch, logs=None):
print("Finished prediction on batch {}!".format(batch))
We now fit the model to the data, and specify that we would like to use our custom callback LossAndMetricCallback().
# Train the model
history = model.fit(train_data, train_targets, epochs=20,
batch_size=100, callbacks=[LossAndMetricCallback()], verbose=False)
We can also use our callback in the evaluate function...
# Evaluate the model
model_eval = model.evaluate(test_data, test_targets, batch_size=10,
callbacks=[LossAndMetricCallback()], verbose=False)
...And also the predict function.
# Get predictions from the model
model_pred = model.predict(test_data, batch_size=10,
callbacks=[LossAndMetricCallback()], verbose=False)
Application - learning rate scheduler¶
Let's now look at a more sophisticated custom callback.
We are going to define a callback to change the learning rate of the optimiser of a model during training. We will do this by specifying the epochs and new learning rates where we would like it to be changed.
First we define the auxillary function that returns the learning rate for each epoch based on our schedule.
# Define the learning rate schedule. The tuples below are (start_epoch, new_learning_rate)
lr_schedule = [
(4, 0.03), (7, 0.02), (11, 0.005), (15, 0.007)
]
def get_new_epoch_lr(epoch, lr):
# Checks to see if the input epoch is listed in the learning rate schedule
# and if so, returns index in lr_schedule
epoch_in_sched = [i for i in range(len(lr_schedule)) if lr_schedule[i][0]==int(epoch)]
if len(epoch_in_sched)>0:
# If it is, return the learning rate corresponding to the epoch
return lr_schedule[epoch_in_sched[0]][1]
else:
# Otherwise, return the existing learning rate
return lr
Let's now define the callback itself.
# Define the custom callback
class LRScheduler(tf.keras.callbacks.Callback):
def __init__(self, new_lr):
super(LRScheduler, self).__init__()
# Add the new learning rate function to our callback
self.new_lr = new_lr
def on_epoch_begin(self, epoch, logs=None):
# Make sure that the optimizer we have chosen has a learning rate, and raise an error if not
if not hasattr(self.model.optimizer, 'lr'):
raise ValueError('Error: Optimizer does not have a learning rate.')
# Get the current learning rate
curr_rate = float(tf.keras.backend.get_value(self.model.optimizer.lr))
# Call the auxillary function to get the scheduled learning rate for the current epoch
scheduled_rate = self.new_lr(epoch, curr_rate)
# Set the learning rate to the scheduled learning rate
tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_rate)
print('Learning rate for epoch {} is {:7.3f}'.format(epoch, scheduled_rate))
Let's now train the model again with our new callback.
# Build the same model as before
new_model = tf.keras.Sequential([
Dense(128, activation='relu', input_shape=(train_data.shape[1],)),
Dense(64,activation='relu'),
tf.keras.layers.BatchNormalization(),
Dense(64, activation='relu'),
Dense(64, activation='relu'),
Dense(1)
])
# Compile the model
new_model.compile(loss='mse',
optimizer="adam",
metrics=['mae', 'mse'])
# Fit the model with our learning rate scheduler callback
new_history = new_model.fit(train_data, train_targets, epochs=20,
batch_size=100, callbacks=[LRScheduler(get_new_epoch_lr)], verbose=False)