Edit a TensorFlow training model for hyperparameter tuning

Before uploading a TensorFlow training model, edit the model to work with the hyperparameter tuning insights feature in IBM Spectrum Conductor Deep Learning Impact.

1. Make sure your model imports the hyperparameter tuning package.
   For example:

   import tf_parameter_mgr
   import monitor_cb
   from monitor_cb import CMonitor
   

2. Make sure to get the learning rate operator and optimizer.
   For example:

   def get_train_op(total_loss, global_step, return_grad=False):
       lr = tf_parameter_mgr.getLearningRate(global_step)
       # Compute gradients.
       opt = tf_parameter_mgr.getOptimizer(lr)
       grads = opt.compute_gradients(total_loss)
       # Apply gradients.
       apply_gradient_op = opt.apply_gradients(grads, global_step=global_step)
       with tf.control_dependencies([apply_gradient_op] + tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
           train_op = tf.no_op(name='train')
       if return_grad:
           return apply_gradient_op, grads
       return train_op
   

3. To return the loss value from the model to the hyperparameter tuning optimizer, the total_loss of the test dataset needs to be added to the summary and periodically output to the summary log file.
   For example:

   monitor.SummaryScalar('train loss', total_loss)
   monitor.SummaryScalar('test loss', total_loss)