In this video you can see a comparison between different optimizers and learning rates.
The input are two images, “image” and “msg”.
A randomly initialized variable tensor with the same dimensions as the inputs is altered to minimize the two losses.
Both losses are the mean squared error(mse) of the difference between the output and the input tensor.
The “image” loss i simply the mse, comparing the “image” input to the variable tensor.
The “msg” loss is the mse comparison of the variable tensor that has undergone a 2d convolution with three 5×5 kernels, with the “msg” input.
The “image” loss only alters the variable tensor, but the “msg” loss also optimizes the 2d conv kernels.