Initialization

Welcome to the first assignment of "Improving Deep Neural Networks".

Training your neural network requires specifying an initial value of the weights. A well chosen initialization method will help learning.

If you completed the previous course of this specialization, you probably followed our instructions for weight initialization, and it has worked out so far. But how do you choose the initialization for a new neural network? In this notebook, you will see how different initializations lead to different results.

A well chosen initialization can:

  • Speed up the convergence of gradient descent

  • Increase the odds of gradient descent converging to a lower training (and generalization) error

To get started, run the following cell to load the packages and the planar dataset you will try to classify.

import numpy as np
import matplotlib.pyplot as plt
import sklearn
import sklearn.datasets
from init_utils import sigmoid, relu, compute_loss, forward_propagation, backward_propagation
from init_utils import update_parameters, predict, load_dataset, plot_decision_boundary, predict_dec

%matplotlib inline
plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# load image dataset: blue/red dots in circles
train_X, train_Y, test_X, test_Y = load_dataset()

You would like a classifier to separate the blue dots from the red dots.

1 - Neural Network model

You will use a 3-layer neural network (already implemented for you). Here are the initialization methods you will experiment with:

  • Zeros initialization

    -- setting

    initialization = "zeros"

    in the input argument.

  • Random initialization

    -- setting

    initialization = "random"

    in the input argument. This initializes the weights to large random values.

  • He initialization

    -- setting

    initialization = "he"

    in the input argument. This initializes the weights to random values scaled according to a paper by He et al., 2015.

Instructions: Please quickly read over the code below, and run it. In the next part you will implement the three initialization methods that thismodel()calls.

2 - Zero initialization

There are two types of parameters to initialize in a neural network:

  • the weight matrices(W[1],W[2],W[3],...,W[L−1],W[L])(W^{[1]}, W^{[2]}, W^{[3]}, ..., W^{[L-1]}, W^{[L]})

  • the bias vectors(b[1],b[2],b[3],...,b[L−1],b[L])(b^{[1]}, b^{[2]}, b^{[3]}, ..., b^{[L-1]}, b^{[L]})

Exercise: Implement the following function to initialize all parameters to zeros. You'll see later that this does not work well since it fails to "break symmetry", but lets try it anyway and see what happens. Use np.zeros((..,..)) with the correct shapes.

The performance is really bad, and the cost does not really decrease, and the algorithm performs no better than random guessing. Why? Lets look at the details of the predictions and the decision boundary:

The model is predicting 0 for every example.

In general, initializing all the weights to zero results in the network failing to break symmetry. This means that every neuron in each layer will learn the same thing, and you might as well be training a neural network withn[l]=1n[l]=1for every layer, and the network is no more powerful than a linear classifier such as logistic regression.

What you should remember:

The weightsW[l]W^{[l]}should be initialized randomly to break symmetry.

It is however okay to initialize the biaseb[l]b^{[l]}to zeros. Symmetry is still broken so long asW[l]W^{[l]}is initialized randomly.

3 - Random initialization

To break symmetry, lets intialize the weights randomly. Following random initialization, each neuron can then proceed to learn a different function of its inputs. In this exercise, you will see what happens if the weights are intialized randomly, but to very large values.

Exercise: Implement the following function to initialize your weights to large random values (scaled by *10) and your biases to zeros. Usenp.random.randn(..,..) * 10for weights andnp.zeros((.., ..))for biases. We are using a fixednp.random.seed(..)to make sure your "random" weights match ours, so don't worry if running several times your code gives you always the same initial values for the parameters.

Run the following code to train your model on 15,000 iterations using random initialization.

If you see "inf" as the cost after the iteration 0, this is because of numerical roundoff; a more numerically sophisticated implementation would fix this. But this isn't worth worrying about for our purposes.

Anyway, it looks like you have broken symmetry, and this gives better results. than before. The model is no longer outputting all 0s.

Observations:

  • The cost starts very high. This is because with large random-valued weights, the last activation (sigmoid) outputs results that are very close to 0 or 1 for some examples, and when it gets that example wrong it incurs a very high loss for that example. Indeed, whenlog⁡(a[3])=log⁡(0)\log(a^{[3]}) = \log(0), the loss goes to infinity.

  • Poor initialization can lead to vanishing/exploding gradients, which also slows down the optimization algorithm.

  • If you train this network longer you will see better results, but initializing with overly large random numbers slows down the optimization.

In summary:

  • Initializing weights to very large random values does not work well.

  • Hopefully intializing with small random values does better. The important question is: how small should be these random values be? Lets find out in the next part!

4 - He initialization

Finally, try "He Initialization"; this is named for the first author of He et al., 2015. (If you have heard of "Xavier initialization", this is similar except Xavier initialization uses a scaling factor for the weightsW[l]W^{[l]}ofsqrt(1./layers_dims[l-1])where He initialization would usesqrt(2./layers_dims[l-1]).)

Exercise: Implement the following function to initialize your parameters with He initialization.

Hint: This function is similar to the previousinitialize_parameters_random(...). The only difference is that instead of multiplyingnp.random.randn(..,..)by 10, you will multiply it by2dimension of the previous layer\sqrt{\frac{2}{\text{dimension of the previous layer}}}, which is what He initialization recommends for layers with a ReLU activation.

Run the following code to train your model on 15,000 iterations using He initialization.

Observations:

  • The model with He initialization separates the blue and the red dots very well in a small number of iterations.

5 - Conclusions

You have seen three different types of initializations. For the same number of iterations and same hyperparameters the comparison is:

Model

Train accuracy

Problem/Comment

3-layer NN with zeros initialization

50%

fails to break symmetry

3-layer NN with large random initialization

83%

too large weights

3-layer NN with He initialization

99%

recommended method

What you should remember from this notebook:

  • Different initializations lead to different results

  • Random initialization is used to break symmetry and make sure different hidden units can learn different things

  • Don't intialize to values that are too large

  • He initialization works well for networks with ReLU activations.

Previous1.14 梯度检验应用的注意事项（Gradient Checking Implementation Notes）NextGradient Checking

Last updated

Was this helpful?