> For the complete documentation index, see [llms.txt](https://baozoulin.gitbook.io/neural-networks-and-deep-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://baozoulin.gitbook.io/neural-networks-and-deep-learning/di-er-men-ke-gai-shan-shen-ceng-shen-jing-wang-luo-chao-can-shu-tiao-shi-zheng-ze-hua-yi-ji-you-hua/improving-deep-neural-networks/practical-aspects-of-deep-learning/111-shen-jing-wang-luo-de-quan-zhong-chu-shi-hua-ff08-weight-initialization-for-deep-networks.md).

# 1.11 神经网络的权重初始化（Weight Initialization for Deep Networks）

深度神经网络模型中，以单个神经元为例，该层（$$l$$）的输入个数为$$n$$，其输出为：

$$
z=w\_1x\_1+w\_2x\_2+\cdots+w\_nx\_n
$$

$$
a=g(z)
$$

![](/files/-Le0cdOaU55-K_SUCzPo)

> 忽略了常数项b

为了让$$z$$不会过大或者过小，$$w$$应该越小才好。方法是在初始化$$w$$时，令其方差为$$\frac{1}{n}$$

激活函数是$$tanh$$相应的python伪代码为：

```python
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(1/n[l-1])
```

如果激活函数是$$ReLU$$，权重$$w$$的初始化一般令其方差为$$\frac{2}{n}$$：

```python
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(2/n[l-1])
```

另外一种初始化$$w$$的方法，令其方差为$$\frac{2}{n^{\[l-1]}+n^{\[l]}}$$：

```c
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(2/(n[l-1] + n[l]))
```