# 1.11 神经网络的权重初始化（Weight Initialization for Deep Networks）

深度神经网络模型中，以单个神经元为例，该层（$$l$$）的输入个数为$$n$$，其输出为：

$$
z=w\_1x\_1+w\_2x\_2+\cdots+w\_nx\_n
$$

$$
a=g(z)
$$

![](/files/-Le0cdOaU55-K_SUCzPo)

> 忽略了常数项b

为了让$$z$$不会过大或者过小，$$w$$应该越小才好。方法是在初始化$$w$$时，令其方差为$$\frac{1}{n}$$

激活函数是$$tanh$$相应的python伪代码为：

```python
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(1/n[l-1])
```

如果激活函数是$$ReLU$$，权重$$w$$的初始化一般令其方差为$$\frac{2}{n}$$：

```python
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(2/n[l-1])
```

另外一种初始化$$w$$的方法，令其方差为$$\frac{2}{n^{\[l-1]}+n^{\[l]}}$$：

```c
w[l] = np.random.randn(n[l],n[l-1])*np.sqrt(2/(n[l-1] + n[l]))
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://baozoulin.gitbook.io/neural-networks-and-deep-learning/di-er-men-ke-gai-shan-shen-ceng-shen-jing-wang-luo-chao-can-shu-tiao-shi-zheng-ze-hua-yi-ji-you-hua/improving-deep-neural-networks/practical-aspects-of-deep-learning/111-shen-jing-wang-luo-de-quan-zhong-chu-shi-hua-ff08-weight-initialization-for-deep-networks.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.