# 2.2 残差网络（Residual Networks (ResNets)）

人为地让神经网络某些层跳过下一层神经元的连接，隔层相连，弱化每层之间的强联系，这种神经网络被称为**Residual Networks(ResNets)**

Residual Networks由许多隔层相连的神经元子模块组成，称之为Residual block（残差块）。单个Residual block的结构如下图所示：

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/f0a8471f869d8062ba59598c418da7fb.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit#)

紫色线是skip connection（跳跃连接），直接建立$$a^{\[l]}$$与$$a^{\[l+2]}$$之间的隔层联系。相应的表达式如下：

$$
z^{\[l+1]}=W^{\[l+1]}a^{\[l]}+b^{\[l+1]}
$$

$$
a^{\[l+1]}=g(z^{\[l+1]})
$$

$$
z^{\[l+2]}=W^{\[l+2]}a^{\[l+1]}+b^{\[l+2]}
$$

$$
a^{\[l+2]}=g(z^{\[l+2]}+a^{\[l]})
$$

$$a^{\[l]}$$直接隔层与下一层的线性输出相连，$$a^{\[l]}$$插入的时机是在线性激活之后，**ReLU**激活之前，与$$z^{\[l+2]}$$共同通过激活函数（ReLU）输出$$a^{\[l+2]}$$

这种模型结构对于训练非常深的神经网络效果很好。非Residual Networks称为Plain Network

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/131e538bb527859430280becd65b049b.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit#)

> Residual Network的结构

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/48bded15cca17581084e3fe0853673b5.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit#)

> Plain Network

与Plain Network相比，Residual Network能够训练更深层的神经网络，有效避免发生发生梯度消失和梯度爆炸

* 随着神经网络层数增加，Plain Network实际性能会变差，training error甚至会变大
* Residual Network的训练效果却很好，training error一直呈下降趋势

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/6077958a616425d76284cecb43c2f458.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit#)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://baozoulin.gitbook.io/neural-networks-and-deep-learning/di-si-men-ke-juan-ji-shen-jing-wang-luo-convolutional-neural-networks/convolutional-neural-networks/deep-convolutional-models-case-studies/23-can-cha-wang-luo-ff08-residual-networks-resnets.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.