# 2.2 残差网络（Residual Networks (ResNets)）

人为地让神经网络某些层跳过下一层神经元的连接，隔层相连，弱化每层之间的强联系，这种神经网络被称为**Residual Networks(ResNets)**

Residual Networks由许多隔层相连的神经元子模块组成，称之为Residual block（残差块）。单个Residual block的结构如下图所示：

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/f0a8471f869d8062ba59598c418da7fb.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit)

紫色线是skip connection（跳跃连接），直接建立$$a^{\[l]}$$与$$a^{\[l+2]}$$之间的隔层联系。相应的表达式如下：

$$
z^{\[l+1]}=W^{\[l+1]}a^{\[l]}+b^{\[l+1]}
$$

$$
a^{\[l+1]}=g(z^{\[l+1]})
$$

$$
z^{\[l+2]}=W^{\[l+2]}a^{\[l+1]}+b^{\[l+2]}
$$

$$
a^{\[l+2]}=g(z^{\[l+2]}+a^{\[l]})
$$

$$a^{\[l]}$$直接隔层与下一层的线性输出相连，$$a^{\[l]}$$插入的时机是在线性激活之后，**ReLU**激活之前，与$$z^{\[l+2]}$$共同通过激活函数（ReLU）输出$$a^{\[l+2]}$$

这种模型结构对于训练非常深的神经网络效果很好。非Residual Networks称为Plain Network

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/131e538bb527859430280becd65b049b.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit)

> Residual Network的结构

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/48bded15cca17581084e3fe0853673b5.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit)

> Plain Network

与Plain Network相比，Residual Network能够训练更深层的神经网络，有效避免发生发生梯度消失和梯度爆炸

* 随着神经网络层数增加，Plain Network实际性能会变差，training error甚至会变大
* Residual Network的训练效果却很好，training error一直呈下降趋势

[\
![](https://github.com/fengdu78/deeplearning_ai_books/raw/master/images/6077958a616425d76284cecb43c2f458.png)](https://legacy.gitbook.com/book/baozou/neural-networks-and-deep-learning/edit)