# 2.9 学习率衰减(Learning rate decay)

减小学习因子$$\alpha$$也能有效提高神经网络训练速度，这种方法被称为learning rate decay, Learning rate decay就是随着迭代次数增加，学习因子$$\alpha$$逐渐减小

下图中，蓝色折线表示使用恒定的学习因子$$\alpha$$，由于每次训练$$\alpha$$相同，步进长度不变，在接近最优值处的振荡也大，在最优值附近较大范围内振荡，与最优值距离就比较远。绿色折线表示使用不断减小的$$\alpha$$，随着训练次数增加，$$\alpha$$逐渐减小，步进长度减小，使得能够在最优值处较小范围内微弱振荡，不断逼近最优值。相比较恒定的$$\alpha$$来说，learning rate decay更接近最优值

![](/files/-Le0caaNPHjB4Z1ydzei)

Learning rate decay中对$$\alpha$$的公式：

$$
\alpha=\frac{1}{1+decay\_rate\*epoch}\alpha\_0
$$

deacy\_rate是参数（可调），epoch是迭代次数。随着epoch增加，$$\alpha$$会不断变小

其它计算公式：

$$
\alpha=0.95^{epoch}\cdot \alpha\_0
$$

$$
\alpha=\frac{k}{\sqrt{epoch}}\cdot \alpha\_0\ \ \ \ or\ \ \ \ \frac{k}{\sqrt{t}}\cdot \alpha\_0
$$

$$k$$为可调参数，$$t$$为mini-bach number

还可以设置$$\alpha$$为关于$$t$$的离散值，随着$$t$$增加，$$\alpha$$呈阶梯式减小。也可以根据训练情况灵活调整当前的$$\alpha$$值，但会比较耗时间


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://baozoulin.gitbook.io/neural-networks-and-deep-learning/di-er-men-ke-gai-shan-shen-ceng-shen-jing-wang-luo-chao-can-shu-tiao-shi-zheng-ze-hua-yi-ji-you-hua/improving-deep-neural-networks/optimization-algorithms/29-xue-xi-lv-shuai-51cf28-learning-rate-decay.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.