2.4 逻辑回归的梯度下降（Logistic Regression Gradient Descent）

对单个样本而言，逻辑回归Loss function表达式如下：

$$z=w^Tx+b$$

$$\hat y=a=\sigma(z)$$

$$L(a,y)=-(y\log(a)+(1-y)\log(1-a))$$

计算该逻辑回归的反向传播过程:

$$da=\frac{\partial L}{\partial a}=-\frac ya+\frac{1-y}{1-a}$$

$$dz=\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\cdot \frac{\partial a}{\partial z}=(-\frac ya+\frac{1-y}{1-a})\cdot a(1-a)=a-y$$

$$dw_1=\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial z}\cdot \frac{\partial z}{\partial w_1}=x_1\cdot dz=x_1(a-y)$$

$$dw_2=\frac{\partial L}{\partial w_2}=\frac{\partial L}{\partial z}\cdot \frac{\partial z}{\partial w_2}=x_2\cdot dz=x_2(a-y)$$

$$db=\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z}\cdot \frac{\partial z}{\partial b}=1\cdot dz=a-y$$

则梯度下降算法可表示为：

$$w_1:=w_1-\alpha\ dw_1$$

$$w_2:=w_2-\alpha\ dw_2$$

$$b:=b-\alpha\ db$$