线性回归/11. 梯度下降的简单例子.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据前面介绍的梯度，我们来看几个使用梯度下降法求解极值的例子。对于函数 $J(\\theta)$，我们首先选取一个初始点 $\\theta_0$，然后沿着梯度方向对初始值进行更新得到 $\\theta_1$：\n",
    "\n",
    "$$\n",
    "\\theta_1 = \\theta_0 - \\eta \\frac{\\partial J(\\theta)}{\\partial \\theta}\n",
    "$$\n",
    "\n",
    "然后再对 $\\theta_1$ 进行更新得到 $\\theta_2$，以此类推，直到损失函数的值不再有显著变化，最终得到 $\\theta_k$ 就是我们要求的参数，它使损失函数达到最小。其中 $\\eta$ 是一个待确定的常数，通常被称为 **学习率**（learning rate）或**步长**（step），它是一个超参数。\n",
    "\n",
    "上面的迭代过程一般写成下面的形式，其中的 $:=$ 表示使用右侧的值更新左侧的值：\n",
    "\n",
    "$$\n",
    "\\theta := \\theta - \\eta \\frac{\\partial J(\\theta)}{\\partial \\theta}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一元函数的梯度下降\n",
    "\n",
    "我们首先看一个最简单的一元函数：\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\theta^2\n",
    "$$\n",
    "\n",
    "求一元函数的梯度，也就对其求导：\n",
    "\n",
    "$$\n",
    "\\nabla J(\\theta) = J'(\\theta) = 2\\theta\n",
    "$$\n",
    "\n",
    "给 $\\theta$ 一个初始值 $\\theta_0 = 1$，并设学习率 $\\eta = 0.4$，根据梯度下降的更新公式，我们有：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\theta_0 &= 1 \\\\\n",
    "\\theta_1 &= \\theta_0 - \\eta \\nabla J(\\theta_0) = 1 - 0.4 * (2*1) = 0.2 \\\\\n",
    "\\theta_2 &= \\theta_1 - \\eta \\nabla J(\\theta_1) = 0.2 - 0.4 * (2*0.2) = 0.04 \\\\\n",
    "\\theta_3 &= 0.008 \\\\\n",
    "\\theta_4 &= 0.0016 \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "经过4步，基本上已经到达函数的最低点了。\n",
    "\n",
    "![](../images/gradient-sample-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 二元函数的梯度下降\n",
    "\n",
    "我们再来看一个二元函数：\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\theta_1^2 + \\theta_2^2\n",
    "$$\n",
    "\n",
    "该函数的梯度为：\n",
    "\n",
    "$$\n",
    "\\nabla J(\\theta) = \\langle 2\\theta_1, 2\\theta_2 \\rangle\n",
    "$$\n",
    "\n",
    "给 $\\theta$ 一个初始值 $\\theta^0 = \\langle 1,3 \\rangle$，并设学习率 $\\eta = 0.1$，开始迭代更新：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\theta^0 &= \\langle 1,3 \\rangle \\\\\n",
    "\\theta^1 &= \\theta^0 - \\eta \\nabla J(\\theta^0) = \\langle 1,3 \\rangle - 0.1*\\langle 2,6 \\rangle = \\langle 0.8,2.4 \\rangle \\\\\n",
    "\\theta^2 &= \\theta^1 - \\eta \\nabla J(\\theta^1) = \\langle 0.8,2.4 \\rangle - 0.1*\\langle 1.6,4.8 \\rangle = \\langle 0.64,1.92 \\rangle\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "可以写个程序来自动完成迭代过程，可以发现经过若干步的迭代，函数也达到了最低点。\n",
    "\n",
    "![](../images/gradient-sample-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 关于学习率\n",
    "\n",
    "梯度下降算法中的 $\\eta$ 称为 **学习率**（或者 **步长**），它是一个超参数，关于它的值，不宜太大，也不宜太小。学习率用于控制每一步走的距离，如果太小，参数更新每次也很小，梯度下降的速度会非常慢，如果太大，参数更新也会很大，可能会直接越过极小值，甚至无法收敛到达最低点。\n",
    "\n",
    "![](../images/learning-rate.png)\n",
    "\n",
    "### 参考\n",
    "\n",
    "* [深入浅出--梯度下降法及其实现 - 简书](https://www.jianshu.com/p/c7e642877b0e)\n",
    "* [Gradient Descent - Problem of Hiking Down a Mountain | Udacity](https://storage.googleapis.com/supplemental_media/udacityu/315142919/Gradient%20Descent.pdf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}