L05_grad-descent/code/second-order-deriv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STAT 479: Deep Learning (Spring 2019)  \n",
    "Instructor: Sebastian Raschka (sraschka@wisc.edu)  \n",
    "Course website: http://pages.stat.wisc.edu/~sraschka/teaching/stat479-ss2019/\n",
    "GitHub repository: https://github.com/rasbt/stat479-deep-learning-ss19"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.1\n",
      "IPython 7.2.0\n",
      "\n",
      "torch 1.0.1\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Second Order Partial Derivatives (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those who are interested, in a nutshell, the second order\n",
    "partial derivative of a function is the partial derivative of the partial\n",
    "derivative. For instance, we write the second derivative of a function $f$\n",
    "with respect to $x$ as\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:partial-def}\n",
    "\\frac{\\partial}{\\partial x} \\bigg(\\frac{\\partial f}{\\partial x} \\bigg) =\n",
    "\\frac{\\partial^2 f}{\\partial x^2}.\n",
    "\\end{equation}\n",
    "\n",
    "For example, we compute the second partial derivative of a function\n",
    "$f(x, y) = x^2y+y$ as follows:\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:partial-example}\n",
    "\\frac{\\partial^2 f}{\\partial x^2}  = \\frac{\\partial}{\\partial x} \\bigg( \n",
    "\\frac{\\partial}{\\partial x} x^2y+y \\bigg) = \\frac{\\partial}{\\partial x}  2xy =\n",
    "\\frac{\\partial}{\\partial x} = 2y.\n",
    "\\end{equation}\n",
    "\n",
    "Note that in the initial definition (1st equation)\n",
    "and the example (2nd equation) both the first and second order\n",
    "partial derivatives were computed with respect to the same input argument, $x$. However,\n",
    "depending on what measurement we are interested in, the second order partial\n",
    "derivative can involve a different input argument. For instance, \n",
    "given a multivariable function with two input arguments, we can in fact compute four\n",
    "distinct second order partial derivatives:\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial^2 f}{\\partial x^2}, \\; \\frac{\\partial^2 f}{\\partial y^2}, \\;\n",
    "\\frac{\\partial^2 f}{\\partial x \\partial y}, \\text{ and } \\frac{\\partial^2\n",
    "f}{\\partial y \\partial x},\n",
    "\\end{equation}\n",
    "\n",
    "where, for example, $\\frac{\\partial^2 f}{\\partial y \\partial x}$ is defined as\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial^2 f}{\\partial y \\partial x} = \\frac{\\partial}{\\partial y}\n",
    "\\bigg(\\frac{\\partial f}{\\partial x} \\bigg).\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: First Order Derivative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([24.]),)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([3.], requires_grad=True)\n",
    "y  = torch.tensor([4.])\n",
    "\n",
    "f = x**2 * y + y\n",
    "\n",
    "grad(f, x) # 2xy = 24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Second Order Derivative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([8.]),)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([3.], requires_grad=True)\n",
    "y  = torch.tensor([4.])\n",
    "\n",
    "f = x**2 * y + y\n",
    "\n",
    "df_dx = grad(f, x, create_graph=True) # first order deriv\n",
    "grad(df_dx, x) # 2y = 8"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}