Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.3\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.990848162756775\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.300672\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.6\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.75\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.991640741515118\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.750232\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.3, 0.6, 0.75]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.8929028107279455e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  0\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.25\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 5.37e-09\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.274000; val_acc: 0.192000\n",
      "(Epoch 1 / 25) train acc: 0.410000; val_acc: 0.263000\n",
      "(Epoch 2 / 25) train acc: 0.518000; val_acc: 0.269000\n",
      "(Epoch 3 / 25) train acc: 0.550000; val_acc: 0.248000\n",
      "(Epoch 4 / 25) train acc: 0.684000; val_acc: 0.297000\n",
      "(Epoch 5 / 25) train acc: 0.758000; val_acc: 0.292000\n",
      "(Epoch 6 / 25) train acc: 0.782000; val_acc: 0.266000\n",
      "(Epoch 7 / 25) train acc: 0.860000; val_acc: 0.241000\n",
      "(Epoch 8 / 25) train acc: 0.868000; val_acc: 0.283000\n",
      "(Epoch 9 / 25) train acc: 0.898000; val_acc: 0.281000\n",
      "(Epoch 10 / 25) train acc: 0.894000; val_acc: 0.265000\n",
      "(Epoch 11 / 25) train acc: 0.946000; val_acc: 0.291000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.293000\n",
      "(Epoch 13 / 25) train acc: 0.940000; val_acc: 0.280000\n",
      "(Epoch 14 / 25) train acc: 0.972000; val_acc: 0.279000\n",
      "(Epoch 15 / 25) train acc: 0.980000; val_acc: 0.298000\n",
      "(Epoch 16 / 25) train acc: 0.978000; val_acc: 0.318000\n",
      "(Epoch 17 / 25) train acc: 0.972000; val_acc: 0.291000\n",
      "(Epoch 18 / 25) train acc: 0.988000; val_acc: 0.298000\n",
      "(Epoch 19 / 25) train acc: 0.980000; val_acc: 0.299000\n",
      "(Epoch 20 / 25) train acc: 0.998000; val_acc: 0.306000\n",
      "(Iteration 101 / 125) loss: 0.016472\n",
      "(Epoch 21 / 25) train acc: 0.992000; val_acc: 0.304000\n",
      "(Epoch 22 / 25) train acc: 0.986000; val_acc: 0.310000\n",
      "(Epoch 23 / 25) train acc: 1.000000; val_acc: 0.301000\n",
      "(Epoch 24 / 25) train acc: 0.992000; val_acc: 0.295000\n",
      "(Epoch 25 / 25) train acc: 0.996000; val_acc: 0.305000\n",
      "0.75\n",
      "(Iteration 1 / 125) loss: 17.318479\n",
      "(Epoch 0 / 25) train acc: 0.232000; val_acc: 0.172000\n",
      "(Epoch 1 / 25) train acc: 0.372000; val_acc: 0.253000\n",
      "(Epoch 2 / 25) train acc: 0.416000; val_acc: 0.256000\n",
      "(Epoch 3 / 25) train acc: 0.516000; val_acc: 0.306000\n",
      "(Epoch 4 / 25) train acc: 0.560000; val_acc: 0.299000\n",
      "(Epoch 5 / 25) train acc: 0.592000; val_acc: 0.294000\n",
      "(Epoch 6 / 25) train acc: 0.612000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.642000; val_acc: 0.294000\n",
      "(Epoch 8 / 25) train acc: 0.674000; val_acc: 0.297000\n",
      "(Epoch 9 / 25) train acc: 0.700000; val_acc: 0.301000\n",
      "(Epoch 10 / 25) train acc: 0.766000; val_acc: 0.316000\n",
      "(Epoch 11 / 25) train acc: 0.792000; val_acc: 0.319000\n",
      "(Epoch 12 / 25) train acc: 0.784000; val_acc: 0.288000\n",
      "(Epoch 13 / 25) train acc: 0.808000; val_acc: 0.309000\n",
      "(Epoch 14 / 25) train acc: 0.818000; val_acc: 0.321000\n",
      "(Epoch 15 / 25) train acc: 0.850000; val_acc: 0.330000\n",
      "(Epoch 16 / 25) train acc: 0.844000; val_acc: 0.314000\n",
      "(Epoch 17 / 25) train acc: 0.830000; val_acc: 0.316000\n",
      "(Epoch 18 / 25) train acc: 0.890000; val_acc: 0.334000\n",
      "(Epoch 19 / 25) train acc: 0.894000; val_acc: 0.330000\n",
      "(Epoch 20 / 25) train acc: 0.908000; val_acc: 0.326000\n",
      "(Iteration 101 / 125) loss: 5.317433\n",
      "(Epoch 21 / 25) train acc: 0.882000; val_acc: 0.285000\n",
      "(Epoch 22 / 25) train acc: 0.918000; val_acc: 0.299000\n",
      "(Epoch 23 / 25) train acc: 0.912000; val_acc: 0.327000\n",
      "(Epoch 24 / 25) train acc: 0.910000; val_acc: 0.312000\n",
      "(Epoch 25 / 25) train acc: 0.892000; val_acc: 0.313000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0.75]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3X2YnHV58P3v6SbIipgFkqLZBAgvpmISSbsElYpA1KBAiGhjsKL4hrQird6mhPtp09y5a0nBW2wKT58bq4JWCZHSiJU2tbxoQYVsDAYBI5GXZjdUQyQR7AJJOJ8/ZjZMlt2wm92Za2fm+zmOPWau3/Wba855ySZnzt9LZCaSJEmSpObxkqIDkCRJkiTVlomgJEmSJDUZE0FJkiRJajImgpIkSZLUZEwEJUmSJKnJmAhKkiRJUpMxEZQkNYWIaImIpyLisKJjkSSpaOE+gpKk0Sginqo4fBnwDLCrfPyxzPxa7aOSJKkxmAhKkka9iHgE+Ehm/vte+ozJzJ21i6p2Gvm1SZKK4dBQSVJdioi/jIjrI+K6iHgSeF9EvCEifhgR2yLisYhYHhFjy/3HRERGxBHl438on/+XiHgyIn4QEVMGeK6XRMQNEfFf5WvfHhGvqTj/soi4IiL+MyK2R8T3IuKl5XMnlWPaHhGbIuLccvsdEXFexTU+EhG394n1jyJiI/DTcvuVEdEVEb+OiDUR8caKx4+JiD+PiJ+Xz3dGxMSI+L8R8dd9Xs/NEfGJ4X8KkqR6ZSIoSapn7wS+DowDrgd2An8MjAdOBE4DPraXx78X+HPgYOA/gf+9l77/DBwDvBL4CfDVinNXADOAE8rX+p/Ac+XE8mbgc8AhwEzg3iG8vrnA8cD08vFd5ec5GLgB+EZvwgksBN5N6TW3AR8BngauBc6JiACIiEOBk4HrhhCHJKnBmAhKkurZHZn5rcx8LjN7MnNNZt6VmTsz8yHgauDNe3n8DZnZmZk7gK8Bx/XXqXz9azLzycx8GlgC/G5EHBARLcB5wEWZ+Vhm7srMO8rXfB/wL5m5shzT45l5zxBe319l5hOZ2VOO46uZ+avyMNHLgFcAR5f7fgT4n5n5YDnee8p9v08pITy53O8c4N8z8/EhxCFJajAmgpKkerap8iAifjsivl0ewvlrYCml6uBA/qvi/n8DL++vU3nF0csi4qHydTeWT40HDgX2A37ez0MnD9A+WH1f359GxE8jYjvwBHAAz7++vT3XVyglpZRvvzpAP0lSkzARlCTVs74rnv1fSsM2j87MVwCLgRiB53k/8A7gVErDUHurcAH8AngWOKqfx20aoB3gN5RWQ+31yn767H59EXEK8CngXZSGfh4EPMXzr29vz/VV4J0RMbPc51sD9JMkNQkTQUlSIzkQ2A78pryYy97mBw71us8AWyklb5/pPZGZu4BrgM9HxCvL1cMTy4vU/ANwWkS8q7yYy/iIeF35ofcA74qI1oh4NfChQcSwE3gcGEtpeOoBFef/HvjLiDgqSo6LiIPLMT4K/JjSfMFvlIe3SpKamImgJKmR/A/gA8CTlKqD14/Qdb8MbC7/3Ad8v8/5TwIPAGuBXwF/RWmLpoeBM4GLy+0/4vmFXz5LqeL3S+BLlJLGvbkZ+HfgQeAR4NfAYxXnLwdWAbeUz10N7F9x/tryczssVJLkPoKSJDWDiDgV+CJwZPqXvyQ1PSuCkiQ1uIjYj9K2Gl8wCZQkgYmgJEkNLSKmU1ph9GBgecHhSJJGCYeGSpIkSVKTsSIoSZIkSU3GRFCSJEmSmsyYogMYKePHj88jjjii6DAkSZIkqRBr1659PDMnDKZvwySCRxxxBJ2dnUWHIUmSJEmFiIhHB9vXoaGSJEmS1GRMBCVJkiSpyZgISpIkSVKTMRGUJEmSpCZjIihJkiRJTaaQRDAivhQRv4yInwxwPiJieURsjIj1EfE7tY5RkiRJkhpVUdtHXANcCXxlgPNvB44p/5wA/F35VpIkSdIosWpdN5ev3sDmbT1MbGtl4ZypzJvZXnRYGoRCEsHM/F5EHLGXLmcBX8nMBH4YEW0R8arMfKwmAUqSJKkumZjUzqp13Vxy47307NgFQPe2Hi658V6Ahn3PG+n7NVo3lG8HNlUcd5XbTAQlSZLUr6ITk0ZKEgbj8tUbdr/XvXp27OLy1Ruq/rqLeK+L/n6NtLpeLCYizo+Izojo3LJlS9HhSJIkjSqr1nVz4rJbmbLo25y47FZWresuOqSq2ltiUm29SUL3th6S55OERn7PN2/rGVL7SCnqvS7y+1UNozUR7AYmVxxPKrftITOvzsyOzOyYMGFCzYKTJEka7UxMXrx9JBWdhBaR8E9sax1S+0gp6r0u8vtVDaM1EbwJeH959dDXA9udHyhJkjR4jVa9GIyiEhNovuoYwMI5U2kd27JHW+vYFhbOmVrV5y3qvS7y+1UNRW0fcR3wA2BqRHRFxIcj4oKIuKDc5WbgIWAj8AXgj4qIU5IkjbyiqhfNNkyy6OpFEe93UYkJNF91DErz4i49ezrtba0E0N7WyqVnT6/6fLmi3usiv1/VUNSqoee8yPkEPl6jcCRJUo0UtdhCoy3yMBgT21rp7ifpq0X1oqj3u/faRSzYsnDO1D1eMzR2dazXvJntNf8zVNR7XeT3qxpG66qhkiSpyopYda+oVQaLXN0Qinmvi/rHMhT7fheRmPQ+L9Q+SSgy4S9KkQlZUd+vajARlCSpCRVVsSmqelFk1aQZq2NFV6mK0kzVsaI1UkJWFBNBSZKaUFEVm6KqF0VWTZqxOtaMVaqiNNpwRdWOiaAkSTTfRtBFVWyKql4UWTVpxupYs1apimJ1TPvCRFCS1PRcSGTP9moqqnpRZNWkGatjVqmk0S9KC3TWv46Ojuzs7Cw6DElSHTpx2a39/kO9va2VOxedWkBE1dc3+YVSxaYWS783G99rSbUSEWszs2Mwfa0ISpKaXjMO3bNiUzu+15JGIxNBSdKoUdQ8vWYcugfOK6ol32tJo81Lig5AkiR4fvhc97Yekufn6a1a11315144ZyqtY1v2aHNhC0lSI7MiKEkaFYpeYr83hlpXI5tttVJJ0uhgIihJGhWKnqdXxNC9ZlytVJI0Ojg0VJI0Kgw0H6+R5+ntrQoqSVI1mQhKkkaFZpynV3QVVJLUvEwEJUmjwryZ7Vx69nTa21oJSnv4Nfo+a81YBZUkjQ7OEZQkjRrNtsT+wjlT+91ovJGroJKk0cFEUJKkgrjRuKQRsX4l3LIUtnfBuEkwezHMmF90VBrlTAQlSSpQs1VBJY2w9SvhWxfBjvLc4u2bSsdgMqi9co6gJEmSVK9uWfp8EthrR0+pvRbWr4QrpsGSttLt+pW1eV4NmxVBSZIkNY5mGya5vWto7SOpyGpks33OVVBIRTAiTouIDRGxMSIW9XP+8Ii4JSLWR8TtETGpiDglSZJUR3oTk+2bgHw+MWnkKtW4Af6ZPFD7SCqqGtmMn3MV1DwRjIgW4Crg7cCxwDkRcWyfbp8FvpKZM4ClwKW1jVKSJEl1p+hhkkWYvRjG9tlyZmxrqb3aiqpGNuPnXAVFDA2dBWzMzIcAImIFcBZwf0WfY4FPle/fBqyqaYSSNAqsWtftapKSNBRFDpMsSu9wyCKGSY6bVK7K9dNeTc34OVdBEYlgO1D5jekCTujT58fA2cDfAO8EDoyIQzJza21ClKRirVrXvcf+ct3berjkxnsBapIMmoRKGrYi5nAVlZgUbcb8YubHzV685xxBqE01slk/5xE2WlcN/TTw5ohYB7wZ6AZ29e0UEedHRGdEdG7ZsqXWMUpS1Vy+esMem4wD9OzYxeWrN1T9uXuT0O5tPSTPJ6Gr1nVX/bkljbCiVnQsag5XkcMkm9GM+XDmchg3GYjS7ZnLq5+UFvk5N9AqqUVUBLuByRXHk8ptu2XmZkoVQSLi5cC7MnNb3wtl5tXA1QAdHR1ZrYAlqdY2b+sZUvtI2lsSalVQqiNFrui4tzlc1XzuIodJNqsiqpFFfc4NtmdjEYngGuCYiJhCKQFcALy3skNEjAd+lZnPAZcAX6p5lJJUoIltrXT3k/RNbGvtp/fIKjIJlTSCikrGoNg5XEUNk1RtFfE5F/lnqgpqPjQ0M3cCFwKrgQeAlZl5X0QsjYi55W4nAxsi4mfAocBnah2nJBVp4ZyptI5t2aOtdWwLC+dMrfpzD5Rs1iIJlTSCikzGitzSoEgNNGxQ/WiwRWoKmSOYmTdn5qsz86jM/Ey5bXFm3lS+f0NmHlPu85HMfKaIOCWpKPNmtnPp2dNpb2slgPa2Vi49e3pNhmYWmYRKGkFFJmPNOFfPve0aX4P9B0cRQ0MlSYMwb2Z7IXPyep/TVUOlOlfUio7QnHP1GmzYoPpR5J+pKjARlCS9QFFJqKQRVHQy1mxz9Rps2KD6UfSfqRFmIihJL8I99STVrWZLxork3nbNoYH+TI3WfQQlaVRwTz1J0qA047xI1TUTQUl1Y9W6bk5cditTFn2bE5fdWpNkrMiN3SVJdaSozdWlfeTQUEl1obcy15uU9VbmgKoO03RPPUnSoDXQsEE1PiuCkupCUZU599STJEmNyERQUl0oqjLnnnpSFbjptiQVzqGhkurCxLZWuvtJ+qpdmXNPPWmE9W663bsPV++m21D9IXXrVzbMsu+SNFwmgpLqwsI5U/eYIwi1q8y5p540goradLvIBFSSRiGHhkqqC/NmtnPp2dNpb2slgPa2Vi49e7oJmlRvitp0e28JqCQ1ISuCkuqGlTmpARS16XZRCagkjVJWBCVJUu0Uten2QIlmtRPQXi6QI2mUMRGUJEm1U9Sm20UloPD8/MTtm4B8fn6iyaCkAjk0VJIk1VYRm273Pl8Rq4YWtUCOJO2FiaAkSWoORSSg4PxESaOSQ0MlSWpWzlurjaLnJ0pSP0wEJUlqRs5bq50i5ydK0gBMBCVJakbuq1c7RS2QI0l74RxBSZKKtH5lMQuYOG+ttoqanyhJAyikIhgRp0XEhojYGBGL+jl/WETcFhHrImJ9RLyjiDglSaqqIodnOm9NkppazRPBiGgBrgLeDhwLnBMRx/bp9mfAysycCSwA/t/aRilJUg0UOTzTeWuS1NSKqAjOAjZm5kOZ+SywAjirT58EXlG+Pw7YXMP4JEmqjSKHZzpvTZKaWhFzBNuBTRXHXcAJffosAf4tIj4BHAC8pTahSZIKVdR8uaKMm1QeFtpPey04b02SmtZoXTX0HOCazJwEvAP4akS8INaIOD8iOiOic8uWLTUPUpI0gppxOwOHZ0qSClJEItgNTK44nlRuq/RhYCVAZv4A2B8Y3/dCmXl1ZnZkZseECROqFK4kqSaK3s6giM3VHZ4pSSpIEUND1wDHRMQUSgngAuC9ffr8JzAbuCYiXkMpEbTkJ40Cq9Z1c/nqDWze1sPEtlYWzpnKvJntRYelRlDkfLneamRvItpbjYTqJ2UOz5QkFaDmFcHM3AlcCKwGHqC0Ouh9EbE0IuaWu/0P4KMR8WPgOuC8zMxaxyppT6vWdXPJjffSva2HBLq39XDJjfeyal3for60D4rczqDoaqQkSTVWyIbymXkzcHOftsUV9+8HTqx1XJL27vLVG+jZsWuPtp4du7h89Qarghq+2Yv3rMpB7ebLubm6JKnJjNbFYiSNQpu39QypXRqSIufLubm6JKnJFFIRlFSfJra10t1P0jexrbWf3tI+KGq+XJHVSEmSCmBFUNKgLZwzldaxLXu0tY5tYeGcqQVFpKopYgXNIrl6pySpyVgRlDRovfMAXTW0wRW5gmaRXL1TktREolEW4+zo6MjOzs6iw5Bqxm0cVDVXTCtv6t7HuMnwyZ/UPh5JkjQoEbE2MzsG09eKoFSHerdx6F3Bs3cbB8BkUMPnCpqSJDU85whKdWhv2zhIw+YKmpIkNTwTQakOuY2Dqmr24tKKmZVcQVOSpIZiIijVoYG2a3AbB40IV9CUJKnhOUdQqkML50zdY44guI2DRpgraEqS1NBMBKU65DYOkiRJGg4TQalOzZvZbuInSZKkfeIcQUmSJElqMiaCkiRJktRkTAQlSZIkqcmYCEqSJElSkzERlCRJkqQmYyIoSZIkSU3GRFCSJEmSmkwhiWBEnBYRGyJiY0Qs6uf8FRFxT/nnZxGxrYg4JUmSJKkR1XxD+YhoAa4C3gp0AWsi4qbMvL+3T2Z+sqL/J4CZtY5Tkgq3fiXcshS2d8G4STB7McyYX3RUkiSpARRREZwFbMzMhzLzWWAFcNZe+p8DXFeTyCRptFi/Er51EWzfBGTp9lsXldolSZKGqYhEsB3YVHHcVW57gYg4HJgC3FqDuCRp9LhlKezo2bNtR0+pXZIkaZhG+2IxC4AbMnNXfycj4vyI6IyIzi1bttQ4NEmqou1dQ2uXJEkagiISwW5gcsXxpHJbfxawl2GhmXl1ZnZkZseECRNGMERJKti4SUNrlyRJGoIiEsE1wDERMSUi9qOU7N3Ut1NE/DZwEPCDGscnDdqqdd2cuOxWpiz6Nicuu5VV6wb6Pw1piGYvhrGte7aNbS21S5IkDdOwEsGI+EREHDSUx2TmTuBCYDXwALAyM++LiKURMbei6wJgRWbmcGKUqmXVum4uufFeurf1kED3th4uufFek0GNjBnz4czlMG4yEKXbM5e7aqgkSRoRMZw8KyL+klLC9iPgS8DqohK3jo6O7OzsLOKp1aROXHYr3dt6XtDe3tbKnYtOLSAiSZIkNbOIWJuZHYPpO6yKYGb+GXAM8EXgPODBiPiriDhqONeV6sHmfpLAvbWrjq1fCVdMgyVtpVu3cJAkSXVu2HMEyxXA/yr/7KQ0r++GiLhsuNeWRrOJba1Daledcj8/SZLUgIY7R/CPI2ItcBlwJzA9M/8Q+F3gXSMQnzRqLZwzldaxLXu0tY5tYeGcqQVFpKpwPz9JktSAxgzz8QcDZ2fmo5WNmflcRJwxzGtLo9q8me0AXL56A5u39TCxrZWFc6bubleDcD8/SZLUgIabCP4L8Kveg4h4BfCazLwrMx8Y5rWlUW/ezHYTv0Y3blJ5WGg/7ZIkSXVquHME/w54quL4qXKbJDUG9/OTJEkNaLiJYFRuF5GZzzH8KqMkjR7u5ydJkhrQcJO2hyLiIp6vAv4R8NAwrylJo8uM+SZ+kiSpoQy3IngB8EagG+gCTgDOH25QkiRJkqTqGVZFMDN/CSwYoVgkae/Wryxt27C9q7RYy+zFVuokSZL2wbASwYjYH/gw8Fpg/972zPzQMOOSpD31buzeu6df78buYDIoSZI0RMMdGvpV4JXAHOC7wCTgyeEGJUkv4MbukiRJI2a4ieDRmfnnwG8y81rgdErzBCVpZLmxuyRJ0ogZbiK4o3y7LSKmAeOA3xrmNSXphQbawN2N3SVJkoZsuIng1RFxEPBnwE3A/cBfDzsqSerLjd0lSZJGzD4vFhMRLwF+nZlPAN8DjhyxqCSpr94FYVw1VJIkadj2ORHMzOci4k+BlSMYjyQNzI3dJUmSRsRwh4b+e0R8OiImR8TBvT8jEpmk0Wn9SrhiGixpK92u9/+CJEmS6s2w9hEE3lO+/XhFW+IwUakxuZefJElSQxhWIpiZU0YqEEl1YG97+ZkISpIk1Y1hJYIR8f7+2jPzKy/yuNOAvwFagL/PzGX99JkPLKFUYfxxZr53OLFKGgHu5SdJktQQhjs09PiK+/sDs4EfAQMmghHRAlwFvBXoAtZExE2ZeX9Fn2OAS4ATM/OJiHBvQmk0GDepNBy0v3ZJkiTVjeEODf1E5XFEtAErXuRhs4CNmflQ+TErgLMo7UHY66PAVeWtKcjMXw4nTkkjZPbiPecIgnv5SZIk1aHhrhra12+AF5s32A5UlhS6ym2VXg28OiLujIgfloeSSqpUxOqdM+bDmcth3GQgSrdnLnd+oCRJUp0Z7hzBb1GawwelpPJYRmZfwTHAMcDJwCTgexExPTO39Xn+84HzAQ477LAReFqpThS5eqd7+UmSJNW94c4R/GzF/Z3Ao5n5YqtGdAOTK44nldsqdQF3ZeYO4OGI+BmlxHBNZafMvBq4GqCjoyORmoWrd0qSJGkYhjs09D8pJWzfzcw7ga0RccSLPGYNcExETImI/YAFwE19+qyiVA0kIsZTGir60DBjlRqHq3dKkiRpGIabCH4DeK7ieFe5bUCZuRO4EFgNPACszMz7ImJpRMwtd1tNKam8H7gNWJiZW4cZq9Q4Blql09U7JUmSNAjDHRo6JjOf7T3IzGfLVb69ysybgZv7tC2uuJ/Ap8o/kvpy9U5JkiQNw3ArglsqqnhExFnA48O8pqQX4+qdkiRJGobhVgQvAL4WEVeWj7uA9w/zmpIGw9U7JUmStI+Gu6H8z4HXR8TLy8dPjUhUkiRJkqSqGdbQ0Ij4q4hoy8ynMvOpiDgoIv5ypIKTRr0iNnWXJEmShmm4cwTfXrnJe2Y+AbxjmNeU6kPvpu7bNwH5/KbuJoOSJEka5YabCLZExEt7DyKiFXjpXvpLjWNvm7pLkiRJo9hwF4v5GnBLRHwZCOA84NrhBiUN1ap13Vy+egObt/Uwsa2VhXOmMm9me3Wf1E3dJUmSVKeGu1jMX0fEj4G3AElpI/jDRyIwabBWrevmkhvvpWfHLgC6t/VwyY33AlQ3GRw3qTwstJ92SZIkaRQb7tBQgF9QSgJ/HzgVeGAErikN2uWrN+xOAnv17NjF5as3VPeJZy8ubeJeyU3dJUmSVAf2qSIYEa8Gzin/PA5cD0RmnjKCsUmDsnlbz5DaR0zvHn63LC0NBx03qZQEurefJEmSRrl9HRr6U+A/gDMycyNARHxyxKKShmBiWyvd/SR9E9ta++k9wtzUXZIkSXVoX4eGng08BtwWEV+IiNmUFouRam7hnKm0jm3Zo611bAsL50wtKCJJkiRpdNunRDAzV2XmAuC3gduAPwF+KyL+LiLeNpIBSi9m3sx2Lj17Ou1trQTQ3tbKpWdPr/6qoZIkSVKdiswcmQtFHERpwZj3ZObsEbnoEHR0dGRnZ2etn1aSJEmSRoWIWJuZHYPpOxKrhgKQmU9k5tVFJIGSJEmSpMEb7oby0m6FbOrea/1KV++UJEmSBslEUCOisE3doZQEfusi2FFeOXT7ptIxmAxKkiRJ/RixoaFqboVt6g6lSuCOPttH7OgptUuSJEl6ARNBjYjCNnWH0nDQobRLkiRJTa6QRDAiTouIDRGxMSIW9XP+vIjYEhH3lH8+UkScGryBNm+vyabu4yYNrV2SJElqcjVPBCOiBbgKeDtwLHBORBzbT9frM/O48s/f1zRIDVmhm7rPXgxj+yScY1tL7ZIkSZJeoIiK4CxgY2Y+lJnPAiuAswqIQyOo0E3dZ8yHM5fDuMlAlG7PXO5CMZIkSdIAilg1tB3YVHHcBZzQT793RcRJwM+AT2bmpn76aBSZN7O9dttF9DVjvomfJEmSNEijdbGYbwFHZOYM4DvAtf11iojzI6IzIjq3bNlS0wAlSZIkqV4VkQh2A5MrjieV23bLzK2Z+Uz58O+B3+3vQpl5dWZ2ZGbHhAkTqhKsJEmSJDWaIhLBNcAxETElIvYDFgA3VXaIiFdVHM4FHqhhfJIkSZLU0Go+RzAzd0bEhcBqoAX4UmbeFxFLgc7MvAm4KCLmAjuBXwHn1TpOSZIkSWpUkZlFxzAiOjo6srOzs+gwJEmSJKkQEbE2MzsG03e0LhYjSZIkSaoSE0FJkiRJajImgpIkSZLUZEwEJUmSJKnJmAhKkiRJUpMxEZQkSZKkJmMiKEmSJElNxkRQkiRJkpqMiaAkSZIkNRkTQUmSJElqMiaCkiRJktRkTAQlSZIkqcmYCEqSJElSkzERlCRJkqQmYyIoSZIkSU3GRFCSJEmSmoyJoEbO+pVwxTRY0la6Xb+y6IgkSZIk9WNM0QGoQaxfCd+6CHb0lI63byodA8yYX1xckiRJkl7ARLABrVrXzeWrN7B5Ww8T21pZOGcq82a2V/dJb1n6fBLYa0dPqd1EUJIkSRpVChkaGhGnRcSGiNgYEYv20u9dEZER0VHL+OrZqnXdXHLjvXRv6yGB7m09XHLjvaxa113dJ97eNbR2SZIkSYWpeSIYES3AVcDbgWOBcyLi2H76HQj8MXBXbSOsb5ev3kDPjl17tPXs2MXlqzdU94nHTRpauyRJkqTCFFERnAVszMyHMvNZYAVwVj/9/jfw18DTtQyu3m3e1jOk9hEzezGMbd2zbWxrqV2SJEnSqFJEItgObKo47iq37RYRvwNMzsxv1zKwRjCxrXVI7SNmxnw4czmMmwxE6fbM5c4PlCRJkkahUbdYTES8BPgccN4g+p4PnA9w2GGHVTewOrFwzlQuufHePYaHto5tYeGcqdV/8hnzTfwkSZKkOlBERbAbmFxxPKnc1utAYBpwe0Q8ArweuKm/BWMy8+rM7MjMjgkTJlQx5Poxb2Y7l549nfa2VgJob2vl0rOnV3/VUEmSJEl1o4iK4BrgmIiYQikBXAC8t/dkZm4HxvceR8TtwKczs7PGcdateTPbTfwkSZIkDajmFcHM3AlcCKwGHgBWZuZ9EbE0IubWOh5JkiRJajaFzBHMzJuBm/u09bu8ZGaeXIuYJEmSJKlZFLKhvKps/Uq4YhosaSvdrl9ZdESSJEmSRpFRt2qohmn9SvjWRbCjvG/g9k2lY3BFT0mSJEmAFcHGc8vS55PAXjt6Su2SJEmShIlg49neNbR2SZIkSU3HRLDRjJs0tHZJkiRJTcdEsNHMXgxjW/dsG9taapckSZIkTAQbz4z5cOZyGDcZiNLtmctdKEaSJEnSbq4a2ohmzDfxkyRJkjQgK4KSJEmS1GRMBCVJkiSpyZgISpIkSVKTMRGUJEmSpCZjIihJkiRJTcZEUJIkSZKajImgJEmSJDUZE0FJkiRJajImgpIkSZLUZEwEJUmSJKnJmAhKkiRJUpMxEZQkSZKkJlNIIhgRp0XEhojYGBGL+jl/QUTcGxH3RMQdEXFsEXFKkiRJUiOqeSIYES3AVcDbgWOBc/pJ9L6emdMz8zjgMuBzNQ5TkiRJkhpWERXBWcDGzHwoM58FVgBnVXZ8uBAiAAAgAElEQVTIzF9XHB4AZA3jkyRJkqSGNqaA52wHNlUcdwEn9O0UER8HPgXsB5xam9AkSZIkqfGN2sViMvOqzDwKuBj4s/76RMT5EdEZEZ1btmypbYCSJEmSVKeKSAS7gckVx5PKbQNZAczr70RmXp2ZHZnZMWHChBEMUZIkSZIaVxGJ4BrgmIiYEhH7AQuAmyo7RMQxFYenAw/WMD5JkiRJamg1nyOYmTsj4kJgNdACfCkz74uIpUBnZt4EXBgRbwF2AE8AH6h1nJIkSZLUqIpYLIbMvBm4uU/b4or7f1zzoEbYqnXdXL56A5u39TCxrZWFc6Yyb2Z70WFJkiRJUjGJYKNbta6bS268l54duwDo3tbDJTfeC2AyKEmSJKlwo3bV0Hp2+eoNu5PAXj07dnH56g0FRSRJkiRJz7MiWAWbt/UMqV2SJKkZ7Nixg66uLp5++umiQ5Hq2v7778+kSZMYO3bsPl/DRLAKJra10t1P0jexrbWAaCRJkkaHrq4uDjzwQI444ggiouhwpLqUmWzdupWuri6mTJmyz9dxaGgVLJwzldaxLXu0tY5tYeGcqQVFJEmSVLynn36aQw45xCRQGoaI4JBDDhl2Zd2KYBX0LgjjqqGSJEl7MgmUhm8k/hxZEaySeTPbuXPRqTy87HTuXHSqSaAkSdIo8K//+q9MnTqVo48+mmXLlvXb55lnnuE973kPRx99NCeccAKPPPLI7nOXXnopRx99NFOnTmX16tUv+nzXXHMNF1544UiFP2Sf//zn+e///u/Cnn+kDebz++QnP8lxxx3Hcccdx6tf/Wra2tp2n2tpadl9bu7cuS/6fI888gjTpk0bsfiHatWqVdx///1VubYVQUmSJDWFXbt28fGPf5zvfOc7TJo0ieOPP565c+dy7LHH7tHvi1/8IgcddBAbN25kxYoVXHzxxVx//fXcf//9rFixgvvuu4/Nmzfzlre8hZ/97Ge0tLQM8IwD27lzJ2PGVP+f4p///Od53/vex8te9rKqP1e1Dfbzu+KKK3bf/9u//VvWrVu3+7i1tZV77rln2LHU6vNbtWoVZ5xxxgte40iwIihJkqRRadW6bk5cditTFn2bE5fdyqp13cO63t13383RRx/NkUceyX777ceCBQv45je/+YJ+3/zmN/nABz4AwLvf/W5uueUWMpNvfvObLFiwgJe+9KVMmTKFo48+mrvvvvsFj//yl7/Mq1/9ambNmsWdd965u/28887jggsu4IQTTuBP//RP+dWvfsW8efOYMWMGr3/961m/fj0AS5Ys4dxzz+UNb3gDxxxzDF/4wheA0iIhCxcuZNq0aUyfPp3rr78egNtvv50zzjhj9/NceOGFXHPNNSxfvpzNmzdzyimncMoppwzrvdsn61fCFdNgSVvpdv3KYV1usJ9fpeuuu45zzjlnSM+zdu1aXve61/G6172Oq666anf7Nddcw9y5czn11FOZPXv2Xj+Pk046idNPP52pU6dywQUX8Nxzz+2OZ/r06UybNo2LL75497Vf/vKX775/ww03cN555/H973+fm266iYULF3Lcccfx85//fEiv48VYEZQkSdKos2pdN5fceO/uvZm7t/VwyY33AuzzlJvu7m4mT568+3jSpEncdddde+03ZswYxo0bx9atW+nu7ub1r3/9Ho/v7t4zOX3sscf4i7/4C9auXcu4ceM45ZRTmDlz5u7zXV1dfP/736elpYVPfOITzJw5k1WrVnHrrbfy/ve/f3e1av369fzwhz/kN7/5DTNnzuT000/nBz/4Affccw8//vGPefzxxzn++OM56aSTBny9F110EZ/73Oe47bbbGD9+/D69Z/ts/Ur41kWwo7yS/vZNpWOAGfP36ZKD/fx6Pfroozz88MOceuqpu9uefvppOjo6GDNmDIsWLWLevHkveNwHP/hBrrzySk466SQWLly4x7kf/ehHrF+/noMPPph//Md/HPDzuPvuu7n//vs5/PDDOe2007jxxht54xvfyMUXX8zatWs56KCDeNvb3saqVav6jQHgjW98I3PnzuWMM87g3e9+95Deq8GwIihJkqRR5/LVG3Yngb16duzi8tUbCopocO666y5OPvlkJkyYwH777cd73vOePc7//u///u6hpHfccQfnnnsuAKeeeipbt27l17/+NQBnnXUWra2tjB8/nlNOOYW7776bO+64g3POOYeWlhYOPfRQ3vzmN7NmzZravsDBumXp80lgrx09pfYaWbFiBe9+97v3GLr76KOP0tnZyde//nX+5E/+5AVVtm3btrFt27bdCV3v59PrrW99KwcffDDAXj+PWbNmceSRR9LS0sI555zDHXfcwZo1a3Z/N8aMGcMf/MEf8L3vfa+ab8FemQhWywiXwiVJkprJ5n72ZN5b+2C0t7ezadOm3cddXV20t7+wuljZb+fOnWzfvp1DDjlk0I/fmwMOOGBQ/fquCrm3VSLHjBmze+ghMOxtBUbE9q6htQ/CUN//FStWvGBYaG//I488kpNPPnmP+YODUY3Pr+/5Wn1+JoLV0FsK374JyOdL4SaDkiRJgzKxrXVI7YNx/PHH8+CDD/Lwww/z7LPPsmLFin5Xjpw7dy7XXnstUJqvdeqppxIRzJ07lxUrVvDMM8/w8MMP8+CDDzJr1qw9HnvCCSfw3e9+l61bt7Jjxw6+8Y1vDBjPm970Jr72ta8BpXll48eP5xWveAVQmqf49NNPs3XrVm6//XaOP/543vSmN3H99deza9cutmzZwve+9z1mzZrF4Ycfzv33388zzzzDtm3buOWWW3Y/x4EHHsiTTz65z+/ZPhs3aWjtgzDYzw/gpz/9KU888QRveMMbdrc98cQTPPPMMwA8/vjj3HnnnS9YhKWtrY22tjbuuOMOgN2fT38G+jygNDT04Ycf5rnnnuP666/n937v95g1axbf/e53efzxx9m1axfXXXcdb37zmwE49NBDeeCBB3juuef4p3/6p93PUc3PzzmC1bC3Uvg+jomWJElqJgvnTN1jjiBA69gWFs6Zus/XHDNmDFdeeSVz5sxh165dfOhDH+K1r30tAIsXL6ajo4O5c+fy4Q9/mHPPPZejjz6agw8+mBUrVgDw2te+lvnz53PssccyZswYrrrqqhesGPqqV72KJUuW8IY3vIG2tjaOO+64AeNZsmQJH/rQh5gxYwYve9nLdiefADNmzOCUU07h8ccf58///M+ZOHEi73znO/nBD37A6173OiKCyy67jFe+8pUAzJ8/n2nTpjFlypQ95iSef/75nHbaaUycOJHbbrttn9+7IZu9eM85ggBjW0vt+2iwnx+UqoELFizYo9L2wAMP8LGPfYyXvOQlPPfccyxatKjf1Ti//OUv86EPfYiI4G1ve9uA8Qz0efz0pz/l+OOP58ILL2Tjxo2ccsopvPOd7+QlL3kJy5Yt45RTTiEzOf300znrrLMAWLZsGWeccQYTJkygo6ODp556CoAFCxbw0Y9+lOXLl3PDDTdw1FFH7fP711dk5ohdrEgdHR3Z2dlZdBglS9qA/t7XgCXbah2NJEnSqPDAAw/wmte8ZtD9V63r5vLVG9i8rYeJba0snDO1KfZmXrJkCS9/+cv59Kc/XXQow7N+ZakQsr2rVAmcvbgpiiK33347n/3sZ/nnf/7nqj5Pf3+eImJtZnYM5vFWBKth3KTysNB+2iVJkjQo82a2N0Xi17BmzG+KxK9emQhWQxVK4ZIkSWoOS5YsKToEDcPJJ5/MySefXHQYL8rFYqphxnw4czmMmwxE6fbM5f6PiCRJkqRRwYpgtVgKlyRJeoHMfNGl9CXt3Uis81JIRTAiTouIDRGxMSIW9XP+UxFxf0Ssj4hbIuLwIuKUJEnSyNl///3ZunXriPwjVmpWmcnWrVvZf//9h3WdmlcEI6IFuAp4K9AFrImImzLz/opu64COzPzviPhD4DLgPbWOVZIkSSNn0qRJdHV1sWXLlqJDkera/vvvz6RJw1uIsoihobOAjZn5EEBErADOAnYngplZucnJD4H31TRCSZIkjbixY8cyZcqUosOQRDFDQ9uByr0VusptA/kw8C9VjUiSJEmSmsioXiwmIt4HdABvHuD8+cD5AIcddlgNI5MkSZKk+lVERbAbmFxxPKnctoeIeAvw/wBzM/OZ/i6UmVdnZkdmdkyYMKEqwUqSJElSo4lar9oUEWOAnwGzKSWAa4D3ZuZ9FX1mAjcAp2Xmg4O87hbg0ZGPeNjGA48XHYQamt8xVZPfL1WT3y9Vk98vVdNo/X4dnpmDqpDVPBEEiIh3AJ8HWoAvZeZnImIp0JmZN0XEvwPTgcfKD/nPzJxb80BHQER0ZmZH0XGocfkdUzX5/VI1+f1SNfn9UjU1wverkDmCmXkzcHOftsUV999S86AkSZIkqUkUsqG8JEmSJKk4JoLVd3XRAajh+R1TNfn9UjX5/VI1+f1SNdX996uQOYKSJEmSpOJYEZQkSZKkJmMiWEURcVpEbIiIjRGxqOh41Fgi4pGIuDci7omIzqLjUf2LiC9FxC8j4icVbQdHxHci4sHy7UFFxqj6NcD3a0lEdJd/j91TXlVcGrKImBwRt0XE/RFxX0T8cbnd32Eatr18v+r6d5hDQ6skIloo7Zf4VqCL0n6J52Tm/YUGpoYREY8AHZk5GvewUR2KiJOAp4CvZOa0cttlwK8yc1n5P7QOysyLi4xT9WmA79cS4KnM/GyRsan+RcSrgFdl5o8i4kBgLTAPOA9/h2mY9vL9mk8d/w6zIlg9s4CNmflQZj4LrADOKjgmSRpQZn4P+FWf5rOAa8v3r6X0F580ZAN8v6QRkZmPZeaPyvefBB4A2vF3mEbAXr5fdc1EsHragU0Vx100wBdGo0oC/xYRayPi/KKDUcM6NDMfK9//L+DQIoNRQ7owItaXh446bE/DFhFHADOBu/B3mEZYn+8X1PHvMBNBqX79Xmb+DvB24OPlYVdS1WRpLoHzCTSS/g44CjgOeAz4P8WGo3oXES8H/hH4k8z8deU5f4dpuPr5ftX17zATwerpBiZXHE8qt0kjIjO7y7e/BP6J0nBkaaT9ojw3oneOxC8LjkcNJDN/kZm7MvM54Av4e0zDEBFjKf0j/WuZeWO52d9hGhH9fb/q/XeYiWD1rAGOiYgpEbEfsAC4qeCY1CAi4oDyZGUi4gDgbcBP9v4oaZ/cBHygfP8DwDcLjEUNpvcf6GXvxN9j2kcREcAXgQcy83MVp/wdpmEb6PtV77/DXDW0ispLyH4eaAG+lJmfKTgkNYiIOJJSFRBgDPB1v18aroi4DjgZGA/8AvgLYBWwEjgMeBSYn5ku+KEhG+D7dTKlIVUJPAJ8rGI+lzRoEfF7wH8A9wLPlZv/J6V5XP4O07Ds5ft1DnX8O8xEUJIkSZKajENDJUmSJKnJmAhKkiRJUpMxEZQkSZKkJmMiKEmSJElNxkRQkiRJkpqMiaAkSX1ExK6IuKfiZ9EIXvuIiKirvaYkSY1nTNEBSJI0CvVk5nFFByFJUrVYEZQkaZAi4pGIuCwi7o2IuyPi6HL7ERFxa0Ssj4hbIuKwcvuhEfFPEfHj8s8by5dqiYgvRMR9EfFvEdFa2IuSJDUlE0FJkl6otc/Q0PdUnNuemdOBK4HPl9v+Frg2M2cAXwOWl9uXA9/NzNcBvwPcV24/BrgqM18LbAPeVeXXI0nSHiIzi45BkqRRJSKeysyX99P+CHBqZj4UEWOB/8rMQyLiceBVmbmj3P5YZo6PiC3ApMx8puIaRwDfycxjyscXA2Mz8y+r/8okSSqxIihJ0tDkAPeH4pmK+7twzr4kqcZMBCVJGpr3VNz+oHz/+8CC8v0/AP6jfP8W4A8BIqIlIsbVKkhJkvbG/4GUJOmFWiPinorjf83M3i0kDoqI9ZSqeueU2z4BfDkiFgJbgA+W2/8YuDoiPkyp8veHwGNVj16SpBfhHEFJkgapPEewIzMfLzoWSZKGw6GhkiRJktRkrAhKkiRJUpOxIihJairlzd8zIpwnL0lqWiaCkqS6EhH/GhFL+2k/KyL+ywRPkqQXZyIoSao31wLvi4jo034u8LXM3FlATCMiSvy7WZJUdf5lI0mqN6uAQ4A39TZExEHAGcBXysenR8S6iPh1RGyKiCWDvXhELIqIn0fEkxFxf0S8s8/5j0bEAxXnf6fcPjkiboyILRGxNSKuLLcviYh/qHj8HkNTI+L2iPhMRNwJ/DdwZER8sOI5HoqIj/WJ4ayIuKf8+n4eEadFxO9HxNo+/T4VEd8c7GuXJDUPE0FJUl3JzB5gJfD+iub5wE8z88fl49+Uz7cBpwN/GBHzBvkUP6eUZI4D/hfwDxHxKoCI+H1gSfnarwDmAlsjogX4Z+BR4AigHVgxhJd1LnA+cGD5Gr+klNi+gtKehFdUJJyzKCW8C8uv7yTgEeAmYEpEvKbPdb8yhDgkSU3CRFCSVI+uBd4dEfuXj99fbgMgM2/PzHsz87nMXA9cB7x5MBfOzG9k5ubyY68HHgRmlU9/BLgsM9dkycbMfLR8fiKwMDN/k5lPZ+YdQ3g912TmfZm5MzN3ZOa3M/Pn5ef4LvBvPF8B/TDwpcz8TjnG7sz8aWY+A1wPvA8gIl5LKSn95yHEIUlqEiaCkqS6U06yHgfmRcRRlBKxr/eej4gTIuK28jDN7cAFwPjBXDsi3l8edrktIrYB0yoeO5lSxbCvycCjw5ifuKlPDG+PiB9GxK/KMbxjEDFAKRl+b3n+5LnAynKCKEnSHkwEJUn16iuUKoHvA1Zn5i8qzn2d0lDJyZk5Dvj/gL6Ly7xARBwOfAG4EDgkM9uAn1Q8dhNwVD8P3QQcNsCKpb8BXlZx/Mp++uze1DciXgr8I/BZ4NByDDcPIgYy84fAs5Sqh+8FvtpfP0mSTAQlSfXqK8BbgI9SMSy07EDgV5n5dHlO3XsHec0DKCVlWwAi4oOUKoK9/h74dET8bnmFz6PLyePdwGPAsog4ICL2j4gTy4+5BzgpIg6LiHHAJS8Sw37AS8sx7IyItwNvqzj/ReCDETE7Il4SEe0R8dsV578CXAnsGOLwVElSEzERlCTVpcx8BPg+peTtpj6n/whYGhFPAospLS4zmGveD/wf4AfAL4DpwJ0V578BfIZSxfFJSiuYHpyZu4AzgaOB/wS6gPeUH/MdSnP31gNreZE5e5n5JHBROeYnKCWxN1Wcv5vyAjLAduC7wOEVl/gqpeT1H5AkaQCRmS/eS5Ik1YWIaKW06ujvZOaDRccjSRqdrAhKktRY/hBYYxIoSdqb/ia1S5KkOhQRj1BaVGaweyZKkpqUQ0MlSZIkqck4NFSSJEmSmkxVE8GIOC0iNkTExohY1M/5CyLi3vLGvXdExLF9zh8WEU9FxKerGackSZIkNZOqDQ2NiBbgZ8BbKS2jvQY4p7w0d2+fV2Tmr8v35wJ/lJmnVZy/gdJ+Tndl5mf39nzjx4/PI444YsRfhyRJkiTVg7Vr1z6emRMG07eai8XMAjZm5kMAEbECOAvYnQj2JoFlvZv4Uu4/D3gY+M1gnuyII46gs7NzBMKWJEmSpPoTEY8Otm81h4a2A5sqjrvKbXuIiI9HxM+ByyhtoEtEvBy4GPhfVYxPkiRJkppS4YvFZOZVmXkUpcTvz8rNS4ArMvOpvT02Is6PiM6I6NyyZUuVI5UkSZKkxlDNoaHdwOSK40nltoGsAP6ufP8E4N0RcRnQBjwXEU9n5pWVD8jMq4GrATo6OtwHQ5IkSZIGoZqJ4BrgmIiYQikBXAC8t7JDRByTmQ+WD08HHgTIzDdV9FkCPNU3CZQkSZIk7ZuqJYKZuTMiLgRWAy3AlzLzvohYCnRm5k3AhRHxFmAH8ATwgWrFI0mSJEkqqdr2EbXW0dGRrhoqSZIkqVlFxNrM7BhM38IXi5EkSVKVrF8JV0yDJW2l2/Uri45I0ihRzTmCkiRJKsr6lfCti2BHT+l4+6bSMcCM+cXFJWlUsCIoSZLUiG5Z+nwS2GtHT6ldUtMzEZQkSWpE27uG1i6pqZgISpIkNaJxk4bWLqmpmAhKkqTacgGT2pi9GMa27tk2trXUXgt+ztKo5mIxkiSpdlzApHZ6389blpaGg46bVEoCa/E++zlLo577CEqSpNq5YlopKehr3GT45E9qH4+qw89ZKoT7CEqSpNHJBUyag5+zNOqZCEqSpNpxAZPm4OfcPJwLWrdMBCVJUu0UvYCJasPPuTn0zgXdvgnI5+eCmgzWBRNBSZJUOzPmw5nLS3PFiNLtmctdQKTRFPk5W6GqnVuWPr8gUK8dPaV2jXquGipJkmprxnwTv2ZQxOfsaqW15VzQumZFUJIkSY3BClVtORe0rpkISpIkqTFYoaot54LWNRNBSZIkNQYrVLXlnN+65hxBSZIkNYbZi/ecIwhWqKrNOb91y4qgJEnNytUV1WisUEmDVtWKYEScBvwN0AL8fWYu63P+AuDjwC7gKeD8zLw/It4KLAP2A54FFmbmrdWMVZKkpuLqimpUVqikQalaRTAiWoCrgLcDxwLnRMSxfbp9PTOnZ+ZxwGXA58rtjwNnZuZ04APAV6sVpyRJTcnVFSWpqVVzaOgsYGNmPpSZzwIrgLMqO2TmrysODwCy3L4uMzeX2+8DWiPipVWMVZI0GjhUsXZcXVGSmlo1h4a2A5sqjruAE/p2ioiPA5+iNAz01H6u8y7gR5n5TDWClCSNEg5VrK1xk0rvcX/tDWrVum4uX72Bzdt6mNjWysI5U5k3s73osCSpEIUvFpOZV2XmUcDFwJ9VnouI1wJ/DXysv8dGxPkR0RkRnVu2bKl+sJKk6nGoYm012f5fq9Z1c8mN99K9rYcEurf1cMmN97JqXXfRoUlSIaqZCHYDkyuOJ5XbBrICmNd7EBGTgH8C3p+ZP+/vAZl5dWZ2ZGbHhAkTRiBkSVJhHKpYW022uuLlqzfQs2PXHm09O3Zx+eoNBUUkScWq5tDQNcAxETGFUgK4AHhvZYeIOCYzHywfng48WG5vA74NLMrMO6sYoySpP+tXlipx27tKQwVnL65+gtCEQxUL10SrK27e1jOk9hFXxJ8pSdqLqlUEM3MncCGwGngAWJmZ90XE0oiYW+52YUTcFxH3UJon+IHeduBoYHFE3FP++a1qxSpJqtA7V2/7JiCfn6tX7YVbmmyoomprYlvrkNpHVFF/piRpLyIzi45hRHR0dGRnZ2fRYUhS/bti2gCVucnwyZ9U97mtmqhKeucIVg4PbR3bwqVnT6/+gjFF/pmS1FQiYu3/3979h1ld1wn/f74aIKcsxx+sxSCGoXxTRNl7wNT8gZrQ6iJ5m8Jm5equtRu11RWJd7vG5X3vLbd0l+vqtVe2bbr3t0J0CW37wXqjVpYmGAYLRJI/cgZLJccyR0R43X/MGRxGBs7AfOYzc87zcV1ccz7vz+dzzuuc85nDvM77/Xq/M7OlmmMLXVBekjQElVmrV0dDFTWwZk5qpvnJf+ewny7kj/IZno6RPPnHc5k8aXrxD279q6RBqPRZQyVJg0xvNXnW6mkoW72YyWs+x1t4htcFvIVnmLzmcwMzPNPfKal21NB6tyaCkqSdWaunWlTm8iT+Tkn9r4yErMbqfU0EJUk7q7NlBVQnyh7y7O+U1H/KSshqbL1bawQlSa9lrZ5qTdnLk/g7JfWf3SVkRf6e1Vi9rz2CkiSp9jk8UzVs6ao2Tl5wN2PnfZuTF9zN0lVtZYdUrLISshqr9zURlCRJtc/hmapRXUujtLV3kEBbewdXLllT28lgWQlZjX2h5NBQSZLK5NqJA8fhmapBC5dt2Gl9TICOrdtYuGxD8WtkluXMqzprArsPDx2IhKzr86NGPrNNBCVJKkvXhAddf8x0TXgAQ/YPC0kDa1N7R5/aa0KZCVkNfaFkIihJUlnKmvBAUs0Y1dRI2y6SvlFNjbs4uobUUEJWFmsEJUkqS43NQCdp4M2dNp7G4Q07tTUOb2DutPElRaShwh5B1QZrbCQNRWUvaSDVoKWr2li4bAOb2jsY1dTI3Gnja7dWDnY8t3p6zuofJoIa+qyxkTRUlTXhgVSjumbQ7Jo8pWsGTaCmE6OZk5pr+vkNJrX0RYNDQzX07a7GRpIGM5c0kPrV7mbQHAh1t55fnam1pTrsEdTQZ42NpKHMCQ+kflPmDJr12htZT2ptqQ57BDX0lbWoqCRJGlR6mylzIGbQLLs3UsWrtaU6TAQ19J15VWdNTXfW2EiSVHfKnEGz1pIEvVaZXzQUwURQQ581NqpVqxfDFyfA/KbOn6sXlx2RJA1qMyc1c835x9Lc1EgAzU2NXHP+sQMybK/WkoRq1VNdZK0t1RGZWdydR0wH/gFoAP45Mxf02P8R4KPANuAF4PLMXFfZdyVwWWXfxzNz2e4eq6WlJVeuXNn/T0KSytBzNlzo7On2Sw5JGpR61ghCZ5IwUIloGer1OQ/mWUMj4qHMbKnq2KISwYhoAH4BvBtoBVYAs7sSvcoxb87M31VuzwD+OjOnR8TRwDeAKcAo4P8CR2XmNnphIiippnxxQi/ryx0Gn/zPgY9HkrRHgz1J6G8nL7ibtl0MfW1uauRH884oISL1JREsctbQKcDGzHy0EtQi4DxgRyLYlQRWvBHoykrPAxZl5hbgsYjYWLm/+wuMV+o7F7JXUZwNV9IQV29JEZS4nl9Jf49YFzm0FZkINgPdv85uBU7oeVBEfBT4FDAC6PrqoBl4oMe5tf3JoaHHhexVpANG99Ij6Gy4GvrqMUEoS1mvtUspDKAS/x4Z1dS4yx7BWq+LrBWlTxaTmTdm5tuBK4C/7cu5EXF5RKyMiJXPPPNMMQFKvXEhexXJ2XBVo2ptQebBrMzX2qUUBlCJf4/U2uQp9abIRLANOKzb9uhKW28WATP7cm5m3pSZLZnZMnLkyH0MV+ojh+7VjxwpEEMAACAASURBVDJm73Q2XNUoE4SBU+Zr7ZDBAVTi3yNlztKqfVfk0NAVwJERMZbOJG4W8GfdD4iIIzPzkcrmOUDX7TuBr0fEF+icLOZI4MECY5X67MXGt/CGjqd23V5CPCpImUOAJ15o4qeaY4IwcMp8rR0yOIBKLiUorS5S+6ywHsHMfAWYAywD1gOLM3NtRFxdmSEUYE5ErI2Ih+msE/xQ5dy1wGI6J5b5HvDR3c0YKpXh2q0X8WKO2KntxRzBtVsvKikiFcIhwFK/qte11spQ5mvtkMEBVKelBPW0fmFRCq0RzMzvZOZRmfn2zPz7SttVmXln5fbfZOYxmXl8Zk6tJIBd5/595bzxmfndIuOU9sYtL0xh3ta/oHX7IWzPoHX7Iczb+hfc8sKUskNTf3IIsNSvTBAGTpmvtUMGB1AdlhJYa9w/ihwaKtW0UU2N3Nn+Lu58+V07tTcP1Lfa9bh0RRnP2dk7pX7VlQg4a2jxyn6tHTI4gOqslGB39a9ec9UzEZT20txp43eaGhsG8Fvtely6oqznfOZVOz8u1MWQm3rkkgYDxwRh4PhaqxZZa9w/Sl8+QhqqSh32Uo91a2U95zocclOPHGYkSUOHtcb9wx5BaR+U9U1rPt9K9KG9JpRZq1dnQ27qkcOMJGnoKHVUVg0xEZSGoN9wCG/hmV7aa5S1enWhrOGZDjNS0Rx6LPWfsutfa4WJoDQEXfPy+7hm+D/zhnh5R9uLOYJrtr6PfygxrkJZq1fzuoZndn3D2zU8Eyj8P3fXPFORyry2pVpl/eu+s0ZQGoJWvvndu1y6YuWb3112aMWxVq/m7W54ZtFc0kBFKvPalqTe2CMoDUGdY+Nf3mnpisbhDVxT63+0WqtX08ocnukwIxXJoceSBiMTQWkI8o9W1aKyh2fObPgRM19/NezXCq8fDQ1XAX7xoH1X9rUtSbtiIigNUY6NV61xbU7VKmc4lDQYWSMoSRoUXJtTtarUa1uSemGPoCRp0Citp7vMdSpVFxzFIWmwsUdQkqTe1qN0nUpJfbF6MXxxAsxv6vy5enHZEUm9MhGUJOnMqzrXpezOdSol9UVXrfHzTwL5aq2xyaAGKRNBSZJcp1LSvrLWWEOMNYKStAdLV7W5VEc9cJ1KSfvCWmMNMfYISuqbOqt/WLqqjSuXrKGtvYME2to7uHLJGpauais7NEnSYGKtsYYYE0FJ1avD+oeFyzbstPYXQMfWbSxctqHwx166qo2TF9zN2Hnf5uQFd5t8StJgZq2xhphCE8GImB4RGyJiY0TM28X+T0XEuohYHRHLI+LwbvuujYi1EbE+Iq6PiCgyVklVqMP6h03tHX1q7y/2RErSEGOtsYaYwmoEI6IBuBF4N9AKrIiIOzNzXbfDVgEtmfliRPwVcC1wUUScBJwMTKwcdx9wGnBvUfFKqkId1j+MamqkbRdJ36imxl0c3X921xNpfWJtsQZVqiHWGmsIKbJHcAqwMTMfzcyXgUXAed0PyMx7MvPFyuYDQNcg6gT2A0YArweGA78pMFZJ1ajD+oe508bTOLxhp7bG4Q3MnTa+0MctqydSA6vsnl+HH0tS/SoyEWwGnuy23Vpp681lwHcBMvN+4B7gqcq/ZZm5vqA4JVWrDusfZk5q5przj6W5qZEAmpsaueb8Ywvvsemtx7HonkgNrLJrUB1+LEn1a1AsHxERFwMtdA7/JCLGAe/g1R7CuyLilMz8YY/zLgcuBxgzZszABSzVq67hLsuv7hwOesDoziSwxofBzJzUPOBD9eZOG8+VS9bslCQMRE+kBlaZPb8OP5ak+lZkItgGHNZte3SlbScRcRbwWeC0zNxSaX4v8EBmvlA55rvAicBOiWBm3gTcBNDS0pL9/QQk7YL1DwOi6w9xa8dqW1k1qODwY0mqd0UmgiuAIyNiLJ0J4Czgz7ofEBGTgC8B0zPz6W67fgX8ZURcAwSdPYXXFRirJA06ZfREamCV2fNbZhIqSSpfYTWCmfkKMAdYBqwHFmfm2oi4OiJmVA5bCOwP3BYRD0fEnZX224FfAmuAnwE/y8xvFRWrJEllKKsGFcqbCEmSNDhEZm2MqGxpacmVK1eWHYYkSUOGS1dIUm2JiIcys6WaYwfFZDGSJGngOfxYkupXkctHSJIkSZIGIRNBSZIkSaozJoKSJEmSVGdMBCVJkiSpzpgISpIkSVKdcdZQ9RunIZckSZKGBhNB9Yulq9q4cskaOrZuA6CtvYMrl6wBMBmUJEmSBhmHhqpfLFy2YUcS2KVj6zYWLttQUkSSJEmSemMiqH6xqb2jT+2SJEmSyrPHoaER8THg/8/M5wYgHg1Ro5oaadtF0jeqqbGEaFQka0ElSZKGvmp6BA8FVkTE4oiYHhFRdFAaeuZOG0/j8Iad2hqHNzB32viSIlIRumpB29o7SF6tBV26qq3s0CRJktQHe0wEM/NvgSOBrwCXAI9ExP+MiLcXHJuGkJmTmrnm/GNpbmokgOamRq45/1h7imqMtaD1Y+mqNk5ecDdj532bkxfcbbIvSVKNqWrW0MzMiPg18GvgFeBA4PaIuCszP1NkgBo6Zk5qNvGrcdaC1gdnAZYkqfbtsUcwIv4mIh4CrgV+BBybmX8F/BfgvxYcn/aC3+SrKL3VfFoLWlvs+ZUkqfZVUyN4EHB+Zk7LzNsycytAZm4Hzi00OvWZNVwqkrWg9cGeX0mSal81ieB3gd92bUTEmyPiBIDMXF9UYNo7fpOvIlkLWh/s+ZUkqfZVUyP4T8Afd9t+YRdtuxQR04F/ABqAf87MBT32fwr4CzrrDp8BLs3MJyr7xgD/DBwGJPAnmfl4FfHWNb/JV9GsBa19c6eN36lGEOz5lSSp1lTTIxiZmV0blSGh1aw/2ADcCLwHOBqYHRFH9zhsFdCSmROB2+msQ+zyr8DCzHwHMAV4uopY657f5EvaV/b8SpJU+6rpEXw0Ij5OZy8gwF8Dj1Zx3hRgY2Y+ChARi4DzgHVdB2TmPd2OfwC4uHLs0cCwzLyrctwLVTye8Jt8Sf3Dnl9JkmpbNT2CHwFOAtqAVuAE4PIqzmsGnuy23Vpp681ldNYjAhwFtEfEkohYFRELKz2M2gO/yZckSZK0J3vsEczMp4FZRQYRERcDLcBp3eI6BZgE/Aq4lc7F7L/S47zLqSSlY8aMKTLEIcVv8iVJkiTtTjW1fvvR2Vt3DLBfV3tmXrqHU9vonOily+hKW8/7Pwv4LHBaZm6pNLcCD3cbVroUeCc9EsHMvAm4CaClpSWRJEmSJO1RNUND/w/wFmAa8H06E7rfV3HeCuDIiBgbESPo7FW8s/sBETEJ+BIwo9Lz2P3cpogYWdk+g261hZIkSZKkvVdNIjguM/8O+ENm3gKcQ2ed4G5l5ivAHGAZsB5YnJlrI+LqiJhROWwhsD9wW0Q8HBF3Vs7dBnwaWB4Ra4AAvtzH56aBtnoxfHECzG/q/Ll6cdkRSZIkSdqFamYN3Vr52R4RE4BfA39UzZ1n5neA7/Rou6rb7bN2c+5dwMRqHkeDwOrF8K2Pw9bKeoXPP9m5DTDxwvLikiRJkvQa1fQI3hQRBwJ/S+fQznXA/yo0Kg09y69+NQnssrWjs12SJEnSoLLbHsGIeB3wu8x8DvgBcMSARKWh5/nWvrVLkiRJKs1uewQzczvwmQGKRf2ljFq9A0b3rV2SJElSaaoZGvp/I+LTEXFYRBzU9a/wyLR3umr1nn8SyFdr9YpOBs+8CoY37tw2vLGzXZIkSdKgUs1kMRdVfn60W1viMNHBaXe1ekVO2tJ138uv7hwOesDoziTQiWIkSZKkQWePiWBmjh2IQNRPyqzVm3ihiZ8kSZI0BOwxEYyID+6qPTP/tf/D0T47YHRlWOgu2iVJkiSJ6moEJ3f7dwowH5ixuxNUImv1JEmSJO1BNUNDP9Z9OyKagEWFRaR9Y62eJEmSpD2oZrKYnv4AWDc4mNVhrd7SVW0sXLaBTe0djGpqZO608cyc1Fx2WJIkSdKgVE2N4LfonCUUOoeSHg0MwMJ0UnWWrmrjyiVr6Ni6DYC29g6uXLIGwGRQkiRJ2oVqegQ/3+32K8ATmTkAU1BK1Vm4bMOOJLBLx9ZtLFy2wURQkiRJ2oVqEsFfAU9l5ksAEdEYEW/LzMcLjUyq0qb2jj61S5IkSfWumllDbwO2d9veVmmTBoVRTY19apckSZLqXTWJ4LDMfLlro3J7RHEhSX0zd9p4Goc37NTWOLyBudPGlxSRJEmSNLhVkwg+ExE71g2MiPOAZ4sLSeqbmZOaueb8Y2luaiSA5qZGrjn/WOsDJUmSpF5UUyP4EeBrEXFDZbsV+GBxIUl9N3NSs4mfJEmSVKVqFpT/JfDOiNi/sv1C4VFJkiRJkgqzx6GhEfE/I6IpM1/IzBci4sCI+B/V3HlETI+IDRGxMSLm7WL/pyJiXUSsjojlEXF4j/1vjojWbr2RkurY0lVtnLzgbsbO+zYnL7ibpavayg5JkiRpSKqmRvA9mdnetZGZzwF/sqeTIqIBuBF4D52L0M+OiKN7HLYKaMnMicDtwLU99v934AdVxCipxi1d1caVS9bQ1t5BAm3tHVy5ZI3JoCRJ0l6oJhFsiIjXd21ERCPw+t0c32UKsDEzH63MNLoIOK/7AZl5T2a+WNl8ABjd7XH+C3Ao8B9VPJakGrdw2QY6tm7bqa1j6zYWLttQUkSSJElDVzWTxXwNWB4RXwUCuAS4pYrzmoEnu223Aifs5vjLgO8CRMTrgP8NXAycVcVjSapxm9o7+tQuSZKk3lUzWcz/ioif0ZmQJbAMOHz3Z/VNRFwMtACnVZr+GvhOZrZGxO7Ouxy4HGDMmDH9GZKkQWZUUyNtu0j6RjU1lhCNJEnS0FbN0FCA39CZBL4POANYX8U5bcBh3bZHV9p2EhFnAZ8FZmTmlkrzicCciHgc+DzwwYhY0PPczLwpM1sys2XkyJFVPhVJQ9HcaeNpHN6wU1vj8AbmThtfUkSSJElDV689ghFxFDC78u9Z4FYgMnNqlfe9AjgyIsbSmQDOAv6sx2NMAr4ETM/Mp7vaM/P93Y65hM4JZV4z66ik+tG1TuTCZRvY1N7BqKZG5k4b7/qRkiRJe2F3Q0N/DvwQODczNwJExCervePMfCUi5tA5lLQB+JfMXBsRVwMrM/NOYCGwP3BbZQjorzJzxt49FUm1buakZhM/SZKkfrC7RPB8Onvx7omI79E562fvBXu7kJnfAb7To+2qbrf3OBFMZt4M3NyXx5UkSZIk9a7XGsHMXJqZs4D/D7gH+ATwRxHxTxFx9kAFKEmSJEnqX3ucLCYz/5CZX8/MP6VzwpdVwBWFRyZJkiRJKkS1s4YCkJnPVWbqPLOogCRJkiRJxepTIihJkiRJGvpMBCVJkiSpzpgISpIkSVKdMRGUJEmSpDpjIihJkiRJdcZEUJIkSZLqjImgJEmSJNUZE0FJkiRJqjMmgpIkSZJUZ0wEJUmSJKnOmAhKkiRJUp0xEZQkSZKkOmMiKEmSJEl1xkRQkiRJkuqMiaAkSZIk1ZlCE8GImB4RGyJiY0TM28X+T0XEuohYHRHLI+LwSvvxEXF/RKyt7LuoyDglSZIkqZ4UlghGRANwI/Ae4GhgdkQc3eOwVUBLZk4EbgeurbS/CHwwM48BpgPXRURTUbFKkiRJUj0pskdwCrAxMx/NzJeBRcB53Q/IzHsy88XK5gPA6Er7LzLzkcrtTcDTwMgCY5UkSZKkulFkItgMPNltu7XS1pvLgO/2bIyIKcAI4Jf9Gp0kSZIk1alhZQcAEBEXAy3AaT3a3wr8H+BDmbl9F+ddDlwOMGbMmAGIVJIkSZKGviJ7BNuAw7ptj6607SQizgI+C8zIzC3d2t8MfBv4bGY+sKsHyMybMrMlM1tGjnTkqCRJkiRVo8hEcAVwZESMjYgRwCzgzu4HRMQk4Et0JoFPd2sfAXwT+NfMvL3AGCVJkiSp7hSWCGbmK8AcYBmwHlicmWsj4uqImFE5bCGwP3BbRDwcEV2J4oXAqcAllfaHI+L4omKVJEmSpHoSmVl2DP2ipaUlV65cWXYYkiRJklSKiHgoM1uqObbQBeUlSZIkSYOPiaAkSZIk1RkTQUmSJEmqMyaCkiRJklRnTAQlSZIkqc6YCEqSJElSnTERlCRJkqQ6YyIoSZIkSXXGRFCSJEmS6oyJoCRJkiTVGRNBSZIkSaozJoKSJEmSVGdMBCVJkiSpzpgISpIkSVKdMRGUJEmSpDpjIihJkiRJdcZEUJIkSZLqjImgJEmSJNWZQhPBiJgeERsiYmNEzNvF/k9FxLqIWB0RyyPi8G77PhQRj1T+fajIOCVJkiSpnhSWCEZEA3Aj8B7gaGB2RBzd47BVQEtmTgRuB66tnHsQ8DngBGAK8LmIOLCoWCVJkiSpnhTZIzgF2JiZj2bmy8Ai4LzuB2TmPZn5YmXzAWB05fY04K7M/G1mPgfcBUwvMFZJkiRJqhtFJoLNwJPdtlsrbb25DPjuXp4rSZIkSarSsLIDAIiIi4EW4LQ+nnc5cDnAmDFjCohMkiRJkmpPkT2CbcBh3bZHV9p2EhFnAZ8FZmTmlr6cm5k3ZWZLZraMHDmy3wKXJEmSpFpWZCK4AjgyIsZGxAhgFnBn9wMiYhLwJTqTwKe77VoGnB0RB1YmiTm70iZJkiRJ2keFDQ3NzFciYg6dCVwD8C+ZuTYirgZWZuadwEJgf+C2iAD4VWbOyMzfRsR/pzOZBLg6M39bVKySJEmSVE8iM8uOoV+0tLTkypUryw5DkiRJkkoREQ9lZks1xxa6oLwkSZIkafAxEZQkSZKkOmMiKEmSJEl1ZlCsI1iLlq5qY+GyDWxq72BUUyNzp41n5qTmssOSJEmSJBPBIixd1caVS9bQsXUbAG3tHVy5ZA2AyaAkSZKk0jk0tAALl23YkQR26di6jYXLNpQUkSRJkiS9ykSwAJvaO/rULkmSJEkDyUSwAKOaGvvULkmSJEkDyUSwAHOnjadxeMNObY3DG5g7bXxJEUmSJEnSq5wspgBdE8I4a6gkSZKkwchEsCAzJzWb+EmSJEkalBwaKkmSJEl1xkRQkiRJkuqMiaAkSZIk1RkTQUmSJEmqMyaCkiRJklRnTAQlSZIkqc6YCEqSJElSnSk0EYyI6RGxISI2RsS8Xew/NSJ+GhGvRMQFPfZdGxFrI2J9RFwfEVFkrJIkSZJULwpLBCOiAbgReA9wNDA7Io7ucdivgEuAr/c49yTgZGAiMAGYDJxWVKySJEmSVE+GFXjfU4CNmfkoQEQsAs4D1nUdkJmPV/Zt73FuAvsBI4AAhgO/KTBWSZIkSaobRSaCzcCT3bZbgROqOTEz74+Ie4Cn6EwEb8jM9f0foiRJkgbK1q1baW1t5aWXXio7FGlI22+//Rg9ejTDhw/f6/soMhHcaxExDngHMLrSdFdEnJKZP+xx3OXA5QBjxowZ2CAlSZLUJ62trbzpTW/ibW97G07/IO2dzGTz5s20trYyduzYvb6fIieLaQMO67Y9utJWjfcCD2TmC5n5AvBd4MSeB2XmTZnZkpktI0eO3OeAJUmSVJyXXnqJgw8+2CRQ2gcRwcEHH7zPPetFJoIrgCMjYmxEjABmAXdWee6vgNMiYlhEDKdzohiHhkqSJA1xJoHSvuuP36PCEsHMfAWYAyyjM4lbnJlrI+LqiJgBEBGTI6IVeB/wpYhYWzn9duCXwBrgZ8DPMvNbRcUqSZKk+vC9732P8ePHM27cOBYsWLDLY7Zs2cJFF13EuHHjOOGEE3j88cd37LvmmmsYN24c48ePZ9myZXt8vJtvvpk5c+b0V/h9dt111/Hiiy+W9vj9rZr375Of/CTHH388xx9/PEcddRRNTU079jU0NOzYN2PGjD0+3uOPP86ECRP6Lf6+Wrp0KevWrdvzgXuh0BrBzPwO8J0ebVd1u72CV+sAux+zDfhwkbFJkiRpcFu6qo2Fyzawqb2DUU2NzJ02npmTmvf6/rZt28ZHP/pR7rrrLkaPHs3kyZOZMWMGRx+98wpnX/nKVzjwwAPZuHEjixYt4oorruDWW29l3bp1LFq0iLVr17Jp0ybOOussfvGLX9DQ0NDnWF555RWGDSt+uo7rrruOiy++mDe84Q2FP9ZrrF4My6+G51vhgNFw5lUw8cK9vrtq378vfvGLO27/4z/+I6tWrdqx3djYyMMPP7zXMXQZqPdv6dKlnHvuua95jv2h0AXlJUmSpL2xdFUbVy5ZQ1t7Bwm0tXdw5ZI1LF1V7ZQTr/Xggw8ybtw4jjjiCEaMGMGsWbO44447XnPcHXfcwYc+9CEALrjgApYvX05mcscddzBr1ixe//rXM3bsWMaNG8eDDz74mvO/+tWvctRRRzFlyhR+9KMf7Wi/5JJL+MhHPsIJJ5zAZz7zGX77298yc+ZMJk6cyDvf+U5Wr14NwPz58/nABz7AiSeeyJFHHsmXv/xloHOSkLlz5zJhwgSOPfZYbr31VgDuvfdezj333B2PM2fOHG6++Wauv/56Nm3axNSpU5k6depev257ZfVi+NbH4fkngez8+a2Pd7bvpWrfv+6+8Y1vMHv27D49zkMPPcRxxx3Hcccdx4033rij/eabb2bGjBmcccYZnHnmmbt9P0499VTOOeccxo8fz0c+8hG2b9++I55jjz2WCRMmcMUVV+y47/3333/H7dtvv51LLrmEH//4x9x5553MnTuX448/nl/+8pd9eh57MihnDZUkSVJ9W7hsAx1bt+3U1rF1GwuXbdjrXsG2tjYOO+zVuQxHjx7NT37yk90eN2zYMA444AA2b95MW1sb73znO3c6v61t58T0qaee4nOf+xwPPfQQBxxwAFOnTmXSpEk79re2tvLjH/+YhoYGPvaxjzFp0iSWLl3K3XffzQc/+MEdvVWrV6/mgQce4A9/+AOTJk3inHPO4f777+fhhx/mZz/7Gc8++yyTJ0/m1FNP7fX5fvzjH+cLX/gC99xzD4cccshevWZ7bfnVsLVj57atHZ3te9krWO371+WJJ57gscce44wzztjR9tJLL9HS0sKwYcOYN28eM2fOfM15f/7nf84NN9zAqaeeyty5c3fa99Of/pTVq1dz0EEH8W//9m+9vh8PPvgg69at4/DDD2f69OksWbKEk046iSuuuIKHHnqIAw88kLPPPpulS5fuMgaAk046iRkzZnDuuedywQUX9Om1qoY9gpIkSRp0NrV39Kl9sPjJT37C6aefzsiRIxkxYgQXXXTRTvvf97737RhKet999/GBD3wAgDPOOIPNmzfzu9/9DoDzzjuPxsZGDjnkEKZOncqDDz7Ifffdx+zZs2loaODQQw/ltNNOY8WKFQP7BKv1fGvf2guwaNEiLrjggp2G7j7xxBOsXLmSr3/963ziE594TS9be3s77e3tOxK6rveny7vf/W4OOugggN2+H1OmTOGII46goaGB2bNnc99997FixYod18awYcN4//vfzw9+8IMiX4LdMhEsyurF8MUJML+p8+c+dINLkiTVm1FNjX1qr0ZzczNPPvnkju3W1laam1/bu9j9uFdeeYXnn3+egw8+uOrzd+eNb3xjVcf1nBVyd7NEDhs2bMfQQ2CflxXoFwe8ZhqQ3bdXoa+v/6JFi14zLLTr+COOOILTTz99p/rBahTx/vXcP1Dvn4lgEQoYEy1JklRP5k4bT+PwnSdhaRzewNxp4/f6PidPnswjjzzCY489xssvv8yiRYt2OXPkjBkzuOWWW4DOeq0zzjiDiGDGjBksWrSILVu28Nhjj/HII48wZcqUnc494YQT+P73v8/mzZvZunUrt912W6/xnHLKKXzta18DOuvKDjnkEN785jcDnXWKL730Eps3b+bee+9l8uTJnHLKKdx6661s27aNZ555hh/84AdMmTKFww8/nHXr1rFlyxba29tZvnz5jsd405vexO9///u9fs322plXwfAeSfvwxs72vVTt+wfw85//nOeee44TT3x1KfLnnnuOLVu2APDss8/yox/96DWTsDQ1NdHU1MR9990HsOP92ZXe3g/oHBr62GOPsX37dm699Vbe9a53MWXKFL7//e/z7LPPsm3bNr7xjW9w2mmnAXDooYeyfv16tm/fzje/+c0dj1Hk+2eNYBEKGBMtSZJUT7rqAPtz1tBhw4Zxww03MG3aNLZt28all17KMcccA8BVV11FS0sLM2bM4LLLLuMDH/gA48aN46CDDmLRokUAHHPMMVx44YUcffTRDBs2jBtvvPE1M4a+9a1vZf78+Zx44ok0NTVx/PHH9xrP/PnzufTSS5k4cSJveMMbdiSfABMnTmTq1Kk8++yz/N3f/R2jRo3ive99L/fffz/HHXccEcG1117LW97yFgAuvPBCJkyYwNixY3eqSbz88suZPn06o0aN4p577tnr167Puv7m7cdZQ6t9/6CzN3DWrFk79bStX7+eD3/4w7zuda9j+/btzJs3b5ezcX71q1/l0ksvJSI4++yze42nt/fj5z//OZMnT2bOnDls3LiRqVOn8t73vpfXve51LFiwgKlTp5KZnHPOOZx33nkALFiwgHPPPZeRI0fS0tLCCy+8AMCsWbP4y7/8S66//npuv/123v72t+/169dTZGa/3VmZWlpacuXKlWWH0Wl+E7Cr1zVgfvtARyNJkjQorF+/nne84x1lhzHozZ8/n/33359Pf/rTZYeivXDvvffy+c9/nn//938v9HF29fsUEQ9lZks15zs0tAgFjImWJEmSpP7i0NAinHlVZ01g9+Gh+zgmWpIkSfVh/vz5ZYegfXD66adz+umnlx3GHtkjWISJF8KfXg8HHAZE588/vd76QEmSJEmDgj2CRZl4oYmfJElSD5m5x6n0Je1ef8zzYo+gJEmSBsR+++3H5s2b++WPWKleZSabN29mv/3226f7sUdQjiCkMwAABXxJREFUkiRJA2L06NG0trbyzDPPlB2KNKTtt99+jB69bxNRmghKkiRpQAwfPpyxY8eWHYYkHBoqSZIkSXXHRFCSJEmS6oyJoCRJkiTVmaiVWZsi4hngibLj2IVDgGfLDkI1zWtMRfL6UpG8vlQkry8VabBeX4dn5shqDqyZRHCwioiVmdlSdhyqXV5jKpLXl4rk9aUieX2pSLVwfTk0VJIkSZLqjImgJEmSJNUZE8Hi3VR2AKp5XmMqkteXiuT1pSJ5falIQ/76skZQkiRJkuqMPYKSJEmSVGdMBAsUEdMjYkNEbIyIeWXHo9oSEY9HxJqIeDgiVpYdj4a+iPiXiHg6Iv6zW9tBEXFXRDxS+XlgmTFq6Orl+pofEW2Vz7GHI+JPyoxRQ1dEHBYR90TEuohYGxF/U2n3M0z7bDfX15D+DHNoaEEiogH4BfBuoBVYAczOzHWlBqaaERGPAy2ZORjXsNEQFBGnAi8A/5qZEypt1wK/zcwFlS+0DszMK8qMU0NTL9fXfOCFzPx8mbFp6IuItwJvzcyfRsSbgIeAmcAl+BmmfbSb6+tChvBnmD2CxZkCbMzMRzPzZWARcF7JMUlSrzLzB8BvezSfB9xSuX0Lnf/xSX3Wy/Ul9YvMfCozf1q5/XtgPdCMn2HqB7u5voY0E8HiNANPdttupQYuGA0qCfxHRDwUEZeXHYxq1qGZ+VTl9q+BQ8sMRjVpTkSsrgwdddie9llEvA2YBPwEP8PUz3pcXzCEP8NMBKWh612Z+cfAe4CPVoZdSYXJzloC6wnUn/4JeDtwPPAU8L/LDUdDXUTsD/wb8InM/F33fX6GaV/t4voa0p9hJoLFaQMO67Y9utIm9YvMbKv8fBr4Jp3DkaX+9ptKbURXjcTTJcejGpKZv8nMbZm5Hfgyfo5pH0TEcDr/SP9aZi6pNPsZpn6xq+trqH+GmQgWZwVwZESMjYgRwCzgzpJjUo2IiDdWipWJiDcCZwP/ufuzpL1yJ/Chyu0PAXeUGItqTNcf6BXvxc8x7aWICOArwPrM/EK3XX6GaZ/1dn0N9c8wZw0tUGUK2euABuBfMvPvSw5JNSIijqCzFxBgGPB1ry/tq4j4BnA6cAjwG+BzwFJgMTAGeAK4MDOd8EN91sv1dTqdQ6oSeBz4cLd6LqlqEfEu4IfAGmB7pfm/0VnH5WeY9slurq/ZDOHPMBNBSZIkSaozDg2VJEmSpDpjIihJkiRJdcZEUJIkSZLqjImgJEmSJNUZE0FJkiRJqjMmgpIk9RAR2yLi4W7/5vXjfb8tIobUWlOSpNozrOwAJEkahDoy8/iyg5AkqSj2CEqSVKWIeDwiro2INRHxYESMq7S/LSLujojVEbE8IsZU2g+NiG9GxM8q/06q3FVDRHw5ItZGxH9ERGNpT0qSVJdMBCVJeq3GHkNDL+q27/nMPBa4Abiu0vaPwC2ZORH4GnB9pf164PuZeRzwx8DaSvuRwI2ZeQzQDvzXgp+PJEk7icwsOwZJkgaViHghM/ffRfvjwBmZ+WhEDAd+nZkHR8SzwFszc2ul/anMPCQingFGZ+aWbvfxNuCuzDyysn0FMDwz/0fxz0ySpE72CEqS1DfZy+2+2NLt9jas2ZckDTATQUmS+uaibj/vr9z+MTCrcvv9wA8rt5cDfwUQEQ0RccBABSlJ0u74DaQkSa/VGBEPd9v+XmZ2LSFxYESsprNXb3al7WPAVyNiLvAM8OeV9r8BboqIy+js+fsr4KnCo5ckaQ+sEZQkqUqVGsGWzHy27FgkSdoXDg2VJEmSpDpjj6AkSZIk1Rl7BCVJkiSpzpgISpIkSVKdMRGUJEmSpDpjIihJkiRJdcZEUJIkSZLqjImgJEmSJNWZ/wc8pReQfc7bywAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f49b70dd208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question\n",
    "Explain what you see in this experiment. What does it suggest about dropout?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answer\n",
    "Model with dropout has lower train accuracy than model without dropout, but in general higer validation accuracy. This suggest dropout is a regularization approach."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}