Stock Prices from News.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/Users/Abhi/anaconda3/lib/python3.6/site-packages/nltk/twitter/__init__.py:20: UserWarning: The twython library has not been installed. Some functionality from the twitter package will not be available.\n",
      "  warnings.warn(\"The twython library has not been installed. \"\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effect of News on Stock Price Movement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project consists of 2 main parts, sentiment analysis and multivariate regression. \n",
    "\n",
    "## Problem Statement\n",
    "Given a stock's previous data and some news information, can we predict if it's price will go up or down?\n",
    "\n",
    "## Simplifying the Problem\n",
    "\n",
    "Since we just want to maximize stock return, we can reduce this problem to binary classification, 0 representing a loss (net change < 0) and 1 representing a win (net change ≥ 0). Now, the second part of this problem is also classification.\n",
    "\n",
    "In order to effectively predict the effect of news on stock prices, the news must have a sentiment score, which can be applied to a separate model that predicts the effect of this sentiment on the stock's price.\n",
    "\n",
    "Also, due to time constraints, I've opted to train and test the model on a single index, the Dow Jones industrial average.\n",
    "\n",
    "I began by gathering news data, as well as corresponding stock data, from this link: https://www.kaggle.com/aaron7sun/stocknews#RedditNews.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DOW Jones From 2008 to 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD8CAYAAAC2PJlnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VNXZwPHfyb6TkIQEQjDsu7hEQFRUBAX1FVu1Vfsq9VVpXdpqq3Wrr1qXarW1WpfWKnWpQq1LpS8qooKogBBAdoGwhy37vk7mvH/cO3f2JExCZiZ5vp8PH+6ce+6dO5bOM2d7jtJaI4QQQgQiItgPIIQQInxJEBFCCBEwCSJCCCECJkFECCFEwCSICCGECJgEESGEEAGTICKEECJgEkSEEEIETIKIEEKIgEUF+wEClZGRofPy8oL9GEIIEVbWrl1bqrXO7Kr7hW0QycvLo6CgINiPIYQQYUUpta8r7yfdWUIIIQImQUQIIUTAJIgIIYQImAQRIYQQAZMgIoQQImASRIQQQgRMgogQQoiASRARQvRqK3aVsnR7cbAfI2xJEBFC9Fhaaz7ZcoT6ZpvfOlf/7Ruu+/sabK32bnyynkOCiBCix3pg4RbmvrGWJxdvR2vtdX5PaZ11/O2Byu58tB5DgogQosd6faWR4WPLwWpO/93nvLhsF/XNNhZtPMwbq/Zx7lPLrLoPL9rG5oNVnPbop+wvqw/K89rtmi93lvgMeKGq3SCilJqnlCpWSm12KTtJKbVKKfWtUqpAKTXRLFdKqWeVUoVKqY1KqVNcrpmjlNpp/pnjUn6qUmqTec2zSinV1R9SCNGzHapsoLS2ya2surHFOt5RXMOR6kae+Pg7fvLGWm55ax33/3uzW/0NByq5+M9fUVLTxCtf7ebPn+0k7+5F1Ljc53hbsOYA17yymv/beLjb3rOzOtISeRWY6VH2e+AhrfVJwP+arwFmAcPNP3OBFwGUUn2BB4BJwETgAaVUmnnNi2Zdx3We7yWEEH412+xMefxz8h/51K18RWEpAJERisp6ZyD4cmep1z2GZCS6va5psvGHJTuA7u3m2l9utIB+Nn89dnt4tEbaDSJa6+VAuWcxkGIe9wEOmcezgde1YRWQqpTqD1wALNFal2utK4AlwEzzXIrWeqU22m+vA5d2+lMJIXqFDQcq2VDk/JJ37Ya6732jpTFzXHab9xiVncyCuZPdyg5XNlrHni0cf2qbbLR0cnDe7tKNdaiqAYC6Jhs3/WMt/1yzn2lPLeN/P9hMXZP/iQLdLdAxkduAJ5VSB4CngHvM8hzggEu9IrOsrfIiH+U+KaXmmt1nBSUlJQE+uhCiJ7C12pn9/Ndc8ZeVVtlXhaXY7ZqNRZWU1TUDcMmEAV7XPnTJWCtw5KTGk5kc63Z+9V7n7+byuo51Z533h2Vc/OxXaK0DHtN4aflu6/i9dQepb7axfn8lH20+wl3vbmJ3aR2vr9xHbFToDGcHup/ITcDtWut3lVI/AF4BpgO+xjN0AOU+aa1fAl4CyM/PD4+2nhCiyx2pamTy7z7zKr/3/U0crKzn+aW7APjp2UNJjPH+mvtBfi7xMZH8c+5kRmWn4DkU2+rSlVRZ39yhZzpa3cTR6iYG3/MhAD85ewj3zBrd4c/U2NLq9vqPS3bw3roi9voY5I+KDJ0gEuiTzAHeM4//hTHOAUZLItel3kCMrq62ygf6KBdCCL9eX7nX7zlHAAHIS08gwsdP1fiYSAAmDUmnT0K027npo/u5vXa0aNrSZGv1KvvrF7s73BWmtWbpd8aCxwm5qVa5rwASagINIoeAs83jacBO83ghcK05S2syUKW1PgwsBs5XSqWZA+rnA4vNczVKqcnmrKxrgQ8C/TBCiN4hJy2+Q/XOG51FSwcHqKePzmJ4vyQGewyyv/XNfj749mCb164oLPNZnv/Ip7z69Z52x0qunbeam95cB8DpQ9I79LyhoiNTfOcDK4GRSqkipdT1wI3AH5RSG4DHMGZXAXwI7AYKgb8BNwNorcuBh4E15p/fmmVgdI29bF6zC/ioaz6aEKInamm1W4PmT1w23m+9i0/sT2ZyLPHRkW7lN5412Gf9l+fks+SXZ1NS4916uPe9TW0+04pd3jO+HB78z1Zu/+e3bV7vOmNs4uA0xuf0cTs/fXRWm9cHU0dmZ12lte6vtY7WWg/UWr+itf5Ka32q1nqC1nqS1nqtWVdrrW/RWg/VWo/XWhe43Gee1nqY+efvLuUFWutx5jW36nBaZSOE6HavfLXHOu7fx3+LJMHssjotL40nLz+RNfdN5zcXjebudsYpbps+goykGD65fapVVtfc6jZO4ulvX+5xe/3wpePcXre17uNgZYPb64mD0/n95Sdy6glpVtkTl4136+YKJaEzOiOEEB3wdoEx0fPNGyaRGOt/blCCOaCulOKK/Fwyk2O54awhRPoaJHGRl5FIwW9mMCIr2a38mle+8Tnrytd6jgkD+3iV+XPrW+us49X3nkdSbBSj+6dw30XOYJeeFMsfrpjQ4Xt2JwkiQoiw0WrXHKlqZFi/JM4YlkGSSxAZ0z/Fra5j8Lwz0lwG3VfsKmP+6gNederNWVW/nDHCKuubGNPh9yiuNrrPEmIi6ZcSZ5Wne9wjqp3gFywSRIQQYaO0ton65lbmTMkDIDHWGSj+72dnunUBJUR3Pogsvm0qF43vb71evcd7AL3eXPiXnuT80k9PdF930r9PHP4kxxmB8NrT89zKHYFoVLbRImqvBRUsEkSEEGFj/f4KAPqbv9hdWyIREYo3b5jEjyYNAqC5C1K790uJY7xL11SajxZGrRlEXNejeLaCPAf3XTW0tDJ9dD9+fcFIt/LkuGjW3DedRT8/C4CoSCOIhFoskSAihAhJrXbN/NX73dZgvGwOYA/tlwS4BxGAuOhIJptTZIsq3AesAzX7JOeKd1/pRuqajOdLjI3i01+ezTs/Pd2rTnWj/zQllfUt5KTGE+EjOmQmx3q1QNoaBwoGCSJCiJD0/vqD3PPeJrfZWKkJMaQnxlhrOaIiI1h5zzRW3D3NqjNzXDY3nDmYOz1+2Qeqf5949j5+EUMyE6lr9l5UWNNkpEVJjIlkWL8k8vP6AvD5r8626niuRndotWuqGlrok9D+GEp6Yiwjs5J59qqTA/kYx01ohTQhhDAdNqe+rttXYZVVN7ZYrRAHz2m+0ZER/ObiMV3+PDGRETTbvLvIdh6tBWBgWoJb+ZBM53PWN9vQWnulV/l021EA9pXV0Z6YqAgWu0w7DhXSEhFCBFVtk81t6mxtk43C4hqO1hiZdD/d5tz/vLqhhT7x0V736A6xUd5BxG7XPLBwCwPT4snt671mZcXd07jxrMHYNTS2eAcgR1qUK08bdHweuhtIS0QIETQ7jtZw/tPLjeNHZhETFcH0P3zBkepGt3rzvtrD/5w5mOqGFlLighNEoiMjvNKXvL/eSIdSVNHg1coAGJAaz/iBxiLBvWV1jPaYhrx+v5HG3nVWWbiRlogQIihW7Cq1AgjAW9/so7bJ5hVAAH77f1vR2hw/CFJLJMZHS2SdOVusrfGXvuZ4R63HoHxjSyvvrC2y7h2uwvfJhRBhSWtNdWMLP563xq38wf9sZU+JMTbwxGXjSY6N4vsnO7cXKiyupa65lZT44HSgxERFeE0bPlrdyJj+Kdxy7jC/18XHGF+zDR6D8oervINlOJLuLCFEtyksrmH6H5f7Pb+71BikPmVQGl/dNY2kuCjeM7uMXl+5DyB4LRGXgfW1+yrITYunvK6ZtMS2nyfOXCPS4DFD65A5ceDNGyYdh6ftPhJEhBDd5isf+5vfd+Fovi2qZNuhanYV1xIZoRiUnkBslPsCvXrzl3zQxkRcWiKXvbiCfsmxREdGMHFw3zavcyw09Jzm+92RGgCGZiZ5XRNOpDtLCNFtKhvct5q9bfpwbpw6hIzEGEprm9hVUkduWrxXAAHnL/egzc4yWyKObL7FNU0crW4ku42UJuBcve7ZnbXlYBX9kmPbvT7USRARQhx3drux7/h765ybO82/cbI1lhAXE0mTzc6uklqvX+ZL7zgHgIJ9xhZEnutEuktMlDE7y3WA3GbXZHnsz+4p3k931nvrD1LsY++ScCPdWUKI46rZZmfK458xaUg6+8uN7V7/8t+ncvpQ5w5+sVFGENlTWseZwzLcrh+ckUhOajwHKxuMrq6+7ov6uku02RLxnGWV0E4aEl9jIr7Sx4craYkIIbrc7pJa68v2uyPVlNY2s8jcmOmVOfnMHJftVj/WnOLaZLOT57E9LTiz9WanxAUtm61SUFHfwhmPf+5WHtvO9NzYqAiUgkaX7qx31xlTe88fE7o7FnaUBBEhRIf9e/1BCvaWt1mnrLaJaX/4ggc+2MLafeVc8tzXbufP87HVq63V+cs8O8V7jMCxwdSQTO8A010ciRY9xUS2/TWqlEJrePbzQqvsznc2AjApzPZT90W6s4QQHdJss3ObuVf4L84bztWTBpHl8oVf3dhCYkwUH20+Ahi/th2/uB3mnH6Cz3u77rdxmo/ZTo49N7J8BJju4mNBOnBsCwUr6pqxu6R4aWj2n903XEgQEUJ0yJKtR63jZz7bybvrivjqLiN7bn2zjRMf/ISpIzJZvqPE7z3OGp7ps/zSk3P49bvGr3Nfs68cZQldsFthVzuWIHLyw0vcXof79F6QICKE6KB6j1/NroPDz3y6E8BnAHnnp6czqG8Cn31XzHmj+/m8d0xUBI9/fzwnmnmmPDmm/O4tqw/o2Y+nzgS2WS67JoYrGRMRQnTIEY80Ha679/11+W6/1+Xn9aVfShxXTRzkM0mhw5UTBzFmQIrPc2V1xlTYqvrmY3nkLuXvyVM7sBfIq9ed1rUPE0IkiAghOqSwpJb+feL47FdnkxQbRVWD0TI5UO6/dZCR1P4XbEc4fu3fcNaQLrlfIKJcBtBdEy6mdSCI+Oqi++WMEV3zYEEmQUQI0SGFxbWMyEpmaGYSP8jPtdJ4XPbiCr/XdNXqcscMKF+bQnWXn00bxsUn9mfNfdMZ5rLgsSOf0XMbXwjN8Z1ASBARQvilteaNVfuoqGtmT2mdtS1tQkwk9c02bK12a9X15acOBIwv2/k3Tga6Log4xkr6BzFFyIDUeJ67+hQyk2OZ4TJNuSPrVnwtSPQVWMJRu0FEKTVPKVWslNrsUf4zpdR2pdQWpdTvXcrvUUoVmucucCmfaZYVKqXudikfrJT6Rim1Uyn1T6VU17R/hRCdNn/1Ae7/92auf20N9c2tjDXHLOJjIrFr2FBUZdV1BJFzRmaSl2GsKr+xi7qfrjsjj3/fcgZTPFazB0vEMS54TIrxDhiJPSSIdORTvAo8B7zuKFBKnQvMBk7UWjcppfqZ5WOAK4GxwADgU6WUo+PveWAGUASsUUot1FpvBZ4AntZaL1BK/QW4HnixKz6cEKJz3jPXeawzd+BztCwc+aAOmkkRX73uNCYPSWfno7OINruedj924TF/2fqjlOKkXN8zt4Ll5nOGUl7XsYH+hFjvrque0hJp91NorZcrpfI8im8CHtdaN5l1HJsgzwYWmOV7lFKFwETzXKHWejeAUmoBMFsptQ2YBlxt1nkNeBAJIkIE3Vc7SynYV+FW5sgD5ejP/2iTkcpkSIYxRhDtMvjcVQEkVP165qgO13X97zIiK4kdR2t7TEsk0DGREcBZZjfUF0opx/y1HOCAS70is8xfeTpQqbW2eZT7pJSaq5QqUEoVlJT4X9AkhOi8bw9UeJU5gohjeu+KXWXkpMYzKD04SRHDzSUTBlip5BN9tE7CUaBBJApIAyYDdwJvK2MCuK+fHjqAcp+01i9prfO11vmZmb5XvgohusaB8gYyk2NxbVA4umDGDugDQFVDC+ldNI23p9vzuwt55sqTiDDXymS2k0I+XAQaRIqA97RhNWAHMszyXJd6A4FDbZSXAqlKqSiPciFEEGmt2Xakmty0eD645UyrfHT/ZACG9UtiYp6R48oxPiLappRCKcVfrzmVu2eNIjOpdweRf2OMZWAOnMdgBISFwJVKqVil1GBgOLAaWAMMN2dixWAMvi/UWmtgKXC5ed85wAeBfhghRNd46pPtbCyqIiMplvED+xBlNkdcV5xnphhfgtHtZLEV7oZkJvHTs4e2uXo/nLQ7sqOUmg+cA2QopYqAB4B5wDxz2m8zMMcMCFuUUm8DWwEbcIvWutW8z63AYiASmKe13mK+xV3AAqXUI8B64JUu/HxCiHa8/OVuTspNJd9sWRQW1/L80l2Ac/D4i1+fS3G1e9qTVHOm1sjs5G58WhFqOjI76yo/p/7bT/1HgUd9lH8IfOijfDfOGVxCiONEa83jH3/HgD7xzJmSZ5U9smgbAHsfvwiABxdusa5xrMzOSY0nJzXe7X6O7LU9pW9fBKZnzDETQrRp88EqLv7zV9ZrRxBp8pFGxDFr6Lbpw9u8p6OLS/ecnV5FAKQzU4geZMWuUk7/3WdUemS7dQ0gAF/uNKbIF1U0uJXb7ZrFW45y1vAMbpvedoLAyAjj66PVHrx8ViL4JIgI0YNc/bdvOFzVyJx5q62yvaV11vHILGP84ppXjPP7y53ntNa8tXo/ADuP1rb7XjGRRkukVWJIryZBRIgeyDWn1eZDxvEffzCBj35xllu9Wpd9ww9WNrBuv7HA8OZzh7b7Hv9z5mAuPrE/c6b43vJW9A4SRIToofaW1rGntI6qhhYAzhiWQUSE4vQh6VYd126vP326k7VmmpMfnpZLe1ITYnju6lM6tCmT6LkkiAjRQxyqdB/fOOepZZz71DL2lNQRGxVhbZ40dYSR7aGhuZVVu8tITzTK95XVsa+sngvGZlnb0QrRHpmdJUQPsdGlC8vVgYp6BvVNsKbk9k001neU1zdzpKqR0f1TsNntrNpdDsCZIZJuXYQHaYkI0UOU1BiLAZ+6YoJbeXFNk9taDkeLpKKumaqGFvrERzM007lT35nDJS+d6DgJIkL0EPd/YCwSPH1oulv5/rJ6tyDS1+y+Kq9rprS2mfSkGCYO7mud7ynZZUX3kCAiRA9Q32yzjh1jHA5ldc3Y7M4VgVkpxhazLy7bRVVDC/2SY60dCwESfezCJ4Q/EkSE6AHeXmNs1/PCj06x9vxwFeWSz31gmpG+ZOXuMgD6Jce5dWdJVl5xLCSICNEDfLT5CIC1hez6+2fwl/8+1Tr/0CVjrWOlFJEuQWXMgBS3jLI9fUdC0bUkiAgRxl5ctouPNh1mdP8UkmKjGGAmSUxLjCEtwZiFlZEU47WW44NbzrCOXbuyhDhW0vkpRJiy2zVPfPyd9TrGY18Pxx7eURHevxVdM/L2lH0tRHBIEBEiTJV7JFl0dGU5pJkD7K4zr1zPvXXjJGu6L8CKu6dR3dhyHJ5U9GQSRIQIUwV7K9xe/+p896y7OanxLJg7mVMGpfm8fspQ90WFA1LjGUC8z7pC+CNBRIgw1Gyz89N/rHUrmzQk3aveZB9lQnQlGVgXIgxtP1IT7EcQApAgIkTY+eMn2/mv54xNpq6ZbKRhH5qZGMxHEr2YdGcJEUa01jz7eSFgLBp8+NJxTB2RycmDUtu5UojjQ4KIEGGksNjYcfCWc4dyx/kjAZgxJiuYjyR6OenOEiKMLNtu7I3+o0knyPoOERIkiAgRRpbtKGZEVpK1Ml2IYJMgIkQY2VRU5XPxoBDB0m4QUUrNU0oVK6U2+zh3h1JKK6UyzNdKKfWsUqpQKbVRKXWKS905Sqmd5p85LuWnKqU2mdc8q6SNLoRPWmuqG21uq8yFCLaOtEReBWZ6FiqlcoEZwH6X4lnAcPPPXOBFs25f4AFgEjAReEAp5VhG+6JZ13Gd13sJIeAvX+wGoLpBUpOI0NFuENFaLwfKfZx6Gvg1oF3KZgOva8MqIFUp1R+4AFiitS7XWlcAS4CZ5rkUrfVKrbUGXgcu7dxHEqLnKdhbbiVbnDEmO8hPI4RTQFN8lVKXAAe11hs8ep9ygAMur4vMsrbKi3yUCyGAqvoWJvz2E7eyM4dn+KktRPc75iCilEoA7gPO93XaR5kOoNzfe8/F6Ppi0KBB7T6rEOHulrfWBfsRhGhTILOzhgKDgQ1Kqb3AQGCdUioboyWR61J3IHConfKBPsp90lq/pLXO11rnZ2ZmBvDoQoSXg5UNACy89Yx2agoRHMfcEtFabwL6OV6bgSRfa12qlFoI3KqUWoAxiF6ltT6slFoMPOYymH4+cI/WulwpVaOUmgx8A1wL/LlzH0mInkFrTWV9M1dNzOXEgamsv38Gdu23oS5EULQbRJRS84FzgAylVBHwgNb6FT/VPwQuBAqBeuA6ADNYPAysMev9VmvtGKy/CWMGWDzwkflHiF6vtLaZivoWhvdLBpybTAkRStoNIlrrq9o5n+dyrIFb/NSbB8zzUV4AjGvvOYTobXYeNdK9j8hKDvKTCOGfrFgXohus3VfB8Ps+5PmlhVz6/NccKK9v95pt5p4hI7KTjvfjCREwCSJCdIPLXlxBS6vmycXb+fZAJRf8aTkrCkvbvGb1njIG9IkjMym2m55SiGMnQUSIbjAq271Lqr65latf/sZvfa01i7ccpcWuJVuvCGkSRIToBgPTEo6p/vKdRitl3ICU4/E4QnQZCSJCdIPqxhZGZCURG9X+/+X2lNYxZ95qAB793vjj/WhCdIoEESGOweo95TQ0t3a4/r6yOux2TWlNEyekJ7L9kVk8ZgaGCAV2u3Pdh9aae9/fxGxz/3RA9g0RIU+CiBB+LN1ezJc7S7j/35s5WNlASU0TP/jrSi7685cdun7N3nLOfnIZc98oYHdpHZPMfUCunjSIh2ePxa5h7AOLqWk0svK+sGwXb32zn+pGGwB///Fpx+eDCdGFZI91IXzQWnPd39dYr7cfqWFsjjE+sbukrt3rK+qaueIvKwH4dFsxAFec6sz841g42NDSyl3vbuSFH53K4i1H3O4xITe1cx9CiG4gLREhfCgsrnV7XVbXRHldc4ev/8U/v/UqS4l3/marbrBZx1sOVQMQ4TELqyPjJ0IEm7REhPBhv8diwF0ldewyWyAxURFo7X/q7Ttri1i+o8Sr3LV+XoZztlZUhEJrbSVbdIiRICLCgPwrFcLDisJS3i444Pd8s81OTZPN7/k7/rUBgFnjsll//wyfdaYMzeCtGycBkH9CX2Y8vZySmiYGZyQydYSRoToqQtaHiNAnLREhXHy46TA3v+ncw+OT26eyaONhnvlsp1u9stpmUuKiva6vbnRuXVta20RaYgxf/vpcfDVapgzNYEhmIlsOV1ndZzPGZPHz84bT2NIqiwxFWJCWiBCmo9WNbgEEjCm2rt1KL/zoFAA2FlW61dtbWseo+z/ixAeduxCOy+kDQG7fBL+LDZNjo9h8sNp6fdfMUSTFRpEhqU5EmJAgIkLO5oNVLNte3K3v2dJqZ9JjnwFwyiDnrKik2CgizW6l688cbM2YenXFXqvOr97ewDlPLaOxxW6V5aTGc8+s0e2+74aiKuu44DfTrfcSIlxIEBEh5+I/f8WP/76GVnv3bMB0tLqR4fc5t7F58JKxAAzOSAQgK8VoFWQkxZKTGs+IrCS3mVTvrivyuueYASnHPDCeFCu9yyL8yL9aEbKm/WEZX9x5rlvZgfJ6Dlc1Utdk45RBafRJ8B6XOFZPfPSddbzk9qkMz0pm1T3nkRRn/N9j9oQcWu0w+6QBgNFNtWpXGYC1UNBTY0vHVrX//brTrPUocdGRAX8GIYJFgogIKU0255fvvrJ6qupb3ALFhc98ac2MOnlQKu/f3Pm9x99bf9A6TjfHIrL7xFllERGKy08daL0enJ7I++sPUt3YwnZzzw9PHU242DfBWHQYFy2dAiI8yb9cEVKKq5vcXr+xaq/ba9eptev3uw9ud4W+HdiCdnBmIlrDkapGXlha6HX+zgtGcvfMUR16v8RYo/WhkLEQEZ4kiIiQUlzTCMCj3zN2TH7qkx0U7C0HOC5jJM02OzFREaQlRPP2T07v0DWO1sP5Ty9n6XbvRYVzpuR1uJstIcboDOifGtdOTSFCk3RniZCyp9RYKT5uQB+r7PK/rGTzQxdQ18YCv0C9+c0+mm12nrpiAhPNBIntSfPTWvnd98fzvZNzjmlso3+fOG6bPpzpo7M6fI0QoUSCiAgJWmsWbjjEO2uNleJ56YlMyE1lwwGjy6quyUatjyDS2NIa8ID0oo2Heeg/WwEY07/jmz+lJ7kHkTsvGMkt5w4L6BmUUtw2fURA1woRCqQ7SwTF/rJ6txlMP3r5G36x4FtW7S5nRFYSfRKiufP8kdb5phY7tWaK9BvOHEya2V1UUuM+hnIsbnnLubAwt2/H9+3olxzHH38wwXrtmAosRG8kQUR0u6qGFqY+uZQHF24BjIV+K8wps2CkFAHIz0uz1mg02lqpqDfKZ43P5o8/PAmA4k4EEceMqL//+DRio46tNfP9UwbSJ94IZPExMjVX9F4SRES3u/f9TQAsWHOAfWV1XoGgzEy5HhcdyaOXGrsANjS3WqnY+ybG0i/ZCC4l5kC8P0eqGvl061Gv8qXbi2lssRMfHcm5o/p16vMkxkivsOi9JIiIbrX1UDWLNh62Xp/95DI+2mS8fu7qkwGYPMQ5wN3XHH8ormmyruubGEO/5DirvC13vrOBG14v4HCVkWZ9y6Eq8u5exJur9gHGLoOB0tqYLZYgLRHRi7X7E0opNQ+4GCjWWo8zy54E/gtoBnYB12mtK81z9wDXA63Az7XWi83ymcAzQCTwstb6cbN8MLAA6AusA67RWnd89x8RVr4q9J4S+9bq/UQoI4Pttt/OJCrSuWYix9xj/MbXC6yylLgo7BoiI5TXuhJPjnGX03/3OT8/bzjVDcYK80+3FROh4N4L289v5Y9jyrF0Z4nerCMtkVeBmR5lS4BxWusTgR3APQBKqTHAlcBY85oXlFKRSqlI4HlgFjAGuMqsC/AE8LTWejhQgRGARA91qLLRK0fU7pI68tITiY2KJD4mkuhI5z9LX9lslVJERijSE2OsdSX+uN7r2c92ug3E90uO61TCw5nj+gNYYyNC9EbtBhGt9XKg3KPsE621Y75Yn9swAAAXQElEQVTlKsCRE2I2sEBr3aS13gMUAhPNP4Va691mK2MBMFsZGyZMA94xr38NuLSTn0mEKK01a/aWM6hvAi9fm+92bnep733L2/qS7xMfTU1j22tHPLec/XpXqXWcmdy5dOv3XjiK+TdOlrTtolfrijGR/wEcKVBzANct4YrMMn/l6UClS0BylIse6LIXV7DlUDUXjs9m+pgs9j5+EY4YcVpemt/rHjKz6noeJ8RE0tBOosMIjyBUWe9MmHhSbqpn9WOSnhTL6UPTO3UPIcJdp4KIUuo+wAa86SjyUU0HUO7v/eYqpQqUUgUlJd596yJ02e2adWauq34pzhQfjoSHPz17qN9rr5yYC8DEwX2ZMyXPKo+LjqS+ue0g4muvc4cVLq0SIURgAp6bqJSagzHgfp52TFMxWhK5LtUGAofMY1/lpUCqUirKbI241veitX4JeAkgPz8/oERKB8rraWhpZURWciCXiwCt219hHbv+cnCs1WhrXCE2KpK9j1/kVZ4UG8WOYt9ZdMEIXL5MHZHJ8h0lXDJBGr1CdFZALRFzptVdwCVa63qXUwuBK5VSseasq+HAamANMFwpNVgpFYMx+L7QDD5LgcvN6+cAHwT2UTrmrN8v5fynlwP+v2RE1/vVvzYAcOawDC4x9+UArEV+gcxwGpyRyJEq/wPrrl1dP5/mTEsyMS+NXY9dyC+mDz/m9xRCuGs3iCil5gMrgZFKqSKl1PXAc0AysEQp9a1S6i8AWustwNvAVuBj4BatdavZyrgVWAxsA94264IRjH6plCrEGCN5pUs/oR9vfrOPIfd+SGlt4CueRcdsP1LDvjLjt8Yb1090Wx1+4XhjhlMgg9Mp8dG0tGpaWu0+zzsSNj5y6Thun+HMT5XosuWtEKJz2u3O0lpf5aPY7xe91vpR4FEf5R8CH/oo340xe6tb3ff+ZsDIvSSza46f6sYWLviT0fKbfdIAlMdsqdvOG87VEweRlXLsqdDjzcSLjS2tfLz5CB9vOcJzV51svYcjYWNSbJQ1LbjVrkmUbWiF6DK9bsX61BGZbq/tWrq0jqcvdzgHr+9wSajoEBGh3HYRPBZxZhdYQ0srP5u/nkUbD9PY4myV3PCasUDRETQcjQ/Zy1yIrtPrgsiVp+W6vV68+UiQnqTns7XarUy5z1x5Erl9O7ZlbEdZLZFmZ+BwrFBftbvMWnvi2Afd0UKRlogQXafXBRFHH7zDs597b28qOk9rzQ0uqUpmjevfRu3AOILIff/eZJWd/PAS8u5exP5y53wPx1oOZ0tE0pQI0VV6XRDx5cwnPj8uu+b1Jlpr1u6rsPJJLVhzgGXbS4iMUOx+7EJiorr+n1p8jHHPL3d6r/dwJEW8auIg+vcx8m85Vq8nxUqaEiG6Sq8MIj/Md+/SKqpocFvHII7d8p2lXPbiCp5esgNbq5173jNaB/+59UyvVeNdpa0dDQ9VGll7b3WZ2htpdWdJS0SIrtIrg8jvvj+e+TdOdiurbpCWSGc4Ehu+u66IT1z27xjd//gt6mxrH4/HPvyOuOgIKwswQGqi0QIJdDtdIYS3XjnCGBGhvHIe2ey+1xqIjmloNoLw4apGdhw1VpEX/Ga615TertRei8IzyLx23UQWbjhEemKMnyuEEMeqV7ZEfLG19p6pvq12Td7di/jTpzu65H7PLy3k/g+2WK/fXVdEdKQ67l/W7c2ySvAIMkMyk7ht+ojjGtiE6G16dRB57HvjreOPt/Seqb7vrDUSKv/p050UV7e9H0d7DpTX8+Ti7R5lDbS06uP+Zd0vOY78E/xn/5Vta4U4/np1EHHdGnXJ1qPY/KTP6EnKapu4613nlNjPvivu1P3O+v3Szj5SwCIjFO/cNMXvedcdEoUQx0evDiKe2tuboifYcbTW7bXr/hqd8dvZY/noF2d1yb26ivK504AQoiv1+iDym4uce2w32Xp2S6SxpZWb31wLwEvXnApAfXPgs9IcA+h3zRzFtafnMbp/inXu0e+N68STBuYf109ym40lhDj+en2n8Q1nDeGRRdsAZ8qMnuhodSOTHvvMen3+2GySY6NYs7ec2ibbMeeTqm+2WSn1XVPJ7H7sQpSiWwevP7l9Kk0tdsYP7MPtM0Zwh5l2XsbPhTj+en1LxNX+svr2K4Up18Fvx/7mNU02Vu0u54Wlx5765bEPt1nHaS6zsCIiVLfPfhqRlcz4gX0AY+W89SwSRYQ47iSIAG/eMAmAFbvKgvwkx88H3x4EjC6f6WOy3M5tPVx9TPcqrW3iH6v2AzAyxHaIdJ2oLXuGCHH8SRABzhiWQWZyLEc6Od01FPiaYaa1psVcB3Pm8Ayv821tTevL59ucM7oWzJ3cRs3ud6o55TcjKYYnLz8xyE8jRM8nQcSUEhcV9rOz7vjXBobd9xErCt0TEhZVGHmkHr7U92D3B9/63daezQeryLt7EQ8udC4mbDW7jO67cLRbV1YoGJqZxN7HL6LgNzMYkpkU7McRoseTIGKKjoygJYxnZzW2tPLO2iIA1u5zTyb5rwJjceEEc9zAF+1nc66fvGHM5np1xV6rrLjayJM1Z0peoI8rhOghJIiYYqIi/O7VHeoaW1oZdf/H1utal7T2JTVN1p4po7JTvK513sOO1pp73tvIejOjcatdc9DMhgtGoNFa85+NhxjQJ+64pHcXQoQX+RYwRUUoa9wgHNha7bz1zX6eXPwd5z61zCpPTYimxiWILN9RAhjrNjy/9H8ydYh1XNtko7yumfmrD3DNK6vdrnVoadXsLq2jsLiWc0f16+qPJIQIQ71+nYhDdGQEzWHSErG12hl230de5evun8H3Xvja2mCrsaWV+auNWVT/NWGAV/17LhzNyOxkfvn2BuqabDTajDEhR4usrK7Zrf5fv9jFoSqjZXLxid73E0L0PtISMYVTd5ZrxlyHWeOy6ZsYQ2JMlBVELn3+awrM8ZFkP4sJHZlwa5tsfG7m0RqYFs+SrUd55jMjy++dF4wE4A9LdjB/tTG+Mi7Hf9eYEKL3kJaIKToyPIKI1tpqXbgalJ4AGHts1DTaqG+28d0RIy3JZacM9LsAMNkliKzbVwlAfEwkd/xrA1UNLUQovFK6D++XRHKcbDErhJAgYomOVLTYQn9MZNvhGus4NiqCJpudW88dZm0DW1HfQmFxLWP+d7FV7ydnD/G6j4OjJXLlS6usspKaJqoajMSMdm2so3F15cRBCCEESBCxhEtLZH+5kZrlNxeN5oazvINDYbF7lt53b5rCiDZWlfva2OmoOYUXICMplty+CeSfkGZ1jQ3JSAzo2YUQPU+7YyJKqXlKqWKl1GaXsr5KqSVKqZ3m32lmuVJKPauUKlRKbVRKneJyzRyz/k6l1ByX8lOVUpvMa55VQdp2LiZMBtZ3lRhBwnUvFFcf3+aejj0rJbbN+3muVo+Ldv8nMSLLWLD32PedG3ilxMtvDyGEoSMD668CMz3K7gY+01oPBz4zXwPMAoabf+YCL4IRdIAHgEnAROABR+Ax68x1uc7zvbpFdGRESG+R+9B/tpB39yKKKhqIjFDER/veX3xUdgqzT3LOnMpMbjuIeJ6fPSHHOh6SmcizV50MQFZynFUe5+e9hRC9T7tBRGu9HCj3KJ4NvGYevwZc6lL+ujasAlKVUv2BC4AlWutyrXUFsASYaZ5L0Vqv1MaS6ddd7tWtoqNUyHZn2Vrt/P3rvQDMX72fxJjINjPlTh/tTLAYG3VsX/gnD0q1jh+ZPY6MJCPISOtDCOFLoFN8s7TWhwHMvx0rz3KAAy71isyytsqLfJR3u1BeJ7L9aI3ba8cXuz8pZhfVzecM7dD9Xfeaz3MZ70hx6epSSrHynmk8fOk4xvSX6b1CCENXrxPx9fNYB1Du++ZKzVVKFSilCkpKSvxVC0hMCA6s1zfb2FVSy/r9lW7l6UltJz2cOjyDD245g9tnjOjQ+1w9aZCVV2tAH+fOgJ7jJf37xHPN5BO6fb8QIUToCrSP4qhSqr/W+rDZJeXIDV4E5LrUGwgcMsvP8ShfZpYP9FHfJ631S8BLAPn5+V06gGHMzgqtMZHvv7DCWuvhqrqh7S1tlVJMyE1ts46nv12bz8INh8jt6wwiKceYIl4I0fsE2hJZCDhmWM0BPnApv9acpTUZqDK7uxYD5yul0swB9fOBxea5GqXUZHNW1rUu9+pWUZGKVrt2S14YTMXVjV4BxDHIfTxaTP1S4rjhrCFurYxj3TJXCNH7dGSK73xgJTBSKVWklLoeeByYoZTaCcwwXwN8COwGCoG/ATcDaK3LgYeBNeaf35plADcBL5vX7AK8k0J1g+hI4z/FSQ99Eoy397K7tM7tdU5qPJMH9wVg9IDuGZOQnQGFEO1p96em1voqP6fO81FXA7f4uc88YJ6P8gLA925J3SjGDCI2e2h0aR2saHB7Xd3YQr+UOBbMncz4HP/7gnSFeT/O51Bl+O/yKIQ4/qS/whQd6fzVvXJXGW+t3s8zPzyJiG7+Nd7Q3MrCDQf53UffuZU/d7WxbnPykPTj/gzTRmW1X0kIIZAgYolxWU9x5zsbKKpo4I7zR3BCevem+Jjy+GdU1LdYr4f1S6KwuJaMdmZkCSFEMEgqeFNirDOIOMaWgzHI7tqbdt+Fo7GZg+iySlwIEYokiJgSY5yNsgPlxnhEczfuua61MTMsJ9U5xTa3bwKPX3Yip+WlkZuW0G3PIoQQHSXdWaaEWO9f+t0ZRF5dsZeH/rPVrSwnNZ7xA/vwr59O6bbnEEKIYyEtEZNjiq+r7kyD8nZBkVdZTlq8j5pCCBE6JIiYfK2J6M6WyP6yOq+ytARZMS6ECG3SnWWKjvDREummILKxqJK65lbr9R9/MIGZ47IlR5UQIuRJEDH5aonsLavvlvf+dJuReuzmc4ZSWFzLBWOzSYiR/2mEEKFPurNMvn70P/Hxd15lxTWNtHbxqvbXV+4F4IazhvDStfk+t6wVQohQJEHEpNuIC7VNNmytdr7ZXcbERz/j4f/b6r/yMSqva6bSXFyYHCfBQwgRXuRby6T9b2PCuAcWc9H4/taCvw++PciDl4ztkvctr2sC4JFLx/mcISaEEKFMgojJX0vEbnZdLdp0mLho40v+jGEZXfa+jhQnMp1XCBGO5Kevyd/eGYu3HLGOG1uM2VqdmTW1+WAVL3+523q99VA1ACOzkgO+pxBCBIsEEVNeRiKvzMn3Kr/jXxu8yppaWr3KOqKirpmL//wVjyzaRoM5pfdwVSPRkYrslLiA7imEEMEkQcTFeaOz8Jzp67p+w+HrwtKA7j/rmS+t43X7KwA4UtVAv+S4bk85L4QQXUGCiIeoDgxuB9qddaTaudFTcY1xvPlQNUMyuzfdvBBCdBUJIh5io4z/JPFtpF53TPk9VqOykxlqBozdJUaak8r6FnL7SoZeIUR4kiDiYVi/JAD+es2pbuUnDnTfkrayoYWOaLK18sAHm7nlrXV8d6SGUdnG/uh//ryQ55cWUtvUQmKM7BUihAhPEkQ8vHRNPg9dMpZR2e6zpa4/c7Db68r6jgWRtXsreG3lPhZtPAxAvEvAeHLxdhpb7JLiRAgRtiSIeMhMjmXOlDz6JrpvR3tSbioAUeYAeElNU4fut6vUPTvvhgOV/OK84W5liT72MhFCiHAgQcQPzwH2E9ITWXzbVD6+bSoA/1i1j0+3Hm33PhsPVLq9HpAa77U2XloiQohwJUGkA9756ekAjMxOJtXc42PRpsPc8HpBuwPsZXXNbq+fu/pkkjxaHtISEUKEKwkiHeAYbAfvle1THv/cSo3iy+ffGWne//TDk3j6hxNIjovmujMG88Rl4606eekyxVcIEZ4kiHSAa9dWnMfU3+KaJor9jI80uqxsv/TkHL538kDA2Ir3h6cNss71k9XqQogw1akgopS6XSm1RSm1WSk1XykVp5QarJT6Rim1Uyn1T6VUjFk31nxdaJ7Pc7nPPWb5dqXUBZ37SF0vymM1+dypQ9xeN/hIg/LFjhJG3f9xh+4vKeCFEOEq4CCilMoBfg7ka63HAZHAlcATwNNa6+FABXC9ecn1QIXWehjwtFkPpdQY87qxwEzgBaVUSA0SeO56eN6ofm6vm2zeQWTOvNXW8fNXn+LzvsvuOId7LxxFSpzspS6ECE+d7c6KAuKVUlFAAnAYmAa8Y55/DbjUPJ5tvsY8f54y8ofMBhZorZu01nuAQmBiJ5+rSziSInq2RCYNSedHk5zdUU0t7oProz1aIAP9pHnPy0hk7tShXfGoQggRFAH3o2itDyqlngL2Aw3AJ8BaoFJrbTOrFQE55nEOcMC81qaUqgLSzfJVLrd2vSao3rt5Cuv3V/rMleW6gZTn6nXP7q0J5hoTIYToaTrTnZWG0YoYDAwAEoFZPqo6pi75ylqo2yj39Z5zlVIFSqmCkpKSY3/oYzQgNZ6LTuzv89yt04ZZq9pvfL3AKl+7r+K4P5cQQoSKznRnTQf2aK1LtNYtwHvAFCDV7N4CGAgcMo+LgFwA83wfoNy13Mc1brTWL2mt87XW+ZmZmZ149M7LSIrlPz87E4Bcl+6qn89fbx0/PHssux+7sNufTQghuktngsh+YLJSKsEc2zgP2AosBS4368wBPjCPF5qvMc9/rrXWZvmV5uytwcBwwDkqHcKiIyOYMLAPA9OcWXgPVjZYx1fk58o+IUKIHq0zYyLfKKXeAdYBNmA98BKwCFiglHrELHvFvOQV4A2lVCFGC+RK8z5blFJvYwQgG3CL1jqwrQODICEmytqlsLbJZpXPHJvttaZECCF6mk4tUNBaPwA84FG8Gx+zq7TWjcAVfu7zKPBoZ54lWOJjIq1kjBUuKU5GZMue6UKInk9WrHdSfHQkFfVG8GiyGVN9Z580gJ9PGxbMxxJCiG4hQaSTIiMURRUNNNvs7DXTvs8al92hbXaFECLcyTddJ40dYOxUWNXQwk/+sRZwtkiEEKKnkyDSSdl9jFXt1Y0ttJrZfDu6YZUQQoQ7CSKd5Ejj/tTi7WSlxAIwa7zvBYpCCNHTSBDppJHmLKyPNh8hQikm5KaSk+o7V5YQQvQ0EkQ6KS46knE5KWSlxFLTaOOUQZInSwjRe0gQ6QInDkyl2WantslGsqR1F0L0IhJEukBSbBQV9UYm37QECSJCiN5DgkgXSIxxLvzvmxgTxCcRQojuJUGkCyTGOnNkSRARQvQmEkS6QGKssyWSliBBRAjRe0gQ6QKO3FkAfeJlTEQI0XtIEOkCdrtzI8bkuE4lRhZCiLAiQaQLnDwozTpOipUgIoToPSSIdIEzhmVYx5K9VwjRm8g3nhBCiIBJ30sXmX/jZA5U1Af7MYQQoltJEOkipw9N53TSg/0YQgjRraQ7SwghRMAkiAghhAiYBBEhhBABkyAihBAiYBJEhBBCBEyCiBBCiIBJEBFCCBEwCSJCCCECprTW7dcKQUqpEmBfgJePB6IBDSgff9PGufb+Dta1vf39w/nZg/3+4fzswX7/YDw7wHoCd4LWOrMT17sJ2yDSGUqp3vehhRA9htZatV+re0h3lhBCiIBJEBFCCBGw3pqAcQ8wAGjBGBvx/Js2zrX3d7Cu7e3vH87PHuz3D+dnD/b7B+PZQ0qvHBMRQgjRNaQ7SwghRMBCojtLKfVjYB7O6W5CCCGC72Wt9Y1tVQiVlkgdsBpjrOJgkJ9FCCGE4XqlVExbFUJyTEQp1UroBDghhOjN+mutj/g7GXJf1EqpmwjB5xJCiF5qbFsnQ+rLWik1Fngh2M8hhBACu/l3dFuVQiaIKKXSgE3Bfg4hhBCAMz40tVUpJMZElFIKaCZEZosJIYRAY7RGsrXWpf4qhUpL5C0kgAghRChRQHlbAQRCpCUihBAiPIVKS0QIIUQYkiAihBAiYBJEhBBCBEyCiBBCiIBJEBFCCBEwCSJCCCECJkFECCFEwCSICCGECNj/A8yyecdQQP3KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(stocks['Date'], stocks['High'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of Pipeline\n",
    "#### News, Stock Chart $\\rightarrow$ Sentiment Class for Stock $\\rightarrow$ Prediction (Increase/Decrease)\n",
    "\n",
    "## Sentiment Analysis Model\n",
    "This relies heavily on NLTK. Given more time, I'd use the implementation from Homework 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These 20 Companies have had a great year. : 1\n",
      "Many businesses are experiencing loss. : 0\n",
      "It has been a successful year for tech. : 1\n",
      "Net sentiment: 0.8926000000000001\n"
     ]
    }
   ],
   "source": [
    "class SentimentModel:\n",
    "    def __init__(self):\n",
    "        self.model = SentimentIntensityAnalyzer()\n",
    "    \n",
    "    def predict(self, text):\n",
    "        tup = self.model.polarity_scores(text)['compound']\n",
    "        sent = 1 if tup >= 0 else 0\n",
    "        return sent\n",
    "    \n",
    "    def net_sentiment(self, text_list):\n",
    "        total = 0\n",
    "        \n",
    "        for t in text_list:\n",
    "            tup = self.model.polarity_scores(str(t))['compound']\n",
    "            total += tup\n",
    "        \n",
    "        return total \n",
    "        \n",
    "            \n",
    "m = SentimentModel()\n",
    "tests = ['These 20 Companies have had a great year.', 'Many businesses are experiencing loss.', 'It has been a successful year for tech.']\n",
    "for t in tests:\n",
    "    print(\"{} : {}\".format(t, m.predict(t)))  \n",
    "print(\"Net sentiment: {}\".format(m.net_sentiment(tests)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stock Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "stocks = pd.read_csv('stocks.csv')\n",
    "news = pd.read_csv('stocknews/Combined_News_DJIA.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>return</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>previous return</th>\n",
       "      <th>previous close</th>\n",
       "      <th>change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1988</td>\n",
       "      <td>2008-08-08</td>\n",
       "      <td>11432.089844</td>\n",
       "      <td>11759.959961</td>\n",
       "      <td>11388.040039</td>\n",
       "      <td>11734.320312</td>\n",
       "      <td>212830000</td>\n",
       "      <td>11734.320312</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.7360</td>\n",
       "      <td>0</td>\n",
       "      <td>11431.430000</td>\n",
       "      <td>302.230468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1987</td>\n",
       "      <td>2008-08-11</td>\n",
       "      <td>11729.669922</td>\n",
       "      <td>11867.110352</td>\n",
       "      <td>11675.530273</td>\n",
       "      <td>11782.349609</td>\n",
       "      <td>183190000</td>\n",
       "      <td>11782.349609</td>\n",
       "      <td>1</td>\n",
       "      <td>-3.0185</td>\n",
       "      <td>1</td>\n",
       "      <td>11734.320312</td>\n",
       "      <td>52.679687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1986</td>\n",
       "      <td>2008-08-12</td>\n",
       "      <td>11781.700195</td>\n",
       "      <td>11782.349609</td>\n",
       "      <td>11601.519531</td>\n",
       "      <td>11642.469727</td>\n",
       "      <td>173590000</td>\n",
       "      <td>11642.469727</td>\n",
       "      <td>1</td>\n",
       "      <td>-5.4389</td>\n",
       "      <td>1</td>\n",
       "      <td>11782.349609</td>\n",
       "      <td>-139.230468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1985</td>\n",
       "      <td>2008-08-13</td>\n",
       "      <td>11632.809570</td>\n",
       "      <td>11633.780273</td>\n",
       "      <td>11453.339844</td>\n",
       "      <td>11532.959961</td>\n",
       "      <td>182550000</td>\n",
       "      <td>11532.959961</td>\n",
       "      <td>1</td>\n",
       "      <td>-4.2181</td>\n",
       "      <td>1</td>\n",
       "      <td>11642.469727</td>\n",
       "      <td>-99.849609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1984</td>\n",
       "      <td>2008-08-14</td>\n",
       "      <td>11532.070312</td>\n",
       "      <td>11718.280273</td>\n",
       "      <td>11450.889648</td>\n",
       "      <td>11615.929688</td>\n",
       "      <td>159790000</td>\n",
       "      <td>11615.929688</td>\n",
       "      <td>0</td>\n",
       "      <td>-3.2687</td>\n",
       "      <td>1</td>\n",
       "      <td>11532.959961</td>\n",
       "      <td>83.859376</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0        Date          Open          High           Low  \\\n",
       "0        1988  2008-08-08  11432.089844  11759.959961  11388.040039   \n",
       "1        1987  2008-08-11  11729.669922  11867.110352  11675.530273   \n",
       "2        1986  2008-08-12  11781.700195  11782.349609  11601.519531   \n",
       "3        1985  2008-08-13  11632.809570  11633.780273  11453.339844   \n",
       "4        1984  2008-08-14  11532.070312  11718.280273  11450.889648   \n",
       "\n",
       "          Close     Volume     Adj Close  return  sentiment  previous return  \\\n",
       "0  11734.320312  212830000  11734.320312       1    -7.7360                0   \n",
       "1  11782.349609  183190000  11782.349609       1    -3.0185                1   \n",
       "2  11642.469727  173590000  11642.469727       1    -5.4389                1   \n",
       "3  11532.959961  182550000  11532.959961       1    -4.2181                1   \n",
       "4  11615.929688  159790000  11615.929688       0    -3.2687                1   \n",
       "\n",
       "   previous close      change  \n",
       "0    11431.430000  302.230468  \n",
       "1    11734.320312   52.679687  \n",
       "2    11782.349609 -139.230468  \n",
       "3    11642.469727  -99.849609  \n",
       "4    11532.959961   83.859376  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Data Cleaning, adding sentiment scores to each row\n",
    "\"\"\"return_ints = []\n",
    "nets = []\n",
    "\n",
    "for i in range(1989):\n",
    "    news_row = news.loc[i]\n",
    "    stock_row = stocks.loc[i]\n",
    "    \n",
    "    news_texts = [news_row['Top{}'.format(x)] for x in range(1, 26)]\n",
    "    net_score = m.net_sentiment(news_texts)\n",
    "    \n",
    "    nets.append(net_score)\n",
    "    \n",
    "    inc = stock_row['return']\n",
    "    as_int = 1 if inc else 0\n",
    "    return_ints.append(as_int)\n",
    "    \n",
    "score_series = pd.Series(nets)\n",
    "ints = pd.Series(return_ints)\n",
    "\n",
    "stocks['return'] = ints.values\n",
    "stocks['sentiment'] = score_series.values\n",
    "\"\"\"\n",
    "prev_close = [11431.43] + list(stocks['Close'])[:-1]\n",
    "prev_return = [0] + list(stocks['return'])[:-1]\n",
    "\n",
    "stocks['previous return'] = prev_return\n",
    "stocks['previous close'] = prev_close\n",
    "stocks['change'] = stocks['Close']-stocks['Open']\n",
    "\n",
    "def class_data(stocks):\n",
    "    X, y = stocks[['Open', 'previous return', 'previous close', 'sentiment']], stocks['return']\n",
    "    split = int(len(X)*0.7)\n",
    "    end_num = len(X)-split\n",
    "    X_train, y_train, X_test, y_test = X.head(split), y[:split], X.tail(end_num), y[split:]\n",
    "    \n",
    "    return (X_train, list(y_train), X_test, list(y_test))\n",
    "\n",
    "def reg_data(stocks):\n",
    "    X, y = stocks[['Open', 'previous return', 'previous close', 'sentiment']], stocks['change']\n",
    "    split = int(len(X)*0.7)\n",
    "    end_num = len(X)-split\n",
    "    X_train, y_train, X_test, y_test = X.head(split), y[:split], X.tail(end_num), y[split:]\n",
    "    \n",
    "    return (X_train, list(y_train), X_test, list(y_test))\n",
    "\n",
    "X_train, y_train, X_test, y_test = class_data(stocks)\n",
    "classifier_data = [X_train, y_train, X_test, y_test]\n",
    "\n",
    "X_train, y_train, X_test, y_test = reg_data(stocks)\n",
    "regressor_data = [X_train, y_train, X_test, y_test]\n",
    "\n",
    "stocks.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment $\\rightarrow$ Stock Prediction\n",
    "\n",
    "At this point, we can use the 'Open', 'previous return', 'previous close', and 'sentiment' columns to predict either 'return' or 'change'. By predicting one of two columns, we can re-model this problem as either classification or regression.\n",
    "\n",
    "### Goal\n",
    "Ideally, we need classification accuracy to be at least 50%, and regression accuracy (mean absolute percent error) to be within 15%. \n",
    "\n",
    "## Approaches\n",
    "I implemented neural network, random forest, and a few other approaches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Neural Network (keras)\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.optimizers import adam\n",
    "\n",
    "class StockPredict:\n",
    "    \n",
    "    def __init__(self):\n",
    "        model = Sequential()\n",
    "        model.add(Dense(64, input_dim=4, activation='relu'))\n",
    "        model.add(Dropout(0.5))\n",
    "        model.add(Dense(64, activation='tanh'))\n",
    "        model.add(Dropout(0.5))\n",
    "        model.add(Dense(1, activation='sigmoid'))\n",
    "        \n",
    "        self.nn = model\n",
    "        \n",
    "        self.nn.compile(loss='binary_crossentropy',\n",
    "              optimizer='rmsprop', metrics=['accuracy'])\n",
    "        \n",
    "    def train(self, dataX, dataY, e=50, batch=32):\n",
    "        self.nn.fit(X_train, y_train, epochs=e, batch_size=batch, verbose=0)\n",
    "        \n",
    "    def predict(self, x):\n",
    "        return self.nn.predict(x)\n",
    "        \n",
    "    def score(self, testX, testY, bs=100):\n",
    "        return self.nn.evaluate(testX, testY, batch_size=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "597/597 [==============================] - 0s 269us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.6907956510332961, 0.5376884302801423]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stock_model = StockPredict()\n",
    "\n",
    "X_train, y_train, X_test, y_test = classifier_data\n",
    "\n",
    "stock_model.train(X_train, y_train, e=30)\n",
    "score = stock_model.score(X_test, y_test)\n",
    "\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Function for sklearn models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_error(model, x, y, classifier=True):\n",
    "    index = 0\n",
    "    error = 0\n",
    "    \n",
    "    for yhat in model.predict(x):\n",
    "        ystar = y[index]\n",
    "        \n",
    "        if classifier:\n",
    "            if yhat == ystar:\n",
    "                error += 1\n",
    "        else:\n",
    "            error += abs(yhat-ystar)\n",
    "\n",
    "        index += 1\n",
    "        \n",
    "    return error/index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Regression Score (MAPE): 118.94290698442212\n",
      "Classification Score: 0.5309882747068677%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Abhi/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/forest.py:248: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "/Users/Abhi/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/forest.py:248: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "#Random Forests\n",
    "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor\n",
    "\n",
    "model = RandomForestRegressor()\n",
    "X_train, y_train, X_test, y_test = regressor_data\n",
    "model.fit(X_train, y_train)\n",
    "score = model_error(model, X_test, y_test, classifier=False)\n",
    "print(\"Regression Score (MAPE)%: {}\".format(score))\n",
    "\n",
    "model = RandomForestClassifier()\n",
    "X_train, y_train, X_test, y_test = classifier_data\n",
    "model.fit(X_train, y_train)\n",
    "score = model_error(model, X_test, y_test, classifier=True)\n",
    "print(\"Classification Score: {}%\".format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Score: 0.5376884422110553%\n",
      "Regression Score (MAPE): 110.11160779466063%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression, LinearRegression, SGDClassifier, SGDRegressor\n",
    "\n",
    "model = LogisticRegression(solver='lbfgs')\n",
    "X_train, y_train, X_test, y_test = classifier_data\n",
    "model.fit(X_train, y_train)\n",
    "score = model_error(model, X_test, y_test, classifier=True)\n",
    "print(\"Classification Score: {}%\".format(score))\n",
    "\n",
    "model = LinearRegression()\n",
    "X_train, y_train, X_test, y_test = regressor_data\n",
    "model.fit(X_train, y_train)\n",
    "score = model_error(model, X_test, y_test, classifier=False)\n",
    "print(\"Regression Score (MAPE): {}%\".format(score))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}