LDA_gensim_vs_sklearn_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LDA gensim and sklearn implementations\n",
    "\n",
    "Here is a quick comparisson between the two models to ensure that is ok to use one or the other (a matter of preference).\n",
    "\n",
    "For a more in-depth comparison one can have a look to this [issue](https://github.com/RaRe-Technologies/gensim/issues/457) and the code cited there. \n",
    "\n",
    "Let's start by reading the docs, select a randomm sample of 2500 documents and split the dataset into train/test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda2/lib/python2.7/site-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 174 of the file /home/ubuntu/anaconda2/lib/python2.7/runpy.py. To get rid of this warning, change code that looks like this:\n",
      "\n",
      " BeautifulSoup([your markup])\n",
      "\n",
      "to this:\n",
      "\n",
      " BeautifulSoup([your markup], \"lxml\")\n",
      "\n",
      "  markup_type=markup_type))\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import random\n",
    "from nlp_utils import SimpleTokenizer, read_docs\n",
    "\n",
    "# logging.basicConfig(format='%(levelname)s : %(message)s', level=logging.DEBUG)\n",
    "# logging.root.level = logging.DEBUG\n",
    "TEXT_DATA_DIR = '/home/ubuntu/working/text_classification/20_newsgroup/'\n",
    "NB_TOPICS = 10\n",
    "\n",
    "docs, doc_classes  = read_docs(TEXT_DATA_DIR)\n",
    "# picking documents at random\n",
    "random.seed(1981)\n",
    "rand_docs = random.sample(docs,2500)\n",
    "\n",
    "# Apply a simple tokenizer based in gensim's simple_preprocess\n",
    "rand_docs = [SimpleTokenizer(doc) for doc in rand_docs]\n",
    "\n",
    "#train,test\n",
    "train_docs, test_docs =  rand_docs[:2000], rand_docs[2000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare the data before building the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from gensim.matutils import Sparse2Corpus\n",
    "\n",
    "# PREPARE DATA\n",
    "# gensim corpus can be prepared using just gensim...\n",
    "# id2word = gensim.corpora.Dictionary(train_docs)\n",
    "# id2word.filter_extremes(no_below=20, no_above=0.5)\n",
    "# gensim_tr_corpus = [id2word.doc2bow(doc) for doc in train_docs]\n",
    "# gensim_te_corpus = [id2word.doc2bow(doc) for doc in test_docs]\n",
    "\n",
    "# or using sklearn vectorizer and the very convenient Sparse2Corpus\n",
    "vectorizer = CountVectorizer(min_df=20, max_df=0.5,\n",
    "    preprocessor = lambda x: x, tokenizer=lambda x: x)\n",
    "sklearn_tr_corpus = vectorizer.fit_transform(train_docs)\n",
    "sklearn_te_corpus = vectorizer.transform(test_docs)\n",
    "\n",
    "id2word = dict()\n",
    "for k, v in vectorizer.vocabulary_.iteritems():\n",
    "    id2word[v] = k\n",
    "gensim_tr_corpus = Sparse2Corpus(sklearn_tr_corpus, documents_columns=False)\n",
    "gensim_te_corpus = Sparse2Corpus(sklearn_te_corpus, documents_columns=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now define the model parameters that will be passed to both, gensim and sklearn LDA. Information on these parameters can be found [here](https://radimrehurek.com/gensim/models/ldamulticore.html) and [here](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# MODEL PARAMETERS\n",
    "decay = 0.5\n",
    "offset = 1.\n",
    "max_iterations = 10\n",
    "batch_size = 200\n",
    "max_e_steps = 100\n",
    "eval_every = 1\n",
    "mode = \"online\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And build and run the models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sklearn/decomposition/online_lda.py:812: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  if doc_topic_distr != 'deprecated':\n",
      "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sklearn/decomposition/online_lda.py:816: DeprecationWarning: Argument 'doc_topic_distr' is deprecated and is being ignored as of 0.19. Support for this argument will be removed in 0.21.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "from gensim.models import LdaMulticore\n",
    "\n",
    "#SKLEARN\n",
    "lda_sklearn = LatentDirichletAllocation(\n",
    "    n_components=NB_TOPICS,\n",
    "    batch_size=batch_size,\n",
    "    learning_decay=decay,\n",
    "    learning_offset=offset,\n",
    "    n_jobs=-1,\n",
    "    random_state=0,\n",
    "    max_iter=max_iterations,\n",
    "    learning_method=mode,\n",
    "    max_doc_update_iter=max_e_steps,\n",
    "    evaluate_every=eval_every)\n",
    "\n",
    "start = time.time()\n",
    "lda_sklearn.fit(sklearn_tr_corpus)\n",
    "sk_time = time.time() - start\n",
    "\n",
    "gamma = lda_sklearn.transform(sklearn_te_corpus)\n",
    "sklearn_perplexity = lda_sklearn.perplexity(sklearn_te_corpus, gamma)\n",
    "\n",
    "# GENSIM\n",
    "start = time.time()\n",
    "lda_gensim_mc = LdaMulticore(\n",
    "    gensim_tr_corpus,\n",
    "    id2word=id2word,\n",
    "    decay=decay,\n",
    "    offset=offset,\n",
    "    num_topics=NB_TOPICS,\n",
    "    passes=max_iterations,\n",
    "    batch=False, #for online training\n",
    "    chunksize=batch_size,\n",
    "    iterations=max_e_steps,\n",
    "    eval_every=eval_every)\n",
    "gn_time = time.time() - start\n",
    "\n",
    "log_prep_gensim_mc   = lda_gensim_mc.log_perplexity(gensim_te_corpus)\n",
    "preplexity_gensim_mc = np.exp(-1.*log_prep_gensim_mc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look to the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gensim run time and perplexity: 62.3879930973, 1878.04572157\n",
      "sklearn run time and perplexity: 17.2567749023, 1737.68489529\n"
     ]
    }
   ],
   "source": [
    "print(\"gensim run time and perplexity: {}, {}\".format(gn_time, preplexity_gensim_mc))\n",
    "print(\"sklearn run time and perplexity: {}, {}\".format(sk_time,sklearn_perplexity))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results are on a AWS-p2 instance.\n",
    "\n",
    "I have run a few times the script with different seeds. Normally perplexity values are quite similar but `sklearn` is $\\sim$3 times faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sklearn \n",
      "\n",
      "      topic_0    topic_1  topic_2      topic_3    topic_4  topic_5  topic_6  \\\n",
      "0  government       said     like   discussion    science      new      use   \n",
      "1      people       time  article      general  objective      gov     need   \n",
      "2     turkish       know      mac       number       uiuc   ground    space   \n",
      "3        jews      years     know    reference      moral     nasa      com   \n",
      "4      israel  insurance   people         copy      point     book  windows   \n",
      "5   president       went      com      article       mean    price      bit   \n",
      "6      health      think      key         news      frank  subject  problem   \n",
      "7    armenian        way    going  information    article      man     mail   \n",
      "8         war       came      lot    situation     theory  appears     work   \n",
      "9         fbi        old     sure        islam   morality      old     file   \n",
      "\n",
      "   topic_7  topic_8  topic_9  \n",
      "0   people  article     year  \n",
      "1      god      com     game  \n",
      "2    think     good      com  \n",
      "3  article      use     team  \n",
      "4  believe      car    cable  \n",
      "5    right    power      pay  \n",
      "6     like     know  article  \n",
      "7     know     like     good  \n",
      "8      com      new     like  \n",
      "9     time    sound   better  \n",
      "\n",
      "\n",
      "Gensim \n",
      "\n",
      "   topic_0  topic_1    topic_2   topic_3     topic_4  topic_5  topic_6  \\\n",
      "0    space      mac    turkish      game     article      com      com   \n",
      "1     year     file     people      time      ground  article     card   \n",
      "2     nasa    image       said     think        good     know    drive   \n",
      "3      new      gif   armenian      like  encryption      apr     mail   \n",
      "4     team  windows        day     sound        like      key     know   \n",
      "5  article      bit  president     games  government    phone      use   \n",
      "6     like      use  armenians       got      better      use     work   \n",
      "7      gov    files   genocide      year       right     need      dos   \n",
      "8    radio   images      turks      best         car      new  article   \n",
      "9     play  program      today  baseball     average     like  problem   \n",
      "\n",
      "      topic_7    topic_8  topic_9  \n",
      "0      people       jews      god  \n",
      "1        like    article   people  \n",
      "2   insurance  objective    think  \n",
      "3      israel      black  article  \n",
      "4     private     jewish     know  \n",
      "5     article        art      way  \n",
      "6  government     values  believe  \n",
      "7      health   morality     like  \n",
      "8        batf      moral      com  \n",
      "9       money      frank    right  \n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "topic_words = dict()\n",
    "gensim_topics = lda_gensim_mc.show_topics(formatted=False)\n",
    "def sklearn_show_topics(model, feature_names, n_top_words):\n",
    "    sk_topics = []\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        tot_score = np.sum(topic)\n",
    "        top_words = [(feature_names[i],topic[i]/tot_score)\n",
    "            for i in topic.argsort()[:-n_top_words - 1:-1]]\n",
    "        sk_topics.append([topic_idx,top_words])\n",
    "    return sk_topics\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "sklearn_topics = sklearn_show_topics(lda_sklearn, feature_names,10)\n",
    "topic_words['gensim']  = gensim_topics\n",
    "topic_words['sklearn'] = sklearn_topics\n",
    "\n",
    "# or in data frame formta\n",
    "topic_words_df = dict()\n",
    "for model, result in topic_words.iteritems():\n",
    "    df = pd.DataFrame()\n",
    "    for topic in result:\n",
    "        cols =  [[word[0] for word in topic[1]] for topic in result]\n",
    "        for i,c in enumerate(cols):\n",
    "            df[\"topic_\"+str(i)] = c\n",
    "    topic_words_df[model] = df\n",
    "\n",
    "print('Sklearn \\n')\n",
    "print(topic_words_df['sklearn'])\n",
    "print('\\n')\n",
    "print('Gensim \\n')\n",
    "print(topic_words_df['gensim'])"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}