Online NMF

docs/notebooks/nmf_tutorial.ipynb

@@ -0,0 +1,302 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tutorial on Online Non-Negative Matrix Factorization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebooks explains basic ideas behind NMF implementation, training examples and use-cases."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "lines_to_next_cell": 2
+   },
+   "source": [
+    "**Matrix Factorizations** are useful for many things: recomendation systems, bi-clustering, image compression and, in particular, topic modeling.\n",
+    "\n",
+    "Why **Non-Negative**? It makes the problem more strict and allows us to apply some optimizations.\n",
+    "\n",
+    "Why **Online**? Because corpora are large and RAM is limited. Online NMF can learn topics iteratively.\n",
+    "\n",
+    "This particular implementation is based on [this paper](arxiv.org/abs/1604.02634)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "from gensim import matutils\n",
+    "from gensim.models.nmf import Nmf\n",
+    "from gensim.models import CoherenceModel\n",
+    "from gensim.parsing.preprocessing import preprocess_string\n",
+    "from sklearn.datasets import fetch_20newsgroups"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dataset preprocessing"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "categories = [\n",
+    "    'alt.atheism',\n",
+    "    'comp.graphics',\n",
+    "    'rec.motorcycles',\n",
+    "    'talk.politics.mideast',\n",
+    "    'sci.space'\n",
+    "]\n",
+    "\n",
+    "trainset = fetch_20newsgroups(subset='train', categories=categories, random_state=42)\n",
+    "testset = fetch_20newsgroups(subset='test', categories=categories, random_state=42)\n",
+    "\n",
+    "train_documents = [preprocess_string(doc) for doc in trainset.data]\n",
+    "test_documents = [preprocess_string(doc) for doc in testset.data]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dictionary compilation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from gensim.corpora import Dictionary\n",
+    "\n",
+    "dictionary = Dictionary(train_documents)\n",
+    "\n",
+    "dictionary.filter_extremes()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Corpora compilation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_corpus = [\n",
+    "    dictionary.doc2bow(document)\n",
+    "    for document\n",
+    "    in train_documents\n",
+    "]\n",
+    "\n",
+    "test_corpus = [\n",
+    "    dictionary.doc2bow(document)\n",
+    "    for document\n",
+    "    in test_documents\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Training\n",
+    "\n",
+    "The API works in the way similar to [Gensim.models.LdaModel](https://radimrehurek.com/gensim/models/ldamodel.html).\n",
+    "\n",
+    "Specific parameters:\n",
+    "\n",
+    "- `use_r` - whether to use residuals. Effectively adds regularization to the model\n",
+    "- `kappa` - optimizer step size coefficient.\n",
+    "- `lambda_` - residuals coefficient. The larger it is, the less more regularized result gets.\n",
+    "- `sparse_coef` - internal matrices sparse coefficient. The more it is, the faster and less accurate training is."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 12.4 s, sys: 1.08 s, total: 13.5 s\n",
+      "Wall time: 13.7 s\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%time\n",
+    "\n",
+    "nmf = Nmf(\n",
+    "    corpus=train_corpus,\n",
+    "    chunksize=1000,\n",
+    "    num_topics=5,\n",
+    "    id2word=dictionary,\n",
+    "    passes=5,\n",
+    "    eval_every=10,\n",
+    "    minimum_probability=0,\n",
+    "    random_state=42,\n",
+    "    use_r=True,\n",
+    "    lambda_=1000,\n",
+    "    kappa=1,\n",
+    "    sparse_coef=3\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Topics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[(0,\n",
+       "  '0.035*\"god\" + 0.030*\"atheist\" + 0.021*\"believ\" + 0.020*\"exist\" + 0.019*\"atheism\" + 0.016*\"religion\" + 0.013*\"christian\" + 0.013*\"religi\" + 0.013*\"peopl\" + 0.012*\"argument\"'),\n",
+       " (1,\n",
+       "  '0.055*\"imag\" + 0.054*\"jpeg\" + 0.033*\"file\" + 0.024*\"gif\" + 0.021*\"color\" + 0.019*\"format\" + 0.015*\"program\" + 0.014*\"version\" + 0.013*\"bit\" + 0.012*\"us\"'),\n",
+       " (2,\n",
+       "  '0.053*\"space\" + 0.034*\"launch\" + 0.024*\"satellit\" + 0.017*\"nasa\" + 0.016*\"orbit\" + 0.013*\"year\" + 0.012*\"mission\" + 0.011*\"data\" + 0.010*\"commerci\" + 0.010*\"market\"'),\n",
+       " (3,\n",
+       "  '0.022*\"armenian\" + 0.021*\"peopl\" + 0.020*\"said\" + 0.018*\"know\" + 0.011*\"sai\" + 0.011*\"went\" + 0.010*\"come\" + 0.010*\"like\" + 0.010*\"apart\" + 0.009*\"azerbaijani\"'),\n",
+       " (4,\n",
+       "  '0.024*\"graphic\" + 0.017*\"pub\" + 0.015*\"mail\" + 0.013*\"data\" + 0.013*\"ftp\" + 0.012*\"send\" + 0.011*\"imag\" + 0.011*\"rai\" + 0.010*\"object\" + 0.010*\"com\"')]"
+      ]
+     },
+     "execution_count": 31,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "nmf.show_topics()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Coherence"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "-1.6698708891486376"
+      ]
+     },
+     "execution_count": 32,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "CoherenceModel(\n",
+    "    model=nmf,\n",
+    "    corpus=test_corpus,\n",
+    "    coherence='u_mass'\n",
+    ").get_coherence()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Perplexity"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def perplexity(model, corpus):\n",
+    "    W = model.get_topics().T\n",
+    "\n",
+    "    H = np.zeros((W.shape[1], len(corpus)))\n",
+    "    for bow_id, bow in enumerate(corpus):\n",
+    "        for topic_id, proba in model[bow]:\n",
+    "            H[topic_id, bow_id] = proba\n",
+    "    \n",
+    "    dense_corpus = matutils.corpus2dense(corpus, W.shape[0])\n",
+    "    \n",
+    "    return np.exp(-(np.log(W.dot(H), where=W.dot(H)>0) * dense_corpus).sum() / dense_corpus.sum())\n",
+    "\n",
+    "perplexity(nmf, test_corpus)"
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "text_representation": {
+    "extension": ".py",
+    "format_name": "percent",
+    "format_version": "1.1",
+    "jupytext_version": "0.8.3"
+   }
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}