week10_nlp&w2v/1. Train_w2v.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Как научить компьютер читать? \n",
    "\n",
    "В этой тетрадке мы обучим свой собственный word2vec. Делать мы это будем на каком-нибудь не очень большом тексте, который вам предстоит выбрать самому. На выбор есть [несколько сказок](https://github.com/nevmenandr/word2vec-russian-novels/tree/master/vector-school) и других [литературных штук](https://github.com/nevmenandr/word2vec-russian-novels/tree/master/books_before) из школьной программы. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ссылка на выбранное вами произведение\n",
    "# Я взял преступление и наказание (ненвижу Достоевского)\n",
    "url = 'https://github.com/nevmenandr/word2vec-russian-novels/master/books_before/CrimeAndPunishment.txt'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Спарсим текст из файлика."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "смотреть на эти семь лет, как на семь\n",
      "дней. Он даже и не знал того, что новая жизнь не даром же ему достается, что ее надо еще дорого купить, заплатить за нее великим, будущим подвигом...\n",
      "Но тут уж начинается новая история, история постепенного обновления человека, история постепенного перерождения его, постепенного перехода из одного мира в другой, знакомства с новою, доселе совершенно неведомою действительностью. Это могло бы составить тему нового рассказа, - но теперешний рассказ наш окончен.\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "resp = requests.get(url)\n",
    "text = resp.text \n",
    "\n",
    "# Последние 500 символов. Аккуратно! Спойлеры!\n",
    "print(text[-500:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Предобработка\n",
    "\n",
    "Теперь нам надо его немного предобработать.  Пусть все слова пишутся с маленькой буквы. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = text.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разобьём весь текст на предложения. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re \n",
    "# выкидываем лишние символы! \n",
    "text = re.sub('\\n|\\t|\\r', ' ', text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !/usr/local/opt/python@3.9/bin/python3.9 -m pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/fulyankin/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /Users/fulyankin/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import nltk\n",
    "\n",
    "# nltk.download('all') # если хотим всё и сразу \n",
    "\n",
    "# нам хватит вот этого: \n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13702"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "sents = sent_tokenize(text)\n",
    "len(sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'действительно, на его платье и даже в волосах кое-где виднелись прилипшие былинки сена.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents[220]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разобьём каждое предложение на отдельные слова."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['действительно',\n",
       " 'на',\n",
       " 'его',\n",
       " 'платье',\n",
       " 'и',\n",
       " 'даже',\n",
       " 'в',\n",
       " 'волосах',\n",
       " 'кое',\n",
       " 'где',\n",
       " 'виднелись',\n",
       " 'прилипшие',\n",
       " 'былинки',\n",
       " 'сена']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "\n",
    "tokenizer = RegexpTokenizer('\\w+')\n",
    "tokenizer.tokenize(sents[220])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# разбейте все предложения на токены \n",
    "sents_tokenize  =  [tokenizer.tokenize(item) for item in sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten без numpy :) \n",
    "words = [item for sent in  sents_tokenize for item in sent]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "173403"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words) # всего слов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24925"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(words)) # уникальных слов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно выбросить все стоп-слова. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['и', 'в', 'во', 'не', 'что', 'он', 'на', 'я', 'с', 'со']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "stopwords_ru = stopwords.words('russian') \n",
    "stopwords_ru[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "151"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(stopwords_ru)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# избавьтесь от стоп-слов \n",
    "sents_tokenize = [[item for item in sent if item not in stopwords_ru]\n",
    "                       for sent in sents_tokenize ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Слов в корпусе не очень много. Давайте лемматизируем их.  В этом нам поможет библиотека **pymorphy2.**\n",
    "\n",
    "**pymorphy2** — это полноценный морфологический анализатор, целиком написанный на Python. Он также умеет ставить слова в нужную форму (спрягать и склонять). [Документация по pymorphy2.](https://pymorphy2.readthedocs.io/en/latest/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Collecting pymorphy2\n",
      "  Downloading pymorphy2-0.9.1-py3-none-any.whl (55 kB)\n",
      "\u001b[K     |████████████████████████████████| 55 kB 774 kB/s eta 0:00:01\n",
      "\u001b[?25hCollecting docopt>=0.6\n",
      "  Using cached docopt-0.6.2.tar.gz (25 kB)\n",
      "Collecting dawg-python>=0.7.1\n",
      "  Using cached DAWG_Python-0.7.2-py2.py3-none-any.whl (11 kB)\n",
      "Collecting pymorphy2-dicts-ru<3.0,>=2.4\n",
      "  Downloading pymorphy2_dicts_ru-2.4.417127.4579844-py2.py3-none-any.whl (8.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 8.2 MB 4.7 MB/s eta 0:00:01\n",
      "\u001b[?25hBuilding wheels for collected packages: docopt\n",
      "  Building wheel for docopt (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13705 sha256=360f985f483b636b99128c6323ea333a11e5fae6a5481e57fd83a00d192a2485\n",
      "  Stored in directory: /Users/fulyankin/Library/Caches/pip/wheels/70/4a/46/1309fc853b8d395e60bafaf1b6df7845bdd82c95fd59dd8d2b\n",
      "Successfully built docopt\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Installing collected packages: pymorphy2-dicts-ru, docopt, dawg-python, pymorphy2\n",
      "\u001b[33m  WARNING: The script pymorphy is installed in '/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Successfully installed dawg-python-0.7.2 docopt-0.6.2 pymorphy2-0.9.1 pymorphy2-dicts-ru-2.4.417127.4579844\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.1.2; however, version 21.2.4 is available.\n",
      "You should consider upgrading via the '/usr/local/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pymorphy2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'филипп пойти в авеньон и пленить папа'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pymorphy2\n",
    "morph = pymorphy2.MorphAnalyzer()\n",
    "\n",
    "text = \"Филипп пошёл в Авеньон и пленил пап!\"\n",
    "tokens = tokenizer.tokenize(text)\n",
    "\n",
    "\" \".join(morph.normal_forms(token)[0] for token in tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parse(word='стали', tag=OpencorporaTag('VERB,perf,intr plur,past,indc'), normal_form='стать', score=0.975342, methods_stack=((DictionaryAnalyzer(), 'стали', 945, 4),)),\n",
       " Parse(word='стали', tag=OpencorporaTag('NOUN,inan,femn sing,gent'), normal_form='сталь', score=0.010958, methods_stack=((DictionaryAnalyzer(), 'стали', 13, 1),)),\n",
       " Parse(word='стали', tag=OpencorporaTag('NOUN,inan,femn plur,nomn'), normal_form='сталь', score=0.005479, methods_stack=((DictionaryAnalyzer(), 'стали', 13, 6),)),\n",
       " Parse(word='стали', tag=OpencorporaTag('NOUN,inan,femn sing,datv'), normal_form='сталь', score=0.002739, methods_stack=((DictionaryAnalyzer(), 'стали', 13, 2),)),\n",
       " Parse(word='стали', tag=OpencorporaTag('NOUN,inan,femn sing,loct'), normal_form='сталь', score=0.002739, methods_stack=((DictionaryAnalyzer(), 'стали', 13, 5),)),\n",
       " Parse(word='стали', tag=OpencorporaTag('NOUN,inan,femn plur,accs'), normal_form='сталь', score=0.002739, methods_stack=((DictionaryAnalyzer(), 'стали', 13, 9),))]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = morph.parse('стали')\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['стать', 'сталь']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "morph.normal_forms('стали')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обработаем все слова из датасета. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# лемматизируйте все слова из датасета\n",
    "sents_tokenize = [[morph.normal_forms(item)[0] for item in sent] \n",
    "                    for sent in sents_tokenize]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten без numpy :) \n",
    "words = [item for sent in  sents_tokenize for item in sent]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "93069"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words) # всего слов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11084"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(words)) # уникальных слов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Хватит обработок! Мы тут не анализом текстов занимаемся, а нейросетками. Если хочешь больше предобработки, [читай мой мануал](https://nbviewer.jupyter.org/github/FUlyankin/hse_texts_do/blob/master/sem_1/texts_sem1.ipynb) об этом.  Давайте построим словарик с частотностями и перейдём к моделированию. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('это', 1449),\n",
       " ('всё', 963),\n",
       " ('знать', 630),\n",
       " ('раскольник', 567),\n",
       " ('свой', 549),\n",
       " ('один', 548),\n",
       " ('сказать', 544),\n",
       " ('говорить', 536),\n",
       " ('человек', 501),\n",
       " ('весь', 442),\n",
       " ('стать', 441),\n",
       " ('мочь', 441),\n",
       " ('который', 430),\n",
       " ('сам', 430),\n",
       " ('такой', 395),\n",
       " ('очень', 387),\n",
       " ('какой', 379),\n",
       " ('соня', 379),\n",
       " ('рука', 369),\n",
       " ('петрович', 369)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_dict = Counter(words)\n",
    "word_dict.most_common()[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4768"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = word_dict.most_common()\n",
    "len([item for item in words if item[1] >= 3])  # совсем мало :) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Моделирование\n",
    "\n",
    "__Основные параметры:__\n",
    "\n",
    "* данные должны быть итерируемым объектом \n",
    "* `size` — размер вектора, \n",
    "* `window` — размер окна наблюдения,\n",
    "* `min_count` — мин. частотность слова в корпусе,\n",
    "* `sg` — используемый алгоритм обучения (0 — CBOW, 1 — Skip-gram),\n",
    "* `sample` — порог для downsampling'a высокочастотных слов,\n",
    "* `workers` — количество потоков,\n",
    "* `alpha` — learning rate,\n",
    "* `iter` — количество итераций,\n",
    "* `max_vocab_size` — позволяет выставить ограничение по памяти при создании словаря (т.е. если ограничение привышается, то низкочастотные слова будут выбрасываться). Для сравнения: 10 млн слов = 1Гб RAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Collecting gensim\n",
      "  Downloading gensim-4.0.1.tar.gz (23.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 23.1 MB 3.5 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.9/site-packages (from gensim) (1.19.4)\n",
      "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.9/site-packages (from gensim) (1.5.4)\n",
      "Collecting smart_open>=1.8.1\n",
      "  Downloading smart_open-5.1.0-py3-none-any.whl (57 kB)\n",
      "\u001b[K     |████████████████████████████████| 57 kB 3.6 MB/s eta 0:00:011\n",
      "\u001b[?25hBuilding wheels for collected packages: gensim\n",
      "  Building wheel for gensim (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for gensim: filename=gensim-4.0.1-cp39-cp39-macosx_10_12_x86_64.whl size=23825345 sha256=ec2195eeb3b6cefd175c6108dfad3cd0f3127975652714abc8a3a17d00244d8d\n",
      "  Stored in directory: /Users/fulyankin/Library/Caches/pip/wheels/20/74/75/72ec1172891bdecb4ee73fbc2c71d5a150f165b1d0c2ea04e1\n",
      "Successfully built gensim\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Installing collected packages: smart-open, gensim\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "Successfully installed gensim-4.0.1 smart-open-5.1.0\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -tatsmodels (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -ip (/usr/local/lib/python3.9/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.1.2; however, version 21.2.4 is available.\n",
      "You should consider upgrading via the '/usr/local/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4.0.1'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gensim\n",
    "gensim.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#%%time \n",
    "from gensim.models.word2vec import Word2Vec\n",
    "\n",
    "# size - размерность векторов, которые мы хотим обучить\n",
    "# window - ширина окна контекста\n",
    "# min_count - если слово встречается реже, для него не учим модель\n",
    "model = Word2Vec(vector_size=100, window=2, min_count=3, workers=-1)\n",
    "\n",
    "# строительство словаря, чтобы обучение шло быстрее\n",
    "model.build_vocab(sents_tokenize)\n",
    "\n",
    "# обучение модели \n",
    "# первый аргумент - наша выборка, генератор будет вкидывать в модель наши тексты, пока они не кончатся\n",
    "# второй аргумент - число примеров в выборке \n",
    "# третий аргумент - количество эпох обучения: сколько раз модель пройдётся по всему корпусу текстов\n",
    "model.train(sents_tokenize, total_examples=model.corpus_count, epochs=100)\n",
    "\n",
    "# !NB в ситуации, когда у нас огромный корпус, 100 эпох это слишком много! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13702"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.corpus_count # число примеров в обучающей выборке"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Смотрим, сколько в модели слов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4768"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model.wv.key_to_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'старуха' in model.wv.key_to_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Свойства модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00158984, -0.00595314, -0.00605843,  0.00848177, -0.00034839,\n",
       "        0.00706192, -0.00888268, -0.00192542, -0.00676588, -0.0072364 ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# вектор слова\n",
    "model.wv['старуха'][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100,)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# размерность вектора\n",
    "model.wv['старуха'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.03793418"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# похожести слов \n",
    "model.wv.similarity('тварь', 'право')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('выехать', 0.3434045910835266),\n",
       " ('вицмундир', 0.3418233096599579),\n",
       " ('нравиться', 0.3186732232570648),\n",
       " ('капернаум', 0.3163142502307892),\n",
       " ('преступник', 0.3118082284927368),\n",
       " ('случайность', 0.29301321506500244),\n",
       " ('поворотить', 0.2834480106830597),\n",
       " ('обыкновение', 0.2801940143108368),\n",
       " ('буля', 0.27475351095199585),\n",
       " ('поручить', 0.2660590708255768)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# самые похожие\n",
    "model.wv.most_similar('топор')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('съехать', 0.3254740834236145),\n",
       " ('раздражать', 0.3184410631656647),\n",
       " ('линия', 0.31684330105781555),\n",
       " ('покоситься', 0.30329668521881104),\n",
       " ('детски', 0.29883092641830444),\n",
       " ('чашка', 0.29052451252937317),\n",
       " ('вырываться', 0.2874678075313568),\n",
       " ('влево', 0.2853025794029236),\n",
       " ('вызывать', 0.2818695604801178),\n",
       " ('холодно', 0.28120535612106323)]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# арифметика\n",
    "model.wv.most_similar(positive=['раскольников','соня'], \n",
    "                       negative=['тварь'])[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Как дообучить модель? \n",
    "\n",
    "Ради чистоты эксперимента сохраним текущую модель и заново подгрузим её. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = \"./our_w2v.model\"\n",
    "model.save(model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "our_model = Word2Vec.load(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подгрузим другое произведение и сделаем для него предобработку. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " узнает...\n",
      "В нем взыграло ретивое!\n",
      "\"Что я вижу? что такое?\n",
      "Как!\" - и дух в нем занялся...\n",
      "Царь слезами залился,\n",
      "Обнимает он царицу,\n",
      "И сынка, и молодицу,\n",
      "\n",
      "\n",
      "\n",
      "И садятся все за стол;\n",
      "И веселый пир пошел.\n",
      "А ткачиха с поварихой,\n",
      "С сватьей бабой Бабарихой\n",
      "Разбежались по углам;\n",
      "Их нашли насилу там.\n",
      "Тут во всем они признались,\n",
      "Повинились, разрыдались;\n",
      "Царь для радости такой\n",
      "Отпустил всех трех домой.\n",
      "День прошел - царя Салтана\n",
      "Уложили спать вполпьяна.\n",
      "Я там был; мед, пиво пил -\n",
      "И усы лишь обмочил.\n",
      "\n",
      "1831\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = 'https://github.com/nevmenandr/word2vec-russian-novels/master/vector-school/SkazkaOCareSaltane.txt'\n",
    "\n",
    "resp = requests.get(url)\n",
    "text2 = resp.text \n",
    "\n",
    "# Последние 500 символов. Аккуратно! Спойлеры!\n",
    "print(text2[-500:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "text2 = text2.lower()\n",
    "sents2 = sent_tokenize(text2)\n",
    "\n",
    "sents_tokenize2 = [tokenizer.tokenize(sent) for sent in sents2]\n",
    "sents_tokenize2 = [[morph.normal_forms(word)[0] for word in text_cur if word not in stopwords_ru]\n",
    "                      for text_cur in sents_tokenize2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['сени', 'выйти', 'царь', 'отец']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents_tokenize2[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sents_tokenize2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дополняем модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# обновили словарь\n",
    "our_model.build_vocab(sents_tokenize2, update=True)\n",
    "\n",
    "# дообучили модель\n",
    "our_model.train(sents_tokenize2, \n",
    "                total_examples=our_model.corpus_count, \n",
    "                epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'ядро' in model.wv.key_to_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'ядро' in our_model.wv.key_to_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('право', 0.3656421899795532),\n",
       " ('первый', 0.3411722481250763),\n",
       " ('изнеможение', 0.33265551924705505),\n",
       " ('рассеянность', 0.3231242001056671),\n",
       " ('механически', 0.3048047721385956),\n",
       " ('зе', 0.2968287169933319),\n",
       " ('далеко', 0.2877439260482788),\n",
       " ('опыт', 0.286418080329895),\n",
       " ('выхлопотать', 0.2828163504600525),\n",
       " ('голубчик', 0.2819357216358185)]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_model.wv.most_similar('ядро')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Пример со старым словом."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('помолчать', 0.3307487666606903),\n",
       " ('набережный', 0.32425227761268616),\n",
       " ('лесть', 0.3182099759578705),\n",
       " ('нарушить', 0.3141959011554718),\n",
       " ('нетерпеливо', 0.3112514019012451),\n",
       " ('гоголь', 0.30818116664886475),\n",
       " ('кончиться', 0.30746209621429443),\n",
       " ('сесть', 0.3021995723247528),\n",
       " ('снисходительный', 0.30076098442077637),\n",
       " ('интересоваться', 0.29836300015449524)]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_model.wv.most_similar('сын')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('помолчать', 0.3307487666606903),\n",
       " ('набережный', 0.32425227761268616),\n",
       " ('лесть', 0.3182099759578705),\n",
       " ('нарушить', 0.3141959011554718),\n",
       " ('нетерпеливо', 0.3112514019012451),\n",
       " ('гоголь', 0.30818116664886475),\n",
       " ('кончиться', 0.30746209621429443),\n",
       " ('сесть', 0.3021995723247528),\n",
       " ('снисходительный', 0.30076098442077637),\n",
       " ('интересоваться', 0.29836300015449524)]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('сын')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}