Keras-Tutorials/4. LSTM Text Generation/Keras LSTM Text Generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2KEVdYD8GdsK"
      },
      "source": [
        "# Keras LSTM Text Generation\n",
        "Text generation is a important nlp problem which can enable computers to write."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1PD0sLRMGdsR"
      },
      "source": [
        "<table align=\"left\"><td>\n",
        "  <a target=\"_blank\"  href=\"https://colab.research.google.com/github/TannerGilbert/Tutorials/blob/master/Keras-Tutorials/4.%20LSTM%20Text%20Generation/Keras%20LSTM%20Text%20Generation.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab\n",
        "  </a>\n",
        "</td><td>\n",
        "  <a target=\"_blank\"  href=\"https://github.com/TannerGilbert/Tutorials/blob/master/Keras-Tutorials/4.%20LSTM%20Text%20Generation/Keras%20LSTM%20Text%20Generation.ipynb\">\n",
        "    <img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "</td></table>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "crXWQg1vGdsT"
      },
      "outputs": [],
      "source": [
        "from __future__ import print_function\n",
        "from tensorflow.keras.callbacks import LambdaCallback\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation\n",
        "from tensorflow.keras.layers import LSTM\n",
        "from tensorflow.keras.optimizers import RMSprop\n",
        "import numpy as np\n",
        "import random\n",
        "import sys"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "55y3JXJYGvzD",
        "outputId": "d95f1a9a-cc6c-4281-dcee-b781e328c0b1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2021-03-24 07:32:09--  http://www.gutenberg.org/files/1661/1661-0.txt\n",
            "Resolving www.gutenberg.org (www.gutenberg.org)... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47\n",
            "Connecting to www.gutenberg.org (www.gutenberg.org)|152.19.134.47|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607792 (594K) [text/plain]\n",
            "Saving to: ‘sherlock_homes.txt’\n",
            "\n",
            "sherlock_homes.txt  100%[===================>] 593.55K  3.40MB/s    in 0.2s    \n",
            "\n",
            "2021-03-24 07:32:09 (3.40 MB/s) - ‘sherlock_homes.txt’ saved [607792/607792]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget -O sherlock_homes.txt http://www.gutenberg.org/files/1661/1661-0.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OClKM8W6GdsW",
        "outputId": "c6dce257-40e8-4e44-e8ab-0cefefd34c49"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "text length 581889\n"
          ]
        }
      ],
      "source": [
        "text = open('sherlock_homes.txt', 'r').read().lower()\n",
        "print('text length', len(text))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "puWNoMs1GdsW",
        "outputId": "b26de140-c1f5-4066-f35c-21ea76aa7d06"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "project gutenberg's the adventures of sherlock holmes, by arthur conan doyle\n",
            "\n",
            "this ebook is for the use of anyone anywhere at no cost and with\n",
            "almost no restrictions whatsoever.  you may copy it, give it away or\n",
            "re-use it under the terms of the project gutenberg license included\n",
            "with this ebook or online at www.gutenberg.org\n",
            "\n",
            "\n",
            "title: the adventures of sherlock holmes\n",
            "\n",
            "author: arthur conan doyle\n",
            "\n",
            "release date: november 29, 2002 [ebook #1661]\n",
            "last updated: may 20, 2019\n",
            "\n",
            "language: english\n",
            "\n",
            "character set encoding: utf-8\n",
            "\n",
            "*** start of this project gutenberg ebook the adventures of sherlock holmes ***\n",
            "\n",
            "\n",
            "\n",
            "produced by an anonymous project gutenberg volunteer and jose menendez\n",
            "\n",
            "\n",
            "\n",
            "cover\n",
            "\n",
            "\n",
            "\n",
            "the adventures of sherlock holmes\n",
            "\n",
            "\n",
            "\n",
            "by arthur conan doyle\n",
            "\n",
            "\n",
            "\n",
            "contents\n",
            "\n",
            "\n",
            "   i.     a scandal in bohemia\n",
            "   ii.    the red-headed league\n",
            "   iii.   a case of identity\n",
            "   iv.    the boscombe valley mystery\n",
            "   v.     the five orange pips\n",
            "   vi.    the man with the twisted lip\n",
            "   vii.   the adventure of the blue \n"
          ]
        }
      ],
      "source": [
        "print(text[:1000])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVysZNijGdsX"
      },
      "source": [
        "## Map chars to integers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HKGcwfhSGdsY",
        "outputId": "d96c7d2d-9bb1-47aa-d12c-ca5289897cd3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "total chars:  73\n"
          ]
        }
      ],
      "source": [
        "chars = sorted(list(set(text)))\n",
        "print('total chars: ', len(chars))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9ZUHkl80GdsZ"
      },
      "outputs": [],
      "source": [
        "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
        "indices_char = dict((i, c) for i, c in enumerate(chars))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hT5KU09SGdsZ"
      },
      "source": [
        "## Split up into subsequences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WQaXsgQJGdsa",
        "outputId": "4802fcc3-7b7a-4e51-d432-c0cab959be24"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "nb sequences: 193950\n"
          ]
        }
      ],
      "source": [
        "maxlen = 40\n",
        "step = 3\n",
        "sentences = []\n",
        "next_chars = []\n",
        "for i in range(0, len(text) - maxlen, step):\n",
        "    sentences.append(text[i: i + maxlen])\n",
        "    next_chars.append(text[i + maxlen])\n",
        "print('nb sequences:', len(sentences))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "165pGeTLGdsa",
        "outputId": "c75ca0bb-b9ff-4346-893b-ec347b810f6c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\"\\ufeff\\nproject gutenberg's the adventures of \", \"roject gutenberg's the adventures of she\", \"ect gutenberg's the adventures of sherlo\"]\n",
            "['s', 'r', 'c']\n"
          ]
        }
      ],
      "source": [
        "print(sentences[:3])\n",
        "print(next_chars[:3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dtThqXy_Gdsc"
      },
      "outputs": [],
      "source": [
        "x = np.zeros((len(sentences), maxlen, len(chars)), dtype=bool)\n",
        "y = np.zeros((len(sentences), len(chars)), dtype=bool)\n",
        "for i, sentence in enumerate(sentences):\n",
        "    for t, char in enumerate(sentence):\n",
        "        x[i, t, char_indices[char]] = 1\n",
        "    y[i, char_indices[next_chars[i]]] = 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DtTb0587Gdsc",
        "outputId": "59126279-bd80-46f7-f286-4cca828ce432"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[[[False False False ... False False  True]\n",
            "  [ True False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  ...\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  [False  True False ... False False False]]\n",
            "\n",
            " [[False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  ...\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]]\n",
            "\n",
            " [[False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  ...\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]\n",
            "  [False False False ... False False False]]]\n",
            "[[False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False  True False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False]\n",
            " [False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False  True False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False]\n",
            " [False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False  True\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False False False False False False False False False False False False\n",
            "  False]]\n"
          ]
        }
      ],
      "source": [
        "print(x[:3])\n",
        "print(y[:3])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-kDWNjNGdsd"
      },
      "source": [
        "## Building Model\n",
        "\n",
        "In this notebook a small recurrent neural networks is used for both simplicity and because of the training time but if you want to train a more sophisticated model you can increase the size of the network. You can also use a model pretrained on some other text like wikipedia text to both speed up the training process and get better results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8-8-wlhYGdse"
      },
      "outputs": [],
      "source": [
        "model = Sequential()\n",
        "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n",
        "model.add(Dense(len(chars)))\n",
        "model.add(Activation('softmax'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6gJqXggRGdse"
      },
      "outputs": [],
      "source": [
        "optimizer = RMSprop(learning_rate=0.01)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AbKTa2IsGdsf"
      },
      "source": [
        "## Helper Functions\n",
        "These helper functions are taken from the [official Keras text generation notebook](https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ocMf4eOfGdsf"
      },
      "outputs": [],
      "source": [
        "def sample(preds, temperature=1.0):\n",
        "    # helper function to sample an index from a probability array\n",
        "    preds = np.asarray(preds).astype('float64')\n",
        "    preds = np.log(preds) / temperature\n",
        "    exp_preds = np.exp(preds)\n",
        "    preds = exp_preds / np.sum(exp_preds)\n",
        "    probas = np.random.multinomial(1, preds, 1)\n",
        "    return np.argmax(probas)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i55LQ-1EGdsf"
      },
      "outputs": [],
      "source": [
        "def on_epoch_end(epoch, logs):\n",
        "    # Function invoked at end of each epoch. Prints generated text.\n",
        "    print()\n",
        "    print('----- Generating text after Epoch: %d' % epoch)\n",
        "\n",
        "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
        "    for diversity in [0.2, 0.5, 1.0, 1.2]:\n",
        "        print('----- diversity:', diversity)\n",
        "\n",
        "        generated = ''\n",
        "        sentence = text[start_index: start_index + maxlen]\n",
        "        generated += sentence\n",
        "        print('----- Generating with seed: \"' + sentence + '\"')\n",
        "        sys.stdout.write(generated)\n",
        "\n",
        "        for i in range(400):\n",
        "            x_pred = np.zeros((1, maxlen, len(chars)))\n",
        "            for t, char in enumerate(sentence):\n",
        "                x_pred[0, t, char_indices[char]] = 1.\n",
        "\n",
        "            preds = model.predict(x_pred, verbose=0)[0]\n",
        "            next_index = sample(preds, diversity)\n",
        "            next_char = indices_char[next_index]\n",
        "\n",
        "            generated += next_char\n",
        "            sentence = sentence[1:] + next_char\n",
        "\n",
        "            sys.stdout.write(next_char)\n",
        "            sys.stdout.flush()\n",
        "        print()\n",
        "print_callback = LambdaCallback(on_epoch_end=on_epoch_end)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7FeR2KQ3Gdsf"
      },
      "source": [
        "## Defining callbacks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hZBfLCbfGdsg"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.callbacks import ModelCheckpoint\n",
        "\n",
        "filepath = \"weights.hdf5\"\n",
        "checkpoint = ModelCheckpoint(filepath, monitor='loss',\n",
        "                             verbose=1, save_best_only=True,\n",
        "                             mode='min')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YHFMBrXUGdsg"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.callbacks import ReduceLROnPlateau\n",
        "reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.2,\n",
        "                              patience=1, min_lr=0.001)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kR_hzINvGdsh"
      },
      "outputs": [],
      "source": [
        "callbacks = [print_callback, checkpoint, reduce_lr]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mIIuZ1B1Gdsi"
      },
      "source": [
        "## Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cqImw0CcGdsi",
        "outputId": "fca36fe9-73a3-4e66-b5b8-a8cbe4962799"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "1516/1516 [==============================] - 189s 123ms/step - loss: 2.2542\n",
            "\n",
            "----- Generating text after Epoch: 0\n",
            "----- diversity: 0.2\n",
            "----- Generating with seed: \"veted, and he gave a little cry of\n",
            "satis\"\n",
            "veted, and he gave a little cry of\n",
            "satister of the starter of the sint of the stant of the stant of the can the stant of the bent had been the count to the state in the counter in the start in the saters of the man to the sent the startion of the stant of the sall be to the stoner and the care to the stoner what he was a can the started to the bention and the countly to the some to the starter of the some to the stoner man to the contre\n",
            "----- diversity: 0.5\n",
            "----- Generating with seed: \"veted, and he gave a little cry of\n",
            "satis\"\n",
            "veted, and he gave a little cry of\n",
            "satistation which he was been a firder so the sa a to matter farish and the been the wind be a seart be in the\n",
            "bent in the\n",
            "counted of the can out on the bearss a stare and be the best in the tan is in the bent stently with the fall in it was the bent in the can windot which i have been been man explared the counted that i thong were the banbar in erglaced when his grefing me was have been man have seen\n",
            "----- diversity: 1.0\n",
            "----- Generating with seed: \"veted, and he gave a little cry of\n",
            "satis\"\n",
            "veted, and he gave a little cry of\n",
            "satis were hand my a“kere his head, fantilly in our iralman” stepple aschis sauled tinned in hisy sune themeghance\n",
            "which and the beg to lipple now\n",
            "of the rotherm refuted it and in his canstant, that his waysy and that\n",
            "it asten cloafd. there should be mince,’ said i’ have agroc“tom. it was gook the nome could sunk aschaw from the might weeply. “naw the beg, halm.\n",
            "“only starding there, and be to seee i c\n",
            "----- diversity: 1.2\n",
            "----- Generating with seed: \"veted, and he gave a little cry of\n",
            "satis\"\n",
            "veted, and he gave a little cry of\n",
            "satisn.\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "“which she wourn beit kinds. o, sax the man say looked’s me overhey-and\n",
            "cenole as hocsely comeention, i to mr, “venied restdon’s you sen\n",
            "unto, hommermat carc and down. in that he\n",
            "said sasted seem headycesmon, whicis know,’ed centen matres backir?” i.. serredid, exploie he seaw, sokes which you veen eye a\n",
            "shotfum.\n",
            " he was defired! i drint.\n",
            "\n",
            "“was paunt bank, ; amlering\n",
            "aovelsserved to maireed\n",
            "\n",
            "Epoch 00001: loss improved from inf to 1.96468, saving model to weights.hdf5\n",
            "Epoch 2/5\n",
            "1516/1516 [==============================] - 191s 126ms/step - loss: 1.6393\n",
            "\n",
            "----- Generating text after Epoch: 1\n",
            "----- diversity: 0.2\n",
            "----- Generating with seed: \"i was clear. the question now was, who w\"\n",
            "i was clear. the question now was, who was seen me the secter of the state the secter of the served the state and the state the work and the went in the was a little had been and the was a little the window which we went the conticuate and the work and the state which we went the sever the contine and the wend the condicual seart of the window which he had been the state the state and the state the secter of the secter in the wend the s\n",
            "----- diversity: 0.5\n",
            "----- Generating with seed: \"i was clear. the question now was, who w\"\n",
            "i was clear. the question now was, who was sented the will which we was doubt the wropped the probable when he was so him leane i was so the wenter the cepone which was seen the white of excensial wenter in the seat the secort and the dear in the secked in the west and in the states when he has see the one and the connice which wis he had seen in the contered me the winding the was he down the far years i wis come i wis the state it whi\n",
            "----- diversity: 1.0\n",
            "----- Generating with seed: \"i was clear. the question now was, who w\"\n",
            "i was clear. the question now was, who wwenteney and it’s mortay, he looked.\n",
            "she, you \n",
            "\n",
            "\n",
            "“that you clmile, she with her let se. sims at your slacked.” i wen? gerted.”\n",
            "\n",
            "“wnew reseed his venter!’ sciet then all and the little in there came and he wigh did it post so- holmes, but i some in dingeece. then that you shall a look in that i trusted over holmes so heard i went with his was ”\n",
            "led im?” he raked his foar at be permin andutertive of\n",
            "----- diversity: 1.2\n",
            "----- Generating with seed: \"i was clear. the question now was, who w\"\n",
            "i was clear. the question now was, who would under the, womno lifter much puy.\n",
            "\n",
            "“hisch of the \n",
            "centally and he had -en. onmer trunk; you lathered thery\n",
            "hound the stmen’ d. i remp! frot seen dresenty untinste stowers.”\n",
            "\n",
            "“as she all the gi, we st\n",
            "andoleft and disprronvil .”\n",
            "\n",
            "“are rove duyase there grasped decen death, sid a own ong back scauch one.’\n",
            "\n",
            "“no, raad of eacyom\n",
            "unce wull, wenter veryonp, in you,” he tood,” said hoq\n",
            "verly and my e\n",
            "\n",
            "Epoch 00002: loss improved from 1.96468 to 1.61977, saving model to weights.hdf5\n",
            "Epoch 3/5\n",
            "1516/1516 [==============================] - 190s 125ms/step - loss: 1.5184\n",
            "\n",
            "----- Generating text after Epoch: 2\n",
            "----- diversity: 0.2\n",
            "----- Generating with seed: \" was on board of a ship.”\n",
            "\n",
            "“excellent. w\"\n",
            " was on board of a ship.”\n",
            "\n",
            "“excellent. what in the cried which we have a late. it was a string the present of the string and the door which we have the door in the strengther the way of the clair and the propest of the present of the string and any man which i have the consideral late and which we have the country which i am she had been of the propessed and which i cauld street the strengther and the constant of the strength the profes\n",
            "----- diversity: 0.5\n",
            "----- Generating with seed: \" was on board of a ship.”\n",
            "\n",
            "“excellent. w\"\n",
            " was on board of a ship.”\n",
            "\n",
            "“excellent. we are man was a matter, and suspertance in the concluantle of the let of his from his propect of the room. in the confession of the streets of the strong business darked of the\n",
            "ground and strengther of the detersed the days of a drighted from\n",
            "the course which a bark any dingled to face and of one of course propest\n",
            "as remark when the nawration was a father and a head a way weened the from all the d\n",
            "----- diversity: 1.0\n",
            "----- Generating with seed: \" was on board of a ship.”\n",
            "\n",
            "“excellent. w\"\n",
            " was on board of a ship.”\n",
            "\n",
            "“excellent. while writiques is a case in a wilat.\n",
            "my muries can take gresiand resettracrail is keep as showen.\n",
            "‘led steps your\n",
            " one business as of a\n",
            "worance that his crias catter evatences which he was saw unmide’s agreawsages. string pass of the crieds of his listent for mirrainked him here ret” curry-amady droved spee as every thin! but he disesting. we amil from the dark in the rever a bawraperstooes. the n\n",
            "----- diversity: 1.2\n",
            "----- Generating with seed: \" was on board of a ship.”\n",
            "\n",
            "“excellent. w\"\n",
            " was on board of a ship.”\n",
            "\n",
            "“excellent. we have been inoye?’ bece‘mens, slight to humainly bus fror the sund elseer, his evimined onrawnorg frightion instant, and nevimed you her more to me tatenedyhable. “then\n",
            "absorn it put from\n",
            "his\n",
            "wins over him been make fundgesting inbleftsoriplip fasions of\n",
            "icre. advicked the babirablenow, shndwained\n",
            "anything wamsheaps and bee\n",
            "syaring and lerfess?”\n",
            "\n",
            "“yes led it drowry hour as he caunde. way doyinis.\n",
            "\n",
            "Epoch 00003: loss improved from 1.61977 to 1.52188, saving model to weights.hdf5\n",
            "Epoch 4/5\n",
            "1516/1516 [==============================] - 190s 125ms/step - loss: 1.4685\n",
            "\n",
            "----- Generating text after Epoch: 3\n",
            "----- diversity: 0.2\n",
            "----- Generating with seed: \"nd went up to my bedroom again,\n",
            "where i \"\n",
            "nd went up to my bedroom again,\n",
            "where i have the strengther which was a confiderable which was a confidered and street of the reach of the conniced and the stone which was a man of the confection of the way of the ware of the confidered and the there was a condical was and was a considerable which was and the continue and the forture which was a confection of the face and the confidered and the face and the states of the way of the stat\n",
            "----- diversity: 0.5\n",
            "----- Generating with seed: \"nd went up to my bedroom again,\n",
            "where i \"\n",
            "nd went up to my bedroom again,\n",
            "where i have been me and and the man who was a deather. she was a the barrates of the continue the states which was and a cight which seemed his weared all in to the strengther stard and the stands of the\n",
            "still of the arrested of the servant of the\n",
            "way we was a month of business. and there had been but with a tever and for seem of the concearing from himself and and there was a bard of the man which was a\n",
            "----- diversity: 1.0\n",
            "----- Generating with seed: \"nd went up to my bedroom again,\n",
            "where i \"\n",
            "nd went up to my bedroom again,\n",
            "where i resse at was purpine a gind. he had ever excentret though ready before, from go maliny here-broad, on\n",
            "ween\n",
            "this cornor who was a busikes not took with the eventsmeed for toodabthed, for he did go water. ‘of a dak.\n",
            "\n",
            "“he would not anyn shigting, there is to sook watson there was junt and half for it was any indeatford to cernightured tow. there murdying anrepain that you see as towar. he is streetsw\n",
            "----- diversity: 1.2\n",
            "----- Generating with seed: \"nd went up to my bedroom again,\n",
            "where i \"\n",
            "nd went up to my bedroom again,\n",
            "where i arecup ebgra spraat?”\n",
            "\n",
            "“but the cuntier you. there, and then aref didcell.\n",
            "they comver to leesing courd,\n",
            "mids of viouse in coupsings of the\n",
            "kind-dable is then, which and eburim.for. mr. my other would reabsotters pirced me theel’s, a mccart engla heid. twent. liblingd by h probably he beporbardine implios even with in dohsom of vered withed bet at yard?’ you gase,” he beg had two\n",
            "mdoped. ejccvicat\n",
            "\n",
            "Epoch 00004: loss improved from 1.52188 to 1.47073, saving model to weights.hdf5\n",
            "Epoch 5/5\n",
            "1516/1516 [==============================] - 191s 126ms/step - loss: 1.4254\n",
            "\n",
            "----- Generating text after Epoch: 4\n",
            "----- diversity: 0.2\n",
            "----- Generating with seed: \"ock holmes was wrong in his conjecture, \"\n",
            "ock holmes was wrong in his conjecture, and the commence of the shoulder and the door of the shoulder and the companion and all the shoulder and a should be a conterest of the comply and a the comply that i should be some was a crime and the door of the shoulder and the compries and the colone of the secouse and a companion of the station of the companion and the comply in the conteller of the state and the resure of the shoulder of the\n",
            "----- diversity: 0.5\n",
            "----- Generating with seed: \"ock holmes was wrong in his conjecture, \"\n",
            "ock holmes was wrong in his conjecture, and the quick. it was the companion. it was not the haded in the purpence to the crime and the silence. i am a lock the compricts in the drasing of the reservical of the small and his appeared to the ross that we am some part in several prong the lady of scare his done so in a cring of the been firstal part in the companion of the matter is have a shure and for his chair and myself in the change o\n",
            "----- diversity: 1.0\n",
            "----- Generating with seed: \"ock holmes was wrong in his conjecture, \"\n",
            "ock holmes was wrong in his conjecture, and that he pirp wat what horse, whoel sh“vely by one nard, you sckluming a when piving as apthe nor lens,\n",
            "which small roohs. my more has fourdperent, i find, which an upionemered.”\n",
            "\n",
            "“you keeply i summs of you can invisioncial is an abtloving mal\n",
            "head i samiculad with as burbow shide’s suffide black our said bag. therefirs . the\n",
            "fire, asked away slipsed apand.\n",
            "\n",
            "“this i\n",
            "feely also right was no unme\n",
            "----- diversity: 1.2\n",
            "----- Generating with seed: \"ock holmes was wrong in his conjecture, \"\n",
            "ock holmes was wrong in his conjecture, you being men now you rark find my bi, mr.\n",
            "\n",
            "“you say round away a hroubs, as the lesswomen tiunas mr. will passmant. i never only was told firs for k   wequcentiffered, now, it. there cambing hisriatemes should yand,” casely i one thrould returning that he is that the lagal of such and\n",
            "as is. on\n",
            "saped, it was shwute’s have not\n",
            "ago person what well gently criw his mid.fibed,” said holmes\n",
            "shell litt\n",
            "\n",
            "Epoch 00005: loss improved from 1.47073 to 1.43966, saving model to weights.hdf5\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f7691b259d0>"
            ]
          },
          "execution_count": 18,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.fit(x, y, batch_size=128, epochs=5, callbacks=callbacks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UNkTgpPGGdsj"
      },
      "source": [
        "## Testing the model\n",
        "Now that we have a trained network we can test it using a method simular to the ``on_epoch_end`` method above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "86882eXFGdsj"
      },
      "outputs": [],
      "source": [
        "def generate_text(length, diversity):\n",
        "    # Get random starting text\n",
        "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
        "    generated = ''\n",
        "    sentence = text[start_index: start_index + maxlen]\n",
        "    generated += sentence\n",
        "    for i in range(length):\n",
        "            x_pred = np.zeros((1, maxlen, len(chars)))\n",
        "            for t, char in enumerate(sentence):\n",
        "                x_pred[0, t, char_indices[char]] = 1.\n",
        "\n",
        "            preds = model.predict(x_pred, verbose=0)[0]\n",
        "            next_index = sample(preds, diversity)\n",
        "            next_char = indices_char[next_index]\n",
        "\n",
        "            generated += next_char\n",
        "            sentence = sentence[1:] + next_char\n",
        "    return generated"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MmZxXlaKGdsj",
        "outputId": "d7687244-5422-4afc-a210-815c4245e0aa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "dy between the wharf and the\n",
            "house. it seemed to the comply and the companion of the shoulder the shoulder and and the corner of the state of the shoulder of the station of the was and a comply and a comprict of the state and had been a charce and a comprict of the state of the some of the companion and the compries of the part of the ble of the companion and the confiated the state and a companion of the trust of the comprict in the london and the companion. he was a crime and the state and and the companion of the shoulder and the s\n"
          ]
        }
      ],
      "source": [
        "print(generate_text(500, 0.2))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Keras LSTM Text Generation.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}