examples/train_lenet_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "two4two_leNet.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOW4fdcQTwVyJmLA6A3PH9q",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mschuessler/two4two/blob/LoMedHiVarSamplers/examples/train_lenet_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fdgP1RGA4hzg"
      },
      "source": [
        "import pathlib\n",
        "import os\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras as keras\n",
        "from tensorflow.keras import layers\n",
        "import pandas as pd\n",
        "from keras_preprocessing.image import ImageDataGenerator\n"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7WVdllt_HB7V"
      },
      "source": [
        "# Two4two data training\n",
        "The notebook demonstrates how to train a modern LeNet CNN on a Dataset pregenerated with the [two4two Module](https://github.com/mschuessler/two4two).\n",
        "\n",
        "If you open this notebook in Colab please make sure to request a GPU Instance. Training times will be excessively slow otherwise.\n",
        "\n",
        "As a first step we define a relative path where the trained model should be saved."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RntmEihRfpPv"
      },
      "source": [
        "relative_model_path = \"two4two_example_model\""
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPVYMHZkJuj8"
      },
      "source": [
        "# Mounting Google drive to save trained model later\n",
        "Since colab is a free resource it runtimes are limited. Hence we want to save our model to reuse it late when our reserved instance terminates. To do this we mount a google drive. If this notebook is not run inside of collab the following cells will skip the mounting of Google drive and save the model to your local directory from which this notebook is executed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q4tF_WKwe3Ia"
      },
      "source": [
        "if 'google.colab' in str(get_ipython()):\n",
        "  from google.colab import drive\n",
        "  mount_drive = True\n",
        "else:\n",
        "  mount_drive = False"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g9FWWAq3J19A",
        "outputId": "e395f9e9-b2cc-4a1a-a5a9-2e2eaee11b85"
      },
      "source": [
        "if mount_drive:\n",
        "  from google.colab import drive\n",
        "  drive.mount(\"/content/gdrive\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "giYoHx93manY"
      },
      "source": [
        "We will now define the path to which or trained model will be saved. You may alter the name of the dictionary according to your preference. If you run this notebook outside of Google Colab just change this path to a local path of your notebook server."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S_h3e1rxmHuk"
      },
      "source": [
        "model_filepath = os.path.join(\"/content/gdrive/My Drive\", relative_model_path) if mount_drive else relative_model_path"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nlYVeSFmm3xK"
      },
      "source": [
        "We will use the the callback functionality of keras to save our model whenever we achieved a new highest validation accuracy when training the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B4_PRQ0sKrY0"
      },
      "source": [
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=model_filepath,\n",
        "    save_weights_only=False,\n",
        "    monitor='val_accuracy',\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hk7DT8FzJcrA"
      },
      "source": [
        "# Defining model architecture / loading previously trained model\n",
        "The follwoing cell tests wheter the dictionary where the model should be saved exist, if so we try to load it.\n",
        "\n",
        "If the directory does not exist we define a new model according to a modern LeNet architecture and complie it using the ADAM optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NfhE_gqFkawN"
      },
      "source": [
        "trained_model_exists = os.path.exists(model_filepath)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cxWnqQLbFarq"
      },
      "source": [
        "if trained_model_exists:\n",
        "  modernLenetModel = keras.models.load_model(model_filepath)\n",
        "else:\n",
        "  modernLenetModel = keras.models.Sequential([\n",
        "        layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\"),\n",
        "        layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "        layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n",
        "        layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "        layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n",
        "        layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "        layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n",
        "        layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "        layers.Flatten(),\n",
        "        layers.Dropout(0.5),\n",
        "        layers.Dense(2, activation=\"softmax\"),\n",
        "    ])\n",
        "  modernLenetModel.compile(loss=\"categorical_crossentropy\",\n",
        "                             optimizer=\"adam\", metrics=[\"accuracy\"])\n",
        "    "
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-XkRyErrlmTG"
      },
      "source": [
        "# Download dataset\n",
        "In this example we will be using a pregenerated dataset package. It contains a larger dataset called \"spherical_color_bias\". This dataset contains two biases. The sphercitiy of the blocks as well as their color are somewhat predictive of the label peaky or stretchy. The package contains 80.000 images for training, 500 for validation and 3.000 for testing.\n",
        "\n",
        "It also contains the testing data of 3000 images each for four other datasets that contain only one or none of the two biases. One dataset does not even have the correct arm positions.\n",
        "\n",
        "Hence, we would expect that if we train a model on the dataset with two biases it should before worse on the other test sets, if the biases are used by the model for its predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PgTNO0q6JaAa"
      },
      "source": [
        "datasets = [\"spherical_color_bias\", \"no_arms\", \"no_bias\", \"spherical_bias\", \"color_bias\"]"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rAN-gfkP4jnx",
        "outputId": "6a77f0fe-ecf2-42f4-d0fa-d439a0e068d0"
      },
      "source": [
        "data_dir = keras.utils.get_file(\n",
        "    origin=\"https://f001.backblazeb2.com/file/two4two/datasets_models/golden80k.tar.gz\",\n",
        "    fname=\"two4two_datasets\",\n",
        "    untar=True\n",
        ")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://f001.backblazeb2.com/file/two4two/datasets_models/golden80k.tar.gz\n",
            "2189754368/2189754121 [==============================] - 57s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g-f2GEpIJO0V"
      },
      "source": [
        "# Reading dataframe from jsonl"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "CkEYzrNUoEfq",
        "outputId": "d7a5de6d-a997-4b1e-ead0-33d03735ec9e"
      },
      "source": [
        "data_dir"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'/root/.keras/datasets/two4two_datasets'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m7Qi_MIL4r0p"
      },
      "source": [
        "train_dir = os.path.join(data_dir, datasets[0], \"train\")\n",
        "train_df = pd.read_json(os.path.join(train_dir, \"parameters.jsonl\"), lines=True)\n",
        "train_df[\"filename\"] = train_df[\"id\"] + \".png\""
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xmx53eyxFPlj"
      },
      "source": [
        "valid_dir = os.path.join(data_dir, datasets[0], \"validation\")\n",
        "valid_df = pd.read_json(os.path.join(valid_dir, \"parameters.jsonl\"), lines=True)\n",
        "valid_df[\"filename\"] = valid_df[\"id\"] + \".png\""
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7jgyTApTJUY2"
      },
      "source": [
        "# Creating Datagenerator from dataframes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SNaYUXbBFXJG",
        "outputId": "0bc8e35c-1247-4230-c175-e15ff46dfbfc"
      },
      "source": [
        "datagen = ImageDataGenerator(rescale=1. / 255)\n",
        "train_generator = datagen.flow_from_dataframe(dataframe=train_df, directory=train_dir,\n",
        "                                              x_col=\"filename\", y_col=\"label\", batch_size=64)\n",
        "valid_generator = datagen.flow_from_dataframe(dataframe=valid_df, directory=valid_dir,\n",
        "                                              x_col=\"filename\", y_col=\"label\", batch_size=64)\n",
        "STEP_SIZE_TRAIN = train_generator.n // train_generator.batch_size\n",
        "STEP_SIZE_VALID = valid_generator.n // valid_generator.batch_size"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 80000 validated image filenames belonging to 2 classes.\n",
            "Found 500 validated image filenames belonging to 2 classes.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QkR0thUgJqNB"
      },
      "source": [
        "# Train Model\n",
        "We highly recommend to train the model at least for 10 or even better 20 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sCeQePhAFvRy",
        "outputId": "88ca0392-977e-4b55-f898-640751709864"
      },
      "source": [
        "modernLenetModel.fit(train_generator,\n",
        "                     steps_per_epoch=STEP_SIZE_TRAIN,\n",
        "                     validation_data=valid_generator,\n",
        "                     validation_steps=STEP_SIZE_VALID,\n",
        "                     epochs=5,\n",
        "                     callbacks = [model_checkpoint_callback]\n",
        "                     )"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "1250/1250 [==============================] - 241s 191ms/step - loss: 0.3685 - accuracy: 0.8190 - val_loss: 0.2235 - val_accuracy: 0.9129\n",
            "INFO:tensorflow:Assets written to: /content/gdrive/My Drive/two4two_example_model/assets\n",
            "Epoch 2/5\n",
            "1250/1250 [==============================] - 197s 157ms/step - loss: 0.1799 - accuracy: 0.9248 - val_loss: 0.1315 - val_accuracy: 0.9464\n",
            "INFO:tensorflow:Assets written to: /content/gdrive/My Drive/two4two_example_model/assets\n",
            "Epoch 3/5\n",
            "1250/1250 [==============================] - 193s 154ms/step - loss: 0.1534 - accuracy: 0.9356 - val_loss: 0.1229 - val_accuracy: 0.9487\n",
            "INFO:tensorflow:Assets written to: /content/gdrive/My Drive/two4two_example_model/assets\n",
            "Epoch 4/5\n",
            "1250/1250 [==============================] - 192s 154ms/step - loss: 0.1351 - accuracy: 0.9436 - val_loss: 0.1048 - val_accuracy: 0.9598\n",
            "INFO:tensorflow:Assets written to: /content/gdrive/My Drive/two4two_example_model/assets\n",
            "Epoch 5/5\n",
            "1250/1250 [==============================] - 191s 153ms/step - loss: 0.1277 - accuracy: 0.9455 - val_loss: 0.0976 - val_accuracy: 0.9621\n",
            "INFO:tensorflow:Assets written to: /content/gdrive/My Drive/two4two_example_model/assets\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7fb7b9344c50>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bstPOz7spG57"
      },
      "source": [
        "# Evaluating the model\n",
        "We now use our trained model on the test sets of all datasets. As expected the model perfroms worse on the models were not all biases are present."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IqkiCAWvXq_T",
        "outputId": "85a1f03e-299d-4e8b-a2f4-ed12a79a6a3d"
      },
      "source": [
        "for dataset_name in datasets:\n",
        "        test_dir = os.path.join(data_dir, dataset_name, \"test\")\n",
        "        test_df = pd.read_json(os.path.join(test_dir, \"parameters.jsonl\"), lines=True)\n",
        "        test_df[\"filename\"] = test_df[\"id\"] + \".png\"\n",
        "\n",
        "        datagen = ImageDataGenerator(rescale=1. / 255)\n",
        "        test_generator = datagen.flow_from_dataframe(dataframe=test_df, directory=test_dir,\n",
        "                                                     x_col=\"filename\", y_col=\"label\",\n",
        "                                                     batch_size=64)\n",
        "\n",
        "        print(\"Evaluating on \" + dataset_name)\n",
        "        modernLenetModel.evaluate(test_generator)[1]"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 3000 validated image filenames belonging to 2 classes.\n",
            "Evaluating on spherical_color_bias\n",
            "47/47 [==============================] - 9s 184ms/step - loss: 0.1076 - accuracy: 0.9570\n",
            "Found 3000 validated image filenames belonging to 2 classes.\n",
            "Evaluating on no_arms\n",
            "47/47 [==============================] - 8s 171ms/step - loss: 1.5017 - accuracy: 0.6977\n",
            "Found 3000 validated image filenames belonging to 2 classes.\n",
            "Evaluating on no_bias\n",
            "47/47 [==============================] - 8s 172ms/step - loss: 2.3019 - accuracy: 0.5550\n",
            "Found 3000 validated image filenames belonging to 2 classes.\n",
            "Evaluating on spherical_bias\n",
            "47/47 [==============================] - 8s 171ms/step - loss: 0.9158 - accuracy: 0.7653\n",
            "Found 3000 validated image filenames belonging to 2 classes.\n",
            "Evaluating on color_bias\n",
            "47/47 [==============================] - 8s 175ms/step - loss: 1.0872 - accuracy: 0.7663\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}