Load-and-Inspect-Model-Predictions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing/inspecting the learning rate over time and what the model learned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jan 14 03:57:40 2019       \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |\r\n",
      "|-------------------------------+----------------------+----------------------+\r\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
      "|===============================+======================+======================|\r\n",
      "|   0  GeForce GTX 108...  Off  | 00000000:02:00.0 Off |                  N/A |\r\n",
      "|  0%   32C    P8    17W / 280W |   9470MiB / 11178MiB |      0%      Default |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "|   1  GeForce GTX 108...  Off  | 00000000:81:00.0 Off |                  N/A |\r\n",
      "|  0%   30C    P8    16W / 280W |     10MiB / 11178MiB |      0%      Default |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "                                                                               \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| Processes:                                                       GPU Memory |\r\n",
      "|  GPU       PID   Type   Process name                             Usage      |\r\n",
      "|=============================================================================|\r\n",
      "|    0      9591      C   ...o/miniconda3/envs/tensorflow/bin/python   737MiB |\r\n",
      "|    0     14274      C   ...o/miniconda3/envs/tensorflow/bin/python  8723MiB |\r\n",
      "+-----------------------------------------------------------------------------+\r\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pytest\r\n",
      "pytest-cov\r\n",
      "joblib\r\n",
      "torch==1.0.0\r\n",
      "torchvision==0.2.1\r\n",
      "scikit-learn==0.20.1\r\n",
      "numpy==1.15.4\r\n"
     ]
    }
   ],
   "source": [
    "# !pip install joblib\n",
    "# !echo \"joblib\" >> requirements.txt\n",
    "# !pip freeze | grep -i torch >> requirements.txt\n",
    "# !pip freeze | grep -i numpy >> requirements.txt\n",
    "!cat requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.data.read_txt import *\n",
    "from src.data.config import *\n",
    "from src.data.training_data import *\n",
    "from src.data.sgnn_projection_layer import *\n",
    "from src.model.loss import *\n",
    "from src.model.transformer import *\n",
    "from src.model.save_load_model import *\n",
    "from src.training import *\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.metrics import jaccard_similarity_score, f1_score, accuracy_score\n",
    "from joblib import dump, load\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import math\n",
    "import copy\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 160\n",
    "train_iters_per_epoch = 24000\n",
    "max_epoch = 11\n",
    "cuda_device_id = 1  # None for CPU, 0 for first GPU, 1 for second GPU, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my-model.pytorch.epoch_00000.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00000.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00001.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00001.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00002.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00002.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00003.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00003.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00004.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00004.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00005.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00005.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00006.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00006.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00007.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00007.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00008.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00008.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00009.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00009.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00010.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00010.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00011.notebook_run.gpu0\r\n",
      "my-model.pytorch.epoch_00011.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00012.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00013.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00014.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00015.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00016.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00017.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00018.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00019.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00020.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00021.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00022.notebook_run.gpu1\r\n",
      "my-model.pytorch.epoch_00023.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00000.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00000.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00001.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00001.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00002.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00002.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00003.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00003.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00004.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00004.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00005.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00005.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00006.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00006.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00007.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00007.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00008.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00008.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00009.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00009.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00010.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00010.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00011.notebook_run.gpu0\r\n",
      "my-model.sklearn.epoch_00011.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00012.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00013.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00014.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00015.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00016.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00017.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00018.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00019.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00020.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00021.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00022.notebook_run.gpu1\r\n",
      "my-model.sklearn.epoch_00023.notebook_run.gpu1\r\n"
     ]
    }
   ],
   "source": [
    "!ls -1 models_weights/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded model from files: ./models_weights/my-model.sklearn.epoch_00011.notebook_run.gpu0 ./models_weights/my-model.pytorch.epoch_00011.notebook_run.gpu0\n"
     ]
    }
   ],
   "source": [
    "preproc_sgnn_sklearn_pipeline, sentence_projection_model = load_model(\n",
    "    \"my-model{}.epoch_00011.notebook_run.gpu0\", cuda_device_id)\n",
    "# preproc_sgnn_sklearn_pipeline, sentence_projection_model = load_most_recent_model(MY_MODEL_NAME, cuda_device_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_trainer = TrainerModel(sentence_projection_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the learning rate over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ0AAAEICAYAAACEdClSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsnXlcVdX6/98Ph0lUEBBUQAUFBxBHTLs2WGZqNpfNWb/M5tvg7Zbdoek2D7fhW91uXbtl82CD3ayszMoyzbGcR1IUBQFRZDoH1u+PvcHD4QCHGfR5v1682HuNzx4/51lr7bXEGIOiKIqitAR+rW2AoiiKcvSgoqMoiqK0GCo6iqIoSouhoqMoiqK0GCo6iqIoSouhoqMoiqK0GEel6IjI5yJyRWvboTQ/IhIvIkZE/H1M/6qIPNDcdjWU+h5Pc2PbktiI/AtF5OqmtKmpEZGxIpLRiPwvisjfm9Km9kyLio6IpIvIKS1ZpzeMMZOMMa+1th3QPh46pfVoK89MUyAi94rIG81Y/k0iskxESkTkVS/x40Rkg4gUisi3ItLbLS5IRF4RkQMiskdEZjTQhitFZJF7mDHmOmPMPxpSXlPRlu6jI87TaSu/AKFt2VIf2qvdSnWOsmu5G3gAeMUzQkS6Ah8CfwcigGXAu25J7gWSgN7AScAdIjKxme09OjHGtNgfkA6cUkPc6cAqYD/wEzDYLW4msBU4CKwDznGLuxL4EXgKyMG66a4EFgFPAHnAdmCSW56FwNVu+WtLmwB8b9f9NfA88EYNxzAWyADuBPYArwPhwP+AbLv8/wFxdvoHgTKgGCgAnrPDBwBfAbnARuCCGuo7CfjNbf8r4Be3/R+AsxtxDivC9gPbgD/Y4TuBLOAKb+fU/by67RvgZrucfcDjgF8Nx3UMsNiuNxN4Dgj0KOs6YLOd5nlA7DiHfS332XXdaKf3r6GuYcAK+7y8C7wDPODjfdkT60WWbZ+3iuvXF1hgh+0D3gS62HF/BuZ42PAs8IwX214HyoEi+/64A4i3j+cKYIdd/l/d8twLfAC8ARwArgaCgKexXsq77e0gb9fJ7fwm2tuRwKd2Wb/Y94bndfV6LTzKnAiUAk77WFa73Tf/wLrXDgLzga5u+Ubb530/sBoY68N75gHgVY+wa4Cf3PY72ud1gL2/GzjVLf4fwDvuz7Vb3M1Yz1CcRx0DsZ7lMvsY99vhr1bcUxx+R9yB9QxlAmcDpwGbsJ75v7iV6cfhZzcHeA+IqOG4u2K9X/bb5fxg5692H9V1bu3r8jCw1L72n1TUCwRj3V85dt5fgG4+64CvCZvijxpEB+vBzwJGYb00rrDTVjwYU4AY+wReCBwCerg9NC7gj4A/0MEOcwLT7fKut2+qihfTQqqKTm1pF2O9xAKB4+wLUJvouIBHsR70DlgP7XlACNAZeB/42OPiur+sO2K91P+ffTzDsF4syV7q64B1k3cFAoC9wC67ng72TRbZiHPosu1wYD3IO7BeKkHAqVgviU41HMeVVH85fYv1K7MX1gN2dQ3ncQTWA+GP9ZJdD9zqUdb/gC52WdnARDvuOmADliBE2HV6FR37mv4O3Gafv/Pte6HiBVHjfWnvr8YS5Y5YD+Jxdr5EYLydLgrrR8vTdlwP+9xXiJC/XccIX54ZDovOy/Z1GgKUAAPt+HvtYzjbvtYdgPuBn4Fo256fgH94u05u57dCdN6x/0KAZKx70/O6er0WXo7lXjyeHfu+2Qr0s21dCDxix8VivdhOs49lvL0fVcd7xpvoPAP8yyNsDdazGW4fRze3uPOxf9DhJjrA3Vg/UrzaUMP5fJWqouOyywnAeu9kA29hPbcpWM9tgp3+Fvvaxdn307+Bt2uo+2HgRbvcAOB4Dr/HPO+jWs+tfR12AYOw7u85FdcOuBbrh0gI1nMwAgj1WQfqKxyN+fM8cLfwf2E/BG5hG4ETayhnFXCW20Xe4eXCb3HbD7Fvqu5uJ/TqutJiPUQuIMQt/g1qF51SILiWczAUyPN46Nxf1hcCP3jk+TdwTw3l/QCci/WSno/1S2gilhf0ay12+HION7vtp1L9wcwBhtZwHFdS/eU00W3/BuAbH++bW4GPPMo6zm3/PWCmvb0AuM4t7lRqFp0TcPuBYYf9xOEXRI33JXAs1svCqwflkedsYKXb/ufAdHv7dGCdr88Mh0Unzi1sKXCRvX0v8L1HGVuB09z2JwDp3q6T2/lNxHqhOIH+bnHePB2v18LLsdyLd9H5m8d98YW9fSfwukf6L3HzsGuox5vozMIWM7ewH+3j72kfR7Bb3Hi3czQW6wX8T6xWkbBa6vZ2Pl+lqugUAQ57v7Nd9yi39Ms53EKxHhjnFtfDvibe7uf7sTySRB/uo1rPLW7ib+8nY73bHMBVeHj99flrK306vYE/icj+ij+sGyEGQESmisgqt7hBWL/uK9jppcw9FRvGmEJ7s1MN9deUNgbIdQurqS53so0xxRU7IhIiIv8Wkd9F5ADWr94uIuKoIX9vYJTHubgUSwS98R3WjXyCvb0Q66V4or1fYUdDzuFet+0iAGOMZ1hN59Qb7nX8jn19PRGRfiLyP7tD9wDwkIet4HbNgEI3O2K81FMTMcAuYz9VXtLXdl/2BH43xri82N9NRN4RkV22/W942P8acJm9fRlW80d9qen4ofq1jPE4rhrPvQdRWJ6Ye3m1PmtebPGFmvL3BqZ4nP/jsF689aUACPUIC8Xy1gvc9j3jKuiC1UT3sDEmvwH1u5NjjCmzt4vs/zU9V72Bj9yOfz1W8103L+U+DmwB5ovINhGZWYsNvpxbz+coAOs+fh1LoN4Rkd0i8piIBNRxzJW0FdHZCTxojOni9hdijHnbHmHyMnATVlNRFyy3WNzyGy9lNgWZQISIhLiF9awjj6ctfwL6Y/2SCcUSBzhsv2f6ncB3HueikzHm+hrq8xSd7/AQnRY6h4ewvMQKvImk+7nrheVleONfWE1kSfY5+wtVba2NTC/11JY2VkTcy3ZPX+N9acf1qqGj/iGs85lq23+Zh/0fA4NFZBCWp/NmLTY25Lp45tmN9ZKpwP3cV7luIuJ+3bKxPP04t7C67v/62FUXO7F+jbuf/47GmEcaUPdarKZIAESkI1bf21pjTB7WvTDELf0QO08FeVjX6r8iMqaWepr6XbQTq4/Z/RwEG2N2VavYmIPGmD8ZY/oAZwIzRGRcDXb5cm49nyMnsM8Y4zTG3GeMScbq5z0dmOrrAbWG6ASISLDbnz/WC/E6ERklFh1FZLKIdMZqTzRYDwAi8v+wfqU3O8aY37FGudwrIoEicixwRj2L6Yz1y2W/iEQA93jE7wX6uO3/D+gnIpeLSID9N1JEBtZQ/k9YonYMsNQYsxbbW8LyqqBlzuEq4Fzbs0sEpnlJ82cRCReRnlht1e96SQPWOTsAFIjIAKx+Nl95D7hZROJEJByrE7YmFmO9VG+2z/O5WOexgtruy6VYL6pH7PBgt5dRZ6xfz/kiEos1eKAS2xP+AKsdf6kxZkctNnreHw3hbeBvIhJlj+K6G8v7AqtfKkVEhopIMFYTWIWdZVgDJe61r+sA6vFy8cJeIF5EfH3vvAGcISITRMRhn+OxIhLnLbGI+NvH4AAq0lf8KPgIGCQi59lp7sZqft5gx8/GOkfh9nFOx2oWq8QYsxCr1eFDEXG/TzyPMU5EAn08xrp4EXjQ/uGIfQ3P8pZQRE4XkUT7R1Q+lkdU7maX+33ky7m9TESS7R/d9wMfGGPKROQkEUm1W2sOYIlROT7SGqIzD+slXPF3rzFmGdZFfg7rF8UWrLZRjDHrgCexXhB7sfoWfmxBey/Far+vGNX1LlbHra88jdVBug+rQ/ALj/hngPNFJE9EnjXGHMTqh7gI69foHg4PTKiGMeYQVsfmWmNMqR28GKvpJ8tO0xLn8CmsNt+9WM1H3n69f4LVXr0K+Ayrnd0btwOXYDVvvEzN4uSNl7Fc/9VY5+XDmhLa5+tcrHstF6s/7UO3+NruyzKsHyCJWAMsMuz8APcBw7Ee/M9qsOE1rOtQV9Paw1gvw/0icnsdaWviAawfT78Cv2Gdlwfs49iE9UL5GmsE2iKPvDcBYRwejfk29bv/3Xnf/p8jIivqSmyM2QmcheXpZmP9Ov8zNb+3/ob1TpmJ5V0W2WEYY7KxBg08iHUtR2E9YxXcg9X39TtWC8HjxhjPZxVjzFdYfRqfishwLzYswPKQ9ojIvrqO0QeeAeZiNZkdxHqHjKohbRLWdSzAetZfMMZ8a8dVuY98PLevYwnvHqyBMjfb4d2xfjQdwGru+85OW/Eh7Iu1HVDFyAbFR0TkXWCDMcbTY1FqQUQMVnPZlta2pS0gIr2wmhC7G2MOtLY9viIij2LZfEVr26I0HyKyEGvQx3+auuy20qfTZrGbtvqKiJ9YH4udhdUmrygNwm5emoH1HUibFhwRGSAig+3mxWOwmk0/am27lPbL0fS1ckPpjtU8EonVhHK9MWZl65qktFfsDuy9WM047eGL985YTWoxWHY/idVMqigNQpvXFEVRlBZDm9cURVGUFuOIaF7r2rWriY+Pb20zFEVR2hXLly/fZ4yJask6jwjRiY+PZ9myZa1thqIoSrtCRGqbsaNZ0OY1RVEUpcVQ0VEURVFaDBUdRVEUpcU4Ivp0FEVpGzidTjIyMiguLq47sdJiBAcHExcXR0CAz5NBNxsqOoqiNBkZGRl07tyZ+Ph4qk7erbQWxhhycnLIyMggISGhtc3xrXlNRCaKyEYR2SJe1mgQkSARedeOXyIi8W5xd9nhG0Vkglv4KyKSJSJraqjzTyJi7FlxFUVpBxQXFxMZGamC04YQESIjI9uM91mn6NjTVz8PTMJaPe5iEUn2SDYNazXMRKzZhh+18yZjzeSagjXlxwtyePGyV6lhGhB76vtTsWbvVRSlHaGC0/ZoS9fEF0/nGKzlnLfZU8G/gzXppTtnYU3VDtaU1+PsNR3OwprUsMQYsx1ravhjAIwx32NNJ++Np4A7aL7F2Wrk4Lff4szMbOlqFUVRjgp8EZ1Yqi5bmmGHeU1jL9+bjzVBpi95q2AvULTLGLO6jnTXiMgyEVmWnZ3tw2H4Rsb1N7D9/ClNVp6iKC1PfHw8qampDB06lLS0NADef/99UlJS8PPzq/Ix+VdffcWIESNITU1lxIgRLFiwoNayn3zySUSEffus5XKMMdx8880kJiYyePBgVqw4vFTQa6+9RlJSEklJSbz22muV4cuXLyc1NZXExERuvvlmjqY5MNvUQAJ7hbq/YDWt1Yox5iXgJYC0tLQmuWKm3Fr8riwnpymKUxSlFfn222/p2vVwl/CgQYP48MMPufbaa6uk69q1K59++ikxMTGsWbOGCRMmsGtXtdWgAdi5cyfz58+nV6/Dq5p//vnnbN68mc2bN7NkyRKuv/56lixZQm5uLvfddx/Lli1DRBgxYgRnnnkm4eHhXH/99bz88suMGjWK0047jS+++IJJkyY1z4loY/ji6eyi6lrZcXaY1zT28rBhWCtt+pLXnb5AArBaRNLt9Cuk6rrtzYZxuVqiGkVRWoGBAwfSv3//auHDhg0jJiYGgJSUFIqKiigp8b446m233cZjjz1WpY/kk08+YerUqYgIo0ePZv/+/WRmZvLll18yfvx4IiIiCA8PZ/z48XzxxRdkZmZy4MABRo8ejYgwdepUPv746FmiyxdP5xcgSUQSsATjIqylhN2ZC1yBtUTq+cACY4wRkbnAWyLyT6z1OJKw1pb3ijHmNyC6Yt8WnjRjTFMs+1onptTZEtUoylHBfZ+uZd3upl2jLjkmlHvOSKkznYhw6qmnIiJce+21XHPNNT6VP2fOHIYPH05QkLU6/NVXX811111HWloan3zyCbGxsQwZMqRKnl27dtGz5+Hf1nFxcezatavW8Li4uGrhRwt1io4xxiUiN2GtO+8AXjHGrBWR+4Flxpi5WGvdvy4iW7AGB1xk510rIu8B6wAXcKO9tjwi8jYwFugqIhnAPcaYWU1+hPXAOEtbs3pFUZqIRYsWERsbS1ZWFuPHj2fAgAGccMIJteZZu3Ytd955J/Pnz68M+89/rNWaCwsLeeihh6rEKQ3Dpz4dY8w8YJ5H2N1u28WA1953Y8yDwINewi/2od54X+xrKoxTPR1FaSp88Uiai9hYa7xSdHQ055xzDkuXLq1VdDIyMjjnnHOYPXs2ffv2rRa/detWtm/fXunlZGRkMHz4cJYuXUpsbCw7d+6sUlZsbCyxsbEsXLiwSvjYsWOJjY0lIyOjWvqjBZ17zR0VHUVp9xw6dIiDBw9Wbs+fP59BgwbVmH7//v1MnjyZRx55hDFjxnhNk5qaSlZWFunp6aSnpxMXF8eKFSvo3r07Z555JrNnz8YYw88//0xYWBg9evRgwoQJzJ8/n7y8PPLy8pg/fz4TJkygR48ehIaG8vPPP2OMYfbs2Zx1ludXKEcuKjpuqKejKO2fvXv3ctxxxzFkyBCOOeYYJk+ezMSJE/noo4+Ii4tj8eLFTJ48mQkTrAlSnnvuObZs2cL999/P0KFDGTp0KFlZWYDVp1PXWl2nnXYaffr0ITExkenTp/PCCy8AEBERwd///ndGjhzJyJEjufvuu4mIiADghRde4OqrryYxMZG+ffseNSPXAORIGB+elpZmmmIRt5LNm9l2xpkADNywvtHlKcrRxvr16xk4cGBrm6F4wdu1EZHlxpi0lrRDPR033D0dU6qDChRFUZoaFR033EWn7NChVrREURTlyERFxw130SlX0VEURWlyVHTcqCI6BQWtaImiKMqRiYqOGyo6iqIozYuKjhtV+nRUdBRFUZocFR03qno62qejKO2V5ljaYNWqVYwePbqyzKVLrWkkdWmDemKMafd/I0aMME3B/rlzzbr+A8y6/gNM7tvvNEmZinI0sW7dutY2wRhjTO/evU12dnaVsHXr1pkNGzaYE0880fzyyy+V4StWrDC7du0yxhjz22+/mZiYGK9ljh8/3sybN88YY8xnn31mTjzxxMrtiRMnmvLycrN48WJzzDHHGGOMycnJMQkJCSYnJ8fk5uaahIQEk5uba4wxZuTIkWbx4sWmvLzcTJw4sbLc5sTbtcGaP7NF39fq6bjhPst0+SFtXlOUI4nGLm0gIhw4YM2anZ+fX5lHlzaoH21qEbfWpkqfzoGDrWiJohwBfD4T9vzWtGV2T4VJj9SZrDmWNnj66aeZMGECt99+O+Xl5fz000+ALm1QX1R03KgqOvmtaImiKI2hqZc2APjXv/7FU089xXnnncd7773HtGnT+Prrr5vtGI5UVHTcqBAdR1RXyvNVdBSlUfjgkTQXTb20AViDAp555hkApkyZwtVXX11Zly5t4Dvap+NGxXLV/l2jKNuvoqMo7ZHmWNoAICYmhu+++w6ABQsWkJSUBKBLG9QT9XTcqFg51D8ykrK8vFa2RlGUhrB3717OOeccAFwuF5dccknl0gZ//OMfyc7OZvLkyQwdOpQvv/yyytIG999/PwDz588nOjq6Sp/Oyy+/zC233ILL5SI4OJiXXnoJsJY2mDdvHomJiYSEhPDf//4XqLq0AVBtaYMrr7ySoqIiJk2apEsbtDeaammDrKefJufl/xA6+TSKVqwk8euvmsA6RTl60KUN2i66tEEbxDidSEAAjrAulGmfjqIoSpPjk+iIyEQR2SgiW0Rkppf4IBF5145fIiLxbnF32eEbRWSCW/grIpIlIms8ynpcRDaIyK8i8pGIdGn44dWPw6ITRvnBg5V9PIqiKErTUKfoiIgDeB6YBCQDF4tIskeyaUCeMSYReAp41M6bDFwEpAATgRfs8gBetcM8+QoYZIwZDGwC7qrnMTUYd9EBKDuo3+ooiqI0Jb54OscAW4wx24wxpcA7gOdQi7OAiomFPgDGiYjY4e8YY0qMMduBLXZ5GGO+B3I9KzPGzDfGVLgYPwNxnmmai0rR6WKLzv79LVW1oijKUYEvohML7HTbz7DDvKaxBSMfiPQxb21cBXzuLUJErhGRZSKyLDs7ux5F1oKHp6Pf6iiKojQtbXYggYj8FXABb3qLN8a8ZIxJM8akRUVFNUmd1ZrXVHQURVGaFF9EZxfQ020/zg7zmkZE/IEwIMfHvNUQkSuB04FLTQuO6VbRUZT2zc6dOznppJNITk4mJSWlcgaBe++9l9jYWIYOHcrQoUOZN29eZZ5ff/2VY489lpSUFFJTUykuLq6x/CeffBIRYd++fYAua9Ag6pqGGusD0m1AAhAIrAZSPNLcCLxob18EvGdvp9jpg+z82wCHW754YI1HWROBdUCUr1NlN9XSBjuuudZsO/c848zNNev6DzA5r81uknIV5WihtZc22L17t1m+fLkxxpgDBw6YpKQks3btWnPPPfeYxx9/vFp6p9NpUlNTzapVq4wxxuzbt8+4XC6vZe/YscOceuqpplevXpXLJrSXZQ2MaUdLGxirj+Ym4EtgvS0oa0XkfhE50042C4gUkS3ADGCmnXct8J4tIl8ANxpjygBE5G1gMdBfRDJEZJpd1nNAZ+ArEVklIi/6LqGNo9LTCQ0FPz9cedXGOSiK0obp0aMHw4cPB6Bz584MHDiw1hmc58+fz+DBgxkyZAgAkZGROBwOr2lvu+02HnvsMawxUha6rEH98WkaHGPMPGCeR9jdbtvFwJQa8j4IPOgl/OIa0if6YlNzUCE64nDgCA+nLEdFR1EayqNLH2VD7oYmLXNAxADuPOZOn9Kmp6ezcuVKRo0axY8//shzzz3H7NmzSUtL48knnyQ8PJxNmzYhIkyYMIHs7Gwuuugi7rjjDqDqsgaffPIJsbGxleJUgS5rUH/a7ECC1qBCdAD8IyJw5eS0skWKojSEgoICzjvvPJ5++mlCQ0O5/vrr2bp1K6tWraJHjx786U9/Aqy52RYtWsSbb77JokWL+Oijj/jmm28Aa1mDtLQ0CgsLeeihhyrnZVMah0746Ya76Di6RlKmoqMoDcZXj6SpcTqdnHfeeVx66aWce+65AHTr1q0yfvr06Zx++umA5WmccMIJdO3aFbAm71yxYgXjxo2rTL9161a2b99e6eVkZGQwfPhwli5dqssaNAD1dNwwTicSaHs6kV3V01GUdoYxhmnTpjFw4EBmzJhRGZ6ZmVm5/dFHH1UudTBhwgR+++03CgsLcblcfPfddyQnV51wJTU1laysLNLT00lPTycuLo4VK1bQvXt3XdagAain40aV5rXICPV0FKWd8eOPP/L666+TmprK0KFDAXjooYd4++23WbVqFSJCfHw8//73vwEIDw9nxowZjBw5EhHhtNNOY/LkyUDVPp2a0GUN6o8ubeDGllPGEzJiODGPPsq+l14m+5//pP/KFfh16NAEVirKkY8ubdB20aUN2iDG6QQ3TwfQJjZFUZQmREXHjSoDCSIjAbSJTVEUpQlR0XGjap+ONZpFPR1FUZSmQ0XHDc+BBKCioyiK0pSo6LihzWuKoijNi4qOjSkrg7KyStHxCwrCLywMV1YTrdWjKIqiqOhUYFzWYqXiH1AZFtCtG869e1vLJEVR6klOTk7l8gXdu3evXM6gS5cu1T76bAuMHTuWpvjcoz2homNjnE6ASk8HwL97N1xuXzIritK2iYyMZNWqVaxatYrrrruO2267rXLfz09fd20BvQo23kQnoFt39XQU5QihrKyM6dOnk5KSwqmnnkpRURFgza02ceJERowYwfHHH8+GDdVnxs7NzeXss89m8ODBjB49ml9//RWwFoe76qqrGDt2LH369OHZZ5+tzPPGG29wzDHHMHToUK699lrKysp8tjU+Pr5yobhly5YxduxYAL777rtKT27YsGEcPHgQgMcff5yRI0cyePBg7rnnngadn5ZCp8GxMaVePJ0e3SnLyaG8tBS/wMDWMk1R2iV7HnqIkvVNu7RB0MABdP/LXxqUd/Pmzbz99tu8/PLLXHDBBcyZM4fLLruMa665hhdffJGkpCSWLFnCDTfcwIIFC6rkveeeexg2bBgff/wxCxYsYOrUqaxatQqADRs28O2333Lw4EH69+/P9ddfz5YtW3j33Xf58ccfCQgI4IYbbuDNN99k6tSpjTr+J554gueff54xY8ZQUFBAcHAw8+fPZ/PmzSxduhRjDGeeeSbff/89J5xwQqPqai5UdGxq8nQAXFlZBLqtgaEoSvsjISGhcj62ESNGkJ6eTkFBAT/99BNTphxeDqykpKRa3kWLFjFnzhwATj75ZHJycjhw4AAAkydPJigoiKCgIKKjo9m7dy/ffPMNy5cvr5x7raioiOjo6EYfw5gxY5gxY0blDNpxcXHMnz+f+fPnM2zYMMBa1mHz5s0qOm0d4ywFqJxlGqw+HQDXnj0qOopSTxrqkTQXQUFBldsOh4OioiLKy8vp0qVLpdfSFOW6XC6MMVxxxRU8/PDDDSrT39+f8vJyAIqLiyvDZ86cyeTJk5k3bx5jxozhyy+/xBjDXXfdxbXXXtvgY2hJtE/Hxqun093ydJx7tF9HUY5EQkNDSUhI4P333wespRFWr15dLd3xxx/Pm2++CcDChQvp2rUroaGhNZY7btw4PvjgA7KysgCrT+j333/32a74+HiWL18OUOlhgdX/lJqayp133snIkSPZsGEDEyZM4JVXXqGgoACwVjOtqLctoqJTQcWQafc+nYrmtb17WsUkRVGanzfffJNZs2YxZMgQUlJS+OSTT6qluffee1m+fDmDBw9m5syZvPbaa7WWmZyczAMPPMCpp57K4MGDGT9+fJU1ferinnvu4ZZbbiEtLQ2Hw1EZ/vTTTzNo0CAGDx5MQEAAkyZN4tRTT+WSSy7h2GOPJTU1lfPPP79ygEFbxKelDURkIvAM4AD+Y4x5xCM+CJgNjABygAuNMel23F3ANKAMuNkY86Ud/gpwOpBljBnkVlYE8C4QD6QDFxhj8mqzrymWNihavZr0Cy+i50v/ppNbW+jGtJGEnXMO3f/atpoKFKUtoksbtF3azdIGIuIAngcmAcnAxSLi+ZXVNCDPGJMIPAU8audNBi4CUoCJwAt2eQCv2mGezAS+McYkAd/Y+83ACgoIAAAgAElEQVSOt+Y1gIAe3XHu3t0SJiiKohzx+NK8dgywxRizzRhTCrwDeK6vehZQ4W9+AIwTEbHD3zHGlBhjtgNb7PIwxnwP5Hqpz72s14Cz63E8DaZG0YnridNtDXRFURSl4fgiOrGA+1s3ww7zmsYY4wLygUgf83rSzRhT0fi5B+jmLZGIXCMiy0RkWXZ24+dHq1F0esZRmpHBkbDCqqK0BPqstD3a0jVp0wMJjHWmvJ4tY8xLxpg0Y0xaVFRU4+uyRWdB5g9sy99WGR7YsxemsJCyXG9OmaIo7gQHB5OTk9OmXnJHO8YYcnJyCA4Obm1TAN++09kF9HTbj7PDvKXJEBF/IAxrQIEveT3ZKyI9jDGZItIDaJGxfxWi89yaF9mTPYuVl68ELE8HoHTHDvzt5Q4URfFOXFwcGRkZNEXrg9J0BAcHE9dGvjX0RXR+AZJEJAFLMC4CLvFIMxe4AlgMnA8sMMYYEZkLvCUi/wRigCRgaR31VZT1iP2/+vjFZqBCdFwOcJW7KsMDe1qa6czIAPuLX0VRvBMQEEBCQkJrm6G0YepsXrP7aG4CvgTWA+8ZY9aKyP0icqadbBYQKSJbgBnYI86MMWuB94B1wBfAjcaYMgAReRtLpPqLSIaITLPLegQYLyKbgVPs/WanYu41l6NqeECs1QVVqoMJFEVRGo1P0+AYY+YB8zzC7nbbLgameOaz4x4EHvQSfnEN6XOAcb7Y1ZS4ezru+AUH49+tG84dKjqKoiiNpU0PJGhJKkXHyxmxRrCp6CiKojQWFR2bmjwdgMBevSmtx7xJiqIoindUdGw8Rcd9MEFQ3z6UZe+jLD+/NUxTFEU5YlDRsakQnTJbdPJLDgtMYJ8+AJRs3VYtn6IoiuI7Kjo2xunEOBwYEQD2l+yvjAtKTASgdNvWVrFNURTlSEFFx8Y4nRBwuEPHXXQCYmKQoCBKtqjoKIqiNAYVHRvjdGL83USn+LDoiMNBYJ8+lKinoyiK0ihUdGyMs5RyN9HJK6m6hE9Qnz6UqqejKIrSKFR0bKp5Om7NawBBiX1x7t5NeWFhS5umKIpyxKCiY2OcTsodh09HTlFOlfhAezBByZYtLWqXoijKkYSKjo1xOin3t05HVIcosouqzpIbPNBaLLV43foWt01RFOVIQUXHxjidlDms4dJxnePILqwqOgGxMfiFhVG8bl1rmKcoinJEoKJjY5xOyvwFP/Gje8fu1TwdESF4wACK16unoyiK0lBUdCqwPZ0gRxDRHaLJLsyutvphcHIyJRs3Vs5eoCiKotQPFR0bU+qkzA+CHEFEhURRXFZMgbOgSprg5IGY0lJKtm1vJSsVRVHaNyo6NsbpxOUQAh2BdO3QFaBav05wsj2YYL326yiKojQEFR0bS3QMwY5gokOiAar16wTGxyMhIRT/+ltrmKgoitLuUdGxsUSHKp5OVmFWlTTicNBh8GAKV61sDRMVRVHaPSo6NsbppNSvdk8HIGT4MEo2bKT80KGWNlFRFKXdo6JjU9G8FugIpGNARzoHdiazILNaug7DhkF5OUW//toKViqKorRvfBIdEZkoIhtFZIuIzPQSHyQi79rxS0Qk3i3uLjt8o4hMqKtMERknIitEZJWILBKRxMYdom8Yp5NSMQT7BwMQ2ymW3Yd2V0vXYcgQEKFwpTaxKYqi1Jc6RUdEHMDzwCQgGbhYRJI9kk0D8owxicBTwKN23mTgIiAFmAi8ICKOOsr8F3CpMWYo8Bbwt8Ydom8Yp5NSRzmBjkAAYjrGsLuguug4QkMJSkykaOWqljBLURTliMIXT+cYYIsxZpsxphR4BzjLI81ZwGv29gfAOBERO/wdY0yJMWY7sMUur7YyDRBqb4cB1d/8zYBxuSiVcoIdlqcT0ymGXQW7qn0gClYTW9HKlRiXqyVMUxRFOWLwRXRigZ1u+xl2mNc0xhgXkA9E1pK3tjKvBuaJSAZwOfCIN6NE5BoRWSYiy7Kzq3f41xdrIMFhTye2UyxFrqJqSxwAdBw9ivKCAorXrm10vYqiKEcTbXEgwW3AacaYOOC/wD+9JTLGvGSMSTPGpEVFRTW6UuN0UuJXRpAjCLA8HcBrE1vIsccCcOinnxpdr6IoytGEL6KzC+jpth9nh3lNIyL+WM1iObXk9RouIlHAEGPMEjv8XeAPPh1JIzBlZVBWRokcFp3YTpbjtavA81DBPzycoOSBHPppcXObpiiKckThi+j8AiSJSIKIBGINDJjrkWYucIW9fT6wwFidIXOBi+zRbQlAErC0ljLzgDAR6WeXNR5o9mmdK/pmit08nR6degDePR2ATn/4A4WrVun3OoqiKPWgTtGx+2huAr7EEoD3jDFrReR+ETnTTjYLiBSRLcAMYKaddy3wHrAO+AK40RhTVlOZdvh0YI6IrMbq0/lz0x1uDcdozxpd6ldOkL8lOqGBoYQFhfH7wd+95gk59lhwOilcvry5zVMURTli8PclkTFmHjDPI+xut+1iYEoNeR8EHvSlTDv8I+AjX+xqKipEx2XPMl1BQmgC6fnpXvOEjBiBBAdT8N33dDrhhJYwU1EUpd3TFgcStDim1BYdh4fohCWQfiDdax6/4GA6jhnDwQULvA6rVhRFUaqjooObp+NFdPYV7eNA6QGv+TqffDKuzExdwlpRFMVHVHQA4ywFvIsOUGMTW6eTxoKfHwXfLGhuExVFUY4IVHSo2dOJD40HYHu+95VC/SMiCBk+nIPffNPsNiqKohwJqOhQs+jEdo7F38+/RtEB6HTKOEo2bqQ0Pb25zVQURWn3qOgAuI9e8z8sOgF+AfTu3Jut+VtrzBo6aRKIkP/p/5rdTEVRlPaOig41ezoA/SL6sTF3Y415A7p1I2T0KPI//VRHsSmKotSBig7uoiN8/us+1u7Or4wbGDGQzEOZ7C+uPvFnBWFnnIlzxw6KV69udlsVRVHaMyo6VPV0/vP9TiY/u6gyrn9EfwA25G2oMX/nU8cjQUHkz/WcHUhRFEVxR0WHqqJjTECVuIERAwFqbWJzdOpE51NOIf9/n1FeVNR8hiqKorRzVHSoKjqYqjMDhQeH0y2kG+tza593NPziiyg/cIAD86rN7KMoiqLYqOhQde41Ux5QLX5AxADW59QuOh1GjCAoKYm8N9/SAQWKoig1oKKDd0/HVVZeGZ/aNZVt+dtqnA4HQEQIv+Riiteto/i335rVXkVRlPaKig7eRSe7oKQyfmj0UABWZ9U+Oi30jDPxCwkhd/brzWOooihKO0dFh8OzTIt/ICAAZOYXV8andk3FIQ5WZq2stRxHp450ufBCDnz+OaUZGc1mr6IoSntFRYfDng4BgZVhe9xEJyQghP4R/VmVvarOsiKuvBLx8yNn1qwmt1NRFKW9o6LDYdEx/odFx93TARgaNZQ1+9bgLHfWWlZAt2jCzj6b/Dkf4srObnpjFUVR2jEqOrh5Oo7DI9d25hZWSTOs2zCKXEWsy6l77ZzIq6dhXC5y/qPejqIoijsqOliiU+4nOMSady0+MoQdHqIzqvsoBGHx7sV1lhfYuzdhZ59N3ltv4dy1q1lsVhRFaY/4JDoiMlFENorIFhGZ6SU+SETeteOXiEi8W9xddvhGEZlQV5li8aCIbBKR9SJyc+MOsW6M00mZvyAEIAL9u3cmPedQlTThweEMjBzok+gARP3xJvDzI/vZ/2sOkxVFUdoldYqOiDiA54FJQDJwsYgkeySbBuQZYxKBp4BH7bzJwEVACjAReEFEHHWUeSXQExhgjBkIvNOoI/QB43RS5hAw/oQEOEjo2omduYWUlVf9yPMPMX9gdfZqCkoL6iwzoEcPIi6/jPy5cyneWPMUOoqiKEcTvng6xwBbjDHbjDGlWCJwlkeas4DX7O0PgHEiInb4O8aYEmPMdmCLXV5tZV4P3G+MKQcwxmQ1/PB8o0J0xAQQEuRPfGQIzjLD7v1V51H7Q8wfKDNlLN2z1KdyI6dPxy80lL0PPqSzFCiKouCb6MQCO932M+wwr2mMMS4gH4isJW9tZfYFLhSRZSLyuYgkeTNKRK6x0yzLbuQoMeN04nIIxvjTMdBB78iOAPyeU7VfZ2jUUEL8Q/g+43ufynWEhRE9YwaFS5dy4NNPG2WjoijKkUBbHEgQBBQbY9KAl4FXvCUyxrxkjEkzxqRFRUU1qkJLdKx510IC/YnvGgLAdo9+nQBHACfEncC3O7+lrLzMp7K7TDmf4MGD2fvoY5QdqHkaHUVRlKMBX0RnF1YfSwVxdpjXNCLiD4QBObXkra3MDOBDe/sjYLAPNjYK43LichjKy/zpGOSgW+dgQgIdbM2q3ndzSu9TyC3OZUXWCp/KFj8/ut9zN2V5eWQ98WRTm64oitKu8EV0fgGSRCRBRAKxBgZ4rlY2F7jC3j4fWGCsToy5wEX26LYEIAlYWkeZHwMn2dsnApsadmi+Y5xOnH5QVu5PSKA/fn5Cv26d2bCnumdyfOzxBDmC+Pr3r30uv0NKChFXXsn+996j4HvfmuYURVGOROoUHbuP5ibgS2A98J4xZq2I3C8iZ9rJZgGRIrIFmAHMtPOuBd4D1gFfADcaY8pqKtMu6xHgPBH5DXgYuLppDrWWY3Q6cToMZS4HHYMcAAzo3pmNew5WGwAQEhDCmJgxfL3ja8pNubfivBJ1y80EJSWx+69/xZWX16T2K4qitBd86tMxxswzxvQzxvQ1xjxoh91tjJlrbxcbY6YYYxKNMccYY7a55X3QztffGPN5bWXa4fuNMZONManGmGONMbVP7dwUOJ04/QyuMsvTAUt08gqdZB8sqZZ8UsIksgqzWJK5xOcq/IKCiHn8Mcr257Pnnnt1NJuiKEclbXEgQYtTXuqk1M/gdPnRMdDydPp3DwVg/Z6D1dKf1OskOgd25qMtH9WrnuABA4i+9RYOzp9P3uu6/IGiKEcfKjpAubMUlwOcTgchQYc9HYANmdX7dYIcQUxOmMw3v39Dfkl+veqKuOoqOp0yjr2PPU7h8uWNN15RFKUdoaIDlJeWUOYAV5l/pacT3jGQmLBgftvlXVTOTjqb0vJSPt/+udf4mhARYh5+mMDYWDJuvRXn3mb/9lVRFKXNoKLDYU8Hc7hPB2BY73BW7tjvNU9yRDIDIwby9oa36zWgAMDRuTOx//cs5lAhO6+7jrKCQ3VnUhRFOQJQ0cHq03H5gTEBlaPXAIb17MKu/UVkHSiulkdEuDz5crblb+PHXT/Wu87gfv2IfeYZSjZtYtettx5eXkFRFOUIRkWHwzMSUO7h6fQKB2BFDd7OxPiJRHeI5vV1DRsU0On44+hx370cWrSIzL/fjSmvn8ekKIrS3lDRwW0aHBNASOBhT2dQbCiBDj9W7vT+XU2AI4CLB17M4szFrM9Z36C6u5x/Pl1vuon8jz9mz733qfAoinJEo6IDUOHpePTpBPk7GBQbypJtuTVmvaD/BXQO7MwLq15ocPVdb7yByGuuYf9777H3gQf0Gx5FUY5YVHQAXC67ea1qnw7ApJ4upux5kvwD1b/XAQgNDOXKlCtZmLGQX7N/bVD1IkLUbbcSMe0q8t562/p4tMy3CUUVRVHaEyo6AE6X3bxW1dMBOOvAG1zq+Ibff3ijxuyXDryU8KBw/m9lw1cJFRGib7+90uPZdeutlJdUnw1BURSlPaOiA5an4weY6p5OZJj1kWjx1p9qzN4xoCPTB0/n58yf+W7ndw02Q0SInnEb3f5yFwe/+pqd066mLL9+H58qiqK0ZY560TFlZUi5sRZxK6/u6TgOZgLQO+/HWjv5LxpwEX3C+vDI0kcoKWuchxIxdSoxTzxB4erVpF9wISVbtjSqPEVRlLaCio79fYw1kKDq6DUA8tIB6GZy2La25mWqA/wCuGvUXWQUZPDfNf9ttF1hp0+m96v/payggPQLL+LgggWNLlNRFKW1UdFxE50Av0ACHG6nxBjIS6d4wLkA7F06p9ayRvcYzYT4Cbz060tsyWu8dxIyYgQJH7xPYHw8GTfcSNZTT+tHpIqitGtUdNxEJyQgqGrkoWxwFhKccCzrAwcRu2ueJUS1cNcxd9EpoBN/WfQXnOWNF4iAHj3o/eYbhJ17Ljn//je/X3Y5pRkZjS5XURSlNVDRqSI6HapG5v1u/Q+PZ3/i2fQuzyB97c+1lhfZIZK7j72b9bnreenXl5rERr/gYGIeepCYJ5+gZOtWtp99Dvlz5+r3PIqitDtUdCpEx8+PTkEBVSPt/hzCe9PvpMsoNQ4yf5hdZ5mn9D6FM/qcwUu/vsRPu2se9VZfwiZPJuHjj60VSO+4k4zrrseZmdlk5SuKojQ3KjqlluiUOaqPXKsUnS69iIzqwfpOo0jaO4+SkqI6y/3b6L/RJ6wPd35/J5kFTScMgXGx9H7jdbrdNZNDS5ey7fQzyH3rLZ0+R1GUdoGKju3plPk5qn2jQ146dO4BdrObjJxGV/az9us36yw3JCCEp8Y+hbPcya0Lb6XQWdhkNovDQcQVV9Dn07l0GDKYvff/g/QLLqRwxcomq0NRFKU58El0RGSiiGwUkS0iMtNLfJCIvGvHLxGReLe4u+zwjSIyoR5lPisiBQ07LN853KfjxdPZ/zuEx1fuphx/DjulByGrZvlUdnxYPI8c/wgbcjfw5+//jKvc1VRmAxAYF0fPWbOIefwxXFlZ/H7JJez60+3a5KYoSpulTtEREQfwPDAJSAYuFpFkj2TTgDxjTCLwFPConTcZuAhIASYCL4iIo64yRSQNCG/ksfmEcZYC4BJH5aqhleSlQ5felbsOh4NdSZcywLmO9cu/96n8sT3H8tdRf+X7jO954Oemn8xTRAg74wz6fj6PyOuv4+DXX7N10mlkPfEErjzvs2MriqK0Fr54OscAW4wx24wxpcA7wFkeac4CXrO3PwDGiYjY4e8YY0qMMduBLXZ5NZZpC9LjwB2NOzTfqPB0nOJPSJCbp+MqhfyMKp4OQOrkGzlAR0q+edjnOi7ofwHTU6czZ/McHvvlsWYZdebXsSPRt9xC33mf0Xn8eHJmvcLWU8aT/ez/UXbgQJPXpyiK0hB8EZ1YYKfbfoYd5jWNMcYF5AORteStrcybgLnGmFrbiETkGhFZJiLLsrOzfTgM77iLThVPJ38nYKqJTsewCNb0vJShhT+x7TffR6b9cdgfuWzgZbyx/g0eWfpIsw13DoiNJfbxx+gz9xM6Hncc+154gS22+Lhya16iQVEUpSVoUwMJRCQGmALUOV2zMeYlY0yaMSYtKiqqwXVWiE4Jjqp9Onnbrf8eogOQcs6dHKAj++fd73M9IsIdI+9gavJU3trwFvctvq/J+3jcCUpKIu6Zp0n4cA4haWmW+Jx0Mnvuv5/SHTuarV5FUZTa8EV0dgE93fbj7DCvaUTEHwgDcmrJW1P4MCAR2CIi6UCIiDTrbJeHv9PxmHfN7cNQT8IiurKhz1UML1rMmh8+8bkuEeH2tNsrm9puWnATh5yHGmN+nQQnJ9Pzhefp89n/CD3jdPa//wFbJ04i4+ZbOPTzEv3AVFGUFsUX0fkFSBKRBBEJxBoYMNcjzVzgCnv7fGCBsd5mc4GL7NFtCUASsLSmMo0xnxljuhtj4o0x8UChPTih+ahsXguo2qeTlw6OIOjUzWu2wVPuIkO60/nbv1Jaj3VvRISbh9/MPcfew8+7f+aKz69gd8HuxhyBTwT17UvMAw/Q95uviZw2jUNLlrDjyivZNvl0cmfP1iUUFEVpEeoUHbuP5ibgS2A98J4xZq2I3C8iZ9rJZgGRtlcyA5hp510LvAesA74AbjTGlNVUZtMemm8c9nQ8+nTy0iG8N/h5P0XBHTqSPeZeepfvZNk7vjezVXB+v/N5ftzz7CrYxQX/u4AfMn5oiPn1JiA6mug/zSDpu4X0ePhhHJ07s/ehh9l84lh23/UXDi1Zqh+aKorSbMiR0LySlpZmli1b1qC8+z/4gMy//Z2rLxvG38/+JxMHdbciXjweOneHS9+vObMxrHryDJIPLmbPRV/Sa2BavevfcWAHMxbOYGPeRqanTuf6odcT4BdQd8YmpHjdOvLeeZcDn31G+aFD+Mf0IOz0Mwg760yC+vZtUVsURWk5RGS5Mab+L65G0KYGErQGxmV15rvEY9XQvN+99udUQYTYy/9FgYRQ+sE1FBfXPT2OJ71Ce/HGaW9wbtK5vPzby1w277ImWRahPgQnJ9Pj/vtIWvQDMU8+QVBiIjmzZrFt8ulsP38KObNe0ZmtFUVpElR07LnXXBJ4ePRaUR6U5Ff5MLQmorr1ZOeYh0ks28rq/9zQIBuC/YO57w/38eSJT5JZkMkF/7uAWb/NatbRbd7w69CBsMmT6fXSSyQt/JbomXdCeTlZjz/O1lPGs+3cc9n34ouUbNvWonYpinLkcNSLTrk9I4HTz83TqZxdOt6nMoaMv4yfu13MqH0fsvzjOkd718ip8afy0VkfcWLciTy94mmmfDqFX/b80uDyGoN/VBSRV15Jwodz6PvVfKL//Gf8AgLJfvoZtp02ma2nn07Wk09yaOlSXVhOURSfOepFp6zUGnnmIoiOFZ5OPUUHYPi0Z1kTNJRBK+9jzU+fN9ieyA6R/HPsP3l67NMUOgu56suruOO7O9hzaE+Dy2wsgT17EjntKuLffYfEhd/S7a9/xT+yKzn/fZUdU69g07F/IOOWW9k/Zw7OrKxWs1NRlLaPf91JjmxcJcWAPWQ60NPTqbt5rYLAwEB6Xfsee547md7z/x/bQz8iYdCxDbJJRBjXexxjYsfwyppXeGXNK3yz4xsuHHAhV6deTURwRIPKbQoCuncn4vLLiLj8MsoKCji0eDGHvv+egu++5+CXXwIQ1L8/HUePImTUKELS0nCEhraavYqitC1UdEqLcfkBBNKx4judvN8hpCsEda5XWaER3Si44mMO/XcioR9cyDa/D+mT3PCBIcH+wdww9AbOTjybf63+F2+uf5M5m+ZwWfJlTE2eSlhQWIPLbgocnToROn48oePHY4yhZONGCr77nkOLF5P3zrvkvjYb/PwITk4mZNQxdBw1ipARI/Dr2LFV7VYUpfU46odMb33gbg68+z4XXnohG++8BxGB2WdDyQGYvqBBZWZsXkXwm2fhoIzcc96i75ATGlSOJ9vyt/HCqhf4Mv1LOvh34Lyk87g8+XJiOsU0SflNSXlJCUWrVlO4ZAmHli6haPWv1oe4DgdB/fsRMnQYHYZZfwGxMdZ5VxSlRWmNIdNHvehsuvtO8ufO5apLpvLbHXdZgc8MhdjhcP4rDbZp55Y1ON44h1BzkC0nv8jQE89ucFmebMrbxGtrX2PetnkYDBPiJ3DpwEtJ7ZraZl/e5YWFFK5cSeEvv1C0ajVFv/6KKbQWtvOPiqLD0KGWCA0dQvCAAfiFhLSyxYpy5NMaonPUN6+VlRbjckCgf5Ad4LJmmB50bqPK7Zk4iKzp89k36ywGLfh//JK5lrQL7kJqmOGgPvQL78eDxz3IH4f9kdfXvc4Hmz5g3vZ59A/vz5R+U5jcZzKdAjs1up6mxC8khE5jxtBpzBjA+j6qZPNmCleupGjlKopWruTgV1/Zif0I6tuH4OQUglNSCB6UokKkKEcIR72ns2bG9eQtWsjMy27ix5tvtPpznhkMZ/4fDJ/aaNsKDuSx8V+XMqLoR5aGTiB5+kt06tyl0eVWqaO0gHnb5/H+pvfZkLuBDv4dmJQwickJk0nrnoaftI9Biq7sbIp++43iNWspXruWonVrKcveZ0X6+RHYJ4EOKSkEDRhIUL8kgvv1w9G1a5v17hSlraOeTitQXlqCyw86+AdbAftrnl26IXQKDWfY7Z+yZPZMRqa/TOY/R5F5xoskDT+pScoH6BTYiQv6X8CUflNYs28N7296ny+2f8GHmz8kOiSa0xJOY3KfyfQP79+mX9D+UVF0PvlkOp98cmWYc28WxWvXVv4d+mkx+Z8cnm/WER5OUL9+9p8lREGJiTpYQVHaKCo6paW4HG6iUzFc2ofZCHzFz+Fg1P97nLWLTybiyz+S8Mm5/PjL5Yy4/EGCQ+o3Qq42RITUqFRSo1K5a9RdLNy5kM+2fcYb697g1bWvEh8az8m9TubkXieT2jW1XXhAAd2iCegWTeeTD4u0KzeXkk2brL/NmynetIn9c+ZU9hEBBPTsSWCfBIIS+hCYkEBQnwQCExJwREa2aeFVlCMdFR2nJTohAW6i4+cPoZ6LozaelGMnkT9wCatfu4Exma+R+fjnZB33DwaffGGTvwgrmtgmJUxif/F+5v8+n69+/4rZa2fzyppX6NqhK2N7juWknicxsvtIOvh3aNL6mxP/iAj8R4+m4+jRlWGmvBznrl2VYlS8aROl27ZT+PMSjNvSE36hoQQmxFeKkSVMCQTExeEXHNwKR6MoRxdHfZ/O8svOYfeuDXw47Qn+e9lk+OAq2LUCblnVxFZWZc2Pn9Hp6zuINxmsDkoj9IwHGvwxaX04UHqAHzJ+YMGOBSzatYhCVyEBfgEMjx7OsTHHcmzMsQyIGNAuvCBfMOXlOHdnUrp9O6Xbt1OyfRul29Mp3bYNl8fsCf7R0QT06klgz14E9IwjsGcvAnvGEdCrF47wcPWQlCMOHTLdQBojOssuPJ2dOVv5ZvpzPHfhOHh5nPVR6NSPm9jK6jhLi1n1/iP02/wSYRxieeeTiTz9XuL7D2n2ugFKy0r5Zc8v/LT7JxZnLmZz3mYAwoPCGd1jNGnd0xgePZw+XfocMSLkTllBgSVA6emU7tyBc8dOSjN24tyxs5og+XXsaDXZ9exJQM+eBMTEEBDTg4AePQiIicEvNFRFSWl36ECCVsA4nbgcQqcgu3kpLx0Gnt4idQcEBjPy0nvJz72RH9//B8N2v0PwWyeysvPxhI67nb7DTmzW+gMdgYyJHcOYWGsYc3ZhNj9n/sxPu3/i58yf+ZPg3ZwAABuDSURBVDzdmkMuLCiMYdHDGBE9guHdhjMwcmCLr/nTHDg6daJD6iA6pA6qFldeXIwzI4PSHTtxZuy0/u/cScnWrRR89x2mtLRKer+QEPxjehDQI6ZSiCpEyb9HDAHdopGA9n/OFKWxHPWig9OJyw86BQZDyUEo3NdkI9d8JSwiijHXPkve3j/z69wnSM54j9BPzmTtvEEUDZvOkFMuJiAwqNntiAqJ4oy+Z3BG3zMwxrDj4A5W7F3BiqwVrNi7goU7FwIQ5AhiQMQABnUdREpkCildU4gPjT+ivCG/4GCCEhMJSqy+WropL6csNxdnZibO3Zk4d+/Gmbkbl71fvHYtZbm5HgX64R8ZiX+3bvhHR+MfHUVA5fbhMEeXLuoxKUc0R33z2tKJJ7DRfx95Mz7k5mQXvDgGprwKKec0rZH1IH9/Lus/e474LbPpbrLJIYwtMWfR/aRr6J2U2mp27Svax4q9K1iVvYq1+9ayPnc9RS5r4bpOAZ1IjkwmpWsKAyMG0i+8H71De+Pvd3T+rikvKsKZucdNjHbj3LsXV1Y2rr17cWVlUbZ/f7V8EhhoC1A0/t2iCagQpaiuOCIjLeGKjMQREYE4HF5qVhTf0T6dBtIY0Vky7g+s67Qf8+f/cVXkOnj3UrhmIcQMa1IbG0K5y/X/2zvzGDuS+75/fn2++82bg9whh+RySa4WlLWSKGolxfIGtqDTRjZGhGgVQBYSJXJiCXGABLDWDmBBOWUgMhLYsiRLCmJB1hFHjhdG7FWsFSAjkbhLKdIetLjkiiNylsdcb968s8/KH93z9s3JWXLmzfHqAzS6+lfVVfWb7nnf7qrqKp79zjeInv6vPNj6HpbEXLBO0zj5dzj1Cx+gcmBiR+sXxiFXald4bvY5np97nudmn+Ni9WJ38TnHcDgxdIJTlVPcX7mf+yv3c6pyitHs6I7We7cQex7hzAzh9HSy3bpFMD29TJiC6ellQ8G7iGBWKokAjY5gjYy+HB4ewRodwRwZxRodwRoeRhyn/w5qdj27VnRE5F3AfwZM4PNKqf+4It4F/gh4AzAHvE8pNZnGPQZ8CIiAf66UemKjPEXky8BZIACeAn5VKbXhKmF3JToPP8QzIy3Kv/kX/P3wcXjiN+E3JiFbuaP8tovZl65w5VufZ3TyzzkeTxIp4YL7Wuonfon7fvbvcc/EfTtdRSAZnHCldoUXqi9wqXqJF6ov8EL1BWbaM900w5lhjpePc2/pXo6Xj3e3Q/lDmIZ+el9J1GgQzswQzc0Rzs0Tzs0Szc4Rzs2tCM+tLVCAUS5jVSqYlQrm0FCyrwyttg0ldrNc3pIpmzS7m10pOiJiAi8AbwemgKeB9yulLvSk+TXgQaXUPxWRR4FfVkq9T0ROA18BHgIOAX8F3J+etmaeIvIeYGkVtD8GvqOU+oON6nhXzWt/6w2cHw84+tt/yS9N/S786Gvw2NU7yqsfKKV48fmnmf7uVzh64y+ZiK8D8KJ5gtlDP0/5tb/Iydc9jGXtrmat+c58V4QuVS8xuTjJldoVFryXm5gcw+Fo6WhXkI6VjjFRnOBI8Qij2dF91We0XcStFuH8PNHsbCJEsy8LU7RQJVpYIKwuEFWrRPPzqwZEdDEMzFIpFafKyyK1JE7lMkaphFkqY5ZLmKUSRrmMkc/rPqk9xG4dvfYQcFkp9RMAEfkq8AhwoSfNI8DH0/CfAL8nyZ33CPBVpZQHXBGRy2l+rJenUup/LWUqIk8B29qGJGFEaBrJqqHVyVe0cNtOICKc/JmHOPkzD4H6FFcv/oCpp/4n5Wvf4uzVL2Be+zzzf17kSv71RMfeyuHXv5NDJx7c8afW4cwwbxp/E28af9Mye7VTZXJxkslaIkJXale4VL3Ek1efJFJRN51ruhwuHO6K0ERhgoniBBOFCQ4XD++pj1u3EyOXw8nlYOL2/zZKKVS7TVStEi4sEC2JUbVKtFAlrCYiFVUXCKam6Dz7LFG1uvHy5KaJWSwmolQuJ8JVKmGUU4EqlTDLpUSwykMvC1apjJHPacEaADYjOoeBaz3HU8Cb1kujlApFpAaMpPbvrTh36VP/DfMUERv4APDrm6jjHWOEMaFhJauGVidh7IHtLG5rEeHoA2/g6ANvAP4NC7M3+cm5x1GXvsXhhfPcc+E7cOHfM8sQVwpnCCbezNgDb+X4q9+IZe+ONv5KpkIlU+H1B5b3oQVRwPXmdabqU8nWmOJa/RpT9SnO3zxPK1zejDScGeZg7iDj+XHuyd/T3S9tY9kx3XS3AhFBcjmMXA778OZm4FBKETdbxIs1olqNqLZItFgjXlxMw+lxN7xIcO1akrZehyhaP3PTxCgUMAsFjGIRo5DHLBRXhc1iAaOQbGYxjc8XunY9wGJ3s7vaYJbzaZKmtb9eK1JEPgx8GODo0aN3XIgZxYSGSd4xkhmm73/XHee10wyN3sOZX0z+LCqO+emLz3P9h9/EufZ/OL74A0Z//CT8GFp/6nLJvZ/ayOuwjj7E+Omf5dCR4zv+NtSLbdocKx3jWGn1m6dSigVvoStCU40pbjRvcLN5k6v1q5y7eY5m0Fx2jikmB3IHGM+PczB/kIO5g4xmRzmQO8BodpSx7BhjuTHytp4odCNEBLOQxyzksQ+9ssUDE8FqEtdqiSCtIVhxo0HcqBPVG8SNBsH0LeIXXySu14kaDQjD29cxl+sKl5mKU1escnmMfA4jn8fIrdivYZNd1ky9H9jMX/Ql4EjP8URqWyvNlIhYQJlkQMFG566bp4j8NjAG/Op6lVJKfQ74HCR9OpvwY02MKCYUg2I4B5HX9290tgsxDI6dek06xPpfglLcunqRa8/9NcHkOSrzz3Dm+ldwbnwJzsE8JV5yT9AaPo1z+LUM33eGQycf7Mv3Qa8UEem+IT049uCaaep+nZvNm10xWtpuNG/wzMwzzLRm8OPV/RlZK7tMiEazo4zlxrqiNJIZoZKpMOQODexw8DslEazkTWazb1a9KKVQntcVoLjRSMKpQPWKVdSoE9fT+Ead4MYN4nqduNUibrVgk6N2xXU3JU5r23IY2SxGNotkcxi5NDzgQrYZ758GTonIcRJheBT4ByvSPA58EPgu8F7gSaWUEpHHgT8WkU+RDCQ4RTIiTdbLU0T+MfBO4G1Kqfgu/dsQpRRmpAhNk2I71bxd3qdzx4hw8NgDHDz2APBPAPA6LS5dOMfCpe/BzWcZWrzI/de/jnvjy3AePGVzyTxKtXCCaPgkmUOnOXD8Ndxz72nMXdI8tx5Fp0jRKXKqcmrNeKUUi/4is+1ZZtozzLRmuvsl24W5C8y0Z7rfIvUiCGW3zHBmmEqmwnBmeFm4kqkkAuVWGM4OU3bKunnvLhERJJPByGSwxsbuOB8Vx6hOJ3nrajYTIWo2iZpNVKu1bN8bHzdbxK1m0mR480ZynKbZsNlwJbbdFSMjm0VyWYzsywJl5LJItse2dJxZEZeKmH3o0J5a4PC2opP20XwUeIJkePMXlVLPi8gngPNKqceBLwBfSgcKzJOICGm6r5MMOgiBjyiV9A6vlWda5GeAnwLfTTsVv6GU+sSWedxLFGEoCMUk20i7mCrHt6Wo3YibySXr+vSs7eP5Hpcu/oiFKz8guvEsxerfcKz+fQ4ufhMmgf8LgTJ5yRxnLnOMVvkkxuhJiuMnGTv6KsbG78XYA23qIololN0yJ4ZObJi2GTS7ojTfmWe+M0+1U+2G5zvzvLjwIk93nqbm1VCsfoo2xKDsJOUNuUPdsktOadlxd0vTFuyC7lzfYsQwun1Z3IV4LaGUQvn+aoFqNok7bVS7TdxqE7fbxO3W6uM0HNVqBDdvJMedDnE7Ofd2HPncZyk8/PBd+9EvBvrj0Ljd5uLrz/BHb7mHf/eBt2N853fgX0+Dtbuf4neC5mKVqcvPsvDTZwmnL5KtvchoZ5Lx6Aa2vPyU5ymbW+ZBFjKHaeePoirHsEdPUBo/ydjECcpDw/v6RzSMQxa8ha4oVTtV5jpz3eOaV6Pm11j0FlnwFqh5tVWDInoxxeyKU69gFZ0iBbvQfaPrDfceO6a+l/cy3beydipSrVS0usLVIvfGN2IfOHBH+e/WIdP7lqWhn7FpYdSuQnlCC8465EsVXnXmYTiz/IkqDnxuTF1i7toLNG5eJp77CU79KkOdl7iv+QyFmXbyRVZKXWWZM8eouwfxcuPEpQnMoQnckSOUDt7LyPhx8oWtW9iu31iGxWh29BXNuhBEATW/lgjS0rbG8YK3wHRrmkvVS9T9Oo2gseZbVS+u6a4rSCvDOTtH3s6Tt/Lk7Xz3OGfldNPgDrHsrWyfoEUHiE07GS69hauFDgqG7TB+/NWMH3/1qjgVx1TnbqWCdAlv/hrUpnCa1yl4tzjUvsjI3OKq8+ZVkQVzmIY9gpcZJcodRIoHcIfGyQ4fpjByiNLYEQqlyq4acXen2Kb9ioUKIFYxzaBJw2+w6C/SCBrU/XpXkOp+fVVcw29wvXG9e+xF3u0LIhlgkbNSUeoRpLzVE+6NW0O4slaWrJ0la2X3xSzlmjtDiw4QWzZUL8KJt+1wjfYXYhhUxsapjI0Day/TUG/Umb8xSf3WJJ25q0QLUxj1G9jtaXLeLAdq1xheqOLI6qGyHWUzLxVq1jAte4TArRDnRjByI1jFUdzyQXJDByiN3ENp5CButgT7qGnPEKP7ljLO+B3l4Ud+V6SaQZNm0KQVtJJw2BPujQub3X6uyWAysYetNQdcrIdlWIkIpWKWtbJkrEzXtuFmZ8maa9jScMbM7Osm3L3OYItOOubfME2o39g3w6X3EsVCkeKp18AGs2fHUcx8dYb5W1epz14nWLhBVL+JNKax29NkOrOM+FMU2s9TrtaX9TH10lE2NSnRMMs0rSE8e4ggM4zKDCHZClZ+CLswQrY4TK48QmFojOLQKHZm/36745gOI9kRRrIjd51XFEe0wtYq4Vo6boftZVsn7Kyy1f06063pVfZXgiBkrAyu6eKabjecMTO4VmLLWtkN45fCWTO7ypYxM8vOc01Xi9wrYLBFx0/edBwzbRfXorMrMUyD4dGDDI8evG1aFccsLs5Tm71JY/4mrYVpvMUZVHMGac1jdeZx/SqZcIGR5nXKjUUKrN+RD8ngiLoUaBoF2mYRzyri2yVCp4xyS5AdQrJDWNkyTq6EUxgiWxgiV6xQKA2RzZeQAegTMQ2z++a1lcQqxou8l0UoaK8SpXbYphW2ukLWCTt0og5e5OGFHu2ojRd6eJHHfDCPF3rL4jtRhyDecF7hDVlLsFzDxTEdHNPBNVeEjeX2VfGmg2OsH++aLrZhd+17aV7CgRadyO8AkDHTJ2MtOnseMQxKQ6OUhkaB1SuCroWKAhqL8zSqszRqs3QW5/Aa84TNeeLWAqq9gOnVsIMablAn789yoHOFgmpQvI1gAcRKaEiGtuRoSx7PzOGbeXyrQGQViJwCyimi3CKSKWJkSljZInamiJMrkskVcfMlcvkSuUJpV36wu50YYnSbzraTKI7wolSMekSpE6bi1BPuRJ2Xwyv36fl+7ONHPu2wTc2r4Uc+XuThRz5+7HfzjO/yc8RPv+3T/NzEz23RX2H7GWjR8TrJD0bOSJ9w9uuHoZoNEdOmWDlIsXL7N6mVqCik3ajSrFVpNaq0GzW85gJBq0bUWiTuLKI6i+DVMYMGZtDADpu4YYOCd4usapFXbQqy+SYkX5m0JUtHXDzJ4htZAjPZQjNHZOWIrSyxnQcnB3YBcXMYbgHTLWBmCtiZPI6bw8nmcLIFMtkCmWyeTDa3LwZn3AmmYZIzcuTs/o4UC+MwEaI1RGmZLXpZqII46IaPl/fWt4UDLTq+l8zPlTc8sHOQv/sPxTSDhZgWufIYufLd3TtxFNFp1WjVF+jUF/Bai/itOkGnTtBuEnXqxF4D5TfBb0LQwgiamGEbK2phRW2y4SxO3CGj2mSVR442lrzyp+i2cvDExcMhMFx8cQkNl8DIEJkukZklMl2UmSG2sigrg7KziJUFJ4vYOQwni2HnMN0sppPDcrPYjoudyWG7WVw3i5PJ4mby2LY90H0ilmFhGVbfxW6nGHDRSd50CtJOmtYG+MbX7CyGaZIrDpMrDm9dpkrhex06rTpeaxGv1cBvLRJ0GoSdFqHfIvJaRH6L2G+h/A4qaEPYQsIOErYxwg5m5GFGHazIIxvUsZWHqzwcfFzlk8G7I3FbIlKCh4MvNgE2gTiEYhOKQ2g4RN3NJTIcYsMhNl2U6aAsF2W6YLpgZRDLBTsJm7aLYWVSAXQx082yXWzHxXLSsJvBdlwcJ4PjuHtiRo29zECLTuAlTRoF1YTK6u9MNJo9jQhOJnmjYPjOvljfDEopfN/HazfwvSZ+u0XQaRJ4TSKvReh1CP02UdAh9juooEMcdIhDD4I2KvIh7CChh0Q+EnmYkYfEPmbsY8UeVlgno+awlY+tAhx8HBVgE+ASYMjWzawSKJMAi0AsQmwCLCKxCMUmEotIHCLDIhKb2Eg3sYlNh1hslGmjDBtlOmDaKMNJPjo3HDAdxHIQ004E0nIwLQfDcjAsOw27GLaNZTmJUFo2lu1g2S6m7WA5Lk66t0xrzzWHDrToLPXpFFRdfxiq0dwhIoLjujiuS7KMVn9RcUwQ+vidNoHXJvDbXaEL/TaR1yEMPeLAIwx84jQcBz4q8lChjwp94shHQj8RwchH4gCiAIl9JAow48RmxAGmCjCiACdsYank2CLEUiFLUmWrEJtw3SH8W8Uzf/sLPPjz793WMraSgRadVqtFDsiyf5Y00GgGDTEMbCeD7WSAyk5XZxVxFBEEHqHvEQYeoe+nb34+YeARBR2iMCAKfOLQJwqDRBjDgDgKEkEMfVQcQOgTRwFEISpKxHDiyB5aeJIBF512q0kOsEVp0dFoNNuCYZq4Zg43MxgDBW7H3moM3GI66bThjsRadDQajaYPDLToeJ1UdAwFQ3e+5LVGo9FoNocWHUDccvIRnUaj0Wi2lYEWndBLpsExivfscE00Go1mMBhs0fGTtUSsocM7XBONRqMZDAZadKL041B3eG/NXaTRaDR7lYEWHek0AMiOndzhmmg0Gs1gsCnREZF3ichFEbksIh9bI94Vka+l8edE5N6euMdS+0UReeft8hSR42kel9M8nbtzcQO/vCaBCfkDp7arCI1Go9H0cFvRERET+H3g3cBp4P0icnpFsg8BVaXUSeB3gU+m554GHgVeDbwL+LSImLfJ85PA76Z5VdO8twUjaBGa4I7dt11FaDQajaaHzbzpPARcVkr9RCnlA18FHlmR5hHgv6XhPwHeJslc5Y8AX1VKeUqpK8DlNL8180zP+YU0D9I8/+6du7cxRuARGSDFQ9tVhEaj0Wh62IzoHAau9RxPpbY10yilQqBGMvPfeueuZx8BFtI81isLABH5sIicF5HzMzMzm3BjDe49wpVXZWGPzdKq0Wg0e5U9O/eaUupzwOcAzp49e0fzmr/vP3xjS+uk0Wg0mo3ZzCP+S8CRnuOJ1LZmGhGxgDIwt8G569nngKE0j/XK0mg0Gs0eZTOi8zRwKh1V5pAMDHh8RZrHgQ+m4fcCTyqlVGp/NB3ddhw4BTy1Xp7pOd9O8yDN88/u3D2NRqPR7CZu27ymlApF5KPAE4AJfFEp9byIfAI4r5R6HPgC8CURuQzMk4gIabqvAxeAEPiIUioCWCvPtMjfAL4qIv8W+H9p3hqNRqPZB0jycrG3OXv2rDp//vxOV0Oj0Wj2FCLyfaXU2X6WqYdtaTQajaZvaNHRaDQaTd/QoqPRaDSavqFFR6PRaDR9Y18MJBCRGeCnd3j6KDC7hdXZrQyCn4PgI2g/9xM77eMxpdRYPwvcF6JzN4jI+X6P3tgJBsHPQfARtJ/7iUHwcSW6eU2j0Wg0fUOLjkaj0Wj6hhaddNLQAWAQ/BwEH0H7uZ8YBB+XMfB9OhqNRqPpH/pNR6PRaDR9Q4uORqPRaPrGQIuOiLxLRC6KyGUR+dhO12cziMikiDwrIj8UkfOpbVhE/reIXEr3ldQuIvJfUv+eEZEzPfl8ME1/SUQ+2GN/Q5r/5fRc6ZNfXxSRaRF5rse27X6tV0Yfffy4iLyUXs8fish7euIeS+t7UUTe2WNf875Nlwo5l9q/li4bQrq0yNdS+zkRuXe7fEzLOyIi3xaRCyLyvIj8emrfN9dzAx/33fXccpRSA7mRLKnwInAf4AA/Ak7vdL02Ue9JYHSF7XeAj6XhjwGfTMPvAf4CEODNwLnUPgz8JN1X0nAljXsqTSvpue/uk18PA2eA5/rp13pl9NHHjwP/ao20p9N70gWOp/equdF9C3wdeDQNfwb4Z2n414DPpOFHga9t87UcB86k4SLwQurPvrmeG/i4767nlv/tdroCO+Y4vAV4ouf4MeCxna7XJuo9yWrRuQiMp+Fx4GIa/izw/pXpgPcDn+2xfza1jQM/7rEvS9cH3+5l+Q/ytvu1Xhl99HG9H6ll9yPJ2lNvWe++TX98ZwFr5f29dG4attJ00sfr+mfA2/fj9VzDx31/Pe92G+TmtcPAtZ7jqdS221HAN0Xk+yLy4dR2UCl1Iw3fBA6m4fV83Mg+tYZ9p+iHX+uV0U8+mjYrfbGnOeiV+jgCLCilwhX2ZXml8bU0/baTNv28HjjHPr2eK3yEfXw9t4JBFp29yluVUmeAdwMfEZGHeyNV8viz78bB98OvHfrb/QFwAngdcAP4T30uf9sQkQLwP4B/oZRa7I3bL9dzDR/37fXcKgZZdF4CjvQcT6S2XY1S6qV0Pw38KfAQcEtExgHS/XSafD0fN7JPrGHfKfrh13pl9AWl1C2lVKSUioE/JLme8Mp9nAOGRMRaYV+WVxpfTtNvGyJik/wYf1kp9Y3UvK+u51o+7tfruZUMsug8DZxKR4g4JB1yj+9wnTZERPIiUlwKA+8AniOp99LIng+StC+T2n8lHR30ZqCWNj08AbxDRCrp6/87SNqLbwCLIvLmdDTQr/TktRP0w6/1yugLSz+QKb9Mcj2X6vVoOlLpOHCKpPN8zfs2far/NvDe9PyVf68lH98LPJmm3y6fBPgC8DdKqU/1RO2b67mej/vxem45O92ptJMbyaiZF0hGj/zWTtdnE/W9j2R0y4+A55fqTNKe+y3gEvBXwHBqF+D3U/+eBc725PWPgMvp9g977GdJ/lFeBH6PPnVQAl8haY4ISNqvP9QPv9Yro48+fin14RmSH5PxnvS/ldb3Ij2jCNe7b9P746nU9/8OuKk9kx5fTuPv2+Zr+VaSZq1ngB+m23v20/XcwMd9dz23etPT4Gg0Go2mbwxy85pGo9Fo+owWHY1Go9H0DS06Go1Go+kbWnQ0Go1G0ze06Gg0Go2mb2jR0Wg0Gk3f0KKj0Wg0mr7x/wG8awuZftuSeQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Some code may derive from: https://github.com/harvardnlp/annotated-transformer\n",
    "# MIT License, Copyright (c) 2018 Alexander Rush\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "# Three settings of the lrate hyperparameters.\n",
    "opts = [NoamOpt(512, 1, 4000, None), \n",
    "        NoamOpt(512, 1, 8000, None),\n",
    "        NoamOpt(256, 1, 4000, None),\n",
    "        get_std_opt(model_trainer)]\n",
    "plt.plot(\n",
    "    np.arange(1, train_iters_per_epoch * max_epoch),\n",
    "    [[opt.rate(i) for opt in opts] for i in range(1, train_iters_per_epoch * max_epoch)]\n",
    ")\n",
    "plt.title(\"Learning rate warmup and decay through the 100k time steps.\")\n",
    "plt.legend([\"512:4000\", \"512:8000\", \"256:4000\", \"The one I use\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize results on some custom data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrices are symmetric, so on each border is a sentence dotted with annother one in similarity to get something that is almost like covariance of each sentences to each other. We should observe 2x2 activated blocks along the diagonal. The loss function is a binary cross-entropy on this sentence-to-sentence similarity grid we see. I seem to have invented a new similarity loss function but it probably already exists...\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUYAAAD8CAYAAAAL8Q1XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XuUXWWZ5/Hvj8oNAuRC0SGEQEDSAZTuyMSggjbIRWAh2D2IwbENDk5sgb7oONPY9BIXLbPAXt04vUQkrREQuTi2SHqMQEQRHUUIdAzXkHBPgCQkXAK5UVXP/HF2wcnJOVVPnXNSVefw+2TtVfvs/Zy931NVeWrv/e53P4oIzMzsLbsMdQPMzIYbJ0YzswpOjGZmFZwYzcwqODGamVVwYjQzq+DEaGZDTtICSWslPVhjvST9i6SVkpZJOqJs3VxJK4ppbjPa48RoZsPB1cBJfaw/GZheTPOAKwEkTQQuAo4EZgMXSZrQaGOcGM1syEXEXcCGPkJOB66NkruB8ZImAx8GFkfEhoh4CVhM3wk2ZUSjG9gZOid2xLSpI1Oxm6Mnvd3V28anYze/Pjodi3JhI8e8kd5k16bc5wcg/y0gBvITH8CgqBG7dqVjO57P/z3eOjEfO2JTOpSpk9em4p56vTO9zdHP5r8HWyaNSsdqIIPTkr8Lb7y8ge5Nryd/c6v78LFjY/2G7lTsfcu2PgRsKVs0PyLmD2B3U4Bny16vKpbVWt6QYZkYp00dyT23TU3FPrIt/7/hwmdOT8cu+93B6dieUbnf3H1n5P4zAqy7b1I6tmNL/vd7S2c+i3ZszW+38/D8Z9v9kj3SsY+flf8Dtdd9+ST69b+/IhV3zpL8Jat3fH59Onb5F/ZPxw7k57tL8mf29Px/Tm+zlvUburnnttzn6Ji8YktEzGp4p4PEp9JmVpcAepL/mmA1UH60tF+xrNbyhjSUGCWdJGl50VN0QZX1oyXdVKz/naRpjezPzIaPIHgjulNTEywEPlX0Tr8XeCUingduA06UNKHodDmxWNaQuk+lJXUAVwAnUDqvv1fSwoh4uCzsHOCliDhY0hzgMuDjjTTYzIaPJh0NIukG4BigU9IqSj3NIwEi4lvAIuAUYCWwCfh0sW6DpH8A7i02dXFE9NWJk9LINcbZwMqIeAJA0o2Ueo7KE+PpwFeK+R8C35Ck8LPOzFpeEHQ36b9yRJzVz/oAzquxbgGwoCkNKTRyKp3pDXozJiK6gFeAvaptTNI8SUskLVm3vimH3ma2k/UQqanVDJvOl4iYHxGzImLW3nt1DHVzzKwfAXQTqanVNJIYM71Bb8ZIGgGMA/L3NJjZsOYjxh3dC0yXdKCkUcAcSj1H5RYCvTeCnQH83NcXzdpDAG9EpKZWU3fnS0R0STqfUtd4B7AgIh6SdDGwJCIWAt8BvidpJaXhPnOa0WgzG3rRoqfJGQ2NfImIRZS60cuXfblsfgvwsYFud3P0pEe0HDpqt/R2lz27Xzp2IMOwdl2TO/DeMCXf1o4ZG9Oxm9fltzvi1fz1266980MYOwbwDdvwzl3TsXTnbwfZcHi+DT96KTcIo3PP19Pb3Dgr//sVk7amY3dZkf9+9YxMfg8aGgxYCOhuz7w4PIcEmtnwVxr50p6cGM2sTqK7KYeew48To5nVpdT54sRoZvam0n2MToxmZtvp8RGjmdlbfMRoZlYhEN3DZ1RxUzkxmlndfCptZlYmENuiPR/44sRoZnUp3eDtU+lBs3rb+HThqoEM81t57HfTsde+mq8Od/1zR6biRnXknzO58vaD0rET1+XHZR1xzrJ07K+ezrfhoHEvpmOXjt0nHfv+WcvTsc9c9ofp2KdnTUzF7Tl6S/9Bhe5b7knHTpz4vnTsi0fmqw8eM/ORVNxPrtuc3mZf3PliZlYmQnSHjxjNzLbT4yNGM7O3lDpf2jOFtOdxsJntdL2dL5mpP4lSzJdLWlpMj0l6uWxdd9m6yodl16U9072ZDYruJtzHmCnFHBGfL4v/S+DdZZvYHBEzG25IGR8xmlldeke+ZKZ+vFmKOSK2Ab2lmGs5C7ihSR+jKidGM6tbT+ySmoDO3vLIxTSvbDOZUswASDoAOBD4edniMcU275b00WZ8rrpPpSVNBa4FJlG63DA/Iv53RcwxwC3Ak8WiH0XExfXu08yGj9JDJNLHVi9GRK6eRN/mAD+MiPKbgg+IiNWSDgJ+LumBiHi8kZ00co2xC/jvEXG/pD2A+yQtLr8uUPhVRJzawH7MbBgKxBvNGRKYKcXcaw5w3nbtiFhdfH1C0p2Urj82lBjrPpWOiOcj4v5ifiPwCDUOf82s/URAd+ySmvqRKcWMpEOACcBvy5ZNkDS6mO8EjgIqD84GrCm90pKmUcrSv6uy+n2Sfg88B3wxIh6qsY15wDyAjgkTWPa7g3P7HkCVsoEM8/vUnvkhbj9ckxvq99Czk9Pb7JmWr9C31yP5v2+/fiY/zO+QSWvTsU+9ulc6dubHHkzH3v30tHTs+PH5o5flt+SGD77/zP9Ib/PRj8xOx+66IT88dOL9+c+1ZsYeqbg3eppxpKem3OCdLMUMpYR5Y0Vt+kOBqyT1UDrQu7TKWeuANZwYJe0O/BvwNxHxasXq+ymd/78m6RTgx8D0atuJiPnAfIDRU6e2aVFGs/YR0LQhgf2VYi5ef6XK+34DHN6URpRp6FNJGkkpKX4/In5UuT4iXo2I14r5RcDI4nDXzNpAk27XGXYa6ZUW8B3gkYj45xox+wBrIiIkzaaUiNfXu08zGz4C+UG1VRwF/DnwgKSlxbK/A/YHiIhvAWcAn5PUBWwG5lRcHzCzFlUqn9qeg+fq/lQR8Wvo+8prRHwD+Ea9+zCz4Ux+HqOZWbmA3lEtbceJ0czq5iNGM7MyEfIRo5lZuVLni6sEmpmVcc2XwSXoGZW7q2fXNfkfTLaaH+SH+QEsnH5rKu7c3d6b3uZTZx+Qju0Zk/8xrl86Lh37wlH5SnKz/+DpdOytKw9Nx454cPd07G5r88MoXz4k9z07aNf80NDHtvXk9z99VDqWAdzg9ugzuQqMW7aNzG+0hlLni68xmpltpxVHtWQ4MZpZXTzyxcysikyhq1bkxGhmdYmAN3qcGM3M3lQ6lXZiNDPbjke+mJmV8e06ZmY78Km0mdkOmlHzZTgalolx5Jg32HdGrhDThim7pbc7qiM/mmUghauyI1q+OeXu9DZPGvWOdOzyz+a/ByM35IdR/N30Rf0HFd47Zl069raf5ssLb90rP5rk6dPy/0k/8p4lqbgZY55Pb/PO1a+lYzv2n5CO3fCu/M9s/ITXU3FrR+T/L9RS6pVuz7HS7XkcbGY7Xe8N3pmpP5JOkrRc0kpJF1RZf7akdZKWFtNnytbNlbSimOY247MNyyNGM2sNzTiVltQBXAGcAKwC7pW0sEoZ1Jsi4vyK904ELgJmUeoPuq9470uNtKnhI0ZJT0l6oMjiO5yfqORfir8EyyQd0eg+zWzo9fZKN+GIcTawMiKeiIhtwI3A6clmfBhYHBEbimS4GDip3s/Uq1lHjMdGRK3HkJxMqZb0dOBI4Mriq5m1uAH0SndWHDjNL2rJA0wBni1bt4rqOeI/S/og8Bjw+Yh4tsZ7p2QbVctgnEqfDlxbVAe8W9J4SZMjIn9V28yGnQjRlU+ML0ZEvtdtR/8O3BARWyV9FrgG+FAD2+tTMzpfArhd0n2S5lVZn8rokuZJWiJpyRuv5J8DaGZDp0mn0quBqWWv9yuWvSki1kfE1uLlt4H/lH1vPZqRGI+OiCMonTKfVxzqDlhEzI+IWRExa+S4XZvQLDPbmZp4jfFeYLqkAyWNAuYAC8sDJJXfP3ca8EgxfxtwoqQJkiYAJxbLGtLwqXRErC6+rpV0M6ULqXeVheyUjG5mQ68ZQwIjokvS+ZQSWgewICIeknQxsCQiFgJ/Jek0oAvYAJxdvHeDpH+glFwBLo6IDY22qaHEKGkssEtEbCzmTwQurghbCJwv6UZKF1Rf8fVFs9bXzAfVRsQiYFHFsi+XzX8J+FKN9y4AFjSlIYVGjxgnATdL6t3W9RFxq6S/AIiIb1H6sKcAK4FNwKcb3KeZDRMeElhFRDwB/HGV5d8qmw/gvIFst2vTSNbdNykV2zFjY3q7K28/KB3bMy1fWClbuGogw/xu/cn307EzfvWpdOzEqfmzjIu+fnY6dsxL+aF72z6U/97uMyV/n+7GzWPSsb+5MtdBetfY96S3Oeqy/LDI3Ua8kI5d/+Te6diX1u2Riut+o/GhfBHQ5QfVmpltz48dMzMr42JYZmZVhBOjmdn23PliZlYmwtcYzcwqiG73SpuZbc/XGM3MyrhKoJlZpShdZ2xHToxmVjf3Sg+mHujYkvuGb16Xr5A3cV3+z9tej+QvKveMyX0bB1LNbyDD/JZ/4Np07HvuPzMdu/H9+edivrwlP8Rs/P2j0rG7DWBo5vWHXZOOPXHL+f0HAbs8nn8E3qaN+di9x+crCp579B3p2FtW7TBCt6r1o7rS26wl3PliZrYjn0qbmVVwr7SZWZkIJ0Yzsx34dh0zswq+xmhmViYQPe6VNjPbXpseMNZfPlXSDElLy6ZXJf1NRcwxkl4pi/lyre2ZWYspOl8yU38knSRpuaSVki6osv4Lkh6WtEzSHZIOKFvXXZZjFla+tx51HzFGxHJgZtGwDkolUW+uEvqriDi13v2Y2TDWhEPGIn9cAZwArALulbQwIh4uC/sPYFZEbJL0OeBrwMeLdZsjYmbjLXlLsy4QHAc8HhFPN2l7ZtYCmnTEOBtYGRFPRMQ24Ebg9O33E7+IiE3Fy7sp1affaZp1jXEOcEONde+T9HvgOeCLEfFQtSBJ84B5AB0TJrClM1d1bsSr+aFoR5yzLB3762fyFQXXLx2Xihu5If/ndSDV/AYyzO/eI36Qjj3sm+emY7ccvDUdO2pj/vtw+6E/TsdOX/zX6ViNyP1+dY/LVz+cuHhsOnbcJ19Mxx652+Pp2BNmPNx/EPDJMQ3XpC89XacnfbtOp6QlZa/nR8T8Yn4K8GzZulWUatDXcg7w07LXY4ptdwGXRkT+l6aGhhOjpFHAaVQvhn0/cEBEvCbpFODHwPRq2ym+SfMBRu8/tV2v6Zq1jwDy9zG+GBG5mrV9kPRJYBbwJ2WLD4iI1ZIOAn4u6YGIyP81qaIZp9InA/dHxJrKFRHxakS8VswvAkZK6mzCPs1sGIjITf1YDUwte71fsWw7ko4HLgROi4g3T1EiYnXx9QngTuDdDX0ompMYz6LGabSkfSSpmJ9d7G99E/ZpZsNBJKe+3QtMl3RgcQY6B9iud1nSu4GrKCXFtWXLJ0gaXcx3AkcBuesJfWjoVFrSWEo9SZ8tW/YXABHxLeAM4HOSuoDNwJyIdr1X3uztJncrTn8iokvS+cBtQAewICIeknQxsCQiFgL/COwO/J/iWOuZiDgNOBS4SlIPpQOvSyt6s+vSUGKMiNeBvSqWfats/hvANxrZh5kNY006zCkutS2qWPblsvnja7zvN8DhzWnFWzzyxczqExD5XumW4sRoZg1wYjQz216b9hg4MZpZ/ZwYzczKDOwG75YyPBNjQMfW3De8a+98FblfPZ0f5nfIpLX9BxVeOCpXTe/vpi/qP6hw0dfPTscOpJrfQIb5PXzuN9Ox/29LfujcVTOOSccees156dhJ787/zNaszI0z+N6pV6a3+ZcP59v68lX7p2PP2z//M9v92B3GWVT19JZ8RcW+tOvNd8MzMZpZa3CvtJnZ9uQjRjOzMrnhfi3JidHM6iR3vpiZ7cBHjGZmFfI3I7QUJ0Yzq4/vYzQz25F7pc3MKrVpYmxWlUAzs7YxLI8YR+zaRefhueFdHQM4lj9oXL4y21Ov7tV/UGH2H+Sqxr53zLr0Nse8lL+q/fKWfKXEgVTzG8gwv6PG5P/GfvL3h6RjRx/8Wjp27fo907HTD9uhpEj1uJH54ZZ7359v6zOn7JGO3bLftnTsBzpzn+uJEflt9sWn0mZm5YK2HRKY+jMvaYGktZIeLFs2UdJiSSuKrxNqvHduEbNC0txmNdzMhoHmFMMadrLnP1cDJ1UsuwC4IyKmA3cUr7cjaSJwEaXi2bOBi2olUDNrPYrc1GpSiTEi7gI2VCw+Heh9dtE1wEervPXDwOKI2BARLwGL2THBmlmratIRo6STJC2XtFJStYOs0ZJuKtb/TtK0snVfKpYvl/ThJnyqhnqlJ0XE88X8C8CkKjFTgGfLXq8qlplZO2hCYpTUAVwBnAwcBpwl6bCKsHOAlyLiYOBy4LLivYdRqkP9TkoHXd8stteQptyuU9SKbuiAWdI8SUskLel6ZVMzmmVmO1H2NDpxKj0bWBkRT0TENuBGSmek5crPUH8IHKdSgenTgRsjYmtEPAmsLLbXkEYS4xpJkwGKr9Xur1kNTC17vV+xbAcRMT8iZkXErBHjdmugWWY2aHqUm6Cz98CnmOaVbSVzZvlmTER0Aa9Qqmm/U85KG7ldZyEwF7i0+HpLlZjbgP9V1uFyIvClBvZpZsPIADpWXoyIWTuxKU2VvV3nBuC3wAxJqySdQykhniBpBXB88RpJsyR9GyAiNgD/ANxbTBcXy8ysHTSn8yVzZvlmjKQRwDhgffK9A5Y6YoyIs2qsOq5K7BLgM2WvFwAL6mqdmQ1fzbsV515guqQDKSW1OcAnKmJ6z1B/C5wB/DwiQtJC4HpJ/wzsC0wH7mm0QcNy5EvH87uw+yW5IVMb3rlrertLx+6Tjp35sQf7DyrcuvLQVNxtP82fSWz7UL764fj7R6VjR23M/yYPpJrfQIb5PXna/HTsey78XDr2xfd3pWM7PpEbEvfx6yr/f9YWX92Yjh37g93TsVN+mR+aefsrM1Nxr268M73NPjUhMUZEl6TzKV166wAWRMRDki4GlkTEQuA7wPckraR06+Cc4r0PSfoB8DDQBZwXEd2NtmlYJkYzaw1q0oNqI2IRsKhi2ZfL5rcAH6vx3kuAS5rTkhI/XcfMrIKPGM2sfi043C/DidHM6tOi46AznBjNrH5OjGZmFZwYzczeIprXKz3cODGaWX18jdHMrAonRjOzCk6Mg2frxF14/KzRueDu/EWO989ano69++lp6dgRD+aGd23dK9/Wfaa8lI7dbVp++ODth/44HXvoNeelYwdSzW8gw/zuveTKdOwhv/7zdOyKy/dNxXU9NTK9zb3uyf93OvjT+d/FL+x7Wzp2/C65oY5nfDdfMbMvPpU2M6vkxGhmVibcK21mtiMfMZqZbc/XGM3MKjkxmpmVabg26PDlxGhmdRHteyrd74NqJS2QtFbSg2XL/lHSo5KWSbpZ0vga731K0gOSlkpa0syGm9nQa1Jd6WEn8wTvq4GTKpYtBt4VEX8EPEbfJVGPjYiZrVQ60cySmlMlcNjpNzFGxF2Uis+UL7u9KHoNcDelkoVm9nbTpomxGdcY/ytwU411AdwuKYCrIqJmeThJ84B5AKPGTmCv+3LlaDYcnv+uP3PZH6Zjx4/vSMfutjY3JO/p05Te5sbNY9Kx1x92TTp2+uK/TsdOevfadOza9XumYwdSzW8gw/wePfp76djDvnluKq5r//xwy2mfWpGOXbc5XyXwa6tOTseufm1cKu7JLfnfmZoG6TRZ0kRKOWYa8BRwZkS8VBEzE7gS2BPoBi6JiJuKdVcDfwK8UoSfHRFL+9pnQ8WwJF1IqWTh92uEHB0RRwAnA+dJ+mCtbUXE/IiYFRGzRuw6tpFmmdlgGZwjxguAOyJiOnBH8brSJuBTEfFOSpf+vl7R9/E/ikt6M/tLitBAYpR0NnAq8F8ioupHj4jVxde1wM3A7Hr3Z2bDj3pyU4NOB3oPca8BPloZEBGPRcSKYv45YC2wd707rCsxSjoJ+J/AaRGxqUbMWEl79M4DJwL5KvZmNuwNoFe6U9KSsmneAHYzKSKeL+ZfACb12SZpNjAKeLxs8SXFXTSXS+r30V39XmOUdANwDKUPtgq4iFIv9GhgsSSAuyPiLyTtC3w7Ik4pGn9zsX4EcH1E3Nrf/sysRQzsNPnFvu5MkfQzYJ8qqy7cbpcRUfRZ1NrOZOB7wNyI6D1W/RKlhDoKmA/8LXBxX43tNzFGxFlVFn+nRuxzwCnF/BPAH/e3fTNrYU3qfImI42utk7RG0uSIeL5IfFV7BSXtCfwEuDAi7i7bdu/R5lZJ3wW+2F97Gup8MbO3r96RL4Nwg/dCYG4xPxe4ZYe2SKMo9WNcGxE/rFg3ufgqStcn+72k58RoZnVTT6SmBl0KnCBpBXB88RpJsyR9u4g5E/ggcHYx0m5pcQsPwPclPQA8AHQCX+1vhx4rbWb1GaSbtyNiPXBcleVLgM8U89cB19V4/4cGuk8nRjOrWyuOg85wYjSz+jkxDp6pk9fy9b+/IhX7o5fyz6Z4etbEdOzyW/LDB18+JPdt/Mh78g8Y+s2V+c914pbz07Eakb/bds3KznTs9MNWp2M7PpGrZAf5an6QH+YH8PC530zFHfh//1t6m5v+LD/kc9VfTUnHrnnnq/k2PJcbati1tTn/9X3EaGZWyYnRzKyMqwSamW2vnZ/g7cRoZvWr/vyYlufEaGZ18xGjmVm5Fn06d4YTo5nVzZ0vZmYVnBjNzMoF7nwZTE+93sk5S+b2Hwh07vl6ert7jt6Sjn3/mf+Rjj1o1xdTcTPGPN9/UOGuse9Jx+7y+K7p2O5x+T/x3zv1ynTs9JGb07Efv+4T6diup0bmYwdQuCo7ouXJU/81vc0/+ff8Q6kH0mnRvSxX4ArgiOMeS8X9Ytet+Qb0wZ0vZmaVnBjNzN7Szjd49/ugWkkLJK2V9GDZsq9IWl32QMhTarz3JEnLJa2UVK3koZm1qsg9pLYJD6oddJkneF9NqU5rpcvL6rQuqlwpqQO4glJN6cOAsyQd1khjzWyYGZy60oOu38QYEXcBG+rY9mxgZUQ8ERHbgBsp1Yc1szYxSDVfBl0jNV/OL+q0LpA0ocr6KcCzZa9XFcuqkjSvt+Zs96v5nmYzGyIB9ERuajH1JsYrgXcAM4HngX9qtCERMT8iZkXErI49xza6OTMbDINwKi1poqTFklYUX6sdiCGpu6zfY2HZ8gMl/a7o67ipqCjYp7oSY0SsiYjuoqD1v1I6ba60Gpha9nq/YpmZtYlBOpW+ALgjIqYDdxSvq9lc1u9xWtnyyyj1iRwMvASc098O60qMvXVaC39K9Tqt9wLTi2w9CphDqT6smbWJQeqVPh24ppi/hlJt6Fz7SrWkPwT01ppOvT9zu84NwG+BGZJWSToH+JqkByQtA44FPl/E7itpEUBEdAHnA7cBjwA/iIiHsh/IzIa57Gl040eMkyKid9jYC8CkGnFjin6KuyX1Jr+9gJeLfAT99HX06vcG74g4q8ri79SIfQ44pez1ImCHW3n6M/rZLt7x+fWp2I2z9ktvt/uWe9Kxj36k2tWB6h7blhtmd+fq19LbHHXZunTspo35IYETF+ev3/7lw+elY/e+P//Z4qsb07F73ZMfgzDtUyvSsdnCVQMZ5vfLq+anY//onmr/rarbtmx8OvbJ66an4rZuGJ3eZi2lG7zTWa9TUnk1uPkR8eY3TNLPgH2qvO/C8hcREVLNk/MDImK1pIOAn0t6AHgl28ByHvliZvXLD71/MSJqlr6MiONrrZO0RtLkiHi+uIy3tsY2Vhdfn5B0J/Bu4N+A8ZJGFEeNqb6ORm7XMbO3OUWkpgYtBHqfKjMXuGWHdkgTJI0u5juBo4CHIyKAXwBn9PX+Sk6MZlafwbvGeClwgqQVwPHFayTNkvTtIuZQYImk31NKhJdGxMPFur8FviBpJaVrjlUvBZbzqbSZ1WlwxkFHxHrguCrLlwCfKeZ/Axxe4/1PUP2WwpqcGM2sfn5QrZlZmXBpAzOzHfmI0cysQnvmRSdGM6ufetrzXNqJ0czqEwzkBu+WMiwT45ZJo1j+hf1TsTEpX+1s4sT3pWN33dCdjn15er9PMQKgY/+qT0uqarcRL6Rj9x6fH4437pO5ioYAL1+V+xkAPHPKHunYsT/YPR178KeXp2PXbc5vd9Vf9TtcFhjYk2EGMsxv2ewb0rFfPeiQdOy/PTkzF/iz/O93LaIpN28PS8MyMZpZi3BiNDOr4MRoZlbG1xjNzHbkXmkzs+2ET6XNzLYTODGame2gPc+k+0+MkhYApwJrI+JdxbKbgBlFyHhKNRV2uIFK0lPARqAb6OrrCb5m1nrezvcxXg18A7i2d0FEfLx3XtI/0XddhWMjIn9XsZm1jrdrYoyIuyRNq7auKE14JqXyhGb2dhIB3e15Lt3oNcYPAGsiolZ5tgBuL6p6XVVeFaySpHnAPIAR4yfQsSVXxW2XFfkKeS8e2dV/UGHi/R3p2OwTRja8K//Xdf2Te6djzz36jnTskbs9no49b/9z07Fb9tuWjp3yy/x/pi/se1s69murTk7Hrnnnq6m47mXj0tscSDW/gQzz+/vOR9Oxk0e+nIq7ZHS+UmOf3q5HjP04C+hr0OfRRTnDPwAWS3o0Iu6qFlgkzfkAY/ab2p7fbbN206aJse5iWJJGAH8G3FQrpqyc4VrgZgZYd8HMhrEAeiI3tZhGqgQeDzwaEauqrZQ0VtIevfPAicCDDezPzIaVgOjJTQ2QNFHSYkkriq87PKZK0rGSlpZNWyR9tFh3taQny9b1+wiifhOjpBuA3wIzJK2SdE6xag4Vp9GS9pW0qHg5Cfh1Uc7wHuAnEXFrf/szsxYRlDpfMlNjLgDuiIjpwB3F6+2bEvGLiJhZ3Db4IWATcHtZyP/oXR8RS/vbYaZXuupD5iLi7CrLngNOKeafAP64v+2bWQsbnGuMpwPHFPPXAHdSqhVdyxnATyNiU707bORU2sze7iJyE3RKWlI2zRvAXiZFxPPF/AuUzkb7ssPZLHCJpGWSLpc0ur8dekigmdVpQA+ReLGvkW+SfgbsU2XVhdvtMSKK2/9qbWcycDhQfp/Xlygl1FGU7nz5W+DivhrrxGhm9QmgSY8di4jja62TtEbS5Ih4vkh8a/vY1JnAzRHxRtm2e482t0r6LvDF/trjU2kzq1/+VLoRC4F0E1jeAAADrElEQVS5xfxc4JY+Yne4t7pIpr0j9T5K4u4YJ0Yzq1MMVq/0pcAJklZQuk3wUgBJsyR9uzeoGLo8Ffhlxfu/L+kB4AGgE/hqfzscnqfSPbDL1tyQwJ6R+b9Gx8x8JB27Zka+6t2jz1S7NLKj8RNeT2/zpXX5/d+yKt/5f8KMh9Oxux+7Jh37gc7V6djbX0lWsgPG75Ifarj6tfzwvU3P5SoKHnHcY+ltPnnd9HRsupof+WF+AOeMy1WXvKrjjf6D+hMQDd6jmNpNxHrguCrLlwCfKXv9FLBD+ceIGPCzHIZnYjSz1tCCo1oynBjNrH5tOlbaidHM6hPRtF7p4caJ0czq5yNGM7NyQXR3D3UjdgonRjOrT+9jx9qQE6OZ1W8QbtcZCk6MZlaXAMJHjGZmZSJ8xGhmVqldO18Uw7C7XdI64OmKxZ1AO9anbtfPBe372drhcx0QEflSlFVIupXS9yLjxYg4qZH9DaZhmRirkbSkr+e5tap2/VzQvp+tXT+XvcVP1zEzq+DEaGZWoZUS4/yhbsBO0q6fC9r3s7Xr57JCy1xjNDMbLK10xGhmNiicGM3MKrREYpR0kqTlklZKumCo29Mskp6S9ICkpZKWDHV7GiFpgaS1kh4sWzZR0mJJK4qvE4ayjfWo8bm+Iml18XNbKumUoWyjNd+wT4ySOoArgJOBw4CzJB02tK1qqmMjYmYb3Bd3NVB5A+8FwB0RMR24o3jdaq5mx88FcHnxc5sZEYsGuU22kw37xAjMBlZGxBMRsQ24ETh9iNtkFSLiLmBDxeLTgWuK+Wsola5sKTU+l7W5VkiMU4Bny16vokolsBYVwO2S7pM0b6gbsxNMKit2/gIwaSgb02TnS1pWnGq33CUC61srJMZ2dnREHEHpMsF5kj441A3aWaJ0X1i73Bt2JfAOYCbwPPBPQ9sca7ZWSIyrKRXR7rVfsazlRcTq4uta4GZKlw3ayRpJkwGKr2uHuD1NERFrIqI7SkWV/5X2+7m97bVCYrwXmC7pQEmjgDnAwiFuU8MkjZW0R+88cCLwYN/vajkLgbnF/FzgliFsS9P0JvvCn9J+P7e3vWH/PMaI6JJ0PnAb0AEsiIiHhrhZzTAJuFkSlH4O10fErUPbpPpJugE4BuiUtAq4CLgU+IGkcyg9Ru7MoWthfWp8rmMkzaR0aeAp4LND1kDbKTwk0MysQiucSpuZDSonRjOzCk6MZmYVnBjNzCo4MZqZVXBiNDOr4MRoZlbh/wOund+2WK4kIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQgAAAD8CAYAAACLgjpEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADwNJREFUeJzt3XusZWV9xvHv0wEkUKyM6MhlKsZOSaiRqSFDTWkDRbmFONpYO6RpsaUZaySpSZuGtokY+49NY01ajMbLBGwU7W10EkeGybQJmigwEK7KZUowzBEZZSyjouLgr3+cNeT0zH6ZPWftc/aF7yc52evy7rXedfbJk7X2es/6paqQpEF+YdwdkDS5DAhJTQaEpCYDQlKTASGpyYCQ1GRASGoyICQ1GRCSmo4ZdwcGOWX1qjpz7bEj3+7D954w8m1qOv3q658Zqt2s/s38hB/xbP00R2o3kQFx5tpjuX3H2pFv95LT1o98m5pOO3bcPVS7Wf2bua12DdWu1yVGkkuTPJRkT5JrB6x/SZLPd+tvS3Jmn/1JWllLDogkq4CPAJcBZwNXJjl7UbOrge9X1a8AHwb+fqn7k7Ty+pxBbAD2VNWjVfUs8Dlg46I2G4Ebu+l/By5KcsTrHkmToU9AnA48vmB+b7dsYJuqOgg8Dby8xz4lraCJuc2ZZHOS3Ul2f/ep58bdHUn0C4g5YOGthjO6ZQPbJDkG+CXgqUEbq6qPV9W5VXXuK16+qke3JI1Kn4C4A1iX5DVJjgM2AdsWtdkGXNVNvx34r/IRVtLUWPI4iKo6mOQaYAewCthSVQ8k+QCwu6q2AZ8C/iXJHmA/8yEiaUr0GihVVduB7YuWvW/B9E+A3+uzD0njM5EjKaXlNqsjJEdtYu5iSJo8BoSkJgNCUpMBIanJgJDUZEBIajIgJDUZEJKaDAhJTQaEpKaJHGr98L0nLMtQ2B3fHu5BpUfLYbuaVZ5BSGoyICQ1GRCSmgwISU0GhKQmA0JSU5/KWmuT/HeSbyR5IMmfD2hzQZKnk9zd/bxv0LYkTaY+4yAOAn9RVXclOQm4M8nOqvrGonZfqaoreuxH0pgs+Qyiqp6oqru66R8A3+TwylqSpthIvoPoqnb/OnDbgNVvTHJPki8n+bVR7E/Syug91DrJLwL/Aby3qg4sWn0X8Oqq+mGSy4EvAOsa29kMbAY4nhP6dkvSCPQ6g0hyLPPh8Jmq+s/F66vqQFX9sJveDhyb5JRB21pYeu9YXtKnW5JGpM9djDBfOeubVfWPjTav6tqRZEO3v4G1OSVNnj6XGL8J/CFwX5JD/yb5N8AvA1TVx5ivx/nuJAeBHwObrM0pTY8+tTm/CuQIba4Hrl/qPiSNlyMpJTUZEJKaDAhJTQaEpCYDQlKTASGpaSKfar1cluvp08M+LdunX0+fo3kS+ix+vp5BSGoyICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNRkQEhqelGNpFwuw46gm7ZReUfT36MxCcem4XgGIampd0AkeSzJfV1pvd0D1ifJPyXZk+TeJG/ou09JK2NUlxgXVtX3GusuY74WxjrgPOCj3aukCbcSlxgbgU/XvK8DL0ty6grsV1JPowiIAm5JcmdXHWux04HHF8zvxRqe0lQYxSXG+VU1l+SVwM4kD1bVrUe7EUvvSZOn9xlEVc11r/uArcCGRU3mgLUL5s/oli3ejqX3pAnTtzbniUlOOjQNXAzcv6jZNuCPursZvwE8XVVP9NmvpJXR9xJjDbC1K795DPDZqro5yZ/B8+X3tgOXA3uAZ4A/7rlPSSukV0BU1aPAOQOWf2zBdAHv6bMfSeORSayl+9KsrvNy0bi7MVbTNixb0+W22sWB2v+CtXXBodaSXoABIanJgJDUZEBIajIgJDUZEJKaDAhJTQaEpCYDQlKTASGpyadaT6ijGT7tsOx5s/p7GOdxeQYhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkpiUHRJKzunqch34OJHnvojYXJHl6QZv39e+ypJWy5IFSVfUQsB4gySrma11sHdD0K1V1xVL3I2l8RnWJcRHwP1X1rRFtT9IEGNVQ603ATY11b0xyD/Bt4C+r6oFBjSy9t3SzPCx72vo7bsP+vjZc8sxQ7XqfQSQ5DngL8G8DVt8FvLqqzgH+GfhCazuW3pMmzyguMS4D7qqqJxevqKoDVfXDbno7cGySU0awT0krYBQBcSWNy4skr0pXly/Jhm5/T41gn5JWQK/vILqCvW8G3rVg2cK6nG8H3p3kIPBjYFNNYikvSQP1rc35I+Dli5YtrMt5PXB9n31IGh9HUkpqMiAkNRkQkpoMCElNBoSkpkziXceXZnWdl4vG3Y2Rm7Zhw0fT36MxCcc2rGn7zIZ1W+3iQO3Pkdp5BiGpyYCQ1GRASGoyICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNRkQEhqMiAkNQ0VEEm2JNmX5P4Fy1Yn2Znkke715MZ7r+raPJLkqlF1XNLyG/YM4gbg0kXLrgV2VdU6YFc3//8kWQ1cB5wHbACuawWJpMkzVEBU1a3A/kWLNwI3dtM3Am8d8NZLgJ1Vtb+qvg/s5PCgkTSh+nwHsaaqnuimvwOsGdDmdODxBfN7u2WSpsBIvqTsal30evJMks1JdifZ/TN+OopuSeqpT0A8meRUgO5134A2c8DaBfNndMsOY21OafL0CYhtwKG7ElcBXxzQZgdwcZKTuy8nL+6WSZoCw97mvAn4GnBWkr1JrgY+CLw5ySPAm7p5kpyb5JMAVbUf+Dvgju7nA90ySVNgqNJ7VXVlY9VhT5atqt3Any6Y3wJsWVLvJI2VT7XWipvVJ0VPE59qLak3A0JSkwEhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkJgNCUtNQ/4uh2TEJw5yPZruT0N9xO5rfwbA2XPLMUO08g5DUZEBIajIgJDUZEJKaDAhJTQaEpKYjBkSj7N4/JHkwyb1JtiZ5WeO9jyW5L8ndSXaPsuOSlt8wZxA3cHg1rJ3A66rq9cDDwF+/wPsvrKr1VXXu0rooaVyOGBCDyu5V1S1VdbCb/Trz9S4kzZhRfAfxJ8CXG+sKuCXJnUk2j2BfklZQr6HWSf4WOAh8ptHk/KqaS/JKYGeSB7szkkHb2gxsBjieE/p060VnlocjOyx7efr6cD01VLsln0EkeSdwBfAH1Xh2flXNda/7gK3Ahtb2LL0nTZ4lBUSSS4G/At5SVQP/6yPJiUlOOjTNfNm9+we1lTSZhrnNOajs3vXAScxfNtyd5GNd29OSbO/eugb4apJ7gNuBL1XVzctyFJKWxRG/g2iU3ftUo+23gcu76UeBc3r1TtJYOZJSUpMBIanJgJDUZEBIajIgJDUZEJKa0hgEOVbnnnN83b5j7VBtp2nIrCbHsMOyZ/Xv67baxYHanyO18wxCUpMBIanJgJDUZEBIajIgJDUZEJKaDAhJTQaEpCYDQlLTRI6kfGlW13m5aNzdGNrRPCx1WJMwgm85jgsm49iGNasPwnUkpaTellp67/1J5rrnUd6d5PLGey9N8lCSPUmuHWXHJS2/pZbeA/hwV1JvfVVtX7wyySrgI8BlwNnAlUnO7tNZSStrSaX3hrQB2FNVj1bVs8DngI1L2I6kMenzHcQ1XXXvLUlOHrD+dODxBfN7u2WSpsRSA+KjwGuB9cATwIf6diTJ5iS7k+z+GT/tuzlJI7CkgKiqJ6vquar6OfAJBpfUmwMWPvXljG5Za5uW3pMmzFJL7526YPZtDC6pdwewLslrkhwHbAK2LWV/ksbjiJW1utJ7FwCnJNkLXAdckGQ9UMBjwLu6tqcBn6yqy6vqYJJrgB3AKmBLVT2wLEchaVksW+m9bn47cNgtUEnTwaHW0ohM07Bsh1pL6s2AkNRkQEhqMiAkNRkQkpoMCElNBoSkJgNCUpMBIanJgJDUdMT/xZA0nKMZPj0tTwz3DEJSkwEhqcmAkNRkQEhqMiAkNRkQkpqGeSblFuAKYF9Vva5b9nngrK7Jy4D/rarD7q8keQz4AfAccLCqzh1RvyWtgGHGQdwAXA98+tCCqvr9Q9NJPgQ8/QLvv7CqvrfUDkoan2EeWntrkjMHrUsS4B3A74y2W5ImQd/vIH4LeLKqHmmsL+CWJHcm2dxzX5JWWN+h1lcCN73A+vOrai7JK4GdSR7sigEfpguQzQDHc0LPbr24TMuw3aWYpidFT4Jhf18bLnlmqHZLPoNIcgzwu8DnW22qaq573QdsZXCJvkNtLb0nTZg+lxhvAh6sqr2DViY5MclJh6aBixlcok/ShDpiQHSl974GnJVkb5Kru1WbWHR5keS0JIcqaa0BvprkHuB24EtVdfPoui5puS219B5V9c4By54vvVdVjwLn9OyfpDFyJKWkJgNCUpMBIanJgJDUZEBIajIgJDX5VOsZMAlPU9bRmZbPzDMISU0GhKQmA0JSkwEhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkplTVuPtwmCTfBb61aPEpwCwW4JnV44LZPbZZOK5XV9UrjtRoIgNikCS7Z7F036weF8zusc3qcQ3iJYakJgNCUtM0BcTHx92BZTKrxwWze2yzelyHmZrvICStvGk6g5C0wqYiIJJcmuShJHuSXDvu/oxKkseS3Jfk7iS7x92fPpJsSbIvyf0Llq1OsjPJI93ryePs41I0juv9Sea6z+3uJJePs4/LaeIDIskq4CPAZcDZwJVJzh5vr0bqwqpaPwO3zW4ALl207FpgV1WtA3Z189PmBg4/LoAPd5/b+qraPmD9TJj4gGC+Ivieqnq0qp4FPgdsHHOftEhV3QrsX7R4I3BjN30j8NYV7dQINI7rRWMaAuJ04PEF83u7ZbOggFuS3Jlk87g7swzWVNUT3fR3mC/oPCuuSXJvdwkydZdOw5qGgJhl51fVG5i/fHpPkt8ed4eWS83fLpuVW2YfBV4LrAeeAD403u4sn2kIiDlg7YL5M7plU6+q5rrXfcBW5i+nZsmTSU4F6F73jbk/I1FVT1bVc1X1c+ATzN7n9rxpCIg7gHVJXpPkOGATsG3MfeotyYlJTjo0DVwM3P/C75o624CruumrgC+OsS8jcyj0Om9j9j6350184ZyqOpjkGmAHsArYUlUPjLlbo7AG2JoE5j+Hz1bVzePt0tIluQm4ADglyV7gOuCDwL8muZr5/859x/h6uDSN47ogyXrmL5keA941tg4uM0dSSmqahksMSWNiQEhqMiAkNRkQkpoMCElNBoSkJgNCUpMBIanp/wAP6YhAx52cDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQgAAAD8CAYAAACLgjpEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADhhJREFUeJzt3X2sZHV9x/H3p8uigZLyJCsPWyB2Q0JN2Rqy1pQ2UBUWQlxtrF3StNjSLDWS1KRNQ9tEjP3HprEmLUbiwwZsFLUPq5u4AhvaBEl8YCHLk4JsCYa9IKtgQUTFxW//uOeS6935sZc5M3dmru9XcjPn/M5vzvmdTPLJOTPn/r6pKiRpkF+a9AAkTS8DQlKTASGpyYCQ1GRASGoyICQ1GRCSmgwISU0GhKSmIyY9gEFOPH5NnbF+7cj3+617jhr5PqVZ9GN+yPP1kxyu31QGxBnr1/L1m9ePfL8XnbJx5PuUZtHX6tZl9et1i5Fkc5IHk+xLcvWA7a9I8tlu+9eSnNHneJJW1tABkWQN8GHgYuBs4LIkZy/pdgXw/ar6NeBDwD8OezxJK6/PFcQmYF9VPVxVzwOfAbYs6bMFuKFb/g/gjUkOe98jaTr0CYhTgUcXre/v2gb2qaqDwNPACT2OKWkFTc3PnEm2JdmTZM93n3xh0sORRL+AmAMW/9RwWtc2sE+SI4BfAZ4ctLOq+mhVnVtV577qhDU9hiVpVPoExB3AhiRnJjkS2ArsXNJnJ3B5t/x24L/LKaykmTH0cxBVdTDJVcDNwBpge1Xdn+T9wJ6q2gl8Avi3JPuAp5gPEUkzoteDUlW1C9i1pO29i5Z/DPxBn2NImpyp+ZJS0vQxICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNRkQEhqMiAkNRkQkpqmctLab91z1FgmmL35sb0j3yc4Ga5WL68gJDUZEJKaDAhJTQaEpCYDQlKTASGpqU9lrfVJ/ifJN5Lcn+QvB/Q5P8nTSfZ2f+8dtC9J06nPcxAHgb+qqruSHAPcmWR3VX1jSb8vV9WlPY4jaUKGvoKoqser6q5u+QfANzm0spakGTaS7yC6qt2/CXxtwOY3JLk7yZeS/PoojidpZfR+1DrJLwP/Cbynqp5Zsvku4PSqejbJJcDngQ2N/WwDtgG8kqP6DkvSCPS6gkiylvlw+FRV/dfS7VX1TFU92y3vAtYmOXHQvhaX3lvLK/oMS9KI9PkVI8xXzvpmVf1zo8+ru34k2dQdb2BtTknTp88txm8Dfwzcm2Th3yT/DvhVgKq6jvl6nO9KchD4EbDV2pzS7OhTm/N2IIfpcy1w7bDHkDRZPkkpqcmAkNRkQEhqMiAkNRkQkpoMCElNUzmr9biMa/bpccyW7UzZmgZeQUhqMiAkNRkQkpoMCElNBoSkJgNCUpMBIanJgJDUZEBIajIgJDUZEJKaegdEkkeS3NuV1tszYHuS/EuSfUnuSfK6vseUtDJG9c9aF1TV9xrbLma+FsYG4PXAR7pXSVNuJW4xtgCfrHlfBY5NcvIKHFdST6MIiAJuSXJnVx1rqVOBRxet78cantJMGMUtxnlVNZfkJGB3kgeq6raXuxNL70nTp/cVRFXNda8HgB3ApiVd5oD1i9ZP69qW7sfSe9KU6Vub8+gkxywsAxcC9y3pthP4k+7XjN8Cnq6qx/scV9LK6HuLsQ7Y0ZXfPAL4dFXdlOQv4MXye7uAS4B9wHPAn/Y8pqQV0isgquph4JwB7dctWi7g3X2OI2kyfqEmrR2XcUwwO46JcMHJcPXy+Ki1pCYDQlKTASGpyYCQ1GRASGoyICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkJgNCUtPQAZHkrK4e58LfM0nes6TP+UmeXtTnvf2HLGmlDD0nZVU9CGwESLKG+VoXOwZ0/XJVXTrscSRNzqhuMd4I/G9VfXtE+5M0BUY1q/VW4MbGtjckuRt4DPjrqrp/UCdL7/28cc0+7WzZejl6X0EkORJ4C/DvAzbfBZxeVecA/wp8vrUfS+9J02cUtxgXA3dV1RNLN1TVM1X1bLe8C1ib5MQRHFPSChhFQFxG4/YiyavT1eVLsqk73pMjOKakFdDrO4iuYO+bgSsXtS2uy/l24F1JDgI/ArZ2pfgkzYC+tTl/CJywpG1xXc5rgWv7HEPS5PgkpaQmA0JSkwEhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkJgNCUpMBIanJgJDUZEBIajIgJDUZEJKaDAhJTaOa1Vozwtmy9XJ4BSGpaVkBkWR7kgNJ7lvUdnyS3Uke6l6Pa7z38q7PQ0kuH9XAJY3fcq8grgc2L2m7Gri1qjYAt3brPyfJ8cA1wOuBTcA1rSCRNH2WFRBVdRvw1JLmLcAN3fINwFsHvPUiYHdVPVVV3wd2c2jQSJpSfb6DWFdVj3fL3wHWDehzKvDoovX9XZukGTCSLym7Whe96l0k2ZZkT5I9P+UnoxiWpJ76BMQTSU4G6F4PDOgzB6xftH5a13YIa3NK06dPQOwEFn6VuBz4woA+NwMXJjmu+3Lywq5N0gxY7s+cNwJfAc5Ksj/JFcAHgDcneQh4U7dOknOTfBygqp4C/gG4o/t7f9cmaQYs60nKqrqssemNA/ruAf580fp2YPtQo5M0UT5JKanJgJDUZEBIajIgJDUZEJKaDAhJTQaEpCYDQlKTASGpyYCQ1GRASGpyVmuNhLNlr05eQUhqMiAkNRkQkpoMCElNBoSkJgNCUtNhA6JRdu+fkjyQ5J4kO5Ic23jvI0nuTbI3yZ5RDlzS+C3nCuJ6Dq2GtRt4bVX9BvAt4G9f4v0XVNXGqjp3uCFKmpTDBsSgsntVdUtVHexWv8p8vQtJq8wovoP4M+BLjW0F3JLkziTbRnAsSSuo16PWSf4eOAh8qtHlvKqaS3ISsDvJA90VyaB9bQO2AbySo/oMS9KIDH0FkeSdwKXAH3W1OQ9RVXPd6wFgB7CptT9L70nTZ6iASLIZ+BvgLVX1XKPP0UmOWVhmvuzefYP6SppOy/mZc1DZvWuBY5i/bdib5Lqu7ylJdnVvXQfcnuRu4OvAF6vqprGchaSxOOx3EI2ye59o9H0MuKRbfhg4p9foJE2UT1JKajIgJDUZEJKaDAhJTQaEpCYDQlKTs1prqs3SbNmrcaZsryAkNRkQkpoMCElNBoSkJgNCUpMBIanJgJDUZEBIajIgJDUZEJKaDAhJTcOW3ntfkrluPsq9SS5pvHdzkgeT7Ety9SgHLmn8hi29B/ChrqTexqratXRjkjXAh4GLgbOBy5Kc3WewklbWUKX3lmkTsK+qHq6q54HPAFuG2I+kCenzHcRVXXXv7UmOG7D9VODRRev7uzZJM2LYgPgI8BpgI/A48MG+A0myLcmeJHt+yk/67k7SCAwVEFX1RFW9UFU/Az7G4JJ6c8D6ReundW2tfVp6T5oyw5beO3nR6tsYXFLvDmBDkjOTHAlsBXYOczxJk3HYKee60nvnAycm2Q9cA5yfZCNQwCPAlV3fU4CPV9UlVXUwyVXAzcAaYHtV3T+Ws5A0FmMrvdet7wIO+QlU0mxw0lr9QhrHBLPjmAgXJjsZro9aS2oyICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNRkQEhqMiAkNRkQkpoMCElNBoSkJgNCUpMBIanJgJDUZEBIalrOnJTbgUuBA1X12q7ts8BZXZdjgf+rqkOmvUnyCPAD4AXgYFWdO6JxS1oBy5ly7nrgWuCTCw1V9YcLy0k+CDz9Eu+/oKq+N+wAJU3OciatvS3JGYO2JQnwDuD3RjssSdOg73cQvwM8UVUPNbYXcEuSO5Ns63ksSSus76zWlwE3vsT286pqLslJwO4kD3TFgA/RBcg2gFdyVM9hSStvXLNPj2O27E0XPbesfkNfQSQ5Avh94LOtPlU1170eAHYwuETfQl9L70lTps8txpuAB6pq/6CNSY5OcszCMnAhg0v0SZpShw2IrvTeV4CzkuxPckW3aStLbi+SnJJkoZLWOuD2JHcDXwe+WFU3jW7oksZt2NJ7VNU7B7S9WHqvqh4Gzuk5PkkT5JOUkpoMCElNBoSkJgNCUpMBIanJgJDUZEBIajIgJDUZEJKaDAhJTQaEpCYDQlKTASGpyYCQ1GRASGoyICQ1GRCSmlJVkx7DIZJ8F/j2kuYTgdVYgGe1nhes3nNbDed1elW96nCdpjIgBkmyZzWW7lut5wWr99xW63kN4i2GpCYDQlLTLAXERyc9gDFZrecFq/fcVut5HWJmvoOQtPJm6QpC0gqbiYBIsjnJg0n2Jbl60uMZlSSPJLk3yd4keyY9nj6SbE9yIMl9i9qOT7I7yUPd63GTHOMwGuf1viRz3ee2N8klkxzjOE19QCRZA3wYuBg4G7gsydmTHdVIXVBVG1fBz2bXA5uXtF0N3FpVG4Bbu/VZcz2HnhfAh7rPbWNV7RqwfVWY+oBgviL4vqp6uKqeBz4DbJnwmLREVd0GPLWkeQtwQ7d8A/DWFR3UCDTO6xfGLATEqcCji9b3d22rQQG3JLkzybZJD2YM1lXV493yd5gv6LxaXJXknu4WZOZunZZrFgJiNTuvql7H/O3Tu5P87qQHNC41/3PZavnJ7CPAa4CNwOPAByc7nPGZhYCYA9YvWj+ta5t5VTXXvR4AdjB/O7WaPJHkZIDu9cCExzMSVfVEVb1QVT8DPsbq+9xeNAsBcQewIcmZSY4EtgI7Jzym3pIcneSYhWXgQuC+l37XzNkJXN4tXw58YYJjGZmF0Ou8jdX3ub3oiEkP4HCq6mCSq4CbgTXA9qq6f8LDGoV1wI4kMP85fLqqbprskIaX5EbgfODEJPuBa4APAJ9LcgXz/537jsmNcDiN8zo/yUbmb5keAa6c2ADHzCcpJTXNwi2GpAkxICQ1GRCSmgwISU0GhKQmA0JSkwEhqcmAkNT0/1em+lmHwYJfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compute the 2D overlap in the matrix:\n",
      "test_jaccard_score: 0.885\n",
      "test_f1_score: 0.5306122448979592\n",
      "test_accuracy_score: 0.885\n"
     ]
    }
   ],
   "source": [
    "sentences_raw = (\n",
    "    \"This is a test. This is another test. \"\n",
    "    \"I like bacon. I don't like bacon. \"\n",
    "    \"My name is Guillaume. My family name is Chevalier. \"\n",
    "    \"Programming can be used for solving complicated math problems. Let's use the Python language to write some scientific code. \"\n",
    "    \"My family regrouped for Christmast. We met aunts and uncles. \"\n",
    "    \"I like linux. I have an operating system. \"\n",
    "    \"Have you ever been in the situation where you've got Jupyter notebooks (iPython notebooks) so huge that you were feeling stuck in your code?. Or even worse: have you ever found yourself duplicating your notebook to do changes, and then ending up with lots of badly named notebooks?. \"\n",
    "    \"Either and in any ways. For every medium to big application. \"\n",
    "    \"If you're working with notebooks, it is highly likely that you're doing research and development. If doing research and development, to keep your amazing-10x-working-speed-multiplier, it might be a good idea to skip unit tests. \"\n",
    "    \"I hope you were satisfied by this reading. What would you do?.\"\n",
    ").split(\". \")  # each 2 sentence (pairs) above are similar, so we have 10 pairs as such:\n",
    "category_per_sentence = [0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9]\n",
    "\n",
    "plot_a_result(\n",
    "    category_per_sentence, cuda_device_id, preproc_sgnn_sklearn_pipeline, \n",
    "    sentence_projection_model, sentences_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last plot is the expected diagonal block matrix (blocs of 2x2), and the top plot is the prediction. Mid plot is what is above 1 std in the prediction."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}