3- Information gain - mutual information In Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection-Information gain - mutual information In Classification Problem Statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mutual Information \n",
    "\n",
    "MI Estimate mutual information for a discrete target variable.\n",
    "\n",
    "Mutual information (MI) between two random variables is a non-negative value, which measures the dependency between the variables. It is equal to zero if and only if two random variables are independent, and higher values mean higher dependency.\n",
    "\n",
    "The function relies on nonparametric methods based on entropy estimation from k-nearest neighbors distances.\n",
    "\n",
    "Inshort\n",
    "\n",
    "A quantity called mutual information measures the amount of information one can obtain from one random variable given another.\n",
    "\n",
    "The mutual information between two random variables X and Y can be stated formally as follows:\n",
    "\n",
    "<b>I(X ; Y) = H(X) – H(X | Y)<b>\n",
    "Where I(X ; Y) is the mutual information for X and Y, H(X) is the entropy for X and H(X | Y) is the conditional entropy for X given Y. The result has the units of bits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Youtube Videos\n",
    "\n",
    "Entropy: https://www.youtube.com/watch?v=1IQOtJ4NI_0\n",
    "        \n",
    "Information Gain: https://www.youtube.com/watch?v=FuTRucXB9rA\n",
    "        \n",
    "Gini Impurity: https://www.youtube.com/watch?v=5aIFgrrTqOw\n",
    "\n",
    "Statistical test: https://www.youtube.com/watch?v=4-rxTA_5_xA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Wine</th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Malic.acid</th>\n",
       "      <th>Ash</th>\n",
       "      <th>Acl</th>\n",
       "      <th>Mg</th>\n",
       "      <th>Phenols</th>\n",
       "      <th>Flavanoids</th>\n",
       "      <th>Nonflavanoid.phenols</th>\n",
       "      <th>Proanth</th>\n",
       "      <th>Color.int</th>\n",
       "      <th>Hue</th>\n",
       "      <th>OD</th>\n",
       "      <th>Proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Wine  Alcohol  Malic.acid   Ash   Acl   Mg  Phenols  Flavanoids  \\\n",
       "0     1    14.23        1.71  2.43  15.6  127     2.80        3.06   \n",
       "1     1    13.20        1.78  2.14  11.2  100     2.65        2.76   \n",
       "2     1    13.16        2.36  2.67  18.6  101     2.80        3.24   \n",
       "3     1    14.37        1.95  2.50  16.8  113     3.85        3.49   \n",
       "4     1    13.24        2.59  2.87  21.0  118     2.80        2.69   \n",
       "\n",
       "   Nonflavanoid.phenols  Proanth  Color.int   Hue    OD  Proline  \n",
       "0                  0.28     2.29       5.64  1.04  3.92     1065  \n",
       "1                  0.26     1.28       4.38  1.05  3.40     1050  \n",
       "2                  0.30     2.81       5.68  1.03  3.17     1185  \n",
       "3                  0.24     2.18       7.80  0.86  3.45     1480  \n",
       "4                  0.39     1.82       4.32  1.04  2.93      735  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv('https://gist.githubusercontent.com/tijptjik/9408623/raw/b237fa5848349a14a14e5d4107dc7897c21951f5/wine.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3], dtype=int64)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Wine'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 178 entries, 0 to 177\n",
      "Data columns (total 14 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   Wine                  178 non-null    int64  \n",
      " 1   Alcohol               178 non-null    float64\n",
      " 2   Malic.acid            178 non-null    float64\n",
      " 3   Ash                   178 non-null    float64\n",
      " 4   Acl                   178 non-null    float64\n",
      " 5   Mg                    178 non-null    int64  \n",
      " 6   Phenols               178 non-null    float64\n",
      " 7   Flavanoids            178 non-null    float64\n",
      " 8   Nonflavanoid.phenols  178 non-null    float64\n",
      " 9   Proanth               178 non-null    float64\n",
      " 10  Color.int             178 non-null    float64\n",
      " 11  Hue                   178 non-null    float64\n",
      " 12  OD                    178 non-null    float64\n",
      " 13  Proline               178 non-null    int64  \n",
      "dtypes: float64(11), int64(3)\n",
      "memory usage: 19.6 KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Train test split to avoid overfitting\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train,X_test,y_train,y_test=train_test_split(df.drop(labels=['Wine'], axis=1),\n",
    "    df['Wine'],\n",
    "    test_size=0.3,\n",
    "    random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Malic.acid</th>\n",
       "      <th>Ash</th>\n",
       "      <th>Acl</th>\n",
       "      <th>Mg</th>\n",
       "      <th>Phenols</th>\n",
       "      <th>Flavanoids</th>\n",
       "      <th>Nonflavanoid.phenols</th>\n",
       "      <th>Proanth</th>\n",
       "      <th>Color.int</th>\n",
       "      <th>Hue</th>\n",
       "      <th>OD</th>\n",
       "      <th>Proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>13.71</td>\n",
       "      <td>1.86</td>\n",
       "      <td>2.36</td>\n",
       "      <td>16.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.61</td>\n",
       "      <td>2.88</td>\n",
       "      <td>0.27</td>\n",
       "      <td>1.69</td>\n",
       "      <td>3.80</td>\n",
       "      <td>1.11</td>\n",
       "      <td>4.00</td>\n",
       "      <td>1035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>12.22</td>\n",
       "      <td>1.29</td>\n",
       "      <td>1.94</td>\n",
       "      <td>19.0</td>\n",
       "      <td>92</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.04</td>\n",
       "      <td>0.39</td>\n",
       "      <td>2.08</td>\n",
       "      <td>2.70</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.02</td>\n",
       "      <td>312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>13.27</td>\n",
       "      <td>4.28</td>\n",
       "      <td>2.26</td>\n",
       "      <td>20.0</td>\n",
       "      <td>120</td>\n",
       "      <td>1.59</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.35</td>\n",
       "      <td>10.20</td>\n",
       "      <td>0.59</td>\n",
       "      <td>1.56</td>\n",
       "      <td>835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>13.16</td>\n",
       "      <td>3.57</td>\n",
       "      <td>2.15</td>\n",
       "      <td>21.0</td>\n",
       "      <td>102</td>\n",
       "      <td>1.50</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.30</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.60</td>\n",
       "      <td>1.68</td>\n",
       "      <td>830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>13.86</td>\n",
       "      <td>1.51</td>\n",
       "      <td>2.67</td>\n",
       "      <td>25.0</td>\n",
       "      <td>86</td>\n",
       "      <td>2.95</td>\n",
       "      <td>2.86</td>\n",
       "      <td>0.21</td>\n",
       "      <td>1.87</td>\n",
       "      <td>3.38</td>\n",
       "      <td>1.36</td>\n",
       "      <td>3.16</td>\n",
       "      <td>410</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Alcohol  Malic.acid   Ash   Acl   Mg  Phenols  Flavanoids  \\\n",
       "22     13.71        1.86  2.36  16.6  101     2.61        2.88   \n",
       "108    12.22        1.29  1.94  19.0   92     2.36        2.04   \n",
       "175    13.27        4.28  2.26  20.0  120     1.59        0.69   \n",
       "145    13.16        3.57  2.15  21.0  102     1.50        0.55   \n",
       "71     13.86        1.51  2.67  25.0   86     2.95        2.86   \n",
       "\n",
       "     Nonflavanoid.phenols  Proanth  Color.int   Hue    OD  Proline  \n",
       "22                   0.27     1.69       3.80  1.11  4.00     1035  \n",
       "108                  0.39     2.08       2.70  0.86  3.02      312  \n",
       "175                  0.43     1.35      10.20  0.59  1.56      835  \n",
       "145                  0.43     1.30       4.00  0.60  1.68      830  \n",
       "71                   0.21     1.87       3.38  1.36  3.16      410  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.4200991 , 0.29931113, 0.16101569, 0.2190171 , 0.17603501,\n",
       "       0.47478851, 0.71161006, 0.11691792, 0.27784677, 0.61910049,\n",
       "       0.55091963, 0.5514424 , 0.54385001])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import mutual_info_classif\n",
    "# determine the mutual information\n",
    "mutual_info = mutual_info_classif(X_train, y_train)\n",
    "mutual_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Flavanoids              0.711610\n",
       "Color.int               0.619100\n",
       "OD                      0.551442\n",
       "Hue                     0.550920\n",
       "Proline                 0.543850\n",
       "Phenols                 0.474789\n",
       "Alcohol                 0.420099\n",
       "Malic.acid              0.299311\n",
       "Proanth                 0.277847\n",
       "Acl                     0.219017\n",
       "Mg                      0.176035\n",
       "Ash                     0.161016\n",
       "Nonflavanoid.phenols    0.116918\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutual_info = pd.Series(mutual_info)\n",
    "mutual_info.index = X_train.columns\n",
    "mutual_info.sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIEAAAI0CAYAAACK4Tk9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAy1UlEQVR4nO3de7zldV0v/tcbRsJE1H6OWkIOFV7QUAnMW3lLj0aCqSmkp1KT6idpxzKpzvHW6UR6slNeOqLpsZt411FIUlMzu8igiCKRE5LCKUVTMS1uvs8fa23Z7Jk9s9l77fnOWt/n8/HYj9nf7/oO++XXmTVrvdbnUt0dAAAAABbbAUMHAAAAAGDzKYEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIzAlqF+8K1vfevetm3bUD8eAAAAYOGcd955X+jurbt7bLASaNu2bdmxY8dQPx4AAABg4VTVP632mOlgAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIbBk6wKxtO+2soSOsyaWnHz90BAAAAGBEjAQCAAAAGAElEAAAAMAIKIEAAAAARmBNJVBVPbyqLq6qnVV12m4e/52qOn/69Q9V9eWZJwUAAABg3fa6MHRVHZjkZUkemuSyJOdW1fbu/uTSNd39X5Zd//NJ7rkJWQEAAABYp7WMBLpXkp3dfUl3X53kzCQn7uH6k5O8bhbhAAAAAJiNtZRAt0/y2WXHl03P7aKq7pDkiCR/scrjp1TVjqraccUVV9zYrAAAAACs06wXhj4pyZu6+7rdPdjdZ3T3sd197NatW2f8owEAAABYzVpKoMuTHL7s+LDpud05KaaCAQAAAOx31lICnZvkyKo6oqoOyqTo2b7yoqq6c5JbJfmb2UYEAAAAYKP2WgJ197VJTk1yTpKLkryhuy+sqhdU1QnLLj0pyZnd3ZsTFQAAAID12usW8UnS3WcnOXvFueesOH7e7GIBAAAAMEuzXhgaAAAAgP2QEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMwJahA7D/23baWUNHWLNLTz9+6AgAAACwXzISCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFYUwlUVQ+vqouramdVnbbKNY+rqk9W1YVV9aezjQkAAADARmzZ2wVVdWCSlyV5aJLLkpxbVdu7+5PLrjkyya8kuV93f6mqbrNZgQEAAAC48dYyEuheSXZ29yXdfXWSM5OcuOKapyZ5WXd/KUm6+/OzjQkAAADARqylBLp9ks8uO75sem65Oya5Y1V9qKr+tqoevrv/UFWdUlU7qmrHFVdcsb7EAAAAANxos1oYekuSI5M8MMnJSV5ZVbdceVF3n9Hdx3b3sVu3bp3RjwYAAABgb9ZSAl2e5PBlx4dNzy13WZLt3X1Nd386yT9kUgoBAAAAsB9YSwl0bpIjq+qIqjooyUlJtq+45m2ZjAJKVd06k+lhl8wuJgAAAAAbsdcSqLuvTXJqknOSXJTkDd19YVW9oKpOmF52TpIvVtUnk7wvybO6+4ubFRoAAACAG2evW8QnSXefneTsFeees+z7TvLM6RcAAAAA+5lZLQwNAAAAwH5MCQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI7Bl6AAwVttOO2voCGt26enHDx0BAACADVICAQtFuQYAALB7poMBAAAAjIASCAAAAGAElEAAAAAAI2BNIAD2ylpLAAAw/4wEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARWFMJVFUPr6qLq2pnVZ22m8d/qqquqKrzp18/PfuoAAAAAKzXlr1dUFUHJnlZkocmuSzJuVW1vbs/ueLS13f3qZuQEQAAAIANWstIoHsl2dndl3T31UnOTHLi5sYCAAAAYJbWUgLdPslnlx1fNj230mOq6oKqelNVHb67/1BVnVJVO6pqxxVXXLGOuAAAAACsx6wWhn5Hkm3dfXSSdyd57e4u6u4zuvvY7j5269atM/rRAAAAAOzNWkqgy5MsH9lz2PTcN3X3F7v7qunhq5J832ziAQAAADALaymBzk1yZFUdUVUHJTkpyfblF1TVty87PCHJRbOLCAAAAMBG7XV3sO6+tqpOTXJOkgOTvLq7L6yqFyTZ0d3bkzy9qk5Icm2Sf03yU5uYGQAAAIAbaa8lUJJ099lJzl5x7jnLvv+VJL8y22gAAAAAzMqsFoYGAAAAYD+mBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI7Bl6AAAMFbbTjtr6Ahrdunpxw8dAQCADTISCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBHYMnQAAIBZ2nbaWUNHWLNLTz9+6AgAwIgYCQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIrKkEqqqHV9XFVbWzqk7bw3WPqaquqmNnFxEAAACAjdprCVRVByZ5WZJHJDkqyclVddRurrt5kmck+btZhwQAAABgY9YyEuheSXZ29yXdfXWSM5OcuJvrfj3JbyX5jxnmAwAAAGAG1lIC3T7JZ5cdXzY9901VdUySw7v7rD39h6rqlKraUVU7rrjiihsdFgAAAID12fDC0FV1QJIXJ/nFvV3b3Wd097HdfezWrVs3+qMBAAAAWKO1lECXJzl82fFh03NLbp7kbkneX1WXJrl3ku0WhwYAAADYf6ylBDo3yZFVdURVHZTkpCTblx7s7q909627e1t3b0vyt0lO6O4dm5IYAAAAgBttryVQd1+b5NQk5yS5KMkbuvvCqnpBVZ2w2QEBAAAA2Lgta7mou89OcvaKc89Z5doHbjwWAAAAALO04YWhAQAAANj/KYEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIrKkEqqqHV9XFVbWzqk7bzeM/W1Ufr6rzq+qvquqo2UcFAAAAYL32WgJV1YFJXpbkEUmOSnLybkqeP+3u7+3ueyR5YZIXzzooAAAAAOu3lpFA90qys7sv6e6rk5yZ5MTlF3T3lcsOb5akZxcRAAAAgI3asoZrbp/ks8uOL0vy/SsvqqqnJXlmkoOSPHgm6QAAAACYibWUQGvS3S9L8rKq+vEk/zXJT668pqpOSXJKknznd37nrH40AACbaNtpZw0dYc0uPf34oSMAwH5rLdPBLk9y+LLjw6bnVnNmkkft7oHuPqO7j+3uY7du3brmkAAAAABszFpKoHOTHFlVR1TVQUlOSrJ9+QVVdeSyw+OTfGp2EQEAAADYqL1OB+vua6vq1CTnJDkwyau7+8KqekGSHd29PcmpVfVDSa5J8qXsZioYAAAAAMNZ05pA3X12krNXnHvOsu+fMeNcAAAAAMzQWqaDAQAAADDnlEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBHYMnQAAAAYo22nnTV0hDW79PTjh44AwAwYCQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEdgydAAAAIBZ2XbaWUNHWLNLTz9+6AjAyBgJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAisqQSqqodX1cVVtbOqTtvN48+sqk9W1QVV9d6qusPsowIAAACwXnstgarqwCQvS/KIJEclObmqjlpx2UeTHNvdRyd5U5IXzjooAAAAAOu3lpFA90qys7sv6e6rk5yZ5MTlF3T3+7r769PDv01y2GxjAgAAALARaymBbp/ks8uOL5ueW81TkvzZ7h6oqlOqakdV7bjiiivWnhIAAACADZnpwtBV9cQkxyZ50e4e7+4zuvvY7j5269ats/zRAAAAAOzBljVcc3mSw5cdHzY9dwNV9UNJfi3JA7r7qtnEAwAAAGAW1jIS6NwkR1bVEVV1UJKTkmxffkFV3TPJK5Kc0N2fn31MAAAAADZiryVQd1+b5NQk5yS5KMkbuvvCqnpBVZ0wvexFSQ5J8saqOr+qtq/ynwMAAABgAGuZDpbuPjvJ2SvOPWfZ9z8041wAAAAAzNBMF4YGAAAAYP+kBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAEtgwdAAAAgP3bttPOGjrCml16+vFDR4D9lpFAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjMCWoQMAAADAGG077ayhI6zZpacfP3QEZsBIIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIAR2DJ0AAAAAIBZ2XbaWUNHWLNLTz9+n/48I4EAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOwphKoqh5eVRdX1c6qOm03j/9gVX2kqq6tqsfOPiYAAAAAG7HXEqiqDkzysiSPSHJUkpOr6qgVl30myU8l+dNZBwQAAABg47as4Zp7JdnZ3ZckSVWdmeTEJJ9cuqC7L50+9o1NyAgAAADABq1lOtjtk3x22fFl03M3WlWdUlU7qmrHFVdcsZ7/BAAAAADrsE8Xhu7uM7r72O4+duvWrfvyRwMAAACM2lpKoMuTHL7s+LDpOQAAAADmxFpKoHOTHFlVR1TVQUlOSrJ9c2MBAAAAMEt7LYG6+9okpyY5J8lFSd7Q3RdW1Quq6oQkqarjquqyJD+W5BVVdeFmhgYAAADgxlnL7mDp7rOTnL3i3HOWfX9uJtPEAAAAANgP7dOFoQEAAAAYhhIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMgBIIAAAAYASUQAAAAAAjoAQCAAAAGAElEAAAAMAIKIEAAAAARkAJBAAAADACSiAAAACAEVACAQAAAIyAEggAAABgBJRAAAAAACOgBAIAAAAYASUQAAAAwAgogQAAAABGQAkEAAAAMAJKIAAAAIARUAIBAAAAjIASCAAAAGAElEAAAAAAI6AEAgAAABgBJRAAAADACCiBAAAAAEZACQQAAAAwAkogAAAAgBFQAgEAAACMwJpKoKp6eFVdXFU7q+q03Tz+LVX1+unjf1dV22aeFAAAAIB122sJVFUHJnlZkkckOSrJyVV11IrLnpLkS939PUl+J8lvzTooAAAAAOu3lpFA90qys7sv6e6rk5yZ5MQV15yY5LXT79+U5CFVVbOLCQAAAMBGVHfv+YKqxyZ5eHf/9PT4Pyf5/u4+ddk1n5hec9n0+B+n13xhxX/rlCSnTA/vlOTiWf0P2WS3TvKFvV7FjeW+bg73dXO4r7Pnnm4O93VzuK+bw33dHO7r5nBfZ8893Rzu6+aYp/t6h+7eursHtuzLFN19RpIz9uXPnIWq2tHdxw6dY9G4r5vDfd0c7uvsuaebw33dHO7r5nBfN4f7ujnc19lzTzeH+7o5FuW+rmU62OVJDl92fNj03G6vqaotSW6R5IuzCAgAAADAxq2lBDo3yZFVdURVHZTkpCTbV1yzPclPTr9/bJK/6L3NMwMAAABgn9nrdLDuvraqTk1yTpIDk7y6uy+sqhck2dHd25P8QZI/qqqdSf41k6JokczdFLY54b5uDvd1c7ivs+eebg73dXO4r5vDfd0c7uvmcF9nzz3dHO7r5liI+7rXhaEBAAAAmH9rmQ4GAAAAwJxTAgEAAACMgBIIAAAAYASUQOwzVXXEWs4BAAAAs2dh6DWoqgOSHNLdVw6dZZ5V1Ue6+5gV587r7u8bKhPsTVXdP8mR3f2aqtqayXPBp4fONc/c09mrqu9Ocll3X1VVD0xydJI/7O4vD5kLlquqY/b0eHd/ZF9lgT3xZ3VzVdW37eb0V7v7mn0eBm6ERekFlECrqKo/TfKzSa5Lcm6SQ5P8bne/aNBgc6iq7pzkrklemORZyx46NMmzuvuugwRbAFX1oCQ/n+RO01MXJXlpd79/sFALpKqem+TYJHfq7jtW1XckeWN332/gaHPLPd0cVXV+Jvd1W5Kzk7w9yV27+4cHjDWXquolSVZ9cdTdT9+HcRZKVb1v+u3Bmfx5/ViSyqS03NHd9xkq2yKYlupPzeR5YMvS+e5+8lCZ5tWyP6u709394H0WZgFV1aVJDk/ypUyeA26Z5F+SfC7JU7v7vMHCzSl//zfPIvYCW/Z+yWgd1d1XVtUTkvxZktOSnJdkbv/PHtCdkvxIJk/wj1x2/quZPFmxDlV1fJKXJnlBkudn8o/oMUleXVWndvfZQ+ZbED+a5J5JPpIk3f1/q+rmw0aae+7p5vhGd19bVT+a5CXd/ZKq+ujQoebUjqEDLKruflCSVNVbkhzT3R+fHt8tyfMGjLYo3p7kg0nek8mbFdZp6c8qm+bdSd7U3eckSVU9LMljkrwmycuTfP+A2eaVv/+bZ+F6ASXQ6m5SVTdJ8qhMRlZcU1WGTa1Dd789ydur6j7d/TdD51kgz0ryqO7+2LJz51fVjiQvyWQ0ABtzdXf30t/9qrrZ0IEWgHu6Oa6pqpOT/GSuL9tvMmCeudXdr11+XFWHTM//2zCJFtKdlgqgJOnuT1TVXYYMtCC+tbufPXSIRVJVT0vyJ0tTa6vqVklO7u6XDxps/t27u7/5QXB3/3lV/c/u/pmq+pYhg80xf/83z8L1AhaGXt0rklya5GZJ/rKq7pBkruf+7Qd2VtWvVtUZVfXqpa+hQ82x260ogJIk3X1BktsOkGcRvaGqXpHkllX11Ew+XXnlwJnmnXu6OZ6U5D5JfqO7Pz1ddP+PBs4016rqbtPRVBcm+WRVnVdVpi/PxgVV9aqqeuD065VJLhg61AJ4Z1WZAjpbT12+tlp3fylGsc/CP1fVs6vqDtOvX07yuao6MMk3hg43p/z93zwL1wtYE+hGqKot3X3t0DnmVVX9dSbDFM/LsmGK3f3mwULNsT0tqm3B7dmpqocmeVgm0+3O6e53Dxxp7rmnzIPpv1m/1t3vmx4/MMn/6O77DplrEVTVwUl+LskPTk/9ZZLf7+7/GC7V/Kqqr2ayjlVl8iblqiTXTI+7uw8dMN5cq6qPJzm6p2+YpiXFBdaz3JiqunWS5ya5//TUhzJZ2uArSb6zu3cOlW3e+Ps/jHnvBZRAK1TVM/f0eHe/eF9lWTRVdX5332PoHIuiqr6cyQvnXR5Kcv/uvtW+TQTsa9M3KLv7h3zpxd/R+zjSwqiqj3X33fd2DlhcVfWiJHfIZCRAkvxMks909y8NlwrYFxa5F7Am0K6WFii9U5LjkmyfHj8yyYcHSbQ43llVP2zB4pk5cfrrTZMcmckbwZ1J/j3J/xwq1CJZ9ulKkhyUyRorX/OpyvpV1aOT/FaS22RSVPikamN+ZOgAC+ySqvpvuX5a3ROTXDJgnrlXVW/o7setVl4qLTemqt7b3Q/Z2zlulGcnOSWTnYGSybTF2w0XZ75V1fY9Pd7dJ+yrLIumqu6X5Pzu/lpVPTGTzWL+V3d/ZuBo82xhNy4xEmgVVfWXSY7v7q9Oj2+e5Kzu/sE9/05WM31DbZjijEwXKPuNJE9OsvQEf3iS/5PkV7v7moGiLaSqqkyKt3t392lD55lXVbUzySO7+6KhsyyaqrptJh9eJMmHu/vzQ+aZd9MFYJ+f66crfDDJ86ZrgrAOVfXt3f3P0/UUdtHd/7SvMy2C6fS6myX5iyQPzOT1VTLZxvhd3X3ngaIthKq6Z5IfT/K4TIrgN3f3S4dNNZ+q6ookn03yuiR/l+v/rCZJuvsDQ+RaBFV1QZK7Jzk6k/cCr0ryuO5+wJC52D8pgVZRVRdnMgf4qunxt2QyB/hOwyaDiar6nSSHJHnmsrLy0ExGAf17dz9jyHyLqqo+2t33HDrHvKqqD3X3/YbOsWiq6nGZbFX6/kxeVP9Akmd195uGzAW7M124/J+X1gCqqpsmuW13XzposDlVVc9I8gtJviPJ5bn+jfWVSV6psLjxquqOSU6efn0hyeuT/FJ377bAZG2mayo9NJP7enSSs5K8rrsvHDTYAqiqj3T3MVX1nCSXd/cfLJ0bOtu8q6rDMtl5een16weTPKO7Lxsu1cYogVZRVb+WSeP/1umpRyV5fXf/5mCh5lRV3bm7/76qdvsk1N0f2deZFkFVfSrJHXvFX+LpP7B/391HDpNscUynLi05IMmxSR7Q3fcZKNLcq6rfzWQo/dsyGRWYJOnutwyVaRFU1ceSPHRp9E9VbU3yHuvXrN/0TeAvJdmWZdPnu/vBQ2VaFFW1I8l9u/vq6fFBST7U3cft+XeyJ1X18939kqFzLIKq+kYmb/SesrRIcVVd0t3fNWyyxTH9gP3kTD7AeL6ycmOq6gNJ3pXJDIEfSPL5TKaHmWa7QVX17iR/mhtOD39Cdz90uFQbY02gVXT3b1TVn2XylyhJntTdHx0y0xx7ZibzqX97N491Ei+o16dXFkDTk9dVlXZ3Nh657PtrM9ke8sTdX8oaHZrk65nsDrakkyiBNuaAFdO/vphJccn6vTHJ/85kSP11e7mWG2fLUgGUJN199bQIYgO6+yVVdd/sWlz+4WCh5tejk5yU5H1V9a4kZ2bF1CXWZ1r+HJ9JAbQtye/l+g/dWb/HZzJt8Und/S9V9YOZTBNl47Z292uWHf+fqvqFocLMghJohao6tLuvrKpvy+QN36XLHvu27v7XobLNq+4+Zfrrg4bOsmA+WVU/sfLF3XQxuL8fKNNC6e4nDZ1h0binm+ZdVXVOJussJJMXgxbh35hru/v3hw6xoK6oqhO6e3uSVNWJmUy5YQOq6o+SfHeS83N9cdlJlEA3Une/LcnbqupmmXz48wtJblNVv5/krd395wPGm1tV9YdJ7pbJv0/P7+5PDBxpYUyLn/cl+fGq+uMkn07yv4ZNtTC+OH1/tfQa6+RMPmybW6aDrVBV7+zuH6mqT2fyD+fy1r8NA52tqrpdd//L0DnmUVXdPpPRE/+e5Lzp6WMz2S3sR7v78qGyzbuqekl2v+12kqS7n74P4yyEqvrl7n7havfWPd24qnpMls1X726frK7D9EOgJHl6JsPp35obTl30YdAGVdV3J/mTTNawqUwWiv2JpWk3rE9VXZTkqN2NEmbjpovF/1iSx9txbX2m0+y+Nj1c/ufUZjHrZP2qzTfdzOAlSe6TyZ/bv07y9HneeU0JxKCq6qzuPn7oHPOsqh6c5K7Tw09293uHzLMIquonlx0+P8lzlz/e3a/dt4nmX1U9srvfseLefpN7yv5ilQ+BlvgwaIaq6pAk6e5/GzrLIqiqN2byxuSfh84C7BvWr2I9lEB7UFUnJFnaEv793f3OIfPMs+lixe8xJYx5Yzcw5sF0EfPfSnKbTMoLn6qyX6uq4zP5AOPgpXPd/YLhEs2/6VSQeyT5cG44eu2EoTIBm6uqHpXJ+lX3y2Rh6DOTvKq7jxgy1yKZbrbx1Oy63tqTh8q0UdYEWkVVnZ7kuEyGKyfJM6rqvt39qwPGmlvTxYq/UVW36O6vDJ0HbgRN+QxU1Tuy5yl23qRszAuTPLK7Lxo6yKKoqpsk+bks+zAoySu6+5rBQi2IqvrfSb41yYMyWXj7sZkUF2zM84YOAOxb1q/aJ96eyWir92RBNoowEmgVVXVBknt09zemxwcm+aht9tavqt6e5J5J3p3r5wNbC4T9WlV9pLuPGTrHvKuqB+zp8e7+wL7Ksoiq6kPdfb+9X8laVdWrktwkydJUxf+c5Lru/unhUi2Gqrqgu49e9ushSf6su39gr78ZgD2yftVsVdX53X2PoXPMkpFAe3bLJEsLQN5iwByL4i2xDTRzoKq+mutHrXxrVV259FBMsVmX5SXPdCvoO04PLzayYiZ2VNXrk7wtN5wG4jl3/Y7r7rsvO/6LqvrYYGkWy79Pf/16VX1HJrusfPuAeRZCVd07k8VL75LkoCQHJvmaf7NgXLr7S0nOmH6xce+sqh/u7oXZdVUJtLrfTPLR6fzqymQ4+GnDRppv3f1ab/6YB91986EzLKqqemAmIysuzeS59fCq+snu/ssBYy2CQ5N8PcnDlp3rKN434rqq+u7u/sckqarvyoIMA98PvLOqbpnkRUk+ksmf1VcOmmgxvDSTtUHemMluoT+R619zAbA+z0jyq1V1dZKrswAfCpsOtgdV9e2ZrAuUJB+2lfnG7O7NXxJv/mBEquq8JD/e3RdPj++Y5HXd/X3DJoMbqqqHJHlNkksy+TfrDkme1N3vGzTYgqmqb0lysPUCN66qdnT3sUvT7KbnbG4AwA0YCbRnByT5Qib36Y5VdUeFxYb8dpKHrXzzl8SbPxiPmyw9ByRJd//DdAFeNmD6fPr7SW7b3XerqqOTnNDd/33gaHOru99bVUcmudP01MXdfdWefg833vSeXlVVt/Nh24Z9fTri+vyqemGSf87ktSwA61RVleQJSY7o7l+vqsOTfHt3z+2GBkYCraKqfivJ45NcmOQb09NtB5v1W/7J1J7OAYurql6TyZSaP56eekKSA+d5m839QVV9IMmzMtm96p7Tc5/o7rsNm2x+VdXTkvxJd395enyrJCd398sHDbagquqs7j5+6BzzrKrukORzmawH9F8yWc/y5d29c9BgAHNsutPaN5I8uLvvMn098Ofdfdxefut+Swm0iqq6OMnRPvWbnap6dSZ/gbz5g5GaTv14WpL7T099MJM3KZ5rN6Cqzu3u45ZP/VjE3Sz2pd3dP1Nr2N9ZexFgtpZ2Cl7xGutjKzaPmCumg63ukky2hvXGZHZ+LpM3f0tbwn8wiU9UYSSq6sAkH+vuOyd58dB5FswXquq7M93Vrqoem8lUENbvwKqqnn5aNv3ze9DAmRbCdBerC7v7q9PjQ5Pcpbv/bthk883C+wCb4prpa4Cl1wNbc/1MoblkJNAqqurNSe6e5L254Xa7T1/1NwGwR1X19iQ/392fGTrLIpnuXHVGkvsm+VKSTyd5YndfOmSueVZVL8pkMehXTE/9TJLPdvcvDpdqMVTVR5Mcs6xgOyDJju4+Zthk883C+wCzV1VPyGSZmGMyKdofm+S/dvcbBw22AUYCrW779IsNqqqPZ9qc7o41gWBUbpXkwqr6cJKvLZ203trGdPclSX6oqm6W5IClERZsyLMzKX5+bnr87iSvGi7OQvnmCKsk6e5vVJXXpBtn4X2AGevuP5mW7A/JZJTlo7r7ooFjbYiRQGy66UKFq+ruf9pXWYBhVdUDdne+uz+wr7MskulaS49Jsi3LPuDp7hcMlQlWU1VvSfL+THa0S5L/P8mDuvtRQ2VaBBbeB9gc0+lgt80NX2PN7ah2JdAqptvC/maSo5IcvHS+u79rsFALoKpum2RpJfUPd/fnh8wD7BtVdXCSn03yPUk+nuQPuvvaYVMtjqp6V5KvJDkvkzeBSZLu/u3BQs2pvYxe7XleCHJ/UVW3SfJ7SR6cyb1+b5Jf8JpgYyy8DzB7VfXzSZ6bye6L12UyGqjneTaLEmgVVfVXmfyf/TtJHpnkSZkMsX/OoMHmWFU9LsmLMvn0r5L8QJJndfebhswFbL6qen2SazJ5U/KIJP/U3c8YNtXisB387KwyerWSHJ7kV7r7h/dxJNir6afUF04X3gdgRqpqZ5Lv7+4vDp1lVsy/Xt1Nu/u9051B/inJ86ZzAZVA6/drSY5b+qRvurL6e5IogWDxHdXd35skVfUHST48cJ5F89dV9b3d/fGhg8y75VOUq+qeSX48yY9lstj2m4fKtQiq6pe7+4VV9ZLsZrSVzTfWr7uvq6qLq+o753mKAsB+6LOZjLZeGEqg1V013a3iU1V1apLLkxwycKZ5d8CKod5fTHLAUGGAfeqapW+6+9qqGjLLwlg2dWlLkidV1SWZ7Gg590OVhzLdUenk6dcXkrw+k5HTDxo02GJYWkhzx6ApFpeF9wFmpKqeOf32kiTvr6qzcsNdw188SLAZMB1sFVV1XCYvVm6Z5NeTHJrkRd39t0PmmmfT7XaPTvK66anHJ7mgu589XCpgX6iq63L9m5JKctMkX8/1ZcWhQ2WbZxben72q+kYm0xaf0t07p+cusSYg+zsL7wPMTlU9d0+Pd/fz91WWWVMCraKqjunujwydYxFU1fckuW13f6iqHp3rFyz8cpI/6e5/HCwcwByz4PbsVdWjkpyU5H5J3pXkzCSv6u4jhsy1CKrqHVl90W0jVtbJ8wDA5quqQzP54PKrQ2fZKCXQKqrqfUlul8l6Na/v7k8MHGluVdU7M1lM8+Mrzn9vkv/R3Y8cJhnAfLPg9uapqpslOTGTaWEPTvKHSd7a3X8+aLA5ttpIlSVGrKyP5wGAzVNVxyZ5TZKbT099JcmTu/u84VJtjBJoD6rqdkkel8m0pUMzKYP++7Cp5k9Vndvdx63y2MeXFosF4MZZ/hxaVVuSfLi7jxk41sKpqltlsjj047v7IUPngeU8DwBsnqq6IMnTuvuD0+P7J3n5PK+7aFHePejuf+nu38tkiO35sTPYet1yD4/ddF+FAFhAN1hwe8ggi6y7v9TdZyiAZqOqjqyqN1XVJ6vqkqWvoXPNMc8DAJvnuqUCKEm6+6+SzPVzrd3BVlFVd8lkBNBjMtnF6vVJfnHQUPNrR1U9tbtfufxkVf10krkdRgewH7h7VV05/b6S3HR6bMFt9mevSfLcJL+T5EFJnhQfTG6E5wGAzfOBqnpFJpsbdSYdwfur6pgkmcd1hE0HW0VV/U0mi0G+sbv/79B55llV3TbJW5NcnetLn2OTHJTkR7v7X4bKBgDsW1V1Xnd/34ppTOd19/cNnQ0AlpuuFbya7u4H77MwM2Ik0Cq6+z5DZ1gU3f25JPetqgcludv09Fnd/RcDxgIAhnFVVR2Q5FNVdWqSy5McMnAmANhFdz9o6AyzZiTQKqrqyCS/meSoJAcvne/u7xosFADAnKuq45JclMmagb+e5BZJXtjdfztkLgBYi6o6Zh6ngS1RAq2iqv4q189Xf2Sm89W72+LQAAAAMEJV9crufurQOdZLCbQK89UBAGanqrbv6fHuPmFfZQGAsbIm0OrMVwcAmJ37JPlsJjus/F0mu1cBwH5nafev1ZgOtoDMVwcAmJ2qOjDJQ5OcnOToJGcleV13XzhoMABYYdmuYAdnsrP1xzL58OLoJDvmeSMpJRAAAPtUVX1LJmXQi5I8v7tfOnAkANhFVb0lyXO7++PT47sleV53P3bYZOtnOtgKVfWOJKs2Y+arAwCsz7T8OT6TAmhbkt9L8tYhMwHAHtxpqQBKku7+RFXdZchAG2Uk0ApV9YA9Pd7dH9hXWQAAFkVV/WGSuyU5O8mZ3f2JgSMBwB5V1euSfC3JH09PPSHJId198nCpNkYJtEJVfWd3f2boHAAAi6SqvpHJC+nkhqOuK0l396H7PhUArK6qDk7yc0l+cHrqL5P8fnf/x3CpNkYJtEJVfaS7j5l+/+bufszQmQAAAAA2yppAu1q+Xel3DZYCAAAA2Oeq6g3d/biq+nh2s2Zwdx89QKyZUALtqlf5HgAAAFh8z5j++iODptgEpoOtUFXXZTJfvZLcNMnXlx6K+eoAAAAwGlV12yTHTQ8/3N2fHzLPRimBAAAAAFaoqscleVGS92cyMOQHkjyru980ZK6NUAIBAAAArFBVH0vy0KXRP1W1Ncl7uvvuwyZbvwOGDgAAAACwHzpgxfSvL2bOexQLQwMAAADs6l1VdU6S102PH5/k7AHzbJjpYAAAAAC7UVWPSXK/6eEHu/utQ+bZKCUQAAAAwAjM9Vw2AAAAgM1QVY+uqk9V1Veq6sqq+mpVXTl0ro0wEggAAABgharameSR3X3R0FlmxUggAAAAgF19bpEKoMRIIAAAAIBdVNXvJrldkrcluWrpfHe/ZahMG2WLeAAAAIBdHZrk60ketuxcJ5nbEshIIAAAAIARMBIIAAAAYIWqOjjJU5LcNcnBS+e7+8mDhdogC0MDAAAA7OqPMlkT6D8l+UCSw5J8ddBEG2Q6GAAAAMAKVfXR7r5nVV3Q3UdX1U2SfLC77z10tvUyEggAAABgV9dMf/1yVd0tyS2S3GbAPBtmTSAAAACAXZ1RVbdK8t+SbE9yyPT7uWU6GAAAAMAKVXVgd183dI5ZMh0MAAAAYFefrqozquohVVVDh5kFJRAAAADAru6c5D1Jnpbk0qp6aVXdf+BMG2I6GAAAAMAeTNcG+t0kT+juA4fOs15GAgEAAADsRlU9oKpenuS8JAcnedzAkTbESCAAAACAFarq0iQfTfKGJNu7+2vDJto4JRAAAADAClV1aHdfOXSOWVICAQAAAKxQVQcneUqSu2YyFSxJ0t1PHizUBlkTCAAAAGBXf5Tkdkn+U5IPJDksyVcHTbRBRgIBAAAArFBVH+3ue1bVBd19dFXdJMkHu/veQ2dbLyOBAAAAAHZ1zfTXL1fV3ZLcIsltBsyzYVuGDgAAAACwHzqjqm6V5L8l2Z7kkCTPGTbSxpgOBgAAADACRgIBAAAATFXVM/f0eHe/eF9lmTUlEAAAAMD1bj50gM2iBAIAAAC43rd297Or6se6+41Dh5klu4MBAAAAXO+Hq6qS/MrQQWbNSCAAAACA670ryZeSHFJVVy47X0m6uw8dJtbG2R0MAAAAYIWqent3nzh0jllSAgEAAACMgDWBAAAAAFaoqkdX1aeq6itVdWVVfXXF9LC5YyQQAAAAwApVtTPJI7v7oqGzzIqRQAAAAAC7+twiFUCJkUAAAAAAu6iq301yuyRvS3LV0vnufstQmTbKFvEAAAAAuzo0ydeTPGzZuU4ytyWQkUAAAAAAI2BNIAAAAIAVquqwqnprVX1++vXmqjps6FwboQQCAAAA2NVrkmxP8h3Tr3dMz80t08EAAAAAVqiq87v7Hns7N0+MBAIAAADY1Rer6olVdeD064lJvjh0qI0wEggAAABghaq6Q5KXJLlPJruC/XWSp3f3ZwYNtgFKIAAAAIAR2DJ0AAAAAID9RVU9Zw8Pd3f/+j4LM2NGAgEAAABMVdUv7ub0zZI8Jcn/192H7ONIM6MEAgAAANiNqrp5kmdkUgC9Iclvd/fnh021fqaDAQAAACxTVd+W5JlJnpDktUmO6e4vDZtq45RAAAAAAFNV9aIkj05yRpLv7e5/GzjSzJgOBgAAADBVVd9IclWSazPZGv6bD2WyMPShgwSbASUQAAAAwAgcMHQAAAAAADafEggAAABgBJRAAAAAACOgBAIAAAAYgf8H3CCnNGwvGAoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#let's plot the ordered mutual_info values per feature\n",
    "mutual_info.sort_values(ascending=False).plot.bar(figsize=(20, 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Flavanoids', 'Color.int', 'Hue', 'OD', 'Proline'], dtype='object')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#No we Will select the  top 5 important features\n",
    "sel_five_cols = SelectKBest(mutual_info_classif, k=5)\n",
    "sel_five_cols.fit(X_train, y_train)\n",
    "X_train.columns[sel_five_cols.get_support()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Difference Between Information Gain And Mutual Information\n",
    "I(X ; Y) = H(X) – H(X | Y)\n",
    "and\n",
    "IG(S, a) = H(S) – H(S | a)\n",
    "\n",
    "As such, mutual information is sometimes used as a synonym for \n",
    "information gain. Technically, they calculate the same quantity\n",
    "if applied to the same data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparion of Univariate And Mutual Information\n",
    "Comparison of F-test and mutual information\n",
    "https://scikit-learn.org/stable/auto_examples/feature_selection/plot_f_test_vs_mi.html#sphx-glr-auto-examples-feature-selection-plot-f-test-vs-mi-py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}