Strata_For_Participants.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=RuntimeWarning) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import time\n",
    "from scipy import stats\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recap: basic operations in Jupyter notebook, types of cells, executing cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data in a data frame using pandas, take a look at them, check the size of the data set, rename columns to something easier to type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at data properties divided by type to figure out some differences between LAEs and OIIs. Change settings to visualize all the columns in a data frame. Eliminate outliers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform pandas data frame into a numpy array that can be fed to sklearn methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A quick way to see what variables are important"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a simple linear model and use the cofficients of different variables to track variables' importance. Useful when there are many and we want to get rid of some of them, or to just build understanding of the model. Note that this doesn't properly inform one of which variables are redundant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First steps with models: Decision Tree Classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees are nice because they can be interpreted easily. For example, this is a decision tree showing how scientists might decide whether a newly found planet has a good chance to harbor life:\n",
    "\n",
    "Figure from [here](http://www.machinelearningtutorial.net/2017/01/17/decisiontree/).\n",
    "\n",
    "<br><div style=\"text-align: center \">  <b> IS ANYBODY OUT THERE?</b></div>\n",
    "\n",
    "<img src=\"Strata_images/exoplanets.svg\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees work by deciding where to split the data set using values of different features, and where to stop.\n",
    "\n",
    "Mathematically, a good decision tree is one that maximizes the information gain (e.g. the increase in accuracy) at every \"split\".\n",
    "\n",
    "<b> Pros </b> Easy to interpret, fast.\n",
    "\n",
    "<b> Cons </b> Prone to overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's get coding!\n",
    "\n",
    "-  Import model, fit using k-fold (k = 5) cross validation, establish benchmark performance.\n",
    "\n",
    "-  Consider the metric and its potential fallbacks by comparing to \"dummy\" estimator;\n",
    "\n",
    "-  Calculate and plot the confusion matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another easy to interpret algorithm is KNN (K nearest neighbors). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: center \">  <b> LET'S FIND SOME NEIGHBORS!</b></div>\n",
    "<table><tr>\n",
    "<td> <img src=\"Strata_images/KNN_1.png\" width=\"350\"/> </td>\n",
    "<td>  <img src=\"Strata_images/KNN_2.png\" width=\"350\"/> </td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's get coding!\n",
    "\n",
    "Import model, fit using k-fold (k = 5) cross validation, establish benchmark performance, play with basic parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary and 10-minute break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Advanced Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machines (classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Support Vector Machines are a long-term staple of machine learning. Parameter tuning is very important in SVMs, and it's the curse and blessing of this algorithm.\n",
    "\n",
    "<b>Pros: </b> Accurate, Powerful\n",
    "\n",
    "<b>Cons: </b>  SLOW, need standardization\n",
    "\n",
    "It's usually a good idea to do parameter optimization on a (representative) selection of your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVMs in a nutshell"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a classification problem such as this one, SVMs attempts to find the ideal boundary to separate the two classes.\n",
    "\n",
    "<img src=\"Strata_images/SVM_1.png\" width=\"300\"/>\n",
    "\n",
    "This looks easy, but even in this simple cases of completely separable variables, there are many possible choices, with different resulting boundaries.\n",
    "\n",
    "<img src=\"Strata_images/SVM_2.png\" width=\"300\"/> \n",
    "\n",
    "SVM's strategy is to 1. Maximize the separation between classes, called the <b> margin </b> and 2. Use slack variables to attribute a \"penalty\" to misclassifications (soft margin).\n",
    "\n",
    "<img src=\"Strata_images/SVM_3.png\" width=\"300\"/>\n",
    "\n",
    "In the more general case of non-linearly-separable variables, SVMs attempt to map the original feature space (for us a 4D space) to a higher dimensionality space, where instances are more separable. The set of functions used for the mapping is called <b>kernel</b>. \n",
    "<br>\n",
    "<br>\n",
    "<img src=\"Strata_images/SVM_4.png\" width=\"500\"/>\n",
    "\n",
    "\n",
    "The most important parameters of an SVM are:\n",
    "\n",
    "- The type of kernel (linear, polynomial, or Gaussian, \"rbf\" in sklearn);\n",
    "\n",
    "<img src=\"Strata_images/SVM_5.png\" width=\"400\"/>\n",
    "\n",
    "- Gamma, the \"wiggliness\" of the boundary (small gammas = more linear);\n",
    "\n",
    "<img src=\"Strata_images/SVM_6.png\" width=\"300\"/>\n",
    "\n",
    "- C, the soft margin parameter (smaller C values assign a smaller penalty to misclassifications near the boundary, and generates a wider margin).\n",
    "\n",
    "<img src=\"Strata_images/SVM_7.png\" width=\"300\"/>\n",
    "\n",
    "All the figures in this section are from [here](https://www.ncbi.nlm.nih.gov/pubmed/20221922) and [here](https://www.cs.utexas.edu/~mooney/cs391L/slides/svm.ppt)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's get coding! \n",
    "\n",
    "-  Import model;\n",
    "\n",
    "-  Establish benchmark performance for 5 fold cross validation;\n",
    "\n",
    "-  Visualize and briefly describe the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> TASKS (10-15 mins) </b>\n",
    "\n",
    "-  Play with different parameters, such as type of kernel (for time scaling reasons, use only poly and rbf), soft margin C, and gamma, to see if you can beat the benchmark performance above. Tip 1: Trying 2-3 values per parameter will be sufficient for now, especially if your machine is taking long. Tip 2: Use low values of gamma (< 1.0) to reduce fitting time.\n",
    "\n",
    "-  Now do the same thing, but using precision as your scoring method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Coding Solution.\n",
    "\n",
    "Introduce Grid Search CV (params, cv, scoring, verbose, n_jobs) as a method to optimize various parameters simultaneously; use timings to get an idea of the speed of various methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM Summary\n",
    "\n",
    "-  After parameter optimization, SVM's performance improves a bit over the baseline.\n",
    "\n",
    "-  We learned how to optimize parameters with Grid Search.\n",
    "\n",
    "-  The best model depends A LOT on the metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensemble methods: 1. Random Forest Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest Classifiers are combinations of decision trees. The \"random\" part refers to the fact that different trees in the forest are created using random splits of the data, and random subsets of the features. This randomization process makes the algorithm more robust against overfitting, compared to single trees.\n",
    "\n",
    "<b> Pros: </b> Fast (parallel), robust, insensitive to data range.\n",
    "\n",
    "<b> Cons: </b> Fast but not fastest when compared to other ensemble methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forests have many adjustable parameters. One can tune the parameters of each tree, and the way they are combined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"Strata_images/DT1.png\" width=\"700\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tree Parameters\n",
    "\n",
    "The figure above shows an example of possible split. The parameters associated to that are:\n",
    "\n",
    "-  The minimum number of instances in a leaf node;\n",
    "\n",
    "-  The minimum number of instances required in a split node;\n",
    "\n",
    "- The maximum depth of tree.\n",
    "\n",
    "They all deal with reducing overfitting by avoiding to go \"too deep\" in each tree; it makes sense to change two out of three.\n",
    "\n",
    "Additional parameters are:\n",
    "\n",
    "-  The criterion chosen to decide whether a split is \"worth it\", expressed in terms of information gain;\n",
    "\n",
    "-  The number of features that are used in building trees.\n",
    "\n",
    "#### Forest Parameters\n",
    "\n",
    "In Random Forests, the predictions generated by all the trees are simply averaged to produce the final results. The number of trees in the forest can be adjusted, with the general understanding that more trees are better, but at some point performance will plateau, so one can find the trade-off between having more trees and lower runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's get coding! \n",
    "\n",
    "-  Import model;\n",
    "\n",
    "-  Establish benchmark performance for 5 fold cross validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> TASKS (10-15 minutes) </b> \n",
    "\n",
    "-  Use the get_params() method to find out the names and signatures of different parameters, and their default values.\n",
    "\n",
    "-  Play with different values of the number of trees (estimators, using values between 5 and 50), maximum depth of tree (usually around 3-8), the minimum amount of instances in a split (2-10), and the maximum number of features (you can decide this one!) allowed in builiding individual trees to see if you can beat the benchmark performance above.\n",
    "\n",
    "-  Now do the same thing, but using recall as your scoring method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RF Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensemble methods 2: Gradient Boosting Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient Boosting models are another ensemble method where different decision trees are combined together.\n",
    "\n",
    "Unlike Random Forests, the model is built by <b> adding individual trees in a sequential fashion, </b>\n",
    "but choosing which trees we add to the model in a way that minimizes the current loss function. The \"Gradient\" part refers to the fact that we try to move along the gradient of the objective function (by calculating its numerical derivative) as we add more trees.\n",
    "\n",
    "The parameters depend on the particular implementation.\n",
    "\n",
    "In the sklearn formulation, the parameters of each tree are essentially the same we saw above; additionally we have the \"learning_rate\" parameter, which dictates how much each tree contribute to the final estimator, and the \"subsample\" parameters, which allows one to use a < 1.0 fraction of samples.\n",
    "\n",
    "I liked this blog post about parameter tuning for GBMs:\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2016/02/complete-guide-parameter-tuning-gradient-boosting-gbm-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We'll do the usual import and benchmarking:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> TASKS (10 minutes) </b>\n",
    "\n",
    "-  Use the get_params() method to find out the names and signatures of different parameters, and their default values.\n",
    "\n",
    "-  Play with different values of the number of trees (estimators: 5, 10, 20), max depth of tree (2-8), learning rate (0.1-0.5), and the maximum number of features allowed to see how much you can improve the benchmark performance above.\n",
    "\n",
    "-  Compare the timings to Random Forests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note about xgboost (vs sklearn's GBM)\n",
    "\n",
    "Sometimes knowns as \"regularized\" GBM, more robust to overfitting.\n",
    "\n",
    "Has more flexibility in defining weak learners, and objective function.\n",
    "\n",
    "Reputation of being very fast.\n",
    "\n",
    "From the same author as the one above:\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary so far"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subtleties in parameter optimization: See additional notebook\n",
    "\n",
    "-  Use cv_results to look at gradients along algorithms and build understanding;\n",
    "\n",
    "-  Push the edges of your parameter grid search; \n",
    "\n",
    "-  Do nested cross validation to optimize parameters in order to avoid leakage between the parameter optimization and the cross validation procedure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips for advanced optimization (know your data).\n",
    "\n",
    "Flip data so less common class becomes the positive one and check performance (in particular, recall). Introduce the \"class weight\" parameter for unbalanced data sets where we are interested in the \"uncommon\" class; define and use ad-hoc metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The class weight parameter\n",
    "\n",
    "In SVMs, C, the soft margin parameter, can take different values according to class. <b> This is helpful for imbalanced data sets, where we are interested in the less common objects. </b> This parameter is available for other estimators too!\n",
    "\n",
    "<img src=\"Strata_images/SVM_8.png\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### My advice: Define your own evaluation metric \n",
    "\n",
    "This is an example of what we did for this paper (Leung, VA et al 2016), where x0 = 1 - precision and x1 = 1 - recall.\n",
    "\n",
    "<img src=\"Strata_images/Formula_Leung.jpg\" width=\"300\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to do that in code?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Additional Content\n",
    "\n",
    "-  Notebook 1: Nested Cross Validation (the proper way to optimize parameters), Grid Search best practices, Randomized Grid Search.\n",
    "\n",
    "-  Notebook 2: Diagnostic Tools (Learning Curves, Bias/Variance tradeoff, Feature Importance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i in range(cm.shape[0]):\n",
    "        for j in range(cm.shape[1]):\n",
    "            plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}