Loan Repayment/notebooks/Automated Loan Repayment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/featuretools.png)\n",
    "\n",
    "# Predicting Loan Repayment with Automated Feature Engineering in Featuretools\n",
    "\n",
    "Feature engineering is the process of creating new features (also called predictors or explanatory variables) out of an existing dataset. Traditionally, this process is done by hand using domain knowledge to build new features one at a time. In a previous notebook, we saw that feature engineering is crucial for a data science problem and how the manual approach is time-consuming, tedious, error-prone, and must be re-done for each problem. Automated feature engineering aims to aid the data scientist in this critical process by automatically creating hundreds or thousands of new features from a set of related tables in a fraction of the time as the manual approach. In this notebook, we will apply automated feature engineering to the Home Credit Default Risk loan dataset using [Featuretools, an open-source Python library](https://www.featuretools.com/) for automated feature engineering. \n",
    "\n",
    "This problem is a machine learning competition currently running on Kaggle where the objective is to predict if an applicant will default on a loan given comprehensive data on past loans and applicants. The data is spread across seven different tables making this an ideal problem for automated feature engineering: all of the data must be gathered into a single dataframe for training (and one for testing) with the aim of capturing as much usable information for the prediction problem as possible. As we will see, featuretools can efficiently carry out the tedious process of using all of these tables to make new features with only a few lines of code. Moreover, this code is generally applicable to any data science problem! \n",
    "\n",
    "The general idea of automated feature engineering is picture below:\n",
    "\n",
    "![](../../images/AutomatedFeatureEngineering.png)\n",
    "\n",
    "\n",
    "## Approach \n",
    "\n",
    "In this notebook, we will implement an automated feature engineering approach to the loan repayment problem. While Featuretools allows plenty of options for customization of the library to improve accuracy, we'll focus on a fairly high-level implementation. (Later notebooks, including Engine Life and Retail Spending show some of the additional features of the library).\n",
    "\n",
    "Our approach will be as follows with the background covered as we go:\n",
    "\n",
    "1. Read in the set of related data tables\n",
    "2. Create a featuretools `EntitySet` and add `entities` to it \n",
    "    * Identify correct variable types as required\n",
    "    * Identify indices in data\n",
    "3. Add relationships between `entities`\n",
    "4. Select feature primitives to use to create new features\n",
    "    * Use basic set of primitives\n",
    "    * Examine features that will be created\n",
    "5. Run Deep Feature Synthesis to generate thousands of new features\n",
    "\n",
    "\n",
    "## Problem and Dataset\n",
    "\n",
    "The [Home Credit Default Risk competition](https://www.kaggle.com/c/home-credit-default-risk) currently running on Kaggle is a supervised classification task where the objective is to predict whether or not an applicant for a loan (known as a client) will default on the loan. The data comprises socio-economic indicators for the clients, loan specific financial information, and comprehensive data on previous loans at Home Credit (the institution sponsoring the competition) and other credit agencies. The metric for this competition is Receiver Operating Characteristic Area Under the Curve (ROC AUC) with predictions made in terms of the probability of default. We can evaluate our submissions both through cross-validation on the training data (for which we have the labels) or by submitting our test predictions to Kaggle to see where we place on the public leaderboard (which is calculated with only 10% of the testing data). \n",
    "\n",
    "The Home Credit Default Risk dataset ([available for download here](https://www.kaggle.com/c/home-credit-default-risk/data)) consists of seven related tables of data:\n",
    "\n",
    "* application_train/application_test: the main training/testing data for each client at Home Credit. The information includes both socioeconomic indicators for the client and loan-specific characteristics. Each loan has its own row and is uniquely identified by the feature `SK_ID_CURR`. The training application data comes with the `TARGET` indicating 0: the loan was repaid or 1: the loan was not repaid. \n",
    "* bureau: data concerning client's previous credits from other financial institutions (not Home Credit). Each previous credit has its own row in bureau, but one client in the application data can have multiple previous credits. The previous credits are uniquely identified by the feature `SK_ID_BUREAU`.\n",
    "* bureau_balance: monthly balance data about the credits in bureau. Each row has information for one month about a previous credit and a single previous credit can have multiple rows. This is linked backed to the bureau loan data by `SK_ID_BUREAU` (not unique in this dataframe).\n",
    "* previous_application: previous applications for loans at Home Credit of clients who have loans in the application data. Each client in the application data can have multiple previous loans. Each previous application has one row in this dataframe and is uniquely identified by the feature `SK_ID_PREV`. \n",
    "* POS_CASH_BALANCE: monthly data about previous point of sale or cash loans from the previous loan data. Each row is one month of a previous point of sale or cash loan, and a single previous loan can have many rows. This is linked backed to the previous loan data by `SK_ID_PREV` (not unique in this dataframe).\n",
    "* credit_card_balance: monthly data about previous credit cards loans from the previous loan data. Each row is one month of a credit card balance, and a single credit card can have many rows. This is linked backed to the previous loan data by `SK_ID_PREV` (not unique in this dataframe).\n",
    "* installments_payment: payment history for previous loans at Home Credit. There is one row for every made payment and one row for every missed payment. This is linked backed to the previous loan data by `SK_ID_PREV` (not unique in this dataframe).\n",
    "\n",
    "The image below shows the seven tables and the variables linking them:\n",
    "\n",
    "![](../images/kaggle_home_credit/home_credit_data.png)\n",
    "\n",
    "The variables that tie the tables together will be important to understand when it comes to adding `relationships` between entities. __The only domain knowledge we need for a full Featuretools approach to the problem is the indexes of the tables and the relationships between the tables.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pandas and numpy for data manipulation\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# featuretools for automated feature engineering\n",
    "import featuretools as ft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in Data\n",
    "\n",
    "First we can read in the seven data tables. We also replace the anomalous values previously identified (we did the same process with manual feature engineering). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the datasets and replace the anomalous values\n",
    "app_train = pd.read_csv('../input/application_train.csv').replace({365243: np.nan})\n",
    "app_test = pd.read_csv('../input/application_test.csv').replace({365243: np.nan})\n",
    "bureau = pd.read_csv('../input/bureau.csv').replace({365243: np.nan})\n",
    "bureau_balance = pd.read_csv('../input/bureau_balance.csv').replace({365243: np.nan})\n",
    "cash = pd.read_csv('../input/POS_CASH_balance.csv').replace({365243: np.nan})\n",
    "credit = pd.read_csv('../input/credit_card_balance.csv').replace({365243: np.nan})\n",
    "previous = pd.read_csv('../input/previous_application.csv').replace({365243: np.nan})\n",
    "installments = pd.read_csv('../input/installments_payments.csv').replace({365243: np.nan})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will join together the training and testing datasets to make sure we build the same features for each set. Later, after the feature matrix is built, we can separate out the two sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_test['TARGET'] = np.nan\n",
    "\n",
    "# Join together training and testing\n",
    "app = app_train.append(app_test, ignore_index = True, sort = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several of the indexes are an incorrect data type (floats) so we need to make these all the same (integers) for adding relationships. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index in ['SK_ID_CURR', 'SK_ID_PREV', 'SK_ID_BUREAU']:\n",
    "    for dataset in [app, bureau, bureau_balance, cash, credit, previous, installments]:\n",
    "        if index in list(dataset.columns):\n",
    "            dataset[index] = dataset[index].fillna(0).astype(np.int64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Featuretools Basics\n",
    "\n",
    "[Featuretools](https://docs.featuretools.com/#minute-quick-start) is an open-source Python library for automatically creating features out of a set of related tables using a technique called [Deep Feature Synthesis](http://www.jmaxkanter.com/static/papers/DSAA_DSM_2015.pdf). Automated feature engineering, like many topics in machine learning, is a complex subject built upon a foundation of simpler ideas. By going through these ideas one at a time, we can build up our understanding of Featuretools which will later allow for us to get the most out of it.\n",
    "\n",
    "There are a few concepts that we will cover along the way:\n",
    "\n",
    "* [Entities and EntitySets](https://docs.featuretools.com/loading_data/using_entitysets.html): our tables and a data structure for keeping track of them all\n",
    "* [Relationships between tables](https://docs.featuretools.com/loading_data/using_entitysets.html#adding-a-relationship): how the tables can be related to one another\n",
    "* [Feature primitives](https://docs.featuretools.com/automated_feature_engineering/primitives.html): aggregations and transformations that are stacked to build features\n",
    "* [Deep feature synthesis](https://docs.featuretools.com/automated_feature_engineering/afe.html): the method that uses feature primitives to generate thousands of new features\n",
    "\n",
    "# Entities and Entitysets\n",
    "\n",
    "An entity is simply a table or in Pandas, a `dataframe`. The observations must be in the rows and the features in the columns. An entity in featuretools must have a unique index where none of the elements are duplicated.  Currently, only `app`, `bureau`, and `previous` have unique indices (`SK_ID_CURR`, `SK_ID_BUREAU`, and `SK_ID_PREV` respectively). For the other dataframes, when we create entities from them, we must pass in `make_index = True` and then specify the name of the index. \n",
    "\n",
    "Entities can also have time indices that represent when the information in the row became known. (There are not datetimes in any of the data, but there are relative times, given in months or days, that could be treated as time variables, although we will not use them as time in this notebook).\n",
    "\n",
    "An [EntitySet](https://docs.featuretools.com/loading_data/using_entitysets.html) is a collection of tables and the relationships between them. This can be thought of a data structure with its own methods and attributes. Using an EntitySet allows us to group together multiple tables and will make creating the features much simpler than keeping track of individual tables and relationships. __EntitySets and entities are abstractions that can be applied to any dataset because they do not depend on the underlying data.__\n",
    "\n",
    "First we'll make an empty entityset named clients to keep track of all the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entity set with id applications\n",
    "es = ft.EntitySet(id = 'clients')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable Types\n",
    "\n",
    "Featuretools will automatically infer the variable types. However, there may be some cases where we need to explicitly tell featuretools the variable type such as when a boolean variable is represented as an integer. Variable types in featuretools can be specified as a dictionary. \n",
    "\n",
    "We will first work with the `app` data to specify the proper variable types. To identify the `Boolean` variables that are recorded as numbers (1.0 or 0.0), we can iterate through the data and find any columns where there are only 2 unique values and the data type is numeric. We can also use the column definitions to find any other data types that should be identified, such as `Ordinal` variables. Identifying the correct variable types is important because Featuretools applies different operations to different data types (just as we do when manual feature engineering)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import featuretools.variable_types as vtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 32 Boolean variables in the application data.\n"
     ]
    }
   ],
   "source": [
    "app_types = {}\n",
    "\n",
    "# Handle the Boolean variables:\n",
    "for col in app:\n",
    "    if (app[col].nunique() == 2) and (app[col].dtype == float):\n",
    "        app_types[col] = vtypes.Boolean\n",
    "\n",
    "# Remove the `TARGET`\n",
    "del app_types['TARGET']\n",
    "\n",
    "print('There are {} Boolean variables in the application data.'.format(len(app_types)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ordinal variables\n",
    "app_types['REGION_RATING_CLIENT'] = vtypes.Ordinal\n",
    "app_types['REGION_RATING_CLIENT_W_CITY'] = vtypes.Ordinal\n",
    "app_types['HOUR_APPR_PROCESS_START'] = vtypes.Ordinal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `previous` table is the only other `entity` that has features which should be recorded as Boolean. Correctly identifying the type of column will prevent featuretools from making irrelevant features such as the mean or max of a `Boolean`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2 Boolean variables in the previous data.\n"
     ]
    }
   ],
   "source": [
    "previous_types = {}\n",
    "\n",
    "# Handle the Boolean variables:\n",
    "for col in previous:\n",
    "    if (previous[col].nunique() == 2) and (previous[col].dtype == float):\n",
    "        previous_types[col] = vtypes.Boolean\n",
    "\n",
    "print('There are {} Boolean variables in the previous data.'.format(len(previous_types)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to identifying Boolean variables, we want to make sure featuretools does not create nonsense features such as statistical aggregations (mean, max, etc.) of ids. The `credit`, `cash`, and `installments` data all have the `SK_ID_CURR` variable. However, we do not actually need this variable in these dataframes because we link them to `app` through the `previous` dataframe with the `SK_ID_PREV` variable. \n",
    "\n",
    "We don't want to make features from `SK_ID_CURR` since it is an arbitrary id and should have no predictive power. \n",
    "Our options to handle these variables is either to tell featuretools to ignore them, or to drop the features before including them in the entityset. We will take the latter approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "installments = installments.drop(columns = ['SK_ID_CURR'])\n",
    "credit = credit.drop(columns = ['SK_ID_CURR'])\n",
    "cash = cash.drop(columns = ['SK_ID_CURR'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Entities\n",
    "\n",
    "Now we define each entity, or table of data, and add it to the `EntitySet`. We need to pass in an index if the table has one or `make_index = True` if not. In the cases where we need to make an index, we must supply a name for the index. We also need to pass in the dictionary of variable types if there are any specific variables we should identify. The following code adds all seven tables to the `EntitySet`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entities with a unique index\n",
    "es = es.entity_from_dataframe(entity_id = 'app', dataframe = app, index = 'SK_ID_CURR',\n",
    "                              variable_types = app_types)\n",
    "\n",
    "es = es.entity_from_dataframe(entity_id = 'bureau', dataframe = bureau, index = 'SK_ID_BUREAU')\n",
    "\n",
    "es = es.entity_from_dataframe(entity_id = 'previous', dataframe = previous, index = 'SK_ID_PREV',\n",
    "                              variable_types = previous_types)\n",
    "\n",
    "# Entities that do not have a unique index\n",
    "es = es.entity_from_dataframe(entity_id = 'bureau_balance', dataframe = bureau_balance, \n",
    "                              make_index = True, index = 'bureaubalance_index')\n",
    "\n",
    "es = es.entity_from_dataframe(entity_id = 'cash', dataframe = cash, \n",
    "                              make_index = True, index = 'cash_index')\n",
    "\n",
    "es = es.entity_from_dataframe(entity_id = 'installments', dataframe = installments,\n",
    "                              make_index = True, index = 'installments_index')\n",
    "\n",
    "es = es.entity_from_dataframe(entity_id = 'credit', dataframe = credit,\n",
    "                              make_index = True, index = 'credit_index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: clients\n",
       "  Entities:\n",
       "    app [Rows: 356255, Columns: 122]\n",
       "    bureau [Rows: 1716428, Columns: 17]\n",
       "    previous [Rows: 1670214, Columns: 37]\n",
       "    bureau_balance [Rows: 27299925, Columns: 4]\n",
       "    cash [Rows: 10001358, Columns: 8]\n",
       "    installments [Rows: 13605401, Columns: 8]\n",
       "    credit [Rows: 3840312, Columns: 23]\n",
       "  Relationships:\n",
       "    No relationships"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display entityset so far\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `EntitySet` allows us to group together all of our tables as one data structure. This is much easier than manipulating the tables one at a time (as we have to do in manual feature engineering)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Relationships\n",
    "\n",
    "Relationships are a fundamental concept not only in featuretools, but in any relational database. The most common type of relationship is one-to-many. The best way to think of a one-to-many relationship is with the analogy of parent-to-child. A parent is a single individual, but can have mutliple children. In the context of tables, a parent table will have one row (observation) for every individual while a child table can have many observations for each parent.  In a _parent table_, each individual has a single row and is uniquely identified by an index (also called a key). Each individual in the parent table can have multiple rows in the _child table_. Things get a little more complicated because children tables can have children of their own, making these grandchildren of the original parent. \n",
    "\n",
    "As an example of a parent-to-child relationship, the `app` dataframe has one row for each client (identified by `SK_ID_CURR`) while the `bureau` dataframe has multiple previous loans for each client. Therefore, the `bureau` dataframe is the child of the `app` dataframe. The `bureau` dataframe in turn is the parent of `bureau_balance` because each loan has one row in `bureau` (identified by `SK_ID_BUREAU`) but multiple monthly records in `bureau_balance`. When we do manual feature engineering, keeping track of all these relationships is a massive time investment (and a potential source of error), but we can add these relationships to our `EntitySet` and let featuretools worry about keeping the tables straight!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parent: app, Parent Variable of bureau: SK_ID_CURR\n",
      "\n",
      "    SK_ID_CURR  TARGET  TOTALAREA_MODE WALLSMATERIAL_MODE\n",
      "0      100002     1.0          0.0149       Stone, brick\n",
      "1      100003     0.0          0.0714              Block\n",
      "2      100004     0.0             NaN                NaN\n",
      "3      100006     0.0             NaN                NaN\n",
      "4      100007     0.0             NaN                NaN\n",
      "\n",
      "Child: bureau, Child Variable of app: SK_ID_CURR\n",
      "\n",
      "    SK_ID_CURR  SK_ID_BUREAU CREDIT_ACTIVE CREDIT_CURRENCY  DAYS_CREDIT\n",
      "0      215354       5714462        Closed      currency 1       -497.0\n",
      "1      215354       5714463        Active      currency 1       -208.0\n",
      "2      215354       5714464        Active      currency 1       -203.0\n",
      "3      215354       5714465        Active      currency 1       -203.0\n",
      "4      215354       5714466        Active      currency 1       -629.0\n"
     ]
    }
   ],
   "source": [
    "print('Parent: app, Parent Variable of bureau: SK_ID_CURR\\n\\n', app.iloc[:, 111:115].head())\n",
    "print('\\nChild: bureau, Child Variable of app: SK_ID_CURR\\n\\n', bureau.iloc[:, :5].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SK_ID_CURR` 215354 has one row in the parent table and multiple rows in the child. \n",
    "\n",
    "Two tables are linked via a shared variable. The `app` and `bureau` dataframe are linked by the `SK_ID_CURR` variable while the `bureau` and `bureau_balance` dataframes are linked with the `SK_ID_BUREAU`. The linking variable is called the `parent` variable in the parent table and the `child` variable in the child table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parent: bureau, Parent Variable of bureau_balance: SK_ID_BUREAU\n",
      "\n",
      "    SK_ID_CURR  SK_ID_BUREAU CREDIT_ACTIVE CREDIT_CURRENCY  DAYS_CREDIT\n",
      "0      215354       5714462        Closed      currency 1       -497.0\n",
      "1      215354       5714463        Active      currency 1       -208.0\n",
      "2      215354       5714464        Active      currency 1       -203.0\n",
      "3      215354       5714465        Active      currency 1       -203.0\n",
      "4      215354       5714466        Active      currency 1       -629.0\n",
      "\n",
      "Child: bureau_balance, Child Variable of bureau: SK_ID_BUREAU\n",
      "\n",
      "    SK_ID_BUREAU  MONTHS_BALANCE STATUS\n",
      "0       5715448               0      C\n",
      "1       5715448              -1      C\n",
      "2       5715448              -2      C\n",
      "3       5715448              -3      C\n",
      "4       5715448              -4      C\n"
     ]
    }
   ],
   "source": [
    "print('Parent: bureau, Parent Variable of bureau_balance: SK_ID_BUREAU\\n\\n', bureau.iloc[:, :5].head())\n",
    "print('\\nChild: bureau_balance, Child Variable of bureau: SK_ID_BUREAU\\n\\n', bureau_balance.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditionally, we use the relationships between parents and children to aggregate data by grouping together all the children for a single parent and calculating statistics. For example, we might group together all the loans for a single client and calculate the average loan amount. This is straightforward, but can grow extremely tedious when we want to make hundreds of these features. Doing so one at a time is extremely inefficient especially because we end up re-writing much of the code over and over again and this code cannot be used for any different problem! \n",
    "\n",
    "Things get even worse when we have to aggregate the grandchildren because we have to use two steps: first aggregate at the parent level, and then at the grandparent level. Soon we will see that featuretools can do this work automatically for us, generating thousands of features from __all__ of the data tables. When we did this manually it took about 15 minutes per feature (as we saw in the manual feature engineering notebook) so featuretools potentially saves us hundreds of hours.\n",
    "\n",
    "### Adding Relationships\n",
    "\n",
    "Defining the relationships is straightforward using the diagram for the data tables. For each relationship, we need to first specify the parent variable and then the child variable. Altogether, there are a total of 6 relationships between the tables (counting the training and testing relationships as one). Below we specify these relationships and then add them to the EntitySet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Relationship between app_train and bureau\n",
    "r_app_bureau = ft.Relationship(es['app']['SK_ID_CURR'], es['bureau']['SK_ID_CURR'])\n",
    "\n",
    "# Relationship between bureau and bureau balance\n",
    "r_bureau_balance = ft.Relationship(es['bureau']['SK_ID_BUREAU'], es['bureau_balance']['SK_ID_BUREAU'])\n",
    "\n",
    "# Relationship between current app and previous apps\n",
    "r_app_previous = ft.Relationship(es['app']['SK_ID_CURR'], es['previous']['SK_ID_CURR'])\n",
    "\n",
    "# Relationships between previous apps and cash, installments, and credit\n",
    "r_previous_cash = ft.Relationship(es['previous']['SK_ID_PREV'], es['cash']['SK_ID_PREV'])\n",
    "r_previous_installments = ft.Relationship(es['previous']['SK_ID_PREV'], es['installments']['SK_ID_PREV'])\n",
    "r_previous_credit = ft.Relationship(es['previous']['SK_ID_PREV'], es['credit']['SK_ID_PREV'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: clients\n",
       "  Entities:\n",
       "    app [Rows: 356255, Columns: 122]\n",
       "    bureau [Rows: 1716428, Columns: 17]\n",
       "    previous [Rows: 1670214, Columns: 37]\n",
       "    bureau_balance [Rows: 27299925, Columns: 4]\n",
       "    cash [Rows: 10001358, Columns: 8]\n",
       "    installments [Rows: 13605401, Columns: 8]\n",
       "    credit [Rows: 3840312, Columns: 23]\n",
       "  Relationships:\n",
       "    bureau.SK_ID_CURR -> app.SK_ID_CURR\n",
       "    bureau_balance.SK_ID_BUREAU -> bureau.SK_ID_BUREAU\n",
       "    previous.SK_ID_CURR -> app.SK_ID_CURR\n",
       "    cash.SK_ID_PREV -> previous.SK_ID_PREV\n",
       "    installments.SK_ID_PREV -> previous.SK_ID_PREV\n",
       "    credit.SK_ID_PREV -> previous.SK_ID_PREV"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add in the defined relationships\n",
    "es = es.add_relationships([r_app_bureau, r_bureau_balance, r_app_previous,\n",
    "                           r_previous_cash, r_previous_installments, r_previous_credit])\n",
    "# Print out the EntitySet\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we can see the benefits of using an `EntitySet` that is able to track all of the relationships for us. This allows us to work at a higher level of abstraction, thinking about the entire dataset rather than each individual table, greatly increasing our efficiency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Slightly advanced note__: we need to be careful to not create a [diamond graph](https://en.wikipedia.org/wiki/Diamond_graph) where there are multiple paths from a parent to a child. If we directly link `app` and `cash` via `SK_ID_CURR`; `previous` and `cash` via `SK_ID_PREV`; and `app` and `previous` via `SK_ID_CURR`, then we have created two paths from `app` to `cash`. This results in ambiguity, so the approach we have to take instead is to link `app` to `cash` through `previous`. We establish a relationship between `previous` (the parent) and `cash` (the child) using `SK_ID_PREV`. Then we establish a relationship between `app` (the parent) and `previous` (now the child) using `SK_ID_CURR`. Then featuretools will be able to create features on `app` derived from both `previous` and `cash` by stacking multiple primitives. \n",
    "\n",
    "If this doesn't make too much sense, then just remember to only include one path from a parent to any descendents. For example, link a grandparent to a grandchild through the parent instead of directly through a shared variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All entities in the entity can be linked through these relationships. In theory this allows us to calculate features for any of the entities, but in practice, we will only calculate features for the `app` dataframe since that will be used for training/testing. The end outcome will be a dataframe that has one row for each client in `app` with thousands of features for each individual. \n",
    "\n",
    "We are almost to the point where we can start creating thousands of features but we still have a few foundational topics to understand. The next building block to cover is feature primitives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Primitives\n",
    "\n",
    "A [feature primitive](https://docs.featuretools.com/automated_feature_engineering/primitives.html) is an operation applied to a table or a set of tables to create a feature. These represent simple calculations, many of which we already use in manual feature engineering, that can be stacked on top of each other to create complex deep features. Feature primitives fall into two categories:\n",
    "\n",
    "* __Aggregation__: function that groups together children for each parent and calculates a statistic such as mean, min, max, or standard deviation across the children. An example is the maximum previous loan amount for each client. An aggregation covers multiple tables using relationships between tables.\n",
    "* __Transformation__: an operation applied to one or more columns in a single table. An example would be taking the absolute value of a column, or finding the difference between two columns in one table.\n",
    "\n",
    "A list of the available features primitives in featuretools can be viewed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>median</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Finds the median value of any feature with well-ordered values.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>all</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Test if all values are 'True'.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sum</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Sums elements of a numeric or boolean feature.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>avg_time_between</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Computes the average time between consecutive events.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>any</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Test if any value is 'True'.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>n_most_common</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Finds the N most common elements in a categorical feature.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>num_true</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Finds the number of 'True' values in a boolean.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>count</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Counts the number of non null values.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>percent_true</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Finds the percent of 'True' values in a boolean feature.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>last</td>\n",
       "      <td>aggregation</td>\n",
       "      <td>Returns the last value.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name         type  \\\n",
       "0            median  aggregation   \n",
       "1               all  aggregation   \n",
       "2               sum  aggregation   \n",
       "3  avg_time_between  aggregation   \n",
       "4               any  aggregation   \n",
       "5     n_most_common  aggregation   \n",
       "6          num_true  aggregation   \n",
       "7             count  aggregation   \n",
       "8      percent_true  aggregation   \n",
       "9              last  aggregation   \n",
       "\n",
       "                                                       description  \n",
       "0  Finds the median value of any feature with well-ordered values.  \n",
       "1                                   Test if all values are 'True'.  \n",
       "2                   Sums elements of a numeric or boolean feature.  \n",
       "3            Computes the average time between consecutive events.  \n",
       "4                                     Test if any value is 'True'.  \n",
       "5       Finds the N most common elements in a categorical feature.  \n",
       "6                  Finds the number of 'True' values in a boolean.  \n",
       "7                            Counts the number of non null values.  \n",
       "8         Finds the percent of 'True' values in a boolean feature.  \n",
       "9                                          Returns the last value.  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List the primitives in a dataframe\n",
    "primitives = ft.list_primitives()\n",
    "pd.options.display.max_colwidth = 100\n",
    "\n",
    "primitives[primitives['type'] == 'aggregation'].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>multiply</td>\n",
       "      <td>transform</td>\n",
       "      <td>Creates a transform feature that multplies two features.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>time_since</td>\n",
       "      <td>transform</td>\n",
       "      <td>Calculates time since the cutoff time.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>minutes</td>\n",
       "      <td>transform</td>\n",
       "      <td>Transform a Timedelta feature into the number of minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>absolute</td>\n",
       "      <td>transform</td>\n",
       "      <td>Absolute value of base feature.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>add</td>\n",
       "      <td>transform</td>\n",
       "      <td>Creates a transform feature that adds two features.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>characters</td>\n",
       "      <td>transform</td>\n",
       "      <td>Return the characters in a given string.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>seconds</td>\n",
       "      <td>transform</td>\n",
       "      <td>Transform a Timedelta feature into the number of seconds.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>cum_min</td>\n",
       "      <td>transform</td>\n",
       "      <td>Calculates the min of previous values of an instance for each value in a time-dependent entity.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>longitude</td>\n",
       "      <td>transform</td>\n",
       "      <td>Returns the second value on the tuple base feature.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>weekend</td>\n",
       "      <td>transform</td>\n",
       "      <td>Transform Datetime feature into the boolean of Weekend.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          name       type  \\\n",
       "19    multiply  transform   \n",
       "20  time_since  transform   \n",
       "21     minutes  transform   \n",
       "22    absolute  transform   \n",
       "23         add  transform   \n",
       "24  characters  transform   \n",
       "25     seconds  transform   \n",
       "26     cum_min  transform   \n",
       "27   longitude  transform   \n",
       "28     weekend  transform   \n",
       "\n",
       "                                                                                        description  \n",
       "19                                         Creates a transform feature that multplies two features.  \n",
       "20                                                           Calculates time since the cutoff time.  \n",
       "21                                        Transform a Timedelta feature into the number of minutes.  \n",
       "22                                                                  Absolute value of base feature.  \n",
       "23                                              Creates a transform feature that adds two features.  \n",
       "24                                                         Return the characters in a given string.  \n",
       "25                                        Transform a Timedelta feature into the number of seconds.  \n",
       "26  Calculates the min of previous values of an instance for each value in a time-dependent entity.  \n",
       "27                                              Returns the second value on the tuple base feature.  \n",
       "28                                          Transform Datetime feature into the boolean of Weekend.  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "primitives[primitives['type'] == 'transform'].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Feature Synthesis\n",
    "\n",
    "[Deep Feature Synthesis (DFS)](https://www.featurelabs.com/blog/deep-feature-synthesis/) is the method Featuretools uses to make new features. DFS stacks feature primitives to form features with a \"depth\" equal to the number of primitives. For example, if we take the maximum value of a client's previous loans (say `MAX(previous.loan_amount)`), that is a \"deep feature\" with a depth of 1. To create a feature with a depth of two, we could stack primitives by taking the maximum value of a client's average monthly payments per previous loan (such as `MAX(previous(MEAN(installments.payment)))`). In manual feature engineering, this would require two separate groupings and aggregations and took more than 15 minutes to write the code per feature. \n",
    "\n",
    "Deep Feature Synthesis is an extremely powerful method that allows us to overcome our human limitations on time and creativity by building features that we would never be able to think of on our own (or would not have the patience to implement). Furthermore, DFS is applicable to any dataset with only very minor changes in syntax. In feature engineering, we generally apply the same functions to multiple datasets, but when we do it by hand, we have to re-write the code because it is problem-specific. Featuretools code can be applied to any dataset because it is written at a higher level of abstraction.\n",
    "\n",
    "The [original paper on automated feature engineering using Deep Feature Synthesis](https://dai.lids.mit.edu/wp-content/uploads/2017/10/DSAA_DSM_2015.pdf) is worth a read if you want to understand the concepts at a deeper level.\n",
    "\n",
    "To perform DFS in featuretools, we use the `dfs`  function passing it an `entityset`, the `target_entity` (where we want to make the features), the `agg_primitives` to use, the `trans_primitives` to use, the `max_depth` of the features, and a number of other arguments depending on our use case. There are also options for multi-processing with `njobs` and the information that is printed out with `verbose`. \n",
    "\n",
    "One other important argument is __`features_only`__. If we set this to `True`, `dfs` will only make the feature names and not calculate the actual values of the features (called the feature matrix). This is useful when we want to inspect the feature that will be created and we can also save the features to use with a different dataset (for example when we have training and testing data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Feature Synthesis with Default Primitives\n",
    "\n",
    "Without using any domain knowledge we can make thousands of features by using the default primitives in featuretools. This first call will use the default aggregation and transformation primitives,  a max depth of 2, and calculate primitives for the `app` entity. We will only generate the features themselves (the names and not the values) which we can save and inspect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 1575 features\n"
     ]
    }
   ],
   "source": [
    "# Default primitives from featuretools\n",
    "default_agg_primitives =  [\"sum\", \"std\", \"max\", \"skew\", \"min\", \"mean\", \"count\", \"percent_true\", \"num_unique\", \"mode\"]\n",
    "default_trans_primitives =  [\"day\", \"year\", \"month\", \"weekday\", \"haversine\", \"numwords\", \"characters\"]\n",
    "\n",
    "# DFS with specified primitives\n",
    "feature_names = ft.dfs(entityset = es, target_entity = 'app',\n",
    "                       trans_primitives = default_trans_primitives,\n",
    "                       agg_primitives=default_agg_primitives, \n",
    "                       where_primitives = [], seed_features = [],\n",
    "                       max_depth = 2, n_jobs = -1, verbose = 1,\n",
    "                       features_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even a basic call to deep feature synthesis gives us over 1500 features to work with. Granted, not all of these will be important, but this still represents hundreds of hours that we saved. Moreover, `dfs` might be able to find important features that we would never have thought of in the first place. \n",
    "\n",
    "We can look at the some of the feature names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Feature: MIN(bureau.MAX(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MIN(bureau.SKEW(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MIN(bureau.MEAN(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MIN(bureau.COUNT(bureau_balance))>,\n",
       " <Feature: MIN(bureau.NUM_UNIQUE(bureau_balance.STATUS))>,\n",
       " <Feature: MEAN(bureau.SUM(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.STD(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.MAX(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.SKEW(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.MIN(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.MEAN(bureau_balance.MONTHS_BALANCE))>,\n",
       " <Feature: MEAN(bureau.COUNT(bureau_balance))>,\n",
       " <Feature: MEAN(bureau.NUM_UNIQUE(bureau_balance.STATUS))>,\n",
       " <Feature: NUM_UNIQUE(bureau.MODE(bureau_balance.STATUS))>,\n",
       " <Feature: MODE(bureau.MODE(bureau_balance.STATUS))>]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names[-15:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how featuretools stacks multiple primitives on top of each other. This one of the ideas behind Deep Feature Synthesis and automated feature engineering. Rather than having to do these groupings and aggregations by ourselves, Featuretools is able to handle it all using the framework (`entities`, `relationships`, and `primitives`) that we provide. We can also use Featuretools to expand on our domain knowledge. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building on Top of Domain Features\n",
    "\n",
    "Featuretools will automatically build thousands of features for us, but that does not mean we can't use our own knowledge to improve the predictive performance. Featuretools is able to augment our domain knowledge by stacking additional features on top of our domain knowledge based features. We identified and created numerous useful features in the manual feature engineering notebook, based on our own knowledge and that of thousands of data scientists working on this problem on Kaggle. Rather than getting only one domain knowledge feature, we can effectively get dozens or even hundreds. __Here we'll explain the options for using domain knowledge, but we'll stick with the simple implementation of Featuretools for comparison purposes.__\n",
    "\n",
    "For more information on any of these topics, see the [documentation](https://docs.featuretools.com/guides/tuning_dfs.html) or the other notebooks in this repository. \n",
    "\n",
    "### Seed Features \n",
    "\n",
    "Seed features are domain features that we make in the data that Featuretools is then able to build on top of. For example, we saw that the rate of a loan is an important feature because a higher rate loan is likely more risky. In Featuretools, we can encode the loan rate (both for the current loan and for previous loans) as a seed feature and Featuretools will build additional explanatory variables on this domain knowledge wherever possible. \n",
    "\n",
    "### Interesting Values\n",
    "\n",
    "Interesting values have a similar idea to seed features except they allow us to make conditional features. For example, we might want to find for each client the mean amount of previous loans that have been closed and the mean amount of previous loans that are still active. By specifying interesting values in `bureau` on the `CREDIT_ACTIVE` variable we can have Featuretools do exactly that! Carrying this out by hand would be extremely tedious and present numerous opportunities for errors.\n",
    "\n",
    "### Custom Primitives\n",
    "\n",
    "If we aren't satisfied with the primitives available to use in Featuretools, we can write our own functions to transform or aggregate the data. This is one of the most powerful capabilities in featuretools because it allows us to make very specific operations that can then be applied to multiple datasets. \n",
    "\n",
    "__In this notebook we concentrate on a basic implementation of Featuretools, but keep in mind these capabilities are available for optimizing the library and using domain knowledge!__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selecting Primitives\n",
    "\n",
    "For our actual set of features, we will use a select group of primitives rather than just the defaults. This will generate over 1800 features to use for modeling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify primitives\n",
    "agg_primitives =  [\"sum\", \"max\", \"min\", \"mean\", \"count\", \"percent_true\", \"num_unique\", \"mode\"]\n",
    "trans_primitives = ['percentile', 'and']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 1820 features\n"
     ]
    }
   ],
   "source": [
    "# Deep feature synthesis \n",
    "feature_names = ft.dfs(entityset=es, target_entity='app',\n",
    "                       agg_primitives = agg_primitives,\n",
    "                       trans_primitives = trans_primitives,\n",
    "                       n_jobs = -1, verbose = 1,\n",
    "                       features_only = True,\n",
    "                       max_depth = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft.save_features(feature_names, '../input/features.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we save the features, we can then use them with `calculate_feature_matrix`. This is useful when we want to apply the same features across datasets (such as if we have separate trainig/testing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Full Deep Feature Synthesis\n",
    "\n",
    "If we are content with the features that will be built, we can run deep feature synthesis and create the feature matrix. The following call runs the full deep feature synthesis. This might take a long time depending on your machine. Featuretools does allow for parallel processing, but each core must be able to handle the entire entityset. \n",
    "\n",
    "__An actual run of this code was completed using Dask which can be seen in the [Featuretools on Dask notebook](https://github.com/Featuretools/Automated-Manual-Comparison/blob/master/Loan%20Repayment/notebooks/Featuretools%20on%20Dask.ipynb).__ The Dask code takes under 2 hours to run and is a great example of how we can use parallel processing to use our resouces in the most efficient manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total size of entityset: 11.62682 gb.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print('Total size of entityset: {:.5f} gb.'.format(sys.getsizeof(es) / 1e9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of cpus detected: 8.\n",
      "Total size of system memory: 17.17987 gb.\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "\n",
    "print('Total number of cpus detected: {}.'.format(psutil.cpu_count()))\n",
    "print('Total size of system memory: {:.5f} gb.'.format(psutil.virtual_memory().total / 1e9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feature_matrix, feature_names = ft.dfs(entityset=es, target_entity='app',\n",
    "#                                        agg_primitives = agg_primitives,\n",
    "#                                        trans_primitives = trans_primitives,\n",
    "#                                        seed_features = seed_features,\n",
    "#                                         where_features = where_features,\n",
    "#                                        n_jobs = 1, verbose = 1, features_only = False,\n",
    "#                                        max_depth = 2, chunk_size = 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feature_matrix.reset_index(inplace = True)\n",
    "# feature_matrix.to_csv('../input/feature_matrix.csv', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download the feature matrix, head to https://www.kaggle.com/willkoehrsen/home-credit-default-risk-feature-tools and select the `feature_matrix_article.csv`. There are several other versions of automatically engineered feature matrices available there as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "In this notebook, we saw how to implement automated feature engineering for a data science problem. __Automated feature engineering allows us to create thousands of new features from a set of related data tables, significantly increasing our efficiency as data scientists.__ Moreover, we can still use domain knowledge in our features and even augment our domain knowledge by building on top of our own hand-built features. The main takeaways are:\n",
    "\n",
    "* Automated feature engineering took 1 hour to implement compared to 10 hours for manual feature engineering\n",
    "* Automated feature engineering built thousands of features in a few lines of code compared to dozens of lines of code per feature for manual engineering.\n",
    "* Overall, performance of the automated features are comparable or better than those of the manual features (see the Results notebook)\n",
    "\n",
    "The benefits of automated feature engineering are significant and will considerably help us in our role as data scientists. It won't alleviate the need for data scientists, but rather will make us more efficient and build better predictive pipelines in less time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "After creating a full set of features, we can apply feature selection and then proceed with modeling. To optimize the model for the features, we use random search for 100 iterations over a grid of hyperparamters. To see how to use Dask to run Featuretools in parallel, refer to the Featuretools Implementation with Dask notebook. For feature selection refer to the Feature Selection notebook. Final results are presented in the Results notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}