HW4-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Python for Research Homework: Week 4, Case Study 3\n",
    "\n",
    "Homophily is a property of networks.  Homophily occurs when nodes that are neighbors in a network also share a characteristic more often than nodes that are not network neighbors.  In this case study, we will investigate homophily of several characteristics of individuals connected in social networks in rural India."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Network homophily occurs when nodes that share an edge share a characteristic more often than nodes that do not share an edge. In this case study, we will investigate homophily of several characteristics of individuals connected in social networks in rural India.\n",
    "\n",
    "In this exercise, we will calculate the chance homophily for an arbitrary characteristic. Homophily is the proportion of edges in the network whose constituent nodes share that characteristic. How much homophily do we expect by chance? If characteristics are distributed completely randomly, the probability that two nodes $x$ and $y$ share characteristic $a$ is the probability both nodes have characteristic $a$, which is the frequency of $a$ squared. The total probability that nodes $x$ and $y$ share their characteristic is therefore the sum of the frequency of each characteristic in the network. \n",
    "\n",
    "#### Instructions \n",
    "- Create a function that takes a dictionary `chars` with personal IDs as keys and characteristics as values; it should return a dictionary with characteristics as keys and the frequency of their occurrence as values.\n",
    "- Create a function `chance_homophily(chars)` that takes a dictionary `chars` defined as above and computes the chance homophily (homophily due to chance alone) for that characteristic.\n",
    "- A sample of three peoples' favorite colors is given in `favorite_colors`. Use your function to compute the chance homophily in this group, and store as `color_homophily`.\n",
    "- Print `color_homophily`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5555555555555556\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "\n",
    "def frequency(chars):\n",
    "    frequencies = dict(Counter(chars.values()))\n",
    "    sum_frequencies = sum(frequencies.values())\n",
    "    for key in frequencies:\n",
    "        frequencies[key] /= sum_frequencies\n",
    "    return frequencies\n",
    "    \n",
    "    \n",
    "def chance_homophily(chars):\n",
    "    frequencies = frequency(chars)\n",
    "    return np.sum(np.square(list(frequencies.values())))\n",
    "    \n",
    "\n",
    "\n",
    "favorite_colors = {\n",
    "    \"ankit\":  \"red\",\n",
    "    \"xiaoyu\": \"blue\",\n",
    "    \"mary\":   \"blue\"\n",
    "}\n",
    "\n",
    "color_homophily = chance_homophily(favorite_colors)\n",
    "print(color_homophily)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'ankit': 'red', 'xiaoyu': 'blue', 'mary': 'blue'})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars_counts_dict = frequency(favorite_colors)\n",
    "chars_counts_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "chars_counts = np.array(list(chars_counts_dict.values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['red', 'blue', 'blue'], dtype='<U4')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "In the remaining exercises, we will calculate actual homophily in these village and compare the obtained values to those obtained by chance. In this exercise, we subset the data into individual villages and store them.\n",
    "\n",
    "#### Instructions \n",
    "\n",
    "- `individual_characteristics.dta` contains several characteristics for each individual in the dataset such as age, religion, and caste. Use the `pandas` library to read in and store these characteristics as a dataframe called `df`.\n",
    "- Store separate datasets for individuals belonging to Villages 1 and 2 as `df1` and `df2`, respectively.\n",
    "- Note that some attributes may be missing for some individuals. In this case study, we will ignore rows of data where some column information is missing.\n",
    "- Use the head method to display the first few entries of `df1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>village</th>\n",
       "      <th>adjmatrix_key</th>\n",
       "      <th>pid</th>\n",
       "      <th>hhid</th>\n",
       "      <th>resp_id</th>\n",
       "      <th>resp_gend</th>\n",
       "      <th>resp_status</th>\n",
       "      <th>age</th>\n",
       "      <th>religion</th>\n",
       "      <th>caste</th>\n",
       "      <th>...</th>\n",
       "      <th>privategovt</th>\n",
       "      <th>work_outside</th>\n",
       "      <th>work_outside_freq</th>\n",
       "      <th>shgparticipate</th>\n",
       "      <th>shg_no</th>\n",
       "      <th>savings</th>\n",
       "      <th>savings_no</th>\n",
       "      <th>electioncard</th>\n",
       "      <th>rationcard</th>\n",
       "      <th>rationcard_colour</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>100201</td>\n",
       "      <td>1002</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Head of Household</td>\n",
       "      <td>38</td>\n",
       "      <td>HINDUISM</td>\n",
       "      <td>OBC</td>\n",
       "      <td>...</td>\n",
       "      <td>PRIVATE BUSINESS</td>\n",
       "      <td>Yes</td>\n",
       "      <td>0.0</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>GREEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>100202</td>\n",
       "      <td>1002</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Spouse of Head of Household</td>\n",
       "      <td>27</td>\n",
       "      <td>HINDUISM</td>\n",
       "      <td>OBC</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>GREEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "      <td>100601</td>\n",
       "      <td>1006</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Head of Household</td>\n",
       "      <td>29</td>\n",
       "      <td>HINDUISM</td>\n",
       "      <td>OBC</td>\n",
       "      <td>...</td>\n",
       "      <td>OTHER LAND</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>GREEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>100602</td>\n",
       "      <td>1006</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Spouse of Head of Household</td>\n",
       "      <td>24</td>\n",
       "      <td>HINDUISM</td>\n",
       "      <td>OBC</td>\n",
       "      <td>...</td>\n",
       "      <td>PRIVATE BUSINESS</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>100701</td>\n",
       "      <td>1007</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Head of Household</td>\n",
       "      <td>58</td>\n",
       "      <td>HINDUISM</td>\n",
       "      <td>OBC</td>\n",
       "      <td>...</td>\n",
       "      <td>OTHER LAND</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>GREEN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 48 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   village  adjmatrix_key     pid  hhid  resp_id  resp_gend  \\\n",
       "0        1              5  100201  1002        1          1   \n",
       "1        1              6  100202  1002        2          2   \n",
       "2        1             23  100601  1006        1          1   \n",
       "3        1             24  100602  1006        2          2   \n",
       "4        1             27  100701  1007        1          1   \n",
       "\n",
       "                   resp_status  age  religion caste  ...       privategovt  \\\n",
       "0            Head of Household   38  HINDUISM   OBC  ...  PRIVATE BUSINESS   \n",
       "1  Spouse of Head of Household   27  HINDUISM   OBC  ...               NaN   \n",
       "2            Head of Household   29  HINDUISM   OBC  ...        OTHER LAND   \n",
       "3  Spouse of Head of Household   24  HINDUISM   OBC  ...  PRIVATE BUSINESS   \n",
       "4            Head of Household   58  HINDUISM   OBC  ...        OTHER LAND   \n",
       "\n",
       "  work_outside work_outside_freq shgparticipate shg_no savings savings_no  \\\n",
       "0          Yes               0.0             No    NaN      No        NaN   \n",
       "1          NaN               NaN             No    NaN      No        NaN   \n",
       "2           No               NaN             No    NaN      No        NaN   \n",
       "3           No               NaN            Yes    1.0     Yes        1.0   \n",
       "4           No               NaN             No    NaN      No        NaN   \n",
       "\n",
       "  electioncard rationcard rationcard_colour  \n",
       "0          Yes        Yes             GREEN  \n",
       "1          Yes        Yes             GREEN  \n",
       "2          Yes        Yes             GREEN  \n",
       "3          Yes         No               NaN  \n",
       "4          Yes        Yes             GREEN  \n",
       "\n",
       "[5 rows x 48 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df  = pd.read_csv(\"https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@individual_characteristics.csv\", low_memory=False, index_col=0)\n",
    "df1 = df[df.village == 1]\n",
    "df2 = df[df.village == 2]\n",
    "\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 \n",
    "\n",
    "In this exercise, we define a few dictionaries that enable us to look up the sex, caste, and religion of members of each village by personal ID. For Villages 1 and 2, their personal IDs are stored as `pid`.\n",
    "\n",
    "#### Instructions \n",
    "- Define dictionaries with personal IDs as keys and a given covariate for that individual as values. Complete this for the sex, caste, and religion covariates, for Villages 1 and 2.\n",
    "- For Village 1, store these dictionaries into variables named `sex1`, `caste1`, and `religion1`.\n",
    "- For Village 2, store these dictionaries into variables named `sex2`, `caste2`, and `religion2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sex1      = dict(zip(df1.pid, df1.resp_gend))\n",
    "caste1    = dict(zip(df1.pid, df1.caste))\n",
    "religion1 = dict(zip(df1.pid, df1.religion))\n",
    "\n",
    "# Continue for df2 as well.\n",
    "sex2      = dict(zip(df2.pid, df2.resp_gend))\n",
    "caste2    = dict(zip(df2.pid, df2.caste))\n",
    "religion2 = dict(zip(df2.pid, df2.religion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OBC'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "caste2[202802]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "In this exercise, we will print the chance homophily of several characteristics of Villages 1 and 2. \n",
    "\n",
    "#### Instructions \n",
    "-  Use `chance_homophily` to compute the chance homophily for sex, caste, and religion In Villages 1 and 2. Is the chance homophily for any attribute very high for either village?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5027299861680701\n",
      "0.5005945303210464\n",
      "0.6741488509791551\n",
      "0.425368244800893\n",
      "0.9804896988521925\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "# Enter your code here.\n",
    "print(chance_homophily(sex1))\n",
    "print(chance_homophily(sex2))\n",
    "\n",
    "print(chance_homophily(caste1))\n",
    "print(chance_homophily(caste2))\n",
    "\n",
    "print(chance_homophily(religion1))\n",
    "print(chance_homophily(religion2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5\n",
    "\n",
    "In this exercise, we will create a function that computes the observed homophily given a village and characteristic.\n",
    "\n",
    "#### Instructions \n",
    "- Complete the function `homophily()`, which takes a network `G`, a dictionary of node characteristics `chars`, and node IDs `IDs`. For each node pair, determine whether a tie exists between them, as well as whether they share a characteristic. The total count of these is `num_same_ties` and `num_ties`, respectively, and their ratio is the homophily of chars in `G`. Complete the function by choosing where to increment `num_same_ties` and `num_ties`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def homophily(G, chars, IDs):\n",
    "    \"\"\"\n",
    "    Given a network G, a dict of characteristics chars for node IDs,\n",
    "    and dict of node IDs for each node in the network,\n",
    "    find the homophily of the network.\n",
    "    \"\"\"\n",
    "    num_same_ties = 0\n",
    "    num_ties = 0\n",
    "    for n1, n2 in G.edges():\n",
    "        if IDs[n1] in chars and IDs[n2] in chars:\n",
    "            if G.has_edge(n1, n2):\n",
    "                num_ties += 1\n",
    "                \n",
    "                if chars[IDs[n1]] == chars[IDs[n2]]:\n",
    "                    num_same_ties += 1\n",
    "                    \n",
    "    return (num_same_ties / num_ties)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6\n",
    "\n",
    "In this exercise, we will obtain the personal IDs for Villages 1 and 2. These will be used in the next exercise to calculate homophily for these villages.\n",
    "\n",
    "#### Instructions \n",
    "- In this dataset, each individual has a personal ID, or PID, stored in `key_vilno_1.csv` and `key_vilno_2.csv` for villages 1 and 2, respectively. `data_filepath1` and `data_filepath2` contain the URLs to the datasets used in this exercise. Use `pd.read_csv` to read in and store `key_vilno_1.csv` and `key_vilno_2.csv` as `pid1` and `pid2` respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_filepath1 = \"https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@key_vilno_1.csv\"\n",
    "data_filepath2 = \"https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@key_vilno_2.csv\"\n",
    "\n",
    "pid1 = pd.read_csv(data_filepath1)\n",
    "pid2 = pd.read_csv(data_filepath2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0       100\n",
       "0             102205\n",
       "Name: 100, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pid1.iloc[100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 7\n",
    "\n",
    "In this exercise, we will compute the homophily of several network characteristics for Villages 1 and 2 and compare them to homophily due to chance alone. The networks for these villages have been stored as networkx graph objects `G1` and `G2`.\n",
    "\n",
    "#### Instructions \n",
    "\n",
    "- Use your `homophily()` function to compute the observed homophily for sex, caste, and religion in Villages 1 and 2. Print all six values.\n",
    "- Use the `chance_homophily()` to compare these values to chance homophily. Are these values higher or lower than that expected by chance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Village 1 observed proportion of same sex: 0.5908629441624366\n",
      "Village 1 observed proportion of same caste: 0.7959390862944162\n",
      "Village 1 observed proportion of same religion: 0.9908629441624366\n",
      "Village 2 observed proportion of same sex: 0.5658073270013568\n",
      "Village 2 observed proportion of same caste: 0.8276797829036635\n",
      "Village 2 observed proportion of same religion: 1.0\n",
      "Village 1 chance of same sex: 0.5027299861680701\n",
      "Village 1 chance of same caste: 0.6741488509791551\n",
      "Village 1 chance of same religion: 0.9804896988521925\n",
      "Village 2 chance of same sex: 0.5005945303210464\n",
      "Village 2 chance of same caste: 0.425368244800893\n",
      "Village 2 chance of same religion: 1.0\n"
     ]
    }
   ],
   "source": [
    "import networkx as nx\n",
    "\n",
    "A1 = np.array(pd.read_csv(\"https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@adj_allVillageRelationships_vilno1.csv\", index_col=0))\n",
    "A2 = np.array(pd.read_csv(\"https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@adj_allVillageRelationships_vilno2.csv\", index_col=0))\n",
    "G1 = nx.to_networkx_graph(A1)\n",
    "G2 = nx.to_networkx_graph(A2)\n",
    "\n",
    "# pid1 = np.array(pd.read_csv(data_filepath + \"key_vilno_1.csv\", dtype=int, header = None)[0])\n",
    "# pid2 = np.array(pd.read_csv(data_filepath + \"key_vilno_2.csv\", dtype=int, header = None)[0])\n",
    "\n",
    "# Enter your code here!\n",
    "pid1 = np.array(pd.read_csv(data_filepath1, skiprows=1, dtype =int, header = None)[1]) # skiprows=1\n",
    "pid2 = np.array(pd.read_csv(data_filepath2, skiprows=1, dtype =int, header = None)[1])\n",
    "\n",
    "print(\"Village 1 observed proportion of same sex:\", homophily(G1, sex1, pid1))\n",
    "print(\"Village 1 observed proportion of same caste:\", homophily(G1, caste1, pid1))\n",
    "print(\"Village 1 observed proportion of same religion:\", homophily(G1, religion1, pid1))\n",
    "\n",
    "print(\"Village 2 observed proportion of same sex:\", homophily(G2, sex2, pid2))\n",
    "print(\"Village 2 observed proportion of same caste:\", homophily(G2, caste2, pid2))\n",
    "print(\"Village 2 observed proportion of same religion:\", homophily(G2, religion2, pid2))\n",
    "\n",
    "print(\"Village 1 chance of same sex:\", chance_homophily(sex1))\n",
    "print(\"Village 1 chance of same caste:\", chance_homophily(caste1))\n",
    "print(\"Village 1 chance of same religion:\", chance_homophily(religion1))\n",
    "\n",
    "print(\"Village 2 chance of same sex:\", chance_homophily(sex2))\n",
    "print(\"Village 2 chance of same caste:\", chance_homophily(caste2))\n",
    "print(\"Village 2 chance of same religion:\", chance_homophily(religion2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}