Linkage and Blocking Tuning Tool.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linkage and Blocking Evaluation and Tuning Tool\n",
    "\n",
    "The purpose of this tool is to look at the various options for implementing blocking in [anonlink](https://anonlink-entity-service.readthedocs.io/en/stable/) for CODI datasets. Currently, anonlink uses blocklib library which supports two blocking methods:\n",
    "\n",
    "- “p-sig”: Probabilistic signature\n",
    "- “lambda-fold”: LSH based lambda-fold\n",
    "\n",
    "which are proposed by the following publications:\n",
    "\n",
    "- [Scalable Entity Resolution Using Probabilistic Signatures on Parallel Databases](https://arxiv.org/abs/1712.09691)\n",
    "- [An LSH-Based Blocking Approach with a Homomorphic Matching Technique for Privacy-Preserving Record Linkage](https://www.computer.org/csdl/journal/tk/2015/04/06880802/13rRUxASubY)\n",
    "\n",
    "Adjustments to the blocklib configuration will be made to the type of blocking, encoding, and threshold. We will evaluate multiple runs of our linkage tools using an example data set. The metrics for evaluation include:\n",
    "\n",
    "- Precision\n",
    "- Recall\n",
    "- Reduction Ratio\n",
    "- Set completeness\n",
    "- Performance based on average block size\n",
    "\n",
    "## Useful Terminology\n",
    "\n",
    "- Blocking - a technique that makes record linkage scalable. It is achieved by partitioning datasets into groups, called blocks and only comparing records in corresponding blocks. This can reduce the number of comparisons that need to be conducted to find which pairs of records should be linked.\n",
    "- Bloom filter - a probabilistic data structure used to test set membership. It tells if an element may be in a set, or definitely isn't.\n",
    "- Precision – how many of the found matches are actual matches (found groups : true matches)\n",
    "- Recall – how many of the actual matches we found (true matches : found groups)\n",
    "- Reduction Ratio – measures the proportion of number of comparisons reduced by using blocking\n",
    "- Set Completeness – how many true matches are maintained after blocking\n",
    "- Feature hashing – a fast and space-efficient way of vectorizing features, i.e. turning arbitrary features into indices in a vector or matrix\n",
    "- `p-sig` signature – A subrecord of an entity that can be used to uniquely link commonality between multiple records of an entity\n",
    "- `p-sig` Blocking keys – lower the cost of comparison between datasets by selecting partitions of the raw records (ex. First name, last name, postal code) *its assumed records sharing no blocking keys do not match with each other*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the Environment\n",
    "\n",
    "- The basic requirement is that you have [data-owner-tools](https://github.com/mitre/data-owner-tools) set up with all of Synthetic Denver sites extracted via extract.py and named with the format pii_\\*.csv (e.g. pii_ch.csv) for all 5 sites (scripts can easily be adjusted to work with other data) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "import seaborn as sns\n",
    "from IPython.display import FileLink, FileLinks\n",
    "import qgrid\n",
    "import csv\n",
    "import json\n",
    "from pathlib import Path\n",
    "import dcctools.config\n",
    "from itertools import combinations\n",
    "import importlib\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'dcctools.config' from '/Users/apellitieri/Desktop/CDC/CODI/linkage-agent-tools/dcctools/config.py'>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Only needed if edits are made to dcctools.config\n",
    "importlib.reload(dcctools.config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up a new run:\n",
    "\n",
    "Set an optional run identifier / description string for the row that will be created in the table of run data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_description = 'File reorg run'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set path to data-owner-tools project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_OWNER_TOOLS_DIR = '/Users/apellitieri/Desktop/CDC/CODI/data-owner-tools/'\n",
    "SECRET_FILE = '/Users/apellitieri/Desktop/CDC/CODI/deidentification_secret.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure the blocking schema file to use for the run is set correctly in `config.json`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Config file in use:\n",
      "{\n",
      "  \"systems\": [\n",
      "    \"site_a\",\n",
      "    \"site_b\",\n",
      "    \"site_c\",\n",
      "    \"site_d\",\n",
      "    \"site_e\",\n",
      "    \"site_f\"\n",
      "  ],\n",
      "  \"projects\": [\n",
      "    \"name-sex-dob-phone\",\n",
      "    \"name-sex-dob-zip\",\n",
      "    \"name-sex-dob-parents\",\n",
      "    \"name-sex-dob-addr\"\n",
      "  ],\n",
      "  \"schema_folder\": \"/Users/apellitieri/Desktop/CDC/CODI/data-owner-tools/example-schema\",\n",
      "  \"inbox_folder\": \"/Users/apellitieri/Desktop/CDC/CODI/inbox\",\n",
      "  \"matching_results_folder\": \"/Users/apellitieri/Desktop/CDC/CODI/results\",\n",
      "  \"output_folder\": \"/Users/apellitieri/Desktop/CDC/CODI/output\",\n",
      "  \"entity_service_url\": \"http://localhost:8851/api/v1\",\n",
      "  \"matching_threshold\": 0.75,\n",
      "  \"mongo_uri\": \"localhost:27017\",\n",
      "  \"blocked\": false,\n",
      "  \"blocking_schema\": \"/Users/apellitieri/Desktop/CDC/CODI/data-owner-tools/example-blocking-schema/lambda.json\",\n",
      "  \"household_match\": true,\n",
      "  \"household_schema\": \"/Users/apellitieri/Desktop/CDC/CODI/data-owner-tools/example-schema/household-schema/fn-phone-addr-zip.json\"\n",
      "}\n",
      "\n",
      "No blocking set to be used on this run.\n"
     ]
    }
   ],
   "source": [
    "CONFIG = dcctools.config.Configuration(\"config.json\")\n",
    "print('Config file in use:')\n",
    "with open('config.json', 'r') as config:\n",
    "    config_data = json.load(config)\n",
    "    print(json.dumps(config_data, indent=2))\n",
    "blocking_schema_file = ''\n",
    "if CONFIG.blocked:\n",
    "    blocking_schema_file = CONFIG.blocking_schema\n",
    "    with open(blocking_schema_file, 'r') as blocking_schema:\n",
    "        schema_data = json.load(blocking_schema)\n",
    "        print('\\nBlocking schema being used:')\n",
    "        print(json.dumps(schema_data, indent=2))\n",
    "else:\n",
    "    blocking_schema_file = 'None'\n",
    "    print('\\nNo blocking set to be used on this run.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Garble and block with anonlink\n",
    "\n",
    "The following block runs the script to garble the pii_\\*.csv files and then block and package the CLKs into the inbox folder.\n",
    "\n",
    "[TODO]: Make the garble scripts not specific to the synthetic denver sites - currently need to change name of script depending on using synthetic denver or the new site\n",
    "\n",
    "[TODO]: Figure out a way to record the blocking statistics from the run. The anonlink client blocking program prints out statistics about the blocking run but does not provide the output in an easily digestable format. The important output to look at is the maximum and average block size. From the [anonlink-client documentation](https://anonlink-client.readthedocs.io/en/stable/tutorial/Blocking%20with%20Anonlink%20Entity%20Service.html#Blocking):\n",
    "```\n",
    "The record linkage run time will be largely dominated by the maximum block size, and the number of blocks. In general the smaller the average block size, the better.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cleaning inbox...\n",
      "Running garble.py for site_a\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 751/751 [01:38<00:00,  7.64it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Running garble.py for site_b\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 751/751 [01:33<00:00,  8.02it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Running garble.py for site_c\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 751/751 [01:33<00:00,  8.00it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Running garble.py for site_d\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 750/750 [01:35<00:00,  7.84it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Running garble.py for site_e\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 750/750 [01:39<00:00,  7.56it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Running garble.py for site_f\n",
      "\u001b[31mCLK data written to output/name-sex-dob-phone.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-zip.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-parents.json\u001b[0m\n",
      "\u001b[31mCLK data written to output/name-sex-dob-addr.json\u001b[0m\n",
      "Zip file created at: output/garbled.zip\n",
      "Grouping individuals into households: 100%|███| 751/751 [01:32<00:00,  8.13it/s]\n",
      "\u001b[31mCLK data written to output/households/fn-phone-addr-zip.json\u001b[0m\n",
      "Zip file created at: output/garbled_households.zip\n",
      "Garbled zip files created:\n",
      "site_a.zip            site_c.zip            site_e.zip\n",
      "site_a_households.zip site_c_households.zip site_e_households.zip\n",
      "site_b.zip            site_d.zip            site_f.zip\n",
      "site_b_households.zip site_d_households.zip site_f_households.zip\n"
     ]
    }
   ],
   "source": [
    "inbox_folder = CONFIG.inbox_folder\n",
    "garble_start = time.perf_counter()\n",
    "if CONFIG.blocked:\n",
    "    shell_script = \"{}testing-and-tuning/blocking-garble.sh\".format(DATA_OWNER_TOOLS_DIR)\n",
    "    !$shell_script {blocking_schema_file} {inbox_folder}\n",
    "else:\n",
    "    shell_script = \"{}testing-and-tuning/garble.sh\".format(DATA_OWNER_TOOLS_DIR)\n",
    "    !$shell_script {inbox_folder} {SECRET_FILE}\n",
    "garble_end = time.perf_counter()\n",
    "garble_time = garble_end - garble_start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Garble and block (if enabled) took 626.09 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f\"Garble and block (if enabled) took {garble_time:0.2f} seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All necessary input is present\r\n"
     ]
    }
   ],
   "source": [
    "!python validate.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Database cleared.\r\n"
     ]
    }
   ],
   "source": [
    "# Need to drop database collection here if previous run took place\n",
    "!python drop.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 3441434, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 3441434, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 3441434, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'completed', 'time_added': '2021-09-12T23:16:40.243880+00:00', 'time_completed': '2021-09-12T23:16:56.312192+00:00', 'time_started': '2021-09-12T23:16:43.528666+00:00'}\n",
      "Inserting 777 records into the local database: 100%|█| 777/777 [00:02<00:00, 262\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'completed', 'time_added': '2021-09-12T23:17:02.563037+00:00', 'time_completed': '2021-09-12T23:17:21.310520+00:00', 'time_started': '2021-09-12T23:17:10.049761+00:00'}\n",
      "Inserting 1351 records into the local database: 100%|█| 1351/1351 [00:04<00:00, \n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'completed', 'time_added': '2021-09-12T23:17:28.321381+00:00', 'time_completed': '2021-09-12T23:17:47.460539+00:00', 'time_started': '2021-09-12T23:17:35.970545+00:00'}\n",
      "Inserting 1299 records into the local database: 100%|█| 1299/1299 [00:09<00:00, \n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'completed', 'time_added': '2021-09-12T23:18:00.221856+00:00', 'time_completed': '2021-09-12T23:18:17.893526+00:00', 'time_started': '2021-09-12T23:18:07.810075+00:00'}\n",
      "Inserting 1340 records into the local database: 100%|█| 1340/1340 [00:09<00:00, \n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'waiting for CLKs', 'number': 1, 'progress': {'absolute': 6, 'description': 'number of parties already contributed', 'relative': 1.0}}, 'stages': 3, 'state': 'created', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'queued', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': None}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute similarity scores', 'number': 2, 'progress': {'absolute': 8452506, 'description': 'number of already computed similarity scores', 'relative': 1.0}}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'running', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "{'current_stage': {'description': 'compute output', 'number': 3}, 'stages': 3, 'state': 'completed', 'time_added': '2021-09-12T23:18:29.115037+00:00', 'time_completed': '2021-09-12T23:18:51.789516+00:00', 'time_started': '2021-09-12T23:18:36.983225+00:00'}\n",
      "Inserting 1420 records into the local database: 100%|█| 1420/1420 [00:09<00:00, \n"
     ]
    }
   ],
   "source": [
    "match_start = time.perf_counter()\n",
    "!python match.py\n",
    "match_end = time.perf_counter()\n",
    "match_time = match_end - match_start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matching took 161.51 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f\"Matching took {match_time:0.2f} seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/apellitieri/Desktop/CDC/CODI/results/link_ids.csv created\n",
      "Before deconflict: 856\n",
      "After deconflict and before add singles: 856\n",
      "Final linkage count: 1783\n",
      "Exact individual links found in pprl household links: 1092\n",
      "Number of individual links conflicting with pprl links: 1240\n",
      "Number of individual links combined into PPRL links: 1119\n",
      "Number of individual links skipped from previous conflict: 0\n",
      "/Users/apellitieri/Desktop/CDC/CODI/results/household_link_ids.csv created\n"
     ]
    }
   ],
   "source": [
    "!python link_ids.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "results/patid_link_ids.csv created\r\n"
     ]
    }
   ],
   "source": [
    "!python -m tuning-files-scripts.patid_translate --dotools {DATA_OWNER_TOOLS_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Record Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pair-wise scoring:\r\n",
      "Precision: 0.944079810974009 Recall: 0.7094101400670744 F-Score: 0.8100923631448524\r\n",
      "Perfect scoring:\r\n",
      "Precision: 0.5027932960893855 Recall: 0.5990016638935108 F-Score: 0.5466970387243737\r\n",
      "Partial scoring:\r\n",
      "Precision: 0.9040307101727447 Recall: 0.78369384359401 F-Score: 0.8395721925133689\r\n"
     ]
    }
   ],
   "source": [
    "!python -m tuning-files-scripts.household_score --dotools {DATA_OWNER_TOOLS_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "RESULTS_PATH = '/Users/apellitieri/Desktop/CDC/CODI/results'\n",
    "ANSWER_KEY_CSV = '/Users/apellitieri/Desktop/CDC/CODI/new_answer_key.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_precision = 0\n",
    "run_recall = 0\n",
    "run_f_score = 0\n",
    "answer_key_length = 0\n",
    "proposed_pairs_count = 0\n",
    "\n",
    "systems = CONFIG.systems\n",
    "threshold = CONFIG.matching_threshold\n",
    "\n",
    "true_positives = 0\n",
    "false_positives = 0\n",
    "\n",
    "answer_key = []\n",
    "\n",
    "with open(ANSWER_KEY_CSV) as ak_csv:\n",
    "  ak_reader = csv.reader(ak_csv)\n",
    "  next(ak_reader)\n",
    "  for row in ak_reader:\n",
    "    if row[3] == '1':\n",
    "      answer_pair = [row[1], row[2]]\n",
    "      answer_pair.sort()\n",
    "      answer_key.append(answer_pair)\n",
    "\n",
    "answer_key_length = len(answer_key)\n",
    "\n",
    "patid_csv_path = Path(RESULTS_PATH) / \"patid_link_ids.csv\"\n",
    "\n",
    "with open(patid_csv_path) as patid_csv:\n",
    "  pat_id_reader = csv.reader(patid_csv)\n",
    "  next(pat_id_reader)\n",
    "  for row in pat_id_reader:\n",
    "    patids = row[1:6]\n",
    "    patids = list(filter(lambda id: len(id) > 0, patids))\n",
    "    combos = combinations(patids, 2)\n",
    "    for a, b in combos:\n",
    "      pair = [a, b]\n",
    "      pair.sort()\n",
    "      if pair in answer_key:\n",
    "        true_positives += 1\n",
    "      else:\n",
    "        false_positives += 1\n",
    "\n",
    "run_precision = true_positives / (true_positives + false_positives)\n",
    "run_recall = true_positives / answer_key_length\n",
    "run_f_score = 2 * ((run_precision * run_recall) / (run_precision + run_recall))\n",
    "proposed_pairs_count = true_positives + false_positives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.99\n",
      "Recall: 0.60\n",
      "F-Score: 0.74\n"
     ]
    }
   ],
   "source": [
    "print(f\"Precision: {run_precision:0.2f}\\nRecall: {run_recall:0.2f}\\nF-Score: {run_f_score:0.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully added run to example_run_data.csv!\n"
     ]
    }
   ],
   "source": [
    "with open('tuning-files-scripts/example_run_data.csv', 'a', newline='') as csvfile:\n",
    "    fieldnames = ['Run Description', 'Blocking', 'Match Threshold', 'Precision',\n",
    "                  'Recall', 'F-Score', 'Answer Key Size',\n",
    "                  'Proposed Pairs', 'True Positives', 'Garble & Block Time (s)',\n",
    "                  'Match Time (s)', 'Blocking Config']\n",
    "    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
    "    # writer.writeheader() # Remove after initial run\n",
    "    writer.writerow({\n",
    "        'Run Description': run_description, 'Blocking': CONFIG.blocked,\n",
    "        'Match Threshold': threshold,\n",
    "        'Precision': run_precision, 'Recall': run_recall, 'F-Score': run_f_score,\n",
    "        'Answer Key Size': answer_key_length, 'Proposed Pairs': proposed_pairs_count,\n",
    "        'True Positives': true_positives,\n",
    "        'Garble & Block Time (s)': garble_time,\n",
    "        'Match Time (s)': match_time, 'Blocking Config': blocking_schema_file\n",
    "    })\n",
    "print(\"Successfully added run to example_run_data.csv!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Run Description</th>\n",
       "      <th>Blocking</th>\n",
       "      <th>Match Threshold</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F-Score</th>\n",
       "      <th>Answer Key Size</th>\n",
       "      <th>Proposed Pairs</th>\n",
       "      <th>True Positives</th>\n",
       "      <th>Garble &amp; Block Time (s)</th>\n",
       "      <th>Match Time (s)</th>\n",
       "      <th>Blocking Config</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Lambda-fold run 1 (Synthetic Denver)</td>\n",
       "      <td>True</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.941725</td>\n",
       "      <td>0.832990</td>\n",
       "      <td>0.884026</td>\n",
       "      <td>970</td>\n",
       "      <td>858</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.998019</td>\n",
       "      <td>81.662922</td>\n",
       "      <td>/Users/apellitieri/Desktop/CDC/CODI/data-owner...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No Blocking Run 1 (Synthetic Denver)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.942263</td>\n",
       "      <td>0.841237</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>970</td>\n",
       "      <td>866</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.903272</td>\n",
       "      <td>76.427395</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No Blocking Run 1 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.996756</td>\n",
       "      <td>0.563965</td>\n",
       "      <td>0.720354</td>\n",
       "      <td>7082</td>\n",
       "      <td>4007</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15.183644</td>\n",
       "      <td>89.882657</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No Blocking Run 2 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.996756</td>\n",
       "      <td>0.563965</td>\n",
       "      <td>0.720354</td>\n",
       "      <td>7082</td>\n",
       "      <td>4007</td>\n",
       "      <td>3994.0</td>\n",
       "      <td>15.443283</td>\n",
       "      <td>100.035062</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>No Blocking Run 3 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.988751</td>\n",
       "      <td>0.595736</td>\n",
       "      <td>0.743502</td>\n",
       "      <td>7082</td>\n",
       "      <td>4267</td>\n",
       "      <td>4219.0</td>\n",
       "      <td>15.563336</td>\n",
       "      <td>91.285207</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>No Blocking Run 4 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.970907</td>\n",
       "      <td>0.607879</td>\n",
       "      <td>0.747655</td>\n",
       "      <td>7082</td>\n",
       "      <td>4434</td>\n",
       "      <td>4305.0</td>\n",
       "      <td>16.600817</td>\n",
       "      <td>97.703846</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>No Blocking Run 4 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.955512</td>\n",
       "      <td>0.615645</td>\n",
       "      <td>0.748819</td>\n",
       "      <td>7082</td>\n",
       "      <td>4563</td>\n",
       "      <td>4360.0</td>\n",
       "      <td>15.385254</td>\n",
       "      <td>92.289256</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>No Blocking Run 4 (New Data Set)</td>\n",
       "      <td>False</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.895652</td>\n",
       "      <td>0.450861</td>\n",
       "      <td>0.599793</td>\n",
       "      <td>7082</td>\n",
       "      <td>3565</td>\n",
       "      <td>3193.0</td>\n",
       "      <td>15.295187</td>\n",
       "      <td>134.370978</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>Lambda Run 1 (New Data Set)</td>\n",
       "      <td>True</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.955566</td>\n",
       "      <td>0.601243</td>\n",
       "      <td>0.738083</td>\n",
       "      <td>7082</td>\n",
       "      <td>4456</td>\n",
       "      <td>4258.0</td>\n",
       "      <td>28.138004</td>\n",
       "      <td>103.387432</td>\n",
       "      <td>/Users/apellitieri/Desktop/CDC/CODI/data-owner...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>File reorg run</td>\n",
       "      <td>False</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.989220</td>\n",
       "      <td>0.596018</td>\n",
       "      <td>0.743854</td>\n",
       "      <td>7082</td>\n",
       "      <td>4267</td>\n",
       "      <td>4221.0</td>\n",
       "      <td>626.093749</td>\n",
       "      <td>161.507163</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Run Description  Blocking  Match Threshold  Precision  \\\n",
       "0  Lambda-fold run 1 (Synthetic Denver)      True             0.80   0.941725   \n",
       "1  No Blocking Run 1 (Synthetic Denver)     False             0.80   0.942263   \n",
       "2      No Blocking Run 1 (New Data Set)     False             0.80   0.996756   \n",
       "3      No Blocking Run 2 (New Data Set)     False             0.80   0.996756   \n",
       "4      No Blocking Run 3 (New Data Set)     False             0.75   0.988751   \n",
       "5      No Blocking Run 4 (New Data Set)     False             0.70   0.970907   \n",
       "6      No Blocking Run 4 (New Data Set)     False             0.65   0.955512   \n",
       "7      No Blocking Run 4 (New Data Set)     False             0.50   0.895652   \n",
       "8           Lambda Run 1 (New Data Set)      True             0.65   0.955566   \n",
       "9                        File reorg run     False             0.75   0.989220   \n",
       "\n",
       "     Recall   F-Score  Answer Key Size  Proposed Pairs  True Positives  \\\n",
       "0  0.832990  0.884026              970             858             NaN   \n",
       "1  0.841237  0.888889              970             866             NaN   \n",
       "2  0.563965  0.720354             7082            4007             NaN   \n",
       "3  0.563965  0.720354             7082            4007          3994.0   \n",
       "4  0.595736  0.743502             7082            4267          4219.0   \n",
       "5  0.607879  0.747655             7082            4434          4305.0   \n",
       "6  0.615645  0.748819             7082            4563          4360.0   \n",
       "7  0.450861  0.599793             7082            3565          3193.0   \n",
       "8  0.601243  0.738083             7082            4456          4258.0   \n",
       "9  0.596018  0.743854             7082            4267          4221.0   \n",
       "\n",
       "   Garble & Block Time (s)  Match Time (s)  \\\n",
       "0                22.998019       81.662922   \n",
       "1                13.903272       76.427395   \n",
       "2                15.183644       89.882657   \n",
       "3                15.443283      100.035062   \n",
       "4                15.563336       91.285207   \n",
       "5                16.600817       97.703846   \n",
       "6                15.385254       92.289256   \n",
       "7                15.295187      134.370978   \n",
       "8                28.138004      103.387432   \n",
       "9               626.093749      161.507163   \n",
       "\n",
       "                                     Blocking Config  \n",
       "0  /Users/apellitieri/Desktop/CDC/CODI/data-owner...  \n",
       "1                                               None  \n",
       "2                                               None  \n",
       "3                                               None  \n",
       "4                                               None  \n",
       "5                                               None  \n",
       "6                                               None  \n",
       "7                                               None  \n",
       "8  /Users/apellitieri/Desktop/CDC/CODI/data-owner...  \n",
       "9                                               None  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('tuning-files-scripts/example_run_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}