Twitter_opinion_mining_example_run.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter Opinion Mining\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the different modules\n",
    "from buildDatabase import buildDatabse\n",
    "from makeHTnetwork import makeHTNetwork\n",
    "from selectInitialHashtags import selectInitialHashtags\n",
    "from propagateLabels import propagateLabels\n",
    "from addStatSigniHT import addStatSigniHT\n",
    "from selectHashtags import selectHashtags\n",
    "from updateHTGroups import updateHTGroups\n",
    "from buildTrainingSet import buildTrainingSet\n",
    "from crossValOptimize import crossValOptimize\n",
    "from trainClassifier import trainClassifier\n",
    "from classifyTweets import classifyTweets\n",
    "from makeProbaDF import makeProbaDF\n",
    "from analyzeProbaDF import analyzeProbaDF\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# create empty dictionary and add parameters\n",
    "job = {}\n",
    "# list of directories containing the tweet archive files (TAJ)\n",
    "job['tweet_archive_dirs'] = ['tweet_dir1', 'tweet_dir2']\n",
    "\n",
    "# SQLite database that will be created\n",
    "job['sqlite_db_filename'] = 'test.sqlite'\n",
    "\n",
    "# hashtag co-occurrence graph that will be created\n",
    "job['graph_file'] = 'graph_file.graphml'\n",
    "\n",
    "# pickle files where the training set features will be saved\n",
    "job['features_pickle_file'] = 'features.pickle'\n",
    "\n",
    "# pickle file where the training set labels will be saved\n",
    "job['labels_pickle_file'] = 'labels.pickle'\n",
    "\n",
    "# vectorized features file\n",
    "job['features_vect_file'] = 'features.mmap'\n",
    "\n",
    "# vectorized labels file\n",
    "job['labels_vect_file'] = 'labels.mmap'\n",
    "\n",
    "# mapping between labels names and numbers\n",
    "job['labels_mappers_file'] = 'labels_mappers.pickle'\n",
    "\n",
    "# JSON file with the classifier best parameters obtained from cross-validation\n",
    "job['best_params_file'] = 'best_params.json'\n",
    "\n",
    "# where the trained calssifier will be saved\n",
    "job['classifier_filename'] = 'classifier.pickle'\n",
    "\n",
    "# DataFrame with the results of the label propagation\n",
    "# on the hashtag network\n",
    "job['propag_results_filename'] = 'propag_results.pickle'\n",
    "\n",
    "# DataFrame with the classification probability of\n",
    "# every tweets in the database\n",
    "job['df_proba_filename'] = 'df_proba.pickle'\n",
    "\n",
    "# DataFrame with the number of tweets in each camp per day\n",
    "job['df_num_tweets_filename'] = 'df_num_tweets.pickle'\n",
    "\n",
    "# DataFrame with the number of users in each camp per day\n",
    "job['df_num_users_filename'] = 'df_num_users.pickle'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To collect tweets, dowload the code from https://github.com/kcore0/kcore and follow the instructions in the README file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Build the SQLite database with the extracted information from the tweets\n",
    "Read the tweets from all the .taj files in the directories `tweet_archive_dirs`\n",
    "and add them to the database `sqlite_db_filename`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "buildDatabse(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.  Make the Hashtag co-occurrences network\n",
    "Reads all the co-occurences from the SQLite database and builds the network\n",
    "of where nodes are hashtags and edges are co-occurrences.\n",
    "The graph is a [*graph-tool*](https://graph-tool.skewed.de/) object and is saved in graphml format to `graph_file`.\n",
    "\n",
    "Nodes of the graph have two properties: `counts` is the number of single occurrences of the hashtag and `name` is the name of the hashtag.\n",
    "\n",
    "Edges have a property `weights` equal to the number of co-occurrences they represent.\n",
    "\n",
    "The graph has the following properties saved with it:\n",
    "- `Ntweets`: number of tweets with at least one hashtag used to build the graph.\n",
    "- `start_date` : date of the first tweet.\n",
    "- `stop_date` : date of the last tweet.\n",
    "- `weight_threshold` : co-occurrence threshold. Edges with less than `weight_threshold` co-occurrences are discarded.\n",
    "\n",
    "*Optional parameters that can be added to `job`:*\n",
    "- `start_date` and `stop_date` to specify a time range for the tweets. (Default is `None`, i.e. select all the tweets in the database).\n",
    "- `weight_threshold` is the minimum number of co-occurences between to hashtag to be included in the graph. (Default is 3).\n",
    "\n",
    "To add a parameter to job, simply execute `job[\"parameter name\"] = parameter value`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "makeHTNetwork(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Add statistical significance value to edges\n",
    "Adds a property `s` to edges of the graph corresponding to the statistical significance (`s = log10(p_0/p)`)\n",
    "of the co-occurence computed from a null model [1].\n",
    "The computation is done using `p0=1e-6` and `p0` is saved as a graph property.\n",
    "Different values of `p0` can be tested latter.\n",
    "The resulting graph is saved to `graph_file`.\n",
    "\n",
    "*Optional parameters that can be added to `job`:*\n",
    "- `ncpu` : number of processors to be used. (Default is the number of cores on your machine minus 1).\n",
    "\n",
    "\n",
    "[1] Martinez-Romo, J. et al. Disentangling categorical relationships through a graph of co-occurrences. Phys. Rev. E 84, 1–8 (2011).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addStatSigniHT(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Select the initial hashtags to start the propagation\n",
    "This will display to top occurring hashtags.\n",
    "\n",
    "*Optional parameters that can be added to `job`:*\n",
    "- `num_top_htgs` : (Default is top 100)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selectInitialHashtags(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select seeds hashtags you want to use from the list (minimum two) \n",
    "and add them to the `job` dictionary with the key `initial_htgs_lists`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial_htgs_lists is a list of list with hashtags seeds for each camp:\n",
    "job['initial_htgs_lists'] = [['ht1group1','ht2group1'],\n",
    "                             ['ht1group2', 'ht2group2']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Propagate labels to neighboring hashtags\n",
    "This part can be looped by updating the `htgs_lists` in `job` with the result of the label propagation to reach a larger number of hashtags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start with the hashtag seeds selected above.\n",
    "job['htgs_lists'] = job['initial_htgs_lists']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loop has two steps:\n",
    "1. `propagateLabels` uses the graph from `graph_file` and the initial hashtags from `htgs_lists` to propagate their labels to their neighbors taking into account the statistical significance of edges. The results are saved in a pandas DataFrame in `propag_results_filename`.\n",
    "    - *Optional parameters that can be added to `job`:*\n",
    "        - `count_ratio` : threshold, $r$, for removing hashtags with a number of single occurrences smaller than $r \\max\\limits_{v_j\\in C_k} c_j$ where $c_i$ is the number of occurrences of the hashtag associated with vertex $v_i$, $C_k$ is the class to which $v_i$ belong. (Default = 0.001).\n",
    "        - `p0` : significance threshold. to keep only edges with p_val <= p0. (Default = 1e-5).\n",
    "\n",
    "2. Visualisation of the results using `selectHashtags`, and updating the `htgs_lists` list. This will print a list of hashtags, $i$, for each camp $C_k$ satisfying: $\\sum_{j \\in C_k} s_{ij} > \\sum_{j \\notin C_k} s_{ij}$, where $\\{i : i \\notin C_k \\}$ represents all the other camps than $C_k$.\n",
    "    - *Optional parameters that can be added to `job`:*\n",
    "        - `num_top_htgs` : number of top hashtags to be displayed in each camp. (Default is 100)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1st step of the loop:\n",
    "propagateLabels(job).run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2nd step of the loop:\n",
    "selectHashtags(job).run()\n",
    "# the signification of the displayed columns are:\n",
    "# count (= total number of occurrences),\n",
    "# label_init (= initial label before propagation, -1 means no initial labels)\n",
    "# vertex_id  (= ID of the vertex in the hashtag graph)\n",
    "# label_sum1 (= number of neighbors with label 1)\n",
    "# signi_sum1 (= sum of the significance of edges with neighbors having label 1)\n",
    "# label_sum2 (= number of neighbors with label 2)\n",
    "# signi_sum2 (= sum of the significance of edges with neighbors having label 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can now update the hashtag list and return the 1st step.\n",
    "job['htgs_lists'] = [['ht1group1','ht2group1','ht3group1'],\n",
    "                     ['ht1group2', 'ht2group2','ht3group2']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.  Mark the selected hashtags in the database and build the training set\n",
    "`updateHTGroups` takes the lists of hashtags `htgs_lists` and mark then in the database `sqlite_db_filename`.\n",
    "\n",
    "*Optional parameters that can be added to `job`:*\n",
    "- `column_name_ht_group` : name of the column added to the database (Default is `'ht_class'`). Different names can be used to test different `htgs_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "updateHTGroups(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`buildTrainingSet` reads tweets from the database with hashtags marked above, extract the features and labels of each tweets and saves them in `features_pickle_file` and `labels_pickle_file`, respectively.\n",
    "Vectorized versions of the features and labels are saved to `features_vect_file` and `labels_vect_file` for the cross-validation. A mapper between label names and label number is saved to `labels_mappers_file`.\n",
    "\n",
    "*Optional parameters:*\n",
    "\n",
    "- If the optional parameter `column_name_ht_group` has been changed in `job` in the step before, it will be used here to select the corresponding hashtag lists.\n",
    "- `undersample_maj_class` : whether to undersample the majority class in order to balance the training set. Default is True, if False, unbalanced training set will be used and [class weight](http://scikit-learn.org/0.18/modules/generated/sklearn.linear_model.SGDClassifier.html) will be adjusted accrodingly during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "buildTrainingSet(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Cross-Validation\n",
    "Estimate the performance of the classifier and optimize classifier parameters with cross-validation. `crossValOptimize` loads the vectorized features and labels (`features_vect_file` and `labels_vect_file`) and saves the results of the optimization to `best_params_file` in JSON format.\n",
    "\n",
    "*Optional parameters:*\n",
    "- if `undersample_maj_class` was set to `False` when building the training set, class weights will be adjusted to take into account different sizes of classes.\n",
    "- `ncpu` : number of cores to use (default is the number of cpus on your machine minus one).\n",
    "- `scoring` : The score used to optimize (default is `'f1_micro'`). See the [documentation](http://scikit-learn.org/0.18/modules/generated/sklearn.model_selection.GridSearchCV.html) for explanation and other possibilities. \n",
    "- `n_splits` : number of [folds](http://scikit-learn.org/0.18/modules/generated/sklearn.model_selection.KFold.html) (default is 10).\n",
    "- `loss` : [loss function](scikit-learn.org/0.18/modules/generated/sklearn.linear_model.SGDClassifier.html) to be used. Default is `'log'` for Logistic Regression.\n",
    "- `penalty` : [penalty](scikit-learn.org/0.18/modules/generated/sklearn.linear_model.SGDClassifier.html) of the regularization term (default is `'l2`).\n",
    "- `n_iter` : [number of iterations](scikit-learn.org/0.18/modules/generated/sklearn.linear_model.SGDClassifier.html) of the gradient descent algorithm. Default is `5e5/(number of training samples)`. See the sklearn Stochastic Gradient Descent [user guide](http://scikit-learn.org/0.18/modules/sgd.html#sgd) for recommended settings.\n",
    "- `grid_search_parameters` : parameter space to explore during the cross-validation. Default is `{'classifier__alpha' : np.logspace(-1,-7, num=20)}`, i.e. optimizing the [regularization strength](http://scikit-learn.org/0.18/modules/sgd.html#sgd) (`alpha`) between 1e-1 and 1e-7 with 20 logarithmic steps.\n",
    "- `verbose` : verbosity level of the calssifier (default is 1).\n",
    "\n",
    "To change parameters use, e.g.:\n",
    "\n",
    "- `job['grid_search_parameters'] = {'classifier__alpha' : np.logspace(-1,-7, num=10)}`\n",
    "- `job['grid_search_parameters'] = {'classifier__penalty' : ['l1', 'l2']}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# here we set n_iter=2 just for testing purposes\n",
    "job['n_iter'] = 2\n",
    "CV = crossValOptimize(job)\n",
    "CV.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Train Classifier\n",
    "Uses features and labels from `features_pickle_file` and `labels_pickle_file` to train the classifier using the parameters from `best_params_file`. The trained classifier is then saved to `classifier_filename`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainClassifier(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Classify the tweets\n",
    "Adds two tables `class_proba` and `retweet_class_proba` to the SQLite database with the result of the classification of each tweets and original retweeted status.\n",
    "\n",
    "*Optional parameters:*\n",
    "- `propa_table_name_suffix` : add a suffix to the two table names in order to compare different classifiers. Default is '' (empty string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifyTweets(job).run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Analyze classification results\n",
    "`makeProbaDF` reads the classification results from the database and processes them to:\n",
    "- Replace the classification probability of retweets with the classification results of the original tweets.\n",
    "- Replace the classification probability of tweets having a hashtag of one of the two camps (and not of the other camp) with 0 (for camp1) or 1 (for camp2).\n",
    "- Discard tweets emanating from unoffical Twitter clients.\n",
    "\n",
    "The list of unofficial clients is located in the file `TwSentiment.py` and was manually constructed by taking only the most frequently used clients that do not allow to autmomate the posting of tweets.\n",
    "\n",
    "The results are saved as a pandas dataframe in `df_proba_filename`.\n",
    "\n",
    "*Optional parameters:*\n",
    "- `use_official_clients` : whether you want to keep only tweets from official clients (`True`) or all tweets (`False`). Default is `True`.\n",
    "- `propa_table_name_suffix` can be changed to use the classification of different classifiers if it was used with `classifyTweets`.\n",
    "- `column_name_ht_group` is also used if it was changed to create a different training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mProbDF = makeProbaDF(job)\n",
    "mProbDF.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`analyzeProbaDF` reads `df_proba_filename` and returns the number of tweets and the number of users in each camp per day. The results are displayed and saved as pandas dataframes to `df_num_tweets_filename` and `df_num_users_filename`.\n",
    "\n",
    "*Optional parameters:*\n",
    "- `ncpu` : number of cores to use. Default is number of cores of the machine minus one.\n",
    "- `resampling_frequency` : frequency at which tweets are grouped. Default is `'D'`, i.e. daily. (see [here](http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases) for different possibilities.)\n",
    "- `threshold` : threshold for the classifier probability (threshold >= 0.5). Tweets with p > threshold are classified in camp2 and tweets with p < 1-threshold are classified in camp1. Default is 0.5.\n",
    "- `r_threshold` : threshold for the ratio of classified tweets needed to classify a user. Default is 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the distribution of probabilities:\n",
    "plot = mProbDF.df_proba_all.p_1.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = analyzeProbaDF(job)\n",
    "analysis.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot1 = analysis.df_num_tweets.plot(title='Number of tweets per day')\n",
    "\n",
    "plot2 = analysis.df_num_users.plot(title='Number of users per day')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print all job parameters\n",
    "job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}