nlp_metrices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy\n",
    "If a machine is trying to classsify some text (e.g., tell if it’s “positive” or “negative”), one may look at a metric called accuracy, which indicates what % of the classifications are correct.\n",
    "\n",
    "### Precision and Recall\n",
    "If a machine is “only” trying to determine whether something is true or not (e.g., if there’s a positive statement in some text), one oftentimes looks at two metrics, called recall (what % of the items are found) and precision (what % of the items identified have been correctly identified). There’s a tradeoff between those two metrics (since naive algorithms can declare each example as “yes” or as “no”), and sometimes people look at the average between the two (it’s a special type of average called harmonic mean, marked as F1 score - Wikipedia).\n",
    "\n",
    "### BLEU, ROUGE (Recall-Oriented Understudy for Gisting Evaluation)\n",
    "If a machine is trying to generate text in cases like translation and summarization, measurement is more complex. In fact, no two humans will generate the same text. Metrics such as BLEU, ROUGE and others, usually come into play.\n",
    "\n",
    "### Perplexity or Likelihood\n",
    "If a machine is trying to model text (for example, a so-called language model problem), one may look at metrics identifying how well does the model match actual text - via metrics such as likelihood or Perplexity - Wikipedia.\n",
    "\n",
    "When unsupervised tasks - like word vectors, and to a lesser degree topic modeling - come into play, there isn’t a “ground truth” that one can compare against. In such case, how to evaluate is a challenge of its own. One approach is to measure how well the task (e.g., word vectors) helps in some other, more measurable, task (e.g., sentiment evaluation).\n",
    "\n",
    "### WER (Word Error Rate)\n",
    "The bottom line is that the measurement/evaluation is context and task specific, and may also vary between academic work (that tends to follow past measurements, to show progress) and industry work (that tends to focus on the oftentimes soft aspects of how the method helps the product). For example, in the speech-to-text field (adjacent to traditional NLP), a common measure is WER, which stands for word error rate. This captures each error (superfluous words, replacements and omissions). But, in a business context, some errors may be less critical (e.g., a vs the; or, dogs vs. dog). Normally, there is no “standard” way to capture those nuances in a specific industry context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bleu measures precision: how much the words (and/or n-grams) in the machine generated summaries appeared in the human reference summaries.\n",
    "\n",
    "Rouge measures recall: how much the words (and/or n-grams) in the human reference summaries appeared in the machine generated summaries.\n",
    "\n",
    "Naturally - these results are complementing, as is often the case in precision vs recall. If you have many words from the system results appearing in the human references you will have high Bleu, and if you have many words from the human references appearing in the system results you will have high Rouge.\n",
    "\n",
    "In your case it would appear that sys1 has a higher Rouge than sys2 since the results in sys1 consistently had more words from the human references appear in them than the results from sys2. However, since your Bleu score showed that sys1 has lower recall than sys2, this would suggest that not so many words from your sys1 results appeared in the human references, in respect to sys2.\n",
    "\n",
    "This could happen for example if your sys1 is outputting results which contain words from the references (upping the Rouge), but also many words which the references didn't include (lowering the Bleu). sys2, as it seems, is giving results for which most words outputted do appear in the human references (upping the Blue), but also missing many words from its results which do appear in the human references.\n",
    "\n",
    "BTW, there's something called brevity penalty, which is quite important and has already been added to standard Bleu implementations. It penalizes system results which are shorter than the general length of a reference (read more about it here). This complements the n-gram metric behavior which in effect penalizes longer than reference results, since the denominator grows the longer the system result is.\n",
    "\n",
    "You could also implement something similar for Rouge, but this time penalizing system results which are longer than the general reference length, which would otherwise enable them to obtain artificially higher Rouge scores (since the longer the result, the higher the chance you would hit some word appearing in the references). In Rouge we divide by the length of the human references, so we would need an additional penalty for longer system results which could artificially raise their Rouge score.\n",
    "\n",
    "Finally, you could use the F1 measure to make the metrics work together: F1 = 2 * (Bleu * Rouge) / (Bleu + Rouge)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}