Code/Day50-FindTopPages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning Your Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a web access log, and figure out the most-viewed pages on a website from it! Sounds easy, right?\n",
    "\n",
    "Let's set up a regex that lets us parse an Apache access log line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "format_pat= re.compile(\n",
    "    r\"(?P<host>[\\d\\.]+)\\s\"\n",
    "    r\"(?P<identity>\\S*)\\s\"\n",
    "    r\"(?P<user>\\S*)\\s\"\n",
    "    r\"\\[(?P<time>.*?)\\]\\s\"\n",
    "    r'\"(?P<request>.*?)\"\\s'\n",
    "    r\"(?P<status>\\d+)\\s\"\n",
    "    r\"(?P<bytes>\\S*)\\s\"\n",
    "    r'\"(?P<referer>.*?)\"\\s'\n",
    "    r'\"(?P<user_agent>.*?)\"\\s*'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the path to the log file I'm analyzing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "logPath = \"../datasets/access_log.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll whip up a little script to extract the URL in each access, and use a dictionary to count up the number of times each one appears. Then we'll sort it and print out the top 20 pages. What could go wrong?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "not enough values to unpack (expected 3, got 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-1360e8ecb80e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      7\u001b[0m             \u001b[0maccess\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m             \u001b[0mrequest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0maccess\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'request'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m             \u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mURL\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mURL\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mURLCounts\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m                 \u001b[0mURLCounts\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mURL\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mURLCounts\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mURL\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: not enough values to unpack (expected 3, got 1)"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            (action, URL, protocol) = request.split()\n",
    "            if URL in URLCounts:\n",
    "                URLCounts[URL] = URLCounts[URL] + 1\n",
    "            else:\n",
    "                URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hm. The 'request' part of the line is supposed to look something like this:**\n",
    "\n",
    "GET /blog/ HTTP/1.1\n",
    "\n",
    "There should be an HTTP action, the URL, and the protocol. But it seems that's not always happening. Let's print out requests that don't contain three items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "['_\\\\xb0ZP\\\\x07tR\\\\xe5']\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n[]\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) != 3):\n",
    "                print(fields)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Huh. In addition to empty fields, there's one that just contains garbage. Well, let's modify our script to check for that case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "/xmlrpc.php: 68494\n/wp-login.php: 1923\n/: 440\n/blog/: 138\n/robots.txt: 122\n/sitemap_index.xml: 118\n/post-sitemap.xml: 118\n/page-sitemap.xml: 117\n/category-sitemap.xml: 117\n/orlando-headlines/: 95\n/san-jose-headlines/: 85\nhttp://51.254.206.142/httptest.php: 81\n/comics-2/: 76\n/travel/: 74\n/entertainment/: 72\n/business/: 70\n/national/: 70\n/national-headlines/: 70\n/world/: 70\n/weather/: 70\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) == 3):\n",
    "                URL = fields[1]\n",
    "                if URL in URLCounts:\n",
    "                    URLCounts[URL] = URLCounts[URL] + 1\n",
    "                else:\n",
    "                    URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked! But, the results don't really make sense. What we really want is pages accessed by real humans looking for news from our little news site. What the heck is xmlrpc.php? A look at the log itself turns up a lot of entries like this:\n",
    "\n",
    "46.166.139.20 - - [05/Dec/2015:05:19:35 +0000] \"POST /xmlrpc.php HTTP/1.0\" 200 370 \"-\" \"Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)\"\n",
    "\n",
    "I'm not entirely sure what the script does, but it points out that we're not just processing GET actions. We don't want POSTS, so let's filter those out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "/: 434\n/blog/: 138\n/robots.txt: 122\n/sitemap_index.xml: 118\n/post-sitemap.xml: 118\n/page-sitemap.xml: 117\n/category-sitemap.xml: 117\n/orlando-headlines/: 95\n/san-jose-headlines/: 85\nhttp://51.254.206.142/httptest.php: 81\n/comics-2/: 76\n/travel/: 74\n/entertainment/: 72\n/business/: 70\n/national/: 70\n/national-headlines/: 70\n/world/: 70\n/weather/: 70\n/about/: 69\n/defense-sticking-head-sand/: 69\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) == 3):\n",
    "                (action, URL, protocol) = fields\n",
    "                if (action == 'GET'):\n",
    "                    if URL in URLCounts:\n",
    "                        URLCounts[URL] = URLCounts[URL] + 1\n",
    "                    else:\n",
    "                        URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's starting to look better. But, this is a news site - are people really reading the little blog on it instead of news pages? That doesn't make sense. Let's look at a typical /blog/ entry in the log:\n",
    "\n",
    "54.165.199.171 - - [05/Dec/2015:09:32:05 +0000] \"GET /blog/ HTTP/1.0\" 200 31670 \"-\" \"-\"\n",
    "\n",
    "Hm. Why is the user agent blank? Seems like some sort of malicious scraper or something. Let's figure out what user agents we are dealing with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0): 68484\n-: 4035\nMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0): 1724\nW3 Total Cache/0.9.4.1: 468\nMozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html): 278\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html): 247\nMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 158\nMozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0: 144\nMozilla/5.0 (iPad; CPU OS 8_4 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12H143 Safari/600.1.4: 120\nMozilla/5.0 (Linux; Android 5.1.1; SM-G900T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36: 47\nMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm): 43\nMozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+): 41\nOpera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14: 40\nMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots): 27\nRuby: 15\nMozilla/5.0 (Linux; Android 5.1.1; SM-G900T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.76 Mobile Safari/537.36: 15\nMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 13\nMozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/): 11\nMozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2: 11\nMobileSafari/600.1.4 CFNetwork/711.4.6 Darwin/14.0.0: 10\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36: 9\nMozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots): 9\nMozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/): 7\nMozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html): 6\nMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp): 6\nMozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/3.6.28 (.NET CLR 3.5.30729): 4\nMozilla/5.0 zgrab/0.x: 4\nMozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.36: 4\nMozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/): 4\nMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1): 4\nMozilla/5.0: 3\nMozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0: 3\nOpera/9.80 (Windows NT 5.1; U; ru) Presto/2.9.168 Version/11.50: 3\nMozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot ): 3\nMozilla/4.0 (compatible: FDSE robot): 3\nMozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 2Pac; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022): 3\nMozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0: 3\nMozilla/5.0 (Windows NT 6.2; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0: 2\nMozilla/5.0 (Windows NT 5.1; rv:36.0) Gecko/20100101 Firefox/36.0: 2\nMozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0: 2\nMozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/5.0: 2\nMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 2\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 2\nGooglebot-Image/1.0: 2\nnetEstate NE Crawler (+http://www.website-datenbank.de/): 2\nMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0: 2\nMozilla/5.0 (X11; U; FreeBSD x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16: 2\nMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36: 2\nOpera/9.80 (Windows NT 6.1); U) Presto/2.10.289 Version/12.02: 2\nMozilla/5.0 (compatible; MSIE 9.0; Windows NT 5.1; Trident/5.0; .NET CLR 2.0.50727; .NET CLR 3.5.30729): 2\nMozilla/5.0 (Windows NT 6.2; rv:24.0) Gecko/20100101 Firefox/24.0: 2\nMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/5.0; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E): 2\nMozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0: 2\nMozilla/5.0 (Windows NT 5.1; rv:28.0) Gecko/20100101 Firefox/28.0: 2\nMozilla/5.0 (Windows NT 6.0; rv:29.0) Gecko/20120101 Firefox/29.0: 2\nMozilla/5.0 (Windows NT 6.0; rv:31.0) Gecko/20100101 Firefox/31.0: 2\nMozilla/5.0 (Windows NT 6.2; rv:31.0) Gecko/20100101 Firefox/31.0: 2\nMozilla/4.0 (compatible; Netcraft Web Server Survey): 2\nMozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MAGWJS; rv:11.0) like Gecko: 1\nMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36: 1\nMozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.0.2) Gecko/20121223 Ubuntu/9.25 (jaunty) Firefox/3.8: 1\nMozilla/5.0 (Windows NT 10.0; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0: 1\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36: 1\nOpera/9.80 (Windows NT 6.2; WOW64); U) Presto/2.12.388 Version/12.14: 1\nMozilla/5.0 (Windows NT 6.1; rv:33.0) Gecko/20100101 Firefox/33.0: 1\nMozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729): 1\nScrapy/1.0.3 (+http://scrapy.org): 1\nMozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows 95) Opera 7.03 [de]: 1\nTelesphoreo: 1\nMozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0: 1\nMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36: 1\nMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3.1.1) Gecko/20101203 Firefox/3.6.12 (.NET CLR 3.5.30309): 1\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 Scanning for research (researchscan.comsys.rwth-aachen.de): 1\nMozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20130405 Firefox/22.0: 1\nMozilla/5.0 (Windows; U; Windows NT 5.0; fr-FR; rv:0.9.4) Gecko/20011019 Netscape6/6.2: 1\nMozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/31.0: 1\nfacebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php): 1\nMozilla/3.0 (compatible; Indy Library): 1\nMozilla/5.0 (X11; Ubuntu; Linux i686; rv:42.0) Gecko/20100101 Firefox/42.0: 1\nMozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0: 1\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.59.10 (KHTML, like Gecko) Version/5.1.9 Safari/534.59.10: 1\nMozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko: 1\nNokiaE5-00/SymbianOS/9.1 Series60/3.0 3gpp-gba: 1\nMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36: 1\nMozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36: 1\n"
    }
   ],
   "source": [
    "UserAgents = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if agent in UserAgents:\n",
    "                UserAgents[agent] = UserAgents[agent] + 1\n",
    "            else:\n",
    "                UserAgents[agent] = 1\n",
    "\n",
    "results = sorted(UserAgents, key=lambda i: int(UserAgents[i]), reverse=True)\n",
    "\n",
    "for result in results:\n",
    "    print(result + \": \" + str(UserAgents[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes! In addition to '-', there are also a million different web robots accessing the site and polluting my data. Filtering out all of them is really hard, but getting rid of the ones significantly polluting my data in this case should be a matter of getting rid of '-', anything containing \"bot\" or \"spider\", and W3 Total Cache."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "/: 77\n/orlando-headlines/: 36\n/?page_id=34248: 28\n/wp-content/cache/minify/000000/M9AvyUjVzUstLy7PLErVz8lMKkosqtTPKtYvTi7KLCgpBgA.js: 27\n/wp-content/cache/minify/000000/M9bPKixNLarUy00szs8D0Zl5AA.js: 27\n/wp-content/cache/minify/000000/lY7dDoIwDIVfiG0KxkfxfnbdKO4HuxICTy-it8Zw15PzfSftzPCckJem-x4qUWArqBPl5mygZLEgyhdOaoxToGyGaiALiOfUnIz0qDLOdSZGE-nOlpc3kopDzrSyavVVt_veb5qSDVhjsQ6dHh_B_eE_z2pYIGJ7iBWKeEio_eT9UQe4xHhDll27mGRryVu_pRc.js: 27\n/wp-content/cache/minify/000000/fY45DoAwDAQ_FMvkRQgFA5ZyWLajiN9zNHR0O83MRkyt-pIctqYFJPedKyYzfHg2PzOFiENAzaD07AxcpKmTolORvDjZt8KEfhBUGjZYCf8Fb0fvA1TXCw.css: 25\n/?author=1: 21\n/wp-content/cache/minify/000000/hcrRCYAwDAXAhXyEjiQ1YKAh4SVSx3cE7_uG7ASr4M9qg3kGWyk1adklK84LHtRj_My6Y0Pfqcz-AA.js: 20\n/wp-content/uploads/2014/11/nhn1.png: 19\n/wp-includes/js/wp-emoji-release.min.js?ver=4.3.1: 17\n/wp-content/cache/minify/000000/BcGBCQAgCATAiUSaKYSERPk3avzuht4SkBJnt4tHJdqgnPBqKldesTcN1R8.js: 17\n/wp-login.php: 16\n/comics-2/: 12\n/world/: 12\n/favicon.ico: 10\n/wp-content/uploads/2014/11/violentcrime.jpg: 6\n/robots.txt: 6\n/wp-content/uploads/2014/11/garfield.jpg: 6\n/wp-content/uploads/2014/11/babyblues.jpg: 6\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (action == 'GET'):\n",
    "                        if URL in URLCounts:\n",
    "                            URLCounts[URL] = URLCounts[URL] + 1\n",
    "                        else:\n",
    "                            URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, our new problem is that we're getting a bunch of hits on things that aren't web pages. We're not interested in those, so let's filter out any URL that doesn't end in / (all of the pages on my site are accessed in that manner - again this is applying knowledge about my data to the analysis!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "/: 77\n/orlando-headlines/: 36\n/comics-2/: 12\n/world/: 12\n/weather/: 4\n/australia/: 4\n/about/: 4\n/national-headlines/: 3\n/feed/: 2\n/sample-page/feed/: 2\n/science/: 2\n/technology/: 2\n/entertainment/: 1\n/san-jose-headlines/: 1\n/business/: 1\n/travel/feed/: 1\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (URL.endswith(\"/\")):\n",
    "                        if (action == 'GET'):\n",
    "                            if URL in URLCounts:\n",
    "                                URLCounts[URL] = URLCounts[URL] + 1\n",
    "                            else:\n",
    "                                URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is starting to look more believable! But if you were to dig even deeper, you'd find that the /feed/ pages are suspect, and some robots are still slipping through. However, it is accurate to say that Orlando news, world news, and comics are the most popular pages accessed by a real human on this day.\n",
    "\n",
    "The moral of the story is - know your data! And always question and scrutinize your results before making decisions based on them. If your business makes a bad decision because you provided an analysis of bad source data, you could get into real trouble.\n",
    "\n",
    "Be sure the decisions you make while cleaning your data are justifiable too - don't strip out data just because it doesn't support the results you want!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results still aren't perfect; URL's that include \"feed\" aren't actually pages viewed by humans. Modify this code further to strip out URL's that include \"/feed\". Even better, extract some log entries for these pages and understand where these views are coming from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "/: 77\n/orlando-headlines/: 36\n/comics-2/: 12\n/world/: 12\n/weather/: 4\n/australia/: 4\n/about/: 4\n/national-headlines/: 3\n/science/: 2\n/technology/: 2\n/entertainment/: 1\n/san-jose-headlines/: 1\n/business/: 1\n"
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (URL.endswith(\"/\")):\n",
    "                        if(not('/feed' in URL)):\n",
    "                            if (action == 'GET'):\n",
    "                                if URL in URLCounts:\n",
    "                                    URLCounts[URL] = URLCounts[URL] + 1\n",
    "                                else:\n",
    "                                    URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "GET\n/feed/\nHTTP/1.1\n\nGET\n/sample-page/feed/\nHTTP/1.1\n\nGET\n/feed/\nHTTP/1.1\n\nGET\n/sample-page/feed/\nHTTP/1.1\n\nGET\n/travel/feed/\nHTTP/1.1\n\n"
    }
   ],
   "source": [
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (URL.endswith(\"/\")):\n",
    "                        if('/feed' in URL):\n",
    "                            if (action == 'GET'):\n",
    "                                print(action+'\\n'+URL+'\\n'+protocol+'\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the log file for feed page looks like(through file manager)\n",
    "\n",
    "**46.166.139.20 - - [29/Nov/2015:22:39:44 +0000] \"GET /feed/ HTTP/1.1\" 200 17592 \"-\" \"-\"**\n",
    "\n",
    "So it confirms that this is nothing but malcious data by seeing the \"-\" user agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}