Web-scrapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping with BeautifulSoup : Error handling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Installing Beautifulsoup python libraries**\n",
    "  \n",
    "  BeautifulSoup is a Python library for extracting data from HTML files. It is widely used in Web Scraping to extract information from web pages, and creates an analysis tree from HTML documents. This feature makes web page content more readable than what we've seen with Requests. In practice, to analyze these HTML files, BeautifulSoup uses parsers, which are software libraries used to analyze and extract data from HTML documents. The parser transforms the HTML code into a tree-like representation, called a DOM (Document Object Model) tree, facilitating the navigation and extraction of specific data.\n",
    "\n",
    "  There are several parsers available, each with its own advantages and disadvantages. The three most commonly used parsers are:\n",
    "  - **`html.parser`**: `html.parser` is Python's default parser, which means no additional installation is required. Simple enough to use for basic parsing tasks, it is however slower than `lxml` and `html5lib`. Also, this parser is not very tolerant of syntax errors, and may have difficulty parsing complex `HTML` files. To use html.parser, you need to specify it as an optional argument when creating the `BeautifulSoup` object.\n",
    "  - **`html5lib`**: `html5lib` is a relatively slow parser compared to some other `HTML` parsers. This is due to its exhaustive parsing approach and `HTML5` compatibility, which may result in slightly slower performance. The `html5lib` library can have a relatively larger memory footprint due to its comprehensive design and support for various `HTML5` features. It is a more complete library, but performs less well.\n",
    "  - **`lxml`:** `lxml` is a faster, more syntax-tolerant parser than html.parser. It is also capable of parsing `XML` files, in addition to `HTML` files. However, it must be installed separately. This is the parser we recommend you use.\n",
    "\n",
    "  Once we've created an instance of the `BeautifulSoup` class with the contents of our file, commonly known as `soup`, we can use various functions to extract data from the page.\n",
    "\n",
    "  <img src=\"https://assets-datascientest.s3.eu-west-1.amazonaws.com/133_webscraping/soup.png\"  width=\"600\" height=\"300\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting beautifulsoup4\n",
      "  Downloading beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)\n",
      "     ---------------------------------------- 0.0/143.0 kB ? eta -:--:--\n",
      "     -- ------------------------------------- 10.2/143.0 kB ? eta -:--:--\n",
      "     ------- ----------------------------- 30.7/143.0 kB 330.3 kB/s eta 0:00:01\n",
      "     ------------------ ------------------ 71.7/143.0 kB 563.7 kB/s eta 0:00:01\n",
      "     ------------------------------------ 143.0/143.0 kB 948.1 kB/s eta 0:00:00\n",
      "Collecting soupsieve>1.2 (from beautifulsoup4)\n",
      "  Obtaining dependency information for soupsieve>1.2 from https://files.pythonhosted.org/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl.metadata\n",
      "  Downloading soupsieve-2.5-py3-none-any.whl.metadata (4.7 kB)\n",
      "Downloading soupsieve-2.5-py3-none-any.whl (36 kB)\n",
      "Installing collected packages: soupsieve, beautifulsoup4\n",
      "Successfully installed beautifulsoup4-4.12.2 soupsieve-2.5\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.2.1 -> 23.3.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "%pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Installing an additional parser**\n",
    "\n",
    "As you probably know by now, BeautifulSoup uses a parser to analyze documents. By default, Python has a built-in parser called html.parser. However, some other parsers can be used for specific tasks, such as lxml, which is commonly recommended. To install lxml on your own machines, use the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting lxml\n",
      "  Obtaining dependency information for lxml from https://files.pythonhosted.org/packages/5f/df/6d15cc415e04724ba4c141051cf43709e09bbcdd9868a6c2e7a7073ef498/lxml-4.9.4-cp312-cp312-win_amd64.whl.metadata\n",
      "  Downloading lxml-4.9.4-cp312-cp312-win_amd64.whl.metadata (3.8 kB)\n",
      "Downloading lxml-4.9.4-cp312-cp312-win_amd64.whl (3.8 MB)\n",
      "   ---------------------------------------- 0.0/3.8 MB ? eta -:--:--\n",
      "   ---------------------------------------- 0.0/3.8 MB ? eta -:--:--\n",
      "   ---------------------------------------- 0.0/3.8 MB 435.7 kB/s eta 0:00:09\n",
      "   - -------------------------------------- 0.1/3.8 MB 1.2 MB/s eta 0:00:04\n",
      "   ----- ---------------------------------- 0.5/3.8 MB 3.0 MB/s eta 0:00:02\n",
      "   -------- ------------------------------- 0.8/3.8 MB 4.1 MB/s eta 0:00:01\n",
      "   ------------ --------------------------- 1.2/3.8 MB 4.7 MB/s eta 0:00:01\n",
      "   ---------------- ----------------------- 1.5/3.8 MB 5.1 MB/s eta 0:00:01\n",
      "   ------------------- -------------------- 1.9/3.8 MB 5.4 MB/s eta 0:00:01\n",
      "   ----------------------- ---------------- 2.2/3.8 MB 5.6 MB/s eta 0:00:01\n",
      "   -------------------------- ------------- 2.5/3.8 MB 5.8 MB/s eta 0:00:01\n",
      "   ------------------------------ --------- 2.9/3.8 MB 5.9 MB/s eta 0:00:01\n",
      "   ---------------------------------- ----- 3.2/3.8 MB 6.0 MB/s eta 0:00:01\n",
      "   ------------------------------------- -- 3.6/3.8 MB 6.2 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 3.8/3.8 MB 6.0 MB/s eta 0:00:00\n",
      "Installing collected packages: lxml\n",
      "Successfully installed lxml-4.9.4\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.2.1 -> 23.3.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "%pip install lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in c:\\users\\hp\\appdata\\local\\programs\\python\\python312\\lib\\site-packages (2.31.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\hp\\appdata\\local\\programs\\python\\python312\\lib\\site-packages (from requests) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\hp\\appdata\\local\\programs\\python\\python312\\lib\\site-packages (from requests) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\hp\\appdata\\local\\programs\\python\\python312\\lib\\site-packages (from requests) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\hp\\appdata\\local\\programs\\python\\python312\\lib\\site-packages (from requests) (2023.7.22)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.2.1 -> 23.3.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "%pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BeautifulSoup's functions**\n",
    "\n",
    "To extract data from an `HTML` file, we first need to target specific `HTML` elements. To do this, we can use the functions of the `BeautifulSoup` library to select elements in the `HTML` source code.\n",
    "\n",
    "**The find() function:**\n",
    "The `find()` function searches for the first occurrence of an `HTML` element corresponding to the specified criterion. It takes as parameters the name of the `HTML` tag and, if required, a dictionary of attributes. For example, if you want to find the first paragraph (`<p>` tag) on the page with the class \"my-class\", you can use the following function: `soup.find('p', class_ = \"my-class\"})`\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "<i class=\"fa fa-info-circle\"></i> \n",
    "    You may notice that we use <code>class_</code> instead of <code>class</code> in <code>BeautifulSoup</code>. <code>class</code> is a keyword reserved in Python for defining new classes, a type of customizable object. These words cannot be used as variable, function or attribute names. In this way, <code>class_</code> avoids any conflict or ambiguity with the Python interpreter.\n",
    "\n",
    "> Occasionally, you may come across a different syntax for the `find()` function, notably with the use of the '.' accessor. This is an abbreviated function for accessing the first tag, equivalent to `find()`, but its use depends on personal preference and coding style.\n",
    "> Thus, the following two codes are equivalent:\n",
    "> <br /> \n",
    ">\n",
    ">`soup.div.div.a`\n",
    ">\n",
    ">`soup.find('div').find('div').find('a')`\n",
    ">\n",
    ">> They display the first link of the `HTML` soup document hosted in two `div`.\n",
    "> \n",
    "> In addition, we can use several optional arguments with this `find()` function to specify our search based on the elements at our disposal. We can specify attributes such as class name, unique id or any other attribute using the `attrs` parameter in the form of a dictionary that can take one or more attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)** Run the following code cell to see a concrete example of using the `find()` function with different optional arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alan Turing - Wikipedia\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "url = 'https://en.wikipedia.org/wiki/Alan_Turing'\n",
    "\n",
    "response = requests.get(url)\n",
    "soup = BeautifulSoup(response.content, 'lxml')\n",
    "title = soup.title.text\n",
    "print(title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Alan Turing - Wikipedia'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "title = soup.title.text\n",
    "title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Initial content \n",
      " A second English content \n"
     ]
    }
   ],
   "source": [
    "divs = soup.find_all('div', class_ ='content')\n",
    "for each_div in divs:\n",
    "    print(each_div.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Title 1 \n",
      " Title 2  , is the next level 1 title after first_title\n"
     ]
    }
   ],
   "source": [
    "first_title = soup.find('h1', id=\"first\")\n",
    "second_title = first_title.find_next('h1') # We search for the next level 1 title (tag h1) after first_title\n",
    "print(first_title.text)\n",
    "print(second_title.text, \", is the next level 1 title after first_title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Element 1 \n",
      " Element 2  , is the first 'sibling element' after first_item\n",
      " Element 3  , is the first 'sibling element' after second_item\n"
     ]
    }
   ],
   "source": [
    "first_item = soup.find('li')\n",
    "second_item = first_item.find_next_sibling('li')\n",
    "third_item = second_item.find_next_sibling('li') # equivalent to first_item.find_next_sibling('li').find_next_sibling('li')\n",
    "\n",
    "print(first_item.text)\n",
    "print(second_item.text, \", is the first 'sibling element' after first_item\")\n",
    "print(third_item.text, \", is the first 'sibling element' after second_item\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Element .chip class:   Element 1 \n",
      " Element .chip class:   Element 2 \n",
      " Element .chip class:   Element 3 \n",
      " Title 2 \n",
      " A new paragraph \n"
     ]
    }
   ],
   "source": [
    "elements_css_class = soup.select('.chip')     # an HTML element whose class ('.' in css) is \"chip\".\n",
    "elements_css_id = soup.select('#second')      # an HTML element whose id ('#' in css) is \"second\".\n",
    "elements_parent_children = soup.select_one('div > p') # the first paragraph inside a div\n",
    "\n",
    "\n",
    "for element in elements_css_class : # browses the list of tags affected by the 'chip' class\n",
    "    print(\" Element .chip class: \", element.text)\n",
    "    \n",
    "print(elements_css_id[0].text)      # We access to the 1st element of the returned list\n",
    "print(elements_parent_children.text)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "element_by_id :   Unique main content \n",
      "element_by_class :   Initial content \n",
      "element_by_attrs :   A second English content \n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup as bs\n",
    "\n",
    "code_source = '''\n",
    "<html>\n",
    "  <body>\n",
    "    <h1 id=\"first\"> Title 1 </h1>\n",
    "    <div id=\"main-content\"> Unique main content </div>\n",
    "    <div class=\"content\"> Initial content </div>\n",
    "    <div class=\"content\" data-lang=\"en\"> A second English content </div>\n",
    "    <h1 id=\"second\"> Title 2 </h1>\n",
    "    <ul id=\"lists\">\n",
    "        <li class=\"chip\"> Element 1 </li>\n",
    "        <li class=\"chip\"> Element 2 </li>\n",
    "        <li class=\"chip\"> Element 3 </li>\n",
    "    </ul>\n",
    "    <div> \n",
    "        <p class=\"paragraph\"> A new paragraph </p>\n",
    "    </div>\n",
    "  </body>\n",
    "</html>\n",
    "'''\n",
    "\n",
    "soup = bs(code_source, 'html.parser')\n",
    "element_by_id = soup.find('div', id= 'main-content')\n",
    "element_by_class = soup.find('div', class_= 'content')\n",
    "element_by_attrs = soup.find('div', attrs={'class': 'content', 'data-lang': 'en'})\n",
    "\n",
    "print(\"element_by_id : \",element_by_id.text)\n",
    "print(\"element_by_class : \",element_by_class.text)\n",
    "print(\"element_by_attrs : \",element_by_attrs.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comprehensive list of Beautiful Soup methods for web scraping:\n",
    "\n",
    "**1. find():**\n",
    "\n",
    "The `find()` method finds the first instance of a specific element or string within the parsed HTML document.\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "soup.find(tag, attributes, recursive, text, **kwargs)\n",
    "```\n",
    "\n",
    "**Parameters:**\n",
    "\n",
    "* `tag`: The tag name of the element you want to find.\n",
    "* `attributes`: A dictionary of attributes and their values to match.\n",
    "* `recursive`: A Boolean value that indicates whether to search recursively within the document tree.\n",
    "* `text`: The text content of the element you want to find.\n",
    "* `**kwargs`: Keyword arguments for more advanced filtering options.\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "```python\n",
    "# Find the first `<a>` element with the `href` attribute containing \"google.com\"\n",
    "link = soup.find('a', href='google.com')\n",
    "\n",
    "# Find the first `<div>` element with the class \"content\"\n",
    "content_div = soup.find('div', class_='content')\n",
    "\n",
    "# Find the first `<p>` element whose text contains \"Beautiful Soup\"\n",
    "paragraph = soup.find('p', text='Beautiful Soup')\n",
    "```\n",
    "\n",
    "**2. find_all():**\n",
    "\n",
    "The `find_all()` method returns a list of all instances of a specific element or string within the parsed HTML document.\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "soup.find_all(tag, attributes, recursive, text, **kwargs)\n",
    "```\n",
    "\n",
    "**Parameters:**\n",
    "\n",
    "Same as `find()`, but returns a list instead of a single element.\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "```python\n",
    "# Find all `<a>` elements within the document\n",
    "links = soup.find_all('a')\n",
    "\n",
    "# Find all `<div>` elements with the class \"content\"\n",
    "content_divs = soup.find_all('div', class_='content')\n",
    "\n",
    "# Find all `<p>` elements whose text contains \"Beautiful Soup\"\n",
    "paragraphs = soup.find_all('p', text='Beautiful Soup')\n",
    "```\n",
    "\n",
    "**3. select():**\n",
    "\n",
    "The `select()` method uses CSS selectors to find elements within the parsed HTML document.\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "soup.select(selector)\n",
    "```\n",
    "\n",
    "**Parameter:**\n",
    "\n",
    "A CSS selector string.\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "```python\n",
    "# Find all `<a>` elements with the class \"link\" and the href attribute containing \"google.com\"\n",
    "links = soup.select('a.link[href=\"google.com\"]')\n",
    "\n",
    "# Find all `<div>` elements with the class \"content\"\n",
    "content_divs = soup.select('div.content')\n",
    "\n",
    "# Find all `<p>` elements whose text contains \"Beautiful Soup\"\n",
    "paragraphs = soup.select('p:contains(\"Beautiful Soup\")')\n",
    "```\n",
    "\n",
    "**4. select_one():**\n",
    "\n",
    "The `select_one()` method applies a CSS selector and returns the first matching element, similar to `find()`.\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "soup.select_one(selector)\n",
    "```\n",
    "\n",
    "**Parameter:**\n",
    "\n",
    "Same as `select()`, but returns a single element instead of a list.\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "```python\n",
    "# Find the first `<a>` element with the class \"link\" and the href attribute containing \"google.com\"\n",
    "link = soup.select_one('a.link[href=\"google.com\"]')\n",
    "\n",
    "# Find the first `<div>` element with the class \"content\"\n",
    "content_div = soup.select_one('div.content')\n",
    "\n",
    "# Find the first `<p>` element whose text contains \"Beautiful Soup\"\n",
    "paragraph = soup.select_one('p:contains(\"Beautiful Soup\")')\n",
    "```\n",
    "\n",
    "**5. descendants():**\n",
    "\n",
    "The `descendants()` method returns a list of all descendant elements within the specified element.\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "element.descendants()\n",
    "```\n",
    "\n",
    "**Parameter:**\n",
    "\n",
    "The element to search for descendants.\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "```python\n",
    "# Find all descendant `<span>` elements within the `<div>` element with the class \"content\"\n",
    "spans = content_div.descendants('span')\n",
    "\n",
    "# Find all descendant elements (excluding text content) within the `<p>` element containing \"Beautiful Soup\"\n",
    "elements = paragraph.descendants('*')\n",
    "```\n",
    "\n",
    "**6. parents():**\n",
    "\n",
    "The `parents()` method returns a list of all parent elements of the specified element"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}