geopandas_reading_files.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Reading files with Geopandas\n",
        "\n",
        "Before doing anything else, we need to import the geopandas package!\n",
        "\n",
        "The geopandas team don't recommend using an alias (i.e. we're not going to shorten the way we refer to geopandas).\n"
      ],
      "id": "b91548b2"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import geopandas"
      ],
      "id": "7e2ab2e4",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Importing geojsons, geopackages or shape files\n",
        "\n",
        "When working with prepackaged geographic data types, they will usually be stored in the GeoJSON format, the geopackage (gpkg) format, or as a .shp file.\n",
        "\n",
        ":::{.callout-warning}\n",
        "Shapefiles are a little more complex as they are a number of files with different extensions that all need to be distributed together - even though it's only the file with the extension '.shp' that we read in.\n",
        "\n",
        "Geojson and geopackages are often easier to distribute and download!\n",
        ":::\n",
        "\n",
        "### Files stored locally\n",
        "\n",
        "You can refer to a range of geographic data file types stored locally.\n"
      ],
      "id": "57f3969a"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "countries_gdf = geopandas.read_file(\"package.gpkg\")"
      ],
      "id": "26209416",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Files stored on the web\n",
        "\n",
        "You can also directly refer to files stored on the web.\n"
      ],
      "id": "17673bfd"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "df = geopandas.read_file(\"http://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_110m_land.geojson\")"
      ],
      "id": "e0cba971",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Zipped files\n",
        "\n",
        "You can also directly refer to files stored as zip files by prefixing the file path with zip:///.\n"
      ],
      "id": "45f7b792"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "states = geopandas.read_file(\"zip:///Users/name/Downloads/cb_2017_us_state_500k.zip\")"
      ],
      "id": "73dbbff3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can read more about file imports in the geopandas documentation, which is embedded below.\n",
        "\n",
        "\n",
        "```{=html}\n",
        "<iframe width=\"780\" height=\"500\" src=\"https://geopandas.org/en/stable/docs/user_guide/io.html\" title=\"IO in geopandas\"></iframe>\n",
        "```\n",
        "\n",
        "\n",
        "# Exploring geopandas dataframes\n",
        "\n",
        "Once you’ve read it in, it looks a lot like a pandas dataframe!\n",
        "\n",
        "Even better, you can do all of your normal pandas commands with it - like ‘head’ to view the first 5 rows.\n"
      ],
      "id": "68815ca9"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import geopandas\n",
        "\n",
        "crime_figures = geopandas.read_file(\"https://github.com/hsma-programme/h6_3b_advanced_qgis_mapping_python/raw/main/h6_3b_advanced_qgis_and_mapping_in_python/example_code/lsoa_2011_sw5forces_crime_figures.gpkg\")\n",
        "\n",
        "crime_figures.head()"
      ],
      "id": "88f64bb9",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When we check the type of the dataframe, we will see that it has come through as a GeoDataFrame\n"
      ],
      "id": "75b48703"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "type(crime_figures)"
      ],
      "id": "e5fd6b85",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Turning existing data into a GeoDataFrame\n",
        "\n",
        "However - a lot of the time you may be extracting data from your data warehouse and turning this into a geodataframe.\n",
        "\n",
        "Let’s go back to our crime dataset from the QGIS section.\n"
      ],
      "id": "dbf6807f"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "\n",
        "sw_stop_search_5_forces = pd.read_csv(\"https://github.com/hsma-programme/h6_3b_advanced_qgis_mapping_python/raw/main/h6_3b_advanced_qgis_and_mapping_in_python/example_code/sw5forces_stop_and_search.csv\")\n",
        "\n",
        "# view the first row\n",
        "sw_stop_search_5_forces.head(1)"
      ],
      "id": "7363c69b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we’ve imported it as a csv - but if we’d extracted data from a database and saved it as pandas dataframe, the following steps would be the same!\n",
        "\n",
        "So let's just check the type first.\n"
      ],
      "id": "bdd52aa6"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "type(sw_stop_search_5_forces)"
      ],
      "id": "3627d522",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First, we need to know what the columns that identify the geometry are.\n",
        "\n",
        "In this case, they are ‘Latitude’ and ‘Longitude’\n",
        "\n",
        "We can now construct a geopandas geodataframe from this .csv file.\n"
      ],
      "id": "232d793a"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "sw_5forces_stop_and_search_gdf = geopandas.GeoDataFrame(\n",
        "    sw_5forces_stop_and_search_df, # Our pandas dataframe\n",
        "    geometry = geopandas.points_from_xy(\n",
        "        sw_5forces_stop_and_search_df['Longitude'], # Our 'x' column (horizontal position of points)\n",
        "        sw_5forces_stop_and_search_df['Latitude'] # Our 'y' column (vertical position of points)\n",
        "        ),\n",
        "    crs = 'EPSG:4326' # the coordinate reference system of the data - use EPSG:4326 if you are unsure\n",
        "    )"
      ],
      "id": "23efcb19",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's view this new object.\n"
      ],
      "id": "8200c724"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "sw_5forces_stop_and_search_gdf.head()"
      ],
      "id": "e02e1a54",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "And let's view the type of object it is.\n"
      ],
      "id": "3380b6e8"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "type(sw_5forces_stop_and_search_gdf)"
      ],
      "id": "bf5b831a",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Joining area data to boundary data\n",
        "\n",
        "We can also combine pandas dataframes with geopandas dataframes.\n",
        "\n",
        "When might we want to do this?\n",
        "\n",
        "Imagine we have a dataset of patients who are using a particular type of service.\n",
        "\n",
        "We can use pandas to count the number of patients per LSOA.\n",
        "\n",
        "However - the LSOA code alone isn’t going to allow us to plot this dataset - it doesn’t contain the geometry.\n",
        "\n",
        "Instead, we\n",
        "\n",
        "- import a shapefile, geoJSON or geopackage of boundaries\n",
        "- join it to our pandas dataframe using a common column (like LSOA code)\n",
        "\n",
        "If we join our dataframe to our geodataframe, the result will be a geodataframe - so you can make use of all the useful features of geodataframes.\n"
      ],
      "id": "32b96d28"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| eval: false\n",
        "\n",
        "my_lsoa_boundary_gdf = geopandas.read_file(\"lsoa_boundaries.gpkg\")\n",
        "\n",
        "my_count_df = pd.read_csv(“counts_by_lsoa.csv”)"
      ],
      "id": "c3ce3610",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let’s imagine the geodataframe has a column called ‘LSOA11CD’\n",
        "\n",
        "The count dataframe has a column called ‘LSOA’\n"
      ],
      "id": "21d2e96d"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "#| eval: false\n",
        "\n",
        "my_final_df = pd.merge(\n",
        "\n",
        "\tleft=my_lsoa_boundary_gdf,\n",
        "right=my_count_df,\n",
        "left_on=”LSOA11CD”\n",
        "right_on=”LSOA”\n",
        "how=”right”\n",
        "\n",
        ")"
      ],
      "id": "b14a096b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ":::{.callout-warning}\n",
        "\n",
        "We need to be careful about the order we join things in to ensure we end up with the right type of object at the end.\n",
        "\n",
        "> “The stand-alone merge function will work if the GeoDataFrame is in the left argument; if a DataFrame is in the left argument and a GeoDataFrame is in the right position, the result will no longer be a GeoDataFrame.” - https://geopandas.org/en/v0.8.0/mergingdata.html\"\n",
        "\n",
        "This would result in a geodataframe:\n"
      ],
      "id": "51f73a77"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "my_final_df = pd.merge(\n",
        "\n",
        "\tleft=my_lsoa_boundary_gdf,\n",
        "right=my_count_df,\n",
        "left_on=”LSOA11CD”\n",
        "right_on=”LSOA”\n",
        "how=”right”\n",
        ")"
      ],
      "id": "ec970c2c",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "But this would not.\n"
      ],
      "id": "1e7c28d4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "my_final_df = pd.merge(\n",
        "\n",
        "\n",
        "left=my_count_df,\n",
        "left=my_lsoa_boundary_gdf,\n",
        "left_on=”LSOA”\n",
        "right_on=”LSOA11CD”\n",
        "how=\"left\"\n",
        "\n",
        ")"
      ],
      "id": "279434aa",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ":::\n",
        "\n",
        "\n",
        ":::{.callout-tip}\n",
        "#### The 'how' argument\n",
        "\n",
        "The ‘how’ argument\n",
        "\n",
        "If you set how = ‘left’, all of the rows from the geodataframe will be kept, even if there is no value in your dataframe of counts\n",
        "\n",
        "If you set how = ‘right’, all of the rows from the counts dataframe will be kept, even if there is no value in your geodataframe\n",
        "Check you have no missing values in the ‘geometry’ column after this!\n",
        "\n",
        "If you set how = ‘full’, all of the rows from both dataframes will be kept - so you may end up with empty geometry in some cases and/or empty counts in others\n",
        ":::"
      ],
      "id": "a4473b86"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3 (ipykernel)"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}