Executive summary: My goal for this project was to collect data from various chocolate makers and distributors in France, and eventually build a comprehensive dataset about the chocolate online market in France.
Methodology:
- Web scraping: I scraped product data from several French e-commerce websites (chocolate section), using Python scrapy and requests libraries.
- Python: I build an additional script to:
- do some minor transformations to the .csv files resulting from web scraping (data cleaning)
- automatically load the resulting dataframes in BigQuery using the bigquery.client
- GCP BigQuery: I created an account and an empty project in GCP.
- dbt: I created a dbt Cloud project that is connected to my BigQuery project. In this dbt project, I created 3 sections:
- Bronze (staging): raw data, with few modifications.
- Silver (transformations): the raw data with some transformations (new columns, filters, etc)
- Gold (final): final datasets that will be used for analysis. Thanks to dbt, I could 'export' the SQL views and tables to GCP BigQuery. I also managed to create the data lineage, QoD tests and documentation. All dbt modifications were saved thanks to dbt-Github.
- PowerBI: Finally, I could plug my Gold final datasets to Power BI to create a visual overview of the market.
Final output
- A comprehensive dataset about the French chocolates online market
- A visual Power BI dashboard
Go further To learn more about the project, you may read my non technical article here: https://margot-marchais-maurice.webflow.io/chocolate-french-market
Technical learnings: I did this project to help me acquire new skills such as: build a datawharehouse in GCP Bigquery, automatically feed this DWH with scraped data thanks to a Python script, learn how to use dbt (data transformations, tests and docs generation),... It also made me refresh my web scraping skills (scrapy and requests libraries).
Aperçu du dashboard:
Brands positioning:
