Skip to content

COM6012 Scalable Machine Learning - University of Sheffield

Notifications You must be signed in to change notification settings

SinaTabakhi/ScalableML

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

COM6012 Scalable Machine Learning - University of Sheffield

Spring 2022 by Haiping Lu and Mauricio A Álvarez

In this module, we will learn how to do machine learning at large scale using Apache Spark. We will use the High Performance Computing (HPC) cluster systems of our university. If you are NOT on the University's network, you must use VPN (Virtual Private Network) to connect to the HPC.

This edition uses PySpark 3.2.1, the latest stable release of Spark (Jan 26, 2022), and has 10 sessions below. You can refer to the overview slides for more information, e.g. timetable and assessment information.

  • Session 1: Introduction to Spark and HPC
  • Session 2: RDD, DataFrame, ML pipeline, & parallelization
  • Session 3: Scalable decision trees and ensemble models
  • Session 4: Scalable logistic regression
  • Session 5: Scalable generalized linear models
  • Session 6: Scalable neural networks
  • Session 7: Scalable matrix factorisation for collaborative filtering in recommender systems
  • Session 8: Scalable k-means clustering and Spark configuration
  • Session 9: Scalable PCA for dimensionality reduction and Spark data types
  • Session 10: Apache Spark in the Cloud (guest lecture by Dr Michael Smith)

You can also download the Spring 2021 version for preview or reference.

Acknowledgement

The materials are built with references to the following sources:

Many thanks to

  • Mike Croucher, Neil Lawrence, Will Furnass, Twin Karmakharm, and Vamsi Sai Turlapati for their inputs and inspirations since 2016.
  • Our teaching assistants and students who have contributed in many ways since 2017.

About

COM6012 Scalable Machine Learning - University of Sheffield

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HTML 63.6%
  • Python 22.3%
  • Shell 14.1%