src/deep-demand-forecast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Demand Forecasting with Amazon SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Jupyter Kernel**:\n",
    "\n",
    "* If you are in SageMaker Notebook instance, please make sure you are using **conda_mxnet_p36** kernel\n",
    "* If you are in SageMaker Studio, please make sure you are using **SageMaker JumpStart MXNet 1.0** kernel\n",
    "\n",
    "**Run All**: \n",
    "\n",
    "* If you are in SageMaker Notebook instance, you can *go to Cell tab -> Run All*\n",
    "* If you are in SageMaker Studio, you can *go to Run tab -> Run All Cells*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import sagemaker\n",
    "\n",
    "session = sagemaker.Session()\n",
    "sagemaker_config = json.load(open(\"stack_outputs.json\"))\n",
    "role = sagemaker_config[\"IamRole\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy raw data to S3\n",
    "\n",
    "The dataset we use here is the multivariate time-series [electricity consumptions](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014) data taken from Dua, D. and Graff, C. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/index.php), Irvine, CA: University of California, School of Information and Computer Science. A cleaned version of the data containing **321** time-series with **1 Hour** frequency, starting from **2012-01-01** with **26304** time-steps, is available to download directly via [GluonTS](https://gluon-ts.mxnet.io/). We have also provided the [exchange rate](https://github.com/laiguokun/multivariate-time-series-data/tree/master/exchange_rate) dataset in case you want to try with other datasets as well.\n",
    "\n",
    "For the ease of access, with have made both of the cleaned datasets available in the following S3 bucket.\n",
    "\n",
    "**Note:** To reproduce the results from the [blog post](https://towardsdatascience.com/deep-demand-forecasting-with-amazon-sagemaker-e0226410763a) we use `DATASET_NAME='electricity'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_NAME = \"electricity\"\n",
    "NUM_TS = 321"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "original_data_bucket = \"sagemaker-solutions-prod-{}\".format(sagemaker_config[\"AWSRegion\"])\n",
    "original_data_prefix = \"sagemaker-deep-demand-forecast/data/{}\".format(DATASET_NAME)\n",
    "original_data = \"s3://{}/{}\".format(original_data_bucket, original_data_prefix)\n",
    "print(\"original data: \")\n",
    "S3Downloader.list(original_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, setup the S3 bucket name and prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = sagemaker_config[\"SolutionS3Bucket\"]\n",
    "prefix = \"tst\"  # example\n",
    "raw_data = \"s3://{}/{}\".format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the `original_data` to our `raw_data` if does not exist already"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "if not S3Downloader.list(raw_data):\n",
    "    !aws s3 cp --recursive $original_data $raw_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a few variables that will be used throughout the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessed_data = \"s3://{}/{}/processed_data\".format(bucket, prefix)\n",
    "train_data = preprocessed_data\n",
    "train_output = \"s3://{}/{}/output\".format(bucket, prefix)\n",
    "code_location = \"s3://{}/{}/code\".format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Container for Preprocessing and Feature Engineering\n",
    "\n",
    "Data preprocessing and feature engineering is an important component of the ML lifecycle. Amazon SageMaker Processing allows you to do these easily on a managed infrastructure. Here, we create a lightweight container that will serve as the environment for our data preprocessing. The container can also be easily customized to add more dependencies when needed.\n",
    "\n",
    "Build and registering the container below takes *less than 2 minutes*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "region = sagemaker_config['AWSRegion']\n",
    "account_id = sagemaker_config['AccountID']\n",
    "ecr_repository = sagemaker_config['SageMakerProcessingJobContainerName']\n",
    "container_build_project = sagemaker_config['SageMakerProcessingJobContainerBuild']\n",
    "\n",
    "if container_build_project == 'local':\n",
    "    !cd preprocess && bash container/local/build_and_push.sh $ecr_repository $region $account_id\n",
    "else:\n",
    "    import boto3\n",
    "    import container_build\n",
    "    container_build.build(container_build_project, boto3.session.Session(region_name=region))\n",
    "\n",
    "ecr_repository_uri = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account_id,\n",
    "                                                                    region,\n",
    "                                                                    ecr_repository)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Preprocessing job with Amazon SageMaker Processing\n",
    "\n",
    "The script `src/preprocess/preprocess.py` max-normalizes the training data (correctly) and uses the found scales to normalize the testing data. We use SageMaker `ScriptProcessor` to perform these transformations on the `raw_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ScriptProcessor\n",
    "\n",
    "script_processor = ScriptProcessor(\n",
    "    command=[\"python3\"],\n",
    "    image_uri=ecr_repository_uri,\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the processing job takes around *7 minutes* to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "script_processor.run(\n",
    "    code=\"preprocess/preprocess.py\",\n",
    "    inputs=[ProcessingInput(source=raw_data, destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[ProcessingOutput(destination=preprocessed_data, source=\"/opt/ml/processing/output\")],\n",
    "    arguments=[f\"--dataset-name={DATASET_NAME}\"],\n",
    "    logs=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Results of Data Preprocessing\n",
    "\n",
    "Once the preprocessing job is complete, we can take a look at the contents of the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "processed_files = S3Downloader.list(preprocessed_data)\n",
    "print(\"\\n\".join(processed_files))\n",
    "S3Downloader.download(preprocessed_data, preprocessed_data.split(\"/\")[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your LSTNet Model with GluonTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSTNet** is a Deep Learning model that incorporates traditional *auto-regressive* linear models *in parallel* to the non-linear neural network part, which makes the *non-linear* deep learning model more *robust* for the time series which *violate scale changes*. \n",
    "\n",
    "For more details, please checkout the paper [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://arxiv.org/abs/1703.07015)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters\n",
    "\n",
    "Here is a set of hyperparameters (from the paper) for LSTNet model to train for **5 epoch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTEXT_LENGTH = 12\n",
    "PREDICTION_LENGTH = 6\n",
    "\n",
    "hyperparameters = {\n",
    "    \"context_length\": CONTEXT_LENGTH,  # sliding window size for training\n",
    "    \"prediction_length\": PREDICTION_LENGTH,  # sliding window size for predicting\n",
    "    \"skip_size\": 4,  # skip size is used in skip-rnn layer\n",
    "    \"ar_window\": 4,  # auto-regressive window size\n",
    "    \"channels\": 72,  # number of convolution channels for the first layer\n",
    "    \"scaling\": False,  # whether to scale the data or not\n",
    "    \"output_activation\": \"sigmoid\",  # output activation function either None, sigmoid or tanh\n",
    "    \"epochs\": 5,  # number of epochs for training\n",
    "    \"batch_size\": 32,  # number of batch size\n",
    "    \"learning_rate\": 1e-2,  # learning rate for weight update\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and Fit SageMaker Estimator\n",
    "\n",
    "With the hyperparameters defined, we can execute the training job. We will be using the [GluonTS](https://gluon-ts.mxnet.io/), with **MXNet** as the backend deep learning framework, to define and train our *LSTNet* model. **Amazon SageMaker** makes it do this with the Framework estimators which have the deep learning frameworks already setup. Here, we create a SageMaker MXNet estimator and pass in our model training script, hyperparameters, as well as the number and type of training instances we want.\n",
    "\n",
    "We can then `fit` the estimator on the the training data location in S3.\n",
    "\n",
    "Note: training the estimator (for 5 epochs) takes around **10 minutes**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import logging\n",
    "from sagemaker.mxnet import MXNet\n",
    "\n",
    "estimator = MXNet(\n",
    "    entry_point=\"train.py\",\n",
    "    source_dir=\"deep_demand_forecast\",\n",
    "    role=role,\n",
    "    train_instance_count=1,\n",
    "    train_instance_type=\"ml.g4dn.2xlarge\",  # or 'ml.c4.2xlarge' without GPU\n",
    "    framework_version=\"1.6.0\",\n",
    "    py_version=\"py3\",\n",
    "    hyperparameters=hyperparameters,\n",
    "    output_path=train_output,\n",
    "    code_location=code_location,\n",
    "    sagemaker_session=session,\n",
    "    # container_log_level=logging.DEBUG,  # display debug logs\n",
    ")\n",
    "\n",
    "estimator.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the Training Evaluation\n",
    "\n",
    "We can now access the training artifacts from the specified `output_path` in the above estimator and visual the training results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_files = S3Downloader.list(train_output)\n",
    "print(\"\\n\".join(output_files))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "output_path = os.path.join(train_output, estimator._current_job_name, \"output\")\n",
    "\n",
    "S3Downloader.download(output_path, \"output\")\n",
    "!tar -xvf output/output.tar.gz -C output/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "item_metrics = pd.read_csv(\"output/item_metrics.csv.gz\", compression=\"gzip\")\n",
    "item_metrics.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Outputs\n",
    "\n",
    "For the visualization we will use [altair package](https://github.com/altair-viz/altair) with declarative API. If you want to export to different file formats, follow [altair_saver](https://github.com/altair-viz/altair_saver). \n",
    "\n",
    "Note that after exporting to `html` you can go to `output` and open the generated `html` files inside notebook.\n",
    "\n",
    "Here, we compare the [**Mean Absolute Scaled Error (MASE)**](https://en.wikipedia.org/wiki/Mean_absolute_scaled_error) against the [**symmetric Mean Absolute Percentage Error (sMAPE)**](https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "\n",
    "col_a = \"MASE\"\n",
    "col_b = \"sMAPE\"\n",
    "\n",
    "scatter = (\n",
    "    alt.Chart(item_metrics)\n",
    "    .mark_circle(size=100, fillOpacity=0.8)\n",
    "    .encode(\n",
    "        alt.X(col_a, scale=alt.Scale(domain=[-0.5, 10])),\n",
    "        alt.Y(col_b, scale=alt.Scale(domain=[0, 2.5])),\n",
    "        tooltip=[col_a, col_b],\n",
    "    )\n",
    "    .interactive()\n",
    ")\n",
    "scatter.save(os.path.join(\"output\", f\"{col_a}_vs_{col_b}.html\"))\n",
    "scatter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_a_plot = (\n",
    "    alt.Chart(item_metrics)\n",
    "    .mark_bar()\n",
    "    .encode(\n",
    "        alt.X(col_a, bin=True),\n",
    "        y=\"count()\",\n",
    "    )\n",
    ")\n",
    "col_b_plot = (\n",
    "    alt.Chart(item_metrics)\n",
    "    .mark_bar()\n",
    "    .encode(\n",
    "        alt.X(col_b, bin=True),\n",
    "        y=\"count()\",\n",
    "    )\n",
    ")\n",
    "\n",
    "col_a_b_plot = col_a_plot | col_b_plot\n",
    "col_a_b_plot.save(os.path.join(\"output\", f\"{col_a}_{col_b}_barplots.html\"))\n",
    "col_a_b_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy an Endpoint\n",
    "\n",
    "To serve the model, we can deploy an endpoint (which takes around **7 minutes**) where the `src/deep_demand_forecast/inference.py` script handles the predictions using the trained model as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sagemaker.mxnet import MXNetModel\n",
    "\n",
    "model = MXNetModel(\n",
    "    model_data=os.path.join(output_path, \"model.tar.gz\"),\n",
    "    role=role,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=\"deep_demand_forecast\",\n",
    "    py_version=\"py3\",\n",
    "    name=sagemaker_config[\"SolutionPrefix\"] + \"-model\",\n",
    "    framework_version=\"1.6.0\",\n",
    ")\n",
    "\n",
    "predictor = model.deploy(\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    endpoint_name=sagemaker_config[\"SolutionPrefix\"] + \"-endpoint\",\n",
    "    initial_instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the endpoint\n",
    "\n",
    "To do sanity checking, here we can test the endpoint by requesting predictions for a randomly generated data. The `predictor` handles serialization and deserialization of the requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(1)\n",
    "random_test = np.random.randn(NUM_TS, PREDICTION_LENGTH)\n",
    "\n",
    "# json serializable request format\n",
    "random_test_data = {}\n",
    "random_test_data[\"target\"] = random_test.tolist()\n",
    "random_test_data[\"start\"] = \"2014-01-01\"\n",
    "random_test_data[\"source\"] = []\n",
    "\n",
    "random_ret = predictor.predict(random_test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and then loads the return JSON objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecasts = np.array(random_ret[\"forecasts\"][\"samples\"])\n",
    "print(\"Forecasts shape with 10 samples: {}\".format(forecasts.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactive visualization\n",
    "\n",
    "It is important to visualize how the model is performing given the test data. Since we have trained our model given the hyperparameters defined earlier `CONTEXT_LENGTH` (input length) and `PREDICTION_LENGTH` (output length), we can now input the final window to our model where the last training time is **2014-05-26 19:00:00** so we test from **2014-05-26 19:00:00** onwards and get the predictions and visualize the perfomance of the model for a sample of time-series.\n",
    "\n",
    "We have provided some utilities in `deep_demand_forecast/monitor.py` for creating the visualization data from train, test and predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run deep_demand_forecast/monitor.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, test_df = prepare_data(\"processed_data\")\n",
    "print(f\"prepared train data shape {train_df.shape}, test data shape {test_df.shape}\")\n",
    "ts_col_names = [f\"ts_{i}\" for i in range(NUM_TS + 1)]\n",
    "train_df_viz, test_df_viz, selected_cols = create_data_viz(\n",
    "    train_df, test_df, CONTEXT_LENGTH, PREDICTION_LENGTH, num_sample_ts=11\n",
    ")\n",
    "train_df_viz.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selection = alt.selection_multi(fields=[\"covariate\"], bind=\"legend\", nearest=True)\n",
    "\n",
    "train_plot = (\n",
    "    alt.Chart(train_df_viz, title=\"Train data\")\n",
    "    .mark_line()\n",
    "    .encode(\n",
    "        alt.X(\"time:T\", axis=alt.Axis(title=\"Time\")),\n",
    "        alt.Y(\"value:Q\", axis=alt.Axis(title=\"Normalized electricity consumption (kW)\")),\n",
    "        alt.Color(\"covariate:N\"),\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n",
    "    )\n",
    "    .add_selection(selection)\n",
    ")\n",
    "\n",
    "test_plot = (\n",
    "    alt.Chart(test_df_viz, title=\"Test data\")\n",
    "    .mark_line()\n",
    "    .encode(\n",
    "        alt.X(\"time:T\", axis=alt.Axis(title=\"Time\")),\n",
    "        alt.Y(\"value:Q\", axis=alt.Axis(title=\"Normalized electricity consumption (kW)\")),\n",
    "        alt.Color(\"covariate:N\"),\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n",
    "    )\n",
    "    .add_selection(selection)\n",
    ")\n",
    "\n",
    "train_plot | test_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we prepare our test data for prediction as before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "num_train = train_df.shape[0] - 1\n",
    "test_data = {}\n",
    "test_data[\"target\"] = (\n",
    "    test_df.iloc[num_train : num_train + PREDICTION_LENGTH].set_index(\"time\").values.T.tolist()\n",
    ")\n",
    "test_data[\"start\"] = \"2014-05-26 19:00:00\"\n",
    "test_data[\"source\"] = []\n",
    "\n",
    "predictions = predictor.predict(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally prepare for the final interactive visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gluonts.dataset.common import ListDataset\n",
    "from gluonts.dataset.field_names import FieldName\n",
    "\n",
    "forecasts = np.transpose(np.array(predictions[\"forecasts\"][\"samples\"][0]))\n",
    "preds = ListDataset(\n",
    "    [{FieldName.TARGET: forecasts, FieldName.START: predictions[\"forecasts\"][\"start_date\"]}],\n",
    "    freq=predictions[\"forecasts\"][\"freq\"],\n",
    "    one_dim_target=False,\n",
    ")\n",
    "\n",
    "preds_df = multivar_df(next(iter(preds)))\n",
    "preds_df_filter = preds_df.loc[:, [\"time\"] + selected_cols]\n",
    "preds_df_filter = pd.melt(preds_df_filter, id_vars=[\"time\"], value_vars=selected_cols)\n",
    "preds_df_filter.rename(columns={\"variable\": \"covariate\"}, inplace=True)\n",
    "preds_df_filter.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_plot = (\n",
    "    alt.Chart(preds_df_filter, title=\"Predictions\")\n",
    "    .mark_line()\n",
    "    .encode(\n",
    "        alt.X(\"time:T\", axis=alt.Axis(title=\"Time\")),\n",
    "        alt.Y(\"value:Q\", axis=alt.Axis(title=\"Normalized electricity consumption (kW)\")),\n",
    "        alt.Color(\"covariate:N\"),\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n",
    "    )\n",
    "    .add_selection(selection)\n",
    ")\n",
    "\n",
    "(train_plot | test_plot) & preds_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Delete the endpoint and model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you're done with the endpoint, you should clean it up.\n",
    "\n",
    "All of the training jobs, models and endpoints we created can be viewed through the SageMaker console of your AWS account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_mxnet_p36",
   "language": "python",
   "name": "conda_mxnet_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}