pytorch_torchrun_c10d_imagenet.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# Copyright 2022 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "# Vertex AI SDK: Using PyTorch torchrun to simplify multi-node training with custom containers\n",
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/sdk/sdk_pytorch_torchrun_custom_container_training_imagenet.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/sdk/sdk_pytorch_torchrun_custom_container_training_imagenet.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/sdk/sdk_pytorch_torchrun_custom_container_training_imagenet.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>                                                                                               \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This tutorial uses the Tiny ImageNet dataset to run multi-node distributed training on Vertex AI with Torchrun. It will run distributed training on multiple nodes with GPUs\n",
    "\n",
    "### Objective\n",
    "\n",
    "In this tutorial, you will learn how to train an Imagenet model using PyTorch's Torchrun on multiple nodes:\n",
    "\n",
    "    * Install necessary libraries\n",
    "    * Create a training script using code from PyTorch Elastic's Github repository\n",
    "    * Create a custom training container that downloads training data directly to the image\n",
    "    * Train the model using multiple nodes with GPUs using PyTorch's C10D rendezvous backend\n",
    "\n",
    "### Dataset\n",
    "\n",
    "For the sake of training time, the Tiny ImageNet dataset is used in this tutorial: https://image-net.org/data/tiny-imagenet-200.zip\n",
    "\n",
    "This dataset consists of many small (~2KB) images. To avoid network bottlenecks with the large volume of network transfers from Cloud Storage to the GPUs, we will download this dataset to the containers\n",
    "\n",
    "The training code is based on this PyTorch Torchrun example for ImageNet: https://github.com/pytorch/elastic/blob/master/examples/imagenet/main.py\n",
    "\n",
    "### Costs \n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertex AI Training w/ GPUs\n",
    "* Vertex AI TensorBoard\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [Vertex AI\n",
    "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "## Installation\n",
    "\n",
    "Install the following packages required to execute this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "# Install the packages\n",
    "! pip3 install --user --upgrade google-cloud-aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "58707a750154"
   },
   "source": [
    "### Colab only: Uncomment the following cell to restart the kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f200f10a1da3"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "## Before you begin\n",
    "\n",
    "### Set up your Google Cloud project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
    "\n",
    "3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
    "\n",
    "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, try the following:\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "\n",
      "\n",
      "To take a quick anonymous survey, run:\n",
      "  $ gcloud survey\n",
      "\n"
     ]
    }
   ],
   "source": [
    "PROJECT_ID = \"gcp-ml-sandbox\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "! gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "region"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "region"
   },
   "outputs": [],
   "source": [
    "REGION = \"us-central1\"  # @param {type: \"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sBCra4QMA2wR"
   },
   "source": [
    "### Authenticate your Google Cloud account\n",
    "\n",
    "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "74ccc9e52986"
   },
   "source": [
    "**1. Vertex AI Workbench**\n",
    "* Do nothing as you are already authenticated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "de775a3773ba"
   },
   "source": [
    "**2. Local JupyterLab instance, uncomment and run:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "254614fa0c46"
   },
   "outputs": [],
   "source": [
    "# ! gcloud auth login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ef21552ccea8"
   },
   "source": [
    "**3. Colab, uncomment and run:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "603adbbf0532"
   },
   "outputs": [],
   "source": [
    "# from google.colab import auth\n",
    "# auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f6b2ccc891ed"
   },
   "source": [
    "**4. Service account or other**\n",
    "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "Create a storage bucket to store intermediate artifacts such as datasets.\n",
    "\n",
    "- *{Note to notebook author: For any user-provided strings that need to be unique (like bucket names or model ID's), append \"-unique\" to the end so proper testing can occur}*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_URI = \"gs://gcp-ml-sandbox-scratch\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://gcp-ml-sandbox-scratch/...\n",
      "ServiceException: 409 A Cloud Storage bucket named 'gcp-ml-sandbox-scratch' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n"
     ]
    }
   ],
   "source": [
    "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "960505627ddf"
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "PyQmSRbKA8r-"
   },
   "outputs": [],
   "source": [
    "from google.cloud import aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "source": [
    "### Initialize Vertex AI SDK for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "outputs": [],
   "source": [
    "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a7a84e2e4c4e"
   },
   "source": [
    "### Service Account\n",
    "\n",
    "You use a service account to create the Vertex AI Training job. If you do not want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e0c9c4f84849"
   },
   "outputs": [],
   "source": [
    "SERVICE_ACCOUNT = \"[your-service-account]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2d9440bb3017"
   },
   "source": [
    "If you do not provide a service account, run the code below to get the Compute Engine service account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "304a9ea0b6d0"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "if (\n",
    "    SERVICE_ACCOUNT == \"\"\n",
    "    or SERVICE_ACCOUNT is None\n",
    "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
    "):\n",
    "    # Get your service account from gcloud\n",
    "    if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "        shell_output = !gcloud auth list 2>/dev/null\n",
    "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
    "\n",
    "    if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
    "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
    "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
    "\n",
    "    print(\"Service Account:\", SERVICE_ACCOUNT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gar_enable_api"
   },
   "source": [
    "### Enable Artifact Registry API\n",
    "\n",
    "First, you must enable the Artifact Registry API service for your project.\n",
    "\n",
    "Learn more about [Enabling service](https://cloud.google.com/artifact-registry/docs/enable-service)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "gar_enable_api"
   },
   "outputs": [],
   "source": [
    "! gcloud services enable artifactregistry.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gar_create_repo"
   },
   "source": [
    "### Create a private Docker repository\n",
    "\n",
    "Your first step is to create your own Docker repository in Artifact Registry.\n",
    "\n",
    "1. Run the `gcloud artifacts repositories create` command to create a new Docker repository with your region with the description \"docker repository\".\n",
    "\n",
    "2. Run the `gcloud artifacts repositories list` command to verify that your repository was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "ea7cf85d87d2"
   },
   "outputs": [],
   "source": [
    "REPOSITORY = \"torchrun-imagenet-repo\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "gar_create_repo"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;31mERROR:\u001b[0m (gcloud.artifacts.repositories.create) ALREADY_EXISTS: the repository already exists\n",
      "Listing items under project gcp-ml-sandbox, location us-central1.\n",
      "\n",
      "                                                                                            ARTIFACT_REGISTRY\n",
      "REPOSITORY                       FORMAT  MODE                 DESCRIPTION                                    LOCATION     LABELS  ENCRYPTION          CREATE_TIME          UPDATE_TIME          SIZE (MB)\n",
      "custom-container-prediction-sdk  DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-09-28T17:33:04  2022-11-11T23:04:06  560.739\n",
      "docker-ray-repo                  DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-02-15T21:39:45  2022-02-15T22:44:23  2985.346\n",
      "docker-repo                      DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2021-06-07T16:53:06  2022-07-25T19:50:54  2481.024\n",
      "loan-eligiability-spark-demo     DOCKER  STANDARD_REPOSITORY  loan eligibility spark docker repository       us-central1          Google-managed key  2022-10-10T20:08:51  2022-10-10T20:46:09  427.490\n",
      "nvidia-triton                    DOCKER  STANDARD_REPOSITORY  NVIDIA Triton Docker repository                us-central1          Google-managed key  2022-08-05T17:23:25  2022-08-24T19:39:18  13355.884\n",
      "quickstart-kfp-repo              KFP     STANDARD_REPOSITORY  kfp-template-repo                              us-central1          Google-managed key  2022-10-21T19:50:52  2022-10-21T19:58:42  0.010\n",
      "quickstart-python-repo           PYTHON  STANDARD_REPOSITORY  Python package repository                      us-central1          Google-managed key  2023-01-17T16:30:11  2023-01-17T17:22:10  0.009\n",
      "ray-ml                           DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-03-07T16:30:06  2022-08-23T22:51:04  18448.873\n",
      "torchrun-imagenet-repo           DOCKER  STANDARD_REPOSITORY  Docker repository                              us-central1          Google-managed key  2022-12-08T16:02:58  2022-12-28T21:00:27  16631.888\n",
      "varian-ray-repo                  DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-02-15T23:13:48  2022-02-16T22:31:46  13496.837\n",
      "vertex-custom-containers         DOCKER  STANDARD_REPOSITORY  Image repository for Vertex custom containers  us-central1          Google-managed key  2022-09-23T20:15:16  2023-03-24T14:11:59  17176.408\n",
      "workbench                        DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-07-21T19:37:31  2022-07-26T19:46:55  6496.577\n",
      "workbench-test                   DOCKER  STANDARD_REPOSITORY                                                 us-central1          Google-managed key  2022-07-26T17:49:59  2022-09-08T14:59:50  15585.002\n"
     ]
    }
   ],
   "source": [
    "! gcloud artifacts repositories create {REPOSITORY} --repository-format=docker --location={REGION} --description=\"Docker repository\"\n",
    "\n",
    "! gcloud artifacts repositories list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gar_auth"
   },
   "source": [
    "### Configure authentication to your private repo\n",
    "\n",
    "Before you push or pull container images, configure Docker to use the `gcloud` command-line tool to authenticate requests to `Artifact Registry` for your region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gar_auth"
   },
   "outputs": [],
   "source": [
    "! gcloud auth configure-docker {REGION}-docker.pkg.dev --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/pytorch\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8f3ea1210749"
   },
   "source": [
    "## Vertex AI Training with GPUs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2f6e7336d1a0"
   },
   "source": [
    "### Create files for the host container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "fd9228f35579"
   },
   "outputs": [],
   "source": [
    "%mkdir -p trainer\n",
    "%cat /dev/null > trainer/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "70886930375a"
   },
   "source": [
    "#### Create the Dockerfile\n",
    "Installs necessary libraries, and downloads the tiny ImageNet data for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "b7da2e37e767"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trainer/Dockerfile\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/5065195394783248384 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/5065195394783248384 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/5065195394783248384 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/5065195394783248384 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/Dockerfile\n",
    "FROM gcr.io/deeplearning-platform-release/pytorch-gpu.1-13\n",
    "# Look at 1.13 - prior versions, something broke in the NVidia code\n",
    "\n",
    "RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \\\n",
    "    # Install reduction server plugin on GPU containers. google-fast-socket is\n",
    "    # previously installed in GPU dlenv containers only and it is not compatible\n",
    "    # with google-reduction-server.\n",
    "    if dpkg -s google-fast-socket; then \\\n",
    "      apt remove -y google-fast-socket && \\\n",
    "      apt install -y google-reduction-server; \\\n",
    "    fi\n",
    "\n",
    "COPY . /trainer\n",
    "\n",
    "WORKDIR /trainer\n",
    "\n",
    "RUN pip install -r requirements.txt\n",
    "\n",
    "RUN chmod 777 main.sh\n",
    "\n",
    "#robv@ - added to disable ipv6 in container\n",
    "RUN echo \"net.ipv6.conf.all.disable_ipv6 = 1\" >> /etc/sysctl.conf\n",
    "RUN echo \"net.ipv6.conf.default.disable_ipv6 = 1\" >> /etc/systctl.conf\n",
    "\n",
    "# download data to the container\n",
    "RUN wget -q -P /trainer/data https://image-net.org/data/tiny-imagenet-200.zip\n",
    "RUN unzip -q /trainer/data/tiny-imagenet-200.zip\n",
    "RUN rm /trainer/data/tiny-imagenet-200.zip\n",
    "\n",
    "CMD [\"/bin/bash\", \"-c\", \"TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_id $CLOUD_ML_JOB_ID --rdzv_endpoint=$(if [[ $RANK -gt 0 ]]; then echo $MASTER_ADDR;else echo localhost;fi):$MASTER_PORT main.py \"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "id": "cbcfa70f5d54"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trainer/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/requirements.txt\n",
    "torch==1.13.0\n",
    "torchvision==0.14.0\n",
    "tensorboard==2.5.0\n",
    "protobuf==3.20.*\n",
    "python-json-logger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "10fdcf6db6bb"
   },
   "source": [
    "#### Create the main.py file \n",
    "Main trainer for the ImageNet training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BINARIES_PATH: /opt/deeplearning/binaries\n",
      "CLICOLOR: 1\n",
      "CONDA_DEFAULT_ENV: py-3.9_tf-2.11_aip-1.21\n",
      "CONDA_EXE: /opt/conda/bin/conda\n",
      "CONDA_PREFIX: /opt/conda/envs/py-3.9_tf-2.11_aip-1.21\n",
      "CONDA_PREFIX_1: /opt/conda\n",
      "CONDA_PROMPT_MODIFIER: (py-3.9_tf-2.11_aip-1.21) \n",
      "CONDA_PYTHON_EXE: /opt/conda/bin/python\n",
      "CONDA_ROOT: /opt/conda\n",
      "CONDA_SHLVL: 2\n",
      "DL_ANACONDA_HOME: /opt/conda\n",
      "DL_BIN_PATH: /opt/deeplearning/bin\n",
      "DL_METADATA_PATH: /opt/deeplearning/metadata\n",
      "DL_PATH: /opt/deeplearning\n",
      "DL_PATH_DEPS: /opt/deeplearning/deps\n",
      "ENV_URI_FILE_PATH: /opt/deeplearning/metadata/env_uri\n",
      "ENV_VERSION_FILE_PATH: /opt/deeplearning/metadata/env_version\n",
      "FRAMEWORK_FILE_PATH: /opt/deeplearning/metadata/framework\n",
      "GIT_PAGER: cat\n",
      "GIT_PYTHON_REFRESH: quiet\n",
      "GRPC_FORK_SUPPORT_ENABLED: 0\n",
      "HOME: /home/jupyter\n",
      "INVOCATION_ID: 5d55583bab3d4bcebb30554cdae34929\n",
      "JOURNAL_STREAM: 9:12057\n",
      "JPY_PARENT_PID: 982\n",
      "JUPYTER_DEPS_PATH: /opt/deeplearning/jupyter\n",
      "LANG: C.UTF-8\n",
      "LD_LIBRARY_PATH: /opt/conda/envs/py-3.9_tf-2.11_aip-1.21/lib:/opt/conda/envs/py-3.9_tf-2.11_aip-1.21/lib/python3.9/site-packages/tensorrt\n",
      "LIT_PROXY_URL: /proxy/%PORT%/\n",
      "LOGNAME: jupyter\n",
      "MPLBACKEND: module://matplotlib_inline.backend_inline\n",
      "OPAL_PREFIX: /usr\n",
      "OS_DEBIAN10: debian-10\n",
      "OS_DEBIAN_PROJECT: debian-cloud\n",
      "OS_IMAGE_FAMILY: debian-10\n",
      "OS_IMAGE_PROJECT: debian-cloud\n",
      "OS_UBUNTU2004: ubuntu-2004-lts\n",
      "PACKAGE_SOURCE_CODE_PATH: /opt/deeplearning/src/third_party\n",
      "PAGER: cat\n",
      "PATH: /opt/conda/envs/py-3.9_tf-2.11_aip-1.21/bin:/opt/conda/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games\n",
      "POST_STARTUP_SCRIPT_PATH: /opt/c2d/post_start.sh\n",
      "PWD: /home/jupyter/pytorch\n",
      "RESTRICTION_TYPE_FILE_PATH: /opt/deeplearning/restriction\n",
      "SHELL: /bin/bash\n",
      "SHLVL: 1\n",
      "SRC_PATH: /opt/deeplearning/src\n",
      "TENSORBOARD_PROXY_URL: /proxy/%PORT%/\n",
      "TERM: xterm-color\n",
      "TITLE_FILE_PATH: /opt/deeplearning/metadata/title\n",
      "TUTORIALS_PATH: /opt/deeplearning/workspace/tutorials\n",
      "USER: jupyter\n",
      "VERSION_FILE_PATH: /opt/deeplearning/metadata/version\n",
      "WORKSPACE_PATH: /opt/deeplearning/workspace\n",
      "_: /opt/conda/envs/py-3.9_tf-2.11_aip-1.21/bin/python\n",
      "_CE_CONDA: \n",
      "_CE_M: \n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "for key,value in sorted(os.environ.items()):                                                                                                                                                                                                              \n",
    "        print(\"{0}: {1}\".format(key, value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "id": "24793c94ebff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trainer/main.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/main.py\n",
    "#!/usr/bin/env python3\n",
    "\n",
    "# Copyright (c) Facebook, Inc. and its affiliates.\n",
    "# All rights reserved.\n",
    "#\n",
    "# This source code is licensed under the BSD-style license found in the\n",
    "# LICENSE file in the root directory of this source tree.\n",
    "\n",
    "r\"\"\"\n",
    "Source: `pytorch imagenet example <https://github.com/pytorch/examples/blob/master/imagenet/main.py>`_ # noqa B950\n",
    "Modified and simplified to make the original pytorch example compatible with\n",
    "torchelastic.distributed.launch.\n",
    "Changes:\n",
    "1. Removed ``rank``, ``gpu``, ``multiprocessing-distributed``, ``dist_url`` options.\n",
    "   These are obsolete parameters when using ``torchelastic.distributed.launch``.\n",
    "2. Removed ``seed``, ``evaluate``, ``pretrained`` options for simplicity.\n",
    "3. Removed ``resume``, ``start-epoch`` options.\n",
    "   Loads the most recent checkpoint by default.\n",
    "4. ``batch-size`` is now per GPU (worker) batch size rather than for all GPUs.\n",
    "5. Defaults ``workers`` (num data loader workers) to ``0``.\n",
    "Usage\n",
    "::\n",
    " >>> python -m torchelastic.distributed.launch\n",
    "        --nnodes=$NUM_NODES\n",
    "        --nproc_per_node=$WORKERS_PER_NODE\n",
    "        --rdzv_id=$JOB_ID\n",
    "        --rdzv_backend=etcd\n",
    "        --rdzv_endpoint=$ETCD_HOST:$ETCD_PORT\n",
    "        main.py\n",
    "        --arch resnet18\n",
    "        --epochs 20\n",
    "        --batch-size 32\n",
    "        <DATA_DIR>\n",
    "\"\"\"\n",
    "\n",
    "import traceback\n",
    "import argparse\n",
    "import io\n",
    "import os\n",
    "import shutil\n",
    "import time\n",
    "from contextlib import contextmanager\n",
    "from datetime import timedelta\n",
    "from typing import List, Tuple\n",
    "\n",
    "import numpy\n",
    "import torch\n",
    "import torch.backends.cudnn as cudnn\n",
    "import torch.distributed as dist\n",
    "import torch.nn as nn\n",
    "import torch.nn.parallel\n",
    "import torch.optim\n",
    "import torch.utils.data\n",
    "import torch.utils.data.distributed\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.models as models\n",
    "import torchvision.transforms as transforms\n",
    "from torch.distributed.elastic.utils.data import ElasticDistributedSampler\n",
    "from torch.nn.parallel import DistributedDataParallel\n",
    "from torch.optim import SGD\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "import json\n",
    "\n",
    "# Environment variables set by torch.distributed.launch\n",
    "LOCAL_RANK = int(os.environ['LOCAL_RANK'])\n",
    "WORLD_SIZE = int(os.environ['WORLD_SIZE'])\n",
    "WORLD_RANK = int(os.environ['RANK'])\n",
    "\n",
    "\n",
    "model_names = sorted(\n",
    "    name\n",
    "    for name in models.__dict__\n",
    "    if name.islower() and not name.startswith(\"__\") and callable(models.__dict__[name])\n",
    ")\n",
    "\n",
    "parser = argparse.ArgumentParser(description=\"PyTorch Elastic ImageNet Training\")\n",
    "parser.add_argument(\"--data\", \n",
    "    metavar=\"DIR\", \n",
    "    default=\"/trainer/tiny-imagenet-200\",\n",
    "    help=\"path to dataset\")\n",
    "parser.add_argument(\n",
    "    \"-a\",\n",
    "    \"--arch\",\n",
    "    metavar=\"ARCH\",\n",
    "    default=\"resnet18\",\n",
    "    choices=model_names,\n",
    "    help=\"model architecture: \" + \" | \".join(model_names) + \" (default: resnet18)\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"-j\",\n",
    "    \"--workers\",\n",
    "    default=WORLD_SIZE,\n",
    "    type=int,\n",
    "    metavar=\"N\",\n",
    "    help=\"number of data loading workers\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--epochs\", default=90, type=int, metavar=\"N\", help=\"number of total epochs to run\"\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"-b\",\n",
    "    \"--batch-size\",\n",
    "    default=32,\n",
    "    type=int,\n",
    "    metavar=\"N\",\n",
    "    help=\"mini-batch size (default: 32), per worker (GPU)\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--lr\",\n",
    "    \"--learning-rate\",\n",
    "    default=0.1,\n",
    "    type=float,\n",
    "    metavar=\"LR\",\n",
    "    help=\"initial learning rate\",\n",
    "    dest=\"lr\",\n",
    ")\n",
    "parser.add_argument(\"--momentum\", default=0.9, type=float, metavar=\"M\", help=\"momentum\")\n",
    "parser.add_argument(\n",
    "    \"--wd\",\n",
    "    \"--weight-decay\",\n",
    "    default=1e-4,\n",
    "    type=float,\n",
    "    metavar=\"W\",\n",
    "    help=\"weight decay (default: 1e-4)\",\n",
    "    dest=\"weight_decay\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"-p\",\n",
    "    \"--print-freq\",\n",
    "    default=10,\n",
    "    type=int,\n",
    "    metavar=\"N\",\n",
    "    help=\"print frequency (default: 10)\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--dist-backend\",\n",
    "    default=\"nccl\",\n",
    "    choices=[\"nccl\", \"gloo\"],\n",
    "    type=str,\n",
    "    help=\"distributed backend\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--checkpoint-file\",\n",
    "    default=\"/tmp/checkpoint.pth.tar\",\n",
    "    type=str,\n",
    "    help=\"checkpoint file path, to load and save to\",\n",
    ")\n",
    "\n",
    "def main():\n",
    "    \n",
    "    \n",
    "    args = parser.parse_args()\n",
    "    print(args)\n",
    "    device_id = LOCAL_RANK\n",
    "    torch.cuda.set_device(device_id)\n",
    "    print(f\"=> set cuda device = {device_id}\")\n",
    "\n",
    "    print (f\"LOCAL_RANK={os.environ['LOCAL_RANK']} RANK={os.environ['RANK']} WORLD_SIZE={os.environ['WORLD_SIZE']}\")    \n",
    "    print (f\"Host address: {os.environ['MASTER_ADDR']}:{os.environ['MASTER_PORT']}\")\n",
    "    \n",
    "    print('Resetting MASTER_ADDR and MASTER PORT env')\n",
    "    \n",
    "    cluster_spec = json.loads(os.environ['CLUSTER_SPEC']) \n",
    "    _master_addr,_master_port = cluster_spec['cluster']['workerpool0'][0].split(':')\n",
    "    os.environ['MASTER_ADDR']=_master_addr\n",
    "    os.environ['MASTER_PORT']='3333'\n",
    "    \n",
    "    print (f\"Updated Host address: {os.environ['MASTER_ADDR']}:{os.environ['MASTER_PORT']}\")\n",
    "    \n",
    "    dist.init_process_group(args.dist_backend) \n",
    "    \n",
    "    model, criterion, optimizer = initialize_model(\n",
    "        args.arch, args.lr, args.momentum, args.weight_decay, device_id\n",
    "    )\n",
    "\n",
    "    train_loader, val_loader = initialize_data_loader(\n",
    "        args.data, args.batch_size, args.workers\n",
    "    )\n",
    "\n",
    "    # resume from checkpoint if one exists;\n",
    "    state = load_checkpoint(\n",
    "        args.checkpoint_file, device_id, args.arch, model, optimizer\n",
    "    )\n",
    "\n",
    "    start_epoch = state.epoch + 1\n",
    "    print(f\"=> start_epoch: {start_epoch}, best_acc1: {state.best_acc1}\")\n",
    "\n",
    "    print_freq = args.print_freq\n",
    "    for epoch in range(start_epoch, args.epochs):\n",
    "        state.epoch = epoch\n",
    "        train_loader.batch_sampler.sampler.set_epoch(epoch)\n",
    "        adjust_learning_rate(optimizer, epoch, args.lr)\n",
    "\n",
    "        # train for one epoch\n",
    "        train(train_loader, model, criterion, optimizer, epoch, device_id, print_freq)\n",
    "\n",
    "        # evaluate on validation set\n",
    "        acc1 = validate(val_loader, model, criterion, device_id, print_freq)\n",
    "\n",
    "        # remember best acc@1 and save checkpoint\n",
    "        is_best = acc1 > state.best_acc1\n",
    "        state.best_acc1 = max(acc1, state.best_acc1)\n",
    "\n",
    "        if device_id == 0:\n",
    "            save_checkpoint(state, is_best, args.checkpoint_file)\n",
    "\n",
    "\n",
    "class State:\n",
    "    \"\"\"\n",
    "    Container for objects that we want to checkpoint. Represents the\n",
    "    current \"state\" of the worker. This object is mutable.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, arch, model, optimizer):\n",
    "        self.epoch = -1\n",
    "        self.best_acc1 = 0\n",
    "        self.arch = arch\n",
    "        self.model = model\n",
    "        self.optimizer = optimizer\n",
    "\n",
    "    def capture_snapshot(self):\n",
    "        \"\"\"\n",
    "        Essentially a ``serialize()`` function, returns the state as an\n",
    "        object compatible with ``torch.save()``. The following should work\n",
    "        ::\n",
    "        snapshot = state_0.capture_snapshot()\n",
    "        state_1.apply_snapshot(snapshot)\n",
    "        assert state_0 == state_1\n",
    "        \"\"\"\n",
    "        return {\n",
    "            \"epoch\": self.epoch,\n",
    "            \"best_acc1\": self.best_acc1,\n",
    "            \"arch\": self.arch,\n",
    "            \"state_dict\": self.model.state_dict(),\n",
    "            \"optimizer\": self.optimizer.state_dict(),\n",
    "        }\n",
    "\n",
    "    def apply_snapshot(self, obj, device_id):\n",
    "        \"\"\"\n",
    "        The complimentary function of ``capture_snapshot()``. Applies the\n",
    "        snapshot object that was returned by ``capture_snapshot()``.\n",
    "        This function mutates this state object.\n",
    "        \"\"\"\n",
    "\n",
    "        self.epoch = obj[\"epoch\"]\n",
    "        self.best_acc1 = obj[\"best_acc1\"]\n",
    "        self.state_dict = obj[\"state_dict\"]\n",
    "        self.model.load_state_dict(obj[\"state_dict\"])\n",
    "        self.optimizer.load_state_dict(obj[\"optimizer\"])\n",
    "\n",
    "    def save(self, f):\n",
    "        torch.save(self.capture_snapshot(), f)\n",
    "\n",
    "    def load(self, f, device_id):\n",
    "        # Map model to be loaded to specified single gpu.\n",
    "        snapshot = torch.load(f, map_location=f\"cuda:{device_id}\")\n",
    "        self.apply_snapshot(snapshot, device_id)\n",
    "\n",
    "\n",
    "def initialize_model(\n",
    "    arch: str, lr: float, momentum: float, weight_decay: float, device_id: int\n",
    "):\n",
    "    print(f\"=> creating model: {arch}\")\n",
    "    model = models.__dict__[arch]()\n",
    "    # For multiprocessing distributed, DistributedDataParallel constructor\n",
    "    # should always set the single device scope, otherwise,\n",
    "    # DistributedDataParallel will use all available devices.\n",
    "    model.cuda(device_id)\n",
    "    cudnn.benchmark = True\n",
    "    model = DistributedDataParallel(model, device_ids=[device_id])\n",
    "    # define loss function (criterion) and optimizer\n",
    "    criterion = nn.CrossEntropyLoss().cuda(device_id)\n",
    "    optimizer = SGD(\n",
    "        model.parameters(), lr, momentum=momentum, weight_decay=weight_decay\n",
    "    )\n",
    "    return model, criterion, optimizer\n",
    "\n",
    "\n",
    "def initialize_data_loader(\n",
    "    data_dir, batch_size, num_data_workers\n",
    ") -> Tuple[DataLoader, DataLoader]:\n",
    "    traindir = os.path.join(data_dir, \"train\")\n",
    "    valdir = os.path.join(data_dir, \"val\")\n",
    "    normalize = transforms.Normalize(\n",
    "        mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]\n",
    "    )\n",
    "    train_dataset = datasets.ImageFolder(\n",
    "        traindir,\n",
    "        transforms.Compose(\n",
    "            [\n",
    "                transforms.RandomResizedCrop(224),\n",
    "                transforms.RandomHorizontalFlip(),\n",
    "                transforms.ToTensor(),\n",
    "                normalize,\n",
    "            ]\n",
    "        ),\n",
    "    )\n",
    "    train_sampler = ElasticDistributedSampler(train_dataset)\n",
    "    train_loader = DataLoader(\n",
    "        train_dataset,\n",
    "        batch_size=batch_size,\n",
    "        num_workers=num_data_workers,\n",
    "        pin_memory=True,\n",
    "        sampler=train_sampler,\n",
    "    )\n",
    "    val_loader = DataLoader(\n",
    "        datasets.ImageFolder(\n",
    "            valdir,\n",
    "            transforms.Compose(\n",
    "                [\n",
    "                    transforms.Resize(256),\n",
    "                    transforms.CenterCrop(224),\n",
    "                    transforms.ToTensor(),\n",
    "                    normalize,\n",
    "                ]\n",
    "            ),\n",
    "        ),\n",
    "        batch_size=batch_size,\n",
    "        shuffle=False,\n",
    "        num_workers=num_data_workers,\n",
    "        pin_memory=True,\n",
    "    )\n",
    "    return train_loader, val_loader\n",
    "\n",
    "\n",
    "def load_checkpoint(\n",
    "    checkpoint_file: str,\n",
    "    device_id: int,\n",
    "    arch: str,\n",
    "    model: DistributedDataParallel,\n",
    "    optimizer,  # SGD\n",
    ") -> State:\n",
    "    \"\"\"\n",
    "    Loads a local checkpoint (if any). Otherwise, checks to see if any of\n",
    "    the neighbors have a non-zero state. If so, restore the state\n",
    "    from the rank that has the most up-to-date checkpoint.\n",
    "    .. note:: when your job has access to a globally visible persistent storage\n",
    "              (e.g. nfs mount, S3) you can simply have all workers load\n",
    "              from the most recent checkpoint from such storage. Since this\n",
    "              example is expected to run on vanilla hosts (with no shared\n",
    "              storage) the checkpoints are written to local disk, hence\n",
    "              we have the extra logic to broadcast the checkpoint from a\n",
    "              surviving node.\n",
    "    \"\"\"\n",
    "\n",
    "    state = State(arch, model, optimizer)\n",
    "\n",
    "    if os.path.isfile(checkpoint_file):\n",
    "        print(f\"=> loading checkpoint file: {checkpoint_file}\")\n",
    "        state.load(checkpoint_file, device_id)\n",
    "        print(f\"=> loaded checkpoint file: {checkpoint_file}\")\n",
    "\n",
    "    # logic below is unnecessary when the checkpoint is visible on all nodes!\n",
    "    # create a temporary cpu pg to broadcast most up-to-date checkpoint\n",
    "    with tmp_process_group(backend=\"gloo\") as pg:\n",
    "        rank = dist.get_rank(group=pg)\n",
    "\n",
    "        # get rank that has the largest state.epoch\n",
    "        epochs = torch.zeros(dist.get_world_size(), dtype=torch.int32)\n",
    "        epochs[rank] = state.epoch\n",
    "        dist.all_reduce(epochs, op=dist.ReduceOp.SUM, group=pg)\n",
    "        t_max_epoch, t_max_rank = torch.max(epochs, dim=0)\n",
    "        max_epoch = t_max_epoch.item()\n",
    "        max_rank = t_max_rank.item()\n",
    "\n",
    "        # max_epoch == -1 means no one has checkpointed return base state\n",
    "        if max_epoch == -1:\n",
    "            print(f\"=> no workers have checkpoints, starting from epoch 0\")\n",
    "            return state\n",
    "\n",
    "        # broadcast the state from max_rank (which has the most up-to-date state)\n",
    "        # pickle the snapshot, convert it into a byte-blob tensor\n",
    "        # then broadcast it, unpickle it and apply the snapshot\n",
    "        print(f\"=> using checkpoint from rank: {max_rank}, max_epoch: {max_epoch}\")\n",
    "\n",
    "        with io.BytesIO() as f:\n",
    "            torch.save(state.capture_snapshot(), f)\n",
    "            raw_blob = numpy.frombuffer(f.getvalue(), dtype=numpy.uint8)\n",
    "\n",
    "        blob_len = torch.tensor(len(raw_blob))\n",
    "        dist.broadcast(blob_len, src=max_rank, group=pg)\n",
    "        print(f\"=> checkpoint broadcast size is: {blob_len}\")\n",
    "\n",
    "        if rank != max_rank:\n",
    "            # pyre-fixme[6]: For 1st param expected `Union[List[int], Size,\n",
    "            #  typing.Tuple[int, ...]]` but got `Union[bool, float, int]`.\n",
    "            blob = torch.zeros(blob_len.item(), dtype=torch.uint8)\n",
    "        else:\n",
    "            blob = torch.as_tensor(raw_blob, dtype=torch.uint8)\n",
    "\n",
    "        dist.broadcast(blob, src=max_rank, group=pg)\n",
    "        print(f\"=> done broadcasting checkpoint\")\n",
    "\n",
    "        if rank != max_rank:\n",
    "            with io.BytesIO(blob.numpy()) as f:\n",
    "                snapshot = torch.load(f)\n",
    "            state.apply_snapshot(snapshot, device_id)\n",
    "\n",
    "        # wait till everyone has loaded the checkpoint\n",
    "        dist.barrier(group=pg)\n",
    "\n",
    "    print(f\"=> done restoring from previous checkpoint\")\n",
    "    return state\n",
    "\n",
    "\n",
    "@contextmanager\n",
    "def tmp_process_group(backend):\n",
    "    cpu_pg = dist.new_group(backend=backend)\n",
    "    try:\n",
    "        yield cpu_pg\n",
    "    finally:\n",
    "        dist.destroy_process_group(cpu_pg)\n",
    "\n",
    "\n",
    "def save_checkpoint(state: State, is_best: bool, filename: str):\n",
    "    checkpoint_dir = os.path.dirname(filename)\n",
    "    os.makedirs(checkpoint_dir, exist_ok=True)\n",
    "\n",
    "    # save to tmp, then commit by moving the file in case the job\n",
    "    # gets interrupted while writing the checkpoint\n",
    "    tmp_filename = filename + \".tmp\"\n",
    "    torch.save(state.capture_snapshot(), tmp_filename)\n",
    "    os.rename(tmp_filename, filename)\n",
    "    print(f\"=> saved checkpoint for epoch {state.epoch} at {filename}\")\n",
    "    if is_best:\n",
    "        best = os.path.join(checkpoint_dir, \"model_best.pth.tar\")\n",
    "        print(f\"=> best model found at epoch {state.epoch} saving to {best}\")\n",
    "        shutil.copyfile(filename, best)\n",
    "\n",
    "\n",
    "def train(\n",
    "    train_loader: DataLoader,\n",
    "    model: DistributedDataParallel,\n",
    "    criterion,  # nn.CrossEntropyLoss\n",
    "    optimizer,  # SGD,\n",
    "    epoch: int,\n",
    "    device_id: int,\n",
    "    print_freq: int,\n",
    "):\n",
    "    batch_time = AverageMeter(\"Time\", \":6.3f\")\n",
    "    data_time = AverageMeter(\"Data\", \":6.3f\")\n",
    "    losses = AverageMeter(\"Loss\", \":.4e\")\n",
    "    top1 = AverageMeter(\"Acc@1\", \":6.2f\")\n",
    "    top5 = AverageMeter(\"Acc@5\", \":6.2f\")\n",
    "    progress = ProgressMeter(\n",
    "        len(train_loader),\n",
    "        [batch_time, data_time, losses, top1, top5],\n",
    "        prefix=\"Epoch: [{}]\".format(epoch+1),\n",
    "    )\n",
    "\n",
    "    # switch to train mode\n",
    "    model.train()\n",
    "\n",
    "    end = time.time()\n",
    "    for i, (images, target) in enumerate(train_loader):\n",
    "        # measure data loading time\n",
    "        data_time.update(time.time() - end)\n",
    "\n",
    "        images = images.cuda(device_id, non_blocking=True)\n",
    "        target = target.cuda(device_id, non_blocking=True)\n",
    "\n",
    "        # compute output\n",
    "        output = model(images)\n",
    "        loss = criterion(output, target)\n",
    "\n",
    "        # measure accuracy and record loss\n",
    "        acc1, acc5 = accuracy(output, target, topk=(1, 5))\n",
    "        losses.update(loss.item(), images.size(0))\n",
    "        top1.update(acc1[0], images.size(0))\n",
    "        top5.update(acc5[0], images.size(0))\n",
    "\n",
    "        # compute gradient and do SGD step\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # measure elapsed time\n",
    "        batch_time.update(time.time() - end)\n",
    "        end = time.time()\n",
    "\n",
    "        if i % print_freq == 0:\n",
    "            progress.display(i)\n",
    "\n",
    "\n",
    "def validate(\n",
    "    val_loader: DataLoader,\n",
    "    model: DistributedDataParallel,\n",
    "    criterion,  # nn.CrossEntropyLoss\n",
    "    device_id: int,\n",
    "    print_freq: int,\n",
    "):\n",
    "    batch_time = AverageMeter(\"Time\", \":6.3f\")\n",
    "    losses = AverageMeter(\"Loss\", \":.4e\")\n",
    "    top1 = AverageMeter(\"Acc@1\", \":6.2f\")\n",
    "    top5 = AverageMeter(\"Acc@5\", \":6.2f\")\n",
    "    progress = ProgressMeter(\n",
    "        len(val_loader), [batch_time, losses, top1, top5], prefix=\"Test: \"\n",
    "    )\n",
    "\n",
    "    # switch to evaluate mode\n",
    "    model.eval()\n",
    "\n",
    "    with torch.no_grad():\n",
    "        end = time.time()\n",
    "        for i, (images, target) in enumerate(val_loader):\n",
    "            if device_id is not None:\n",
    "                images = images.cuda(device_id, non_blocking=True)\n",
    "            target = target.cuda(device_id, non_blocking=True)\n",
    "\n",
    "            # compute output\n",
    "            output = model(images)\n",
    "            loss = criterion(output, target)\n",
    "\n",
    "            # measure accuracy and record loss\n",
    "            acc1, acc5 = accuracy(output, target, topk=(1, 5))\n",
    "            losses.update(loss.item(), images.size(0))\n",
    "            top1.update(acc1[0], images.size(0))\n",
    "            top5.update(acc5[0], images.size(0))\n",
    "\n",
    "            # measure elapsed time\n",
    "            batch_time.update(time.time() - end)\n",
    "            end = time.time()\n",
    "\n",
    "            if i % print_freq == 0:\n",
    "                progress.display(i)\n",
    "\n",
    "        # TODO: this should also be done with the ProgressMeter\n",
    "        print(\n",
    "            \" * Acc@1 {top1.avg:.3f} Acc@5 {top5.avg:.3f}\".format(top1=top1, top5=top5)\n",
    "        )\n",
    "\n",
    "    return top1.avg\n",
    "\n",
    "\n",
    "class AverageMeter(object):\n",
    "    \"\"\"Computes and stores the average and current value\"\"\"\n",
    "\n",
    "    def __init__(self, name: str, fmt: str = \":f\"):\n",
    "        self.name = name\n",
    "        self.fmt = fmt\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self) -> None:\n",
    "        self.val = 0\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, val, n=1) -> None:\n",
    "        self.val = val\n",
    "        self.sum += val * n\n",
    "        self.count += n\n",
    "        self.avg = self.sum / self.count\n",
    "\n",
    "    def __str__(self):\n",
    "        fmtstr = \"{name} {val\" + self.fmt + \"} ({avg\" + self.fmt + \"})\"\n",
    "        return fmtstr.format(**self.__dict__)\n",
    "\n",
    "\n",
    "class ProgressMeter(object):\n",
    "    def __init__(self, num_batches: int, meters: List[AverageMeter], prefix: str = \"\"):\n",
    "        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)\n",
    "        self.meters = meters\n",
    "        self.prefix = prefix\n",
    "\n",
    "    def display(self, batch: int) -> None:\n",
    "        entries = [self.prefix + self.batch_fmtstr.format(batch)]\n",
    "        entries += [str(meter) for meter in self.meters]\n",
    "        print(\"\\t\".join(entries))\n",
    "\n",
    "    def _get_batch_fmtstr(self, num_batches: int) -> str:\n",
    "        num_digits = len(str(num_batches // 1))\n",
    "        fmt = \"{:\" + str(num_digits) + \"d}\"\n",
    "        return \"[\" + fmt + \"/\" + fmt.format(num_batches) + \"]\"\n",
    "\n",
    "\n",
    "def adjust_learning_rate(optimizer, epoch: int, lr: float) -> None:\n",
    "    \"\"\"\n",
    "    Sets the learning rate to the initial LR decayed by 10 every 30 epochs\n",
    "    \"\"\"\n",
    "    learning_rate = lr * (0.1 ** (epoch // 30))\n",
    "    for param_group in optimizer.param_groups:\n",
    "        param_group[\"lr\"] = learning_rate\n",
    "\n",
    "\n",
    "def accuracy(output, target, topk=(1,)):\n",
    "    \"\"\"\n",
    "    Computes the accuracy over the k top predictions for the specified values of k\n",
    "    \"\"\"\n",
    "    with torch.no_grad():\n",
    "        maxk = max(topk)\n",
    "        batch_size = target.size(0)\n",
    "\n",
    "        _, pred = output.topk(maxk, 1, True, True)\n",
    "        pred = pred.t()\n",
    "        correct = pred.eq(target.view(1, -1).expand_as(pred))\n",
    "\n",
    "        res = []\n",
    "        for k in topk:\n",
    "            correct_k = correct[:k].reshape(1, -1).view(-1).float().sum(0, keepdim=True)\n",
    "            res.append(correct_k.mul_(100.0 / batch_size))\n",
    "        return res\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    try:\n",
    "        main()\n",
    "    except Exception as e:\n",
    "        trace_str = ''.join(traceback.format_tb(e.__traceback__))\n",
    "        print(trace_str)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "Copying file://trainer/main.py [Content-Type=text/x-python]...\n",
      "/ [1 files][ 19.0 KiB/ 19.0 KiB]                                                \n",
      "Operation completed over 1 objects/19.0 KiB.                                     \n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "!gsutil cp trainer/main.py gs://gcp-ml-sandbox-scratch/pytorch/trainer/main.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "93002a20a2a6"
   },
   "source": [
    "### Build custom container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "8ea6fc98b2a9"
   },
   "outputs": [],
   "source": [
    "CONTENT_NAME = \"pytorch-torchrun-imagenet-multi-node\"\n",
    "CONTAINER_NAME = CONTENT_NAME + \"-gpu\"\n",
    "TAG = \"latest\"\n",
    "\n",
    "custom_container_host_image_uri = (\n",
    "    f\"{REGION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{CONTAINER_NAME}:{TAG}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "id": "ee1a0a06d0b4",
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating temporary tarball archive of 7 file(s) totalling 25.9 KiB before compression.\n",
      "Uploading tarball of [trainer] to [gs://gcp-ml-sandbox_cloudbuild/source/1680240559.375589-e21dd603129645a19b7d6507b181b488.tgz]\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/gcp-ml-sandbox/locations/us-central1/builds/5dbe988f-b187-4096-a744-7527bbe30fb3].\n",
      "Logs are available at [ https://console.cloud.google.com/cloud-build/builds;region=us-central1/5dbe988f-b187-4096-a744-7527bbe30fb3?project=357746845324 ].\n",
      "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
      "starting build \"5dbe988f-b187-4096-a744-7527bbe30fb3\"\n",
      "\n",
      "FETCHSOURCE\n",
      "Fetching storage object: gs://gcp-ml-sandbox_cloudbuild/source/1680240559.375589-e21dd603129645a19b7d6507b181b488.tgz#1680240559593754\n",
      "Copying gs://gcp-ml-sandbox_cloudbuild/source/1680240559.375589-e21dd603129645a19b7d6507b181b488.tgz#1680240559593754...\n",
      "/ [1 files][  9.3 KiB/  9.3 KiB]                                                \n",
      "Operation completed over 1 objects/9.3 KiB.\n",
      "BUILD\n",
      "Already have image (with digest): gcr.io/cloud-builders/docker\n",
      "Sending build context to Docker daemon  32.77kB\n",
      "Step 1/12 : FROM gcr.io/deeplearning-platform-release/pytorch-gpu.1-13\n",
      "latest: Pulling from deeplearning-platform-release/pytorch-gpu.1-13\n",
      "846c0b181fff: Pulling fs layer\n",
      "6fc9dd88827c: Pulling fs layer\n",
      "0b311d7060d0: Pulling fs layer\n",
      "326d76058f67: Pulling fs layer\n",
      "ed7e4c52c661: Pulling fs layer\n",
      "4f05f5570c7a: Pulling fs layer\n",
      "ab264e292103: Pulling fs layer\n",
      "527d1d5ab821: Pulling fs layer\n",
      "fc46d4e99009: Pulling fs layer\n",
      "9ccf692754fa: Pulling fs layer\n",
      "c6611ece70c4: Pulling fs layer\n",
      "fe6ff819b45f: Pulling fs layer\n",
      "ee0f38e835c2: Pulling fs layer\n",
      "349a0ddfd68b: Pulling fs layer\n",
      "4f4fb700ef54: Pulling fs layer\n",
      "b3937eaecccb: Pulling fs layer\n",
      "5179479cab85: Pulling fs layer\n",
      "a75d9881db73: Pulling fs layer\n",
      "2a664f049df2: Pulling fs layer\n",
      "572f222f95f6: Pulling fs layer\n",
      "c44175e333ea: Pulling fs layer\n",
      "0a4af5e8aba0: Pulling fs layer\n",
      "145e1534f679: Pulling fs layer\n",
      "d73cc1e7595a: Pulling fs layer\n",
      "4950a2170284: Pulling fs layer\n",
      "0e2345f04d6f: Pulling fs layer\n",
      "d1b8cbed8a32: Pulling fs layer\n",
      "15a25bf2c8d0: Pulling fs layer\n",
      "0d79a94270ec: Pulling fs layer\n",
      "68923b1fad9d: Pulling fs layer\n",
      "16d04ccdc7e6: Pulling fs layer\n",
      "0e9f5d102aa3: Pulling fs layer\n",
      "56c67d4e466a: Pulling fs layer\n",
      "c6c3b72399ec: Pulling fs layer\n",
      "68a892b6e229: Pulling fs layer\n",
      "527d1d5ab821: Waiting\n",
      "fc46d4e99009: Waiting\n",
      "9ccf692754fa: Waiting\n",
      "c6611ece70c4: Waiting\n",
      "fe6ff819b45f: Waiting\n",
      "ee0f38e835c2: Waiting\n",
      "349a0ddfd68b: Waiting\n",
      "ed7e4c52c661: Waiting\n",
      "4f05f5570c7a: Waiting\n",
      "ab264e292103: Waiting\n",
      "326d76058f67: Waiting\n",
      "4f4fb700ef54: Waiting\n",
      "0e2345f04d6f: Waiting\n",
      "d1b8cbed8a32: Waiting\n",
      "15a25bf2c8d0: Waiting\n",
      "0d79a94270ec: Waiting\n",
      "68923b1fad9d: Waiting\n",
      "16d04ccdc7e6: Waiting\n",
      "0e9f5d102aa3: Waiting\n",
      "56c67d4e466a: Waiting\n",
      "b3937eaecccb: Waiting\n",
      "c6c3b72399ec: Waiting\n",
      "c44175e333ea: Waiting\n",
      "0a4af5e8aba0: Waiting\n",
      "68a892b6e229: Waiting\n",
      "2a664f049df2: Waiting\n",
      "d73cc1e7595a: Waiting\n",
      "a75d9881db73: Waiting\n",
      "572f222f95f6: Waiting\n",
      "6fc9dd88827c: Verifying Checksum\n",
      "6fc9dd88827c: Download complete\n",
      "0b311d7060d0: Download complete\n",
      "326d76058f67: Download complete\n",
      "ed7e4c52c661: Verifying Checksum\n",
      "ed7e4c52c661: Download complete\n",
      "ab264e292103: Download complete\n",
      "527d1d5ab821: Verifying Checksum\n",
      "527d1d5ab821: Download complete\n",
      "fc46d4e99009: Verifying Checksum\n",
      "fc46d4e99009: Download complete\n",
      "846c0b181fff: Verifying Checksum\n",
      "846c0b181fff: Download complete\n",
      "c6611ece70c4: Verifying Checksum\n",
      "c6611ece70c4: Download complete\n",
      "846c0b181fff: Pull complete\n",
      "6fc9dd88827c: Pull complete\n",
      "0b311d7060d0: Pull complete\n",
      "326d76058f67: Pull complete\n",
      "ed7e4c52c661: Pull complete\n",
      "4f05f5570c7a: Verifying Checksum\n",
      "4f05f5570c7a: Download complete\n",
      "ee0f38e835c2: Verifying Checksum\n",
      "ee0f38e835c2: Download complete\n",
      "349a0ddfd68b: Verifying Checksum\n",
      "349a0ddfd68b: Download complete\n",
      "4f4fb700ef54: Verifying Checksum\n",
      "4f4fb700ef54: Download complete\n",
      "9ccf692754fa: Verifying Checksum\n",
      "9ccf692754fa: Download complete\n",
      "b3937eaecccb: Verifying Checksum\n",
      "b3937eaecccb: Download complete\n",
      "a75d9881db73: Verifying Checksum\n",
      "a75d9881db73: Download complete\n",
      "2a664f049df2: Verifying Checksum\n",
      "2a664f049df2: Download complete\n",
      "572f222f95f6: Verifying Checksum\n",
      "572f222f95f6: Download complete\n",
      "c44175e333ea: Verifying Checksum\n",
      "c44175e333ea: Download complete\n",
      "5179479cab85: Verifying Checksum\n",
      "5179479cab85: Download complete\n",
      "145e1534f679: Verifying Checksum\n",
      "145e1534f679: Download complete\n",
      "d73cc1e7595a: Verifying Checksum\n",
      "d73cc1e7595a: Download complete\n",
      "4950a2170284: Verifying Checksum\n",
      "4950a2170284: Download complete\n",
      "0e2345f04d6f: Verifying Checksum\n",
      "0e2345f04d6f: Download complete\n",
      "d1b8cbed8a32: Verifying Checksum\n",
      "d1b8cbed8a32: Download complete\n",
      "15a25bf2c8d0: Verifying Checksum\n",
      "15a25bf2c8d0: Download complete\n",
      "0d79a94270ec: Verifying Checksum\n",
      "0d79a94270ec: Download complete\n",
      "68923b1fad9d: Download complete\n",
      "16d04ccdc7e6: Verifying Checksum\n",
      "16d04ccdc7e6: Download complete\n",
      "0e9f5d102aa3: Verifying Checksum\n",
      "0e9f5d102aa3: Download complete\n",
      "0a4af5e8aba0: Verifying Checksum\n",
      "0a4af5e8aba0: Download complete\n",
      "c6c3b72399ec: Verifying Checksum\n",
      "c6c3b72399ec: Download complete\n",
      "68a892b6e229: Verifying Checksum\n",
      "68a892b6e229: Download complete\n",
      "fe6ff819b45f: Verifying Checksum\n",
      "fe6ff819b45f: Download complete\n",
      "4f05f5570c7a: Pull complete\n",
      "ab264e292103: Pull complete\n",
      "527d1d5ab821: Pull complete\n",
      "fc46d4e99009: Pull complete\n",
      "56c67d4e466a: Verifying Checksum\n",
      "56c67d4e466a: Download complete\n",
      "9ccf692754fa: Pull complete\n",
      "c6611ece70c4: Pull complete\n",
      "fe6ff819b45f: Pull complete\n",
      "ee0f38e835c2: Pull complete\n",
      "349a0ddfd68b: Pull complete\n",
      "4f4fb700ef54: Pull complete\n",
      "b3937eaecccb: Pull complete\n",
      "5179479cab85: Pull complete\n",
      "a75d9881db73: Pull complete\n",
      "2a664f049df2: Pull complete\n",
      "572f222f95f6: Pull complete\n",
      "c44175e333ea: Pull complete\n",
      "0a4af5e8aba0: Pull complete\n",
      "145e1534f679: Pull complete\n",
      "d73cc1e7595a: Pull complete\n",
      "4950a2170284: Pull complete\n",
      "0e2345f04d6f: Pull complete\n",
      "d1b8cbed8a32: Pull complete\n",
      "15a25bf2c8d0: Pull complete\n",
      "0d79a94270ec: Pull complete\n",
      "68923b1fad9d: Pull complete\n",
      "16d04ccdc7e6: Pull complete\n",
      "0e9f5d102aa3: Pull complete\n",
      "56c67d4e466a: Pull complete\n",
      "c6c3b72399ec: Pull complete\n",
      "68a892b6e229: Pull complete\n",
      "Digest: sha256:a6634bc0439b5bfd864719757a75ec685fc0e03b1e9a8ced63f338adeafbb043\n",
      "Status: Downloaded newer image for gcr.io/deeplearning-platform-release/pytorch-gpu.1-13:latest\n",
      " ---> 94aabbb74950\n",
      "Step 2/12 : RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - &&     if dpkg -s google-fast-socket; then       apt remove -y google-fast-socket &&       apt install -y google-reduction-server;     fi\n",
      " ---> Running in a420e932a6dd\n",
      "\u001b[91m  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                 \u001b[0m\u001b[91m                Dload  Upload   Total   Spent    Left  Speed\n",
      "100  1210  100  1210    0     0  50416      0 --:--:-- --:--:-- --:--:-- 50416:--  \u001b[0m\u001b[91m   0\u001b[0m\u001b[91m\n",
      "\u001b[0m\u001b[91mWarning: apt-key output should not be parsed (stdout is not a terminal)\n",
      "\u001b[0mOK\n",
      "Package: google-fast-socket\n",
      "Status: install ok installed\n",
      "Priority: optional\n",
      "Section: contrib/devel\n",
      "Maintainer: Chang Lan <changlan@google.com>\n",
      "Architecture: amd64\n",
      "Version: 0.0.5\n",
      "Recommends: libnccl2\n",
      "Description: Fast Socket for NCCL 2\n",
      "\u001b[91m\n",
      "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
      "\u001b[0mReading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "The following packages will be REMOVED:\n",
      "  google-fast-socket\n",
      "0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.\n",
      "After this operation, 0 B of additional disk space will be used.\n",
      "(Reading database ... 91684 files and directories currently installed.)\n",
      "Removing google-fast-socket (0.0.5) ...\n",
      "Processing triggers for libc-bin (2.31-0ubuntu9.9) ...\n",
      "\u001b[91m\n",
      "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
      "\u001b[0mReading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "The following NEW packages will be installed:\n",
      "  google-reduction-server\n",
      "0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.\n",
      "Need to get 514 kB of archives.\n",
      "After this operation, 0 B of additional disk space will be used.\n",
      "Get:1 https://packages.cloud.google.com/apt google-fast-socket/main amd64 google-reduction-server amd64 2.2.3 [514 kB]\n",
      "\u001b[91mdebconf: unable to initialize frontend: Dialog\n",
      "debconf: (TERM is not set, so the dialog frontend is not usable.)\n",
      "debconf: falling back to frontend: Readline\n",
      "\u001b[0m\u001b[91mdebconf: unable to initialize frontend: Readline\n",
      "debconf: (This frontend requires a controlling tty.)\n",
      "debconf: falling back to frontend: Teletype\n",
      "\u001b[0m\u001b[91mdpkg-preconfigure: unable to re-open stdin: \n",
      "\u001b[0mFetched 514 kB in 0s (9154 kB/s)\n",
      "Selecting previously unselected package google-reduction-server.\n",
      "(Reading database ... 91681 files and directories currently installed.)\n",
      "Preparing to unpack .../google-reduction-server_2.2.3_amd64.deb ...\n",
      "Unpacking google-reduction-server (2.2.3) ...\n",
      "Setting up google-reduction-server (2.2.3) ...\n",
      "Processing triggers for libc-bin (2.31-0ubuntu9.9) ...\n",
      "Removing intermediate container a420e932a6dd\n",
      " ---> 51c8e759a531\n",
      "Step 3/12 : COPY . /trainer\n",
      " ---> 720cad97a3f5\n",
      "Step 4/12 : WORKDIR /trainer\n",
      " ---> Running in c22a5b69d0cd\n",
      "Removing intermediate container c22a5b69d0cd\n",
      " ---> d34dc7a1d205\n",
      "Step 5/12 : RUN pip install -r requirements.txt\n",
      " ---> Running in eeea3c6964a9\n",
      "Collecting torch==1.13.0\n",
      "  Downloading torch-1.13.0-cp37-cp37m-manylinux1_x86_64.whl (890.2 MB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 890.2/890.2 MB 2.3 MB/s eta 0:00:00\n",
      "Collecting torchvision==0.14.0\n",
      "  Downloading torchvision-0.14.0-cp37-cp37m-manylinux1_x86_64.whl (24.3 MB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.3/24.3 MB 62.7 MB/s eta 0:00:00\n",
      "Collecting tensorboard==2.5.0\n",
      "  Downloading tensorboard-2.5.0-py3-none-any.whl (6.0 MB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 79.2 MB/s eta 0:00:00\n",
      "Requirement already satisfied: protobuf==3.20.* in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (3.20.3)\n",
      "Collecting python-etcd\n",
      "  Downloading python-etcd-0.4.5.tar.gz (37 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting python-json-logger\n",
      "  Downloading python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0->-r requirements.txt (line 1)) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0->-r requirements.txt (line 1)) (11.7.99)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0->-r requirements.txt (line 1)) (4.5.0)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0->-r requirements.txt (line 1)) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0->-r requirements.txt (line 1)) (11.10.3.66)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchvision==0.14.0->-r requirements.txt (line 2)) (1.21.6)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/lib/python3.7/site-packages (from torchvision==0.14.0->-r requirements.txt (line 2)) (9.4.0)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from torchvision==0.14.0->-r requirements.txt (line 2)) (2.28.2)\n",
      "Collecting markdown>=2.6.8\n",
      "  Downloading Markdown-3.4.3-py3-none-any.whl (93 kB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.9/93.9 kB 17.5 MB/s eta 0:00:00\n",
      "Collecting google-auth<2,>=1.6.3\n",
      "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 152.9/152.9 kB 26.1 MB/s eta 0:00:00\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard==2.5.0->-r requirements.txt (line 3)) (67.6.1)\n",
      "Collecting google-auth-oauthlib<0.5,>=0.4.1\n",
      "  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)\n",
      "Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.7/site-packages (from tensorboard==2.5.0->-r requirements.txt (line 3)) (0.40.0)\n",
      "Collecting tensorboard-plugin-wit>=1.6.0\n",
      "  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 781.3/781.3 kB 59.1 MB/s eta 0:00:00\n",
      "Collecting werkzeug>=0.11.15\n",
      "  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 34.8 MB/s eta 0:00:00\n",
      "Requirement already satisfied: absl-py>=0.4 in /opt/conda/lib/python3.7/site-packages (from tensorboard==2.5.0->-r requirements.txt (line 3)) (1.4.0)\n",
      "Collecting tensorboard-data-server<0.7.0,>=0.6.0\n",
      "  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 87.7 MB/s eta 0:00:00\n",
      "Requirement already satisfied: grpcio>=1.24.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard==2.5.0->-r requirements.txt (line 3)) (1.53.0)\n",
      "Requirement already satisfied: urllib3>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from python-etcd->-r requirements.txt (line 5)) (1.26.15)\n",
      "Collecting dnspython>=1.13.0\n",
      "  Downloading dnspython-2.3.0-py3-none-any.whl (283 kB)\n",
      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.7/283.7 kB 36.8 MB/s eta 0:00:00\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard==2.5.0->-r requirements.txt (line 3)) (0.2.8)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard==2.5.0->-r requirements.txt (line 3)) (4.9)\n",
      "Collecting cachetools<5.0,>=2.0.0\n",
      "  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)\n",
      "Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard==2.5.0->-r requirements.txt (line 3)) (1.16.0)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard==2.5.0->-r requirements.txt (line 3)) (1.3.1)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard==2.5.0->-r requirements.txt (line 3)) (6.0.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->torchvision==0.14.0->-r requirements.txt (line 2)) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.7/site-packages (from requests->torchvision==0.14.0->-r requirements.txt (line 2)) (2.1.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->torchvision==0.14.0->-r requirements.txt (line 2)) (2022.12.7)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/conda/lib/python3.7/site-packages (from werkzeug>=0.11.15->tensorboard==2.5.0->-r requirements.txt (line 3)) (2.1.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard==2.5.0->-r requirements.txt (line 3)) (3.15.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard==2.5.0->-r requirements.txt (line 3)) (0.4.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard==2.5.0->-r requirements.txt (line 3)) (3.2.2)\n",
      "Building wheels for collected packages: python-etcd\n",
      "  Building wheel for python-etcd (setup.py): started\n",
      "  Building wheel for python-etcd (setup.py): finished with status 'done'\n",
      "  Created wheel for python-etcd: filename=python_etcd-0.4.5-py3-none-any.whl size=38483 sha256=11076bb68076a2bbbe4772f5eb6e73f4b5f87ad96d836483dd69f979b70150dc\n",
      "  Stored in directory: /root/.cache/pip/wheels/4b/c1/8f/237c058a9a16e922bc31e593f1cd4b210d2070336de2930735\n",
      "Successfully built python-etcd\n",
      "Installing collected packages: tensorboard-plugin-wit, werkzeug, tensorboard-data-server, python-json-logger, dnspython, cachetools, python-etcd, markdown, google-auth, torch, google-auth-oauthlib, torchvision, tensorboard\n",
      "  Attempting uninstall: cachetools\n",
      "    Found existing installation: cachetools 5.3.0\n",
      "    Uninstalling cachetools-5.3.0:\n",
      "      Successfully uninstalled cachetools-5.3.0\n",
      "  Attempting uninstall: google-auth\n",
      "    Found existing installation: google-auth 2.17.0\n",
      "    Uninstalling google-auth-2.17.0:\n",
      "      Successfully uninstalled google-auth-2.17.0\n",
      "  Attempting uninstall: torch\n",
      "    Found existing installation: torch 1.13.1\n",
      "    Uninstalling torch-1.13.1:\n",
      "      Successfully uninstalled torch-1.13.1\n",
      "  Attempting uninstall: google-auth-oauthlib\n",
      "    Found existing installation: google-auth-oauthlib 1.0.0\n",
      "    Uninstalling google-auth-oauthlib-1.0.0:\n",
      "      Successfully uninstalled google-auth-oauthlib-1.0.0\n",
      "  Attempting uninstall: torchvision\n",
      "    Found existing installation: torchvision 0.14.1\n",
      "    Uninstalling torchvision-0.14.1:\n",
      "      Successfully uninstalled torchvision-0.14.1\n",
      "\u001b[91mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "google-cloud-aiplatform 1.23.0 requires packaging<22.0.0dev,>=14.3, but you have packaging 23.0 which is incompatible.\n",
      "\u001b[0mSuccessfully installed cachetools-4.2.4 dnspython-2.3.0 google-auth-1.35.0 google-auth-oauthlib-0.4.6 markdown-3.4.3 python-etcd-0.4.5 python-json-logger-2.0.7 tensorboard-2.5.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 torch-1.13.0 torchvision-0.14.0 werkzeug-2.2.3\n",
      "\u001b[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "\u001b[0mRemoving intermediate container eeea3c6964a9\n",
      " ---> 3169c46ae72d\n",
      "Step 6/12 : RUN chmod 777 main.sh\n",
      " ---> Running in 980eaab951f5\n",
      "Removing intermediate container 980eaab951f5\n",
      " ---> 648773959ea2\n",
      "Step 7/12 : RUN echo \"net.ipv6.conf.all.disable_ipv6 = 1\" >> /etc/sysctl.conf\n",
      " ---> Running in 30b3d3fcfbc1\n",
      "Removing intermediate container 30b3d3fcfbc1\n",
      " ---> 8ba2f159b182\n",
      "Step 8/12 : RUN echo \"net.ipv6.conf.default.disable_ipv6 = 1\" >> /etc/systctl.conf\n",
      " ---> Running in 4cd6c8f71b94\n",
      "Removing intermediate container 4cd6c8f71b94\n",
      " ---> 75679a27cf17\n",
      "Step 9/12 : RUN wget -q -P /trainer/data https://image-net.org/data/tiny-imagenet-200.zip\n",
      " ---> Running in 08762f2c47ec\n",
      "Removing intermediate container 08762f2c47ec\n",
      " ---> 173e1b938d14\n",
      "Step 10/12 : RUN unzip -q /trainer/data/tiny-imagenet-200.zip\n",
      " ---> Running in 5c9a82f2d0cb\n",
      "Removing intermediate container 5c9a82f2d0cb\n",
      " ---> 8b73440476ff\n",
      "Step 11/12 : RUN rm /trainer/data/tiny-imagenet-200.zip\n",
      " ---> Running in 3af7859c3b83\n",
      "Removing intermediate container 3af7859c3b83\n",
      " ---> 62a7fc26d8fa\n",
      "Step 12/12 : CMD [\"/bin/bash\", \"-c\", \"TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_id $CLOUD_ML_JOB_ID --rdzv_endpoint=$(if [[ $RANK -gt 0 ]]; then echo $MASTER_ADDR;else echo localhost;fi):$MASTER_PORT main.py \"]\n",
      " ---> Running in 3d623858bed0\n",
      "Removing intermediate container 3d623858bed0\n",
      " ---> a366396537b5\n",
      "Successfully built a366396537b5\n",
      "Successfully tagged us-central1-docker.pkg.dev/gcp-ml-sandbox/torchrun-imagenet-repo/pytorch-torchrun-imagenet-multi-node-gpu:latest\n",
      "PUSH\n",
      "Pushing us-central1-docker.pkg.dev/gcp-ml-sandbox/torchrun-imagenet-repo/pytorch-torchrun-imagenet-multi-node-gpu:latest\n",
      "The push refers to repository [us-central1-docker.pkg.dev/gcp-ml-sandbox/torchrun-imagenet-repo/pytorch-torchrun-imagenet-multi-node-gpu]\n",
      "5b71a6953654: Preparing\n",
      "80b08cd759e9: Preparing\n",
      "fad4a538d4b0: Preparing\n",
      "75c9e8493ce6: Preparing\n",
      "2211d651db3b: Preparing\n",
      "1a89f6a77159: Preparing\n",
      "1663886d0fd3: Preparing\n",
      "5b106c3d8135: Preparing\n",
      "48f15297fec6: Preparing\n",
      "38ce229f0745: Preparing\n",
      "d375defb6347: Preparing\n",
      "f048dc232163: Preparing\n",
      "8469538cc1ff: Preparing\n",
      "afc8fc800afb: Preparing\n",
      "c4024cf738ec: Preparing\n",
      "56e222dc5f10: Preparing\n",
      "9d0408702b5a: Preparing\n",
      "1c5d9e19c354: Preparing\n",
      "ab6a3ff0d682: Preparing\n",
      "1bdeb40809bf: Preparing\n",
      "38ce229f0745: Waiting\n",
      "727e9f2242f4: Preparing\n",
      "9bfd3e05412e: Preparing\n",
      "80b5e2addf82: Preparing\n",
      "268fd46801ca: Preparing\n",
      "06b2faf97f0f: Preparing\n",
      "d375defb6347: Waiting\n",
      "466a1095f5ae: Preparing\n",
      "1a89f6a77159: Waiting\n",
      "f048dc232163: Waiting\n",
      "60772402e8ee: Preparing\n",
      "1590187f68fc: Preparing\n",
      "229d7a2ea066: Preparing\n",
      "48f15297fec6: Waiting\n",
      "1663886d0fd3: Waiting\n",
      "5f70bf18a086: Preparing\n",
      "f6b1096ebe88: Preparing\n",
      "5b106c3d8135: Waiting\n",
      "afc8fc800afb: Waiting\n",
      "522fdb540358: Preparing\n",
      "b3936e4c67d2: Preparing\n",
      "ab6a3ff0d682: Waiting\n",
      "07d37209b7a9: Preparing\n",
      "c4024cf738ec: Waiting\n",
      "1bdeb40809bf: Waiting\n",
      "fed0e8aa65b5: Preparing\n",
      "9bfd3e05412e: Waiting\n",
      "727e9f2242f4: Waiting\n",
      "1c5d9e19c354: Waiting\n",
      "ad8fec0b36f1: Preparing\n",
      "06b2faf97f0f: Waiting\n",
      "3a217af3edf9: Preparing\n",
      "f6b1096ebe88: Waiting\n",
      "3297f5de02be: Preparing\n",
      "522fdb540358: Waiting\n",
      "466a1095f5ae: Waiting\n",
      "b3936e4c67d2: Waiting\n",
      "f3717d7fdfb7: Preparing\n",
      "07d37209b7a9: Waiting\n",
      "60772402e8ee: Waiting\n",
      "e1eace4c0976: Preparing\n",
      "ad8fec0b36f1: Waiting\n",
      "fed0e8aa65b5: Waiting\n",
      "1590187f68fc: Waiting\n",
      "959a7375cb04: Preparing\n",
      "d79c672e1e8b: Preparing\n",
      "3a217af3edf9: Waiting\n",
      "5f70bf18a086: Waiting\n",
      "7b7c9e761223: Preparing\n",
      "3297f5de02be: Waiting\n",
      "56e222dc5f10: Waiting\n",
      "f3717d7fdfb7: Waiting\n",
      "e1eace4c0976: Waiting\n",
      "0002c93bdb37: Preparing\n",
      "959a7375cb04: Waiting\n",
      "7b7c9e761223: Waiting\n",
      "268fd46801ca: Waiting\n",
      "0002c93bdb37: Waiting\n",
      "9d0408702b5a: Waiting\n",
      "75c9e8493ce6: Pushed\n",
      "5b71a6953654: Pushed\n",
      "2211d651db3b: Pushed\n",
      "1a89f6a77159: Pushed\n",
      "5b106c3d8135: Pushed\n",
      "38ce229f0745: Layer already exists\n",
      "d375defb6347: Layer already exists\n",
      "f048dc232163: Layer already exists\n",
      "8469538cc1ff: Layer already exists\n",
      "afc8fc800afb: Layer already exists\n",
      "c4024cf738ec: Layer already exists\n",
      "56e222dc5f10: Layer already exists\n",
      "48f15297fec6: Pushed\n",
      "9d0408702b5a: Layer already exists\n",
      "1c5d9e19c354: Layer already exists\n",
      "1bdeb40809bf: Layer already exists\n",
      "ab6a3ff0d682: Layer already exists\n",
      "9bfd3e05412e: Layer already exists\n",
      "80b5e2addf82: Layer already exists\n",
      "268fd46801ca: Layer already exists\n",
      "727e9f2242f4: Layer already exists\n",
      "06b2faf97f0f: Layer already exists\n",
      "466a1095f5ae: Layer already exists\n",
      "60772402e8ee: Layer already exists\n",
      "1590187f68fc: Layer already exists\n",
      "229d7a2ea066: Layer already exists\n",
      "5f70bf18a086: Layer already exists\n",
      "f6b1096ebe88: Layer already exists\n",
      "522fdb540358: Layer already exists\n",
      "b3936e4c67d2: Layer already exists\n",
      "07d37209b7a9: Layer already exists\n",
      "fed0e8aa65b5: Layer already exists\n",
      "ad8fec0b36f1: Layer already exists\n",
      "3a217af3edf9: Layer already exists\n",
      "3297f5de02be: Layer already exists\n",
      "f3717d7fdfb7: Layer already exists\n",
      "e1eace4c0976: Layer already exists\n",
      "959a7375cb04: Layer already exists\n",
      "d79c672e1e8b: Layer already exists\n",
      "0002c93bdb37: Layer already exists\n",
      "7b7c9e761223: Layer already exists\n",
      "fad4a538d4b0: Pushed\n",
      "80b08cd759e9: Pushed\n",
      "1663886d0fd3: Pushed\n",
      "latest: digest: sha256:012dcbe0c4820129770a09a45f219a2dfddc343e77a4ab32fd8226a25eed1370 size: 9548\n",
      "DONE\n",
      "--------------------------------------------------------------------------------\n",
      "ID                                    CREATE_TIME                DURATION  SOURCE                                                                                        IMAGES                                                                                                               STATUS\n",
      "5dbe988f-b187-4096-a744-7527bbe30fb3  2023-03-31T05:29:19+00:00  7M43S     gs://gcp-ml-sandbox_cloudbuild/source/1680240559.375589-e21dd603129645a19b7d6507b181b488.tgz  us-central1-docker.pkg.dev/gcp-ml-sandbox/torchrun-imagenet-repo/pytorch-torchrun-imagenet-multi-node-gpu (+1 more)  SUCCESS\n"
     ]
    }
   ],
   "source": [
    "!gcloud builds submit \\\n",
    "   --region $REGION \\\n",
    "   --tag $custom_container_host_image_uri \\\n",
    "   --timeout \"2h\" \\\n",
    "   --machine-type=e2-highcpu-32 \\\n",
    "   trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RJV - Refactor training to use c10d backend, simplify code for chief, workers \n",
    "#### Notes:\n",
    "* Consolidate compute instances (use the same compute)\n",
    "* add logic to training script to detect whether or not we are a chief\n",
    "* execute `torchrun` as `python -m torch.distributed.run` with other command line arguments\n",
    "* test out on different ports\n",
    "* Use GCS and use filestore\n",
    "`echo $CLUSTER_SPEC | jq \".cluster.chief[0]\"`\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "BUCKET_NAME = BUCKET_URI.replace(\"gs://\", \"\")\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "MACHINE_SHAPE = \"n1-highmem-16\"  # RJV - these can be consolidated\n",
    "NUM_CPUS = 14  # Set to a few less than max CPUs per instance for paralle data loading\n",
    "TRAIN_GPU = \"NVIDIA_TESLA_T4\"\n",
    "TRAIN_NGPU = 1\n",
    "BATCH_SIZE = 512\n",
    "REPLICAS = 2\n",
    "EPOCHS = 5\n",
    "ARCH = \"resnet18\"\n",
    "BACKEND = \"nccl\"  # gloo for CPU only, nccl for GPUs\n",
    "TRAIN_DATA_LOCATION = \"/trainer/tiny-imagenet-200\"  # Data location of filed downloaded in Dockerfile\n",
    "\n",
    "display_name = (\n",
    "    CONTAINER_NAME\n",
    "    + \"-C10D-\"\n",
    "    + f\"{REPLICAS}workers-{TRAIN_NGPU}{TRAIN_GPU}-{BATCH_SIZE}batch-\"\n",
    "    + TIMESTAMP\n",
    ")\n",
    "gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
    "\n",
    "RDZV_BACKEND = \"c10d\"\n",
    "\n",
    "# cmd = [\"/bin/bash\", \"-c\", (f\"TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_backend {RDZV_BACKEND} --rdzv_id $CLOUD_ML_JOB_ID \"\n",
    "#        f\"--rdzv_endpoint=$(if [[ $RANK -gt 0 ]]; then echo $MASTER_ADDR;else echo localhost;fi):$MASTER_PORT --nnodes 2 --nproc_per_node 1 main.py \"\n",
    "#        f\"--epochs {EPOCHS} --arch {ARCH} --batch-size {BATCH_SIZE} --dist-backend {BACKEND} \"\n",
    "#        f\"--data {TRAIN_DATA_LOCATION}\")\n",
    "# ]\n",
    "# cmd = ['/bin/bash', '-c', (f'TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_backend {RDZV_BACKEND} --rdzv_id $CLOUD_ML_JOB_ID '\n",
    "#        f'--rdzv_endpoint=$(if [[ $RANK -gt 0 ]]; then NEW_MASTER=`echo \"$MASTER_ADDR\" | tr \"-\" \"_\" | tr \"[:lower:]\" \"[:upper:]\"`_SERVICE_HOST && echo ${{!NEW_MASTER}};else echo localhost;fi):$MASTER_PORT '\n",
    "#        f'--rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo 0;else echo 1;fi)\n",
    "#        f'--nnodes {REPLICAS+1} --nproc_per_node auto main.py '\n",
    "#        f'--epochs {EPOCHS} --arch {ARCH} --batch-size {BATCH_SIZE} --dist-backend {BACKEND} '\n",
    "#        f'--data {TRAIN_DATA_LOCATION}')\n",
    "# ]\n",
    "\n",
    "cmd = ['/bin/bash', '-c', (f'TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL '\n",
    "       f'torchrun --rdzv_backend {RDZV_BACKEND} --rdzv_id $CLOUD_ML_JOB_ID '\n",
    "       f'--rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT '\n",
    "       f'--rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) '\n",
    "       f'--nnodes {REPLICAS+1} --nproc_per_node auto main.py '\n",
    "       f'--epochs {EPOCHS} --arch {ARCH} --batch-size {BATCH_SIZE} --dist-backend {BACKEND} '\n",
    "       f'--data {TRAIN_DATA_LOCATION}')\n",
    "]\n",
    "\n",
    "#cmd = [\"python\",\"-c\", \"import time\\nwhile(True): time.sleep(60)\"]\n",
    "\n",
    "CONTAINER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": cmd\n",
    "}\n",
    "\n",
    "PRIMARY_WORKER_POOL = {\n",
    "    \"replica_count\": 1,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": MACHINE_SHAPE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS = [PRIMARY_WORKER_POOL]\n",
    "\n",
    "TRAIN_WORKER_POOL = {\n",
    "    \"replica_count\": REPLICAS,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": MACHINE_SHAPE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS.append(TRAIN_WORKER_POOL)\n",
    "\n",
    "c10_job = aiplatform.CustomJob(\n",
    "    display_name=display_name+'_CUSTOM_SA',\n",
    "    base_output_dir=gcs_output_uri_prefix,\n",
    "    worker_pool_specs=WORKER_POOL_SPECS)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_backend static --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT --rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) --nnodes 3 --nproc_per_node auto main.py --epochs 5 --arch resnet18 --batch-size 512 --dist-backend nccl --data /trainer/tiny-imagenet-200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/357746845324/locations/us-central1/customJobs/4619690875394457600\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/357746845324/locations/us-central1/customJobs/4619690875394457600')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/4619690875394457600?project=357746845324\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_QUEUED\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 access the interactive shell terminals for the custom job:\n",
      "workerpool1-0:\n",
      "f650a6422ac6afc5-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 access the interactive shell terminals for the custom job:\n",
      "workerpool0-0:\n",
      "be6c85b03aca61ad-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 access the interactive shell terminals for the custom job:\n",
      "workerpool1-1:\n",
      "57d381da223324d6-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/4619690875394457600 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "SERVICE_ACCOUNT='vertexai-test@gcp-ml-sandbox.iam.gserviceaccount.com'\n",
    "\n",
    "c10_job.run(\n",
    "    sync=False,\n",
    "    # comment out the line below to turn off interactive debug\n",
    "    enable_web_access=True\n",
    "    ,service_account=SERVICE_ACCOUNT,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/bin/bash -c TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL MASTER_ADDR=$(NEW_MASTER=`echo \"$MASTER_ADDR\" | tr \"-\" \"_\" | tr \"[:lower:]\" \"[:upper:]\"`_SERVICE_HOST && echo ${!NEW_MASTER}) torchrun --rdzv_backend c10d --rdzv_id $CLOUD_ML_JOB_ID --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT --rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) --nnodes 3 --nproc_per_node auto main.py --epochs 5 --arch resnet18 --batch-size 512 --dist-backend nccl --data /trainer/tiny-imagenet-200'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "#### WORKING\n",
    "BACKEND='nccl'\n",
    "RDZV_BACKEND='c10d'\n",
    "cmd = ['/bin/bash', '-c', (f'TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL '\n",
    "       f'MASTER_ADDR=$(NEW_MASTER=`echo \"$MASTER_ADDR\" | tr \"-\" \"_\" | tr \"[:lower:]\" \"[:upper:]\"`_SERVICE_HOST && echo ${{!NEW_MASTER}}) '\n",
    "       f'torchrun --rdzv_backend {RDZV_BACKEND} --rdzv_id $CLOUD_ML_JOB_ID '\n",
    "       f'--rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT '\n",
    "       f'--rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) '\n",
    "       f'--nnodes {REPLICAS+1} --nproc_per_node auto main.py '\n",
    "       f'--epochs {EPOCHS} --arch {ARCH} --batch-size {BATCH_SIZE} --dist-backend {BACKEND} '\n",
    "       f'--data {TRAIN_DATA_LOCATION}')\n",
    "]\n",
    "\n",
    "str_entry = \" \".join(cmd)\n",
    "str_entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/bin/bash -c TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --rdzv_backend c10d --rdzv_id $CLOUD_ML_JOB_ID --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT --rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) --nnodes 3 --nproc_per_node auto main.py --epochs 5 --arch resnet18 --batch-size 512 --dist-backend nccl --data /trainer/tiny-imagenet-200'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/1238965298866946048 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "BACKEND='nccl'\n",
    "RDZV_BACKEND='c10d'\n",
    "cmd = ['/bin/bash', '-c', (f'TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL '\n",
    "       #f'MASTER_ADDR=$(NEW_MASTER=`echo \"$MASTER_ADDR\" | tr \"-\" \"_\" | tr \"[:lower:]\" \"[:upper:]\"`_SERVICE_HOST && echo ${{!NEW_MASTER}}) '\n",
    "       f'torchrun --rdzv_backend {RDZV_BACKEND} --rdzv_id $CLOUD_ML_JOB_ID '\n",
    "       f'--rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT '\n",
    "       f'--rdzv_conf=is_host=$(if [[ $RANK -gt 0 ]]; then echo false;else echo true;fi) '\n",
    "       f'--nnodes {REPLICAS+1} --nproc_per_node auto main.py '\n",
    "       f'--epochs {EPOCHS} --arch {ARCH} --batch-size {BATCH_SIZE} --dist-backend {BACKEND} '\n",
    "       f'--data {TRAIN_DATA_LOCATION}')\n",
    "]\n",
    "\n",
    "str_entry = \" \".join(cmd)\n",
    "str_entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CHIEF\n",
    "TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --nnodes=2 --nproc_per_node=1 --rdzv_id=\"rjv-1\" --rdzv_backend=c10d --rdzv_endpoint=${if [[ $RANK -gt 0 ]]; then echo $MASTER_ADDR; else echo localhost;fi}:$MASTER_PORT demo.py --backend=nccl\n",
    "#WORKER\n",
    "TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --nnodes=2 --nproc_per_node=1 --rdzv_id=\"rjv-1\" --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT demo.py --backend=nccl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trainer/demo.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/demo.py\n",
    "import os\n",
    "import argparse\n",
    "\n",
    "import torch\n",
    "import torch.distributed as dist\n",
    "\n",
    "# Environment variables set by torch.distributed.launch\n",
    "LOCAL_RANK = int(os.environ['LOCAL_RANK'])\n",
    "WORLD_SIZE = int(os.environ['WORLD_SIZE'])\n",
    "WORLD_RANK = int(os.environ['RANK'])\n",
    "\n",
    "def run(backend):\n",
    "    tensor = torch.zeros(1)\n",
    "    \n",
    "    # Need to put tensor on a GPU device for nccl backend\n",
    "    if backend == 'nccl':\n",
    "        device = torch.device(\"cuda:{}\".format(LOCAL_RANK))\n",
    "        tensor = tensor.to(device)\n",
    "\n",
    "    if WORLD_RANK == 0:\n",
    "        for rank_recv in range(1, WORLD_SIZE):\n",
    "            dist.send(tensor=tensor, dst=rank_recv)\n",
    "            print('worker_{} sent data to Rank {}\\n'.format(0, rank_recv))\n",
    "    else:\n",
    "        dist.recv(tensor=tensor, src=0)\n",
    "        print('worker_{} has received data from rank {}\\n'.format(WORLD_RANK, 0))\n",
    "\n",
    "def init_processes(backend):\n",
    "    dist.init_process_group(backend, rank=WORLD_RANK, world_size=WORLD_SIZE)\n",
    "    run(backend)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--backend\", type=str, default=\"nccl\", choices=['nccl', 'gloo'])\n",
    "    args = parser.parse_args()\n",
    "\n",
    "    init_processes(backend=args.backend)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:Creating CustomJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob created. Resource name: projects/357746845324/locations/us-central1/customJobs/6016222100396703744\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob created. Resource name: projects/357746845324/locations/us-central1/customJobs/6016222100396703744\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this CustomJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:To use this CustomJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "custom_job = aiplatform.CustomJob.get('projects/357746845324/locations/us-central1/customJobs/6016222100396703744')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:custom_job = aiplatform.CustomJob.get('projects/357746845324/locations/us-central1/customJobs/6016222100396703744')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/6016222100396703744?project=357746845324\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/6016222100396703744?project=357746845324\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_QUEUED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.jobs:CustomJob projects/357746845324/locations/us-central1/customJobs/6016222100396703744 current state:\n",
      "JobState.JOB_STATE_PENDING\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4a87f56e869a"
   },
   "source": [
    "### Run training on Vertex AI using `torchrun` with ETCD on host"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "63ca30b33176"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "BUCKET_NAME = BUCKET_URI.replace(\"gs://\", \"\")\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "PRIMARY_COMPUTE = \"n1-highmem-16\"\n",
    "TRAIN_COMPUTE = \"n1-highmem-16\"\n",
    "NUM_CPUS = 14  # Set to a few less than max CPUs per instance for paralle data loading\n",
    "TRAIN_GPU = \"NVIDIA_TESLA_T4\"\n",
    "TRAIN_NGPU = 1\n",
    "BATCH_SIZE = 512\n",
    "REPLICAS = 2\n",
    "EPOCHS = 5\n",
    "ARCH = \"resnet18\"\n",
    "BACKEND = \"nccl\"  # gloo for CPU only, nccl for GPUs\n",
    "TRAIN_DATA_LOCATION = \"/trainer/tiny-imagenet-200\"  # Data location of filed downloaded in Dockerfile\n",
    "\n",
    "display_name = (\n",
    "    CONTAINER_NAME\n",
    "    + \"-LOCAL-ETCD-\"\n",
    "    + f\"{REPLICAS}workers-{TRAIN_NGPU}{TRAIN_GPU}-{BATCH_SIZE}batch-\"\n",
    "    + TIMESTAMP\n",
    ")\n",
    "gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
    "\n",
    "RDZV_BACKEND = \"etcd-v2\"\n",
    "RDZV_BACKEND_STORE = f\"/gcs/{BUCKET_NAME}/sharedfile-{display_name}\"\n",
    "RDZV_ENDPOINT = \"localhost:2379\"\n",
    "\n",
    "# Use letters for each parameter to be processed in the shell script\n",
    "\"\"\"\n",
    "e)epochs=${OPTARG};;\n",
    "a)arch=${OPTARG};;\n",
    "b)batchsize=${OPTARG};;\n",
    "d)distbackend=${OPTARG};;\n",
    "t)data=${OPTARG};;\n",
    "w)workers=${OPTARG};;\n",
    "v)env=${OPTARG};;\n",
    "u)rdvzbackend=${OPTARG};;\n",
    "i)rdvzid=${OPTARG};;\n",
    "p)endpoint=${OPTARG};;\n",
    "n)nnodes=${OPTARG};;\n",
    "r)nprocpernode=${OPTARG};;\n",
    "c)ischief=${OPTARG};;\n",
    "\"\"\"\n",
    "\n",
    "CONTAINER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c y\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "CONTAINER_WORKER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c n\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "PRIMARY_WORKER_POOL = {\n",
    "    \"replica_count\": 1,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": PRIMARY_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS = [PRIMARY_WORKER_POOL]\n",
    "\n",
    "TRAIN_WORKER_POOL = {\n",
    "    \"replica_count\": REPLICAS,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": TRAIN_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_WORKER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS.append(TRAIN_WORKER_POOL)\n",
    "\n",
    "job = aiplatform.CustomJob(\n",
    "    display_name=display_name,\n",
    "    base_output_dir=gcs_output_uri_prefix,\n",
    "    worker_pool_specs=WORKER_POOL_SPECS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "2b4684fde1a6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/357746845324/locations/us-central1/customJobs/8932725071584165888\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/357746845324/locations/us-central1/customJobs/8932725071584165888')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/8932725071584165888?project=357746845324\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 access the interactive shell terminals for the custom job:\n",
      "workerpool1-1:\n",
      "be1e716faacff1d2-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 access the interactive shell terminals for the custom job:\n",
      "workerpool1-0:\n",
      "7a908362bacd7cd9-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 access the interactive shell terminals for the custom job:\n",
      "workerpool0-0:\n",
      "c6a44767022157db-dot-us-central1.aiplatform-training.googleusercontent.com\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/357746845324/locations/us-central1/customJobs/8932725071584165888 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/357746845324/locations/us-central1/customJobs/8932725071584165888\n"
     ]
    }
   ],
   "source": [
    "job.run(\n",
    "    sync=False\n",
    "    # comment out the line below to turn off interactive debug\n",
    "    ,\n",
    "    enable_web_access=True\n",
    "    #service_account=SERVICE_ACCOUNT,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4a87f56e869a"
   },
   "source": [
    "### Run training on Vertex AI using `torchrun` with ETCD on host using NFS for data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "63ca30b33176"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "BUCKET_NAME = BUCKET_URI.replace(\"gs://\", \"\")\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "PRIMARY_COMPUTE = \"n1-highmem-16\"\n",
    "TRAIN_COMPUTE = \"n1-highmem-16\"\n",
    "NUM_CPUS = 14  # Set to a few less than max CPUs per instance for paralle data loading\n",
    "TRAIN_GPU = \"NVIDIA_TESLA_T4\"\n",
    "TRAIN_NGPU = 1\n",
    "BATCH_SIZE = 512\n",
    "REPLICAS = 2\n",
    "EPOCHS = 5\n",
    "ARCH = \"resnet18\"\n",
    "BACKEND = \"nccl\"  # gloo for CPU only, nccl for GPUs\n",
    "TRAIN_DATA_LOCATION = \"/mnt/nfs/vol1/tiny-imagenet-200\"  # Data location of filed downloaded in Dockerfile\n",
    "\n",
    "#Filestore info\n",
    "NFS_SERVER=10.234.137.2:/vol1\n",
    "NFS_PATH=vol1\n",
    "NFS_MOUNTPOINT=vol1\n",
    "\n",
    "display_name = (\n",
    "    CONTAINER_NAME\n",
    "    + \"-LOCAL-ETCD-w-NFS-\"\n",
    "    + f\"{REPLICAS}workers-{TRAIN_NGPU}{TRAIN_GPU}-{BATCH_SIZE}batch-\"\n",
    "    + TIMESTAMP\n",
    ")\n",
    "gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
    "\n",
    "RDZV_BACKEND = \"etcd-v2\"\n",
    "RDZV_BACKEND_STORE = f\"/gcs/{BUCKET_NAME}/sharedfile-{display_name}\"\n",
    "RDZV_ENDPOINT = \"localhost:2379\"\n",
    "\n",
    "# Use letters for each parameter to be processed in the shell script\n",
    "\"\"\"\n",
    "e)epochs=${OPTARG};;\n",
    "a)arch=${OPTARG};;\n",
    "b)batchsize=${OPTARG};;\n",
    "d)distbackend=${OPTARG};;\n",
    "t)data=${OPTARG};;\n",
    "w)workers=${OPTARG};;\n",
    "v)env=${OPTARG};;\n",
    "u)rdvzbackend=${OPTARG};;\n",
    "i)rdvzid=${OPTARG};;\n",
    "p)endpoint=${OPTARG};;\n",
    "n)nnodes=${OPTARG};;\n",
    "r)nprocpernode=${OPTARG};;\n",
    "c)ischief=${OPTARG};;\n",
    "\"\"\"\n",
    "\n",
    "CONTAINER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c y\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "CONTAINER_WORKER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c n\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "PRIMARY_WORKER_POOL = {\n",
    "    \"replica_count\": 1,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": PRIMARY_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_SPEC,\n",
    "    \"nfs_mounts\" {\n",
    "        \"server\": NFS_SERVER,\n",
    "        \"path\": NFS_PATH\n",
    "        \"mount_point\": NFS_MOUNT_POINT\n",
    "    }\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS = [PRIMARY_WORKER_POOL]\n",
    "\n",
    "TRAIN_WORKER_POOL = {\n",
    "    \"replica_count\": REPLICAS,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": TRAIN_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_WORKER_SPEC,\n",
    "    \"nfs_mounts\" {\n",
    "        \"server\": NFS_SERVER,\n",
    "        \"path\": NFS_PATH\n",
    "        \"mount_point\": NFS_MOUNT_POINT\n",
    "    }\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS.append(TRAIN_WORKER_POOL)\n",
    "\n",
    "job = aiplatform.CustomJob(\n",
    "    display_name=display_name,\n",
    "    base_output_dir=gcs_output_uri_prefix,\n",
    "    worker_pool_specs=WORKER_POOL_SPECS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2b4684fde1a6"
   },
   "outputs": [],
   "source": [
    "job.run(\n",
    "    sync=False\n",
    "    # comment out the line below to turn off interactive debug\n",
    "    ,\n",
    "    enable_web_access=True\n",
    "    #service_account=SERVICE_ACCOUNT,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "796de99ae13a"
   },
   "source": [
    "### Run training on Vertex AI using `torchrun` with ETCD on host and reduction server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6815ecb07ad9"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "BUCKET_NAME = BUCKET_URI.replace(\"gs://\", \"\")\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "PRIMARY_COMPUTE = \"n1-highmem-16\"\n",
    "TRAIN_COMPUTE = \"n1-highmem-16\"\n",
    "REDUCTION_COMPUTE = \"n1-highcpu-16\"\n",
    "NUM_CPUS = 14  # Set to a few less than max CPUs per instance for paralle data loading\n",
    "TRAIN_GPU = \"NVIDIA_TESLA_T4\"\n",
    "TRAIN_NGPU = 1\n",
    "BATCH_SIZE = 512\n",
    "REPLICAS = 2\n",
    "EPOCHS = 5\n",
    "ARCH = \"resnet18\"\n",
    "BACKEND = \"nccl\"  # gloo for CPU only, nccl for GPUs\n",
    "TRAIN_DATA_LOCATION = \"/trainer/tiny-imagenet-200\"  # Data location on NFS\n",
    "\n",
    "\n",
    "display_name = (\n",
    "    CONTAINER_NAME\n",
    "    + \"-LOCAL-ETCD-reduc-server-\"\n",
    "    + f\"{REPLICAS}workers-{TRAIN_NGPU}{TRAIN_GPU}-{BATCH_SIZE}batch-\"\n",
    "    + TIMESTAMP\n",
    ")\n",
    "gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
    "\n",
    "RDZV_BACKEND = \"etcd-v2\"\n",
    "RDZV_BACKEND_STORE = f\"/gcs/{BUCKET_NAME}/sharedfile-{display_name}\"\n",
    "RDZV_ENDPOINT = \"localhost:2379\"\n",
    "\n",
    "\n",
    "# Use letters for each parameter to be processed in the shell script\n",
    "\"\"\"\n",
    "e)epochs=${OPTARG};;\n",
    "a)arch=${OPTARG};;\n",
    "b)batchsize=${OPTARG};;\n",
    "d)distbackend=${OPTARG};;\n",
    "t)data=${OPTARG};;\n",
    "w)workers=${OPTARG};;\n",
    "v)env=${OPTARG};;\n",
    "u)rdvzbackend=${OPTARG};;\n",
    "i)rdvzid=${OPTARG};;\n",
    "p)endpoint=${OPTARG};;\n",
    "n)nnodes=${OPTARG};;\n",
    "r)nprocpernode=${OPTARG};;\n",
    "c)ischief=${OPTARG};;\n",
    "\"\"\"\n",
    "\n",
    "CONTAINER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c y\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "CONTAINER_WORKER_SPEC = {\n",
    "    \"image_uri\": custom_container_host_image_uri,\n",
    "    \"command\": [\n",
    "        \"/bin/bash\",\n",
    "        \"main.sh\",\n",
    "        f\"-e {EPOCHS}\",\n",
    "        f\"-a {ARCH}\",\n",
    "        f\"-b {BATCH_SIZE}\",\n",
    "        f\"-d {BACKEND}\",\n",
    "        f\"-t {TRAIN_DATA_LOCATION}\",\n",
    "        f\"-w {NUM_CPUS}\",\n",
    "        f\"-v {RDZV_BACKEND_STORE}\",\n",
    "        f\"-u {RDZV_BACKEND}\",\n",
    "        f\"-i {display_name}\",\n",
    "        f\"-p {RDZV_ENDPOINT}\",\n",
    "        f\"-n {REPLICAS+1}\",\n",
    "        f\"-r {TRAIN_NGPU}\",\n",
    "        \"-c n\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "PRIMARY_WORKER_POOL = {\n",
    "    \"replica_count\": 1,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": PRIMARY_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS = [PRIMARY_WORKER_POOL]\n",
    "\n",
    "TRAIN_WORKER_POOL = {\n",
    "    \"replica_count\": REPLICAS,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": TRAIN_COMPUTE,\n",
    "        \"accelerator_count\": TRAIN_NGPU,\n",
    "        \"accelerator_type\": TRAIN_GPU,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_WORKER_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS.append(TRAIN_WORKER_POOL)\n",
    "\n",
    "# Add Reduction Server worker pool\n",
    "REDUCTION_SERVER_REPLICAS = 3\n",
    "REDUCTION_SERVER_IMAGE_URI = (\n",
    "    \"us-docker.pkg.dev/vertex-ai-restricted/training/reductionserver:latest\"\n",
    ")\n",
    "\n",
    "CONTAINER_REDUCTION_SPEC = {\"image_uri\": REDUCTION_SERVER_IMAGE_URI}\n",
    "\n",
    "REDUCTION_WORKER_POOL = {\n",
    "    \"replica_count\": REDUCTION_SERVER_REPLICAS,\n",
    "    \"machine_spec\": {\n",
    "        \"machine_type\": REDUCTION_COMPUTE,\n",
    "    },\n",
    "    \"container_spec\": CONTAINER_REDUCTION_SPEC,\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPECS.append(REDUCTION_WORKER_POOL)\n",
    "\n",
    "job = aiplatform.CustomJob(\n",
    "    display_name=display_name,\n",
    "    base_output_dir=gcs_output_uri_prefix,\n",
    "    worker_pool_specs=WORKER_POOL_SPECS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "d4591d0a861a"
   },
   "outputs": [],
   "source": [
    "job.run(\n",
    "    sync=True\n",
    "    # comment out the line below to turn off interactive debug\n",
    "    ,\n",
    "    enable_web_access=True,\n",
    "    service_account=SERVICE_ACCOUNT,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sx_vKniMq9ZX"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Delete Cloud Storage objects that were created\n",
    "delete_bucket = False\n",
    "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
    "    ! gsutil -m rm -r $BUCKET_URI"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "sdk_pytorch_torchrun_custom_container_training_imagenet.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "py-3.9_tf-2.11_aip-1.21",
   "name": "common-cpu.m93",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m93"
  },
  "kernelspec": {
   "display_name": "py-3.9_tf-2.11_aip-1.21",
   "language": "python",
   "name": "py-3.9_tf-2.11_aip-1.21"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}