BraTS integration with MLCube [WIP]

brats/README.md

@@ -0,0 +1,170 @@
+# MedPerf - MLCube - BraTs Challange Integration
+
+## Project setup
+
+```bash
+# Create Python environment and install MLCube Docker runner 
+virtualenv -p python3 ./env && source ./env/bin/activate && pip install mlcube-docker
+
+# Fetch the boston housing example from GitHub
+git clone https://github.com/mlcommons/mlcube_examples && cd ./mlcube_examples
+git fetch origin pull/39/head:feature/brats && git checkout feature/brats
+cd ./brats/metrics/mlcube
+```
+
+## Execute docker-based MLCubes with Singularity runner
+
+```bash
+virtualenv -p python3 env && source ./env/bin/activate
+
+git clone https://github.com/mlcommons/mlcube && cd ./mlcube
+
+git fetch origin pull/223/head:feature/singularity_with_docker_images && git checkout feature/singularity_with_docker_images
+
+pip install semver spython && pip install ./mlcube
+
+pip install --no-deps --force-reinstall ./runners/mlcube_singularity
+```
+
+## MedPerf API Server
+
+To run locally, clone this repo:
+
+```Bash
+git clone https://github.com/mlcommons/medperf.git
+```
+
+Go to the `server` folder
+
+```Bash
+cd server
+```
+
+Install all dependencies
+
+```Bash
+pip install -r requirements.txt
+```
+
+Create .env file with your environment settings
+
+Sample .env.example is added to root. Rename `.env.example` to `.env` and modify with your env vars.
+
+```Bash
+cp .env.example .env
+```
+
+Create tables and existing models
+
+```Bash
+python manage.py migrate
+```
+
+Start the server
+
+```Bash
+python manage.py runserver
+```
+
+API Server is running at `http://127.0.0.1:8000/` by default. You can view and experiment Medperf API at `http://127.0.0.1:8000/swagger`
+
+## Medperf CLI
+
+The Medperf CLI is a command-line-interface that provides tools for preparing datasets and executing benchmarks on such datasets.
+
+To install, clone this repo (If you already did skip this step):
+
+```Bash
+git clone https://github.com/mlcommons/medperf.git
+```
+
+Go to the `cli` folder
+
+```Bash
+cd cli
+```
+
+Install using pip
+
+```Bash
+pip install -e .
+```
+
+## How to run
+
+The MedPerf CLI provides the following commands:
+
+- `login`: authenticates the CLI with the medperf backend server
+
+```Bash
+medperf login
+```
+
+- `dataset ls`: Lists all registered datasets by the user
+
+```Bash
+medperf dataset ls
+```
+
+- `dataset create`: Prepares a raw dataset for a specific benchmark
+
+```Bash
+medperf dataset create -b <BENCHMARK_UID> -d <DATA_PATH> -l <LABELS_PATH>
+```
+
+- `dataset submit`: Submits a prepared local dataset to the platform.
+
+```Bash
+medperf dataset submit -d <DATASET_UID> 
+```
+
+- `dataset associate`: Associates a prepared dataset with a specific benchmark
+
+```Bash
+medperf associate -b <BENCHMARK_UID> -d <DATASET_UID>
+```
+
+- `run`: Alias for `result create`. Runs a specific model from a benchmark with a specified prepared dataset
+
+```Bash
+medperf run -b <BENCHMARK_UID> -d <DATASET_UID> -m <MODEL_UID>
+```
+
+- `result ls`: Displays all results created by the user
+
+```Bash
+medperf result ls
+```
+
+
+- `result create`: Runs a specific model from a benchmark with a specified prepared dataset
+
+```Bash
+medperf result create -b <BENCHMARK_UID> -d <DATASET_UID> -m <MODEL_UID>
+```
+
+- `result submit`: Submits already obtained results to the platform
+
+```Bash
+medperf result submit -b <BENCHMARK_UID> -d <DATASET_UID> -m <MODEL_UID>
+```
+
+- `mlcube ls`: Lists all mlcubes created by the user. Lists all mlcubes if `--all` is passed
+
+```Bash
+medperf mlcube ls [--all]
+``` 
+
+- `mlcube submit`: Submits a new mlcube to the platform
+
+```Bash
+medperf mlcube submit
+```
+
+- `mlcube associate`: Associates an MLCube to a benchmark
+
+```Bash
+medperf mlcube associate -b <BENCHMARK_UID> -m <MODEL_UID>
+```
+
+The CLI runs MLCubes behind the scene. This cubes require a container engine like docker, and so that engine must be running before running commands like `prepare` and `execute`

brats/metrics/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+mlcube/workspace/results.yaml

brats/metrics/README.md

@@ -0,0 +1,104 @@
+# BraTS Challenge - MLCube integration - Metrics
+
+Original implementation: ["BraTS Instructions Repo"](https://github.com/BraTS/Instructions)
+
+## Dataset
+
+Please refer to the [BraTS challenge page](http://braintumorsegmentation.org/) and follow the instructions in the data section.
+
+## Project setup
+
+```bash
+# Create Python environment and install MLCube Docker runner 
+virtualenv -p python3 ./env && source ./env/bin/activate && pip install mlcube-docker
+
+# Fetch the boston housing example from GitHub
+git clone https://github.com/mlcommons/mlcube_examples && cd ./mlcube_examples
+git fetch origin pull/39/head:feature/brats && git checkout feature/brats
+cd ./brats/metrics/mlcube
+```
+
+## Important files
+
+These are the most important files on this project:
+
+```bash
+
+├── mlcube
+│   ├── mlcube.yaml                             # MLCube configuration file, it defines the project, author, platform, docker and tasks.
+│   └── workspace
+│       ├── data
+│       │   ├── ground_truth
+│       │   │   └── BraTS_example_seg.nii.gz    # Ground truth example file
+│       │   └── predictions
+│       │       └── BraTS_example_seg.nii.gz    # Prediction example file
+│       ├── parameters.yaml
+│       └── results.yaml                        # Final output file containing result metrics.
+└── project
+    ├── Dockerfile                              # Docker file with instructions to create the image for the project.
+    ├── metrics.py                              # Python file that contains the main logic of the project.
+    ├── mlcube.py                               # Python entrypoint used by MLCube, contains the logic for MLCube tasks.
+    └── requirements.txt                        # Python requirements needed to run the project inside Docker.
+```
+
+## How to modify this project
+
+You can change each file described above in order to add your own implementation.
+
+### Requirements file
+
+In this file (`requirements.txt`) you can add all the python dependencies needed for running your implementation, these dependencies will be installed during the creation of the docker image, this happens when you run the ```mlcube run ...``` command.
+
+### Dockerfile
+
+You can use both, CPU or GPU version for the dockerfile (`Dockerfile_CPU`, `Dockerfile_GPU`), also, you can add or modify any steps inside the file, this comes handy when you need to install some OS dependencies or even when you want to change the base docker image, inside the file you can find some information about the existing steps.
+
+### Parameters file
+
+This is a yaml file (`parameters.yaml`)that contains all extra parameters that aren't files or directories, for example, here you can place all the hyperparameters that you will use for training a model. This file will be passed as an **input parameter** in the MLCube tasks and then it will be read inside the MLCube container.
+
+### MLCube yaml file
+
+In this file (`mlcube.yaml`) you can find the instructions about the docker image and platform that will be used, information about the project (name, description, authors), and also the tasks defined for the project.
+
+In the existing implementation you will find 1 task:
+
+* evaluate:
+
+    This task takes the following parameters:
+
+  * Input parameters:
+    * predictions: Folder path containing predictions
+    * ground_truth: Folder path containing ground truth data
+    * parameters_file: Extra parameters
+  * Output parameters:
+    * output_path: File path where output metrics will be stored
+
+    This task takes the input predictions and ground truth data, perform the evaluation and then save the output result in the output_path.
+
+### MLCube python file
+
+The `mlcube.py` file is the handler file and entrypoint described in the dockerfile, here you can find all the logic related to how to process each MLCube task. If you want to add a new task first you must define it inside the `mlcube.yaml` file with its input and output parameters and then you need to add the logic to handle this new task inside the `mlcube.py` file.
+
+### Metrics file
+
+The `metrics.py` file contains the main logic of the project, you can modify this file and write your implementation here to calculate different metrics, this metrics file is called from the `mlcube.py` file and there are other ways to link your implementation and shown in the [MLCube examples repo](https://github.com/mlcommons/mlcube_examples).
+
+## Tasks execution
+
+```bash
+# Run evaluate task.
+mlcube run --mlcube=mlcube.yaml --task=evaluate
+```
+
+To use Singularity runner add the flag `--platform=singularity`, example:
+
+```bash
+mlcube run --mlcube=mlcube.yaml --task=evaluate --platform=singularity
+```
+
+We are targeting pull-type installation, so MLCube images should be available on Docker Hub. If not, try this:
+
+```Bash
+mlcube run ... -Pdocker.build_strategy=always
+```

brats/metrics/mlcube/mlcube.yaml

@@ -0,0 +1,26 @@
+name: MLCommons Brats metrics
+description: MLCommons Brats integration for metrics
+authors: 
+ - {name: "MLCommons Best Practices Working Group"}
+
+platform:
+  accelerator_count: 0
+
+docker:
+  # Image name.
+  image: mlcommons/brats_metrics:0.0.1
+  # Docker build context relative to $MLCUBE_ROOT. Default is `build`.
+  build_context: "../project"
+  # Docker file name within docker build context, default is `Dockerfile`.
+  build_file: "Dockerfile"
+
+tasks:
+  evaluate:
+  # Executes a number of metrics specified by the params file
+    parameters:
+      inputs: {
+        predictions: data/predictions/,
+        labels: data/ground_truth/,
+        parameters_file: {type: file, default: parameters.yaml}
+      }
+      outputs: {output_path: {type: "file", default: "results.yaml"}}

brats/metrics/mlcube/workspace/parameters.yaml

@@ -0,0 +1,2 @@
+treshold: 0.5
+eps: 0

brats/metrics/project/Dockerfile

@@ -0,0 +1,22 @@
+# for a CPU app use this Dockerfile.
+FROM python:3.8-buster
+
+# fill in your info here
+LABEL author="chuck@norris.org"
+LABEL application="your application name"
+LABEL maintainer="chuck@norris.org"
+LABEL version="0.0.1"
+LABEL status="beta"
+
+# basic
+RUN apt-get -y update && apt -y full-upgrade && apt-get -y install apt-utils wget git tar build-essential curl nano
+
+# install all python requirements
+WORKDIR /workspace
+COPY ./requirements.txt ./requirements.txt
+RUN pip3 install -r requirements.txt
+
+# copy all files
+COPY ./ /workspace
+
+ENTRYPOINT [ "python3", "/workspace/mlcube.py"]