Skip to content

Installation on Abel

Jump to bottom Edit New page
Björn Heyn edited this page Nov 12, 2018 · 7 revisions

Authors

John Naliboff jbnaliboff@ucdavis.edu, Axel Rosén and Björn Heyn.

Overview

Setting up and running ASPECT on Abel is quite straightforward as a precompiled version of both ASPECT and deal.II v9.0.1 can be loaded as a module. For further information on Abel please refer to the Abel information page.

Graphical results of ASPECT scaling tests run on Abel are available through the CIG code scaling page, while the raw results are available on the ASPECT performance statistics repository.

Account

You first need to obtain an an account on Abel, which is available to researchers in Norway through Notur or through the university of Oslo. For more information, see Abel's Getting Access page.

Running with precompiled ASPECT (2.1.0-pre)

A precompiled version of ASPECT is available through the ASPECT module, which can be loaded through the command: module load aspect/2.0.1. Notably, this module should also be loaded in the job script used to submit ASPECT runs. See the Abel Job Scripts page for more information.

For the module, ASPECT is compiled in 'Release' mode. If you want to test a new plugin, consider installing your own version of ASPECT in 'Debug' mode in your home directory to test your runs before using the module. Instructions on how to do that are given further down.

As shown below, loading the ASPECT module also loads supporting modules and sets environment variables pointing to the ASPECT binary and deal.II installation directory.

-bash-4.1$ module load aspect/2.0.1
-bash-4.1$ 
-bash-4.1$ module list
Currently Loaded Modulefiles:
  1) gcc/5.2.0            2) openmpi.gnu/1.10.2   3) aspect/2.0.1
-bash-4.1$ 
-bash-4.1$ which aspect
/cluster/software/VERSIONS/aspect/2.0.1/bin/aspect
-bash-4.1$ 
-bash-4.1$ echo $DEAL_II_DIR
/cluster/software/VERSIONS/aspect/2.0.1/dealii/deal.II-v9.0.1
-bash-4.1$

Installation with precompiled deal.II (v9.0.1)

If you would like to compile your own version of ASPECT, you can use one of the precompiled versions of deal.II v9.0.1. There are two different installations of deal.II v9.0.1 available on Abel as described below.

To speed up installation of ASPECT, one can start an interactive compute node session on Abel and use up to 16 processors when building ASPECT with the 'make' command (e.g., make -j16)

'Release' version for optimized runtime

One is compiled in the optimized 'Release' mode, which has improved performance with respect to run time, but does not include various flags used for debugging. Due to the lack of descriptive error messages, finding mistakes in the code becomes difficult.

The 'Release' version is available under /cluster/software/VERSIONS/aspect/2.0.1/dealii/deal.II-v9.0.1/ and is the one used in the precompiled ASPECT module. To use it, either load the ASPECT module shown above and use the following variable when configuring ASPECT:

-DDEAL_II_DIR=$DEAL_II_DIR

or set the path to deal.II directly during the configuration:

-DDEAL_II_DIR=/cluster/software/VERSIONS/aspect/2.0.1/dealii/deal.II-v9.0.1/

'Debug' version for testing new plugins

The other installation is compiled in 'Debug' mode and runs slower, but is able to capture most mistakes one usually makes when writing a program. It performs extensive checks for various types of errors and prints out descriptive error messages. Thus, if you want to test for example a new plugin, the 'Debug' version is to be preferred. Once you know that your code works as expected, you can recompile ASPECT with the optimized version of deal.II to improve its performance.

It is available under /cluster/software/VERSIONS/aspect/2.0.1/dealii-debugrelese/deal.II-v9.0.1/ (which in fact also contains the optimized version of deal.II). To use it, configure ASPECT with:

-DDEAL_II_DIR=/cluster/software/VERSIONS/aspect/2.0.1/dealii-debugrelese/deal.II-v9.0.1/

Installation of deal.II using candi

In case you want to install deal.II yourself, for example if you require a different version of deal.II, you can use the candi-installer and a cluster-wide version of MKL.

  1. Log in to Abel and start an interactive session on a node, using your project code with --account= 
-bash-4.1$ qlogin --account=PROJECT_CODE --ntasks-per-node=16  --constraint='intel&ib'
  1. Unload previously loaded modules (if any) and load the required modules:
-bash-4.1$ module load cmake
-bash-4.1$ module load openmpi.gnu/1.10.2

This should load the following modules:

-bash-4.1$ module list
Currently Loaded Modulefiles:
    1) cmake/3.7.1      2)gcc/5.2.0     3) openmpi.gnu/1.10.2

and set the environment variables:

-bash-4.1$ export CC=mpicc
-bash-4.1$ export FC=mpif90
-bash-4.1$ export FF=mpif77
-bash-4.1$ export CXX=mpicxx
-bash-4.1$ export MKL_LIB=/cluster/software/VERSIONS/intel-2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/
  1. Create the folder where you want to install deal.II and download candi
-bash-4.1$ mkdir -p Software/dealii
-bash-4.1$ cd Software/dealii
-bash-4.1$ git clone https://github.com/dealii/candi
-bash-4.1$ cd candi
  1. Open candi.cfg and modify the following lines:
  • change DEAL_CONFOPTS to
DEAL_CONFOPTS="-DDEAL_II_WITH_THREADS=OFF \
                -DDEAL_II_WITH_MPI=ON \
                -DLAPACK_LIBRARIES='${MKL_LIB}/libmkl_intel_lp64.so;${MKL_LIB}/libmkl_intel_thread.so;${MKL_LIB}/libmkl_core.so;${MKL_LIB}/../../../compiler/lib/intel64_lin/libiomp5.so;${MKL_LIB}/libmkl_intel_lp64.so;${MKL_LIB}/libmkl_intel_thread.so;${MKL_LIB}/libmkl_core.so;${MKL_LIB}/../../../compiler/lib/intel64_lin/libiomp5.so'"
  • comment out (i.e. add # to beginning of line) the packages
#PACKAGES="${PACKAGES} once:opencascade"
#PACKAGES="${PACKAGES} once:parmetis"

#PACKAGES="${PACKAGES} once:petsc"
#PACKAGES="${PACKAGES} once:slepc"
  • set MKL
MKL=ON
MKL_DIR=/cluster/software/VERSIONS/intel-2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/
  1. Compile deal.II
-bash-4.1$ ./candi.sh -j 8 --platform=./deal.II-toolchain/platforms/supported/linux_cluster.platform -p ~/Software/dealii/install/
  1. Set the environment to use deal.II, i.e. set the paths to the installed packages
-bash-4.1$ source ~/Software/dealii/install/configuration/enable.sh

If you want to use ASPECT continuously, consider adding this line and loading the module files (step 2) to your ~/.bashrc file. Or create a file (e.g. ASPECT_prep.sh) in ~/bin/, which looks like:

!#bin/bash

# Load module files
module load cmake
module load load openmpi.gnu/1.10.2

# Export environment variables
export CC=mpicc
export FC=mpif90
export FF=mpif77
export CXX=mpicxx

# Source package paths
source ~/Software/dealii/install/configuration/enable.sh

and use it with

-bash-4.1$ source ~/bin/ASPECT_prep.sh
Add a custom footer
Add a custom sidebar
Clone this wiki locally