-
Notifications
You must be signed in to change notification settings - Fork 235
Compiling and Running ASPECT on TACC Stampede2
John Naliboff john.naliboff@nmt.edu Haoyuan Li hylli@ucdavis.edu
Portions of the text and formatting below were taken directly from the instructions for Installation on TACC Stampede written by Sarah Stamps, Jonathan Perry-Houts and Eric Heien.
Setting up ASPECT on TACC Stampede2 is much easier than on previous systems since most of the necessary libraries are already correctly installed and configured. Following all the instructions in this document should allow you to start running ASPECT within 30-60 minutes. For further information on Stampede2 you can also refer to the user guide at https://portal.tacc.utexas.edu/user-guides/stampede2.
Please note that many parts of the documentation have changed from the instructions on the original Stampede. Even if you have set up ASPECT before, please follow all sections of this document to make sure your current setup is installed and configured correctly.
Also note that the instructions below refer strictly to installation on the Stampede2 Skylake nodes (SKX). While one can modify the instructions slightly to build and run ASPECT on the Knights Landing nodes (KNL), at present scaling results have only been completed on the SKX nodes.
Graphical results of these scaling tests are available through CIG (https://geodynamics.org/cig/dev/xsede-resources/code-scaling/), while the raw results are available on the ASPECT performance statistics github page (https://github.com/gassmoeller/aspect-performance-statistics).
You first need to obtain an XSEDE account at http://portal.xsede.org/. If you need one send an email to: cig-help@geodynamics.org.
When you have an account created on XSEDE and are assigned resources on Stampede2, a TACC (Texas Advanced Computing Center) account will automatically be created with information sent to the email address associated with your XSEDE account. If you already have a TACC account, additional steps may be needed to link your XSEDE and TACC accounts. Notably, you will also need to setup the TACC two-factor authentication app (TACC Token) on a mobile device if you wish to login into to login into Stampede2 directly via ssh: ssh tacc_username@stampede.tacc.utexas.edu.
Alternatively, one may log in to Stampede2 via the XSEDE Single Sign-On Hub .
The most straightforward method to build ASPECT's key dependencies (deal.II, Trilinos, p4est) is via the candi package, which builds and installs these packages on most Linux-based systems. To speed up the installation significantly we will also use a Skylake development node, which provides access to 48 cores.
- After logging into Stampede2, start by going to the "work2" directory and making a series of folders:
cd $WORK
mkdir software
cd software
While the directory structure and folder names above can be modified to include less or more detail, this will require changing the specified PATHS in a number of steps below.
idev -p skx-dev -N 1 -n 48 -m 120
Respectively, N, n and m refer to the number of nodes, cores and minutes for the interactive session. Note that your account will be charged based on the length of the session, rather than the requested total number of minutes. If you have access to more than one XSEDE allocation with time on Stampede2, you will need to specify the account number by adding -A account_number to the idev command.
In case the compilation is on the knl nodes, use:
idev -p normal -N 1 -n 24 -m 240
Here, we want to use "-n 24" for 24 cpus. The reason for not using all the 68 cpus on a node is the total ram on one knl node is limited to 96 GB. Using fewer cores would ensure each of them has enough ram to get through the compilation. Moreover, it will take longer to complete compared to a compilation on a skx node, thus a long time "-m 240" is assigned here.
- Load the required modules and set environment variables.
Add the following lines to the file .bashrc in your home directory
module unload intel python
module load cmake gcc/9.1.0 mkl/19.1.1
- Download and configure candi for deal.II version 9.3.3
git clone https://github.com/dealii/candi.git
cd candi
Replace the content of the file local.cfg by the following:
NATIVE_OPTIMIZATIONS=ON
BUILD_EXAMPLES=OFF
USE_DEAL_II_CMAKE_MPI_COMPILER=ON
MKL=ON
MKL_DIR=$MKLROOT/lib/intel64
DEAL_II_CONFOPTS="-D DEAL_II_FORCE_BUNDLED_THREADS=ON -D DEAL_II_WITH_COMPLEX_VALUES=OFF -D DEAL_II_COMPONENT_EXAMPLES=OFF -D DEAL_II_WITH_64BIT_INDICES=ON -D DEAL_II_LINKER_FLAGS='-fuse-ld=bfd'"
PACKAGES="load:dealii-prepare once:astyle once:hdf5 once:p4est once:trilinos dealii"
DEAL_II_VERSION=v9.4.2
- Install deal.II 9.4.2 (~ 30-60 minutes)
./candi.sh -p=INSTALL_PATH -j 48
where INSTALL_PATH is the path where you want to install deal.ii, for example $WORK/software/dealii-v9.4.2
- Download, configure and install ASPECT version (tested with the 2.4 release and developer version from April 2023)
Download:
cd $WORK2/software
git clone https://github.com/geodynamics/aspect.git
cd aspect
In case you want to use a specific release, switch to the corresponding branch:
git checkout -b aspect-2.3 origin/aspect-2.3
Configure and install ASPECT:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DDEAL_II_DIR=$WORK/software/dealii-v9.4.2/
make -j48
The most straightforward method to build ASPECT's key dependencies (deal.II, Trilinos, p4est) is via the candi package, which builds and installs these packages on most Linux-based systems.To speed up the installation significantly we will also use a Skylake development node, which provides access to 48 cores.
- After logging into Stampede2, start by going to the "Work2" directory ("Work" is deprecated and now read-only) and making a series of folders:
cd $WORK2
mkdir software
cd software
While the directory structure and folder names above can be modified to include less or more detail, this will require changing the specified PATHS in a number of steps below.
idev -p skx-dev -N 1 -n 48 -m 120
Respectively, N, n and m refer to the number of nodes, cores and minutes for the interactive session. Note that your account will be charged based on the length of the session, rather than the requested total number of minutes. If you have access to more than one XSEDE allocation with time on Stampede2, you will need to specify the account number by adding -A account_number to the idev command.
In case the compilation is on the knl nodes, use:
idev -p normal -N 1 -n 24 -m 240
Here, we want to use "-n 24" for 24 cpus. The reason for not using all the 68 cpus on a node is the total ram on one knl node is limited to 96 GB. Using fewer cores would ensure each of them has enough ram to get through the compilation. Moreover, it will take longer to complete compared to a compilation on a skx node, thus a long time "-m 240" is assigned here.
- Load the required modules and set environment variables.
module load gcc/7.1.0
export FC="${TACC_IMPI_BIN}"/mpif90
export CC="${TACC_IMPI_BIN}"/mpicc
export CXX="${TACC_IMPI_BIN}"/mpicxx
- Download and configure candi for deal.II version 9.3.3
git clone https://github.com/dealii/candi.git
cd candi
Next, search for and edit (or add) the specific line in candi.cfg as follows:
DEAL_II_CONFOPTS="-DDEAL_II_FORCE_BUNDLED_THREADS=ON -DDEAL_II_COMPONENT_EXAMPLES:BOOL=OFF -DDEAL_II_LINKER_FLAGS='-fuse-ld=bfd'"
Also modify the line below so that NATIVE_OPTIMIZATIoNS is set to ON:
NATIVE_OPTIMIZATIONS=ON
Make sure the deal.II version is set to 9.3.3:
DEAL_II_VERSION=v9.3.3
There are multiple packages that can be installed with candi, and packages without a
# in front of PACKAGES= are installed. In candi.cfg, only the following packages
below should be installed. Place a # in front of all of the other packages.
PACKAGES="${PACKAGES} once:openblas"
PACKAGES="${PACKAGES} once:hdf5"
PACKAGES="${PACKAGES} once:p4est"
PACKAGES="${PACKAGES} once:trilinos"
PACKAGES="${PACKAGES} dealii"
- Change the version of OpenBlas built by candi from 0.3.13 to 0.3.7
From the main candi directory, cd into the folder deal.II-toolchain/packages.
Within this folder, open the file openblas.package. Inside openblas.package,
Add a # in front of the line VERSION=0.3.13 and the one beneath it that begins
with CHECKSUM. These are the first two lines in file.
Next, remove the # at the beginning of the line #VERSION=0.3.7. Also remove
the # at the beginning of the next line (#CHECKSUM=48637....)
This will change the version of OpenBlas built by candi from 0.3.13 to 0.3.7.
- Install deal.II 9.3.3 (~ 30-60 minutes)
./candi.sh -j 48 --prefix="$WORK2"/software/candi/install/
Log out and log back in to let the settings take effect (or execute the command in your current shell).
- Download, configure and install ASPECT version 2.3
Download:
cd $WORK2/software
git clone https://github.com/geodynamics/aspect.git
cd aspect
git checkout -b aspect-2.3 origin/aspect-2.3
Configure and install:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DDEAL_II_DIR=$WORK2/software/candi/install/ ../
make -j48
Any runs that require more than one node or multi-hour run times will require submitting jobs to a "normal" compute node (i.e., not development) to the Stampede2 job scheduler system (SLURM). This is accomplished by submitting a job script via the sbatch command.
An example job script is posted below, which would be submitted with sbatch file_name.sh (e.g., if the file is named run.sh then the command is sbatch run.sh). Significantly, note the selected partition (i.e., node type) is skx-normal. Given that we compiled on an skx node above via the interactive idev session, ASPECT and all of the dependencies will only be able to run on the skx nodes (skx-normal or skx-dev)!
Note: specifying the partition (node) to submit to as normal will submit to a knl compute node, rather than skx regular compute node. The correct partition to select is skx-normal (#SBATCH -p skx-normal).
#!/bin/bash
#SBATCH -J aspmod1
#SBATCH -o log_aspect_model_1
#SBATCH -e %j
#SBATCH -p skx-normal
#SBATCH -t 48:00:00
#SBATCH -N 4
#SBATCH -n 192
# load modules
module load gcc/7.1.0
# Aspect executable
ASP="$WORK2/software/build/./aspect"
# Run model. Submit job with "sbatch run.sh"
ibrun $ASP aspect_model_1.prm