Multi-User Cluster Instructions

This document will aid an HPC Administrator on building an environment which multiple end-users will use when compiling and running MOOSE based applications. If you do not have administrative rights, you will not be able to complete these instructions! Please forward these instructions to your HPC Administrator.

Because this document assumes the reader is an administrator, most of the content herein are suggestions, and not 'copy, paste, enter' instructions.

Prerequisites

  • Some sort of environmental module management software. Such as Environment Modules.

  • A working MPI wrapper (MPICH/OpenMPI/MVAPICH) which wraps to a C++11 compliant compiler.

  • Knowledge on Access Control Lists (ACLs) or other means to safeguard a directory from further tampering by others.

warning

Please use a single solitary terminal session throughout and to the completion of these instructions.

Environment Setup

Begin by creating an area for which to build:


export STACK_SRC=`mktemp -d /tmp/stack_src_temp.XXXXXX`
cd $STACK_SRC

Set your umask

Some systems have a secure umask set. We need to adjust our umask so that when you write a file (make install), it is readable by everyone:


umask 0022

Choose a base path

Export a base path variable which will be the home location for the compiler stack. All files related to libraries necessary to build MOOSE, will be stored in this location (choose carefully, as this location should be accessible from all nodes on your cluster):


export PACKAGES_DIR=/opt/moose

Create and chown $PACKGES_DIR

History teaches us, that implicitly trusting scripts we download off the internet with root access, is a very bad idea. So let us create and chown the $PACKAGES_DIR directory before we install anything. That way, the things we do install can be done so without invoking sudo:


sudo mkdir -p $PACKAGES_DIR
sudo chown -R <your user id> $PACKAGES_DIR
warning

Verify that your umask settings are indeed set to 0022 before continuing:


$> umask
0022

Set Up Modules

Even if you are not using Modules, the following provides information on what environment variables are needed for MOOSE developement.

Create a MOOSE module:


mkdir -p $PACKAGES_DIR/modulefiles
vi $PACKAGES_DIR/modulefiles/moose-dev-gcc

Add the following content to that file:

#%Module1.0#####################################################################
##
## MOOSE module

##
set base_path   INSERT PACKAGES_DIR HERE

GCC MPI PATHS

prepend-path    PATH             /GCC and MPI /bin
prepend-path    LD_LIBRARY_PATH  /GCC and MPI /lib

setenv CC       mpicc
setenv CXX      mpicxx
setenv F90      mpif90
setenv F77      mpif77
setenv FC       mpif90

setenv          PETSC_DIR        $base_path/petsc/petsc-__PETSC_DEFAULT__

# Optional if miniconda is installed
prepend-path    PATH             $base_path/miniconda/bin
note

Replace INSERT PACKAGES_DIR HERE with whatever you had set for $PACKAGES_DIR (do not literally enter: $PACKAGES_DIR. As an example, if you left packages_dir as: /opt/moose, then that is what you would enter)

Replace GCC and MPI paths with any additional information needed to make GCC/MPI work on your cluster.

To make the module available in your terminal session, export the following:


export MODULEPATH=$MODULEPATH:$PACKAGES_DIR/modulefiles

The above command should be added to the list of other global profiles (perhaps in /etc/profiles.d). That way, the above is performed as the user logs into the machine.

With the modulefile in place and the MODULEPATH variable set, see if our module is available for loading:


module load moose-dev-gcc

Verify that this module loads properly by attempting to echo $PETSC_DIR:

echo $PETSC_DIR
/opt/moose/petsc/petsc-__PETSC_DEFAULT__

While we are at it, verify that your MPI wrapper works by running a few commands (your results will vary, but they should return something):


which mpicc
/opt/moose/mpich-3.2/gcc-7.3.0/bin/mpicc

mpicc -show
gcc -I/opt/moose/mpich-3.2/gcc-7.3.0/include -L/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,-rpath -Wl,/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,--enable-new-dtags -lmpi

which gcc
/opt/moose/gcc-7.3.0/bin/gcc

Leave this module loaded for the remainder of the instructions (PETSc requirements).

PETSc

Download PETSc 3.8.3

cd $STACK_SRC
curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-__PETSC_DEFAULT__.tar.gz
tar -xf petsc-__PETSC_DEFAULT__.tar.gz -C .

Now we configure, build, and install it

cd $STACK_SRC/petsc-__PETSC_DEFAULT__

./configure \
--prefix=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-scalapack=1 \
--CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 \
--CFLAGS='-fPIC -fopenmp' \
--CXXFLAGS='-fPIC -fopenmp' \
--FFLAGS='-fPIC -fopenmp' \
--FCFLAGS='-fPIC -fopenmp' \
--F90FLAGS='-fPIC -fopenmp' \
--F77FLAGS='-fPIC -fopenmp' \
PETSC_DIR=`pwd`

Once configure is done, we build PETSc

make PETSC_DIR=$STACK_SRC/petsc-__PETSC_DEFAULT__ PETSC_ARCH=arch-linux2-c-opt all

Everything good so far? PETSc should be asking to run more make commands

make PETSC_DIR=$STACK_SRC/petsc-__PETSC_DEFAULT__ PETSC_ARCH=arch-linux2-c-opt install

And now after the install, we can run some built-in tests

make PETSC_DIR=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ PETSC_ARCH="" test

Running the tests should produce some output like the following:

[moose@centos-7 petsc-__PETSC_DEFAULT__]$ make PETSC_DIR=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/opt/moose/petsc-__PETSC_DEFAULT__ and PETSC_ARCH=
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
=========================================

Miniconda

Peacock (an optional MOOSE GUI frontend) uses many libraries. The easiest way to obtain these libraries, is to install miniconda, along with several miniconda/pip packages.

cd $STACK_SRC
curl -L -O https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
sh Miniconda2-latest-Linux-x86_64.sh -b -p $PACKAGES_DIR/miniconda

PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda config --set ssl_verify false
PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda install -c idaholab python=__CONDA_PYTHON__ coverage \
reportlab \
mako \
numpy \
scipy \
scikit-learn \
h5py \
hdf5 \
scikit-image \
requests \
vtk=__CONDA_VTK__ \
pyyaml \
matplotlib \
pip \
lxml \
pyflakes \
pandas \
conda-build \
mock \
yaml \
pyqt \
swig --yes
note

Peacock (as well as the TestHarness sytem in MOOSE), does not work with Python3. Please chose Miniconda2 for Python 2.7 instead.

Next, we need to use pip to install additional libraries not supplied by conda:

PATH=$PACKAGES_DIR/miniconda/bin:$PATH pip install --no-cache-dir pybtex livereload==__PIP_LIVERELOAD__ daemonlite pylint==__PIP_PYLINT__ lxml pylatexenc anytree

Clean Up and Chown

Clean all the temporary stuff and change the ownership to root, so no further writes are possible:


rm -rf $STACK_SRC
sudo chown -R root:root $PACKAGE_DIR

This concludes setting up the environment for MOOSE-based development. However you decide to instruct your users on enabling the above environment, each user will need to perform the instructions provided by the following link: Obtaining and Building MOOSE.