Offline Installation
If you are using a machine (or HPC cluster) with no access to the internet, the following instructions will help you create an environment suitable for MOOSE-based development.
Please be certain, that the machine in which you intend to perform the actual work (which this document will refer to as the 'target' machine), meets the following requirements:
Prerequisites
Minimum System Requirements
In general, the following is required for MOOSE-based development:
GCC/Clang C++17 compliant compiler (GCC @ 7.5.0, Clang @ 10.0.1 or greater)
Note: Intel compilers are not supported.
Memory: 8 GBs of RAM for optimized compilation (16 GBs for debug compilation), 2 GB per core execution
Processor: 64-bit x86 or ARM64 (specifically, Apple Silicon)
Disk: 30GB
A POSIX compliant Unix-like operating system, including the two most recent versions of MacOS and most current versions of Linux.
Git version control system
Python @ 3.7 or greater
Air-Gapped HPC Clusters
If the target machine is an HPC cluster, it is likely your cluster already has a proper toolchain stack available. Please contact your system administrator and ask them how to appropriately make use of your cluster's MPI wrapper and compiler. Please note, Intel compilers are not supported.
Personal Workstation
If the target machine is your personal workstation, that machine must sufficiently achieve the following:
A sane, modern developers environment (toolchain). Examples how to do this for several popular operating systems:
Macintosh: Xcode (
xcode-select --install)Windows: Windows Subsystem for Linux (WSL)
In addition, a chosen Linux flavor below
Ubuntu: (
apt install build-essential)CentOS/Rocky: (
dnf groupinstall 'Development Tools')OpenSUSE: (
zypper install --type pattern devel_basis)
Different operating systems leverage different means by which to obtain a developers environment. We assume the reader is intricately familiar with their platform of choice, and has established such an environment before proceeding.
In addition, the target machine will need an MPI wrapper. We recommend one of the following products:
For Linux users, the easiest way to install an MPI wrapper is via the same system package manager tool used obtaining a developers environment. Examples:
apt install openmpi # Ubuntu, Debian
dnf install mpich # CentOS, Rocky, RHEL
zypper install mpich # OpenSUSE
If you choose this method, more likely than not you will still need to export variables to enable your MPI wrapper:
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77
Exporting variables is only temporary in the session you executed them in.
For this reason you may wish to add the above export command to your shell startup profile so that these variables are always set. Achieving this is different for each shell type (and there are a number of them). You'll want to read up on how to do this for your particular shell.
You can determine what shell your environment operates within by running the following command:
echo $0
Procuring the above on a workstation with no network access is beyond the scope of this document. Please work with your system administrator on satisfying the above prerequisites.
Prepare Directory
PETSc, libMesh and MOOSE can all be obtained from one repository (the moose repository). First, create a top-level directory named ~/projects, which will contain everything we need to transfer over to your target machine.
mkdir -p ~/projects/downloads
Next, enter the ~/projects directory, and perform the cloning operation to obtain all three libraries:
cd ~/projects
git clone https://github.com/idaholab/moose.git
cd moose
git checkout master
git submodule update --init
git submodule foreach --recursive git submodule update --init
With PETSc cloned as part of the group obtained above, we can use a configure option in PETSc, to obtain a list of contributions we will need to download manually (--with-package-download-dir):
cd ~/projects/moose
./scripts/update_and_rebuild_petsc.sh --with-packages-download-dir=~/projects/downloads
As an example, the above command should return something like the following:
===============================================================================
Configuring PETSc to compile on your system
===============================================================================
Download the following packages to /home/you/projects/downloads
fblaslapack ['git://https://bitbucket.org/petsc/pkg-fblaslapack', 'https://bitbucket.org/petsc/pkg-fblaslapack/get/v3.4.2-p3.tar.gz']
hypre ['git://https://github.com/hypre-space/hypre', 'https://github.com/hypre-space/hypre/archive/93baaa8c9.tar.gz']
metis ['git://https://bitbucket.org/petsc/pkg-metis.git', 'https://bitbucket.org/petsc/pkg-metis/get/v5.1.0-p8.tar.gz']
parmetis ['git://https://bitbucket.org/petsc/pkg-parmetis.git', 'https://bitbucket.org/petsc/pkg-parmetis/get/v4.0.3-p6.tar.gz']
ptscotch ['git://https://gitlab.inria.fr/scotch/scotch.git', 'https://gitlab.inria.fr/scotch/scotch/-/archive/v6.0.9/scotch-v6.0.9.tar.gz', 'http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.0.9.tar.gz']
mumps ['git://https://bitbucket.org/petsc/pkg-mumps.git', 'https://bitbucket.org/petsc/pkg-mumps/get/v5.2.1-p2.tar.gz']
scalapack ['git://https://bitbucket.org/petsc/pkg-scalapack', 'https://bitbucket.org/petsc/pkg-scalapack/get/v2.1.0-p1.tar.gz']
superlu_dist ['git://https://github.com/xiaoyeli/superlu_dist', 'https://github.com/xiaoyeli/superlu_dist/archive/v6.3.0.tar.gz']
slepc ['git://https://gitlab.com/slepc/slepc.git', 'https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz']
Your job, will be to parse through the above jargon, and download these packages to ~/projects/downloads. Be certain to preserve the file name. PETSc is listing several means for which you can obtain these contributions (using either git or traditional web link). For the purpose of simplicity, we will use the traditional web method.
Example ONLY, as file names may be deprecated!:
cd ~/projects/downloads
curl -L -O https://bitbucket.org/petsc/pkg-fblaslapack/get/v3.4.2-p3.tar.gz
curl -L -O https://github.com/hypre-space/hypre/archive/93baaa8c9.tar.gz
curl -L -O https://bitbucket.org/petsc/pkg-metis/get/v5.1.0-p8.tar.gz
curl -L -O https://bitbucket.org/petsc/pkg-parmetis/get/v4.0.3-p6.tar.gz
curl -L -O http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.0.9.tar.gz
curl -L -O https://bitbucket.org/petsc/pkg-mumps/get/v5.2.1-p2.tar.gz
curl -L -O https://bitbucket.org/petsc/pkg-scalapack/get/v2.1.0-p1.tar.gz
curl -L -O https://github.com/xiaoyeli/superlu_dist/archive/v6.3.0.tar.gz
curl -L -O https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz
At this point, everything necessary should be downloaded and available in the directory hierarchy at ~/projects. At this time, you may copy this directory to your offline-no-internet access machine's home directory. At the time of this writing, tallying up the disc space used equates to approximately 2GB. Depending on your internet connection, you may want to compress the entire ~/projects/downloads and ~/projects/moose directories into a single compressed tarball archive (saves about 500Mb):
cd ~/
tar -pzcf offline.tar.gz projects/downloads projects/moose
If operating on a Macintosh machine, creating a tarball to be extracted on a Linux machine produces warnings. They are warnings only, and can be safely ignored.
If copying the tarball over, you can extract it with:
tar -xf offline.tar.gz -C ~/
Build Libraries
With ~/projects/moose and ~/projects/downloads directories available on the target machine, we can now build PETSc, libMesh and MOOSE, using your MPI Wrapper/Compiler established in earlier steps.
Verify MPI
Before proceeding verify that your MPI wrapper is accessible:
which $CC $CXX $FC $F77
If the above command returns nothing, produces errors, or fewer paths than the 4 we are asking for, something is wrong. You need to STOP, and figure out how to enable your MPI wrapper before proceeding (discussed in the Prerequisites section).
Build PETSc, libMesh, and WASP
Build PETSc, libMesh, and WASP by instructing these libraries to use the downloads directory you created:
cd ~/projects/moose/scripts
export MOOSE_JOBS=6 METHODS=opt
./update_and_rebuild_petsc.sh --skip-submodule-update --with-packages-download-dir=~/projects/downloads
./update_and_rebuild_libmesh.sh --skip-submodule-update
./update_and_rebuild_wasp.sh --skip-submodule-update
MOOSE_JOBS is a loose influential environment variable that dictates how many cores to use when executing many of our scripts.
METHODS is an influential environment variable that dictates how to build libMesh. If this variable is not set, libMesh will by default build 4 methods (taking 4x longer to finish).
Unfortunately, any errors incurred during the above step is going to be beyond the scope of this document. Most likely, an error will be related to a missing library by one of the myriad contributions we are asking to build PETSc with. Please submit a detailed log of the error, to the MOOSE Discussion forum. But do be prepared to be asked to contact your system administrator; Errors of this nature normally require admin rights to fulfill the dependency.
Build and Test MOOSE
To build MOOSE run the following commands:
cd ~/projects/moose/test
make -j 6
Again, any errors incurred during this step, is going to be beyond the scope of this document. Please submit a detailed log of the error, to the MOOSE Discussion forum.
To test MOOSE, run the following commands:
cd ~/projects/moose/test
./run_tests -j 6
Some tests are SKIPPED. This is normal as some tests are specific to available resources, or some other constraint your machine does not satisfy. If you see failures, or you see MAX FAILURES, thats a problem! And it needs to be addressed before continuing:
Supply a report of the actual failure (scroll up a ways). For example the following snippet does not give the full picture (created with
./run_tests -i always_bad):Final Test Results: -------------------------------------------------------------------------------- tests/test_harness.always_ok .................... FAILED (Application not found) tests/test_harness.always_bad .................................. FAILED (CODE 1) -------------------------------------------------------------------------------- Ran 2 tests in 0.2 seconds. Average test time 0.0 seconds, maximum test time 0.0 seconds. 0 passed, 0 skipped, 0 pending, 2 FAILEDInstead, you need to scroll up and report the actual error:
tests/test_harness.always_ok: Working Directory: /Users/me/projects/moose/test/tests/test_harness tests/test_harness.always_ok: Running command: tests/test_harness.always_ok: tests/test_harness.always_ok: #################################################################### tests/test_harness.always_ok: Tester failed, reason: Application not found tests/test_harness.always_ok: tests/test_harness.always_ok .................... FAILED (Application not found) tests/test_harness.always_bad: Working Directory: /Users/me/projects/moose/test/tests/test_harness tests/test_harness.always_bad: Running command: false tests/test_harness.always_bad: tests/test_harness.always_bad: ################################################################### tests/test_harness.always_bad: Tester failed, reason: CODE 1 tests/test_harness.always_bad: tests/test_harness.always_bad .................................. FAILED (CODE 1)
Now that you have a working MOOSE, and you know how to make your MPI wrapper available, proceed to 'New Users' to begin your tour of MOOSE!