Error running and compiling ASPECT on Stampede2

[SarahChinski]

I have been having issues running ASPECT on Stampede. I was able to run and submit jobs earlier this spring, but now there seems to be an error connected to the ASPECT executable. Here is the exact text for the error:

error while loading shared libraries: libdeal_II.so.9.3.0: cannot open shared object file: No such file or directory

I realized this was an older version of the dealii library, so I followed the wiki installation instructions,) which was updated this April, in order to get the most recent version of ASPECT running. I was able to install dependencies using candi, but I ran into errors when attempting to build ASPECT.

After git cloning ASPECT into my software directory, I tried to make an empty build directory and run the cmake command, but there were various errors referencing necessary files (like CMakeLists.txt) that were in the software directory. So I ran the cmake in the main software directory, which completes without error:

However, trying to run the make command leads to an issue with some Unity files in the source code:

Have any other Stampede users run into issues running ASPECT?

[bangerth]

Sarah – I don’t know whether that is going to fix the issue, but:

• Start from scratch, with an ASPECT directory that does not contain any files that may have been left over from previous attempts at compiling ASPECT.
• Create a separate build directory and go into it. It should be empty.
• In your cmake call, the warning message says that you did not quite use the right syntax when calling cmake. What it should look like is cmake <-D flags> <source directory> where the <source directory> part is the name of a directory in which the ASPECT source files live. You forgot that last part, and cmake only succeeded because you happened to be in the source directory itself – but that’s not the recommended way, you want the build and source directories to be separate.
• Then call make again.

Can you report what happens then?

Best
Wolfgang

[SarahChinski]

Hi Wolfgang,

Thanks for your response- after starting with a fresh ASPECT directory and following your instructions to point to the source directory in the cmake command, I was able to download and compile ASPECT!

Unfortunately, this leads to a new error when I try to submit a job. The error output reads:

/work2/08418/tg877179/stampede2/software/aspect/build/aspect: symbol lookup error: /work2/08418/tg877179/stampede2/software/candi/deal.II-v9.3.3/lib/libdeal_II.so.9.3.3: undefined symbol: _ZN6Kokkos4Impl30serial_resize_thread_team_dataEmmmm

From this it looks like there was some error with the deal_ii library, but there were no issues during the candi installation. Could it have to do with the version of libdeal_ii being referenced? The deal.II-v9.3.3/lib directory contains several executables, including libdeal_II.g.so and libdeal_II.so.

Thanks for the help,
Sarah

[bangerth]

Sarah:
This error confuses me. deal.II 9.3 does not use Kokkos (though 9.5 does), so the missing reference must have come in via another dependency – probably Trilinos. What Trilinos version did you select for candi?

A separate question is why you are trying build such an old version of deal.II. Why not build the latest version of deal.II, and then switch to the latest version of ASPECT at the same time? It is conceivable that you will have to update some of your input files, but the changes should be relatively minor. I would give this a try.

Best
W.

[SarahChinski]

Hi Wolfgang,

You were right about Trilinos- I had added a line in my bashrc to load the most recent version of Trilinos, and that was creating the error above. Once I removed this line and re-submitted a job, it ran without issue!

While I mentioned I was trying to get the most recent deal_ii and ASPECT Installed, I was having trouble getting the most recent version, so I tried getting an earlier version (which had run on my Stampede account before) to work. Now that I was able to get an older version running, I will try to get the latest version of aspect and deal_ii. I also hadn’t realized that deal_ii was already on version 9.5!

Thanks for all the help,
Sarah

[SarahChinski]

I have an update installing the latest deal_ii. Unfortunately the candi install fails due to being unable to find a petsc directory. Here’s most of the error output:

Could not find the petsc library!

Could not find a sufficient PETSc installation:

PETSc has to be configured to use the same number of bits for the global
indices as deal.II, but found:

DEAL_II_WITH_64BIT_INDICES = ON
PETSC_WITH_64BIT_INDICES = (FALSE)

Could not find a sufficient PETSC installation:

PETSC has to be compiled against the same MPI library as deal.II but the
link line of PETSC contains:

/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/debug_mt/libmpi.so

which is not listed in MPI_LIBRARIES:

MPI_LIBRARIES = “/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/libmpicxx.so /opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/release/libmpi.so /lib64/librt.so /lib64/libpthread.so /lib64/libdl.so”

Please ensure that the petsc library version 3.3.0 or newer is installed on
your computer and is configured with the same mpi options as deal.II

If the library is not at a default location, either provide some hints

for the autodetection:

PETSc installed with --prefix=<…> to a destination:

PETSC_DIR="..." cmake <...> cmake -DPETSC_DIR=“…” <…>

PETSc compiled in source tree:

PETSC_DIR="..." PETSC_ARCH="..." cmake <...> cmake -DPETSC_DIR=“…” -DPETSC_ARCH=“…” <…>

or set the relevant variables by hand in ccmake.

I’m able to load a 64-bit version of petsc on Stampede, but even when I add the variable PETSC_DIR pointing to the correct location to my bash, I get the same error when trying to compile.

Digging into the petsc config file, I found a flag in the configure options “–with-64-bit-indices=1” and setting this to =0 doesn’t help the error:

DEAL_II_WITH_64BIT_INDICES = ON
PETSC_WITH_64BIT_INDICES = (FALSE)

There is a lot of info within this whole error output, and I would greatly appreciate some assistance in troubleshooting this new issue.

Thanks for your time,

[MFraters]

Hello Sarah,

You do not need PETSc in Deal ii to install ASPECT. You can turn it off in your candi script. An alternative is to use the aspect specific candi config file you can find here: https://github.com/geodynamics/aspect/blob/main/contrib/install/local.cfg. For more detailed instructions, please see Using candi to compile dependencies — ASPECT 2.6.0-pre.

I hope this helps.

Cheers,

Menno

[SarahChinski]

Hi Menno,

Thank you for the advice! I thought that petsc was a requirement for ASPECT, but commenting out that line in the candi.cfg file allowed me to bypass the error and finish the download. Then ASPECT compiled without issue!

I also appreciate the reference to the ASPECT-specific candi config file. I know the wiki page for installation on Stampede references the local.cfg file, but the standard git clone for candi doesn’t come with a local.cfg file.

Thanks for the help!
Sarah

[jbnaliboff]

Hi Sarah,

Glad to hear the installation worked.

I also appreciate the reference to the ASPECT-specific candi config file. I know the wiki page for installation on Stampede references the local.cfg file, but the standard git clone for candi doesn’t come with a local.cfg file.

Thanks for point this out, as the documentation linked above is specific about what happens when both candi.cfg and local.cfg are present in the directory. In detail, If local.cfg is present it overrides the configuration options set in candi.cfg. This is done here in candi.sh.

I opened a PR to add this bit the online manul.

Cheers,
John