Trilinos installation with candi and MKL fails

ASPECT
1

I have been installing ASPECT/dealii using the candi package. The instructions are very detailed and helpful. I have followed the instructions for using MKL and successfully gotten to the stage of building Trilinos. I initially used an older version of Cmake 3.7 (our default), which produced an error indicating that Cmake 3.10 or higher is required. I updated to Cmake 3.15 and re-ran the configure shell. Thankfully I skipped past everything prior to Trilinos (p4est, hdf5, etc.). I’m stuck and looking for a suggestion.

The log file indicates that Tpetra is disabled because “you are using MKL”. The error message is “Setting Trilinos_ENABLE_Amesos2=OFF which was ‘ON’ because Amesos2 has a required library dependence on disabled package Tpetra. I’m encouraged to look in …/trilinos-release-12-18-1/CMakeFiles/CMakeOutput.log” but there isn’t too much information here. Only one line

“The system is: Linux - 3.1.0-693.11.6.el7.x86_64 - x86_64”

I attach a screenshot of the last few lines of output. Suggestions welcome.

Bruce

Screen Shot 2020-06-17 at 4.08.17 PM
Screen Shot 2020-06-17 at 4.08.17 PM1067×334 74.8 KB

2

Hi Bruce,
I had a similar issue last week on Stampede, but I have not gotten around to looking more deeply into it. I worked around it by installing an older trilinos version (12-10-1 works for me). You can do so by modifying the file deal.II-toolchain/packages/trilinos.package inside candi and commenting the VERSION and CHECKSUM lines about trilinos 12-18-1 and uncommenting the 2 lines for version 12-10-1 (lines 31 and 32 in my version of candi). Rerunning candi should install the older version. Let us know how that goes. If this solves the issue can you open an issue in the candi repository (copying your message from here), because it might affect other deal.II users as well: https://github.com/dealii/candi/issues/new .

Best,
Rene

PS: I have moved your post to its own topic.

2 Likes
3

Thanks for the report. I will look into it as well.

Bruce and Rene: what is the compiler you are using (intel/gcc, which version)?

4

Hi Rene,

Thanks! I will give this a try.

With best wishes,
Bruce

5

Hi Timo,

The compiler I used was intel 2016.7.072

Thanks
Bruce

6

Sorry Timo, I mistyped. The compiler was intel 2016.4.072.

Bruce

7

Compiler: gcc/7.1.0
MPI: Intel MPI/17.0.3
MKL: 17.0.4
Candi: f436827a303b85a3a758eb1f7df269dc0ff8dd54

I attach the candi configure log of trilinos: candi_configure.log (5.0 KB)

To me it seems like MKL disables Tpetra, but Tpetra is needed for Amesos2, which is apparently required (Amesos2 appears under Explicitly enabled packages on input (by user):). Not sure if the dependencies changed between trilinos-12-10 and 12-18, or is something else is going on behind the scenes.

8

Hi Rene,

I followed your advice. I change the version of trilinos to version 12-10-1 and re-ran Candi.sh. It downloaded and starting building trilinos 12-10-1, as expected. I received the message about disabling Tpetra because “you are using MKL” . The BLAS_DIR and LAPACK_DIR are assigned to the MLK directory. All of this looks good.

The error message appeared again: ERROR: Setting Trilinos_ENABLE_Amesos2=OFF which was ‘ON’ because Amesos2 has a required library dependency on disabled package Tpetra! (The exclamation is included in the error message!) Is there any chance that the version of Cmake matters? I used version 3.7 to build everything up to p4est. The trilinos build failed because it needed 3.10 or higher. I shifted to version 3.15 to continue the build, which started with the trilinos package. I wondered if it would be reasonable to rebuild from scratch using Cmake 3.15. (Hard to see how this could matter. On the other hand, if I need to burn incense and chant during the build, so be it.)

Thanks
Bruce

9

Thanks, Rene. I can reproduce it. I will work on a fix in the next few days.

Bruce: Can you try deleting the ./tmp/ directory to make sure trilinos is configured from scratch? Or wait a few days for me to figure out what is going on.

10

Good suggestion. Thanks. I will give it a try.

Bruce

11

I managed to fix compilation with Intel 19.0.5. See https://github.com/dealii/candi/pull/150 (it will be ported to the 9.2 branch after it got merged).

12

Hi Timo,

Great new, thanks! So I just need to update compilers to 19.0.5 and be sure to pull a specific branch of Candi?

Best wishes,
Bruce

13

I would recommend using the 2019 or 2020 versions of the compiler, because these are the ones I have been using over the last 12 months. You might run into some issues with older versions (but the configuration error here has nothing to do with that).
Also, the pull request is not merged yet, sorry. If you don’t want to wait, you could do

git clone https://github.com/tjhei/candi.git
git checkout trilinos_mkl
14

Perfect, thanks!

Bruce

15

Hi Timo,

The version of Candi succeeded in building trilinos. Thanks! I encountered a problem with pets-lite-3.13.1. I’ve now run Candi.sh a couple of times because the first time there appeared to be a problem downloading the package. The fatal error in the build is the absence of “petscvariables” in …/tmp/build/petsc-lite-3.13.1/lib/conf/.

Maybe I could turn off the use the petsc?

Sorry to trouble you.

Bruce

16

Hi Bruce,

I usually only compile Trilinos, p4est, HDF5 (optional), and of course deal.II with candi. Assuming all of the other dependencies are available on the cluster (zlib, cmake, etc), this should be all that is needed to build ASPECT.

Cheers,
John

17

Hi John,

That did the job. Thanks! I was able to finish the deal-ii install and compile aspect.
One question, is there a way to allow a heat sink (i.e. a negative heat source)?
The entry “Radiogenic heating rate” requires a positive value. I would like to override
this restriction or possibly find an equivalent entry.

Thanks

18

Interesting. The easiest would be to remove that restriction by deleting “0.” in the highlighted line then recompile:

19

Interesting indeed – you piqued my curiosity: What physical effect would that
correspond to?

Cheers
W.

20

Thanks. This suggestion worked perfectly. We are trying to model the thermal boundary layer at the core-mantle boundary using strongly temperature dependent viscosity. Modeling the entire convecting layer with a cold temperature at the top and a hot temperature at the bottom produces stagnant lid convection. Our goal is to model the lower half of the mantle (avoiding the stagnant lid at the top) by representing the influence of subduction as a heat sink. A constant temperature is imposed on the lower boundary and insulating conditions are imposed on the sides and top. A heat sink draws heat across the lower boundary. We know that the time averaged heat flux at the lower boundary is equal to the volume-integrated heat sink, so the question is what temperature drop is needed across the lower boundary to achieve the imposed heat flow. This experiment allows us to test the so-called boundary layer model (Nu ~ Ra^{1/3}) in the presence of strongly temperature dependent viscosity. We can also assess whether a large drop in viscosity across the lower boundary layer allows a secondary cell of convection to develop. I’m really impressed with the powerful tools you are putting into the hands of the geodynamics community.

With best wishes,
Bruce