Trilinos installation with candi and MKL fails

bbuffett · June 17, 2020, 11:08pm

I have been installing ASPECT/dealii using the candi package. The instructions are very detailed and helpful. I have followed the instructions for using MKL and successfully gotten to the stage of building Trilinos. I initially used an older version of Cmake 3.7 (our default), which produced an error indicating that Cmake 3.10 or higher is required. I updated to Cmake 3.15 and re-ran the configure shell. Thankfully I skipped past everything prior to Trilinos (p4est, hdf5, etc.). I’m stuck and looking for a suggestion.

The log file indicates that Tpetra is disabled because “you are using MKL”. The error message is “Setting Trilinos_ENABLE_Amesos2=OFF which was ‘ON’ because Amesos2 has a required library dependence on disabled package Tpetra. I’m encouraged to look in …/trilinos-release-12-18-1/CMakeFiles/CMakeOutput.log” but there isn’t too much information here. Only one line

“The system is: Linux - 3.1.0-693.11.6.el7.x86_64 - x86_64”

I attach a screenshot of the last few lines of output. Suggestions welcome.

Bruce

gassmoeller · June 18, 2020, 12:47am

Hi Bruce,
I had a similar issue last week on Stampede, but I have not gotten around to looking more deeply into it. I worked around it by installing an older trilinos version (12-10-1 works for me). You can do so by modifying the file deal.II-toolchain/packages/trilinos.package inside candi and commenting the VERSION and CHECKSUM lines about trilinos 12-18-1 and uncommenting the 2 lines for version 12-10-1 (lines 31 and 32 in my version of candi). Rerunning candi should install the older version. Let us know how that goes. If this solves the issue can you open an issue in the candi repository (copying your message from here), because it might affect other deal.II users as well: https://github.com/dealii/candi/issues/new .

Best,
Rene

PS: I have moved your post to its own topic.

tjhei · June 18, 2020, 1:34am

Thanks for the report. I will look into it as well.

Bruce and Rene: what is the compiler you are using (intel/gcc, which version)?

bbuffett · June 18, 2020, 1:37am

Hi Rene,

Thanks! I will give this a try.

With best wishes,
Bruce

bbuffett · June 18, 2020, 2:23am

Hi Timo,

The compiler I used was intel 2016.7.072

Thanks
Bruce

bbuffett · June 18, 2020, 3:23am

Sorry Timo, I mistyped. The compiler was intel 2016.4.072.

Bruce

gassmoeller · June 18, 2020, 7:25pm

Compiler: gcc/7.1.0
MPI: Intel MPI/17.0.3
MKL: 17.0.4
Candi: f436827a303b85a3a758eb1f7df269dc0ff8dd54

I attach the candi configure log of trilinos: candi_configure.log (5.0 KB)

To me it seems like MKL disables Tpetra, but Tpetra is needed for Amesos2, which is apparently required (Amesos2 appears under Explicitly enabled packages on input (by user):). Not sure if the dependencies changed between trilinos-12-10 and 12-18, or is something else is going on behind the scenes.

bbuffett · June 18, 2020, 7:54pm

Hi Rene,

I followed your advice. I change the version of trilinos to version 12-10-1 and re-ran Candi.sh. It downloaded and starting building trilinos 12-10-1, as expected. I received the message about disabling Tpetra because “you are using MKL” . The BLAS_DIR and LAPACK_DIR are assigned to the MLK directory. All of this looks good.

The error message appeared again: ERROR: Setting Trilinos_ENABLE_Amesos2=OFF which was ‘ON’ because Amesos2 has a required library dependency on disabled package Tpetra! (The exclamation is included in the error message!) Is there any chance that the version of Cmake matters? I used version 3.7 to build everything up to p4est. The trilinos build failed because it needed 3.10 or higher. I shifted to version 3.15 to continue the build, which started with the trilinos package. I wondered if it would be reasonable to rebuild from scratch using Cmake 3.15. (Hard to see how this could matter. On the other hand, if I need to burn incense and chant during the build, so be it.)

Thanks
Bruce

tjhei · June 18, 2020, 7:58pm

Thanks, Rene. I can reproduce it. I will work on a fix in the next few days.

Bruce: Can you try deleting the ./tmp/ directory to make sure trilinos is configured from scratch? Or wait a few days for me to figure out what is going on.

bbuffett · June 18, 2020, 8:56pm

Good suggestion. Thanks. I will give it a try.

Bruce

tjhei · June 23, 2020, 6:06pm

I managed to fix compilation with Intel 19.0.5. See https://github.com/dealii/candi/pull/150 (it will be ported to the 9.2 branch after it got merged).

bbuffett · June 23, 2020, 6:43pm

Hi Timo,

Great new, thanks! So I just need to update compilers to 19.0.5 and be sure to pull a specific branch of Candi?

Best wishes,
Bruce

tjhei · June 23, 2020, 9:17pm

I would recommend using the 2019 or 2020 versions of the compiler, because these are the ones I have been using over the last 12 months. You might run into some issues with older versions (but the configuration error here has nothing to do with that).
Also, the pull request is not merged yet, sorry. If you don’t want to wait, you could do

git clone https://github.com/tjhei/candi.git
git checkout trilinos_mkl

bbuffett · June 23, 2020, 9:28pm

Perfect, thanks!

Bruce

bbuffett · June 26, 2020, 3:40am

Hi Timo,

The version of Candi succeeded in building trilinos. Thanks! I encountered a problem with pets-lite-3.13.1. I’ve now run Candi.sh a couple of times because the first time there appeared to be a problem downloading the package. The fatal error in the build is the absence of “petscvariables” in …/tmp/build/petsc-lite-3.13.1/lib/conf/.

Maybe I could turn off the use the petsc?

Sorry to trouble you.

Bruce

jbnaliboff · June 26, 2020, 5:01am

Hi Bruce,

I usually only compile Trilinos, p4est, HDF5 (optional), and of course deal.II with candi. Assuming all of the other dependencies are available on the cluster (zlib, cmake, etc), this should be all that is needed to build ASPECT.

Cheers,
John

bbuffett · June 28, 2020, 6:00pm

Hi John,

That did the job. Thanks! I was able to finish the deal-ii install and compile aspect.
One question, is there a way to allow a heat sink (i.e. a negative heat source)?
The entry “Radiogenic heating rate” requires a positive value. I would like to override
this restriction or possibly find an equivalent entry.

Thanks

tjhei · June 28, 2020, 6:15pm

Interesting. The easiest would be to remove that restriction by deleting “0.” in the highlighted line then recompile:

github.com

geodynamics/aspect/blob/a44ca1df96d66bb677cf51605e7d68cb6cb46bb2/source/heating_model/constant_heating.cc#L56


      
          
          template <int dim>
          void
          ConstantHeating<dim>::declare_parameters (ParameterHandler &prm)
          {
            prm.enter_subsection("Heating model");
            {
              prm.enter_subsection("Constant heating");
              {
                prm.declare_entry ("Radiogenic heating rate", "0.",
                                   Patterns::Double (0.),
                                   "The specific rate of heating due to radioactive decay (or other bulk sources "
                                   "you may want to describe). This parameter corresponds to the variable "
                                   "$H$ in the temperature equation stated in the manual, and the heating "
                                   "term is $\rho H$. Units: W/kg.");
              }
              prm.leave_subsection();
            }
            prm.leave_subsection();
          }

bangerth · June 29, 2020, 12:53am

Interesting indeed – you piqued my curiosity: What physical effect would that
correspond to?

Cheers
W.

bbuffett · June 29, 2020, 6:37am

Thanks. This suggestion worked perfectly. We are trying to model the thermal boundary layer at the core-mantle boundary using strongly temperature dependent viscosity. Modeling the entire convecting layer with a cold temperature at the top and a hot temperature at the bottom produces stagnant lid convection. Our goal is to model the lower half of the mantle (avoiding the stagnant lid at the top) by representing the influence of subduction as a heat sink. A constant temperature is imposed on the lower boundary and insulating conditions are imposed on the sides and top. A heat sink draws heat across the lower boundary. We know that the time averaged heat flux at the lower boundary is equal to the volume-integrated heat sink, so the question is what temperature drop is needed across the lower boundary to achieve the imposed heat flow. This experiment allows us to test the so-called boundary layer model (Nu ~ Ra^{1/3}) in the presence of strongly temperature dependent viscosity. We can also assess whether a large drop in viscosity across the lower boundary layer allows a secondary cell of convection to develop. I’m really impressed with the powerful tools you are putting into the hands of the geodynamics community.

With best wishes,
Bruce

Topic		Replies	Views
Candi installation ASPECT	4	227	September 6, 2021
Failing to compile trilinos with candi on Ubuntu 22.04 ASPECT installation , trilinos	13	287	October 22, 2022
Errors during dealii installation ASPECT installation	7	341	October 28, 2021
Error compiling deal.II ASPECT installation	10	336	April 19, 2023
Issues with Candi on CentosOS 7 ASPECT	11	233	February 15, 2023

Trilinos installation with candi and MKL fails

Related Topics