Issues with Block GMG solver scheme

ASPECT
1

Hi all,
So I have a working 3D model with a free surface on top and a visco plastic material model using either the ‘single advection, iterated stokes’ or ‘iterated advection and stokes’ (both case use block AMG). Now I want to run parameter test faster so I tried the Block GMG.
My solver scheme looks like this:

set Nonlinear solver scheme                = single Advection, iterated defect correction Stokes
set Max nonlinear iterations               = 20
set Nonlinear solver tolerance             = 1e-5

subsection Solver parameters
  subsection Stokes solver parameters
    set Stokes solver type                      = block GMG
    set Number of cheap Stokes solver steps     = 2000
    set Linear solver tolerance                 = 1e-4
    set GMRES solver restart length             = 150
    set Use full A block as preconditioner      = true
  end
  subsection Newton solver parameters
    set Maximum linear Stokes solver tolerance   = 1e-4
    set Use Eisenstat Walker method for Picard iterations = true
  end
  subsection Advection solver parameters
    set GMRES solver restart length = 100
  end
end

However, this model crashed at the beginning and the error message seems to be saying that it is missing the LAPACK library.

Exception 'ExcNeedsLAPACK()' on rank 182 on processing:

--------------------------------------------------------
An error occurred in line <256> of file </work/n03/n03/mazq/modules/candi/tmp/unpack/deal.II-v9.4.2/source/lac/tridiagonal_matrix.cc> in function
    void dealii::TridiagonalMatrix<number>::compute_eigenvalues() [with number = double]
The violated condition was:
    false
Additional information:
    You are attempting to use functionality that is only available if
    deal.II was configured to use LAPACK, but cmake did not find a valid
    LAPACK library.

Stacktrace:
-----------
#0  /work/n03/n03/mazq/modules/candi/deal.II-v9.4.2/lib/libdeal_II.so.9.4.2: dealii::TridiagonalMatrix<double>::compute_eigenvalues()
#1  /work/n03/n03/mazq/modules/aspect/release/aspect: dealii::SolverCG<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::compute_eigs_and_cond(std::vector<double, std::allocator<double> > const&, std::vector<double, std::allocator<double> > const&, boost::signals2::signal<void (std::vector<double, std::allocator<double> > const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (std::vector<double, std::allocator<double> > const&)>, boost::function<void (boost::signals2::connection const&, std::vector<double, std::allocator<double> > const&)>, boost::signals2::mutex> const&, boost::signals2::signal<void (double), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (double)>, boost::function<void (boost::signals2::connection const&, double)>, boost::signals2::mutex> const&)
#2  /work/n03/n03/mazq/modules/aspect/release/aspect: void dealii::SolverCG<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::solve<aspect::MatrixFreeStokesOperators::ABlockOperator<3, 2, double>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > >(aspect::MatrixFreeStokesOperators::ABlockOperator<3, 2, double> const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>
--------------------------------------------------------
An error occurred in line <256> of file </work/n03/n03/mazq/modules/candi/tmp/unpack/deal.II-v9.4.2/source/lac/tridiagonal_matrix.cc> in function
    void dealii::TridiagonalMatrix<number>::compute_eigenvalues() [with number = double]
The violated condition was:
    false
Additional information:
    You are attempting to use functionality that is only available if
    deal.II was configured to use LAPACK, but cmake did not find a valid
    LAPACK library.

I would greatly appreciate it if you could help me with this. Do I need to install the LAPACK library and then rebuild deal.II on archer2 where I run my models?

Cheers,
ziqi

2

@maziqi96 Yes, that is exactly what you need to do. While you’re there, you might as well see whether you can upgrade to deal.II 9.6.
Best
W.

3

@maziqi96 - Note that you can install lapack through candi through the openblas package. That is the approach I most commonly use when lapack is not available or a different version is needed.

However, since archer2 is a cray system and my recollection that a different approach was required for installing blas/lapack.

Can you search the forums and ASPECT wiki for any previous posts on this? If the search does not turn up anything, I can look through old emails.

@jvanhunen - Does your group by chance have current instructions for installing deal.II and ASPECT on ARCHER2?

Cheers,
John

4

Hi,

I’m using ASPECT on ARCHER2 and ran into the same issue, but never fixed it as LAPACK wasn’t necessary for my project.

I did contact ARCHER2’s help desk (specifically William Lucas who’s helped create ASPECT installation guides in the past) for help and after a bit of back and forth we couldn’t get it working. Unfortunately, since it’s a cray system you can’t just specify -DDEAL_II_WITH_LAPACK=ON in candi and have it work, I also believe it’s set to OFF by default on cray systems.

I do have this email I got from William though which may be of use here since it summarises and explains the problems with trying to install LAPACK as well as a couple of ways they tried to fix the issue.

image
image1882×790 68.5 KB

If you make any progress on getting this working please update this form post as it will be really useful.

Thanks,

Luke

5

@Lukel Thanks for the background information, and for getting this far with the Archer folks. My personal take is that Cray should just stop the silliness of using a different name because that requires every single project on the face of the earth to make changes to their cmake scripts when the whole issue could be fixed by a single project (the Cray software stack) by just changing that one file name. But this is a long-standing issue with Cray we’re not going to solve here.

Pragmatically: For obvious reasons, Cray (and the system administrators on Cray systems) want you to use the Cray-optimized vendor version of LAPACK. But this is not required. You could just as well install a local version of the reference implementation of LAPACK. You’ll get slower performance out of it, but the truth is that whether you spend 0.001% or 0.002% of the ASPECT run time in these functions, it isn’t going to make much difference – in reality, ASPECT does not actually use LAPACK for anything that is at all compute-intensive.
So my suggestion is to just install the reference LAPACK, with its standardized name, and compile deal.II against that. It is possible that you can even install LAPACK with candi (I’d have to look that up) and not have to worry about the vendor version of LAPACK at all.
Best
W.

6

Hi all,
Thanks, @Lukel for the info. @jbnaliboff : I have not used Archer for a while now, so have nothing to add here.
Jeroen