I have been trying to run an Aspect problem on Stampede2. However, I am currently running into a crash when building the Stokes preconditioner.
I am running candi deal.II 9.1.1(typo earlier) based on the wiki guide for setting up Aspect on stampede, with the Aspect version used being the branch in PR 3266.
I am currently running in Release mode, and the only error text produced is(removing some personal path segments):
ML::FATAL ERROR:: 1, /${INSTALLPATH}/candi-install/tmp/unpack/Trilinos-trilinos-release-12-10-1/packages/ml/src/Utils/ml_MultiLevelPreconditioner_NullSpace.cpp, line 98
Checking the file in question, this appears to be produced when an error should also be generated indicating that the null space vectors are NULL.
Does anyone have any suggestions about either what the problem might be, or what I could do to isolate the source of the problem?
Test problem use_full_A_block_preconditioner completed successfully for 1, and 2 processes.
Running the same problem considered with 1 and 2 processes while specifying no mesh refinement did not produce the error. Currently attempting 4, then will try 256. Note that the original error occured before any refinement attempts could be made during the preconditioner building step.
Problem occurs when running for 256 processes. An unsubstantiated guess at the cause is that the problem may be due to the number of cells on each processor.
Problem occurs with the use_full_A_block_preconditioner problem altered for one additional level of refinement for 256 cores, but does not occur for 128.
To clarify, the problem in question was run on another machine (though not on the precise same checkout) beforehand without producing an error, and another problem was successfully run with build used, so I belived I had the correct configuration.
As I noted in my edit, there was no significant change in the error output when Aspect compiled in debug mode.
@egpuckett look for the “!” icon to the right of this message. Click on it and select “Watching”. You can do the same at the category level if you wish to follow the entire ASPECT forum.
Yes, I have seen this exact error message on Frontera (very similar TACC system to Stampede2). I haven’t been able to reproduce it, though. Can you post what compiler and MPI library you are using? Is this with DEAL_II_WITH_64BIT_INDICES?
Test problem use_full_A_block_preconditioner completed successfully for 1, and 2 processes.
Running the same problem considered with 1 and 2 processes while specifying no mesh refinement did not produce the error. Currently attempting 4, then will try 256. Note that the original error occured before any refinement attempts could be made during the preconditioner building step.
Problem occurs when running for 256 processes. An unsubstantiated guess at the cause is that the problem may be due to the number of cells on each processor.
Problem occurs with the use_full_A_block_preconditioner problem altered for one additional level of refinement for 256 cores, but does not occur for 128.
)