A colleague (the HPC pro at Brown) and I (just a user) are having some strange issues with the recent install of ASPECT. He updated ASPECT to 3.0 and downloaded Deal.ii. We originally ran into issues that seemed to be linked with deal.ii being configured with PETSc and SLEPc. Having spoken to another colleague who uses ASPECT with successful runs (for our shared set of parameter files), she mentioned her deal.ii config file had “set(DEAL_II_WITH_PETSC OFF); set(DEAL_II_WITH_SLEPC OFF)”. So deal.ii was reconfigured with those settings to OFF but a different error message came (on several jobs with different parameter files). The run continues and this error message doesn’t result in the job being killed - it just lingers and gets stuck.
I should note that I am trying to use ASPECT with a plug-in for an asthenosphere. The plug-in was built without any errors.
I’ve attached the plug-in and the error message that arises. It appears that if I run ASPECT without the plug-in I don’t have these issues. But from the log file, it doesn’t appear to reach the stage of importing the plug-in.
However, my colleague, who provided the plug-in, says that ASPECT is running fine with it on her cluster.
I don’t know how to fix the problem you’re seeing and I personally haven’t seen this before, but I think the fact that ASPECT runs normally on your cluster without using this plug-in is interesting.
The plug-in that you’re using requires linking to a built .so file, have you tried running a cookbook/benchmark that comes bundled with ASPECT that also requires linking to a .so file? If not, it would be interesting to see if you get this same error. If you do get the error then I would guess there’s something going on with trying to access these .so files on your cluster, and if you don’t get the error then that would help narrow it down to something just within the plug-in you are using.
That is a great suggestion. So I ran one of the cookbooks and the plug-in link was fine. So at least this narrows it down to the plug-in. (Which my colleague did say she used ChatGPT to help write! Even so, it works on her machine fine.. so still very interesting)
Is there anything about the plug-in files you spot that is glaringly problematic? (I’m very unfamiliar with this - but I know you’re busy, so no worries if you don’t have a chance to check!).
Thanks again for such a swift response! Appreciate it.
Very strange! Out of curiosity, which cookbook did you try running? Also, are you running this in debug mode? Could you upload the log.txt file that ASPECT creates when you hit this issue? Seeing exactly what part of the model ASPECT crashes in might be helpful.
I don’t see anything in the plug-in that looks off, and I actually was able to compile and run a test using your plugin on my laptop. I took the existing contintenal_extension cookbook and changed the material model to include your plugin, and the output looked reasonable and it ran for more than 50 timesteps without error… I attached the test that I ran just in case it’s useful, but unfortunately this could be an issue specific to the cluster you are using Maybe the IT people in charge of your cluster can also help with debugging this?
Thanks for the response! I used “free_surface_with_crust”. Interesting that it causes no problem for you. Must be something on my end. I’m afraid that the log file doesn’t appear to be very useful, but here it is (I ran this in debug mode).
Let me know if this is helpful. I can’t interpret it at all! The job doesn’t get canceled. It just lingers at this point.
your issue definitely seems to be machine-related. It would be difficult to diagnose here why using the plugin triggers the error but probably it was compiled with different modules/options/etc. versus your ASPECT executables. To me it seems that your job tries oversubscribe threads, which do not exist. Could you try setting export DEAL_II_NUM_THREADS=1 in your Slurm batch file and report back here if it does help with the plugin?
Thanks for the suggestion. And also keeping with this strange problem!
Different errors popped up - I’ve attached the ASPECT output, but also my slurm batch script to see if I set the deal.ii options incorrectly. (I’m not very familiar with slurm or HPC.)
that’s some progress! Now your job at least proceeds to loading the plugin. This time the error message seems to be actually related to the plugin itself. Debug mode uses a lot of numerical, but also compiler-level assertions to ensure that given parameters and some computed values are reasonable, e.g. within the set of real numbers.
If I interpret the error output right, the root cause is your computed viscosity profile at line 108. With certain parameters, std::pow(viscosity_ratio, -weakening_exponent) produces non-real numbers. For example, with viscosity_ratio=0 and weakening_exponent>=0 you would get undefined or -inf result and that is being caught by debug mode assertions. I believe you would be able to execute the plugin now in release mode without issues.
In case you wish to get rid of this issue in debug mode as well, some defensive checks/assertions should be implemented. This ensures that no invalid input parameters would reach the power function at line 108 in the first place.
Maybe the IT people in charge of your cluster can also help with debugging this?