ASPECT Hanging/Stuck on a Time Step

ASPECT
1

Hi all!

I am requesting some help as I think I am stuck on an issue. I am running ASPECT 3.0.0 on a supercomputer cluster remotely. My setup is fairly simple: 2d annulus of the Earth, I have three main compositional fields (mantle, LLVPs, and continents). I currently have two equally (and reasonably) sized LLVPs and two equally and reasonably sized continents. The models are compressible, isoviscous, and use adiabatic and compositional heating models. The only thing I am varying between the few models that are stuck and ones that have run successfully are the values of compositional heating for the LLVPs, continents, and mantle. I have a grid of 22 total runs, with 6/22 runs getting stuck at very similar timesteps, and the other 16 models ran with no issue. It doesn’t appear to be an issue with the compositional heating values themselves as the problematic runs occur randomly throughout the grid: i.e. a model with slightly smaller compositional heating in one insulator will run fine, the next largest will hang, and then a model with larger compositional heating than the one that hung will run just fine. I have tried adjusting both my solver parameters and mesh refinement parameters to no avail. The computer I am using also unfortunately doesn’t have ASPECT’s debug mode compiled, only the release version. If anyone could help, that would be great. An example of where the models will get stuck for a week before they just time out is: Postprocessing:
RMS, max velocity: 0.35 m/year, 0.794 m/year
Temperature min/avg/max: 283 K, 1664 K, 2.793e+04 K
Heat fluxes through boundary parts: -3.489e+06 W, 6.705e+06 W
Number of advected particles: 2839082
Writing stokes residuals CV5011/stokes_residuals.txt
Mobility: 1.03

*** Timestep 1029: t=4.32171e+06 years, dt=7421.85 years

2

Hi Anna,

you can describe what “hangs” or “being stuck” actually means? Are you saying that you get one line of output, then nothing else, until the scheduler kills the run? Is there a way to figure out whether you are still using CPU time during that time (i.e., something is actually running) or whether the job is technically alive but not doing anything (i.e., a “deadlock”)?

In general, this is the sort of thing that debug mode is there for. It’s very difficult to debug problems in release mode. Can you work with your system administrator to also get the debug version installed?

Best

W.

3

I get that line of output I posted at a specific timestep, it varies between each of the stuck runs but it’s usually around timestep 1030, and then it just stays at that line for a week until the job eventually times out. I am still using CPU during that time and by all appearances the job is still alive and running. We are going to try to talk with the admin to see if we can’t get the debug mode compiled as I’m sure this won’t be the last time we have issues that need debugging, we are just anticipating a long wait time for that.

4

Yes, do that. Our experience has been that trying to debug problems in release mode is a waste of time because we catch so many problems in debug mode. More often that not, one gets decent error messages in debug mode that immediately explain what the problem is. I’m not saying that’s going to be the case for you, but it happens often enough that it’s not worth your time trying to figure out the exact cause unless the same happens in debug mode as well.

Best
WB