I don’t think this is necessarily cluster dependent, I’ve now run it on two different clusters. Also, although the exact time-step that it dies is not reproducible (although I need to try to run it again with the same number of threads and test that) it always dies in exactly same part of the code… rebuilding the preconditioner.
I am suspicious that there is a bug in the material module that I’m using… visco plastic. Am I correct in thinking that when rebuilding the preconditioner, it needs the the viscosity (the stiffness matrix)? I’m guessing that there are not so many people using this specific module, so there could be some bug that I’m just now finding because I trying to really use the module with all its bells and whistles (like the composition-dependent rheology).
Also, I was always taught that a segmentation fault basically means that you are trying to grab information from somewhere in memory that you shouldn’t… so somewhere you are counting wrong. Is that still true? Is it a good way to think about it or not (in terms of trying to track down the problem).
I am trying to think of ways to change the problems so that is still basically runs the same, but that it might help us to figure out what is triggering the problem, but that is more difficult.