Hello All,
I am having some trouble with an external source time function and I was hoping someone could help me trouble-shoot.
I am running a finite fault simulation where I simulate a M7 earthquake on a planar fault using 700 or so external source time function files. I am testing variability in response by changing the distribution of the slip and the hypocenter location. If I prescribe a slip short average rise time for the events (~2s), then everything runs ok. However, if I increase the rise time (~3s), then SPECFEM crashes after completing 50-80% of the simulation for certain slip/hypocenter combinations. The simulation may not always crash at the same time for a given set of starting conditions, but it always crashes. When run with —enable-debug, SPECFEM does not generate any error messages, and the timesteps do not indicate that the simulation is destabilizing. If I run with the MPI handler in debug mode, it spits out a bus error and the following assertion failure:
Assertion failure at /nfs/site/home/phcvs2/gitrepo/ifs-all/Ofed_Delta/rpmbuild/BUILD/libpsm2-11.2.78/ptl_am/ptl.c:153: nbytes == req->req_data.recv_msglen
Have you ever encountered this sort of error before, or do you have any ideas for how I could trouble-shoot it? I figure it has something to do with how specfem is handling the longer subsource rise time, since that’s really the only thing I am changing. For reference, I am running the simulation on 2048 cores (non-gpu), and have compiled the code with gfortran and mvapich2 (I get the same issue compiling with impi).
Thank you,
Ian