Since there was another query posted on Specfem on a Cray Machine, I thought I would add another. Recently, I installed SPECFEM3D_GLOBE on a Cray machine. I believe I have it successfully compiled and installed as I have been able to generate .sac and .ascii files when I am using 1 node with multiple processors/cores with it. However, there seems to be an issue when requesting runs with multiple nodes. When attempting to run a job with multiple nodes, I usually receive an error message when xmeshfem begins along the lines of:
Rank 337 [Thu Aug 5 11:16:43 2021] [c8-1c1s2n3] application called MPI_Abort(MPI_COMM_WORLD, 30) - process 337
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
xmeshfem3D 000000000059DEEB for__signal_handl Unknown Unknown
libpthread-2.26.s 00002AAAABD6E2D0 Unknown Unknown Unknown
libc-2.26.so 00002AAAAC2EB520 gsignal Unknown Unknown
libc-2.26.so 00002AAAAC2ECB01 abort Unknown Unknown
libmpich_intel.so 00002AAAAB319218 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAB226566 MPI_Abort Unknown Unknown
libmpich_intel.so 00002AAAAB26DA05 mpi_abort Unknown Unknown
xmeshfem3D 000000000042F8A7 Unknown Unknown Unknown
xmeshfem3D 00000000004188DA Unknown Unknown Unknown
xmeshfem3D 0000000000455183 Unknown Unknown Unknown
xmeshfem3D 000000000045EF77 Unknown Unknown Unknown
xmeshfem3D 0000000000470B05 Unknown Unknown Unknown
xmeshfem3D 00000000004ABC79 Unknown Unknown Unknown
xmeshfem3D 000000000048C92F Unknown Unknown Unknown
xmeshfem3D 0000000000483C3F Unknown Unknown Unknown
xmeshfem3D 000000000044A21D Unknown Unknown Unknown
xmeshfem3D 000000000040BB52 Unknown Unknown Unknown
libc-2.26.so 00002AAAAC2D634A __libc_start_main Unknown Unknown
xmeshfem3D 000000000040BA6A Unknown Unknown Unknown
If it is not this, I have also noticed an error message like this: " PBS: job killed: walltime 947 exceeded limit 900" is quite common. However, when I increase the available walltime it still runs into issues and give an output such as “PBS: job killed: walltime 2014 exceeded by limit 2000”. This walltime message does not occur at the end of my script but rather when I try to start xmeshfem. I am curious if these two messages could be related or are different issues? I have tried multiple combinations of processors and dimensions (NEX_XI & NPROC_XI, etc.) following the chart on pg. 23 in the 3D_GLOBE manual and runtimes, yet this does not appear to resolve the issue. By any chance, is anyone familiar with what might be causing this or are familiar with the error messages? I am not sure if this is specifically related to it being on a Cray machine or just some of my inexperience with working on programs like this. Any advice or suggestions would be appreciated.