Error when running pylith on the cluster with extended friction model

Dear Pylith developers,

I recently extended pylith to implement my own friction law. I use Pylith installed from source on the cluster.

Now when I try running pylith, I get this error:

pylith ~/Pylith_temp/46_new_dynamic_slip_weak_rate_state_overstressed_02_2020/4_dynamic_3by2m_3_sec_double_slip_weak_overstressed_gentle_to_steep.cfg

/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/utils/PetscManager.py:64:initialize
– petsc(info)
– Initialized PETSc.


A process has executed an operation involving a call to the
“fork()” system call to create a child process. Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption. The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

Local host: [[54552,1],0] (PID 49034)

If you are absolutely sure that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.

/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/apps/PyLithApp.py:103:main
– pylithapp(info)
– Running on 1 process(es).

/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/apps/PyLithApp.py:106:main
– pylithapp(debug)
– [0] CPU time: 00:00:00, Memory usage: 89.75 MB

/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/meshio/MeshIOObj.py:55:read
– meshiocubit(info)
– Reading finite-element mesh
Fatal error. Calling MPI_Abort() to abort PyLith application.
Traceback (most recent call last):
File “/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/apps/PetscApplication.py”, line 74, in onComputeNodes
self.main(*args, **kwds)
File “/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/apps/PyLithApp.py”, line 115, in main
mesh = self.mesher.create(self.problem.normalizer, interfaces)
File “/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/topology/MeshImporter.py”, line 100, in create
mesh = self.reader.read(self.debug, self.interpolate)
File “/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/meshio/MeshIOObj.py”, line 71, in read
ModuleMeshIO.read(self, mesh)
File “/home/bolee/PYLITH_FROM_SOURCE/pylith/lib/python2.7/site-packages/pylith/meshio/meshio.py”, line 86, in read
def read(self, *args): return _meshio.MeshIO_read(self, *args)
RuntimeError: Error while reading Cubit Exodus file ‘mesh/geometry_and_mesh_2by3m.exo’.
Could not open ExodusII file ‘mesh/geometry_and_mesh_2by3m.exo’ for reading.


MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

/home/bolee/PYLITH_FROM_SOURCE/pylith/bin/nemesis: mpirun: exit 255
/home/bolee/PYLITH_FROM_SOURCE/pylith/bin/pylith: /home/bolee/PYLITH_FROM_SOURCE/pylith/bin/nemesis: exit 1

I know my mesh files are fine, because they ran previously with another pylith installation. I know the problem is not in my friction law, because the error is the same with build-in friction laws.
I tried reinstalling pylith and rebuilding friction, that didn’t work.

Do you know what the problem might be?
Any help will be appreciated!

Thanks!

The only time I have seen this error (not being able to open an Exodus file) is when the file was indeed not in the place specified or had the wrong permissions.

Dear Brad,

I checked multiple times before writing here.
It is in place, it is correct and other scripts in the folder (not involving new friction, but using the same mesh) ran with another pylith installation.

Thanks!

Can you run the examples using the PyLith you built?