MPI / Pylith does not work on Macs without internet connection

PyLith
1

Hi all,

During a recent power outage here, we uncovered a surprising fact that Pylith can only run on a mac when connected to the internet. When reconnecting to the internet, it works fine!

Steps to reproduce:

(1) turn off wifi
(2) try to run any pylith example, receive the following error:

box-2d % pylith step01_axialdisp.cfg 
Abort(941216911): Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(66)........: MPI_Init(argc=0x30e02abe8, argv=0x30e02abe0) failed
MPII_Init_thread(234)....: 
MPID_Init(513)...........: 
MPIDI_OFI_init_local(580): OFI fi_fabric failed (ofi_init.c:580:MPIDI_OFI_init_local:Invalid argument)
/Users/elindsey/pylith/pylith-4.1.1-macOS-10.15-x86_64/bin/nemesis: mpiexec: exit 143
/Users/elindsey/pylith/pylith-4.1.1-macOS-10.15-x86_64/bin/pylith: /Users/elindsey/pylith/pylith-4.1.1-macOS-10.15-x86_64/bin/nemesis: exit 1

(3) turn on wifi, run example again and it works fine!

We have reproduced this on two different M3 Pro macs, and I found a thread discussing the same error here but without resolution: Strange gfortran / MPI problem on MacBook - Help - Fortran Discourse

Any ideas or suggestions?

2

I believe a workaround is to manually set the hostname, which makes MPI happy.
We use MPICH in the PyLith binary. There might be some options for the MPICH settings (devices) that will allow us to eliminate this issue. I will create a GitHub issue.

3

Thanks! To clarify, how did you set the hostname? I tried a few options but none seemed to affect this error.

It’s not a critical issue because normally internet is available, but one could imagine some edge cases (trying to run Pylith in the field?)

4

The workaround is to tell MPI to use sockets for all communication by setting the environment variable IF_PROVIDER to sockets. I will add this to the PyLith documentation. We will consider setting this in the setup.sh file.

export FI_PROVIDER=sockets