Problem Pylith2 Installation on a Cluster

Hello, I’m trying to install pylith-installer-2.2.2-0 on a cluster, however, for a couple of days an error with the h5py and NetcCDF libraries persist during run make. The cluster’s administrators gave me root permission on my personal docker.

Thanks.

IMPORTANT

Step 1
BEFORE running “make”, setup your environment so that all of the
dependencies are consistent. See the file setup.sh in this directory
for how to set your variables for the bash shell. You can either run
“source setup.sh” in this terminal (i.e., shell) where you will
build PyLith and it dependencies or add these settings to your
.bashrc or equivalent file and open a new terminal (i.e., shell).

Step 2
Running “make” will install
PyLith dependencies to /home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith.
PyLith to /home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith.

Step 3
If you did not setup your .bashrc with the environment variables, be sure
to run “source setup.sh” every time you open a new terminal (i.e., shell)
BEFORE running PyLith.

root@dgx-node-0-0:/# source setup.sh
root@dgx-node-0-0:/# make
make pylith
make[1]: Entering directory ‘/’
make h5py
make[2]: Entering directory ‘/’
/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/src/pylith/pylith-installer-2.2.2-0/bin/fetch.sh “/usr/bin/wget -nd -r -O” h5py-2.9.0.tar.gz Index of /~buildbot/deps
Found local copy of h5py-2.9.0.tar.gz.
/usr/bin/tar -zxf h5py-2.9.0.tar.gz
cd h5py-2.9.0 &&
python setup.py configure --hdf5=/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith &&
CC=mpicc CPPFLAGS=“-I/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/include -I/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/include " LDFLAGS=”-L/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/lib -L/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/lib64 -L/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/lib -L/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith/lib64 " CFLAGS=“-g -O2” CXXFLAGS=“-g -O2 -DMPICH_IGNORE_CXX_SEEK” FCFLAGS=“” python setup.py build &&
python setup.py install --prefix=/home/diego.cardenas/.local/share/enroot/model/home/diego.cardenas/pylith &&
touch //installed_h5py
Download error on https://pypi.org/simple/pkgconfig/: unknown url type: https – Some packages may not be found!
Couldn’t find index page for ‘pkgconfig’ (maybe misspelled?)
Download error on https://pypi.org/simple/: unknown url type: https – Some packages may not be found!
No local packages or working download links found for pkgconfig
Traceback (most recent call last):
File “setup.py”, line 168, in
cmdclass = CMDCLASS,
File “build/bdist.linux-x86_64/egg/setuptools/init.py”, line 128, in setup
File “build/bdist.linux-x86_64/egg/setuptools/init.py”, line 123, in _install_setup_requires
File “build/bdist.linux-x86_64/egg/setuptools/dist.py”, line 504, in fetch_build_eggs
File “build/bdist.linux-x86_64/egg/pkg_resources/init.py”, line 774, in resolve
File “build/bdist.linux-x86_64/egg/pkg_resources/init.py”, line 1057, in best_match
File “build/bdist.linux-x86_64/egg/pkg_resources/init.py”, line 1069, in obtain
File “build/bdist.linux-x86_64/egg/setuptools/dist.py”, line 571, in fetch_build_egg
File “build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py”, line 667, in easy_install
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse(‘pkgconfig’)
make[2]: *** [Makefile:1227: h5py] Error 1
make[2]: Leaving directory ‘/’
make[1]: *** [Makefile:1238: installed_h5py] Error 2
make[1]: Leaving directory ‘/’
make: *** [Makefile:1436: installed_pylith] Error 2

I believe the latest release of the PyLith installer for v2.2.2 fixes these issues.

You should never need root access to install PyLith. We never install PyLith using root access on any of the machines we use.

1 Like

Thanks Brad. And how can I run pylith (from the source) in parallel from a cluster with SLURM system?

For example, resources reserve: salloc -p PARTITION -n 1 -c #CPUs --mem=#RAMGB --gres=gpu:GPUTypeConsole:#GPUs --time=hh:mm:ss

Then I have to connect to the node ssh dgx-node-0-0

inside the dgx-node, I have root access where pylith is installed.

I tried:

root@user:/home/pylith2-2-2_folder/ mpirun -np 1 (only 1 node with 16 cpus on my resources’ reservation) pylith file.cfg

We don’t have a specific scheduler for SLURM. However, the LSF scheduler is generally compatible. You may have to fine tune the settings for your machine. See Overview of Running PyLith — PyLith 3.0.3 documentation for more information on how to run PyLith on a cluster.

1 Like