Installing v2.2.2 from source: Syntax error in building h5py dependency

PyLith
1

Hello everyone

I am install pylith 2.2.2 on a standalone computer. Everything seems ok expect when building h5py a syntaxError occur in the pkgconfig.py file on line 281 seemingly associated with *_ unpacking. How do I go around this?

If i disable h5py, everything runs to completion but pylith doesn’t install because it finds error when checking for h5py dependency files

Below is the error

cd h5py-2.9.0 &&
python setup.py configure --hdf5=/home/dmanu-ma/pylith &&
CC=mpicc CPPFLAGS="-I/home/dmanu-ma/pylith/include -I/home/dmanu-ma/pylith/include " LDFLAGS="-L/home/dmanu-ma/pylith/lib -L/home/dmanu-ma/pylith/lib64 -L/home/dmanu-ma/pylith/lib -L/home/dmanu-ma/pylith/lib64 " CFLAGS="-g -O2" CXXFLAGS="-g -O2 -DMPICH_IGNORE_CXX_SEEK" FCFLAGS="" python setup.py build &&
python setup.py install --prefix=/home/dmanu-ma/pylith &&
touch /home/dmanu-ma/build/installed_h5py
File “build/bdist.linux-x86_64/egg/pkgconfig/pkgconfig.py”, line 281
flags = _query(packages, *os_opts, *_build_options(option, static=static))
^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents…

Installed /home/dmanu-ma/build/h5py-2.9.0/.eggs/pkgconfig-1.5.5-py2.7.egg
Searching for Cython>=0.23
Reading Links for Cython
Downloading https://files.pythonhosted.org/packages/48/d7/74fefb60a8caeb46681f197073e08243456acbc7244148644c19e5ff3163/Cython-3.0.0a9-cp27-cp27m-manylinux1_x86_64.whl#sha256=8c4c08af0e3f6223aad6a04e82a73781718e820b1b1f75e4ddf49ade8b006851
Best match: Cython 3.0.0a9
Processing Cython-3.0.0a9-cp27-cp27m-manylinux1_x86_64.whl
Installing Cython-3.0.0a9-cp27-cp27m-manylinux1_x86_64.whl to /home/dmanu-ma/build/h5py-2.9.0/.eggs

Installed /home/dmanu-ma/build/h5py-2.9.0/.eggs/Cython-3.0.0a9-py2.7-linux-x86_64.egg
running configure
Autodetection skipped [invalid syntax (pkgconfig.py, line 281)]

                   Summary of the h5py configuration

Path to HDF5: '/home/dmanu-ma/pylith'
HDF5 Version: '1.8.4'
 MPI Enabled: False

Rebuild Required: True

running build
…
copying h5py/tests/hl/test_vds/test_highlevel_vds.py → build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
running build_ext
Autodetection skipped [invalid syntax (pkgconfig.py, line 281)]

                   Summary of the h5py configuration

Path to HDF5: '/home/dmanu-ma/pylith'
HDF5 Version: '1.8.4'
 MPI Enabled: False

Rebuild Required: True

Executing api_gen rebuild of defs
Executing cythonize()
Traceback (most recent call last):
File “setup.py”, line 168, in
cmdclass = CMDCLASS,
File “build/bdist.linux-x86_64/egg/setuptools/init.py”, line 129, in setup
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/core.py”, line 151, in setup
dist.run_commands()
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/dist.py”, line 953, in run_commands
self.run_command(cmd)
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/dist.py”, line 972, in run_command
cmd_obj.run()
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/command/build.py”, line 127, in run
self.run_command(cmd_name)
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/cmd.py”, line 326, in run_command
self.distribution.run_command(command)
File “/home/dmanu-ma/pylith/lib/python2.7/distutils/dist.py”, line 972, in run_command
cmd_obj.run()
File “/home/dmanu-ma/build/h5py-2.9.0/setup_build.py”, line 202, in run
self.extensions = cythonize(self._make_extensions(config),
File “/home/dmanu-ma/build/h5py-2.9.0/setup_build.py”, line 80, in _make_extensions
import pkgconfig
File “build/bdist.linux-x86_64/egg/pkgconfig/init.py”, line 1, in
File “/home/dmanu-ma/build/h5py-2.9.0/.eggs/pkgconfig-1.5.5-py2.7.egg/pkgconfig/pkgconfig.py”, line 281
flags = _query(packages, *os_opts, *_build_options(option, static=static))
^
SyntaxError: invalid syntax
make[2]: *** [Makefile:1227: h5py] Error 1
make[2]: Leaving directory ‘/home/dmanu-ma/build’
make[1]: *** [Makefile:1238: installed_h5py] Error 2
make[1]: Leaving directory ‘/home/dmanu-ma/build’
make: *** [Makefile:1436: installed_pylith] Error 2

2

My guess is that the cython version being installed is incompatible with this version of h5py. Try using Cython version 0.29.24 available from Index of /~buildbot/deps.

In this is not a cluster, I recommend using the PyLith binary.

3

Dear Baagaard,

Thanks for the comment, I used an order version of the pkgconfig file and manage to install pylith.

Yes this is not a cluster but a 6 core, 12 threads computer, I was hoping that installing from source would allow me to run pylith using the multiprocessors. Does that not work?

4

You can use multiple cores and processors with the binary. Just use the --nodes=NUM_PROCESSES command line argument.

The primary reason for building from source on a cluster is to make use of the MPI installed on the cluster that takes advantage of the interconnect hardware like infiniband.

5

This site does not let me attach config.log (it says I am a new user and I can only make 3 reply).
I have continued with my compilation, I have advanced but I still have errors. After compiling netcdf it passes numpy, then to cython and here I have the error with h5py:

writing build/bdist.macosx-10.13-intel/egg/EGG-INFO/native_libs.txt
creating dist
creating ‘dist/Cython-0.29.16-py2.7-macosx-10.13-intel.egg’ and adding ‘build/bdist.macosx-10.13-intel/egg’ to it
removing ‘build/bdist.macosx-10.13-intel/egg’ (and everything under it)
Processing Cython-0.29.16-py2.7-macosx-10.13-intel.egg
creating /Users/cyclo/pylith_222_1/lib/python2.7/site-packages/Cython-0.29.16-py2.7-macosx-10.13-intel.egg
Extracting Cython-0.29.16-py2.7-macosx-10.13-intel.egg to /Users/cyclo/pylith_222_1/lib/python2.7/site-packages
Adding Cython 0.29.16 to easy-install.pth file
Installing cython script to /Users/cyclo/pylith_222_1/bin
Installing cygdb script to /Users/cyclo/pylith_222_1/bin
Installing cythonize script to /Users/cyclo/pylith_222_1/bin

Installed /Users/cyclo/pylith_222_1/lib/python2.7/site-packages/Cython-0.29.16-py2.7-macosx-10.13-intel.egg
Processing dependencies for Cython==0.29.16
Finished processing dependencies for Cython==0.29.16
/Users/cyclo/Pylith222_1/src/pylith/pylith-installer-2.2.2-1/bin/fetch.sh “/opt/local/bin/curl -L -O -o” h5py-2.9.0.tar.gz Index of /~buildbot/deps
Found local copy of h5py-2.9.0.tar.gz.
/usr/bin/tar -zxf h5py-2.9.0.tar.gz
cd h5py-2.9.0 &&
python setup.py configure --hdf5=/Users/cyclo/pylith_222_1 &&
CC=mpicc CPPFLAGS="-I/Users/cyclo/pylith_222_1/include -I/Users/cyclo/pylith_222_1/include " LDFLAGS="-L/Users/cyclo/pylith_222_1/lib -L/Users/cyclo/pylith_222_1/lib64 -L/Users/cyclo/pylith_222_1/lib -L/Users/cyclo/pylith_222_1/lib64 " CFLAGS="-g -O2" CXXFLAGS="-g -O2 -DMPICH_IGNORE_CXX_SEEK" FCFLAGS="-g -O2" python setup.py build &&
python setup.py install --prefix=/Users/cyclo/pylith_222_1 &&
touch /Users/cyclo/Pylith222_1/installed_h5py
Couldn’t find index page for ‘pkgconfig’ (maybe misspelled?)
No local packages or download links found for pkgconfig
Traceback (most recent call last):
File “setup.py”, line 168, in
cmdclass = CMDCLASS,
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py”, line 111, in setup
_setup_distribution = dist = klass(attrs)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py”, line 268, in init
self.fetch_build_eggs(attrs[‘setup_requires’])
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py”, line 313, in fetch_build_eggs
replace_conflicting=True,
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py”, line 843, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py”, line 1088, in best_match
return self.obtain(req, installer)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py”, line 1100, in obtain
return installer(requirement)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py”, line 380, in fetch_build_egg
return cmd.easy_install(req)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/command/easy_install.py”, line 632, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse(‘pkgconfig’)
make[2]: *** [h5py] Error 1
make[1]: *** [installed_h5py] Error 2
make: *** [installed_pylith] Error 2

What am I missing?

6

Please try version 2.2.2-2 of the PyLith installer. I have it working on Linux (tested on CentOS 7, Ubuntu 20.04) and macOS (BigSur). You can download the tarball (temporary location) at
Release Installer version 0 for PyLith v2.2.2 · geodynamics/pylith_installer · GitHub.

Documentation (raw Markdown, temporary): pylith_installer/configs.md at releases/v2.2 · geodynamics/pylith_installer · GitHub

Note: I will make a new release once I have the Docker development container built. The new release will include online documentation with detailed, up-to-date instructions. I will post all of the information for the v2.2.2-2 installer release once it is available (should be mid-December).

7

See docs/configs for examples of which Linux packages to install (Ubuntu 20.04 packages).

Important: For development do NOT use the configure arguments listed in the docs/configs files. Instead, use the ones listed in devenv/configs.md.