Hello I try to install preCICE at the HLRS in vulcan cluster with local version of spack and local pip packages. I was wondering if there is a trick i have to do for the installation, since pyprecice is succesfully installed but i get an import error. I installed preCICE and loaded the required modules in the following way:
spack install precice
spack load precice@3.1.0
module load python/3.10
module load openmpi/4.1.2-gcc-11.2.0
I modified the bashrc file with path:
export PKG_CONFIG_PATH="/zhome/academic/HLRS/isd/isdsg/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-11.2.0/precice-3.1.0-yusoyoeupbrugromwevcy2lrfr7e7oah/lib64/pkgconfig"
I then did the following things:
source ~/.bashrc
pip3 install --user pyprecice
Requirement already satisfied: pyprecice in /zhome/academic/HLRS/isd/isdsg/.local/lib/python3.10/site-packages (3.0.0.0)
Requirement already satisfied: numpy in /sw/vulcan-rocky8/hlrs/non-spack/2022-02/development/python/3.10/lib/python3.10/site-packages (from pyprecice) (1.21.6)
Requirement already satisfied: mpi4py in /zhome/academic/HLRS/isd/isdsg/.local/lib/python3.10/site-packages (from pyprecice) (3.1.5)
Requirement already satisfied: Cython in /sw/vulcan-rocky8/hlrs/non-spack/2022-02/development/python/3.10/lib/python3.10/site-packages (from pyprecice) (0.29.28)
When I try to test pyprecice, I see the following error:
python3 -c "import precice"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/zhome/academic/HLRS/isd/isdsg/.local/lib/python3.10/site-packages/precice/__init__.py", line 5, in <module>
from cyprecice import Participant, get_version_information
ImportError: libprecice.so.3: cannot open shared object file: No such file or directory
Does anyone have an idea of what is going wrong?
P.S. I was recommended by @IshaanDesai not to use the spack package of pyprecice, since its not maintained.
best regards
steffen
1 Like
This looks fishy. Can you run precice-tools version after loading the spack package?
Note that you always need to load the spack package before running a solver too.
Building the cpp solverdummies locally can help to figure out what was going on.
Yes i am aware that i have to load spack and also python 3.10 module itself:D.
bash precice-tools version gives me this:
spack load precice@3.1.0
precice-tools version
3.1.0;develop-2024-04-07-7-ge8ae9a403c-dirty;PRECICE_FEATURE_MPI_COMMUNICATION=Y;PRECICE_FEATURE_PETSC_MAPPING=Y;PRECICE_FEATURE_GINKGO_MAPPING=N;PRECICE_FEATURE_PYTHON_ACTIONS=N;PRECICE_BINDINGS_C=Y;PRECICE_BINDINGS_FORTRAN=Y;CXX=GNU;CXXFLAGS= -O3 -DNDEBUG;LDFLAGS=
By this he means clone the preCICE repository, go to examples/solderdummies and compile the dummy solvers following the instructions in the README. This will help resolving any linking issues with preCICE itself.
This should mean, that dynamic linking works.
Could you check if all version links are present in the spack installation?
It is possible that the link from libprecice.so.3 to libprecice.so.3.1.0 is missing.
Example:
$ ls -l $(where precice-tools | sed "s/bin\/precice-tools/lib/") | grep "libprecice.so"
lrwxrwxrwx 1 x x 15 Apr 4 12:41 libprecice.so -> libprecice.so.3
lrwxrwxrwx 1 x x 19 Apr 4 12:41 libprecice.so.3 -> libprecice.so.3.1.0
-rwxrwxr-x 1 x x 7298672 Apr 4 12:41 libprecice.so.3.1.0
I already compiled the c++ side of our coupling and there were no problems linking with the spack installation
ccmake ..//
precice_DIR /zhome/academic/HLRS/isd/isdsg/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-11.2.0/precice-3.1.0-yusoyoeupbrugromwevcy2lrfr7e7oah/lib64/cmake/
ldd solver_executable
libprecice.so.3 => /zhome/academic/HLRS/isd/isdsg/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-11.2.0/precice-3.1.0-yusoyoeupbrugromwevcy2lrfr7e7oah/lib64/libprecice.so.3 (0x00007f1e18e88000)
The command you suggested was not doable the HPC Os, so i am not sure about the 3.1.0 link:D
From my feeling, i would locate the problem more to the pip installation, i was asking myself if it is enough to provide the pkgconfig for this installation?
Ah lib64 … anyhow I cannot see a reason why this shouldn’t work.
ldd would have been my next suggestion. Looks like libprecice.so.3 is detected at runtime.
It must be something weird with pip.
Could you try installing in a venv?
python3 -m venv venv
. venv/bin/activate
pip install pyprecice==3.1.0
python -c "import precice"
I tried it and got this issue:
python3 -c "import precice"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/zhome/academic/HLRS/isd/isdsg/preCICE/venv/lib/python3.10/site-packages/precice/__init__.py", line 5, in <module>
from cyprecice import Participant, get_version_information
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /zhome/academic/HLRS/isd/isdsg/preCICE/venv/lib/python3.10/site-packages/cyprecice.cpython-310-x86_64-linux-gnu.so)
I also seen this one Interoperability with pip · Issue #28282 · spack/spack · GitHub. I am not sure if its already possible but this looks also natural to me:
spack create myenv
spack activate myenv
spack install py-pip
pip install rich
python -m rich # works
despacktivate
python -m rich # expect ImportError
I dont know why exactly but creating a spack enviroment und using the pip inside the enviroment seems to work. But i think it just worked because i started the terminal and activated the enviroment again:
spack env create myprecice
spack install --add precice
spack install --add py-pip
spack install --add py-numpy
spack get config
spack:
# add package specs to the `specs` list
specs:
- precice
- py-pip
- py-numpy
then installing the packages with the spacks pip
pip install pyprecice==3.1.0
pip show pyprecice
Name: pyprecice
Version: 3.1.0
Summary: Python language bindings for the preCICE coupling library
Home-page: https://github.com/precice/python-bindings
Author: the preCICE developers
Author-email: info@precice.org
License: LGPL-3.0
Location: /zhome/academic/HLRS/isd/isdsg/.local/lib/python3.11/site-packages
Requires: Cython, mpi4py, numpy
Required-by: micro-manager-precice
bash-4.4$ pip show micro-manager-precice
Name: micro-manager-precice
Version: 0.4.0
#Example micro-manager
cd micro_manager
pip install --user .
python3 -c "import precice"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/zhome/academic/HLRS/isd/isdsg/.local/lib/python3.11/site-packages/precice/__init__.py", line 5, in <module>
from cyprecice import Participant, get_version_information
File "cyprecice/cyprecice.pyx", line 1, in init cyprecice
ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).
#Resolve error by deactivating and activating
spack env deactivate
spack env activate myprecice
python3 -c "import precice"
#no return
@fsimonis @IshaanDesai Any other usefull information in mind?
I am wondering if the spack-provided pip conflicts somehow with the system-provided one.
Can you try to use the spack-provided python to create a venv and install the python bindings in there?
I would assume the spacks pip is isolated from the systems one.
which pip
/opt/hlrs/non-spack/2022-02/development/python/vanilla_python/3.10.4/bin/pip
spack env activate myprecice
which pip
~/spack/var/spack/environments/myprecice/.spack-env/view/bin/pip
Your idea also worked, but you have to make sure that spack doesnt uses the systems python (Either installing a package like py-numpy or probably changing paths). And i also had to leave the spack env and activate it again
spack env activate myprecice
. venv/bin/activate
which pip
~/venv/bin/pip
pip show pyprecice
Name: pyprecice
Version: 3.1.0
Summary: Python language bindings for the preCICE coupling library
Home-page: https://github.com/precice/python-bindings
Author: the preCICE developers
Author-email: info@precice.org
License: LGPL-3.0
Location: /zhome/academic/HLRS/isd/isdsg/venv/lib/python3.11/site-packages
Requires: Cython, mpi4py, numpy
Required-by:
I wonder about the installation directory. The import error above suggests the installation path is the same as the one of the system pip.
Have you tried asking in slack of Spack? They are very responsive!
The problem of using spack-pip to install bindings for a spack-provided package cannot be that rare.
They even suggest in the docs that pip is a better fit for python packages that don’t have a compiled component.