OpenFOAM adapter installation on HPC (cluster)

Official adapters and tutorials
1

Hello,

General Problem

I am trying to run the coupled simulation on a supercomputing platform.
I plan to do the 6-2ay CFD-DEM-FEM coupling on the cluster instead of the personal computer.

I have now thoroughly tested out the case and physics on a local workstation, thus the case itself is ready to be run for a longer duration.

I am facing issues trying to run the OpenFOAM participant. I am able to run the XDEM participant, so I can confirm that preCICE was installed properly.

OS

The cluster has RedHat (RHEL) Linux OS, v8.
OpenFOAM v7

Installation process:

  • preCICE: build from source (latest version)
    I add the following to my run script when running or compiling other adapters
PRECICE_PREFIX=/home/users/padhav/0_installs/install_preCICE # set this to your selected prefix
export PATH=$PRECICE_PREFIX/bin:$PATH
export LD_LIBRARY_PATH=$PRECICE_PREFIX/lib:$LD_LIBRARY_PATH
export CPATH=$PRECICE_PREFIX/include:$CPATH
# Enable detection with pkg-config and CMake
export PKG_CONFIG_PATH=$PRECICE_PREFIX/lib64/pkgconfig:$PKG_CONFIG_PATH
export CMAKE_PREFIX_PATH=$PRECICE_PREFIX:$CMAKE_PREFIX_PATH
export PATH="$PATH:/home/users/padhav/0_installs/build_preCICE"
export PATH="$PATH:/home/users/padhav/0_installs/build_preCICE/lib/pkgconfig/libprecice.pc"

I confirm that libprecice.so and libprecice.so.2 are created. This is further confirmed, as I am able to run the XDEM adapter, it initializes and waits for the other participant.

  • OpenFOAM: I use the pre-installed module on HPC to compile the adapter. wmake.log and ldd.log are attached. I see that the adapter library .so file is created in the $FOAM_USER_LIBBIN. I do get an error during compilation, but I know this error exists in the local installation as well, and the adapter works just fine. (wmake_local.log and ldd_local.log added for installations on a local workstation.)

Running the simulation

  • Case set up: The only thing that I had to modify was giving the absolute path of the OpenFOAM adapter in the Fluid/system/controlDict as follows:
functions
{
    preCICE_Adapter
    {
        type preciceAdapterFunctionObject;
        libs ("/home/users/padhav/OpenFOAM/padhav-7/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject_xdem_coupling-HMT-devInterFoam.so");
    }
}
  • Load the environment: I need to load the modules and other environment variables
source /work/projects/luxdem/easybuild/load-luxdem-settings-aion-2020b.sh
module load toolchain/XDEM-devel/5.0-foss-2020b-OF7
source "$FOAM_BASH"
export LD_LIBRARY_PATH=$FOAM_USER_LIBBIN:$LD_LIBRARY_PATH

Note: If I don’t add the $FOAM_USER_LIBBIN to $LD_LIBRARY_PATH, OpenFOAM still complains it cannot find the OpenFOAM adapter .so file. But if I add this the error is modified.

Error

Here is the output with the relevant error:

[6] --> FOAM FATAL ERROR: 
[6] Unknown function type preciceAdapterFunctionObject

Valid functions are : 

6
(
patchProbes
probes
psiReactionThermoMoleFractions
rhoReactionThermoMoleFractions
sets
surfaces
)

[6] 
[6] 
[6]     From function static Foam::autoPtr<Foam::functionObject> Foam::functionObject::New(const Foam::word&, const Foam::Time&, const Foam::dictionary&)
[6]     in file db/functionObjects/functionObject/functionObject.C at line 100.
[6] 
FOAM parallel run exiting
[6] 
[2] 
[2] 
[2] --> FOAM FATAL ERROR: 
[2] Unknown function type preciceAdapterFunctionObject

Valid functions are : 

6
(
patchProbes
probes
psiReactionThermoMoleFractions
rhoReactionThermoMoleFractions
sets
surfaces
)

As per the Troubleshoot page, I should check the following:
check if libpreciceAdapterFunctionObject.so exist (my adapter name is modified but it exists)
check if ldd $FOAM_USER_LIBBIN/libpreciceAdapterFunctionObject.so returns any errors (I run this command, and don’t see any error, not sure if I am looking correctly)

Help Wanted

2

Looking at the end of ldd $FOAM_USER_LIBBIN/libpreciceAdapterFunctionObject.so output, I can see that everything related to SolverInterface has an issue. So I believe could not open file precice/SolverInterface.hpp from wmake.log is the reason.

Now just have to figure out how to get the linking working.

3

I think the following clues are important:

  1. The code compiles and links at compile time, since there is no error in wmake.log. (what you get is only warnings, which are not related to this issue)

  2. As you already spotted, ldd.log shows that the runtime cannot resolve any symbol coming from the preCICE library. It is not that it cannot find one, it cannot find any. This probably excludes a (common) issue of having multiple preCICE versions installed, one picked up by the compiler, and one picked up by the runtime.

To simplify the problem, you could try running the elastic-tube-1d C++ tutorial. This should already reproduce it.

The most common cause I could think of is that preCICE is available in your login node environment, but not in the SLURM job you are executing. Maybe this (quite outdated) documentation page of special systems could help.

By the way,

This worries me. How do you activate your OpenFOAM environment?

1 Like
4

Dear @Makis thank you for your reply, I am trying to solve the issue based on your suggestions. I will reply to this first as I know the answer.

We have a “project” directory on the HPC where @xbesseron has compiled/installed different software. Including OpenFOAM v7. I believe they are installed via Easybuild as a “module” (@xbesseron please correct me if I am wrong).

I load the OpenFOAM via following commands.

source /work/projects/luxdem/easybuild/load-luxdem-settings-aion-2020b.sh # loads all the dependencies 
module load toolchain/XDEM-devel/5.0-foss-2020b-OF7

Then I source the OpenFOAM bashrc source "$FOAM_BASH".

Edit:
I still consider myself a novice when it comes to HPC. But if this still does not clarify, I will ask my colleagues and reply with the updated information.

5

Solution

I was able to compile and run the 1D elastic tube case, but this was in an old terminal.

This is when I realized my colleague @xbesseron has installed preCICE as a module on the HPC as well. Using this preCICE module I recompiled the OpenFOAM adapter.

I did not get any wmake errors. I was able to start the simulation as well.

Note

When I using the paths to manually linking preCICE they were conflicting with the module preCICE

3 Likes
6

Nice that you solved it!

This is expected: module files also modify the environment.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.