Using SLURM with Openfoam and Fenics

Using preCICE
, , ,
1

Hello all,

I’m starting to work with preCICE for (I hope) a long time. The main objective is to couple OF and Fenics for modelling oscillating and deformable hydrofoil. I didn’t have too much problem to install it and make it work on my own computer (ubuntu).

For now, I’m trying to use preCICE (latest version: 2.3.0) on a cluster with SLURM without any success. The cluster is on CentOS (release 7.8.2003). Openfoam v2112 and dolfin/2019.1.0 are both installed and run correctly as stand-alone. The last version of both adapters are were also installed.

I use the tutorial perpendicular-flap to test my SLURM file. I tried several things but basically the attached slurm file (example.run.precice.txt (2.1 KB)) should work and doesn’t and produces the .out and .err files (precice.1316.err.txt (15.8 KB) precice.1316.out (7.5 KB)).

As you can see, the simulation stop at the beginning of the time loop and the usual precice lines in log files doesn’t even appear and then the mpi process crash, producing unreadable error file…

So, this lead to several questions:

  • First thing first, is dolfin (as fenics is based on dolfin) an appropriate participant ?
  • I didn’t install OF, dolfin, precice and the adapters. Therefore, how can I check if the adapters are well installed ?
  • I’ve seen on the forum that the use of --hostfile could help:
mpirun -np 2 --hostfile A.hostfile A &
mpirun -np 4 --hostfile B.hostfile B

Until now, I didn’t manage to make it work

  • I managed at some point to get the simulation to get stuck at:
Setting up master communication to coupling partner/s

but somehow, I can’t manage to get it again.

  • Any help on SLURM with openfoam and fenics would be greatly appreciated !

As you can see, I’m trying to understand what is happening…

Thanks,

Paul

2

Hi Paul,

Be warned: There is a lot of text/ideas coming. :slight_smile:

dolfin/FEniCS is definitely a valid participant. I had some successful simulations of FEniCS codes coupled with other FEniCS codes. Do you use the FEniCS-preCICE adapter for the coupling or preCICE’s Python bindings? I used the bindings in my case.

Error in error output file

Your error output actually shows that OpenFOAM seems to be crashing?! Do you know what is going on there? It could be interesting to see how the mpirun call looks in there. Is this error message shown because you kill the job or is it actually an issue with OpenFOAM?

run.sh files

It might be interesting to see what happens in the run.sh files that you use to start your solvers. Could you share these files with us?

Exchange directory

If the two solvers do not find each other, there might be a problem with the exchange directory that is needed for participants to find each other. It should point to a shared filesystem that is accessible (write and read) by all compute nodes.

I have been on one machine with Slurm where the initialization of the communication took forever. So the whole Setting up master communication would take about 15 minutes. This must be a problem with this particular server. I had other servers with Slurm the coupling setup behaved normally. So you could try waiting longer for the communication to be set up if none of your solver crashes.

Debugging

Regarding deeper debugging since you did not install the software:

  • Try running a FEniCS example in parallel (without preCICE) using SLURM to verify that FEniCS works in parallel.
  • Try running a OpenFOAM example in parallel (without preCICE) using SLURM to verify that OpenFOAM works in parallel.
  • There are some parallel solver dummies that use preCICE which should be easy to use. There you could use the C++ and the Python dummies. This way you could figure out whether preCICE works as expected. I have not tested these dummies with SLURM yet, but they should work (keep the exchange director in mind). Any feedback regarding these dummies would be greatly appreciated.
  • You could also run of the preCICE tutorials to see if everything works as expected. These tutorials are usually well-maintained this should rule out other potential problems. There should be some tutorial using OpenFOAM and FEniCS. I am not sure if all of these tutorials are suitable for parallel runs though.
1 Like
3

Hi @Paul :wave:
Thanks, @ajaust for the detailed help!

Some more material that might help:

https://precice.org/installation-special-systems.html

4

Hi all!
Thank you for your recommendations.
With @Paul, we found that the problem occurs when specifying the openfoam precice adapter in the controlDict file for the fluid participant.
Actually, Fenics and openFoam work as expected in parallel with SLURM and the ldd $FOAM_USER_LIBBIN/libpreciceAdapterFunctionObject.so does not return any error. We suspect that in fact SLURM is not the culprit.

When running the fluid participant of the perpendicular-flap case with OpenFoam without the following lines in controlDict

functions
{
    preCICE_Adapter
    {
        type preciceAdapterFunctionObject;
        libs ("libpreciceAdapterFunctionObject.so");
    }
}

everything perform well, but with the lines above we obtain the following error:

[0] #0  Foam::error::printStack(Foam::Ostream&)[2] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[30517,1],0] (PID 204887)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[1] #0  Foam::error::printStack(Foam::Ostream&)[3] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
[1] #1  Foam::sigFpe::sigHandler(int) at ??:?
[2] #1  Foam::sigFpe::sigHandler(int) at ??:?
[0] #1  Foam::sigFpe::sigHandler(int) at ??:?
[3] #1  Foam::sigFpe::sigHandler(int) at ??:?
[0] #2  ? at ??:?
[1] #2  ? at ??:?
[2] #2  ? at ??:?
[3] #2  ? in /lib64/libpthread.so.0
[2] #3  ? in /lib64/libpthread.so.0
[0] #3  ? in /lib64/libpthread.so.0
[1] #3  ? in /lib64/libpthread.so.0
[3] #3  ? in /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12
[2] #4  ? in /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12
[1] #4  ? in /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12
[3] #4  ? in /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12
[0] #4  ? in /lib64/ld-linux-x86-64.so.2
[2] #5  ? in /lib64/ld-linux-x86-64.so.2
[3] #5  ? in /lib64/ld-linux-x86-64.so.2
[1] #5  ? in /lib64/ld-linux-x86-64.so.2
[0] #5  ? in /lib64/ld-linux-x86-64.so.2
[1] #6  ? in /lib64/ld-linux-x86-64.so.2
[3] #6  ? in /lib64/ld-linux-x86-64.so.2
[2] #6  ? in /lib64/ld-linux-x86-64.so.2
[0] #6  ? in /lib64/ld-linux-x86-64.so.2
[2] #7  ? in /lib64/ld-linux-x86-64.so.2
[1] #7  ? in /lib64/ld-linux-x86-64.so.2
[3] #7  ? in /lib64/ld-linux-x86-64.so.2
[0] #7  ? in /lib64/ld-linux-x86-64.so.2
[1] #8  ? in /lib64/ld-linux-x86-64.so.2
[2] #8  ? in /lib64/ld-linux-x86-64.so.2
[3] #8  ? in /lib64/ld-linux-x86-64.so.2
[0] #8  ? in /lib64/libdl.so.2
[1] #9  ? in /lib64/libdl.so.2
[3] #9  ? in /lib64/libdl.so.2
[2] #9  ? in /lib64/libdl.so.2
[0] #9  ? in /lib64/ld-linux-x86-64.so.2
[1] #10  ? in /lib64/ld-linux-x86-64.so.2
[0] #10  ? in /lib64/ld-linux-x86-64.so.2
[3] #10  ? in /lib64/ld-linux-x86-64.so.2
[2] #10  ? in /lib64/libdl.so.2
[1] #11  dlopen in /lib64/libdl.so.2
[0] #11  dlopen in /lib64/libdl.so.2
[3] #11  dlopen in /lib64/libdl.so.2
[2] #11  dlopen in /lib64/libdl.so.2
[1] #12  Foam::dlOpen(Foam::fileName const&, bool) in /lib64/libdl.so.2
 in /lib64/libdl.so.2
[3] #12  Foam::dlOpen(Foam::fileName const&, bool)[0] #12  Foam::dlOpen(Foam::fileName const&, bool) in /lib64/libdl.so.2
[2] #12  Foam::dlOpen(Foam::fileName const&, bool) at ??:?
[0] #13  Foam::dlOpen(Foam::fileName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ??:?
[1] #13  Foam::dlOpen(Foam::fileName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ??:?
[3] #13  Foam::dlOpen(Foam::fileName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ??:?
[2] #13  Foam::dlOpen(Foam::fileName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ??:?
[3] #14  Foam::dlLibraryTable::openLibrary(Foam::fileName const&, bool) at ??:?
[0] #14  Foam::dlLibraryTable::openLibrary(Foam::fileName const&, bool) at ??:?
[2] #14  Foam::dlLibraryTable::openLibrary(Foam::fileName const&, bool) at ??:?
[1] #14  Foam::dlLibraryTable::openLibrary(Foam::fileName const&, bool) at ??:?
[0] #15  Foam::dlLibraryTable::open(Foam::fileName const&, bool) at ??:?
[3] #15  Foam::dlLibraryTable::open(Foam::fileName const&, bool) at ??:?
[1] #15  Foam::dlLibraryTable::open(Foam::fileName const&, bool) at ??:?
[2] #15  Foam::dlLibraryTable::open(Foam::fileName const&, bool) at ??:?
[0] #16  bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >* const&, bool) at ??:?
[3] #16  bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >* const&, bool) at ??:?
[1] #16  bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >* const&, bool) at ??:?
[2] #16  bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >* const&, bool) at ??:?
[0] #17  Foam::functionObject::New(Foam::word const&, Foam::Time const&, Foam::dictionary const&) at ??:?
[3] #17  Foam::functionObject::New(Foam::word const&, Foam::Time const&, Foam::dictionary const&) at ??:?
[1] #17  Foam::functionObject::New(Foam::word const&, Foam::Time const&, Foam::dictionary const&) at ??:?
[2] #17  Foam::functionObject::New(Foam::word const&, Foam::Time const&, Foam::dictionary const&) at ??:?
[1] #18  Foam::functionObjectList::read() at ??:?
[0] #18  Foam::functionObjectList::read() at ??:?
[3] #18  Foam::functionObjectList::read() at ??:?
[2] #18  Foam::functionObjectList::read() at ??:?
[1] #19  Foam::Time::run() const at ??:?
[3] #19  Foam::Time::run() const at ??:?
[0] #19  Foam::Time::run() const at ??:?
[1] #20   at ??:?
[2] #19  Foam::Time::run() const at ??:?
[0] #20   at ??:?
[3] #20  ?? at ??:?
[2] #20  ? at ??:?
[3] #21  __libc_start_main? at ??:?
[0] #21  __libc_start_main at ??:?
[1] #21  __libc_start_main in /lib64/libc.so.6
[3] #22   in /lib64/libc.so.6
[1] #22   at ??:?
[2] #21  __libc_start_main in /lib64/libc.so.6
[0] #22  ??? in /lib64/libc.so.6
[2] #22   at ??:?
[node009:204888] *** Process received signal ***
[node009:204888] Signal: Floating point exception (8)
[node009:204888] Signal code:  (-6)
[node009:204888] Failing at address: 0x3e900032058
[node009:204888] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204888] [ 1] /lib64/libpthread.so.0(raise+0x2b)[0x7fffefe9f4fb]
[node009:204888] [ 2] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204888] [ 3] /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12(+0x23695)[0x7fffb8a9d695]
[node009:204888] [ 4] /lib64/ld-linux-x86-64.so.2(+0xf9b3)[0x7ffff7dea9b3]
[node009:204888] [ 5] /lib64/ld-linux-x86-64.so.2(+0x1458e)[0x7ffff7def58e]
[node009:204888] [ 6] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204888] [ 7] /lib64/ld-linux-x86-64.so.2(+0x13b7b)[0x7ffff7deeb7b]
[node009:204888] [ 8] /lib64/libdl.so.2(+0xfab)[0x7ffff0949fab]
[node009:204888] [ 9] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204888] [10] /lib64/libdl.so.2(+0x15ad)[0x7ffff094a5ad]
[node009:204888] [11] /lib64/libdl.so.2(dlopen+0x31)[0x7ffff094a041]
[node009:204888] [12] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameEb+0x36)[0x7ffff14c5ec6]
[node009:204888] [13] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xe)[0x7ffff14c704e]
[node009:204888] [14] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable11openLibraryERKNS_8fileNameEb+0x10e)[0x7ffff115debe]
[node009:204888] [15] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openERKNS_8fileNameEb+0x16)[0x7ffff115f026]
[node009:204888] [16] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openIPNS_9HashTableIPFNS_7autoPtrINS_14functionObjectEEERKNS_4wordERKNS_4TimeERKNS_10dictionaryEES6_NS_4HashIS6_EEEEEEbSE_S8_RKT_b+0xb0)[0x7ffff116ff90]
[node009:204888] [17] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14functionObject3NewERKNS_4wordERKNS_4TimeERKNS_10dictionaryE+0x15e)[0x7ffff116d35e]
[node009:204888] [18] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList4readEv+0xc19)[0x7ffff1174e39]
[node009:204888] [19] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x2c7)[0x7ffff118ba87]
[node009:204888] [20] pimpleFoam[0x42b2d4]
[node009:204888] [21] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fffefae4555]
[node009:204888] [22] pimpleFoam[0x4300d8]
[node009:204888] *** End of error message ***
? at ??:?
[node009:204887] *** Process received signal ***
[node009:204887] Signal: Floating point exception (8)
[node009:204887] Signal code:  (-6)
[node009:204887] Failing at address: 0x3e900032057
[node009:204887] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204887] [ 1] /lib64/libpthread.so.0(raise+0x2b)[0x7fffefe9f4fb]
[node009:204887] [ 2] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204887] [ 3] /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12(+0x23695)[0x7fffb8a9d695]
[node009:204887] [ 4] /lib64/ld-linux-x86-64.so.2(+0xf9b3)[0x7ffff7dea9b3]
[node009:204887] [ 5] /lib64/ld-linux-x86-64.so.2(+0x1458e)[0x7ffff7def58e]
[node009:204887] [ 6] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204887] [ 7] /lib64/ld-linux-x86-64.so.2(+0x13b7b)[0x7ffff7deeb7b]
[node009:204887] [ 8] /lib64/libdl.so.2(+0xfab)[0x7ffff0949fab]
[node009:204887] [ 9] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204887] [10] /lib64/libdl.so.2(+0x15ad)[0x7ffff094a5ad]
[node009:204887] [11] /lib64/libdl.so.2(dlopen+0x31)[0x7ffff094a041]
[node009:204887] [12] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameEb+0x36)[0x7ffff14c5ec6]
[node009:204887] [13] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xe)[0x7ffff14c704e]
[node009:204887] [14] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable11openLibraryERKNS_8fileNameEb+0x10e)[0x7ffff115debe]
[node009:204887] [15] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openERKNS_8fileNameEb+0x16)[0x7ffff115f026]
[node009:204887] [16] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openIPNS_9HashTableIPFNS_7autoPtrINS_14functionObjectEEERKNS_4wordERKNS_4TimeERKNS_10dictionaryEES6_NS_4HashIS6_EEEEEEbSE_S8_RKT_b+0xb0)[0x7ffff116ff90]
[node009:204887] [17] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14functionObject3NewERKNS_4wordERKNS_4TimeERKNS_10dictionaryE+0x15e)[0x7ffff116d35e]
[node009:204887] [18] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList4readEv+0xc19)[0x7ffff1174e39]
[node009:204887] [19]  at ??:?
[node009:204890] *** Process received signal ***
[node009:204890] Signal: Floating point exception (8)
[node009:204890] Signal code:  (-6)
[node009:204890] Failing at address: 0x3e90003205a
[node009:204890] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204890] [ 1] /lib64/libpthread.so.0(raise+0x2b)[0x7fffefe9f4fb]
[node009:204890] [ 2] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204890] [ 3] /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12(+0x23695)[0x7fffb8a9d695]
[node009:204890] [ 4] /lib64/ld-linux-x86-64.so.2(+0xf9b3)[0x7ffff7dea9b3]
[node009:204890] [ 5] /lib64/ld-linux-x86-64.so.2(+0x1458e)[0x7ffff7def58e]
[node009:204890] [ 6] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204890] [ 7] /lib64/ld-linux-x86-64.so.2(+0x13b7b)[0x7ffff7deeb7b]
[node009:204890] [ 8] /lib64/libdl.so.2(+0xfab)[0x7ffff0949fab]
[node009:204890] [ 9] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204890] [10] /lib64/libdl.so.2(+0x15ad)[0x7ffff094a5ad]
[node009:204890] [11] /lib64/libdl.so.2(dlopen+0x31)[0x7ffff094a041]
[node009:204890] [12] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameEb+0x36)[0x7ffff14c5ec6]
[node009:204890] [13] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xe)[0x7ffff14c704e]
[node009:204890] [14] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable11openLibraryERKNS_8fileNameEb+0x10e)[0x7ffff115debe]
[node009:204890] [15] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openERKNS_8fileNameEb+0x16)[0x7ffff115f026]
[node009:204890] [16] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openIPNS_9HashTableIPFNS_7autoPtrINS_14functionObjectEEERKNS_4wordERKNS_4TimeERKNS_10dictionaryEES6_NS_4HashIS6_EEEEEEbSE_S8_RKT_b+0xb0)[0x7ffff116ff90]
[node009:204890] [17] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14functionObject3NewERKNS_4wordERKNS_4TimeERKNS_10dictionaryE+0x15e)[0x7ffff116d35e]
[node009:204890] [18] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList4readEv+0xc19)[0x7ffff1174e39]
[node009:204890] [19] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x2c7)[0x7ffff118ba87]
[node009:204890] [20] pimpleFoam[0x42b2d4]
[node009:204890] [21] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fffefae4555]
[node009:204890] [22] pimpleFoam[0x4300d8]
[node009:204890] *** End of error message ***
/opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x2c7)[0x7ffff118ba87]
[node009:204887] [20] pimpleFoam[0x42b2d4]
[node009:204887] [21] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fffefae4555]
[node009:204887] [22] pimpleFoam[0x4300d8]
[node009:204887] *** End of error message ***
 at ??:?
[node009:204889] *** Process received signal ***
[node009:204889] Signal: Floating point exception (8)
[node009:204889] Signal code:  (-6)
[node009:204889] Failing at address: 0x3e900032059
[node009:204889] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204889] [ 1] /lib64/libpthread.so.0(raise+0x2b)[0x7fffefe9f4fb]
[node009:204889] [ 2] /lib64/libpthread.so.0(+0xf630)[0x7fffefe9f630]
[node009:204889] [ 3] /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12(+0x23695)[0x7fffb8a9d695]
[node009:204889] [ 4] /lib64/ld-linux-x86-64.so.2(+0xf9b3)[0x7ffff7dea9b3]
[node009:204889] [ 5] /lib64/ld-linux-x86-64.so.2(+0x1458e)[0x7ffff7def58e]
[node009:204889] [ 6] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204889] [ 7] /lib64/ld-linux-x86-64.so.2(+0x13b7b)[0x7ffff7deeb7b]
[node009:204889] [ 8] /lib64/libdl.so.2(+0xfab)[0x7ffff0949fab]
[node009:204889] [ 9] /lib64/ld-linux-x86-64.so.2(+0xf7c4)[0x7ffff7dea7c4]
[node009:204889] [10] /lib64/libdl.so.2(+0x15ad)[0x7ffff094a5ad]
[node009:204889] [11] /lib64/libdl.so.2(dlopen+0x31)[0x7ffff094a041]
[node009:204889] [12] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameEb+0x36)[0x7ffff14c5ec6]
[node009:204889] [13] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6dlOpenERKNS_8fileNameERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xe)[0x7ffff14c704e]
[node009:204889] [14] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable11openLibraryERKNS_8fileNameEb+0x10e)[0x7ffff115debe]
[node009:204889] [15] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openERKNS_8fileNameEb+0x16)[0x7ffff115f026]
[node009:204889] [16] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14dlLibraryTable4openIPNS_9HashTableIPFNS_7autoPtrINS_14functionObjectEEERKNS_4wordERKNS_4TimeERKNS_10dictionaryEES6_NS_4HashIS6_EEEEEEbSE_S8_RKT_b+0xb0)[0x7ffff116ff90]
[node009:204889] [17] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam14functionObject3NewERKNS_4wordERKNS_4TimeERKNS_10dictionaryE+0x15e)[0x7ffff116d35e]
[node009:204889] [18] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList4readEv+0xc19)[0x7ffff1174e39]
[node009:204889] [19] /opt/lasie/pub/libs/gnu8/openmpi3/openfoam/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x2c7)[0x7ffff118ba87]
[node009:204889] [20] pimpleFoam[0x42b2d4]
[node009:204889] [21] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fffefae4555]
[node009:204889] [22] pimpleFoam[0x4300d8]
[node009:204889] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 204888 on node node009 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
[node009:204882] 3 more processes have sent help message help-opal-runtime.txt / opal_init:warn-fork
[node009:204882] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Any idea on where we missed something?

1 Like
5

When loading the adapter, OpenFOAM also loads preCICE, which will also load OpenMPI. Could it be that OpenFOAM and preCICE are built with different MPI implementations/versions?

6

Hi, Thank you for your reply
On our cluster, we have a single set of compilers (gnu8) and a single implementation of MPI (openmpi3) so it is not possible that we used different compilers/versions.
Best
Antoine

7

But how was OpenFOAM installed? Because, when building OpenFOAM from source, it may use the dependencies bundled in its ThirdParty directory.

8

Hi @Makis,

Thanks for you help. We installed OF again, from source and without the ThirdParty directory. Whether the simulation is parallel or serial, the same error still appears…

#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::sigFpe::sigHandler(int) at ??:?
#2  ? in /lib64/libpthread.so.0
#3  ? in /opt/ohpc/pub/libs/gnu8/openmpi3/trilinos/12.14.1/lib/libteuchoscore.so.12
#4  ? in /lib64/ld-linux-x86-64.so.2
#5  ? in /lib64/ld-linux-x86-64.so.2
#6  ? in /lib64/ld-linux-x86-64.so.2
#7  ? in /lib64/ld-linux-x86-64.so.2
#8  ? in /lib64/libdl.so.2
#9  ? in /lib64/ld-linux-x86-64.so.2
#10  ? in /lib64/libdl.so.2
#11  dlopen in /lib64/libdl.so.2
#12  Foam::dlOpen(Foam::fileName const&, bool) at ??:?
#13  Foam::dlOpen(Foam::fileName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) at ??:?
#14  Foam::dlLibraryTable::openLibrary(Foam::fileName const&, bool) at ??:?
#15  Foam::dlLibraryTable::open(Foam::fileName const&, bool) at ??:?
#16  bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::functionObject> (*)(Foam::word const&, Foam::Time const&, Foam::dictionary const&), Foam::word, Foam::Hash<Foam::word> >* const&, bool) at ??:?
#17  Foam::functionObject::New(Foam::word const&, Foam::Time const&, Foam::dictionary const&) at ??:?
#18  Foam::functionObjectList::read() at ??:?
#19  Foam::Time::run() const at ??:?
#20  ? at ??:?
#21  __libc_start_main in /lib64/libc.so.6
#22  ? at ??:?
/var/spool/slurm/d/job01433/slurm_script : ligne 66 : 170267 Exception en point flottant   pimpleFoam

So I’m guessing the mpi calling is not the problem… In fact, when solving only the fluid part with OpenFoam of the perpendicular-flap or the quickstart tutorials with loading precice, the error occurs.

We run out of idea here… Any guess would be nice :slight_smile: !
Thanks

Paul

9

Did you mean to write “without loading preCICE” here? Because In the last error trace, I see nothing related to preCICE.

10

Hi @Makis ,

To make myself clear, you can find attached a fluid-OpenFoam case based on the quickstart tutorial (here: fluid-quickstart.tar (220 KB)). In the controlDict file, I commented out the Functions lines and I added the libs line which import the libpreciceAdapterFunctionObject.so library.
Running .\run.sh generates the error we talked about before, and it is visible in the run.log. In addition, if I comment out the libs line, the fluid case run fine.

In the mean time, I tried to ran the solverdummies you provide with preCICE and… it ran without any problem. Can I deduce from this test that preCICE is well installed ? If yes, I think you were right when you said preCICE is not responsible for the error.

We’ll continue to look at compiling error in OpenFoam. But the fact that only adding these lines in the controlDict files messes up the fluid case is really strange :
libs ( "/home/pbrousseau/OpenFOAM/pbrousseau-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so" );

Do you have any idea on where to look in the compiling logs of OpenFoam ?

Thanks,
Paul

11

I would look into the ldd.log of the adapter. You can also run ldd <whatever_library>.so to see if all symbols are resolved to some existing library.

12

Related issue in the OpenFOAM GitLab: https://develop.openfoam.com/Development/openfoam/-/issues/2437

1 Like