I’m trying to couple a 3D case using Openfoam with a solid solver that I wrote in python. I also using an HPC to do it. The simulation starts well but at a certant point the simulation explode with this OpenFOAM error
---[precice] e[0m Mapping "Force" for t=64.012 from "Mesh-Fluid" to "Mesh-Solid"
---[precice] e[0m Time window completed
---[precice] e[0m Mapping "Displacement" for t=64.02499999999999 from "Mesh-Solid" to "Mesh-Fluid"
---[precice] e[0m iteration: 1, time-window: 4925, time: 64.012 of 6775, time-window-size: 0.013, max-time-step-size: 0.013, ongoing: yes, time-window-complete: yes,
[78] #0 Foam::error::printStack(Foam::Ostream&)[79] #0 Foam::error::printStack(Foam::Ostream&) at ??:?
at ??:?
[79] #1 Foam::sigFpe::sigHandler(int)[78] #1 Foam::sigFpe::sigHandler(int) at ??:?
[79] #2 ? at ??:?
[78] #2 ? in /lib/x86_64-linux-gnu/libc.so.6
[79] #3 ? in /lib/x86_64-linux-gnu/libc.so.6
[78] #3 ? in /lib/x86_64-linux-gnu/libprecice.so.3
[79] #4 ? in /lib/x86_64-linux-gnu/libprecice.so.3
[78] #4 ? in /lib/x86_64-linux-gnu/libprecice.so.3
[78] #5 preciceAdapter::Interface::readCouplingData(double) in /lib/x86_64-linux-gnu/libprecice.so.3
[79] #5 preciceAdapter::Interface::readCouplingData(double) at ??:?
[79] #6 preciceAdapter::Adapter::readCouplingData(double) at ??:?
[78] #6 preciceAdapter::Adapter::readCouplingData(double) at ??:?
[79] #7 preciceAdapter::Adapter::adjustSolverTimeStepAndReadData() at ??:?
[78] #7 preciceAdapter::Adapter::adjustSolverTimeStepAndReadData() at ??:?
[79] #8 preciceAdapter::Adapter::execute() at ??:?
[78] #8 preciceAdapter::Adapter::execute() at ??:?
[79] #9 Foam::functionObjects::preciceAdapterFunctionObject::execute() at ??:?
[78] #9 Foam::functionObjects::preciceAdapterFunctionObject::execute() at ??:?
at ??:?
[79] #10 Foam::functionObjectList::execute()[78] #10 Foam::functionObjectList::execute() at ??:?
[79] #11 at ??:?
Foam::Time::run() const[78] #11 Foam::Time::run() const at ??:?
[79] #12 at ??:?
[78] #12 ?? in /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/bin/pimpleFoam
[78] #13 ? in /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/bin/pimpleFoam
[79] #13 ? in /lib/x86_64-linux-gnu/libc.so.6
[78] #14 __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
[79] #14 __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
[79] #15 in /lib/x86_64-linux-gnu/libc.so.6
[78] #15 ?? in /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/bin/pimpleFoam
[slurmwkramdepyc:1937059] *** Process received signal ***
[slurmwkramdepyc:1937059] Signal: Floating point exception (8)
[slurmwkramdepyc:1937059] Signal code: (-6)
[slurmwkramdepyc:1937059] Failing at address: 0x2c41001d8ea3
[slurmwkramdepyc:1937059] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f1cf247d520]
[slurmwkramdepyc:1937059] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f1cf24d19fc]
[slurmwkramdepyc:1937059] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f1cf247d476]
[slurmwkramdepyc:1937059] [ 3] in /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/bin/pimpleFoam
[slurmwkramdepyc:1937064] *** Process received signal ***
[slurmwkramdepyc:1937064] Signal: Floating point exception (8)
[slurmwkramdepyc:1937064] Signal code: (-6)
[slurmwkramdepyc:1937064] Failing at address: 0x2c41001d8ea8
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f1cf247d520]
[slurmwkramdepyc:1937059] [ 4] /lib/x86_64-linux-gnu/libprecice.so.3(+0x2830a1)[0x7f1cc0c8c0a1]
[slurmwkramdepyc:1937059] [ 5] /lib/x86_64-linux-gnu/libprecice.so.3(+0x26742a)[0x7f1cc0c7042a]
[slurmwkramdepyc:1937059] [ 6] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter9Interface16readCouplingDataEd+0xd7)[0x7f1cc209f167]
[slurmwkramdepyc:1937059] [ 7] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter16readCouplingDataEd+0x36)[0x7f1cc2125c76]
[slurmwkramdepyc:1937059] [ 8] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter31adjustSolverTimeStepAndReadDataEv+0x1a6)[0x7f1cc21261c6]
[slurmwkramdepyc:1937059] [ 9] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter7executeEv+0x3c8)[0x7f1cc213da58]
[slurmwkramdepyc:1937059] [10] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN4Foam15functionObjects28preciceAdapterFunctionObject7executeEv+0x11)[0x7f1cc216bd61]
[slurmwkramdepyc:1937059] [11] [slurmwkramdepyc:1937064] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7faa0de89520]
[slurmwkramdepyc:1937064] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7faa0dedd9fc]
[slurmwkramdepyc:1937064] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7faa0de89476]
[slurmwkramdepyc:1937064] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7faa0de89520]
[slurmwkramdepyc:1937064] [ 4] /lib/x86_64-linux-gnu/libprecice.so.3(+0x2830a1)[0x7fa9dc5d10a1]
[slurmwkramdepyc:1937064] [ 5] /lib/x86_64-linux-gnu/libprecice.so.3(+0x26742a)[0x7fa9dc5b542a]
[slurmwkramdepyc:1937064] [ 6] /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList7executeEv+0x1db)[0x7f1cf2dae0db]
[slurmwkramdepyc:1937059] [12] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter9Interface16readCouplingDataEd+0xd7)[0x7fa9dc748167]
[slurmwkramdepyc:1937064] [ 7] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter16readCouplingDataEd+0x36)[0x7fa9dc7cec76]
[slurmwkramdepyc:1937064] [ 8] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter31adjustSolverTimeStepAndReadDataEv+0x1a6)[0x7fa9dc7cf1c6]
[slurmwkramdepyc:1937064] [ 9] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN14preciceAdapter7Adapter7executeEv+0x3c8)[0x7fa9dc7e6a58]
[slurmwkramdepyc:1937064] [10] /root/OpenFOAM/user-v2112/platforms/linux64GccDPInt32Opt/lib/libpreciceAdapterFunctionObject.so(_ZN4Foam15functionObjects28preciceAdapterFunctionObject7executeEv+0x11)[0x7fa9dc814d61]
[slurmwkramdepyc:1937064] [11] /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x105)[0x7f1cf2dbf345]
[slurmwkramdepyc:1937059] [13] pimpleFoam(+0x1f1ec)[0x55d4d19df1ec]
[slurmwkramdepyc:1937059] [14] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f1cf2464d90]
[slurmwkramdepyc:1937059] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f1cf2464e40]
[slurmwkramdepyc:1937059] [16] pimpleFoam(+0x2b385)[0x55d4d19eb385]
[slurmwkramdepyc:1937059] *** End of error message ***
/usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam18functionObjectList7executeEv+0x1db)[0x7faa0e7ba0db]
[slurmwkramdepyc:1937064] [12] /usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam4Time3runEv+0x105)[0x7faa0e7cb345]
[slurmwkramdepyc:1937064] [13] pimpleFoam(+0x1f1ec)[0x555806c751ec]
[slurmwkramdepyc:1937064] [14] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7faa0de70d90]
[slurmwkramdepyc:1937064] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7faa0de70e40]
[slurmwkramdepyc:1937064] [16] pimpleFoam(+0x2b385)[0x555806c81385]
[slurmwkramdepyc:1937064] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 78 with PID 1937059 on node slurmwkramdepyc exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
I don’t understand if it’s a problem of the couple set up or just the stability of the simulation.
Hi,
I am not sure but it looks very close to the issue I saw recently and that we are starting to discuss here. I am using OpenFOAM coupled with a rigid body solver. My case crashes after around 6850 time steps and I am using a constant time step of 0.01. I am still trying to figure out what happens.
Claudio
Thank you for the reply. Yes, I have a similar situation, in wihch my simulation stops at around 4920 time steps (my increment is 0.013). I tried to change the type of mapping but the problem is still there.
I also looked at the a timestep in paraview near the explosion, and it doesn’t seem strange, and also the displacement are very small compared to the structure… So I don’t really know what’s happening… I already did some simulations with precice but always with CalculiX and it never gave me this type of error (and never in a HPC environment)
The fact that both incidents of a floating-point exception happen after a few thousands of time steps is interesting and could make them related to each other.
@leonardo how long did it take you to reach this time? Would it be possible to rebuild preCICE in debug mode, enable debug messages and run again? If that works, the line [ 4] /lib/x86_64-linux-gnu/libprecice.so.3(+0x2830a1)[0x7f1cc0c8c0a1] should include more useful information (the exact function where this fails). Enabling trace messages (ideally in a separate file) would be even better, but the logs might get extremely large.
Are any of your data values getting too close to zero?
Edit: Could you please also upload your precice-config.xml?
Ok I’ll try to build preCICE in debug mode, even if I never did this… To reach this timestep it takes 6.30h (with 128 cpu). The forces are not near zero, and also the displacement at the last timestep is around 0.09. This is my configuration file. Thank you so much for the help precice-config.xml (2.3 KB)
This always reads the sample at the time offset passed to readData. Which results in the same behaviour if your solvers don’t sample inside the time-window.
If this doesn’t fail in the same way, then we have a bug in the time interpolation.
I added the tags above for both data (leaving all the rest untouched) and this time the simulation failed on the Solid side at time-step 6401 (4 before the previous simulation) with the following message:
ASSERTION FAILED
Location: precice::time::Sample precice::time::Storage::getSampleAtOrAfter(double) const
File: /home/claudio/Projects/precice/precice-3.1.2/src/time/Storage.cpp:155
Expression: stample != _stampleStorage.end()
Rank: 0
Arguments:
0: "no values found!" == no values found!
Stacktrace:
0# 0x00007F9320DCEF39 in /usr/local/lib/libprecice.so.3
1# 0x00007F9320DBFBFA in /usr/local/lib/libprecice.so.3
2# 0x00007F9320DC00E7 in /usr/local/lib/libprecice.so.3
3# 0x00007F9320DC21F4 in /usr/local/lib/libprecice.so.3
4# 0x00007F9320D298F8 in /usr/local/lib/libprecice.so.3
5# 0x00007F9320DA1A53 in /usr/local/lib/libprecice.so.3
6# 0x00007F9320D8BC43 in /usr/local/lib/libprecice.so.3
7# 0x000055F589606FDF in mbdyn-esm-adapter
8# 0x000055F589604028 in mbdyn-esm-adapter
9# 0x00007F932040A1CA in /lib/x86_64-linux-gnu/libc.so.6
10# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
11# 0x000055F589604105 in mbdyn-esm-adapter
[CSIT-LT1087:335444] *** Process received signal ***
[CSIT-LT1087:335444] Signal: Aborted (6)
[CSIT-LT1087:335444] Signal code: (-6)
[CSIT-LT1087:335444] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7f9320425320]
[CSIT-LT1087:335444] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7f932047eb1c]
[CSIT-LT1087:335444] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7f932042526e]
[CSIT-LT1087:335444] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7f93204088ff]
[CSIT-LT1087:335444] [ 4] /usr/local/lib/libprecice.so.3(+0x438cef)[0x7f9320dbfcef]
[CSIT-LT1087:335444] [ 5] /usr/local/lib/libprecice.so.3(+0x4390e7)[0x7f9320dc00e7]
[CSIT-LT1087:335444] [ 6] /usr/local/lib/libprecice.so.3(+0x43b1f4)[0x7f9320dc21f4]
[CSIT-LT1087:335444] [ 7] /usr/local/lib/libprecice.so.3(+0x3a28f8)[0x7f9320d298f8]
[CSIT-LT1087:335444] [ 8] /usr/local/lib/libprecice.so.3(+0x41aa53)[0x7f9320da1a53]
[CSIT-LT1087:335444] [ 9] /usr/local/lib/libprecice.so.3(+0x404c43)[0x7f9320d8bc43]
[CSIT-LT1087:335444] [10] mbdyn-esm-adapter(+0x9fdf)[0x55f589606fdf]
[CSIT-LT1087:335444] [11] mbdyn-esm-adapter(+0x7028)[0x55f589604028]
[CSIT-LT1087:335444] [12] /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7f932040a1ca]
[CSIT-LT1087:335444] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7f932040a28b]
[CSIT-LT1087:335444] [14] mbdyn-esm-adapter(+0x7105)[0x55f589604105]
[CSIT-LT1087:335444] *** End of error message ***
Hi, I modified the precice-config file as you said and now this is the error message (it stops at the same timestep from the openFOAM part). This is the file with the error (it’s too long to put the text): error.txt (154.8 KB)
Just a little update. I tried to run a simulation with fewer points for the solid mesh to see if a smaller case can solve the problem, but the simulation still explode with the same last error
I forgot to tell something important I think… The 3D simulation is actually a pseudo-2D case, because the z-displacement is always zero… I tell you this beacuse previously you asked me if any data were close to zero…
This is true also for my case. I plan to define the case as 2D and see if anything happens. Besides, I am not sure it is an acceleration-related issue because it looks to me that @leonardo’s case uses explicit coupling. Is that right?
Hi,
I tried defining the case as 2D, setting dimensions="2" in the mesh and z-dead="true" in the mapping. Nothing has changed:
with waveform-degree="0", the simulation crashes on the solid side failing the assert reported here
without waveform-degree="0" the simulation crashes more or less at the same timestep (probably the next timestep) on the fluid side with the error reported here
I have used the preCICE library compiled in optimized debug (i.e. release + debug log + assert).
Claudio