# Forrtl:severe (174) occurred during simulation **URL:** \ **Category:** Using preCICE\ **Tags:** fsi\ **Created:** [March 23, 2022, 2:47pm UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010 "2022-03-23T14:47:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [March 23, 2022, 2:47pm UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/1 "2022-03-23T14:47:25Z")
Hi all, I was doing a FSI simulation using OpenFOAM and an in-house solver with preCICE while a segmentation fault occurred suddenly (after running for several hours): > **forrtl: severe (174)** > > forrtl: severe (174): SIGSEGV, segmentation fault occurred > Image PC Routine Line Source > FSSICAS20220322 000000000054C3AA for\_\_signal\_handl Unknown Unknown > libpthread-2.31.s 000014798E5543C0 Unknown Unknown Unknown > libprecice.so.2.3 000014798E8A4B12 \_ZN7precice3com15 Unknown Unknown > libprecice.so.2.3 000014798E8A5553 Unknown Unknown Unknown > libprecice.so.2.3 000014798E8A66C2 Unknown Unknown Unknown > libprecice.so.2.3 000014798E89AA4D \_ZN5boost4asio6de Unknown Unknown > libprecice.so.2.3 000014798E8840A8 Unknown Unknown Unknown > libstdc++.so.6.0. 000014798948BDE4 Unknown Unknown Unknown > libpthread-2.31.s 000014798E548609 Unknown Unknown Unknown > libc-2.31.so 000014798E46D293 clone Unknown Unknown I searched for this problem through related topics(most of them are related to Calculix adapter’s memory leak) and found that the key points were stack size and memory leak. To get rid of the limitation of stack size, I often used `ulimit -s unlimited` before starting the simulation. As for memory leak, I adopted gnome-system-monitor to see whether the memory kept growing. However, the memory used by our in-house solver was 2.3 GiB and it didn’t increase during the simulation. The total memory of our server is 503.4 GiB while the memory used during the simulatioin is around 23 GiB. So it might not be the reason why the segmentation fault occurred. I also tried to use valgrind to do memory check but I don’t really know the message it provided… [valgrind.log](https://precice.discourse.group/uploads/short-url/9EbFtZAw3sPWrzdDKfk1Sp6ilAG.log) (1.1 MB) Moreover, both fluid(OpenFOAM) and solid part (in-house solver) converged well. The log of OpenFOAM is too large to upload. The log of solid part and config file are uploaded. [FSSICAS.log](https://precice.discourse.group/uploads/short-url/aOIWtZeJkg90VKqlBWfNaNStCP8.log) (1.4 MB) [precice-config.xml](https://precice.discourse.group/uploads/short-url/vJiAYq9eXjNOnvGNMBAprB1nijL.xml) (2.6 KB) What else may lead to this problem? Any hints will be appreciated! ---
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [March 28, 2022, 2:49am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/2 "2022-03-28T02:49:48Z")
My colleague and I are currently identifying whether the problem is caused by our in-house solver or preCICE. The error message is shown in the following picture: ![20220327_55w流体网格_1e7固体出错](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/d/d24a56a1c41a3fde315ed1213bd9c44f42cd8d66.jpeg) If the seg fault is caused by preCICE, there will be some prefix like `--[precice] ERROR`, right? ---
**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\ **Post date:** [March 28, 2022, 8:29am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/3 "2022-03-28T08:29:02Z")
It is very difficult to understand what is going on here, due to the many parts marked as “unknown” in the trace. It looks like the segfault is essentially triggered inside preCICE and it is related to the communication (I see `boost asio`, `precice com`, and `libpthread`). But this could very well be caused by external factors. I guess we could only get more information by rebuilding preCICE in debug mode, but I also guess you don’t want to repeat hours of simulations. Since it is triggered just by time, it could also be some communication buffer access issue (maybe @fsimonis could have an idea here, then). Looking at `valgrind.log`, I get the impression that the simulation does not even get past initialization, which confuses me. Does it correspond to the same problem/situation? This pattern appears near the end (probably written by every process): ```nohighlight ==28628== 2,752 bytes in 1 blocks are definitely lost in loss record 54 of 54 ==28628== at 0xE2507F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==28628== by 0x1CE89702: opal_free_list_grow_st (in /usr/lib/x86_64-linux-gnu/openmpi/lib/libopen-pal.so.40.20.3) ==28628== by 0x3C3D0CC4: ??? ==28628== by 0x1CED79C8: mca_btl_base_select (in /usr/lib/x86_64-linux-gnu/openmpi/lib/libopen-pal.so.40.20.3) ==28628== by 0x3C396527: ??? ==28628== by 0x1B2AD70A: mca_bml_base_init (in /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.40.20.3) ==28628== by 0x1B2ED714: ompi_mpi_init (in /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.40.20.3) ==28628== by 0x1B2910B0: PMPI_Init (in /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.40.20.3) ==28628== by 0x175E2D2D: precice::utils::Parallel::initializeMPI(int*, char***) (in /usr/lib/x86_64-linux-gnu/libprecice.so.2.3.0) ==28628== by 0x175E2DCD: precice::utils::Parallel::initializeManagedMPI(int*, char***) (in /usr/lib/x86_64-linux-gnu/libprecice.so.2.3.0) ==28628== by 0x175AEA08: precice::impl::SolverInterfaceImpl::configure(std::__cxx11::basic_string, std::allocator > const&) (in /usr/lib/x86_64-linux-gnu/libprecice.so.2.3.0) ==28628== by 0x175AF79F: precice::impl::SolverInterfaceImpl::SolverInterfaceImpl(std:: __cxx11::basic_string, std::allocator >, std::__ cxx11::basic_string, std::allocator > const&, int, int, void*) (in /usr/lib/x86_64-linux-gnu/libprecice.so.2.3.0) ``` One side comment, looking at your config file: why are you exchanging velocity on the face nodes? Is this really the velocity of the solid motion, or what? Is it the velocity from the `FF` module, or did you extend the `FSI` module of the adapter? I also noticed in the `FSISICAS.log` the part ` FSSI-CAS-2D_Soil Model For Windows OS`. Is this really running on Windows? ---
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [March 28, 2022, 9:44am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/4 "2022-03-28T09:44:45Z")
Hi Makis, Thanks for your reply! When we compile our solver in linux using intel fortran, no error or warning occur. Besides, we’ve checked both fluid and solid results when the segfault occurred. Actually, both results were fine. I also don’t know why the simulation stops at once while using valgrind to debug. So at present, the best way to figure out this problem is rebuilding preCICE in debug mode and run the simulation again to see more information, right? > [@Makis](#): > > why are you exchanging velocity on the face nodes? Is this really the velocity of the solid motion, or what? Firstly, as we are simulating interaction between wave, soil and structure, there is a seepage velocity at the interface calculated by our solid solver which needs to be passed to fluid domain. The seepage velocity is calculated by using the pressure obtained from OpenFOAM and this velocity is stored on cell nodes (variables are stored on cell nodes in our FEM solver). > [@Makis](#): > > Is it the velocity from the `FF` module, or did you extend the `FSI` module of the adapter? Yes, I extended the FSI module of the adapter (an older version) according to [this topic](https://precice.discourse.group/t/how-to-transfer-pressure-instead-of-force-in-fsi/688/8). Right now, we pass `Pressure` from fluid to solid on face centers, and we also pass `Displacement` and `Velocity` from solid to fluid on face nodes. The modified adapter is uploaded here. Could you please point it out if there is anything wrong in my modification? [openfoam-adapter-OpenFOAM8.zip](https://precice.discourse.group/uploads/short-url/mdbEgczOjfVTjCU9xQmMdhkNC5Z.zip) (1.5 MB) > [@Makis](#): > > I also noticed in the `FSISICAS.log` the part ` FSSI-CAS-2D_Soil Model For Windows OS` . Is this really running on Windows? Since we are currently developing the two-way coupling for our solver in linux, we haven’t changed the name of our solver. It is now running on Ubuntu Mate 20.04. ---
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [March 29, 2022, 12:48pm UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/5 "2022-03-29T12:48:04Z")
Hi, @Makis Since our server has some networking issues, I didn’t succeed in recompiling the preCICE source code in debug mode. Therefore, I decided to recompile preCICE in debug mode on my PC and reran the case to collect information. Right now, the Configuration is Debug in the following picture. Does this mean that the preCICE library is connected well and it’s running in debug mode? (The terminal doesn’t output more message than before) Is there anything needed to collect more information when error happens? ![debug mode](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/0/0316f749c079178e3b135272cad119493f032466.png) ---
**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\ **Post date:** [March 30, 2022, 7:22am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/6 "2022-03-30T07:22:15Z")
It is compiled in Debug mode, as you can also see by the `---[precice] Configuration: Debug`. You have also enabled more information in the `precice-config.xml` (I believe), since you now receive more information. It does not look to me like the initialization has been completed, but maybe it already proceeded in the meantime. ---
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [April 2, 2022, 9:39am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/7 "2022-04-02T09:39:14Z")
Firstly, thanks for your help! In previous tests on our server(Kunlun 9016), we found the timing when such segfault happened has randomness. For an entirely same case, this segfault 174 happened at different times (please see the following pictures). ![4线程不输出_报错](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/1/18d9deeca0d3effbf7294b03d21caf91ab8ce141.jpeg) 16.209s ![有输出时出错](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/8/8d38706c7288df1c7f519b4845c8365d376148be.jpeg) 0.469s My colleague and I have checked our solver carefully during these days. Our solver has an interface to do one-way coupling in both window and linux platforms. Its stability has been tested by many cases. When we run our solver to do one-way coupling with openfoam, no segfault happens no matter in window or linux. Therefore, we assume the problem is data delivery. So, I downloaded the latest version of OF adapter and modified it to meet our need. I rebuilt preCICE in debug mode on another small server and ran a simulation with less fluid mesh. The simulation has been running for days and the segfault has not happened yet. If the segfault happens again, I’ll post the information here to discuss. 😀 ---
**Author:** ![DavidSCN](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@DavidSCN](https://precice.discourse.group/u/DavidSCN)\ **Post date:** [April 25, 2022, 12:36pm UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/8 "2022-04-25T12:36:57Z")
As far as I can judge, this issue was now fixed in [Undefined behavior in communication · Issue #1243 · precice/precice · GitHub](https://github.com/precice/precice/issues/1243) ---
**Author:** ![Stan](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/stan/32/599_2.png) [@Stan](https://precice.discourse.group/u/Stan)\ **Post date:** [April 27, 2022, 2:16am UTC](https://precice.discourse.group/t/forrtl-severe-174-occurred-during-simulation/1010/9 "2022-04-27T02:16:37Z")
Hi David, It’s quite exciting to hear that this problem was fixed! After the previous case was done, such segfault happened once again for another simulation(also running in debug mode). ![segfault_part1](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/4/48ac55576f154b56ee65dbb50cc2cad36069d94f.jpeg) ![segfault_part2](https://global.discourse-cdn.com/free1/uploads/precice/original/2X/d/d00bcf3ddde1b21d7bbdfcca83d91436b00c946d.png) I’ll modify the source code as suggested in the topic and recompile precice again to check whether the problem still remains.