# Running preCICE on a Cluster: The linear system of the RBF mapping from mesh Solid-Mesh to mesh Fluid-Mesh has not converged

**URL:** <https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708>\
**Category:** Using preCICE\
**Tags:** openfoam, adapters, data-mapping, parallel\
**Created:** [August 31, 2021, 4:36pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708 "2021-08-31T16:36:28Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [August 31, 2021, 4:36pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/1 "2021-08-31T16:36:28Z")

</div>

Hi,  
I’m running a 3D coupled simulation with the OpenFOAM-Adapter.  
Launching the simulation on 7 cores works well on my local machine.  
However executing the exact same script on a cluster leads to:

```bash
(0) 18:07:51 [mapping::PetRadialBasisFctMapping]:785 in map: ERROR: The linear system of the RBF mapping from mesh Solid-Mesh to mesh Fluid-Mesh has not converged. This means most probably that the mapping problem is not well-posed. Please check if your coupling meshes are correct. Maybe you need to fix axis-aligned mapping setups by marking perpendicular axes as dead?

```

When I’m running a 2D case (3D case but axis z marked as dead) with the same configuration, it works on the cluster without a problem.  
I’m running preCICE v2.2.1 and the OpenFOAM-Adapter v1.0.0  
The case is run via the command `mpirun -np 7 pimpleFoam -case openfoam -parallel`.  
The full log file can be found here:  
[pimpleFoam.log](https://precice.discourse.group/uploads/short-url/nFY7dznM8lKc3kQ9VgAuq2ER6R4.log) (30.3 KB)

> **Klick here to view my preCICE-config.xml**
>
> ```xml
> <?xml version="1.0" encoding="UTF-8" ?>
> <precice-configuration>
> 
> <solver-interface dimensions="3">
> <data:vector name="Force_Data" />
> <data:vector name="Displacement_Data" />
> 
> <mesh name="Fluid-Mesh">
> <use-data name="Force_Data" />
> <use-data name="Displacement_Data" />
> </mesh>
>     
> <mesh name="Solid-Mesh">
> <use-data name="Displacement_Data" />
> <use-data name="Force_Data" />
> </mesh>
> 
> <participant name="Fluid">
> <use-mesh name="Fluid-Mesh" provide="yes" />
> <use-mesh name="Solid-Mesh" from="Solid" />
> <write-data name="Force_Data" mesh="Fluid-Mesh" />
> <read-data name="Displacement_Data" mesh="Fluid-Mesh" />
> <!--<mapping:rbf-thin-plate-splines direction="write" from="Fluid-Mesh" to="Solid-Mesh" constraint="conservative" z-dead="true"/>
> <mapping:rbf-thin-plate-splines direction="read" from="Solid-Mesh" to="Fluid-Mesh" constraint="consistent" z-dead="true"/>-->
> <mapping:rbf-compact-polynomial-c0 support-radius="5" direction="write" from="Fluid-Mesh" to="Solid-Mesh" constraint="conservative"/>
> <mapping:rbf-compact-polynomial-c0 support-radius="5" direction="read" from="Solid-Mesh" to="Fluid-Mesh" constraint="consistent"/>
> </participant>
> 
> <participant name="Solid">
> <use-mesh name="Solid-Mesh" provide="yes" />
> <write-data name="Displacement_Data" mesh="Solid-Mesh" />
> <read-data name="Force_Data" mesh="Solid-Mesh" />
> </participant>
> 
> <m2n:sockets from="Fluid" to="Solid" exchange-directory="." enforce-gather-scatter="0"/>
> 
> <coupling-scheme:parallel-explicit>
> <time-window-size value="0.0001" />
> <max-time value="0.5" />
> <participants first="Fluid" second="Solid" />
> <exchange data="Force_Data" mesh="Solid-Mesh" from="Fluid" to="Solid" />
> <exchange data="Displacement_Data" mesh="Solid-Mesh" from="Solid" to="Fluid" />
> </coupling-scheme:parallel-explicit>
> </solver-interface>
> </precice-configuration>
> 
> ```

Maybe somebody has a clue what is going on 🙂

---

<div class="post-metadata">

**Author:** ![DavidSCN](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@DavidSCN](https://precice.discourse.group/u/DavidSCN)\
**Post date:** [September 2, 2021, 8:38pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/2 "2021-09-02T20:38:03Z")

</div>

Hi @JulianSchl,

ok so for my understanding again. The overall case is a quasi 2D case, i.e. you have preCICE dimension 3, but only physics in 2D. The simulation works on your local machine without any issues, but on the cluster, you need to mark axis as dead in order to make it work?!

Although the error occurs here in your case on the Fluid side, the ‘crucial’ mesh is here the solid mesh (`from` mesh in case of a read mapping). Which solid solver do you use and is the solid mesh the same as well?

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [September 6, 2021, 7:40am UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/3 "2021-09-06T07:40:15Z")

</div>

If the mapping problem is ill-posed or close to ill-posed, it can happen that one PETSc implementation works and another not. We had similar issues in:

> <https://github.com/precice/tutorials/issues/112>
>
> We are currently defining a few tutorial cases (more prominently the \[FSI/flap\_p…erp\](https://github.com/precice/tutorials/tree/master/FSI/flap\_perp)) in the following way:
> \- the solver meshes are 3D, providing a fake additional layer in one direction, and
> \- the preCICE solver interface is defined as 3D (in the \`precice-config.xml\`).
> 
> After a series of cases that the RBF mapping fails on different systems (see https://github.com/precice/calculix-adapter/issues/37, https://github.com/precice/calculix-adapter/issues/43), @uekerman suggested that these cases are ill-defined and it is a coincidence that they succeed in some systems.
> 
> The reason that we were using 3D solver interfaces until now was that some adapters could not handle 2D meshes & data. Since https://github.com/precice/openfoam-adapter/pull/96, the OpenFOAM adapter can automatically work in a "2D coupling mode" and a nice example case is \[deal.II-OpenFOAM (2D variant)\](https://github.com/precice/tutorials/tree/master/FSI/flap\_perp\_2D/OpenFOAM-deal.II).
> 
> If we had the same functionality for CalculiX, FEniCS, and SU2, we could restructure the pseudo-2D cases to actually use 2D solver interfaces, even if the solvers themselves keep working in 3D internally. Maybe @KyleDavisSA and @IshaanDesai know more on how to implement this in these adapters.

Solution is always to make your mapping problem well-defined (setting axis to dead) or properly go to 2D.

---

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [September 13, 2021, 8:50am UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/4 "2021-09-13T08:50:31Z")

</div>

Hi @DavidSCN and @uekerman,

thank you for your messages and sorry for my late reply!

* * *

> [@DavidSCN](#):
>
> ok so for my understanding again. The overall case is a quasi 2D case, i.e. you have preCICE dimension 3, but only physics in 2D.

Maybe I havent expressed myself that well. The case is a 3D case coupling a 3D airfoil with FSI.

> **view picture of the mesh**
>
> ![airfoil3D](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/6cc08f184428ece5a3e32e9991328c45f7d6754a.png)

  

* * *

> [@DavidSCN](#):
>
> The simulation works on your local machine without any issues, but on the cluster, you need to mark axis as dead in order to make it work?!

Yes marking the z-axis as dead is a way to make the simulation run on the cluster, but would physically lead to wrong results, since it is a 3D case.

* * *

> [@DavidSCN](#):
>
> Which solid solver do you use and is the solid mesh the same as well?

I’m using MBDyn as solid solver and I’m using a simplified mesh version of the fluid mesh.

  

* * *

> [@uekerman](#):
>
> Solution is always to make your mapping problem well-defined (setting axis to dead) or properly go to 2D.

As mentioned, I’m running a 3D case. Is there another way to solve such a problem?

---

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [September 17, 2021, 7:59am UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/5 "2021-09-17T07:59:31Z")

</div>

To visualize the mapping a bit better, I made some screenshots:

> **solid solver mesh**
>
> ![solid_mesh1](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/be585ac787dfb409c1daea0477cdb94416992f9d.png)

> **solid solver mesh (rotated view)**
>
> ![solid_mesh2](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/237e527b553f17ca19990ed860147af2ee6f6168.png)

> **fluid solver patch**
>
> ![fluid_mesh1](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/031706ed6caaf3b02b11b2734a35d839b4e09bc8.png)

> **fluid solver patch (rotated view)**
>
> ![fluid_mesh2](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/1f3c81656b430a0447edc9a655843e230069a373.png)

Note: the airfoil patch in the fluid solver is closed on one side as you can see in the image. In the solid mesh it is opened on both sides. I was expecting this is not a problem, when mapping these meshes. What do you think?

But still I have no clue why the case is not running on the cluster, but on my machine. If you think it could be a PETSc issue: On my local machine, I’ve installed the preCICE Ubuntu package and on the cluster I built preCICE from source (including PETSc). Can I deactivate features on the cluster to acheive a similar environment as on my machine?

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [September 21, 2021, 5:14pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/6 "2021-09-21T17:14:50Z")

</div>

Sry for the late reply.

You use a support radius of 5. How does this value relate to say the span-wise width of your airfoil?  
It could be that your mapping system did not diverge, but neither converge. Good to know: Our next release v2.3 will give more information for such a case.

Just to be sure: both your meshes mesh the same geometry. Meaning, when you plot both at the same time they lie nearly over each other, right?

> [@JulianSchl](#):
>
> Note: the airfoil patch in the fluid solver is closed on one side as you can see in the image. In the solid mesh it is opened on both sides. I was expecting this is not a problem, when mapping these meshes. What do you think?

This could indeed be a problem when you have a fluid mesh partition which is only on this side. But actually, I would expect a different error message then. Sth coming from the re-partitioning.  
Can you visualize your fluid partitions? Any chance to cut this lid?

To get a bit more information: what happens when you run with a different number of ranks?

---

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [September 22, 2021, 10:00am UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/7 "2021-09-22T10:00:54Z")

</div>

Hi 👋

> [@uekerman](#):
>
> You use a support radius of 5.

I selected the support radius just for testing purposes (I thought this value is conservative enough). The airfoil has the following dimensions:  
chord length: 0.144 m  
maximum thickness: 0.017 m  
span length: 0.15 m

> **see screenshot**
>
> ![sizes](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/57a6665066335702b10201d55db252885d8cf87b.png)

Because I’m not exactly sure how to calculate the support radius, I’ve opened a seperate [topic](https://precice.discourse.group/t/support-radius-for-rbf-different-than-gaussian/728).

> [@uekerman](#):
>
> Just to be sure: both your meshes mesh the same geometry. Meaning, when you plot both at the same time they lie nearly over each other, right?

It should be … I was trying to visualize it with paraview using the export function of preCICE, but with the RBF mapping I’m getting many different files, which don’t show the expected mesh when selecting the glyph filter. Is there a way to plot such meshes with preCICE?

Still the mapping works on the local machine for \> 2000 timewindows …

> [@uekerman](#):
>
> Can you visualize your fluid partitions?

You mean you want to see how the case is decomposed?

> [@uekerman](#):
>
> Any chance to cut this lid?

I don’t understand what you mean by that …

> [@uekerman](#):
>
> what happens when you run with a different number of ranks?

I tried it with different number of processors … 1 … 8 … 35,  
but still the same issue.

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [September 22, 2021, 3:18pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/8 "2021-09-22T15:18:59Z")

</div>

Your support radius is definitely too large. You get a dense mapping matrix and GMRES might have problems. Try 3-7 times your mesh width.

> [@JulianSchl](#):
>
> It should be … I was trying to visualize it with paraview using the export function of preCICE, but with the RBF mapping I’m getting many different files, which don’t show the expected mesh when selecting the glyph filter. Is there a way to plot such meshes with preCICE?

You can use the [vtk export of preCICE](https://precice.org/configuration-export.html). Is this what you already do? Why do you get many files?

> [@JulianSchl](#):
>
> > [@uekerman](#):
> >
> > Any chance to cut this lid?
> 
> I don’t understand what you mean by that …

I meant to mesh really the same geometry with fluid and solid. Not to have this additional “closure”. I was referring to:

> [@JulianSchl](#):
>
> Note: the airfoil patch in the fluid solver is closed on one side as you can see in the image. In the solid mesh it is opened on both sides. I was expecting this is not a problem, when mapping these meshes. What do you think?

---

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [September 23, 2021, 12:44pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/9 "2021-09-23T12:44:46Z")

</div>

> [@uekerman](#):
>
> Your support radius is definitely too large. You get a dense mapping matrix and GMRES might have problems. Try 3-7 times your mesh width.

I now ran different radiuses (0.5, 0.8, 1) with the same issue.

> [@uekerman](#):
>
> You can use the [vtk export of preCICE](https://precice.org/configuration-export.html). Is this what you already do? Why do you get many files?

Maybe I selected the wrong output files, as there are many when using the RBF mapping.  
The following files are available (only the fluid participant has time to create its files):

> **view list of files**
>
> ```
> Fluid-Mesh-Fluid.dt1_master.pvtu Solid-Mesh-Fluid.dt1_r1.vtu
> Fluid-Mesh-Fluid.dt1_r0.vtu Solid-Mesh-Fluid.dt1_r2.vtu
> Fluid-Mesh-Fluid.dt1_r1.vtu Solid-Mesh-Fluid.dt1_r3.vtu
> Fluid-Mesh-Fluid.dt1_r2.vtu Solid-Mesh-Fluid.dt1_r4.vtu
> Fluid-Mesh-Fluid.dt1_r3.vtu Solid-Mesh-Fluid.dt1_r5.vtu
> Fluid-Mesh-Fluid.dt1_r4.vtu Solid-Mesh-Fluid.dt1_r6.vtu
> Fluid-Mesh-Fluid.dt1_r5.vtu Solid-Mesh-Fluid.init_master.pvtu
> Fluid-Mesh-Fluid.dt1_r6.vtu Solid-Mesh-Fluid.init_r0.vtu
> Fluid-Mesh-Fluid.init_master.pvtu Solid-Mesh-Fluid.init_r1.vtu
> Fluid-Mesh-Fluid.init_r0.vtu Solid-Mesh-Fluid.init_r2.vtu
> Fluid-Mesh-Fluid.init_r1.vtu Solid-Mesh-Fluid.init_r3.vtu
> Fluid-Mesh-Fluid.init_r2.vtu Solid-Mesh-Fluid.init_r4.vtu
> Fluid-Mesh-Fluid.init_r3.vtu Solid-Mesh-Fluid.init_r5.vtu
> Fluid-Mesh-Fluid.init_r4.vtu Solid-Mesh-Fluid.init_r6.vtu
> Fluid-Mesh-Fluid.init_r5.vtu Solid-Mesh-Solid.dt1.vtk
> Fluid-Mesh-Fluid.init_r6.vtu Solid-Mesh-Solid.dt2.vtk
> Solid-Mesh-Fluid.dt1_master.pvtu Solid-Mesh-Solid.final.vtk
> Solid-Mesh-Fluid.dt1_r0.vtu
> 
> ```

When I try to view `Fluid-Mesh-Fluid.init_master.pvtu` or `Fluid-Mesh-Fluid.dt1_master.pvtu`, paraview doesn’t show anything (scale in glyph is automatically very small and increasing it doesn’t help).  
Now `Fluid-Mesh-Fluid.init_r*.vtu` looks like that:

> **view screenshots**
>
> Blue: Fluid mesh (`Fluid-Mesh-Fluid.init_r*.vtu`)  
> Red: Solid mesh (`Solid-Mesh-Fluid.init_master.pvtu`)
> 
> ![screenshot1](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/36e884b7c579319de6a916927d74a02322cdb486.png)  
> ![screenshot2](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/a32b6074c666155e70f32e38172b0655f585e5a2.png)  
> ![screenshot3](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/71f837d420f59dd10330f9f7c613d746d0604175.png)

> [@uekerman](#):
>
> I meant to mesh really the same geometry with fluid and solid. Not to have this additional “closure”. I was referring to:

At the end it should be closed at the side … and as you know it works on my local machine ^^  
Also since I didn’t create the fluid solver case this would be quite some work for a dummy case,  
so maybe first try some other things before …

  
I tried the visualization of the same case on my machine:

> **view screenshots**
>
> ![screenshot1](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/e758a9bd7cc1630ea6c190c12383affa8b4f01f1.jpeg)  
> ![screenshot2](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/c9d80c08806a9e55e51d2de6d994a5cb29d44dd0.png)  
> ![screenshot3](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/c170f57fd1d74efa047aba9b326c5f135365fa60.png)

In my eyes the mesh looks good, what do you think?

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [September 23, 2021, 1:04pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/10 "2021-09-23T13:04:36Z")

</div>

> [@JulianSchl](#):
>
> I now ran different radiuses (0.5, 0.8, 1) with the same issue.

I would try to go even lower. Your span has a width of 0.15 and you use roughly 10 elements there. 0.015 \* 5 = 0.07. (your solid mesh is the one to consider when you use a consistent mapping from solid to fluid)

Does it work when you use

```xml
use-qr-decomposition="yes"

```

in your RBF mappings?

Meshes look good. I still don’t like the “lid”, but like I said, I would actually expect a different error there.

---

<div class="post-metadata">

**Author:** ![JulianSchl](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/julianschl/32/692_2.png) [@JulianSchl](https://precice.discourse.group/u/JulianSchl)\
**Post date:** [September 23, 2021, 1:47pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/11 "2021-09-23T13:47:44Z")

</div>

With 0.07 the simulation started 🥳  
I will keep you updated how long it runs 👍  
So I didn’t needed

> [@uekerman](#):
>
> `use-qr-decomposition="yes"`

So a big thank you and I will keep you updated 😀

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/67d2a5fa6f80e597e2267ff17fa8c1673cd11caa.svg) [@system](https://precice.discourse.group/u/system)\
**Post date:** [October 8, 2021, 1:39pm UTC](https://precice.discourse.group/t/running-precice-on-a-cluster-the-linear-system-of-the-rbf-mapping-from-mesh-solid-mesh-to-mesh-fluid-mesh-has-not-converged/708/12 "2021-10-08T13:39:00Z")

</div>

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.