# Parallel computing with OpenFOAM **URL:** \ **Category:** Official adapters and tutorials\ **Tags:** openfoam\ **Created:** [December 11, 2019, 4:37pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120 "2019-12-11T16:37:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jun\_leng](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@jun\_leng](https://precice.discourse.group/u/jun_leng)\ **Post date:** [December 11, 2019, 4:37pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/1 "2019-12-11T16:37:53Z")
Hi, everyone, I created a vector `A` in my OpenFOAM solver which is independent with fluid mesh and needs to be sent to the structure solver by call `writeBlockVectorData` and read `BlockVectorData` (`constraints = conservative`). The fluid field is divided into 28 subfields and OpenFOAM uses `mpirun -np 28 pimpleFOAM -parallel` to perform parallel calculations. The structure solver only uses one CPU for computation. ``` ... ... ``` I met a problem that the structure solver receives nearly `28*A` while the fluid solver sent `A` to it. Does this means that preCICE receives `A` for 28 times in one time step and they are added together and sent to the structure solver after mapping? Best, Jun ---
**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\ **Post date:** [December 11, 2019, 5:16pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/2 "2019-12-11T17:16:04Z")
Hi @jun_leng! > I created a vector `A` in my OpenFOAM solver which is independent with fluid mesh What do you mean exactly by this? Does every rank of OpenFOAM only define a part of the vector, or is the same vector with the same elements available in every rank? I assume that you are not using the “official” adapter (or at least you modify it). How do you assign this vector to the interface mesh? How does your `m2n` node in the precice-config look like? Are you using `distribution-type="gather-scatter"`? (see the [wiki page on communication configuration](https://github.com/precice/precice/wiki/Communication-Configuration)) Also, which participant does the mapping? Posting your complete `precice-config.xml` would help. ---
**Author:** ![jun\_leng](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@jun\_leng](https://precice.discourse.group/u/jun_leng)\ **Post date:** [December 12, 2019, 1:57am UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/3 "2019-12-12T01:57:15Z")
Hi, @Makis, Actually, the vector A (’’`Forces`’’ in `precice-config.xml`) is calculated by calling a function I added in PimpleFoam solver. The function will use the displacements and structural meshes calculated from the structure solver to get vector A (similar to forces) which will be used in the structural solver and some other coefficients which is used in the Navier-Stokes equations at every time step. It doesn’t need the information of fluid mesh during coupling and is not related to parallel computation of PimpleFoam. ![image](https://global.discourse-cdn.com/free1/uploads/precice/original/1X/30046204884da759829eae4ac55032cab5142d27.png) I added the part of the adapter into PimpleSolver’s codes directly and it runs well if I don’t use parallel computation. When I run PimpleFoam parallel computation without using ``, however, it says, `(0) 17:21:52 [impl::SolverInterfaceImpl]:133 in configure: ERROR: A parallel participant needs either a master or a server communication configured`. When I run PimpleFoam parallel computation using ``, the value of the vector `A` increase with the number of ranks. I want to know what ‘mpi-single’ is used for and if `mpi-single` will copy data for n times before communicating with structure solver (n is the number of ranks). Additionally, I am using `distribution-type="gather-scatter" `. Are there any documents available for introducing its theory and implementation? Attached is my `precice-config.xml`.[precice-config.xml](https://precice.discourse.group/uploads/short-url/7SVskpyOa0QLfn4nfr3HypnFfcn.xml) (1.9 KB) Best, Jun ---
**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\ **Post date:** [December 12, 2019, 1:20pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/4 "2019-12-12T13:20:35Z")
> [@jun\_leng](#): > > I want to know what ‘mpi-single’ is used for and if `mpi-single` will copy data for n times before communicating with structure solver (n is the number of ranks). Additionally, I am using `distribution-type="gather-scatter" `. Are there any documents available for introducing its theory and implementation? There are two steps where communication is needed: - between the two participants (fluid and solid), which we name “m2n” (M ranks of Fluid to N ranks of Solid). This is the inter-communication. - inside each parallel participant. This is the intra-communication. From your config file: ```xml ``` As the XML node suggests, the `gather-scatter` is part of the m2n communication (with sockets). This means that, with this setting, the `FluidSolver` is gathering all the meshes in one rank, sending them to `StructureSolver`, which can then scatter them to other ranks (here: none). In a scenario where both participants are parallel, the default setting is to setup direct communication channels among all the ranks that need to communicate, but for serial solvers we currently need this workaround (which we try to replace by an automatic solution). Let’s look also at the intra-communication: ```xml ... ``` Here `master` means that one of the `FluidSolver` ranks will synchronize the rest/“slave” processes (e.g. announce when has everybody finished communicating and when is the simulation ready to continue). _Side-note: in the early days, preCICE also had a “server mode”, just like other coupling tools. That server would then be the orchestrator._ The `mpi` means that MPI is used to send these additional messages among participants and `single` means that a single MPI communicator (the one that is already set up by the solver) will be used. You can see other alternatives in the [XML reference](https://github.com/precice/precice/wiki/XML-Reference). More information about these you can find in: - [Communication Configuration](https://github.com/precice/precice/wiki/Communication-Configuration) - [Literature guide](https://github.com/precice/precice/wiki/Literature-guide#communication) About your original question, I am still a bit confused on what exactly you are doing (seeing the source would help). The best way to debug it would be to use the [export to vtk feature](https://github.com/precice/precice/wiki/Exports-Configuration) and visualize the meshes in ParaView to see if problem is introduced in the solver side or during the mapping. The configuration file looks correct to me. ---
**Author:** ![jun\_leng](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@jun\_leng](https://precice.discourse.group/u/jun_leng)\ **Post date:** [January 2, 2020, 3:44pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/5 "2020-01-02T15:44:39Z")
Hi, @Makis! I am sorry for my late reply. I finally found the reason for my problem after I followed your advice and checked my vtk files. The vector that will be sent to the structure solver is the same vector in every rank of OpenFOAM. Therefore, the vector became N times bigger than it was in OpenFOAM when I used `type = " gather-scatter` in `m2n` part (N is the number of ranks of OpenFOAM). To solve the problem, I divided the vector by N in the structure solver and the result turned right. But I don’t think it is a good solution to this problem. Do you have any suggestions? Thanks! ---
**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\ **Post date:** [January 4, 2020, 1:34pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/6 "2020-01-04T13:34:01Z")
Hi @jun_leng What is your vector A exactly? Why do you compute a copy of it in every OpenFOAM rank? It is sth like a force? Then, a conservative mapping (as you use) does exactly what you describe – it’s trying to conserve the sum. Two simple alternatives: - Only compute vector A on one rank on OpenFOAM. That’s probably the cleanest solution. - Use a consistent mapping instead. Then, you have to move the mapping in the preCICE config to the other participants, as explained [in the wiki](https://github.com/precice/precice/wiki/Mapping-Configuration#restrictions-for-parallel-participants). ---
**Author:** ![jun\_leng](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@jun\_leng](https://precice.discourse.group/u/jun_leng)\ **Post date:** [January 4, 2020, 2:27pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/7 "2020-01-04T14:27:22Z")
Hi, @uekerman Thank you for your help. Actually, vector A is the lift and drag forces that will be applied on turbines. It is not projected from the fluid mesh directly but calculated from formulas based on the Actuator Line Method, which is faster and widely used in the field of turbines. The velocity of fluid mesh is needed in the formulas. This model combines a three-dimensional Navier-Stokes solver with a technique in which body forces are distributed radially along lines representing the blades of the wind turbine. More information on ALM can be found in [ALM](https://onlinelibrary.wiley.com/doi/full/10.1002/we.1747?casa_token=a7jGujoSpdgAAAAA:p1QCuVSqP_jMIIPmfUrGIShSvCA2kYu7oC57MRxr4M-CZhw-h7ljUoDZ6p1U8cUdh4-AdrmWJ8dxCH2x). preCICE is quite a useful and strong tool for FSI simulation and it really helps my research a lot. Thank you! I will try it again. ---
**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\ **Post date:** [January 4, 2020, 3:19pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/8 "2020-01-04T15:19:40Z")
So, these forces are actually not mesh based, correct? Then, the best approach for the moment would be to only write them on one rank, like mentioned above, and define the same auxiliary mesh on the solid side. Maybe interesting to know: we have on our road map to also support non-mesh-associated data: [Handling global coupling data (not associated to a mesh) · Issue #202 · precice/precice · GitHub](https://github.com/precice/precice/issues/202) If the forces are mesh-based (and I got the concept wrong), then I would expect them to also be partitioned as the mesh is. > [@jun\_leng](#): > > preCICE is quite a useful and strong tool for FSI simulation and it really helps my research a lot. Thank you! I will try it again. Great! 😃 Probably a good occasion to mention that we are always looking for more [testimonials](https://www.precice.org/testimonials/). ---
**Author:** ![jun\_leng](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@jun\_leng](https://precice.discourse.group/u/jun_leng)\ **Post date:** [January 4, 2020, 3:59pm UTC](https://precice.discourse.group/t/parallel-computing-with-openfoam/120/9 "2020-01-04T15:59:13Z")
OK, I will have a try! 😀 > [@uekerman](#): > > So, these forces are actually not mesh based, correct? Then, the best approach for the moment would be to only write them on one rank, like mentioned above, and define the same auxiliary mesh on the solid side.