# Deadlock Using Three-Solver Explicit Coupling Scheme

**URL:** <https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114>\
**Category:** Using preCICE\
**Tags:** communication, configuration\
**Created:** [December 5, 2019, 11:55am UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114 "2019-12-05T11:55:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![patelrohan008](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@patelrohan008](https://precice.discourse.group/u/patelrohan008)\
**Post date:** [December 5, 2019, 11:55am UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/1 "2019-12-05T11:55:48Z")

</div>

Hello all, I have a quick question about setting up a precice config file using three solvers. The goal is to use serial explicit coupling such that Solver A (using some initial values) runs, passes data to Solver B which then runs and passes data to Solver C. Solver C would then pass data back to Solver A, which would run again beginning the next time step and so on and so forth. The issue I’m having is with defining the participant order for each individual bi-coupling scheme. If I define them as: first A second B, then first B second C, then first C second A, for each bi-coupling scheme, precice hangs on the “initialize: slaves are connected” message. I’m assuming that this is because there is no place to start since each solver requires a different one to be run first. Since the parallel explicit approach would not be appropriate as the solvers must run in series, how do I tell precice to begin by running solver A?

xml config file attached.

[preciceConfig.xml](https://precice.discourse.group/uploads/short-url/ewS2NCtb8DRnsqza0bDfMMteJ9g.xml) (3.5 KB)

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [December 6, 2019, 8:35am UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/2 "2019-12-06T08:35:00Z")

</div>

To cite from [Bernhard’s thesis](https://www5.in.tum.de/pub/Gatzhammer2014_preCICE.pdf), page 140

> A careless setup of coupling schemes and participants can lead to a deadlock in the coupled simulation

I am not too sure about the “careless”, these compositional coupling schemes are really complicated (but also quite powerful), but deadlocks are part of the game.

What you currently have:

```xml
      <coupling-scheme:serial-explicit>
         <participants first="SolverA" second="SolverB"/> ...
         <exchange data="Data1" mesh="MeshB" from="SolverA" to="SolverB"/>
      </coupling-scheme:serial-explicit>
    
      <coupling-scheme:serial-explicit>
         <participants first="SolverB" second="SolverC"/> ...
         <exchange data="Data2" mesh="MeshC" from="SolverB" to="SolverC" />
      </coupling-scheme:serial-explicit>
    
      <coupling-scheme:serial-explicit>
         <participants first="SolverC" second="SolverA"/> ...
         <exchange data="Data3" mesh="MeshA" from="SolverC" to="SolverA"/>
      </coupling-scheme:serial-explicit>

```

Your explanation why this leads to a deadlock is correct. To get the behavior that you want, though, i.e. to have a sequential execution (one solver after the other) and an explicit coupling (to directly go to the next timestep after one cycle), the fix is easy: You simply have to swap the participants in the third coupling scheme as `SolverA` needs to run before `SolverC` in every cycle and the exchange happens to next cycle.

```xml
      <coupling-scheme:serial-explicit>
         <participants first="SolverA" second="SolverC"/> ...
         <exchange data="Data3" mesh="MeshA" from="SolverC" to="SolverA"/>
      </coupling-scheme:serial-explicit>

```

Two things to consider when you move forward:

- For three coupled participants, a sequential coupling might be inefficient. Simply changing `serial` to `parallel`, lets all three participants run in parallel. Then, the order `first` vs `second` can neither lead to deadlocks.
- You only use unidirectional coupling schemes, but yet the combination of all three leads to a closed circle. Thus, an explicit scheme might lead to instabilities. If you want to change to an implicit scheme, simply replacing all `explicit` by `implicit` will not do the job. Then, a [fully implicit scheme](https://github.com/precice/precice/wiki/Multi-Coupling-Configuration#fully-implicit-multi-coupling) is needed.

---

<div class="post-metadata">

**Author:** ![patelrohan008](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@patelrohan008](https://precice.discourse.group/u/patelrohan008)\
**Post date:** [December 6, 2019, 1:42pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/3 "2019-12-06T13:42:01Z")

</div>

Thank you for your response.

> [@uekerman](#):
>
> the fix is easy: You simply have to swap the participants in the third coupling scheme as `SolverA` needs to run before `SolverC` in every cycle and the exchange happens to next cycle.

I tried the fix you suggested and the solution still reaches deadlock. ‘SolverA’ works as expected, the maps to ‘SolverB’ which works as expected. However, after ‘SolverB’ is run, and mapping to ‘SolverC’ is attempted the solution hangs.  
In the terminal window for ‘SolverB’ the following message is displayed:  
[impl::SolverInterfaceImpl]:1418 in mapWrittenData: Compute write mapping from mesh “MeshB” to mesh “MeshC”.  
In the terminal window for ‘SolverC’ the following message is displayed:  
[impl::SolverInterfaceImpl]:240 in initialize: Slaves are connected

At this point the solution is deadlocked and does not progress. It is worth noting that parallel coupling was attempted and ran successfully. However, the results were erroneous and exhibited a ‘lag’ between the solvers as each solver needs data referencing the current timestep from the previous solver, for example to accurately solve for time t = 1 SolverB needs data from SolverA for time t = 1 but when using parallel coupling SolverB only has access to SolverA data from the previous timestep, say t = 0.9, when attempting to solve for the current timestep t = 1.

Thanks, Rohan

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [December 6, 2019, 2:08pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/4 "2019-12-06T14:08:13Z")

</div>

Mmh, I guess I need more insight then. Could you please enable some debug output and upload/paste the output of all three solvers?

To do so, you need to build preCICE in `Debug` mode and use the following in your config:

```xml
<log>
    <sink type="stream" output="stdout" filter= "(%Severity% > debug) or (%Severity% >= debug and %Module% contains SolverInterfaceImpl) or (%Severity% >= debug and %Module% contains partition) or (%Severity% >= debug and %Module% contains cplscheme)" enabled="true" />	
</log> 

```

More information on logging [in the wiki](https://github.com/precice/precice/wiki/Logging-Configuration).

---

<div class="post-metadata">

**Author:** ![patelrohan008](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@patelrohan008](https://precice.discourse.group/u/patelrohan008)\
**Post date:** [December 6, 2019, 3:26pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/5 "2019-12-06T15:26:25Z")

</div>

I’ve included the debug output to this message, since the debugging file was run using non-generic solver names/data, I’ve also included another precice config file to assist in translation of the log files.

In addition, I noticed changes in the behavior of precice depending on the ordering of the bi-coupling schemes, for example moving the heat --\> damage bicoupling scheme above the light --\> heat bicoupling scheme enabled precice to complete one full timestep before becoming deadlocked. How does precice interpret the order of bicoupling schemes?  
Thank you very much!

[preciceConfig.xml](https://precice.discourse.group/uploads/short-url/evj22s3kBCfUgN8uHxlnaPfbt54.xml) (4.3 KB) [debugDamageSolver.log](https://precice.discourse.group/uploads/short-url/1q6wa6qUE5qzuEMuszBZoZH0Yp8.log) (3.7 KB) [debugHEAT.log](https://precice.discourse.group/uploads/short-url/AssrECgl0GeQ1ilAkrCoxwE37xE.log) (7.3 KB) [debugLIGHT.log](https://precice.discourse.group/uploads/short-url/bfZx7ITjjq8kLSK7lNwWi3b8bFW.log) (7.5 KB)

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [December 9, 2019, 9:02am UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/6 "2019-12-09T09:02:59Z")

</div>

This case is really tricky. Yes, the order of bi-coupling schemes plays a crucial role. Each participant “initializes” and “advances” the individual coupling schemes in this order. And this should also do the trick here.  
Can you please try:

```xml
      <coupling-scheme:serial-explicit>
         <participants first="LIGHT" second="DamageSolver"/> ...
         <exchange data="Damage" mesh="mcxyz_mesh" from="DamageSolver" to="LIGHT"/>
      </coupling-scheme:serial-explicit>

      <coupling-scheme:serial-explicit>
         <participants first="HEAT" second="DamageSolver"/> ...
         <exchange data="Temperatures" mesh="damageMesh" from="HEAT" to="DamageSolver" />
      </coupling-scheme:serial-explicit>

      <coupling-scheme:serial-explicit>
         <participants first="LIGHT" second="HEAT"/> ...
         <exchange data="VolumetricHeatSources" mesh="heat_mesh" from="LIGHT" to="HEAT"/>
      </coupling-scheme:serial-explicit>

```

Furthermore, please note that in a serial coupling scheme the first participant already receives data in `intialize`. In fact, data that the second participant sends after the first `advance`. This way you get the staggered behavior.  
[This picture](https://github.com/precice/precice/blob/develop/docs/documents/coupling_steering.pdf) could help. A proper documentation of this behavior is on our list.

---

<div class="post-metadata">

**Author:** ![patelrohan008](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@patelrohan008](https://precice.discourse.group/u/patelrohan008)\
**Post date:** [December 9, 2019, 11:49pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/7 "2019-12-09T23:49:44Z")

</div>

Thank you for the link and help about the order of bi-coupling schemes, it was very informative. I tried this configuration of bi-coupling and the solution hung after the “LIGHT” solved, I tried all possible orders of the bi-coupling schemes and the furthest the solution would go was one full iteration before hanging. This occurred when the bi-coupling schemes were in the following order:  
Heat-\>Damage, Light-\>Heat, Light-\>Damage  
Interestingly, the solution now hangs while both LIGHT and DamageSolver are waiting to receive data, I’m not sure what HEAT is attempting to do when the solution is deadlocked. I’m including the debug files for the aforementioned bi-coupling scheme in the hopes that they yield some insight.  
Thank you for your time.  
[debugDamageSolver2.log](https://precice.discourse.group/uploads/short-url/gWVM6MtdtJ4XazE5Z1y5gFaBVh1.log) (7.4 KB) [debugHEAT2.log](https://precice.discourse.group/uploads/short-url/xiCZACN2fik7251bSHt3LdLq0LK.log) (1.7 KB) [debugLIGHT2.log](https://precice.discourse.group/uploads/short-url/wVqDXZyZL641qjdpLuQpWq16SIY.log) (7.5 KB)

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [December 10, 2019, 10:33am UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/8 "2019-12-10T10:33:28Z")

</div>

After further thinking and drawing many pictures, I am confident that your problem has currently indeed no solution 😕. We would need to sort coupling schemes appropriately. I opened an issue:

> <https://github.com/precice/precice/issues/593>
>
> On \[Discourse\](https://precice.discourse.group/t/deadlock-using-three-solver-exp…licit-coupling-scheme/114/6), @patelrohan008 reported a deadlock for a cyclic dependence of three participants when using serial coupling schemes. I guess that this problem could be solved by sorting coupling schemes where the accessing participant is "first" over those where it is "second" in 
> https://github.com/precice/precice/blob/36ebf6983cfba568d9f59308770c732eba724f49/src/cplscheme/config/CouplingSchemeConfiguration.cpp#L312-L314
> 
> I have not tested this assumption, however. Further integration tests would be necessary.
> 
> A more general study of this could also make a nice student thesis.

But let’s step back a bit.

> [@patelrohan008](#):
>
> It is worth noting that parallel coupling was attempted and ran successfully. However, the results were erroneous and exhibited a ‘lag’ between the solvers as each solver needs data referencing the current timestep from the previous solver, for example to accurately solve for time t = 1 SolverB needs data from SolverA for time t = 1 but when using parallel coupling SolverB only has access to SolverA data from the previous timestep, say t = 0.9, when attempting to solve for the current timestep t = 1.

Such a “lag” is a feature of any explicit coupling scheme. Also for three serial-explicit coupling schemes, `Light` would have a lag to `Damage` by one timestep.

Is your problem time dependent? Or are you only interested in a steady-state solution?  
In both cases, a fully-implicit multi coupling scheme could be beneficial.

---

<div class="post-metadata">

**Author:** ![patelrohan008](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@patelrohan008](https://precice.discourse.group/u/patelrohan008)\
**Post date:** [December 10, 2019, 1:28pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/9 "2019-12-10T13:28:09Z")

</div>

Okay, thank you for your continued help on this issue!  
I will absolutely investigate and consider implementation of a fully-implicit multi coupling scheme.

Thanks!

---

<div class="post-metadata">

**Author:** ![fsimonis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/fsimonis/32/6_2.png) [@fsimonis](https://precice.discourse.group/u/fsimonis)\
**Post date:** [November 7, 2022, 4:04pm UTC](https://precice.discourse.group/t/deadlock-using-three-solver-explicit-coupling-scheme/114/10 "2022-11-07T16:04:14Z")

</div>

## Update regarding this problem!

We have a working prototype for these cases and anticipate to release it with version 3.0.