Platform-dependence of implicit (iterative) coupling


Is there any experience how the coupling behavior (number of coupling iterations per time step are needed) depends on platform (network, CPU etc.) and number of threads used? I run the same simulation on two different machines. I get very similar results, but the number of coupling iterations needed and also the residuals in preCICE differ a lot. See

On both machines most of the software (including preCICE) have been installed with Spack. Thus most packages should be compiled with reasonable optimization flags.

I would need a hint which part of my software stack to blame/debug further. Is this behavior that could happen due to preCICE running on a different machine (distributed memory vs. shared memory) or is it rather a problem that my solver itself behaves differently on different machines?

Thanks in advance!

I have seen similar behavior before. The main problem is that the residual of a single iteration as an “observation function” and, thus, also the number of iterations has a very bad condition. Small changes wherever can lead to big changes here.

And small changes you already get when running your solver with a different number of threads. I guess that you could reproduce your behavior also on the same system with the same software stack, but with a different number of parallel threads (as soon as you use any collective operations in your solver). Can you?

Honestly, I would recommend to not look too deeply into that hole. Drives you crazy :see_no_evil:

The good thing, however, is that the average number of iterations is rather independent of this for a sufficient amount of timesteps, especially if you neglect the initial X timesteps.

1 Like

I was/an interested in the coupling behavior, especially in the beginning. That is why I looked into the coupling iterations in the first place, but if this is a bad idea I will try to avoid it.

I will check that computation with a different number of threads and see what happens then.

I ran a simulation over the weekend on the shared memory system with a different number of processes than before. The coupling behavior actually changes just as you mentioned:

I am still a bit surprised. I did not expect that the coupling is this sensitive. For now, I will try to not to look into this too much.

Thanks for the help!