Is there any experience how the coupling behavior (number of coupling iterations per time step are needed) depends on platform (network, CPU etc.) and number of threads used? I run the same simulation on two different machines. I get very similar results, but the number of coupling iterations needed and also the residuals in preCICE differ a lot. See
On both machines most of the software (including preCICE) have been installed with Spack. Thus most packages should be compiled with reasonable optimization flags.
I would need a hint which part of my software stack to blame/debug further. Is this behavior that could happen due to preCICE running on a different machine (distributed memory vs. shared memory) or is it rather a problem that my solver itself behaves differently on different machines?
I have seen similar behavior before. The main problem is that the residual of a single iteration as an “observation function” and, thus, also the number of iterations has a very bad condition. Small changes wherever can lead to big changes here.
And small changes you already get when running your solver with a different number of threads. I guess that you could reproduce your behavior also on the same system with the same software stack, but with a different number of parallel threads (as soon as you use any collective operations in your solver). Can you?
Honestly, I would recommend to not look too deeply into that hole. Drives you crazy
The good thing, however, is that the average number of iterations is rather independent of this for a sufficient amount of timesteps, especially if you neglect the initial X timesteps.
I was/an interested in the coupling behavior, especially in the beginning. That is why I looked into the coupling iterations in the first place, but if this is a bad idea I will try to avoid it.
I will check that computation with a different number of threads and see what happens then.
I ran a simulation over the weekend on the shared memory system with a different number of processes than before. The coupling behavior actually changes just as you mentioned:
