Myself (Sep 10, 2019):
Let’s assume that there are two participants F (fluid) and S (solid) and that they are being coupled via the conventional serial-implicit approach. That is, one solver solves after another in a serial fashion and each coupling time step has >1 coupling iterations.
Let’s also assume that the mappings are to occur in F. In order to enable this, assume that the necessary changes to the configuration (.xml) file are made.
Also assume that solver F is launched with 3 ranks/processes and S is launched with 4 ranks/processes. Let’s use the broadcast/filter approach to do mesh re-partitioning.
In the first coupling iteration of the first coupling time step, the mesh from S is filtered in F since we asked (via the xml file) that the mappings occur in F. As a result of the filtering, each vertex in any rank of F knows which vertex (or vertices) of S it wants data from. Based on this tagging, communication links are established between individual processes of F and S.
Makes all sense with the only exception that the communication of the solid mesh, the mesh repartitioning of the received mesh, and the establishment of the communication channels, all that happens in
initialize(), so already before the “first coupling iteration of the first coupling time step”.
- I understand that the communication links are established just once in the first coupling iteration of the first coupling step. Is that correct?
Yes (but already in initialize)
In subsequent coupling iterations of the first step and all other steps as well, how are the communication links exactly leveraged? Are only data values exchanged through them? For example, if a given mesh vertex V1 in slave rank #2 of F wants to know what displacements to use before F can solve, does the following happen?
- From the tagging information (is this some sort of a weights matrix?), figure out which vertex/vertices of S have non-zero weights associated with them. That is, figure out which of the mesh vertices in S does V1 map to.
- And then, look for them in all the ranks of S that rank#2 of F is connected to.
- Once found, fetch the displacements data from those ranks and use the weights to arrive at the interpolated value of the displacements vector at V1
Yes, only data values are currently exchanged. And yes, you could regard a data mapping method as a sort of weighting. For RBF interpolation this weighting is, however, only implicit, as we solve the interpolation system in every timestep. The communication is however a bit simpler. Once everything is set up, communication and data mapping are two separate things. In your case, data values are communicated via mesh S. Every rank of S sends the values of its vertices to the connected ranks of F. It could be that one vertex of rank 3 of S has copies on multiple ranks of F. Then, the value is sent to all these ranks. Afterwards, every rank of F can do the mapping. For a nearest-neighbor or a nearest-projection mapping, for example, this last step is a complete local operations. It needs no communication. Let’s say, on every rank, v_F = A * v_S with A being some sort of local weighting matrix.
Good to know: we are also working on dynamic meshes. Meaning that in a future release of preCICE, you will be able to also change the mesh in every iteration (to allow dynamic adaptive meshes or mapping between a background mesh and an immersed mesh)