preCICE- finalize MPI issue in last synchronization window

Using preCICE
1

I’m relatively new to preCICE and the community, and I’d like to ask for advice on an internal coupling setup using our in-house solver MESHFREE (written in Fortran).
Both coupled participants use MESHFREE, and I’m doing a volume–volume fluid–fluid coupling between air and water.

Setup:

  • Both participants: MESHFREE (same solver, two instances: “air” and “water”)
  • Coupling type: volume–volume coupling (air - water)
  • Implementation: Fortran module bindings of preCICE (since MESHFREE is written in Fortran)

The problem appears in the last time synchronization window, when both solvers are finishing:

  1. The air solver reaches the end time Tend first.

  2. As soon as it reaches Tend, the air side calls precicef_finalize and after that MPI_Finalize on the MESHFREE side, which closes the communication.

  3. As a result, the water side hangs, because it still expects data in the last synchronization window but the MPI channels are already closed on the air side.

To avoid this, I changed the behavior so that the MPI communication channels on the air side are only closed when both solvers have reached Tend.

#Normal termination routine- Logic 

      call precicef_finalize()  # Each solver calls this
 
if (precice_L_SecondTermination) then  #G.var-->True, only when called 2nd time
         !Free up memory before exit
         call mffree(precice_vertexIDs)
         call mffree(precice_vertices)
         call mffree(precice_writeData)
         call mffree(precice_readData)
         ! Close the MPI-ranks only when the second participant also has a normal termination
         call SYS_exit(EXIT_SUCCESS)  #Call MPI_Finalize() 
else
    precice_L_SecondTermination = .true.  #First solver, No MPI_Finalize() 

endif

With this change,

  • Both participants terminate normally.

  • The logs from MESHFREE and preCICE show no abnormal termination (see attached files).

Question:

Has anyone experienced a similar issue when using internal coupling (both participants inside the same code base) especially in the last synchronization window ?

Any feedback or suggestions would be very welcome.

2

output_air.txt (2.3 MB)

precice-config.xml (2.3 KB)

output_water.txt (3.1 MB)

I am not sure, if the post already shared the log files, I could not find them in the post. I attach them again as a reply.

3

Hi @cysanghavi,

this is a very important detail. In general, preCICE is not meant to be used like this, but it is something we could eventually add (see Support multiple Participants in a single process · Issue #655 · precice/precice · GitHub - the referenced issue in there sounds similar to yours).

We also have an open issue specifically for the C and Fortran bindings:

How do you currently call preCICE for the two phases?

4

@Makis : Thank you for the links, I went through both issues, but I believe my case is different. In my setup, the two participants (Air and Water) are launched as separate MPI jobs from the same root directory:

How do you currently call preCICE for the two phases ?

# Air participant (4 MPI ranks)

mpirun -np 4 $MESHFREE_EXEC ...   # working directory: root_folder/Air

# Water participant (1 MPI rank)

mpirun -np 1 $MESHFREE_EXEC ...   # working directory: root_folder/Water
  • Debugging findings

I did a little bit debugging and it seems to me that I never actually enter the second loop.

if (precice_L_SecondTermination) then
    mpi_write_1(*,'(2a)') 'preCICE: Second call --> Participant Name: ', PRECICE_ParticipantName
else
    mpi_write_1(*,'(2a)') 'preCICE: First call --> Participant Name: ', PRECICE_ParticipantName
    precice_L_SecondTermination = .true.
endif

The log never outputs "preCICE: Second call", which suggests that the second branch of the termination logic is never reached. This indicates that the execution flow does not return to the coupling loop after the first precicef_finalize() call.

  • Question regarding Finalize part:

Does precicef_finalize() only release the MPI communicators used internally by preCICE (i.e., the inter-participant communicators between Air and Water ranks), or does it also finalize the intra-participant MPI communicators belonging to the MESHFREE solver world?

Do preCICE buffers and internal data structures (e.g., associated with precice_writeData, precice_readData, etc.) get automatically deallocated by call precicef_finalize() ?

Or is it the responsibility of the solver to explicitly free these arrays before or after calling precicef_finalize()

I think the precice_writeData might still be required ?

5

I did a fluid-fluid coupling between two separate instances of my solver a few years ago and did not run into this problem. It is just a matter of where you put your preCICE API calls in your code, as you discovered:

To avoid this, I changed the behavior so that the MPI communication channels on the air side are only closed when both solvers have reached Tend.

I presume your solver has some kind of loop that exits after Tend is reached. The preCICE finalization should occur outside that loop, and then the MPI finalization should happen once preCICE is finalized.

6

Alright, then I understood the “internal coupling (both participants inside the same code base)” differently. We have many tutorials where we use the same code acting as different participants, but starting as different processes (e.g., all OpenFOAM partitioned flow examples, or even the solver dummies).

Good. I am not really sure what the idea here was (since we are not in the other interpretation of “internal coupling”), but it should be enough to have one call to precicef_finalize(), followed by one call to all the termination routines.

See:

You don’t need to modify the MPI_Finalize call of your code.

What you allocate in the adapter code, you also have to deallocate. You explicitly allocate the arrays that you pass to the read/write data, so you also need to explicitly deallocate them. preCICE has no control over them.

Why?

7

Thanks @Makis and @Ray_Scarr for such detailed replies.

Indeed, as you pointed out, the precicef_finalize should be outside of the timeloops. I now have the following steps in our code.

  • call precicef_finalize ,
  • free up memory
  • call MPI_Finalize()

This is now integrated into our CI- tests and works robustly.

See:

Perfect thanks, this is really helpful.

I was of the impression that thecall precice_writedata() from Solver A might be needed my Solver B to perform the mapping in the last time window.

  • I think if I ensure that the call precicef_advance happens before call precicef_finalize then the Solver B already has the interpolated field in its buffers, so then this is seal-proof.

Alright, then I understood the “internal coupling (both participants inside the same code base)” differently. We have many tutorials where we use the same code acting as different participants, but starting as different processes (e.g., all OpenFOAM partitioned flow examples, or even the solver dummies).

Sorry, I did not use the word “internal” correctly.

This is integrated as a CI-job with proper checks and works robustly for us.

8

Yes, this is how it is supposed to be.

So, problem solved? :smiley:

9

Yes, thanks a lot, :slight_smile: .