This is just to document an issue I had when running preCICE on a cluster, and how I isolated and resolved it.
The issue had nothing to do with preCICE, but with the way SLURM was set up. However, this took some time to figure out. Once I knew it, it was way easier to open a reasonable ticket for the cluster support. Here’s what happened:
I ran preCICE with two solvers on separate nodes, using SLURM heterogeneous jobs.
Each solver was started using srun.
Problem Description
If (and only if) ntasks >= 2, preCICE failed with this error message:
---[precice] [0m Setting up primary communication to coupling partner/s
(…)
—[precice] [31mERROR: [0m Accepting a socket connection at 10.33.4.124:50757 failed with the system error: Unable to establish connection as a connection file already exists at "./precice-run/Neumann-Dirichlet/7a/0cfd61335a5bb294bead881d5b0052". This is likely a leftover of a previous crash or stop during communication build-up. Please remove the "precice-run" directory and restart the simulation.
Traceback (most recent call last):
File "/home/it4i-valsc/timeadaptiveqnwr/src/dune_heat.py", line 475, in <module>
run_participant(params)
~~~~~~~~~~~~~~~^^^^^^^^
File "/home/it4i-valsc/timeadaptiveqnwr/src/dune_heat.py", line 355, in run_participant
participant.initialize()
~~~~~~~~~~~~~~~~~~~~~~^^
File "cyprecice/cyprecice.pyx", line 119, in cyprecice.Participant.initialize
Removing precice-run and restarting did not fix it.
It seemed that each participant tried to set up multiple conflicting connections, even though only one should be started per node.
Isolating the issue
@fsimonis suggested the following steps:
- Check the preCICE greeting message in the log of the run to ensure that both scripts run as different participants. This is emitted in the constructor of preCICE.
- Test if the MPI launch works correctly, by printing communicator rank and size in your launch scripts for the various participants. If you see repeated rank numbers, then something is very wrong and you have a good example for the system admins to debug the situation.
Indeed the problem was related to SLURM/srun settings on the cluster.
The easiest way to isolate the issue was taking an MPI hello world example (I just used MPI Hello World · MPI Tutorial ), compiling it, then running it with srun --ntasks 2 to see if they end up in the same communicator.
In the end I had to explicitly set the MPI type for things to work, with: srun --mpi=pmix:
$ srun --ntasks 2 -t 00:01:00 mpi_hello_world
srun: job 2039609 queued and waiting for resources
srun: job 2039609 has been allocated resources
No PMIx server was reachable, but a PMI1/2 was detected.
If srun is being used to launch application, 2 singletons will be started.
Hello world from processor cn14, rank 0 out of 1 processors
Hello world from processor cn14, rank 0 out of 1 processors
$ srun --mpi=pmix --ntasks 2 -t 00:01:00 mpi_hello_world
srun: job 2039610 queued and waiting for resources
srun: job 2039610 has been allocated resources
Hello world from processor cn111, rank 0 out of 2 processors
Hello world from processor cn111, rank 1 out of 2 processors
This was due to SLURM_MPI_TYPE not being properly set on the machine.