Hi Everybody, I got a question regarding the parallelization of Fenics. When I run CHT problem with FEniCS and FEniCS, I get the following error message
Hi @fish,
Welcome to the forum!
From the error my best guess is that a processor which does not have a part of the coupling mesh is somehow trying to update data on an Expression which is leading to a sizeless array being passed to scipy.interpolate.Rbf. Ideally this should not happen as the FEniCS adapter checks within the function update_coupling_expression if the particular processor partition has any vertices or not. Lets try to figure out your case better.
Are you running the partitioned-heat or the flow-over-heated-plate tutorial cases as they are? Or have you modified something? Also, how do you execute the parallel FEniCS code?
Hi@ IshaanDesai, I am running the partitioned-heat-complex tutorial cases, and some fluid equation are added in my code. Besides, anything is not changed. The running way of parallel likes: 1、mpirun -n 8 python3 solid.py … 2、mpirun -n 20 python3 fluid.py … And also, the calculate domain is changed 1e-7*1e-7(2D) and have a small cavity on my mesh.
Hi @fish
Adding fluid equations to the existing FEniCS should not break anything as long as you use FEniCS Expression objects to define the equations and also handle boundary conditions.
The small cavity in the mesh can be problematic as we have not tested the parallelization against scenarios where the coupling interface has discontinuities. Is that the case for your cavity geometry?
What happens if you run the code with less processors? Even just 2 processors. Does it still give the same error?
Hi @IshaanDesai , the strange thing is when I run the code with less processors, everything is ok. So I think as you say the small cavity on the interface of mesh which lead to the problem. But I have no idea to solve it.
Okay, the fact that your code runs on less processors is already a good indicator! If you have directly modified the partitioned-heat-conduction case then you probably have these lines in your code right? These lines will generate VTK files which will highlight the mesh partitions done by FEniCS. Using a VTK file viewer like ParaView we can visualize the mesh partition. Can you show how the mesh partitions look like for the case which runs and the case which fails? Looking at the visual output would give us more insight.
Hi@ IshaanDesai, the following is my fluidmesh. I have marked the mesh partitions using different color. Besides, I want to upload the zip file of my mesh, the file too big to upload.
Okay, is this the case which runs or the case which fails? Can you upload mesh partition figures for both? That would give a better idea of what is going wrong.
Hi @IshaanDesai, I add a minimal code example in zip file and a txt file about how to run the example. I am very appreciate for your help!
minimalcode.zip (22.8 KB)
readme(how to run code).txt (135 Bytes)
Hi @fish
I found it difficult to replicate your issue on my local machine. I think it is only possible to replicate this for a high number of processors. Instead I went through your code and I think the most likely cause of error is that somehow you pass an array to the adapter function update_coupling_expression which does not have any entries in it. That means that the second input argument to the function called values is an empty array. Can you check if this is the case?
While working on your case I also realized that we do not check if the input data to update_coupling_expression is empty or not and we need to check this. Feel free to open an issue in the adapter repo regarding this.
@ IshaanDesaiThank you very much! I will check my code and consider to open an issue in the adapter repo regarding this.
1 Like
@fish If the problem is indeed what we predicted then opening an issue is good but also lets try to resolve it here for your particular case. Hence please also update here 
@IshaanDesai ok, no problem. But I have no idea how to check it now! I will do something to remove this problem.
Actually it is quite simple to check it. In your FEniCS code you have a call precice.update_coupling_expression(coupling_expression, read_data). What we want to find out is for the case that you see the above error is the read_data array empty or does it have data.
Basically the error you have encountered is possible when you have a non-empty rank, that is a rank which has some part of the coupling interface, but you try to update the coupling Expression with empty data.
HiIshaanDesai, My problem is how can I get the read_data value and output it. And also how can I get the coupling Expression and output it to check if it is empty.
Hi @fish
Okay I now understand what you are asking. I would suggest just printing the read data to the terminal before calling precice.update_coupling_expression. Also printing the rank number alongside it is a good idea. By printing the array to the terminal you can find out if it has some data or not.
@ IshaanDesai,OK, I will try it. Thank you very much!
1 Like