Calculix parallelization

Hi Everybody, I got a question regarding the parallelization of CalculiX. Usually, when I run FSI problem with OpenFoam and CalculiX, OpenFoam side can run in parallel, but not CalculiX side (still run with 1 core). Is there anyway to run CaculiX with multiple cores? I’m asking this because my solid volume mesh is big, therefore the OpenFoam finish one step and take long time to wait for CalculiX calculation. Thanks and regards!

Hi @huyfanne!

CalculiX supports shared memory parallelization, which also works in a coupled setup. Should work out of the box. IIRC there were others already who used this feature for coupled simulations.

Let us know if it works.

Benjamin

May I know what is the correct way to do so? Do I need to recompile CalculiX with Makefile_MT? I’m just starting to use precice, sorry for the inconvenience.

I am not sure, but maybe this helps: http://www.dhondt.de/ccx_2.14.pdf (page 10)

Just to clarify: CalculiX is not part of or related to preCICE, it is a completely separate software project. The best place to get answers on CalculiX-specific questions would be at the CalculiX mailing list.

In any case, if you find the answer, a summary here would be helpful for people with the same question. :slight_smile:

Dear huyfanne,
By default CalculiX is compiled to run on a single core. You should compile it with MakefileMT. Or just take my binaries here:
https://github.com/calculix/free_form_fortran#downloads (both for Windows and Linux).

Thank you for your comments, will study further on the matter

I will try to recompile with CalculiX 2.15. If it doesnt work, will get try your binary (since it is 2.16). Thanks alot.

Please note that our CalculiX adapter now also supports 2.16. To install the adapter, you essentially need to rebuild CalculiX, so you also need the code. This is what happens when you run make, which produces a modified copy of ccx, named ccx_preCICE.

I keep going on this discussion because I encountered some issues related to the parallelization of Calculix (see https://gitter.im/precice/Lobby?source=orgpage). Now I solved them and so I give my solutions.

  1. I got a problem with the parallelization of spooles. I got this kind of message in the Solid.log file:

Using up to 4 cpu(s) for the stress calculation.

Using up to 4 cpu(s) for the symmetric stiffness/mass contributions.

Factoring the system of equations using the symmetric spooles solver
Using up to 1 cpu for spooles.

This means that the resolution of the linear system will not be solved in parallel but one 1 CPU. I solved this problem by compiling only the parallel case (Makefile_MT). This problem seems to arise when you compile the serial case (Makefile) and get ccx_2.XX, and then compile the parallel case and get ccx_2.XX_MT. Using ccx_2.XX_MT does not lead to a parallel run in this case. So, only compile the parallel Calculix version. I am not sure that this occurs for everybody.

  1. The restart with Calculix does not work from version 2.15. Actually, it works only for static calculations but not for dynamic. If you need to run a large case with restart, use a lower version (2.13 is ok for sure).

  2. It is not possible to perform parallel FSI calculations with calculiX because some things are missing in the Makefile in the calculix adapter repository. I succeeded in compiling and making a parallel simulation with the following modifications. I only added -DUSE_MT in the CFLAGS and $(SPOOLES)/MT/src/spoolesMT.a \ in the LIBS.

LIBS = \
        \$(SPOOLES)/MT/src/spoolesMT.a \
        \$(SPOOLES)/spooles.a \
        \$(PKGCONF_LIBS) \
        -lstdc++ \
        -L$(YAML)/build -lyaml-cpp \

CFLAGS = -Wall -O3 -fopenmp $(INCLUDES) -DARCH="Linux" -DSPOOLES -DARPACK -DMATRIXSTORAGE -DUSE_MT

Do not forget the tabulations in LIBS.

1 Like

The following may be of interest too: https://web.mit.edu/calculix_v2.7/CalculiX/ccx_2.7/doc/ccx/node3.html