# Calculix parallelization

**URL:** <https://precice.discourse.group/t/calculix-parallelization/167>\
**Category:** Official adapters and tutorials\
**Tags:** openfoam, calculix, fsi\
**Created:** [March 2, 2020, 4:10pm UTC](https://precice.discourse.group/t/calculix-parallelization/167 "2020-03-02T16:10:44Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![huyfanne](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/huyfanne/32/129_2.png) [@huyfanne](https://precice.discourse.group/u/huyfanne)\
**Post date:** [March 2, 2020, 4:10pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/1 "2020-03-02T16:10:44Z")

</div>

Hi Everybody, I got a question regarding the parallelization of CalculiX. Usually, when I run FSI problem with OpenFoam and CalculiX, OpenFoam side can run in parallel, but not CalculiX side (still run with 1 core). Is there anyway to run CaculiX with multiple cores? I’m asking this because my solid volume mesh is big, therefore the OpenFoam finish one step and take long time to wait for CalculiX calculation. Thanks and regards!

---

<div class="post-metadata">

**Author:** ![uekerman](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/uekerman/32/2199_2.png) [@uekerman](https://precice.discourse.group/u/uekerman)\
**Post date:** [March 2, 2020, 4:39pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/2 "2020-03-02T16:39:19Z")

</div>

Hi @huyfanne!

CalculiX supports shared memory parallelization, which also works in a coupled setup. Should work out of the box. IIRC there were others already who used this feature for coupled simulations.

Let us know if it works.

Benjamin

---

<div class="post-metadata">

**Author:** ![huyfanne](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/huyfanne/32/129_2.png) [@huyfanne](https://precice.discourse.group/u/huyfanne)\
**Post date:** [March 2, 2020, 4:59pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/3 "2020-03-02T16:59:36Z")

</div>

May I know what is the correct way to do so? Do I need to recompile CalculiX with Makefile\_MT? I’m just starting to use precice, sorry for the inconvenience.

---

<div class="post-metadata">

**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\
**Post date:** [March 2, 2020, 5:46pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/4 "2020-03-02T17:46:32Z")

</div>

I am not sure, but maybe this helps: [http://www.dhondt.de/ccx\_2.14.pdf](http://www.dhondt.de/ccx_2.14.pdf) (page 10)

Just to clarify: CalculiX is not part of or related to preCICE, it is a completely separate software project. The best place to get answers on CalculiX-specific questions would be at the [CalculiX mailing list](http://groups.yahoo.com/group/calculix).

In any case, if you find the answer, a summary here would be helpful for people with the same question. 🙂

---

<div class="post-metadata">

**Author:** ![imirzov](https://avatars.discourse-cdn.com/v4/letter/i/7ab992/32.png) [@imirzov](https://precice.discourse.group/u/imirzov)\
**Post date:** [March 2, 2020, 7:17pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/5 "2020-03-02T19:17:34Z")

</div>

Dear huyfanne,  
By default CalculiX is compiled to run on a single core. You should compile it with MakefileMT. Or just take my binaries here:  
[https://github.com/calculix/free\_form\_fortran#downloads](https://github.com/calculix/free_form_fortran#downloads) (both for Windows and Linux).

---

<div class="post-metadata">

**Author:** ![huyfanne](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/huyfanne/32/129_2.png) [@huyfanne](https://precice.discourse.group/u/huyfanne)\
**Post date:** [March 3, 2020, 1:25am UTC](https://precice.discourse.group/t/calculix-parallelization/167/6 "2020-03-03T01:25:00Z")

</div>

Thank you for your comments, will study further on the matter

---

<div class="post-metadata">

**Author:** ![huyfanne](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/huyfanne/32/129_2.png) [@huyfanne](https://precice.discourse.group/u/huyfanne)\
**Post date:** [March 3, 2020, 1:26am UTC](https://precice.discourse.group/t/calculix-parallelization/167/7 "2020-03-03T01:26:43Z")

</div>

I will try to recompile with CalculiX 2.15. If it doesnt work, will get try your binary (since it is 2.16). Thanks alot.

---

<div class="post-metadata">

**Author:** ![Makis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/makis/32/25_2.png) [@Makis](https://precice.discourse.group/u/Makis)\
**Post date:** [March 3, 2020, 8:21am UTC](https://precice.discourse.group/t/calculix-parallelization/167/8 "2020-03-03T08:21:06Z")

</div>

Please note that our CalculiX adapter now [also supports 2.16](https://github.com/precice/calculix-adapter/tree/v2.16). To install the adapter, you essentially need to rebuild CalculiX, so you also need the code. This is what happens when you run `make`, which produces a modified copy of `ccx`, named `ccx_preCICE`.

---

<div class="post-metadata">

**Author:** ![fsalmon](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/fsalmon/32/279_2.png) [@fsalmon](https://precice.discourse.group/u/fsalmon)\
**Post date:** [March 10, 2020, 3:46pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/9 "2020-03-10T15:46:43Z")

</div>

I keep going on this discussion because I encountered some issues related to the parallelization of Calculix (see [https://gitter.im/precice/Lobby?source=orgpage](https://gitter.im/precice/Lobby?source=orgpage)). Now I solved them and so I give my solutions.

1. I got a problem with the parallelization of spooles. I got this kind of message in the Solid.log file:

> Using up to 4 cpu(s) for the stress calculation.
> 
> Using up to 4 cpu(s) for the symmetric stiffness/mass contributions.
> 
> Factoring the system of equations using the symmetric spooles solver  
> Using up to 1 cpu for spooles.

This means that the resolution of the linear system will not be solved in parallel but one 1 CPU. I solved this problem by compiling only the parallel case (Makefile\_MT). This problem seems to arise when you compile the serial case (Makefile) and get ccx\_2.XX, and then compile the parallel case and get ccx\_2.XX\_MT. Using ccx\_2.XX\_MT does not lead to a parallel run in this case. So, only compile the parallel Calculix version. I am not sure that this occurs for everybody.

1. The restart with Calculix does not work from version 2.15. Actually, it works only for static calculations but not for dynamic. If you need to run a large case with restart, use a lower version (2.13 is ok for sure).

2. It is not possible to perform parallel FSI calculations with calculiX because some things are missing in the Makefile in the calculix adapter repository. I succeeded in compiling and making a parallel simulation with the following modifications. I only added `-DUSE_MT` in the `CFLAGS` and `$(SPOOLES)/MT/src/spoolesMT.a \` in the `LIBS`.

```Makefile
LIBS = \
        \$(SPOOLES)/MT/src/spoolesMT.a \
        \$(SPOOLES)/spooles.a \
        \$(PKGCONF_LIBS) \
        -lstdc++ \
        -L$(YAML)/build -lyaml-cpp \

CFLAGS = -Wall -O3 -fopenmp $(INCLUDES) -DARCH="Linux" -DSPOOLES -DARPACK -DMATRIXSTORAGE -DUSE_MT

```

Do not forget the tabulations in LIBS.

---

<div class="post-metadata">

**Author:** ![fsimonis](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/fsimonis/32/6_2.png) [@fsimonis](https://precice.discourse.group/u/fsimonis)\
**Post date:** [March 16, 2020, 7:10pm UTC](https://precice.discourse.group/t/calculix-parallelization/167/10 "2020-03-16T19:10:56Z")

</div>

The following may be of interest too: [https://web.mit.edu/calculix\_v2.7/CalculiX/ccx\_2.7/doc/ccx/node3.html](https://web.mit.edu/calculix_v2.7/CalculiX/ccx_2.7/doc/ccx/node3.html)

---

<div class="post-metadata">

**Author:** ![EMahravan](https://avatars.discourse-cdn.com/v4/letter/e/50afbb/32.png) [@EMahravan](https://precice.discourse.group/u/EMahravan)\
**Post date:** [August 13, 2020, 8:32am UTC](https://precice.discourse.group/t/calculix-parallelization/167/11 "2020-08-13T08:32:53Z")

</div>

I had the same problem. I followed this for calculix and it worked fine (also you can find more detail in calculix README.INSTALL):  
[https://www.libremechanics.com/?q=node/9](https://www.libremechanics.com/?q=node/9)

Eventhough, this doesn’t work directly for precice-calculix-adapter makefile, and it needs a few more modifications. I put my final Makefile here, hope it helps:

```
# See our wiki for getting the CalculiX dependencies:
# https://github.com/precice/calculix-adapter/wiki/Installation-instructions-for-CalculiX
# Set the following variables before building:
# Path to original CalculiX source (e.g. $(HOME)/ccx_2.15/src )
CCX = $(HOME)/App/Calculix/ccx_2.15/src
# Path to SPOOLES main directory (e.g. $(HOME)/SPOOLES.2.2 )
SPOOLES = $(HOME)/App/Calculix/spooles
# Path to ARPACK main directory (e.g. $(HOME)/ARPACK )
ARPACK = $(HOME)/App/Calculix/ARPACK
# Path to yaml-cpp prefix (e.g. $(HOME)/yaml-cpp, should contain "include" and "build")
ARPACK = $(HOME)/App/Calculix/ARPACK

# Get the CFLAGS and LIBS from pkg-config (preCICE version >= 1.4.0).
# If pkg-config cannot find the libprecice.pc meta-file, you may need to set the
# path where this is stored into PKG_CONFIG_PATH when building the adapter.
PKGCONF_CFLAGS = $(shell pkg-config --cflags libprecice)
PKGCONF_LIBS = $(shell pkg-config --libs libprecice)

# Specify where to store the generated .o files
OBJDIR = bin

# Includes and libs
INCLUDES = \
	-I./ \
	-I./adapter \
	-I$(CCX) \
	-I$(SPOOLES) \
	-I$(SPOOLES)/MT/src \
	$(PKGCONF_CFLAGS) \
	-I$(ARPACK) \
	-I$(YAML)/include

LIBS = \
    $(SPOOLES)/MT/src/spoolesMT.a \
	$(SPOOLES)/spooles.a \
	$(PKGCONF_LIBS) \
	-lstdc++ \
	-L$(YAML)/build -lyaml-cpp \

# OS-specific options
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Darwin)
	LIBS += $(ARPACK)/libarpack_MAC.a
else
	LIBS += $(ARPACK)/libarpack_ubuntu.a
	LIBS += -lpthread -lm -lc
endif

# Compilers and flags
#CFLAGS = -g -Wall -std=c++11 -O0 -fopenmp $(INCLUDES) -DARCH="Linux" -DSPOOLES -DARPACK -DMATRIXSTORAGE
#FFLAGS = -g -Wall -O0 -fopenmp $(INCLUDES)

CFLAGS = -Wall -O3 -fopenmp $(INCLUDES) -DARCH="Linux" -DSPOOLES -DARPACK -DMATRIXSTORAGE -DUSE_MT=1 

# OS-specific options
CC = cc

FFLAGS = -Wall -O3 -fopenmp $(INCLUDES)
FC = gfortran
# FC = mpif90
# FC = gfortran

# Include a list of all the source files
include $(CCX)/Makefile.inc
SCCXMAIN = ccx_2.15.c

# Append additional sources
SCCXC += nonlingeo_precice.c CCXHelpers.c PreciceInterface.c
SCCXF += getflux.f getkdeltatemp.f

# Source files in this folder and in the adapter directory
$(OBJDIR)/%.o : %.c
	$(CC) $(CFLAGS) -I$(SPOOLES) -c $< -o $@
$(OBJDIR)/%.o : %.f
	$(FC) $(FFLAGS) -I$(SPOOLES) -c $< -o $@
$(OBJDIR)/%.o : adapter/%.c
	$(CC) $(CFLAGS) -I$(SPOOLES) -c $< -o $@
$(OBJDIR)/%.o : adapter/%.cpp
	g++ -std=c++11 -I$(SPOOLES) -I$(YAML)/include -c $< -o $@ $(LIBS)
	#$(CC) $(CFLAGS) $(INCLUDES) -c $< -o $@ $(LIBS)

# Source files in the $(CCX) folder
$(OBJDIR)/%.o : $(CCX)/%.c
	$(CC) $(CFLAGS) -c $< -o $@
$(OBJDIR)/%.o : $(CCX)/%.f
	$(FC) $(FFLAGS) -c $< -o $@

# Generate list of object files from the source files, prepend $(OBJDIR)
OCCXF = $(SCCXF:%.f=$(OBJDIR)/%.o)
OCCXC = $(SCCXC:%.c=$(OBJDIR)/%.o)
OCCXMAIN = $(SCCXMAIN:%.c=$(OBJDIR)/%.o)
OCCXC += $(OBJDIR)/ConfigReader.o

$(OBJDIR)/ccx_preCICE: $(OBJDIR) $(OCCXMAIN) $(OBJDIR)/ccx_2.15.a $(SPOOLES)/MT/src/spoolesMT.a 
	$(FC) -fopenmp -Wall -O3 -o $@ $(OCCXMAIN) $(OBJDIR)/ccx_2.15.a $(LIBS)

$(OBJDIR)/ccx_2.15.a: $(OCCXF) $(OCCXC) 
	ar vr $@ $?

$(OBJDIR):
	mkdir -p $(OBJDIR)

clean:
	rm -f $(OBJDIR)/*.o $(OBJDIR)/ccx_2.15.a $(OBJDIR)/ccx_preCICE
```

---

<div class="post-metadata">

**Author:** ![Rahul\_dev\_meher](https://yyz2.discourse-cdn.com/free1/user_avatar/precice.discourse.group/rahul_dev_meher/32/604_2.png) [@Rahul\_dev\_meher](https://precice.discourse.group/u/Rahul_dev_meher)\
**Post date:** [April 19, 2021, 9:05am UTC](https://precice.discourse.group/t/calculix-parallelization/167/12 "2021-04-19T09:05:52Z")

</div>

Hey @fsalmon ,

Can you please provide a step by step guide for enabling ccx\_preCICE to run in parallel ?