# ASPECT hangs at random spots when using more than one node **URL:** \ **Category:** ASPECT\ **Created:** [March 14, 2023, 6:24pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861 "2023-03-14T18:24:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rbrtmch](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/rbrtmch/32/697_2.png) [@rbrtmch](https://community.geodynamics.org/u/rbrtmch)\ **Post date:** [March 14, 2023, 6:24pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/1 "2023-03-14T18:24:39Z")
Hello all, ## I rectenly installed ASPECT on a new institutional cluster. ## – This is ASPECT, the Advanced Solver for Problems in Earth’s ConvecTion. – . version 2.5.0-pre (main, 4c20fe0) – . using deal.II 9.4.2 – . with 32 bit indices and vectorization level 1 (128 bits) – . using Trilinos 13.2.0 – . using p4est 2.3.2 – . running in OPTIMIZED mode – . running with 128 MPI processes The codes (including cookbooks) run well on a single node from 16 to 56 cores, but as soon as I use more than one node, ASPECT randomly hangs at different time steps, regardless of adaptive gridding or not. Both OPTIMIZED and DEBUG versions exhibit the same behavior. For example, it hangs at: Number of mesh deformation degrees of freedom: 451128 \*\*\* Timestep 56: t=2.79614e+06 years, dt=46144.5 years Solving mesh displacement system… 13 iterations. or at Number of mesh deformation degrees of freedom: 451128 \*\*\* Timestep 13: t=650000 years, dt=50000 years Solving mesh displacement system… 13 iterations. Solving temperature system… 15 iterations. Solving noninitial\_plastic\_strain system … 20 iterations. Solving plastic\_strain system … 19 iterations. Solving crust\_upper system … 16 iterations. Solving crust\_lower system … 17 iterations. Solving mantle\_lithosphere system … 15 iterations. Solving asthenosphere system … 15 iterations. or at Number of mesh deformation degrees of freedom: 87039 Solving mesh displacement system… 0 iterations. \*\*\* Timestep 0: t=0 years, dt=0 years Solving mesh displacement system… 0 iterations. Solving temperature system… 0 iterations. Skipping noninitial\_plastic\_strain composition solve because RHS is zero. Solving plastic\_strain system … 0 iterations. Solving crust\_upper system … 0 iterations. Solving crust\_lower system … 0 iterations. Solving mantle\_lithosphere system … 0 iterations. Solving asthenosphere system … 0 iterations. Rebuilding Stokes preconditioner… Solving Stokes system… 57+0 iterations. Relative nonlinear residual (Stokes system) after nonlinear iteration 1: 1 Rebuilding Stokes preconditioner… Solving Stokes system… 64+0 iterations. Relative nonlinear residual (Stokes system) after nonlinear iteration 2: 0.0816698 Rebuilding Stokes preconditioner… The cluster uses openmpi, 3.1.4 built with gnu 8.3. I do make sure to run on physical cores only and OMP\_NUM\_THREADS=1. I also tried with exclusive nodes to no avail. Any ideas? Thanks, Rob ---
**Author:** ![jbnaliboff](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jbnaliboff](https://community.geodynamics.org/u/jbnaliboff)\ **Post date:** [March 14, 2023, 6:43pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/2 "2023-03-14T18:43:46Z")
Hi Robert, Uh oh ☹ When we have seen these types of nasty random errors in the past, it is usually related to some obscure issue with the MPI version, node interconnect hardware, and/or file server. Do you know if the file server is compatible with MPI-IO (i.e., is it parallel file server that uses lustre or BGFS)? My first thought is that there are newer versions of openmpi available, and in one past similar case upgrading to a newer version resolved a similar issue. My recollection is that had to with something related to openmpi version compatibility with the infiniband hardware, but it could be a number of different things in this case. Others will likely chime in as well. Hopefully there is a quick solution, but this may take a bit of time to pinpoint the specific issue. Cheers, John ---
**Author:** ![tjhei](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/tjhei/32/1163_2.png) [@tjhei](https://community.geodynamics.org/u/tjhei)\ **Post date:** [March 14, 2023, 8:35pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/3 "2023-03-14T20:35:04Z")
From the output it looks like there is not one particular place that causes the hang (solver or postprocessing), so this might be difficult to diagnose. A few things to try: - Does it happen if you run a small example in debug mode (convection box or something) - Does your system have more than one file system? If yes, try to put the output folder on a different filesystem to test - If you are running in debug mode you can, at least on most clusters, connect to one of the compute nodes using ssh and generate a call stack. That would tell us where things are hanging. ---
**Author:** ![rbrtmch](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/rbrtmch/32/697_2.png) [@rbrtmch](https://community.geodynamics.org/u/rbrtmch)\ **Post date:** [March 17, 2023, 4:48pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/4 "2023-03-17T16:48:04Z")
Hi Timo and John, - Does it happen if you run a small example in debug mode (convection box or something) It does not appear to happen with convection-box, ran it distributed in different configurations multiple times without issues. - Does your system have more than one file system? If yes, try to put the output folder on a different filesystem to test Our cluster only uses NFS4.1 I’ll try with openmpi4 to see if that resolves the issue. Or would you suggest MPICH since we are using NFS? Thanks, Rob ---
**Author:** ![tjhei](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/tjhei/32/1163_2.png) [@tjhei](https://community.geodynamics.org/u/tjhei)\ **Post date:** [March 17, 2023, 5:33pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/5 "2023-03-17T17:33:29Z")
One potential issues might be parallel MPI I/O. You can disable it by setting “Number of grouped files = 0”, see [Postprocess — ASPECT 2.5.0-pre](https://aspect-documentation.readthedocs.io/en/latest/parameters/Postprocess.html#parameters-postprocess-visualization-number-20of-20grouped-20files) I have no preference for OpenMPI vs MPICH. Try different things and see if it helps… ---
**Author:** ![jbnaliboff](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jbnaliboff](https://community.geodynamics.org/u/jbnaliboff)\ **Post date:** [March 17, 2023, 6:14pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/6 "2023-03-17T18:14:33Z")
> Our cluster only uses NFS4.1 > One potential issues might be parallel MPI I/O. You can disable it by setting “Number of grouped files = 0”, see [Postprocess — ASPECT 2.5.0-pre](https://aspect-documentation.readthedocs.io/en/latest/parameters/Postprocess.html#parameters-postprocess-visualization-number-20of-20grouped-20files) Yep, this is the issue a number of us encountered on a cluster that had NFS and a version of MPI (openmpi) that did not include MPI I/0. The solution that finally resolved the random segmentation faults was using a version openmpi with MPI I/0, but this is not standard procedure (i.e., not recommended) and can lead to other issues. I agree with Timo that the best solution would be to disable ASPECT using MPI I/O via `Number of grouped files = 0` Cheers, John ---
**Author:** ![rbrtmch](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/rbrtmch/32/697_2.png) [@rbrtmch](https://community.geodynamics.org/u/rbrtmch)\ **Post date:** [March 18, 2023, 12:11am UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/7 "2023-03-18T00:11:10Z")
Setting Number of grouped files = 0 did not resolve the issue. Here is the call stack for the stuck process (this is outside of my wheelhouse): (gdb) where #0 0x00007f5eb8afb3c6 in mca\_btl\_vader\_component\_progress () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/openmpi/mca\_btl\_vader.so #1 0x00007f5ecc02206c in opal\_progress () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libopen-pal.so.40 #2 0x00007f5ecc0287d5 in ompi\_sync\_wait\_mt () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libopen-pal.so.40 #3 0x00007f5ece8975b9 in ompi\_request\_default\_wait () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #4 0x00007f5ece8ec8e3 in ompi\_coll\_base\_sendrecv\_actual () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #5 0x00007f5ece8eccbc in ompi\_coll\_base\_allreduce\_intra\_recursivedoubling () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #6 0x00007f5ece8aded6 in PMPI\_Allreduce () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #7 0x00007f5edbce2c19 in Epetra\_MpiComm::MaxAll (this=, PartialMaxs=, GlobalMaxs=, Count=) at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_MpiComm.cpp:161 #8 0x00007f5edbc205aa in Epetra\_CrsGraph::ComputeIndexState (this=this@entry=0x8f5f2f0) at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_BlockMap.h:770 #9 0x00007f5edbc251dd in Epetra\_CrsGraph::MakeIndicesLocal(Epetra\_BlockMap const&, Epetra\_BlockMap const&) () at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_CrsGraph.cpp:1774 #10 0x00007f5edbc25a16 in Epetra\_CrsGraph::FillComplete(Epetra\_BlockMap const&, Epetra\_BlockMap const&) () at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_CrsGraph.cpp:979 #11 0x00007f5efced86ac in dealii::TrilinosWrappers::SparsityPattern::compress (this=0xa6d5cc0) at /home/rmoucha/bin/tmp/unpack/deal.II-v9.4.2/source/lac/trilinos\_sparsity\_pattern.cc:738 #12 0x00007f5efca458bf in dealii::BlockSparsityPatternBasedealii::TrilinosWrappers::SparsityPattern::compress (this=0x7fff4c610b00) at /home/rmoucha/bin/tmp/unpack/deal.II-v9.4.2/source/lac/block\_sparsity\_pattern.cc:172 #13 0x00000000043768cf in aspect::Simulator\<3\>::setup\_system\_matrix (this=0x7fff4c611700, system\_partitioning=…) at /home/rmoucha/opt/aspect/source/simulator/core.cc:1143 #14 0x0000000004373723 in aspect::Simulator\<3\>::start\_timestep (this=0x7fff4c611700) at /home/rmoucha/opt/aspect/source/simulator/core.cc:633 #15 0x0000000004370d18 in aspect::Simulator\<3\>::run (this=0x7fff4c611700) at /home/rmoucha/opt/aspect/source/simulator/core.cc:2030 #16 0x0000000002b55f23 in run\_simulator\<3\> (raw\_input\_as\_string=…, input\_as\_string=…, output\_xml=false, output\_plugin\_graph=false, validate\_only=false) at /home/rmoucha/opt/aspect/source/main.cc:592 #17 0x0000000002b2f63f in main (argc=2, argv=0x7fff4c615918) at /home/rmoucha/opt/aspect/source/main.cc:784 Here is a different hang point: #0 0x00007fb61a19ef0e in pthread\_mutex\_unlock () from /lib64/libpthread.so.0 #1 0x00007fb604d3529d in ompi\_coll\_libnbc\_progress () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/openmpi/mca\_coll\_libnbc.so #2 0x00007fb61956c06c in opal\_progress () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libopen-pal.so.40 #3 0x00007fb6195727d5 in ompi\_sync\_wait\_mt () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libopen-pal.so.40 #4 0x00007fb61bde15b9 in ompi\_request\_default\_wait () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #5 0x00007fb61be368e3 in ompi\_coll\_base\_sendrecv\_actual () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #6 0x00007fb61be36cbc in ompi\_coll\_base\_allreduce\_intra\_recursivedoubling () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #7 0x00007fb61bdf7ed6 in PMPI\_Allreduce () from /opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib/libmpi.so.40 #8 0x00007fb62922cc19 in Epetra\_MpiComm::MaxAll (this=, PartialMaxs=, GlobalMaxs=, Count=) at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_MpiComm.cpp:161 #9 0x00007fb62916a5aa in Epetra\_CrsGraph::ComputeIndexState (this=this@entry=0x1a445970) at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_BlockMap.h:770 #10 0x00007fb62916f1dd in Epetra\_CrsGraph::MakeIndicesLocal(Epetra\_BlockMap const&, Epetra\_BlockMap const&) () at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_CrsGraph.cpp:1774 #11 0x00007fb62916fa16 in Epetra\_CrsGraph::FillComplete(Epetra\_BlockMap const&, Epetra\_BlockMap const&) () at /home/rmoucha/bin/tmp/unpack/Trilinos-trilinos-release-13-2-0/packages/epetra/src/Epetra\_CrsGraph.cpp:979 #12 0x00007fb64a4226ac in dealii::TrilinosWrappers::SparsityPattern::compress (this=0x10409f60) at /home/rmoucha/bin/tmp/unpack/deal.II-v9.4.2/source/lac/trilinos\_sparsity\_pattern.cc:738 #13 0x00007fb649f8f8bf in dealii::BlockSparsityPatternBasedealii::TrilinosWrappers::SparsityPattern::compress ( this=0x7ffd2e4471d0) at /home/rmoucha/bin/tmp/unpack/deal.II-v9.4.2/source/lac/block\_sparsity\_pattern.cc:172 #14 0x00000000043768cf in aspect::Simulator\<3\>::setup\_system\_matrix (this=0x7ffd2e447dd0, system\_partitioning=…) at /home/rmoucha/opt/aspect/source/simulator/core.cc:1143 #15 0x0000000004373723 in aspect::Simulator\<3\>::start\_timestep (this=0x7ffd2e447dd0) at /home/rmoucha/opt/aspect/source/simulator/core.cc:633 #16 0x0000000004370d18 in aspect::Simulator\<3\>::run (this=0x7ffd2e447dd0) at /home/rmoucha/opt/aspect/source/simulator/core.cc:2030 #17 0x0000000002b55f23 in run\_simulator\<3\> (raw\_input\_as\_string=…, input\_as\_string=…, output\_xml=false, output\_plugin\_graph=false, validate\_only=false) at /home/rmoucha/opt/aspect/source/main.cc:592 #18 0x0000000002b2f63f in main (argc=2, argv=0x7ffd2e44bfe8) at /home/rmoucha/opt/aspect/source/main.cc:784 ---
**Author:** ![tjhei](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/tjhei/32/1163_2.png) [@tjhei](https://community.geodynamics.org/u/tjhei)\ **Post date:** [March 20, 2023, 2:47pm UTC](https://community.geodynamics.org/t/aspect-hangs-at-random-spots-when-using-more-than-one-node/2861/8 "2023-03-20T14:47:06Z")
The call stacks you produced are inside Trilinos, but this does not necessarily mean that the bug is found there. This is because if one process does something different, every other process will hang in the next communication step (which very likely is inside linear algebra routines like you see). Are you using an unmodified prm file from ASPECT (which one?)? What is the minimum number of MPI ranks to see the hangs? It is unlikely that we can reproduce your case, but we can try. Otherwise, you will need to do more experiments: 1. Try to simplify the case (fewer MPI ranks, smaller prm file) 2. Try to produce the call stack of all MPI ranks for a specific run (like you did above), likely all except one of them are inside Trilinos and will look identical.