How much RAM do i need to run a 3D simulation using Steinberger's model with a compositional field?

Francyrad · January 7, 2024, 2:32pm

Dear users,

For several months now, I have been studying LLSVPs under realistic conditions. I’ve conducted simulations using Steinberger’s “material model,” adapting it both to the Earth’s mantle and to my “compositional field” in a stagnant lid setting. The results I have obtained align with those in the scientific literature.

Now, I intend to move to a 3D model. After modifying my script for three-dimensional simulation, I have encountered some difficulties in starting the simulation:

With too low a resolution, the simulation fails to converge.
With a “normal” or high resolution, the program crashes due to insufficient RAM. I am using a MacBook Pro with an M1 Pro and 16 GB of RAM.

The solution seems to be a more powerful computer, capable not only of running the simulation quickly but also of supporting it with a larger amount of RAM.

My questions are as follows:

What is the minimum amount of RAM needed to run a Steinberger-type simulation with a compositional field, using a mesh resolution of 6, 7, 8, and 9?

Regarding the CPU, I am considering the AMD Ryzen Threadripper 7980X for its high computational capacity. However, I do not want to waste money on an excessively expensive CPU that, for only slightly better performance, does not justify the additional cost (as it’s already been experienced with demanding software like DaVinci Resolve). Therefore, I ask for your opinion:

In the attached benchmarks:

It shows that this CPU reaches 800,000 Mops/sec in integer math, while the M1 Pro only 46,000. Considering that ASPECT relies on intensive calculations, is it correct to assume that my simulations will be about 17.4 times faster in terms of integer math and about 7.5 times faster for floating-point calculations, resulting in being about 13 times faster overall?

Thank you very much for your help, I look forward to your response.

Francesco

bangerth · January 7, 2024, 5:28pm

@Francyrad I have no idea what this benchmark tests, but assume that it is a sum over multiple processor cores. The difference would, in that case, primarily be how many cores a machine has.

As for how to run 3d simulations: I suspect that most 3d simulations are run on machines with 100-1000 processor cores. This is more than you can get in a single machine, and so these simulations are almost always run on department- or university-wide resources (i.e., clusters of computers).

Regarding memory: You should probably take approximately 1 kB of memory per degree of freedom as a good order of magnitude, assuming that you use the matrix-free multigrid solver. For a million DoFs, you then need a GB of memory. But many 3d simulations have far far more than one million DoFs.

Best
W.

Francyrad · January 7, 2024, 6:55pm

Thank you very much for the response! Everything is much clearer now!

I’ve done this calculation, hoping not to be too naive (correct me if I’m wrong): I watched your video where you demonstrated the simple shell 3D simulation, conducted 8 years ago, on a cluster with 1000 cores. The simulation also had 64 million cells, according to the description.

This means that you needed at least 64 GB of RAM, which was considerable at that time. I don’t think I would have problems exceeding this limit, considering that modern workstations can have more than 128 GB of RAM, which should guarantee me about 128 million cells (but is this amount considered low compared to current standards for 3D simulations? I mean, would a resolution of 64 million cells be acceptable to reviewers for a 3D simulation?)

Regarding the CPU, the Threadripper 7980 has 64 cores and 128 threads, which is not bad. At the time, you ran the simulation on a cluster with 1000 cores. Considering that 8 years have passed, I assume that this new AMD CPU, in terms of single-core performance, is much more powerful than the one used in the cluster, and perhaps, has a similar or slightly lower overall power. Therefore, with this new setup, maybe taking a few more days, I might be able to replicate that simulation.

I would also like to know how to calculate the expected number of cells in a 3D simulation. For instance, if I know that my 2D simulation has 1 million cells, how many should I expect in a 3D simulation? Because if it were 10 billion, then I could never hope to run my simulation.

Thanks again for the help and suggestions.

Francesco

bangerth · January 7, 2024, 7:30pm

Francesco:
each cell has 25-30 degrees of freedom, so 64 million cells leads to more than 1B degrees of freedom. You won’t be able to fit this onto a single machine.

As far as core performance: Per-core performance has not increased by more than a factor of 2 or 3 over the last 20 years. The amount of memory per core has also not increased substantially. The only thing that has increased is the number of cores. So where I needed 1000 cores 8 years ago, you probably still need approximately this number; your computations will be somewhat faster, but the major difference is that 8 years ago, 1000 cores was considered a large number whereas today it is not.

To estimate the number of degrees of freedom, start with a small simulation and look what is being printed on the screen. Every global refinement increases the number of degrees of freedom by a factor of 4 (in 2d) or 8 (in 3d). You will find that refinement level 8 is perfectly fine in 2d, but very difficult in 3d.

Best
W.

Francyrad · January 7, 2024, 9:07pm

For a resolution of 5, it require 20kk degree of freedom. because it surpass my 16 Gb RAM capacity, the simulation crash because it should need something like 20 or more GB of RAM.
For a resolution of 6:

-----------------------------------------------------------------------------
--                             This is ASPECT                              --
-- The Advanced Solver for Planetary Evolution, Convection, and Tectonics. --
-----------------------------------------------------------------------------
--     . version 2.6.0-pre
--     . using deal.II 9.5.1
--     .       with 32 bit indices and vectorization level 0 (64 bits)
--     . using Trilinos 13.2.0
--     . using p4est 2.3.2
--     . running in OPTIMIZED mode
--     . running with 8 MPI processes
-----------------------------------------------------------------------------

-----------------------------------------------------------------------------
-- For information on how to cite ASPECT, see:
--   https://aspect.geodynamics.org/citing.html?ver=2.6.0-pre&particles=1&pda=1&sha=&src=code
-----------------------------------------------------------------------------
Number of active cells: 3,145,728 (on 6 levels)
Number of degrees of freedom: 155,371,150 (76,088,070+3,195,010+25,362,690+25,362,690+25,362,690)

I don’t even imagine what that would require of a res 7, but i think that a resolution of 6 is enough for my research. Because it needs 155kk DoF, does that means that a RAM of 200 Gb will handle the simulation even if it will require days because of a limited number of cores?
Also, to know if my simulation will converge or not, if my simulation in 2D doesn’t converge at the same resolution of the 3D (it doesn’t with a res of 6), it won’t converge in 3D also? If yes, because it’s impossible to run a res 7 on a normal workstation, i’ll just build a cheaper one

Let me know
Francesco

bangerth · January 8, 2024, 1:16am

Francesco:
I think 200 GB will likely be enough, but you can test this: There is a “memory consumption” or similar postprocessor that you can add to the list of postprocessors, and it will give you an estimate of the memory used.

As for the convergence issue, I’m not sure. Non-convergence can mean many things. You’ll have to show us the error message.
Best
W.

Francyrad · January 8, 2024, 3:35pm

Hi, i have some news

I could not use the postprocessor because it requires that actually a snapshot is created. Because the 3D Mega-sim won’t start, it will never let me know how much memory I need

Regarding the low res simulation error, this is what i get:

-----------------------------------------------------------------------------
--                             This is ASPECT                              --
-- The Advanced Solver for Planetary Evolution, Convection, and Tectonics. --
-----------------------------------------------------------------------------
--     . version 2.6.0-pre
--     . using deal.II 9.5.1
--     .       with 32 bit indices and vectorization level 0 (64 bits)
--     . using Trilinos 13.2.0
--     . using p4est 2.3.2
--     . running in OPTIMIZED mode
--     . running with 8 MPI processes
-----------------------------------------------------------------------------


   Loading Ascii data initial file /Users/francyrad/Documents/aspect_updated/data/initial-temperature/ascii-data/test/P700-p30-2H-2C.txt.

-----------------------------------------------------------------------------
-- For information on how to cite ASPECT, see:
--   https://aspect.geodynamics.org/citing.html?ver=2.6.0-pre&particles=1&pda=1&sha=&src=code
-----------------------------------------------------------------------------
Number of active cells: 49,152 (on 7 levels)
Number of degrees of freedom: 1,040,640 (396,288+49,920+198,144+198,144+198,144)

*** Timestep 0:  t=0 years, dt=0 years
   Solving temperature system... 0 iterations.
   Copying properties into prescribed compositional field density_field... done.
   Rebuilding Stokes preconditioner...
   Solving Stokes system... 232+0 iterations.

Number of active cells: 110,412 (on 8 levels)
Number of degrees of freedom: 2,389,644 (910,048+114,524+455,024+455,024+455,024)

*** Timestep 0:  t=0 years, dt=0 years
   Solving temperature system... 0 iterations.
   Copying properties into prescribed compositional field density_field... done.
   Rebuilding Stokes preconditioner...
   Solving Stokes system... 283+0 iterations.

Number of active cells: 151,206 (on 9 levels)
Number of degrees of freedom: 3,329,826 (1,267,920+160,026+633,960+633,960+633,960)

*** Timestep 0:  t=0 years, dt=0 years
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 130 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x0000000114472ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x0000000114472ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x000000011440f1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x000000011440f1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x000000011440ee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x000000011440ee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x0000000102ca2dbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x0000000102ca2dbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000102751ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000102751ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x00000001030577f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x00000001030577f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x0000000103056ee0 main + 2764: 8   aspect-release                      0x0000000103056ee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

[MacBook-Pro-di-Francesco-3:08398] *** Process received signal ***
[MacBook-Pro-di-Francesco-3:08398] Signal: Abort trap: 6 (6)
[MacBook-Pro-di-Francesco-3:08398] Signal code:  (0)
[MacBook-Pro-di-Francesco-3:08398] [ 0] 0   libsystem_platform.dylib            0x00000001883a7a24 _sigtramp + 56
[MacBook-Pro-di-Francesco-3:08398] [ 1] 0   libsystem_pthread.dylib             0x0000000188378cc0 pthread_kill + 288
[MacBook-Pro-di-Francesco-3:08398] [ 2] 0   libsystem_c.dylib                   0x0000000188288a40 abort + 180
[MacBook-Pro-di-Francesco-3:08398] [ 3] 0   libc++abi.dylib                     0x00000001883306d8 _ZN10__cxxabiv130__aligned_malloc_with_fallbackEm + 0
[MacBook-Pro-di-Francesco-3:08398] [ 4] 0   libc++abi.dylib                     0x00000001883207ac _ZL28demangling_terminate_handlerv + 320
[MacBook-Pro-di-Francesco-3:08398] [ 5] 0   libobjc.A.dylib                     0x0000000187fcb8a4 _ZL15_objc_terminatev + 160
[MacBook-Pro-di-Francesco-3:08398] [ 6] 0   libc++abi.dylib                     0x000000018832fa9c _ZSt11__terminatePFvvE + 16
[MacBook-Pro-di-Francesco-3:08398] [ 7] 0   libc++abi.dylib                     0x000000018832fa40 _ZSt9terminatev + 56
[MacBook-Pro-di-Francesco-3:08398] [ 8] libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 70 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x000000011667aecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x000000011667aecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x00000001166171a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x00000001166171a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000116616e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000116616e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x0000000104eaadbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x0000000104eaadbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000104959ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000104959ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x000000010525f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x000000010525f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x000000010525eee0 main + 2764: 8   aspect-release                      0x000000010525eee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 138 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x00000001167b6ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x00000001167b6ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x00000001167531a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x00000001167531a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000116752e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000116752e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x0000000104fe6dbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x0000000104fe6dbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000104a95ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000104a95ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x000000010539b7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x000000010539b7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x000000010539aee0 main + 2764: 8   aspect-release                      0x000000010539aee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 69 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x000000011496aecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x000000011496aecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x00000001149071a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x00000001149071a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000114906e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000114906e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x000000010319adbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x000000010319adbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000102c49ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000102c49ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x000000010354f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x000000010354f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x000000010354eee0 main + 2764: 8   aspect-release                      0x000000010354eee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 116 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x0000000116c32ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x0000000116c32ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x0000000116bcf1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x0000000116bcf1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000116bcee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000116bcee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x0000000105462dbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x0000000105462dbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000104f11ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000104f11ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x00000001058177f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x00000001058177f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x0000000105816ee0 main + 2764: 8   aspect-release                      0x0000000105816ee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 76 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x0000000114baaecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x0000000114baaecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x0000000114b471a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x0000000114b471a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000114b46e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000114b46e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x00000001033dadbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x00000001033dadbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000102e89ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000102e89ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x000000010378f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x000000010378f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x000000010378eee0 main + 2764: 8   aspect-release                      0x000000010378eee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
0   aspect-release                      0x0000000102ca2dbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64
[MacBook-Pro-di-Francesco-3:08398] [ 9] 0   aspect-release                      0x0000000102751ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56
[MacBook-Pro-di-Francesco-3:08398] [10] 0   aspect-release                      0x00000001030577f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012
[MacBook-Pro-di-Francesco-3:08398] [11] 0   aspect-release                      0x0000000103056ee0 main + 2764
[MacBook-Pro-di-Francesco-3:08398] [12] 0   dyld                                0x0000000187ffd0e0 start + 2360
[MacBook-Pro-di-Francesco-3:08398] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 84 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x00000001168aaecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x00000001168aaecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x00000001168471a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x00000001168471a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x0000000116846e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x0000000116846e04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x00000001050dadbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x00000001050dadbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000104b89ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000104b89ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x000000010548f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x000000010548f7f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x000000010548eee0 main + 2764: 8   aspect-release                      0x000000010548eee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
libc++abi: terminating due to uncaught exception of type dealii::StandardExceptions::ExcMessage: 
--------------------------------------------------------
An error occurred in line <62> of file </Users/francyrad/dealii-candi/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<2, 2>::clear() [dim = 2, spacedim = 2]
The violated condition was: 
    n_open_handles == 0
Additional information: 
    This property pool currently still holds 128 open handles to memory
    that was allocated via allocate_properties_array() but that has not
    been returned via deregister_particle().

Stacktrace:
-----------
#0  2   libdeal_II.9.5.1.dylib              0x00000001169a2ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv + 308: 2   libdeal_II.9.5.1.dylib              0x00000001169a2ecc _ZN6dealii9Particles12PropertyPoolILi2ELi2EE5clearEv 
#1  3   libdeal_II.9.5.1.dylib              0x000000011693f1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv + 224: 3   libdeal_II.9.5.1.dylib              0x000000011693f1a4 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EE15clear_particlesEv 
#2  4   libdeal_II.9.5.1.dylib              0x000000011693ee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev + 40: 4   libdeal_II.9.5.1.dylib              0x000000011693ee04 _ZN6dealii9Particles15ParticleHandlerILi2ELi2EED2Ev 
#3  5   aspect-release                      0x00000001051d2dbc _ZN6aspect8Particle5WorldILi2EED0Ev + 64: 5   aspect-release                      0x00000001051d2dbc _ZN6aspect8Particle5WorldILi2EED0Ev 
#4  6   aspect-release                      0x0000000104c81ee4 _ZN6aspect9SimulatorILi2EED2Ev + 56: 6   aspect-release                      0x0000000104c81ee4 _ZN6aspect9SimulatorILi2EED2Ev 
#5  7   aspect-release                      0x00000001055877f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb + 1012: 7   aspect-release                      0x00000001055877f4 _Z13run_simulatorILi2EEvRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_bbb 
#6  8   aspect-release                      0x0000000105586ee0 main + 2764: 8   aspect-release                      0x0000000105586ee0 main 
#7  9   dyld                                0x0000000187ffd0e0 start + 2360: 9   dyld                                0x0000000187ffd0e0 start 
--------------------------------------------------------

SIGABRT received
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node MacBook-Pro-di-Francesco-3 exited on signal 6 (Abort trap: 6).

This error happens also if I try to increase a little bit the resolution, so idk if this is a bug or some other thing

This is the res that I tried:

subsection Mesh refinement
  set Initial adaptive refinement        = 5

  set Initial global refinement          = 6
  set Minimum refinement level           = 6
  set Refinement fraction                = 0.4
  set Coarsening fraction                = 0.05

  set Strategy                           = nonadiabatic temperature,minimum refinement function, composition, viscosity, boundary

  set Time steps between mesh refinement = 3

  subsection Boundary
    set Boundary refinement indicators = top, bottom
  end

  subsection Minimum refinement function
    set Function constants  =
    set Function expression = if (depth > 2300000,7,if(depth > 500000,5,if(depth > 80000,6,7)))
    set Variable names      = depth,y,z
  end
end

subsection Boundary temperature model
  set Fixed temperature boundary indicators   = 0,1
end

I hope we can solve
Francesco

Francyrad · January 8, 2024, 8:52pm

EDIT: I changed solver (iterated advection and Stokes) instead of single advection and Stokes and now the 3D simulation goes. it has 5kk FoD, but it’s impossibly slow even when using all the CPUs, like if it’s using only one instead of 8 at full power.

bangerth · January 8, 2024, 10:49pm

@Francyrad You can check on the memory consumption on a smaller problem. If you can run a 3d problem with, let’s say, 4 refinement steps, then 5 refinement steps will take ~8x as much memory, and 6 refinement steps will take ~64x as much memory.

As for the speed: A good rule of thumb is that each solve will take ~1 minute if you have 10^5 degrees of freedom per process. In your case, you have 5 million degrees of freedom. With 8 processes, you end up with 600,000 DoFs/process, for which you should expect a run time in release mode of around 6 minutes per time step. You can check how many of the processes are working at full speed if, in a separate terminal window, you run the top program. Any other system monitor program will also do.

Best
W.

bangerth · January 8, 2024, 10:54pm

Separately: The error message you get is not helpful – I am pretty sure that your system is simply out of memory, and what you see is simply a secondary effect of things going wrong. Would you mind sending me the input file that caused this error?
Best
W.

Francyrad · January 9, 2024, 3:21pm

Yes, that is weird. Usually the macbook show “111” for a core that is working at full speed with mpirun, but they are running at weird speed. some times it runs at 40, some times at 20. It’s also running another process called “kernel_task” which operates when the computer is at hight temperatures, and this drain a a full core of computational power.

My truth is that this problem is linked only to Apple Silicon: RAM and CPU are linked toogheter, is not like Intel and AMD CPUs where RAM is separated from CPUs in the motherboard. because of the fact that the sim is consuming a lot of RAM, there are also limitations from the normal simulations (1 milion DoF in 2D with 1kk cells) that runs at full power.

Anyway, the 3D simulation runs with limitations, I should not have problems with a greater setup…

i take the opportunity to say that i’ve finished the ASCII’s cookbook, where, how and to who should I send it?

Francesco

bangerth · January 9, 2024, 8:00pm

@Francyrad I was interested in the simulation that produced the error message in PropertyPool. If you have these input files, please just email them to me.
Best
W.

Francyrad · February 23, 2024, 9:59pm

I sent you the file in private message.

Regarding the RAM, i found this pattern:

subsection Mesh refinement
  set Initial adaptive refinement        = 1

  set Initial global refinement          = 3
  set Minimum refinement level           = 2
  set Refinement fraction                = 0.4
  set Coarsening fraction                = 0.05

  set Strategy                           = nonadiabatic temperature,minimum refinement function, composition, viscosity, boundary

  set Time steps between mesh refinement = 3

  subsection Boundary
    set Boundary refinement indicators = top, bottom
  end

  subsection Minimum refinement function
    set Function constants  =
    set Function expression = if (depth > 2300000,3,if(depth > 500000,3,if(depth > 80000,3,3)))
    set Variable names      = depth,y,z
  end
end

2.5kk DOF -> 2.5 gb ram - then 5.9 kk, 5.9 gb


subsection Mesh refinement
  set Initial adaptive refinement        = 2

  set Initial global refinement          = 4
  set Minimum refinement level           = 3
  set Refinement fraction                = 0.4
  set Coarsening fraction                = 0.05

  set Strategy                           = nonadiabatic temperature,minimum refinement function, composition, viscosity, boundary

  set Time steps between mesh refinement = 3

  subsection Boundary
    set Boundary refinement indicators = top, bottom
  end

  subsection Minimum refinement function
    set Function constants  =
    set Function expression = if (depth > 2300000,4,if(depth > 500000,4,if(depth > 80000,4,4)))
    set Variable names      = depth,y,z
  end
end





Number of active cells: 393,216 (on 5 levels)
Number of degrees of freedom: 19,575,630 (9,585,030+405,570+3,195,010+3,195,010+3,195,010)

20 gb of ram, will this value will be doubled?



subsection Mesh refinement
  set Initial adaptive refinement        = 3

  set Initial global refinement          = 5
  set Minimum refinement level           = 4
  set Refinement fraction                = 0.4
  set Coarsening fraction                = 0.05

  set Strategy                           = nonadiabatic temperature,minimum refinement function, composition, viscosity, boundary

  set Time steps between mesh refinement = 3

  subsection Boundary
    set Boundary refinement indicators = top, bottom
  end

  subsection Minimum refinement function
    set Function constants  =
    set Function expression = if (depth > 2300000,5,if(depth > 500000,5,if(depth > 80000,5,5)))
    set Variable names      = depth,y,z
  end
end

Number of active cells: 3,145,728 (on 6 levels)
Number of degrees of freedom: 155,371,150 (76,088,070+3,195,010+25,362,690+25,362,690+25,362,690)

155 gb of ram, will this value will be doubled?

As you said the ram used is around x8… My doubt is that at the lowest resolution, after 3 timesteps, the value of DoF will be doubled, so i will pass from using 2.5 Gb to 5 Gb of RAM. Will this happens also at higher levels of resolution? I cannot test that for my machine limitations and i also cannot buy 256 Gb of RAM and then discover that i need 400 Gb. Is there someway i can check if the value will be doubled or some way to have a fixed value of DoF? So i can choose with precision the amount of ram for my workstation. Statistics won’t work because i cannot run the first time step above 16 Gb of RAM

Best regards and still thank you for your work!
Francesco

jbnaliboff · February 26, 2024, 6:01pm

@Francyrad - A few responses before @bangerth has a chance to weight (I believe he is currently traveling):

Will this happens also at higher levels of resolution?

I think a reasonable expectation is that doubling the number of cells should result in the total amount of memory used being doubled. If you double the resolution in a 3D model, then this should result in an ~ 8x memory increase.

So i can choose with precision the amount of ram for my workstation.

I often go for fewer CPUS and more RAM on my workstation for this exact reason, it is hard to predict exactly how much will be needed as your projects evolve and more certainly won’t hurt.

To come back to a point @bangerth made earlier, are you currently using the matrix-free (GMG) Stokes solver? This will greatly reduce the total memory consumption and is well suited for the class of models you are running.

Cheers,
John

Francyrad · February 26, 2024, 9:17pm

I tried to use BlockGMG, but with apple silicon i have a vectoralization level of 0 (i already try to upward that, you cannot do that with Apple silicon). My doubt is still if my simulation will double in RAM usage after the refinement

jbnaliboff · February 27, 2024, 6:21pm

@Francyrad - If the issue in question is how much an adaptive refinement step will increase the model size, you can use the solver scheme “no Advection, no Stokes”, which will setup up the initial conditions and run the post processors, but not assemble the matrices for solving the system of equations (i.e., it uses much less RAM). With this, you can at least check how many DOF/cells the model contains.

Beyond that, you may need to find a larger computer to test the model on to get a sense for memory consumption as a function of model size.

Cheers,
John

Francyrad · February 27, 2024, 10:33pm

@jbnaliboff I really thank you for the suggestion. I discovered that “the problem” was:

  set Initial adaptive refinement        = 0

If 1, the DoF will double, if 2 triplicate and so on. So if i put that to 0 at an high res, it won’t change and i can stay with 150 Gb of RAM. I could also check with no advection and no stokes that 20kk of DoF will double at 40KK if the adaptive refinement is 1.

So thank you very much! For me the case is closed

@bangerth I sent you a pm with the prm, let me know!

Francyrad · February 28, 2024, 6:26pm

I take the opportunity to ask one more question: will the RAM’s frequency and latency (CL) affect the speed of my simulation? If not (and it’s only important the amount of RAM) I’ll buy a slower one, also, do you suggest to take ECC one?

bobmyhill · February 28, 2024, 6:59pm

I’ll leave answering your question to someone else, but I’d like to suggest an alternative that you might not have realised was an option: researchers can apply for/buy time on existing clusters. This would avoid you having to buy a very expensive machine that might still not be able to do what you want.

Even if your department doesn’t have a cluster for research purposes, your university almost certainly will (although some are now outsourcing to Google/Amazon), and if you need to go really big every country has supercomputers for scientific research (e.g. Stampede 3 in the States). University clusters might be a good place for you to start - many institutions provide free allocations to get things installed and tested.

bangerth · February 28, 2024, 8:55pm

Francesco:
I don’t think any of us are in a position to give definite advice on any of
these questions. Of course, every specific piece of hardware (processor,
memory, type of memory, etc) has some kind of effect on the speed you will
experience. How much that is, and how much money you’re willing to spend to
get (say) a 5% increase in speed is something none of us can really know. In
general, the price increase for performance increase gets pretty steep at the
end. You can get a good workstation for $15k; you can get a better one for
$30k, but it will not be twice the speed. If your goal is to build a
reasonably well equipped workstation, I would go for something like 32 cores
(not counting hyperthreading) and 128 GB of memory.

Best
Wolfgang

Topic		Replies	Views
How to use more than a tabulated Burman file in the ASCII model ASPECT	4	129	June 16, 2023
Adding New Composition Fields to a Restart Computation	2	410	July 12, 2019
Impossible to use ASCII for setting initial composition in a 2D Spherical Shell ASPECT	1	73	December 26, 2023
Running SPECFEM3D by using GPU SPECFEM specfem3d	1	546	April 8, 2021
Stability problem when solving for temperature and composition ASPECT	5	515	November 5, 2019

How much RAM do i need to run a 3D simulation using Steinberger's model with a compositional field?

Related Topics