Mesh refinement and efficiency

Hi Forum,

Till now, I have been running models with moderate resolution, and these run reasonably fast on the medium-range cluster. A 200 Myr run takes around 1.5 weeks to complete using 160 cores using the gridding below.

subsection Mesh refinement
set Initial adaptive refinement = 1

set Initial global refinement = 4
set Minimum refinement level = 3
set Refinement fraction = 0.4
set Coarsening fraction = 0.05

set Strategy = nonadiabatic temperature,minimum refinement function, composition, vis
cosity

set Time steps between mesh refinement = 3

subsection Minimum refinement function
set Function constants =
set Function expression = if (depth > 2500000,6,if(depth > 660000,4,if(depth > 80000,5,6)))
set Variable names = depth,y,z
end
end

#################################################

Now, I am trying to move toward high-resolution models, and this is the one I got from a recently published paper: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2025JB032510. I was running this model with 960 cores on Stampede3, but it is very slow—I can only run about 7 Myr in 18 hours. Does anyone have any idea how many cores I should use to complete these runs within one week on Stampede3?

subsection Mesh refinement
set Initial adaptive refinement = 3

set Initial global refinement = 5
set Minimum refinement level = 3
set Refinement fraction = 0.35
set Coarsening fraction = 0.15

set Strategy = strain rate, maximum refinement function, minimum refinement function, temperature, viscosity, velocity

set Time steps between mesh refinement = 8

subsection Minimum refinement function
set Function constants =
set Function expression = if (depth < 400e6, 4, if (depth < 660e3, 4,3))
set Variable names = depth,y,z
end

subsection Maximum refinement function
set Function constants =
set Function expression = if (depth < 300e3, 8, if (depth < 450e3, 7, if (depth < 800e3, 7, 6)))set Variable names = depth,y,z

end

end

@DebanjanPal1995 Can you post how many degrees of freedom this model has, and how much model time each time step covers? So the first page or so of your output?

Best
WB

log.txt (211.4 KB)

Thanks @bangerth. Attached here is the log file.

Thanks.

A few thoughts:

  1. Nothing stands out as incorrect (optimized mode, GMG, native optimizations enabled). You spend 30+% in the Stokes solve.

  2. You can likely use more cores: you run with 1.5 billion DoFs (this is a big problem!) and 960 cores meaning you have more than 1 million per core. GMG should scale “down” to 30k DoFs, so you could use 30x more cores and get a faster time to solution.

  3. If you want to tune linear solver performance, you can enable “output details” [1] and try things like “Krylov method for cheap solver steps = IDR(s)” [2]

[1] Solver parameters — ASPECT 3.1.0-pre

[2] Solver parameters — ASPECT 3.1.0-pre

Thanks @tjhei for the suggestions. I will try them out and will keep informed.

Hi Forum,

While trying to load the .vtu files from high-resolution global runs in ParaView on a local system, ParaView crashes. What is the best way to deal with this issue? Any suggestions would be helpful.

@DebanjanPal1995 How bit are the VTU files combined? For big simulations, it’s not uncommon that they are in the range of gigabytes per time step. You shouldn’t expect that you read them on a normal workstation.

Best
WB

Thanks @bangerth for the reply. I ran the simulation using 160 cores, and each generated .vtu file is approximately 13 MB in size. I also tried loading the files in ParaView on the cluster using 160 processes, but I still encountered a segmentation fault issue.

What happens if you try with smaller simulations? I’m just poking in the dark, but 160x13 MB is not that large – about 2 GB, which when uncompressed ends up being perhaps 20 GB, and with in-memory data structures perhaps 100 GB. Most big workstations should be able to handle this. So I’d try to figure out whether it is the fact that you have many files, that the total size is large, whether Paraview can read any files at all, etc.

Low-resolution models, for example with a global refinement of 4, work fine. The problem arises when the global refinement is increased to 5. Also, loading an individual .vtu file works fine; however, ParaView crashes when loading the .pvtu file.

@DebanjanPal1995 I wished I had more suggestions, but don’t. It may be a problem with the size of the data set, the amount of memory you have, with Paraview, or any other number of things. If it doesn’t work on one machine, try it on a different machine. Talk to the system administrator. That’s really all I can offer.

Best
WB