Resetting composition from checkpoint restart

I was wondering if anyone has any experience restarting from a checkpoint, but utilizing only temperature information, such that the composition is reset again by the initial condition? I was unable to find the functionality (which doesn’t mean it isn’t there!), and thought I might ask before rewriting a checkpoint module.

The ascii data model in theory permits this, but requires an awful lot gridding of an output temperature field, to obtain a solution refined enough in the required ascending grid format, and so is a little anachronistic for this purpose.

The ideal is to re-read in the mesh and temperature from the solution files.


Just for reference, here’s a script (python) to construct an ascii dataset in the required format, from an old h5 mesh and solution file.

import h5py
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import griddata


mesh = h5py.File(‘mesh-02630.h5’,‘r’)
nodes = mesh[‘nodes’][:]

radius = np.sqrt(nodes[:,0]*nodes[:,0] + nodes[:,1]nodes[:,1])/1e6 #Calculate R, convert to Mm for gridding
phi = np.arctan2(nodes[:,1],nodes[:,0]) #Calculate phi = atan(y/x)
phi[ phi<0 ] += 2
np.pi #Convert to be between 0–>2pi

f = h5py.File(‘solution-03148.h5’, ‘r’) #Loading temperature solution data
T = f[‘T’][:]

R2 = np.linspace(3481000,6336000,100)/1e6 #Make sure in Mega-m also.
phi2 = np.linspace(0,2*np.pi,1200)

R,P = np.meshgrid(R2,phi2)

out = griddata((radius, phi), T[:,0], (R,P),method=‘nearest’)

print(".2f .2f %.2f",np.c_[R.ravel(),P.ravel(),out.ravel()])

Haha - note the online editor has formatted my format comments in the above!

np.savetxt(‘interpolated_out3.dat’,np.c_[R.ravel()*1e6,P.ravel(),out.ravel()]) #Ouput R,phi and T data. Convert R to m, ascending R order is correct (ie. same as shell_2d.txt)

fig = plt.figure()
plt.tricontourf(P.ravel(),R.ravel(),out.ravel(),levels=100, cmap=‘RdGy_r’)


And the a picture of the imported data (regridded to 300x1200 - which seems to capture most of the details):

The checkpoint/resume feature currently does not support changes like this.

While crude, it is my understanding what others have done the same thing in the past. Without too much development, this is probably the best approach.

While this seems attractive, I would like to point out that the generation of vtk or h5 files is also not “loss-free”. It would seem possible to create an “initial temperature” plugin that reads from a checkpoint or h5 file.

Another option that seems closer to being usable:
I am working on a postprocessor plugin that produces structured data (x,y,z or lat/long/depth) and writes to ascii/netcdf (maybe hdf5 in the future), that would lend itself to be loaded in again as an initial condition. This will take a couple more weeks/months though.

Perhaps another approach could be to write a new material plugin that resets (sets to zero?) the composition at a specific time or after any checkpoint restart? The plugin could use a user-defined time as an input parameter to decide when to reset the composition, or potentially access the global variable that determines whether the model is resuming from a checkpoint.

It would be easy to add a signal that is called right after reading all of the data back in from a checkpoint file. We don’t even need a plugin system for that: the signal would just receive a SimulatorAccess object. Whoever attaches to that signal can then change whatever they want.

That said, that’s definitely not how checkpoint/restart is envisioned – and that’s by design because the intent is to simply continue a previously suspended computation. Craig: Can you explain why you actually want to do that? Why would one want to reset a field in a way that makes the combination of solution variables suddenly inconsistent with each other?


Hi all,
Thanks very much for your thoughts.

Timo - thanks for the clarification. I look forward to the postprocessor addition, but it seems that for now simply recreating the ascii model data is the lowest hanging fruit.

John - yes, that was my thought too, but given there is a simple scripting workaround that is winning my cost-benefit analysis.

Wolfgang - thanks (and I will stay well away from main branch stuff). I’m with you that the main point of checkpointing is a complete restart - here I’m being an annoying user with a specific initial condition requirement.
For me - I have a number of very very long simulations (O~Gyrs) that consumed a lot of kSU to reach some sort of statistical equilibrium* with their thermal BCs (I am using some exotic rheologies, and thus high-resolution and time-consuming convergence). These simulations were for a given problem and composition fields (crust, depletion) that are not relevant to the current one. I also have a module that is utilising the first compositional field for another purpose. Thus… for me the advantage of using just the thermal information is that despite the IC not being exactly right for the new problem, the *equilibration time is far less than beginning the simulation completely from scratch (by about 2 orders of magnitude, which is important on a limited cluster allocation).

Hope that clarifies!



*evolving heat production and core, and thus equilibration is loosely used here.