Hi, CIG Team!
I’m working on the coupling of PyLith with a reservoir simulator to model CO2 storage in an aquifer. During coupled simulation, PyLith is called at certain time instants many times. It performs quasi-static modeling of the reservoir mechanical equilibrium state corresponding to specified pressure and temperature distributions. Based on the stress-strain state, reservoir hydraulic properties are updated.
On every call of PyLith, several steps are repeated, e.g., reading mesh from the GMSH file, defining boundary conditions, materials, and their rheology. Therefore, I would like to ask is it possible to restore the previously created PyLith model from the file. In the restored model, one can introduce modifications in elastic properties and assign a new reference stress state (containing current pressure and temperature distributions) via a spatial database and perform the simulation.
Using model restoration, it is possible to eliminate unnecessary repetitions, which, in turn, can take a long time if the model size is larger. E.g., this option exists in FLAC3D mechanical simulator: “RESTORE – restores an existing (binary) saved state from a previously executed problem”.
Based on the website, it seems that such an option is unavailable in PyLith. Maybe you can provide some recommendations regarding this issue. Is it possible to read a mesh from the “…_info.h5” file created during the first PyLith call and apply this data to attributes of corresponding classes during the second and subsequent calls? I think other repeating steps take much less time than reading mesh.
Thank you in advance!
Best regards,
Evgenii.
PyLith v3 and later are designed with the idea that coupling codes would be done through the API rather than independent invocations of PyLith. We have “observers” and custom implementations could be added to pass information from PyLith to another code. We do not yet have a mechanism in place for transferring information from another code into PyLith to change the state, but we are thinking about how this would be implemented.
With the current PyLith code (v3 and v4), you can do what you are doing. We do not have the ability to create checkpoints and restart from them. Restarting from a checkpoint usually involves restoring an identical state. Changing the state after restarting from a checkpoint is not much different than starting a new simulation because most of effort would be associated with initialization or reinitialization.
There are things you can do to make the initialization more efficient. For example, use the UniformDB spatial database implementation whenever things are uniform and use the SimpleGridDB spatial database implementation whenever there is a spatial variation in values. The SimpleDB implemetation is much less efficient. The AnalyticDB is also efficient but is limited to analytical functions for values.
If you provide more information about how you are setting up your BCs, materials, etc, we might be able to provide additional pointers for doing it more efficiently. You should also think about what discretization is appropriate. For example, a coarser resolution mesh using a basis order of 2 for the displacement might be more accurate and efficient.
Dear Brad,
thank you for your answer.
I attach an archive with PyLith model utilized in the coupled modeling of CO2 storage in an aquifer intersected by a fault. Spatial database file contains initial pressure and temperature distributions.
Reservoir simulator solves dynamic problem of CO2 injection. It is paused at certain time moments, and the coupling python code prepares spatial database containing elastic properties and reference stresses. Elastic properties account for current density distribution. Reference stresses account for current pressure and temperature distribution according to equations: \sigma_{ref} = \sigma_{prev} - (\Delta P + 3\kappa K \Delta T)1, \Delta P = P_{cur} - P_{prev}, \Delta T = T_{cur} - T_{prev}, where \sigma_{prev} is the stress tensor from the previous PyLith call, \Delta P and \Delta T are pressure and temperature alterations betwen current and previous PyLith calls, \kappa is the thermal-expansion coefficient, K is the bulk modulus. Next, the coupling code invokes PyLith model attached to this message.
If the number of cells in the PyLith model is several thousand, PyLith call is executed for several seconds. However, I tested a model with 20 thousand of cells, and “Reading finite-element mesh” step (according to logs in console) takes noticable amont of time. This mesh reading is repeated during each PyLith call. Meanwhile, mechanical model initiation is carried out much faster. Therefore, I have an idea to fill attributes of the mesh class pylith.topology.Mesh by reading data from “_info.h5” file created after the first PyLith call. Probably, this idea won’t work due to the peculiarities of the PyLith code architecture.
Thanks!
Evgenii
pylith_model.zip (810.8 KB)
Reading in the mesh file itself even with 20k cells most likely takes a fraction of a second. The processing of the file is probably taking up much more time. Just switching the format probably won’t change how long it takes to setup.
You can run with the --petsc.log_view command line option to get a profiling summary. This should help identify which routines take a significant amount of time.
I recommend updating your spatial database to be a SimpleGridDB instead of a SimpleDB. This will certainly speed up that part of the runtime.
Dear Brad,
thank you for your recommendations.
I run PyLith with this command for a model with 20k cells and observe that Meshing stage takes 65 % of the total time (archive with screenshots is attached to the message).
Thus, apart from the usage of SimpleGridDB, there is no way to speed up the repeating calls of PyLith model with the same geometry?
Thanks!
Evgenii
Screenshots.zip (200.2 KB)
The Word document you put the screenshots in isn’t working for me. You should be able to just post the screenshots on the forum. Alternatively, just create a zip file containing the screenshot images.
An archive with screenshots is enclosed.
Screenshots.zip (882.4 KB)
I regenerated the mesh for examples/subduction-2d with a minimum grid spacing of 1 km, which yields a mesh of about 20k cells. The Meshing stage takes about 6 sec on my Linux desktop machine. Can you post a complete set of input files for a PyLith run with your mesh with 20k cells, so I can see if I can replicate the performance you see?
Can you also provide information about the computer you are using? What operating system is it running and what version? How much memory does it have? Are all of the filesystems local (physically on the device) or are any mounted over a network?
Dear Brad,
an archive with PyLith model with 20k cells is enclosed.
I performed two tests:
- model with uniform properties (
UniformDB, mesh_model.cfg)
- model with properties read by PyLith from file (
SimpleDB, main_model.cfg)
In the 1st test, meshing takes 40 seconds (80% of PyLith execution).
In the 2nd case, meshing takes the same amont of time (65% of PyLith execution due to reading and appying data from file).
Information about computer: Windows 10 (WSL is used to call PyLith); CPU – Intel Core i7-8850H; RAM – 16 GB; all files are on the local filesystem.
Thanks!
Evgenii
pylith_model_20k_cells.zip (4.9 MB)
Thanks for providing this example. I found a bottleneck in some processing we do after we read in the mesh and rewrote the function. On my desktop machine, the rewrite is 100x faster for your case. This optimization will be available in the next release (target is late April or early May).
If you switch from SimpleDB to SimpleGridDB, it should reduce the setup time significantly.