Dear Pylith team,
I try to run a task on multiple cores (my machine owns 32 cores) to speed up my simulation, but it happens some errors when i add the next words in pylithapp.cfg:
[pylithapp]
scheduler = pbs
[pylithapp.launcher]
command = mpirun -np ${nodes} -machinefile hosts
Could you help me how to modify it so that it runs on multiple cores successfully， I’ve only seen one core being used when running the task.
Thanks for your help with this issue.
The PBS (Portable Batch System) scheduler settings are for clusters that use the PBS batch queue system.
If you are running on a desktop machine without a batch queue system, simply use the command line argument --nodes=NPROCS where NPROCS is the number of processes. The parallel performance depends on the memory architecture; once the memory bus is saturated adding more processes will generally not reduce the runtime.
Dear Brad,
It works now. Thanks!
A post was merged into an existing topic: Single step elastic deformation with gravity
Dear Pylith team,
I have two problems:
Thanks for your help with this issue.
Finding the optimal number of processes when running in parallel to get maximum speedup depends on several factors, including your hardware (number of cores and memory architecture), the solver settings, and the type of problem). If you provide more information about the problem you are running, which version of PyLith you are using, any PETSc settings you are using, and the hardware you are using (processor type), we can provide more guidance.
Do you mean the hard disk storage is insufficient on your desktop or you have insufficient memory on your desktop? If the simulation is too large for your desktop, you can run PyLith on a cluster. In most cases, using multiple compute nodes on a cluster will provide more speedup than using additional processes on a desktop.
Dear Brad:
1.I am using Pylith-v2.2.2, and my processor is 5950x, the PETSc setting is not changed when i run my code. In addition, I found the same issue when I run it on another machine or VMware Workstation.
2.Due to insufficient desktop memory, I set the output path in the code to HDD (writer.filename=), and the linux system was installed on the SSD, which resulted in a much slower running speed.
Please provide all of your parameter (.cfg) files that show which settings you are using. If you are not setting any PETSc solver options, then you should certainly be selection options that are more suitable for the type of problem you are running.
The AMD 5950x processor has 16 cores. With multithreading enabled it will look like it has 32 cores. The AMD specs don’t show the memory architecture, but my guess is that the memory bus gets saturated when you use more than about 4-6 cores.
Hi,Brad:
my codes are here:
[pylithapp]
[pylithapp.journal.info]
timedependent = 1
explicitlumped = 1
explicitlumpedtet4 = 1
petsc = 1
meshiocubit = 1
distributor = 1
elasticityexplicit = 1
elasticityexplicittet4 = 1
meshimporter = 1
#quadrature3d = 1
#fiatlagrange = 1
faultcohesivedyn = 1
absorbingdampers = 1
[pylithapp.journal.debug]
#pylithapp = 1
#problem = 1
#explicit = 1
[pylithapp.mesh_generator]
reader = pylith.meshio.MeshIOCubit
reorder_mesh = True
reader.filename = large200 2000.exo
#refiner = pylith.topology.RefineUniform
#distributor.write_partition = True
#distributor.data_writer = pylith.meshio.DataWriterHDF5Ext
#distributor.data_writer.filename = output/partition.h5
[pylithapp.mesh_generator.reader]
coordsys.space_dim = 3
[pylithapp.timedependent]
dimension = 3
elastic_prestep = False
formulation = pylith.problems.ExplicitTet4
formulation.norm_viscosity = 0.5
normalizer = spatialdata.units.NondimElasticDynamic
normalizer.shear_wave_speed = 3464m/s
normalizer.wave_period = 0.01s
[pylithapp.timedependent.formulation.time_step]
total_time = 15s
dt = 0.003s
[pylithapp.timedependent]
materials = [elastic]
materials.elastic = pylith.materials.ElasticIsotropic3D
[pylithapp.timedependent.materials.elastic]
label = Elastic material
id = 1
db_properties = spatialdata.spatialdb.SimpleDB
db_properties.label = Elastic properties
db_properties.iohandler.filename = matprops.spatialdb
db_properties.query_type = nearest
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 3
[pylithapp.timedependent]
bc = [x_neg,x_pos,y_neg,y_pos,z_pos,z_neg]
bc.x_neg = pylith.bc.AbsorbingDampers
bc.x_pos = pylith.bc.AbsorbingDampers
bc.y_neg = pylith.bc.AbsorbingDampers
bc.y_pos = pylith.bc.AbsorbingDampers
bc.z_neg = pylith.bc.AbsorbingDampers
bc.z_pos = pylith.bc.AbsorbingDampers
[pylithapp.timedependent.bc.x_pos]
label = face_xpos
db.label = Absorbing BC +x
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
[pylithapp.timedependent.bc.x_neg]
label = face_xneg
db.label = Absorbing BC -x
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
[pylithapp.timedependent.bc.y_pos]
label = face_ypos
db.label = Absorbing BC +y
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
[pylithapp.timedependent.bc.y_neg]
label = face_yneg
db.label = Absorbing BC -y
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
[pylithapp.timedependent.bc.z_pos]
label = face_zpos
db.label = Absorbing BC +z
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
up_dir = [1, 1, 0]
[pylithapp.timedependent.bc.z_neg]
label = face_zneg
db.label = Absorbing BC -z
db.iohandler.filename = matprops.spatialdb
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
up_dir = [1, 0, 0]
[pylithapp.timedependent]
interfaces = [fault]
interfaces.fault = pylith.faults.FaultCohesiveDyn
[pylithapp.timedependent.interfaces.fault]
id = 10
label = fault
up_dir = [0, 0, 1]
friction = pylith.friction.SlipWeakening
friction.label = Slip weakening
zero_tolerance = 1.0e-3
zero_tolerance_normal = 1.0e-3
friction.db_properties = spatialdata.spatialdb.SimpleDB
friction.db_properties.label = Slip weakening
friction.db_properties.query_type = nearest
friction.db_properties.iohandler.filename = tpv16_friction.spatialdb
traction_perturbation = pylith.faults.TractPerturbation
quadrature.cell = pylith.feassemble.FIATSimplex
quadrature.cell.dimension = 2
quadrature.cell.quad_order = 2
[pylithapp.timedependent.interfaces.fault.traction_perturbation]
db_initial = spatialdata.spatialdb.SimpleDB
db_initial.label = Initial fault tractions
db_initial.query_type = nearest
db_initial.iohandler.filename = tpv16_tractions.spatialdb
db_change = spatialdata.spatialdb.SimpleDB
db_change.label = Amplitude of Initial traction_perturbation on Nucleation
db_change.iohandler.filename = tractions_axial_pressure.spatialdb
db_change.query_type = nearest
th_change = spatialdata.spatialdb.TimeHistory
th_change.label = Time history for Initial traction_perturbation on Nucleation
th_change.filename = loadcycle.timedb
#output.cell_info_fields = [change_in_value,change_start_time]
#output.writer.filename = output/step19-traction.vtk
#output.cell_filter = pylith.meshio.CellFilterAvg
[pylithapp.timedependent.formulation]
output = [domain]
#output.subdomain = pylith.meshio.OutputSolnSubset
[pylithapp.problem.formulation.output.domain]
vertex_data_fields = [displacement,velocity]
writer.time_format = %06.0f
writer.filename = output/tpv16_hex8_150m.vtk
writer.time_constant = 0.003*s
#[pylithapp.problem.formulation.output.subdomain]
#label = face_zpos
#vertex_data_fields=[displacement,velocity]
[pylithapp.timedependent.interfaces.fault.output]
vertex_info_fields = [strike_dir,normal_dir,traction_initial_value,static_coefficient,dynamic_coefficient,slip_weakening_parameter,cohesion]
vertex_data_fields = [slip,slip_rate,traction]
writer.time_format = %06.0f
writer.time_constant = 0.003*s
writer.filename = output/tpv16_hex8_150m-fault.vtk
[pylithapp.timedependent.materials.elastic.output]
cell_data_fields =
cell_filter = pylith.meshio.CellFilterAvg
writer.time_format = %06.0f
writer.time_constant = 0.003*s
writer.filename = output/tpv16_hex8_150m-elastic.vtk
[pylithapp.petsc]
petscpartitioner_type = parmetis
log_summary = true
It looks like you are running a problem with explicit time stepping. This should scale very well when running in parallel. In PyLith v2.x the mesh is loaded onto a single process, the faults created, and then the mesh is distributed. I recommend monitoring the memory use to make sure you have sufficient memory for these steps of the initialization. As mentioned previously, the speedup is generally controlled by the memory architecture. You can use the PETSc log summary to evaluate the different stages to find the best distribution strategy (number of processes) for your problem.