# Artifacts with HDF5 output **URL:** \ **Category:** ASPECT\ **Created:** [January 18, 2021, 12:37am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649 "2021-01-18T00:37:41Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![maxrudolph](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/maxrudolph/32/248_2.png) [@maxrudolph](https://community.geodynamics.org/u/maxrudolph)\ **Post date:** [January 18, 2021, 12:37am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/1 "2021-01-18T00:37:41Z")
Hello everyone. I am having some issues with the visualization of hdf5 files in ASPECT. The model that I am examining is 3D, cartesian geometry. - There is no adaptive mesh refinement. - The model uses periodic boundary conditions in both the x- and y-directions. The visualization output looks almost correct except for the artifact connecting the top right and bottom left of the image in the attached output. I verified that the visualization contains the artifact in the latest VisIt 3.1.4 as well as the older VisIt 2.13.3. Has anyone else run into something similar? Thank you for your help. ![image](https://canada1.discourse-cdn.com/flex036/uploads/geodynamics/original/1X/e2e49bea543fb3444bb6dafa1a8b0614d0bf4e62.jpeg) This is the aspect version info: – This is ASPECT, the Advanced Solver for Problems in Earth’s ConvecTion. – . version 2.3.0-pre (master, 826c25583) – . using deal.II 9.3.0-pre (master, 250eae6824) – . with 32 bit indices and vectorization level 1 (128 bits) – . using Trilinos 12.10.1 – . using p4est 2.0.0 – . running in OPTIMIZED mode – . running with 128 MPI processes ---
**Author:** ![maxrudolph](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/maxrudolph/32/248_2.png) [@maxrudolph](https://community.geodynamics.org/u/maxrudolph)\ **Post date:** [January 18, 2021, 1:19am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/2 "2021-01-18T01:19:35Z")
Moved to ASPECT forum ---
**Author:** ![bangerth](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/bangerth/32/52_2.png) [@bangerth](https://community.geodynamics.org/u/bangerth)\ **Post date:** [January 18, 2021, 1:25am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/3 "2021-01-18T01:25:39Z")
Max – what happens if you visualize the mesh? Does it have one warped cell? Best W. ---
**Author:** ![maxrudolph](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/maxrudolph/32/248_2.png) [@maxrudolph](https://community.geodynamics.org/u/maxrudolph)\ **Post date:** [January 18, 2021, 1:56am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/4 "2021-01-18T01:56:06Z")
Wolfgang - thanks for your very quick response. Good idea to look at the mesh. There is an artifact in the upper (+x,+y corner). It does not appear to be a single warped cell but maybe corresponds to a single warped cell at a coarser refinement level? ![image](https://canada1.discourse-cdn.com/flex036/uploads/geodynamics/original/1X/28bca30801f62725bc514e8595fd6018cff2fcc1.png) ---
**Author:** ![bangerth](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/bangerth/32/52_2.png) [@bangerth](https://community.geodynamics.org/u/bangerth)\ **Post date:** [January 19, 2021, 7:46am UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/5 "2021-01-19T07:46:30Z")
Max – what’s the coarsest mesh with which you can reproduce this? If you reduce global mesh refinement to a minimum and simplify the setup, can you come up with a minimal testcase that shows the issue? Best Wolfgang ---
**Author:** ![maxrudolph](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/maxrudolph/32/248_2.png) [@maxrudolph](https://community.geodynamics.org/u/maxrudolph)\ **Post date:** [January 19, 2021, 7:11pm UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/6 "2021-01-19T19:11:56Z")
I ran a bunch of test cases and found that the problem depends on the _number of nodes_ not the _number of MPI processes_ used in the calculation. As soon as the computation is distributed across two or more nodes, the visualization output is incorrect. I believe that this is due to using parallel HDF5 on NFS file systems, the same issue that we discussed here: > > > On the UC Davis cluster 'peloton', our scratch filesystem is provided as a NFS mount. Because of potential issues with: > (1) correctness... and here > > > Hi, > I am compiling with ompi/v3.0.x branch each night and I just noticed that since oct 26, our nightly validation tests where... I re-ran a test model on 4 nodes with different values of the OpenMPI fs\_ufs\_lock\_algorithm `export OMPI_MCA_fs_ufs_lock_algorithm=1` has corrupted output `export OMPI_MCA_fs_ufs_lock_algorithm=3` has the correct mesh output The only workaround that I can see is to allow aspect (and deal.ii) to be configured with serial hdf5 even when running in parallel. ---
**Author:** ![bangerth](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/bangerth/32/52_2.png) [@bangerth](https://community.geodynamics.org/u/bangerth)\ **Post date:** [January 21, 2021, 9:06pm UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/7 "2021-01-21T21:06:18Z")
> The only workaround that I can see is to allow aspect (and deal.ii) to be configured with serial hdf5 even when running in parallel. But that isn’t going to work easily either. The HDF5 codes make the assumption that data from one processor is going to be stored correctly in the file even if other processors do the same with theirs. Some of the HDF5 operations may also be collective. So it isn’t just allowing configuration with a sequential HDF5 library, but it would require that the code that _uses_ these interfaces sends its data to a central processor which then does the writing; similar for reading. That would imply a lot of work, I’m afraid. Can’t you just always run with `export OMPI_MCA_fs_ufs_lock_algorithm=3`? Best W. ---
**Author:** ![tjhei](https://yyz2.discourse-cdn.com/flex036/user_avatar/community.geodynamics.org/tjhei/32/1163_2.png) [@tjhei](https://community.geodynamics.org/u/tjhei)\ **Post date:** [January 22, 2021, 4:35pm UTC](https://community.geodynamics.org/t/artifacts-with-hdf5-output/1649/8 "2021-01-22T16:35:51Z")
Hi Max, I agree with Wolfgang. If you want to use serial hdf5, you will have to rewrite all the code inside deal.II that currently uses parallel HDF5. HDF5 uses MPI IO and MPI IO might be talked into using a single writer (which is inefficient, but you might not care that much). Did you look into “cb\_config\_list” and “cb\_nodes” hints? Maybe MPI\_Info\_set(info, “cb\_nodes”, “1”); works? See [https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture33.pdf](https://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture33.pdf) As a side comment (I hope you don’t mind): It looks like you have spent hours and hours on MPI I/O and file system issues on that cluster. NFS is not a parallel file system and I doubt you can make things work well. I would cut my losses and move on: ask a sysadmin to set up a real, working parallel file system or use a different machine. You could start with an XSEDE allocation, for example.