"Singularity" and Aspect

ASPECT
1

Is anyone using Singularity for running Aspect on HPC? I only just starting experimenting with it, but as far as I can tell Singularity is better than Docker for scientific software in just about every way. For one thing, you don’t need to worry about binding local directories or copying things in/out of the container. It’s all pretty transparent to the user. Obtaining and running Aspect would look something like:

wget "https://foo.bar/aspect-v2.sif"
singularity exec aspect-v2.sif aspect input.prm

And the same commands should work identically on a cluster or a laptop or whatever else. XSEDE claims to support it, and I’m pretty sure it is available on Stampede2 (don’t quote me on that), but it doesn’t seem to be available on Expanse (yet).

Searching this forum, it looks like there’s only one mention of it in this old thread. The question that @tjhei raised about how MPI works within a container is answered in the documentation. It seems straightforward, but it does appear that there may be some gotchas that make a single distribution image slightly less portable. Namely, the MPI library in the container has to be compatible with the MPI version used on the cluster, and ideally the image should be configured with a specific cluster’s hardware in mind.

The only reason I could see for not dropping Docker entirely is that Singularity can run Docker images, but Docker is not compatible with Singularity files. So if it turns out there is some use case that’s better suited to Docker, then maybe it’s worth keeping that backwards compatibility. I’m usually sort of cautious about committing to new platforms, but in this case I’m struggling to find any downsides.

I’m interested to hear if anyone has experience with this already. Is there any advantage to Docker (or even VirtualBox, for that matter) over Singularity?

2

I have not used Singularity (except playing with it for a few minutes a long time ago). I see the following:
For workstation/laptop/testing usage, docker is probably the better choice. It has a much wider user base and we as the community already have experience with this (all our testing infrastructure on GitHub uses Docker).
For high performance computing (>1 node) you need to a) use the natively installed and tuned MPI libraries and b) apply machine specific optimizations (tuned BLAS, vectorization AVX2 vs AVX512, etc.).
Docker can not provide a) as MPI is shipped inside the container and b) is also not an option (unless you want to create a docker image for every imaginable machine). Singularity aims to solve the issue a) (as you mention) but this is still a hard problem. I am not sure how b) can be used without compiling for the specific architecture. Another aspect is that singularity is installed on super computers while docker is likely not (as it gives you direct root access).

You might say AVX2 vs AVX512 is not a big issue, but this can make a measurable difference when using the GMG solver. Modern AMD processors do not have AVX512 support, while Intel does.

So, in summary: Singularity has a significant advantage over Docker for >1 machine as it can use native MPI, but I don’t think it is promising enough that I would want to invest time to support it.