Add support for Singularity URIs

[jdidion]

Singularity supports the use of URIs to specify container locations (http://singularity.lbl.gov/user-guide#supported-uris). Most notably, a Docker container can be run without the need for conversion using a docker:// URI.

Requested enhancements are either/both of the following:

1. Nextflow should check whether a Singularity container is a URI before converting it to an absolute path.
2. Add a 'docker = <true/false>' option to the Singularity configuration that tells whether the container is a Docker container. If true, prepend 'docker://' to the container name, else convert to absolute path.

I prefer option 2 because it enables the same workflow to be run on Mac (Docker) and Linux (Singularity), only needing to change the configuration.

If someone can provide guidance on whether either of both of these are acceptable, I will make the change and submit a PR. This is blocking me from completing a project (without a lot of work to manually convert a bunch of Docker images to Singularity images), so I'm motivated to get it done quickly.

[pditommaso]

This is an interesting proposal but it has much more pitfalls than it could seem. The first problem is the pull mechanism of Singularity is a complete different thing compared to the Docker one.

Docker has a running daemon that allow multiple pull requests to be synchronised and managed orderly. When a NF pipeline spawns, let's say 10 jobs, it invokes the execution of the same number of containers. The daemon will take care to check if required image is available and submit a single request to the Docker registry if missing.

Singularity does not have such daemon, hence when spawning the execution of more than one job by specifying a remote container, Singularity will execute an independent download for each of them, even it has been already downloaded(!)

Also the support for remote containers in Singularity is still under heavy development. The upcoming version 2.3 bring a new pull command, the support for the Singularity Hub, an easier import/export procedure, and break the support for older Docker images. I would like to see how this feature evolve before trying to implement such support in NF.

Though I see why it could be useful, I don't understand why you say this is a blocking feature. The docker2singularity tool allows you to convert your docker images to singularity ones. Then you can create two different profiles in your nextflow config file to switch easily from Docker to Singularity.

If you are uncomfortable with the docker2singularity tool, you can just wait a couple of days for Singularity 2.3 (or install the dev branch) which makes make more easier to convert a Docker image to a Singularity one.

[jdidion]

Sorry, “blocking” is overstating, I shouldn’t have used that word. For right now I’m just manually exporting containers on my desktop and copying them to the HPC. I’m glad to hear the next version of Singularity will have better support for pulls from docker and singularity hubs.

I also discovered Udocker, which I'm pleased to see NF supports although it's undocumented, but I'm guessing there's the same issue about lack of a daemon to coordinate the downloads.

[pditommaso]

Yes, true. NF includes also the support Udocker, but it was never documented because it's deadly slow to untar and launch the container images. I don't think it can be used in a production environment.

[pditommaso]

I have added the support for remote Singularity images, implementing the download, caching and synchronisation at NF level.

More info are available at this link. This should solve this feature request.

[jdidion]

!!! This is great! I don't see a link tho.

…

On Jul 31, 2017, at 6:35 AM, Paolo Di Tommaso ***@***.***> wrote: I have added the support for remote Singularity images, implementing the download, caching and synchronisation at NF level. More info are available at this link. This should solve this feature request. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[pditommaso]

Can you open this link ?

https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub

[maxulysse]

There is still a print left from development I'm guessing:

LazyDataflowVariable(value=/path/to/container.img)

[pditommaso]

Update to 0.25.4.. :)

…

On 31 Jul 2017 1:18 p.m., "Maxime Garcia" ***@***.***> wrote: There is still a print left from development I'm guessing: LazyDataflowVariable(value=/path/to/container.img) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#356 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAx3SBEsSL8VjXnDc3LytYilJg2anK9tks5sTbgQgaJpZM4NoxIJ> .

[jdidion]

That works, thanks!

…

On Jul 31, 2017, at 7:02 AM, Paolo Di Tommaso ***@***.***> wrote: Can you open this link ? https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[maxulysse]

The SNAPSHOT fix the pulling from shub as expected, thanks a lot @pditommaso