Singularity support

[ps-account]

Hi, a forked project already seems to include Singularity support, is it possible to include this in the main toil project? Many HPC providers only allow Singularity containers http://singularity.lbl.gov/

Example of Singularity support in toil_vg: https://github.com/vgteam/toil-vg/blob/master/src/toil_vg/singularity.py

Info on how to use Singularity via Toil in the manual would also be helpful.

Gr. Pim

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-178

[ps-account]

The implementation could also be similar to: nextflow-io/nextflow#356

[jessebrennan]

I don't know anything about singularity personally. @glennhickey Is their any reason we cannot just merge your singularity changes into Toil proper?

[glennhickey]

Not really, it's completely independent of anything so it won't hurt. Still, we don't have any unit tests (would need singularity installed on Jenkins), and it's much slower when running Docker images than calling docker directly.

…

On Mon, Jul 31, 2017 at 1:05 PM, jessebrennan ***@***.***> wrote: I don't know anything about singularity personally. @glennhickey <https://github.com/glennhickey> Is their any reason we cannot just merge your singularity changes into Toil proper? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1768 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2_7lOq31QruiJsj1eVoiOHgFTT9wQxks5sTgllgaJpZM4OfTVw> .

[cgcloud-jenkins]

I think it would be good to have this support in Toil, because it allows deployment on HPC systems. On Mon, Jul 31, 2017 at 12:04 PM, Glenn Hickey <notifications@github.com> wrote:

…

Not really, it's completely independent of anything so it won't hurt. Still, we don't have any unit tests (would need singularity installed on Jenkins), and it's much slower when running Docker images than calling docker directly. On Mon, Jul 31, 2017 at 1:05 PM, jessebrennan ***@***.***> wrote: > I don't know anything about singularity personally. @glennhickey > <https://github.com/glennhickey> Is their any reason we cannot just merge > your singularity changes into Toil proper? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1768 (comment) >, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AA2_ 7lOq31QruiJsj1eVoiOHgFTT9wQxks5sTgllgaJpZM4OfTVw> > . > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1768 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMXKjrikKmtT_WMdcZLuyuoJlSugDjgyks5sTiVGgaJpZM4OfTVw> . -- You received this message because you are subscribed to the Google Groups "cgloud-jenkins" group. To unsubscribe from this group and stop receiving emails from it, send an email to ***@***.*** To post to this group, send email to ***@***.*** To view this discussion on the web visit https://groups.google.com/d/ msgid/cgloud-jenkins/BD2KGenomics/toil/issues/1768/319165308%40github.51.al <https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil/issues/1768/319165308%40github.51.al?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout.

[glennhickey]

Sure, I'll make a PR. Just to be clear though, we've only been using Singularity to run command-line programs from Toil scripts from single-machine batch runs, and not to deploy Toil itself. (Speaking of singularities, has Jenkins finally become sentient?) On Mon, Jul 31, 2017 at 3:18 PM, *Jenkins* <notifications@github.com> *wrote*:

…

I think it would be good to have this support in Toil, because it allows deployment on HPC systems. On Mon, Jul 31, 2017 at 12:04 PM, Glenn Hickey ***@***.***> wrote: > Not really, it's completely independent of anything so it won't hurt. > Still, we don't have any unit tests (would need singularity installed on > Jenkins), and it's much slower when running Docker images than calling > docker directly. > > On Mon, Jul 31, 2017 at 1:05 PM, jessebrennan ***@***.***> > wrote: > > > I don't know anything about singularity personally. @glennhickey > > <https://github.com/glennhickey> Is their any reason we cannot just > merge > > your singularity changes into Toil proper? > > > > — > > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > > <#1768 issuecomment-319132274 > >, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/AA2_ > 7lOq31QruiJsj1eVoiOHgFTT9wQxks5sTgllgaJpZM4OfTVw> > > . > > > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1768 (comment) >, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AMXKjrikKmtT_ WMdcZLuyuoJlSugDjgyks5sTiVGgaJpZM4OfTVw> > . > > -- > You received this message because you are subscribed to the Google Groups > "cgloud-jenkins" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ***@***.*** > To post to this group, send email to ***@***.*** > To view this discussion on the web visit https://groups.google.com/d/ > msgid/cgloud-jenkins/BD2KGenomics/toil/issues/1768/319165308% 40github.51.al > <https://groups.google.com/d/msgid/cgloud-jenkins/ DataBiosphere/toil#1768/319165308%40github.51.al?utm_ medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1768 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2_7rmE0Gsk6kbxkmdZSl83ohDqJYUgks5sTiiUgaJpZM4OfTVw> .

[ps-account]

yes it's safer to use than Docker as it limits itself to userspace while still configurable to access accellerators, Infiniband, etc., so it's rapidly becoming the container-system of choice in big (and small) HPC centers. I'm not an expert on the inner workings but it seems similar enough in use to Docker that adding support for Singularity next to Docker should be feasible without heavy rewriting.

[ps-account]

There is one additional remarkt wrt the implementation in toil_vg, though. If I understand correctly, Toil_vg is slower in accessing containers because it converts Docker images to Singularity images. It is however possible to directly access a Singularity hub, I think it is implemented like that in nextflow-io

[ps-account]

I think the request as formulated to nextflow seems therefore a good way to implement singularity support in a workflow system:

Singularity supports the use of URIs to specify container locations (http://singularity.lbl.gov/user-guide#supported-uris). Most notably, a Docker container can be run without the need for conversion using a docker:// URI.

Requested enhancements are either/both of the following:
1.Nextflow should check whether a Singularity container is a URI before converting it to an absolute path.
2.Add a 'docker = <true/false>' option to the Singularity configuration that tells whether the container is a Docker container. If true, prepend 'docker://' to the container name, else convert to absolute path.

[alanhoyle]

We've been moving our containerized workflows to run docker-sourced images in singularity on our local SLURM cluster, and having a tool that allows us to use CWL would be useful. Has there been any progress? I checked the "toil_container" fork, and that one seems to not support CWL.

[mr-c]

@alanhoyle toil-cwl-runner natively supports running Docker format containers using the Singularity runtime

[alanhoyle]

If anyone else is wondering, this is apparently done by having the following on the command line:

toil-cwl-runner --singularity ...

[illusional]

Also, if you don't have node installed, Toil will try to start a docker container with it. To install node without root privileges, you can download and compile it yourself via the first gist:

https://gist.github.com/isaacs/579814#file-node-and-npm-in-30-seconds-sh

[multimeric]

Some of this discussion has been about how to make run Toil's CWL runner use Singularity, but I'm also interested in how it might work for pure Python workflows.

The way I envisage this might work is:

• We add a singularityCall function along the lines of the toil_vg implementation above. Ideally, though, since there's now an official Python API for Python (https://singularityhub.github.io/singularity-cli/), it might be better and easier to use that
• Then, we add a containerCall function that acts as like an abstract layer above Singularity and Docker. It checks the config to see if there was a --containerEngine flag provided, or else defaults to Docker if it wasn't. Then it defers to singularityCall or dockerCall based on this.
• Then we deprecate direct use of singularityCall and dockerCall and encourage new workflows to use containerCall to make these workflows as portable as possible

Thoughts?

[adamnovak]

@multimeric it would be great to have Singularity support alongside the Docker support Toil already provides. It would also be good if containerCall could have a not-actually-in-a-container backend, like toil-vg supports.

One potential issue with actually using the Python Singularity API is that I had to hack Toil's appliance images to wrap Singularity, so it would pull all Docker Hub images through our caching proxy, because Singularity doesn't use a config file with a setting that can do that. If we use the Python API, there's no more command line call to wrap, so that logic would need to be included in singularityCall.

[mr-c]

One potential issue with actually using the Python Singularity API is that I had to hack Toil's appliance images to wrap Singularity, so it would pull all Docker Hub images through our caching proxy, because Singularity doesn't use a config file with a setting that can do that.

Is there an issue for this filed with both the Singularity Python API issue tracker and the Singularity issue tracker? Shouldn't be too much work to fix and it would help a lot of others, I'm sure.

[multimeric]

Yeah do you think you could file an issue against Singularity and link it here, @adamnovak? If and when it's solved in Singularity then it should be pretty simple to implement the rest of this feature in toil.

[adamnovak]

I don't think the Python API needs to worry about mirrors on its own, if it ends up invoking the same Singularity code that would then respect the same configuration, but the issue in Singularity itself for supporting registry mirrors is apptainer/singularity#5780.

[DailyDreaming]

We have singularity support and documentation around it. If support (or docs) could be improved please reopen this issue. For CWL it's --singularity and WDL it's --container='singularity'. CWL defaults to docker, and WDL defaults to singularity.