Should -resume always resubmit processes that failed but had errorStrategy set to ignore?

[chasemc]

New feature

Should -resume always resubmit processes that failed but had errorStrategy set to ignore?

Usage scenario

e.g. Running a tool on 100,000 genomes that will fail on ~10% of genomes and those will always fail on a repeat run. Should -resume always resubmit those 10%?

[bentsherman]

Indeed, failed ignored tasks will be re-tried on a resumed run. In this way, the ignore strategy is designed to ignore errors in a single workflow run but not for subsequent runs.

Whether or not they should be ignored for subsequent runs depends on the situation, i.e. why they failed. Maybe the failure is temporary and the task will succeed if you try again. Or maybe you just want to get as many as samples through as quickly as possible without getting hung up on a few failed samples. In these cases it makes sense to always try them again.

On the other hand, if there is something fundamentally wrong with the sample, such that you would rather discard it entirely, then you should implement that logic into your pipeline. You could catch the error in your task script and emit it as a "failed" sample. It is somewhat clunky to implement in Nextflow but should be possible.

I think the GEMmaker pipeline that I used to work on would write failed samples to a text file of "samples to ignore", then skip them on future runs.

[Rohit-Satyam]

It would be nice to have this as an inbuilt in nextflow as nextflow anyways keep a track of failed samples. I am still struggling on how can I get save the indexes of failed run and remove them from the input channel of the process if custom parameter rerunFailed is set to false.

[bentsherman]

@pditommaso what do you think about adding a runtime setting like workflow.skipIgnoredOnResume to control whether to re-execute ignored tasks from a previous run. It is false by default but setting to true would allow failed samples to be always skipped.

On the other hand, implementing this logic explicitly in the pipeline might be clearer, and we could write up a snippet to show how to do this.

[pditommaso]

Let's keep this as discussion point in the new resume model required for the work dir cleanup