fix: spark issues

[maxulysse]

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - add to the software_versions process and a regex to scrape_software_versions.py
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
• Make sure your code lints (nf-core lint .).
• Ensure the test suite passes (nextflow run . -profile test,docker).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[github-actions]

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit ae806d9

+| ✅ 262 tests passed       |+
#| ❔   5 tests were ignored |#
!| ❗  29 tests had warnings |!

❗ Test warnings:

• params_used - Config variable not found in main.nf: params.cpus
• params_used - Config variable not found in main.nf: params.single_cpu_mem
• actions_awsfulltest - .github/workflows/awsfulltest.yml should test full datasets, not -profile test
• conda_env_yaml - Conda dep outdated: conda-forge::llvm-openmp=8.0.1=hc9558a2_0, 11.1.0 available
• conda_env_yaml - Conda dep outdated: conda-forge::markdown=3.1.1=py_0, 3.3.4 available
• conda_env_yaml - Conda dep outdated: conda-forge::pymdown-extensions=6.0=py_0, 8.2 available
• conda_env_yaml - Conda dep outdated: conda-forge::pygments=2.5.2=py_0, 2.9.0 available
• conda_env_yaml - Conda dep outdated: bioconda::bcftools=1.9=ha228f0b_4, 1.12 available
• conda_env_yaml - Conda dep outdated: bioconda::bwa-mem2=2.0=he513fc3_1, 2.2.1 available
• conda_env_yaml - Conda dep outdated: bioconda::cancerit-allelecount=4.0.2=ha228f0b_1, 4.2.1 available
• conda_env_yaml - Conda dep outdated: bioconda::cnvkit=0.9.6=py27_1, 0.9.8 available
• conda_env_yaml - Conda dep outdated: bioconda::ensembl-vep=99.2=pl526hecc5488_0, 104.2 available
• conda_env_yaml - Conda dep outdated: bioconda::fgbio=1.1.0=0, 1.3.0 available
• conda_env_yaml - Conda dep outdated: bioconda::freebayes=1.3.2=py27h49fb759_2, 1.3.5 available
• conda_env_yaml - Conda dep outdated: bioconda::gatk4-spark=4.1.7.0=0, 4.2.0.0 available
• conda_env_yaml - Conda dep outdated: bioconda::htslib=1.9=ha228f0b_7, 1.12 available
• conda_env_yaml - Conda dep outdated: bioconda::multiqc=1.8=py_2, 1.10.1 available
• conda_env_yaml - Conda dep outdated: bioconda::samblaster=0.1.24=hc9558a2_3, 0.1.26 available
• conda_env_yaml - Conda dep outdated: bioconda::samtools=1.9=h10a08f8_12, 1.12 available
• conda_env_yaml - Conda dep outdated: bioconda::snpeff=4.3.1t=0, 5.0 available
• conda_env_yaml - Conda dep outdated: bioconda::tiddit=2.7.1=py27hb3f55d8_1, 2.12.1 available
• conda_env_yaml - Conda dep outdated: bioconda::trim-galore=0.6.5=0, 0.6.6 available
• conda_env_yaml - Conda dep outdated: conda-forge::pigz=2.3.4=hed695b0_1, 2.6 available
• conda_env_yaml - Conda dep outdated: conda-forge::r-ggplot2=3.3.0=r40h6115d3f_1, 3.3.3 available
• pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
• pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
• pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
• pipeline_todos - TODO string in awstest.yml: You can customise CI pipeline run tests as required
• schema_description - No description provided in schema for parameter: cpus

❔ Tests ignored:

• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.md
• files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
• files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
• files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File found: .github/markdownlint.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: assets/nf-core-sarek_logo.png
• files_exist - File found: bin/markdown_to_html.py
• files_exist - File found: docs/images/nf-core-sarek_logo.png
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: lib/nfcore_external_java_deps.jar
• files_exist - File found: lib/NfcoreSchema.groovy
• files_exist - File found: main.nf
• files_exist - File found: environment.yml
• files_exist - File found: Dockerfile
• files_exist - File found: conf/base.config
• files_exist - File found: .github/workflows/awstest.yml
• files_exist - File found: .github/workflows/awsfulltest.yml
• files_exist - File not found check: Singularity
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.name
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: manifest.homePage
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.show_hidden_params
• nextflow_config - Config variable found: params.schema_ignore_params
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable found: process.container
• nextflow_config - Config variable (correctly) not found: params.version
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config manifest.name began with nf-core/
• nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
• nextflow_config - Config dag.file ended with .svg
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config process.container looks correct: nfcore/sarek:dev
• nextflow_config - Config manifest.version ends in dev: '3.0dev'
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://github.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• params_used - Config variable found in main.nf: params.input
• params_used - Config variable found in main.nf: params.step
• params_used - Config variable found in main.nf: params.genome
• params_used - Config variable found in main.nf: params.genomes_base
• params_used - Config variable found in main.nf: params.save_reference
• params_used - Config variable found in main.nf: params.help
• params_used - Config variable found in main.nf: params.no_intervals
• params_used - Config variable found in main.nf: params.nucleotides_per_second
• params_used - Config variable found in main.nf: params.sentieon
• params_used - Config variable found in main.nf: params.skip_qc
• params_used - Config variable found in main.nf: params.target_bed
• params_used - Config variable found in main.nf: params.tools
• params_used - Config variable found in main.nf: params.trim_fastq
• params_used - Config variable found in main.nf: params.clip_r1
• params_used - Config variable found in main.nf: params.clip_r2
• params_used - Config variable found in main.nf: params.three_prime_clip_r1
• params_used - Config variable found in main.nf: params.three_prime_clip_r2
• params_used - Config variable found in main.nf: params.trim_nextseq
• params_used - Config variable found in main.nf: params.save_trimmed
• params_used - Config variable found in main.nf: params.split_fastq
• params_used - Config variable found in main.nf: params.aligner
• params_used - Config variable found in main.nf: params.markdup_java_options
• params_used - Config variable found in main.nf: params.use_gatk_spark
• params_used - Config variable found in main.nf: params.save_bam_mapped
• params_used - Config variable found in main.nf: params.skip_markduplicates
• params_used - Config variable found in main.nf: params.ascat_ploidy
• params_used - Config variable found in main.nf: params.ascat_purity
• params_used - Config variable found in main.nf: params.cf_coeff
• params_used - Config variable found in main.nf: params.cf_contamination
• params_used - Config variable found in main.nf: params.cf_contamination_adjustment
• params_used - Config variable found in main.nf: params.cf_ploidy
• params_used - Config variable found in main.nf: params.cf_window
• params_used - Config variable found in main.nf: params.generate_gvcf
• params_used - Config variable found in main.nf: params.no_strelka_bp
• params_used - Config variable found in main.nf: params.pon
• params_used - Config variable found in main.nf: params.pon_index
• params_used - Config variable found in main.nf: params.ignore_soft_clipped_bases
• params_used - Config variable found in main.nf: params.umi
• params_used - Config variable found in main.nf: params.read_structure1
• params_used - Config variable found in main.nf: params.read_structure2
• params_used - Config variable found in main.nf: params.annotate_tools
• params_used - Config variable found in main.nf: params.annotation_cache
• params_used - Config variable found in main.nf: params.cadd_cache
• params_used - Config variable found in main.nf: params.cadd_indels
• params_used - Config variable found in main.nf: params.cadd_indels_tbi
• params_used - Config variable found in main.nf: params.cadd_wg_snvs
• params_used - Config variable found in main.nf: params.cadd_wg_snvs_tbi
• params_used - Config variable found in main.nf: params.genesplicer
• params_used - Config variable found in main.nf: params.snpeff_cache
• params_used - Config variable found in main.nf: params.vep_cache
• params_used - Config variable found in main.nf: params.config_profile_contact
• params_used - Config variable found in main.nf: params.config_profile_description
• params_used - Config variable found in main.nf: params.config_profile_url
• params_used - Config variable found in main.nf: params.outdir
• params_used - Config variable found in main.nf: params.publish_dir_mode
• params_used - Config variable found in main.nf: params.sequencing_center
• params_used - Config variable found in main.nf: params.multiqc_config
• params_used - Config variable found in main.nf: params.monochrome_logs
• params_used - Config variable found in main.nf: params.email
• params_used - Config variable found in main.nf: params.email_on_fail
• params_used - Config variable found in main.nf: params.plaintext_email
• params_used - Config variable found in main.nf: params.max_multiqc_email_size
• params_used - Config variable found in main.nf: params.hostnames
• params_used - Config variable found in main.nf: params.validate_params
• params_used - Config variable found in main.nf: params.tracedir
• params_used - Config variable found in main.nf: params.max_cpus
• params_used - Config variable found in main.nf: params.max_memory
• params_used - Config variable found in main.nf: params.max_time
• params_used - Config variable found in main.nf: params.fasta
• params_used - Config variable found in main.nf: params.ac_loci
• params_used - Config variable found in main.nf: params.ac_loci_gc
• params_used - Config variable found in main.nf: params.bwa
• params_used - Config variable found in main.nf: params.chr_dir
• params_used - Config variable found in main.nf: params.chr_length
• params_used - Config variable found in main.nf: params.dbsnp
• params_used - Config variable found in main.nf: params.dbsnp_index
• params_used - Config variable found in main.nf: params.dict
• params_used - Config variable found in main.nf: params.fasta_fai
• params_used - Config variable found in main.nf: params.germline_resource
• params_used - Config variable found in main.nf: params.germline_resource_index
• params_used - Config variable found in main.nf: params.intervals
• params_used - Config variable found in main.nf: params.known_indels
• params_used - Config variable found in main.nf: params.known_indels_index
• params_used - Config variable found in main.nf: params.mappability
• params_used - Config variable found in main.nf: params.snpeff_db
• params_used - Config variable found in main.nf: params.species
• params_used - Config variable found in main.nf: params.vep_cache_version
• files_unchanged - .gitattributes matches the template
• files_unchanged - CODE_OF_CONDUCT.md matches the template
• files_unchanged - LICENSE matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/CONTRIBUTING.md matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
• files_unchanged - .github/markdownlint.yml matches the template
• files_unchanged - .github/workflows/branch.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - .github/workflows/linting.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/sendmail_template.txt matches the template
• files_unchanged - bin/markdown_to_html.py matches the template
• files_unchanged - docs/README.md matches the template
• files_unchanged - lib/nfcore_external_java_deps.jar matches the template
• files_unchanged - lib/NfcoreSchema.groovy matches the template
• files_unchanged - .gitignore matches the template
• files_unchanged - .github/workflows/push_dockerhub_dev.yml matches the template
• files_unchanged - .github/workflows/push_dockerhub_release.yml matches the template
• files_unchanged - assets/multiqc_config.yaml matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - CI is building the correct docker image: docker build --no-cache . -t nfcore/sarek:dev
• actions_ci - CI is pulling the correct docker image: docker pull nfcore/sarek:dev
• actions_ci - CI is tagging docker image correctly: docker tag nfcore/sarek:dev nfcore/sarek:dev
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
• readme - README Nextflow minimum version badge matched config. Badge: 20.04.0, Config: 20.04.0
• readme - README had a bioconda badge
• conda_env_yaml - Conda environment name was correct (nf-core-sarek-3.0dev)
• conda_env_yaml - Conda dep had pinned version number: conda-forge::llvm-openmp=8.0.1=hc9558a2_0
• conda_env_yaml - Conda dep had pinned version number: conda-forge::openmp=8.0.1=0
• conda_env_yaml - Conda package is the latest available: conda-forge::openmp=8.0.1=0
• conda_env_yaml - Conda dep had pinned version number: conda-forge::markdown=3.1.1=py_0
• conda_env_yaml - Conda dep had pinned version number: conda-forge::pymdown-extensions=6.0=py_0
• conda_env_yaml - Conda dep had pinned version number: conda-forge::pygments=2.5.2=py_0
• conda_env_yaml - Conda dep had pinned version number: bioconda::ascat=2.5.2=r40_2
• conda_env_yaml - Conda package is the latest available: bioconda::ascat=2.5.2=r40_2
• conda_env_yaml - Conda dep had pinned version number: bioconda::bcftools=1.9=ha228f0b_4
• conda_env_yaml - Conda dep had pinned version number: bioconda::bwa-mem2=2.0=he513fc3_1
• conda_env_yaml - Conda dep had pinned version number: bioconda::bwa=0.7.17=hed695b0_7
• conda_env_yaml - Conda package is the latest available: bioconda::bwa=0.7.17=hed695b0_7
• conda_env_yaml - Conda dep had pinned version number: bioconda::cancerit-allelecount=4.0.2=ha228f0b_1
• conda_env_yaml - Conda dep had pinned version number: bioconda::cnvkit=0.9.6=py27_1
• conda_env_yaml - Conda dep had pinned version number: bioconda::control-freec=11.6=he1b5a44_0
• conda_env_yaml - Conda package is the latest available: bioconda::control-freec=11.6=he1b5a44_0
• conda_env_yaml - Conda dep had pinned version number: bioconda::ensembl-vep=99.2=pl526hecc5488_0
• conda_env_yaml - Conda dep had pinned version number: bioconda::fastqc=0.11.9=0
• conda_env_yaml - Conda package is the latest available: bioconda::fastqc=0.11.9=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::fgbio=1.1.0=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::freebayes=1.3.2=py27h49fb759_2
• conda_env_yaml - Conda dep had pinned version number: bioconda::gatk4-spark=4.1.7.0=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::genesplicer=1.0=1
• conda_env_yaml - Conda package is the latest available: bioconda::genesplicer=1.0=1
• conda_env_yaml - Conda dep had pinned version number: bioconda::htslib=1.9=ha228f0b_7
• conda_env_yaml - Conda dep had pinned version number: bioconda::manta=1.6.0=py27_0
• conda_env_yaml - Conda package is the latest available: bioconda::manta=1.6.0=py27_0
• conda_env_yaml - Conda dep had pinned version number: bioconda::msisensor=0.5=h25a10a7_1
• conda_env_yaml - Conda package is the latest available: bioconda::msisensor=0.5=h25a10a7_1
• conda_env_yaml - Conda dep had pinned version number: bioconda::multiqc=1.8=py_2
• conda_env_yaml - Conda dep had pinned version number: bioconda::qualimap=2.2.2d=1
• conda_env_yaml - Conda package is the latest available: bioconda::qualimap=2.2.2d=1
• conda_env_yaml - Conda dep had pinned version number: bioconda::samblaster=0.1.24=hc9558a2_3
• conda_env_yaml - Conda dep had pinned version number: bioconda::samtools=1.9=h10a08f8_12
• conda_env_yaml - Conda dep had pinned version number: bioconda::snpeff=4.3.1t=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::strelka=2.9.10=0
• conda_env_yaml - Conda package is the latest available: bioconda::strelka=2.9.10=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::tiddit=2.7.1=py27hb3f55d8_1
• conda_env_yaml - Conda dep had pinned version number: bioconda::trim-galore=0.6.5=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::vcfanno=0.3.2=0
• conda_env_yaml - Conda package is the latest available: bioconda::vcfanno=0.3.2=0
• conda_env_yaml - Conda dep had pinned version number: bioconda::vcftools=0.1.16=he513fc3_4
• conda_env_yaml - Conda package is the latest available: bioconda::vcftools=0.1.16=he513fc3_4
• conda_env_yaml - Conda dep had pinned version number: conda-forge::pigz=2.3.4=hed695b0_1
• conda_env_yaml - Conda dep had pinned version number: conda-forge::r-ggplot2=3.3.0=r40h6115d3f_1
• conda_dockerfile - Found all expected strings in Dockerfile file
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (115 files)
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_params - Schema matched params returned from nextflow config
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: push_dockerhub_dev.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: awsfulltest.yml
• actions_schema_validation - Workflow validation passed: push_dockerhub_release.yml
• actions_schema_validation - Workflow validation passed: awstest.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• merge_markers - No merge markers found in pipeline files

Run details

• nf-core/tools version 1.14
• Run at 2021-05-27 07:25:58

[FriederikeHanssen]

Does this also fix @heuermh problems with the other modules?

[mahesh-panchal]

Out of curiosity what is the difference between fixOwnership and setting the user/group? Does that mean the user needs to be root (or whatever user is set by the container environment) in the docker container? If it needs another user, why doesn't this affect running in a singularity container, which automatically runs as user/group?

[maxulysse]

Basically it's just 2 way of doing the same thing.
The fixOwnership use chown to set up the owner of the files after the process is run.
The runOptions we have, change directly that in Docker, but $HOME becomes / instead of the default /root, which could make some tools fail (when they use $HOME).

[mahesh-panchal]

That would mean it's trying to use some files in /root, since the containers aren't usually started with a writable layer. Or do I misunderstand?
Otherwise I guess one could set the variables in the run option. I also see from the Nextflow code that there is a userEmulation flag that's not documented, which might set all that.

[maxulysse]

@nf-core/core @nf-core/sarek @heuermh @mahesh-panchal Can people test out and tell me if it works?
Thanks a lot, I'd really like to merge this one and prepare a minor release

[pontus]

To reiterate (discussion in slack):

As of today, userEmulation includes the user runOptions as replaced, including mapping passwd and related files and home directory.

It will help in some cases, but will likely not change on e.g. macOS or if using some NSS-provider for user information (common for e.g. cluster setups and if using e.g. AD login).

It will also try to map the home directory which brings (in general) the risk of breaking stuff if something is picked up from the home directory instead of the container.

Both of these issues are not generally likely to appear and should be guarded against in the container (e.g. singularity by default maps the home directory, so it must be protected against anyway).

Overall, this is probably a slight improvement.

[mahesh-panchal]

I should add that userEmulation is an undocumented option.
It's in the Nextflow code here though (https://github.com/nextflow-io/nextflow/blob/b41d126ddde2cf839fb70b57da99b9fd85e2dbb8/src/main/groovy/nextflow/util/DockerBuilder.groovy#L60 , https://github.com/nextflow-io/nextflow/blob/b41d126ddde2cf839fb70b57da99b9fd85e2dbb8/src/main/groovy/nextflow/util/DockerBuilder.groovy#L107)

[maxulysse]

I should add that userEmulation is an undocumented option.

I remember Paolo saying that the code is the documentation.
It was found, explained, we're now using it, it is hence documented 🤓

[maxulysse]

Comment from @pditommaso on Slack:

userEmulation is really an early days Docker hack, don’t use it

[pontus]

An alternative way of expressing it would be that spark requires special care. For the current sarek approach that's addressable, but for DSL2 I see no solution that doesn't involve maintaining a container for it.