Soft matrix support #136

azedinez · 2023-06-15T16:29:49Z

Added support for non Affymetrix platforms via download of SOFT matrices directly from GEO. These matrices are already quantile normalised, but to be comparable to the output of JUSTRMA, a log2 transformation is applied (if applicable).

Tested with added profile test_soft. Also tested on a couple of beadchip mouse datasets.

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/differentialabundance branch on the nf-core/test-datasets repository.
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

…y] track

… array platform)

github-actions · 2023-06-15T16:31:29Z

`nf-core lint` overall result: Passed ✅

Posted for pipeline commit 45dd543

+| ✅ 158 tests passed       |+

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-differentialabundance_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-differentialabundance_logo_light.png
files_exist - File found: docs/images/nf-core-differentialabundance_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowDifferentialabundance.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-differentialabundance_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.3.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://github.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-differentialabundance_logo_light.png matches the template
files_unchanged - docs/images/nf-core-differentialabundance_logo_light.png matches the template
files_unchanged - docs/images/nf-core-differentialabundance_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreTemplate.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (116 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.9
Run at 2023-09-11 15:49:36

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

…cess

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

azedinez · 2023-08-25T15:18:51Z

Apologies for delay in addressing this. I believe I've addressed all your comments but let me know if I need to address more stuff. Otherwise feel free to merge.

pinin4fjords · 2023-08-28T10:41:49Z

@azedinez I had to make a number of fixes and tweaks to make this actually work (and the tests are now running in CI). Could you please test locally to make sure outputs are as you would like?

azedinez · 2023-09-11T15:41:33Z

@pinin4fjords Tested locally and runs. All output looks as expected. I have just added the soft config to the pipeline for convenience.

…abundance into soft_matrix_support

pinin4fjords

Great- we've got there. Thanks for the work @azedinez !

Azedine Zoufir added 28 commits June 13, 2023 10:46

R script read from soft matrix

618a90d

add read_from_soft process to pipeline

5555340

add configuration for read_from_soft process

1d2f9ec

incorporate read from soft process into workflow

0b48cef

add querygse as an input parameter to workflow

4b4aad6

add querygse to schema

27ec512

add 'non_affy_array' as possible value for study type param

cfa223c

fixing configs for read_from_soft process

8e32b08

fix geoquery container url in read_from_soft.nf

3af5ed3

remove raw matrix processing for [non_affy_array] track

1227674

remove raw matrix processing for [non_affy_array] track

baaf791

attempt to fix premature completion of pipeline

e1ff15c

attempt to fix premature completion of pipeline

3b773a8

debug around contrasts & matrix filter

5068295

debug around contrasts & matrix filter

03e30fd

debug matrix input processing for exploratory plots in [non_affy_arra…

054275a

…y] track

do not filter matrix in [non_affy_array] track

4092a93

revert debug and apply log2 transformation to READ_FROM_SOFT output

7cc0a17

add test SOFT matrix track config

8df1f28

add test SOFT matrix track config

b4b8f8b

fix typo in config file

735999a

add extra input to READ_SOFT_MATRIX to process metacolumns (differ by…

ab1cb71

… array platform)

fix typo in config

af0518e

fix typo in config

872023d

fix parameter name in read_soft_matrix

e7bfdc6

fix parameter name in read_soft_matrix

52929b3

change column names to GSM ids in configuration, to match SOFT format

94b9b26

update docs

ed95ab4

azedinez requested a review from pinin4fjords June 15, 2023 16:29

Azedine Zoufir and others added 8 commits August 25, 2023 15:51

remove usage of alias for GEOQUERY_GETGEO

3fa023e

remove read_from_soft files from local folder

6da3cac

fix spacing in conf/modules.config

db1ada5

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

remove querygse as external arg in conf/modules.config

eaf597f

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

add querygse as input for channel workflows/differentialabundance.nf

4a419e4

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

revert to using ch_query_gse as input channel for GEOQUERY_GETGEO pro…

2f4f6fc

…cess

delete join on ch_querygse in workflows/differentialabundance.nf

843c653

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

delete mixing versions for JUSTRMA_NORM

b782445

Co-authored-by: Jonathan Manning <jonathan.manning@deeplife.co>

pinin4fjords added 11 commits August 28, 2023 09:36

Correctly refer to matrices from getgeo as norm

dc1c337

Make sure supplying metadata cols is optional

fed96cc

Sort out config for SOFT

62feb14

prettier

248756c

Fix soft config import

281a18e

Fix channel name

430f530

Fix config selector

33389fc

Fix channel definition

cfaf8eb

Fix CHANGELOG

1c842bb

Final fixes

9790859

limit resources for testing

abed797

add soft config

823b4cd

pinin4fjords added 2 commits September 11, 2023 16:47

Remove rogue tab

6e2480c

Merge branch 'soft_matrix_support' of github.com:nf-core/differential…

45dd543

…abundance into soft_matrix_support

pinin4fjords approved these changes Sep 11, 2023

View reviewed changes

azedinez merged commit 8d61010 into dev Sep 12, 2023
12 checks passed

azedinez deleted the soft_matrix_support branch September 12, 2023 08:12

pinin4fjords mentioned this pull request Oct 6, 2023

Rlog + other small fixes #164

Merged

9 tasks

pinin4fjords added this to the 1.3.0 milestone Oct 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Soft matrix support #136

Soft matrix support #136

azedinez commented Jun 15, 2023 •

edited

Loading

github-actions bot commented Jun 15, 2023 •

edited

Loading

✅ Tests passed:

Run details

azedinez commented Aug 25, 2023

pinin4fjords commented Aug 28, 2023

azedinez commented Sep 11, 2023

pinin4fjords left a comment

Soft matrix support #136

Soft matrix support #136

Conversation

azedinez commented Jun 15, 2023 • edited Loading

PR checklist

github-actions bot commented Jun 15, 2023 • edited Loading

nf-core lint overall result: Passed ✅

✅ Tests passed:

Run details

azedinez commented Aug 25, 2023

pinin4fjords commented Aug 28, 2023

azedinez commented Sep 11, 2023

pinin4fjords left a comment

Choose a reason for hiding this comment

azedinez commented Jun 15, 2023 •

edited

Loading

github-actions bot commented Jun 15, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅