Skip to content
This repository has been archived by the owner on Jan 27, 2020. It is now read-only.

Commit

Permalink
Merge pull request #698 from MaxUlysse/BTB
Browse files Browse the repository at this point in the history
Add configuration profile for munin
  • Loading branch information
maxulysse committed Dec 17, 2018
2 parents e8469c6 + 1901ccd commit 93a0bfb
Show file tree
Hide file tree
Showing 5 changed files with 262 additions and 52 deletions.
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
- [#679](https://github.com/SciLifeLab/Sarek/pull/679) - Add container for `CreateIntervalBeds`
- [#692](https://github.com/SciLifeLab/Sarek/pull/692), [#697](https://github.com/SciLifeLab/Sarek/pull/697) - Add AWS iGenomes possibilities (within `conf/igenomes.conf`)
- [#694](https://github.com/SciLifeLab/Sarek/pull/694) - Add monochrome and grey logos for light or dark background
- [#698](https://github.com/SciLifeLab/Sarek/pull/698) - Add btb profile for munin server
- [#702](https://github.com/SciLifeLab/Sarek/pull/702) - Add font-ttf-dejavu-sans-mono `2.37` and fontconfig `2.12.6` to container

### `Changed`

- [#678](https://github.com/SciLifeLab/Sarek/pull/678) - Changing VEP to v92 and adjusting CPUs for VEP
- [#663](https://github.com/SciLifeLab/Sarek/pull/663) - Update `do_release.sh` script
- [#671](https://github.com/SciLifeLab/Sarek/pull/671) - publishDir modes are now params
- [#677](https://github.com/SciLifeLab/Sarek/pull/677) - Update docs
- [#677](https://github.com/SciLifeLab/Sarek/pull/677), [#698](https://github.com/SciLifeLab/Sarek/pull/698) - Update docs
- [#679](https://github.com/SciLifeLab/Sarek/pull/679) - Update old awsbatch configuration
- [#682](https://github.com/SciLifeLab/Sarek/pull/682) - Specifications for memory and cpus for awsbatch
- [#693](https://github.com/SciLifeLab/Sarek/pull/693) - Qualimap bamQC is now ran after mapping and after recalibration for better QC
Expand Down
140 changes: 140 additions & 0 deletions conf/munin.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
/*
* -------------------------------------------------
* Nextflow config file for Sarek
* -------------------------------------------------
* Configuration for running on munin
* -------------------------------------------------
*/

env {
NXF_WORK="/scratch"
}

params {
genome_base = params.genome == 'GRCh37' ? '/btb/references/Homo_sapiens/GATK/GRCh37/' : params.genome == 'GRCh38' ? '/btb/references/Homo_sapiens/GATK/GRCh38/' : 'References/smallGRCh37'
singleCPUMem = 15.GB
totalMemory = 754.GB
}

executor.$local.cpus = 48

process {
// Default process resources

// A process may use one core,
cpus = 1

// 15 GB of memory,
memory = {params.singleCPUMem}

// and 48 of them are allowed to be launched simultaneously.
maxForks = 48

errorStrategy = {task.exitStatus == 143 ? 'retry' : 'terminate'}
maxErrors = '-1'
maxRetries = 3

// These processes are defined in buildReferences.nf

withName:BuildBWAindexes {
memory = {params.totalMemory} // TODO This is likely too high
}
withName:BuildReferenceIndex {
memory = {params.totalMemory} // TODO This is likely too high
}
withName:BuildSAMToolsIndex {
memory = {params.totalMemory} // TODO This is likely too high
}
withName:BuildVCFIndex {
memory = {params.totalMemory} // TODO This is likely too high
}

// These processes are defined in main.nf

withName:ConcatVCF {
cpus = 8
}
withName:CreateRecalibrationTable {
cpus = 16
memory = {params.totalMemory}
}
withName:MapReads {
cpus = 48
memory = {params.totalMemory}
}
withName:MarkDuplicates {
// Actually the -Xmx value should be kept lower
cpus = 16
memory = {2 * params.singleCPUMem}
}
withName:MergeBams {
cpus = 8
memory = {params.totalMemory}
}
withName:RecalibrateBam {
memory = {params.singleCPUMem * task.attempt}
}
withName:RunAlleleCount {
memory = {params.singleCPUMem * 2 * task.attempt}
}
withName:RunAscat {
memory = {params.singleCPUMem * 2 * task.attempt}
}
withName:RunBamQCmapped {
cpus = 16
memory = {params.totalMemory}
}
withName:RunBamQCrecalibrated {
cpus = 16
memory = {params.totalMemory}
}
withName:RunBcftoolsStats {
}
withName:RunConvertAlleleCounts {
memory = {params.singleCPUMem * 2 * task.attempt}
}
withName:RunFastQC {
cpus = 2 // FastQC is only capable of running one thread per fastq file.
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
}
withName:RunFreeBayes {
memory = {params.singleCPUMem * task.attempt}
}
withName:RunHaplotypecaller {
// Increase memory quadratically
memory = {params.singleCPUMem * task.attempt * task.attempt}
}
withName:RunGenotypeGVCFs {
}
withName:RunManta {
cpus = 48
memory = {params.totalMemory}
}
withName:RunMultiQC {
}
withName:RunMutect2 {
memory = {params.singleCPUMem * task.attempt}
}
withName:RunSamtoolsStats {
}
withName:RunSingleManta {
cpus = 48
memory = {params.totalMemory}
}
withName:RunSingleStrelka {
cpus = 48
memory = {params.totalMemory}
}
withName:RunSnpeff {
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
memory = {params.singleCPUMem * task.attempt}
}
withName:RunStrelka {
cpus = 48
memory = {params.totalMemory}
}
withName:RunVEP {
cpus = 48
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
}
}
85 changes: 73 additions & 12 deletions docs/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,22 @@ The standard ones are designed to work on a Swedish UPPMAX cluster, but can be m
Every configuration file can be modified for your own use.
If you want you can specify the use of a config file using `-c <config file>`

### [`aws-batch.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/aws-batch.config)

Designed for usage with AWS batch.

### [`base.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/base.config)

Define default parameters, is included into every profiles.

### [`binac.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/binac.config)

Define usage limits and Singularity for BINAC cluster in Tuebingen.

### [`cfc.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/cfc.config)

Designed for usage with Singularity on CFC at QBic.

### [`containers.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/containers.config)

Define Containers for all process.
Expand All @@ -21,15 +37,27 @@ Use in your own profile if needed.

### [`docker.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/docker.config)

Define Docker Containers for all process.
Images will be pulled automatically.
Use in your own profile if needed.
Specify Docker options.
To be used with [`containers.config`](#containersconfig)

### [`genomes.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/genomes.config)

Contain path to all references.
Modify it if you want to change genome version, or the path to your references files.

### [`igenomes.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/igenomes.config)

Contain path to all AWS iGenomes references.
Modify it if you want to change genome version, or the path to your references files.

### [`munin.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/munin.config)

Define usage limits and Singularity for munin server at BTB.

### [`resources.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/resources.config)

Define Generalized resource configuration for clusters.

### [`singularity-path.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/singularity-path.config)

Define path to Singularity Containers for all process.
Expand All @@ -39,14 +67,17 @@ You need to set them up before.

### [`singularity.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/singularity.config)

Define Singularity Containers for all process.
Images will be pulled automatically.
Use in your own profile if needed.
Specify Singularity options.
To be used with [`containers.config`](#containersconfig)

### [`travis.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/travis.config)

To be used for Travis (2 cpus) or on small computer for testing purpose

### [`uppmax-localhost.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/uppmax-localhost.config)

Local configuration for a UPPMAX cluster
To be run on a single node
### [`uppmax-slurm.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/uppmax-slurm.config)

Slurm configuration for a UPPMAX cluster
Expand All @@ -59,17 +90,47 @@ The default profile is `standard`, but Sarek has multiple predefined profiles wh
```bash
nextflow run SciLifeLab/Sarek --sample mysample.tsv -profile myprofile
```
awsbatch {
binac {
btb {
cfc {
docker {
singularity {
singularityPath {
slurm {
slurmDownload {
standard {

### `awsbatch`

This is the profile for use with AWS Batch.

### `binac`

This is the profile for use on the german BinAC cluster.

### `btb`

This is the profile for use on the BTB server munin.

### `cfc`

This is the profile for use on the CFC cluster in Tuebingen.

### `docker`

This is the profile for docker testing on a small machine, or on Travis CI.
Docker images will be pulled automatically.

### `standard`
### `singularity`

This is the default profile for use on a localhost on a UPPMAX cluster with Singularity.
Singularity images need to be set up.
This is the profile for Singularity testing on a small machine, or on Travis CI.
Singularity images will be pulled automatically.

### `singularityPath`

This is the profile for Singularity testing on a small machine.
Singularity images needs to be set up.

### `slurm`

Expand All @@ -83,10 +144,10 @@ This is another profile for use on a UPPMAX cluster using the job scheduler slur
Will run the workflow on `/scratch`.
Singularity images will be pulled automatically.

### `singularity`
### `standard`

This is the profile for Singularity testing on a small machine, or on Travis CI.
Singularity images will be pulled automatically.
This is the default profile for use on a localhost on a UPPMAX cluster with Singularity.
Singularity images need to be set up.

## Customisation
The recommended way to use custom settings is to supply Sarek with an additional configuration file. You can use the files in the [`conf/`](https://github.com/SciLifeLab/Sarek/tree/master/conf) directory as an inspiration to make this new `.config` file and specify it using the `-c` flag:
Expand Down
2 changes: 1 addition & 1 deletion docs/REFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Use `--genome smallGRCh37` to map against a small reference genome based on GRCh

## AWS iGenomes
Sarek is using [AWS iGenomes](https://ewels.github.io/AWS-iGenomes/), which facilitate storing and sharing references.
Both `GRCh37` and `GRCh38` are available with `--genome GRCh37` or `--genome GRCh38` respectively with any profile using the `conf/igenomes.config` file (eg.: `awsbatch`), or you can specify it with `-c conf/igenomes.config`, it contains all data previously detailed.
Both `GRCh37` and `GRCh38` are available with `--genome GRCh37` or `--genome GRCh38` respectively with any profile using the `conf/igenomes.config` file (eg.: `awsbatch`, or `btb`), or you can specify it with `-c conf/igenomes.config`, it contains all data previously detailed.

## buildReferences.nf

Expand Down
Loading

0 comments on commit 93a0bfb

Please sign in to comment.