How to get the PATH from working directory

[mudymudy]

Hello. I'm very new to nextflow and I'm trying to translate a very very long bash pipeline into nextflow.
At the moment I only have 4 processes, where the first three are currently working. the last process is not working because I don't know how to get the PATH for a set of files that are in a channel.

#!/usr/bin/env nextflow
params.data = ""
reads = Channel.of(params.data)
params.output = ""
resultsDir = Channel.of(params.output)
params.gcompleteness = 50
geneCompleteness = Channel.of(params.gcompleteness)
params.coverage = 0.5
normalizedCoverage = Channel.of(params.coverage)
params.threads = 10
threadsGlobal = Channel.of(params.threads)
params.tax_id = 0
taxID = Channel.of(params.tax_id)
params.genomes = 100
downloadGenomes = Channel.of(params.genomes)
params.clustering = 0.95
cdHitCluster = Channel.of(params.clustering)
params.core = 0.01
pangenomeThreshold = Channel.of(params.core)
params.clean = "strict"
pangenomeMode = Channel.of(params.clean)
params.config = ""
configFile = Channel.of(params.config)

params.help = false

/*
 * As the name suggest, it will just generate the whole pipeline directory structure.
 */

process dirStructure {

	input:
	path makeDir

	script:
	"""
	#!/bin/bash
	mkdir -p "${makeDir}/results/NCBI/GFF"
	mkdir -p "${makeDir}/results/NCBI/FASTA"
	mkdir -p "${makeDir}/results/CLUSTERING"
	mkdir -p "${makeDir}/results/PROKKA/GFF"
	mkdir -p "${makeDir}/results/PANGENOME"
	mkdir -p "${makeDir}/results/ALIGNMENTS"
	mkdir -p "${makeDir}/results/NORMALIZATION"
	mkdir -p "${makeDir}/results/MATRIX"
	mkdir -p "${makeDir}/results/PLOTS"
	mkdir -p "${makeDir}/results/HETEROPLASMY/intermediate_files"
	mkdir -p "${makeDir}/results/HETEROPLASMY/distributions"
	"""
}

/*
 * entrez() will download FASTA and GenBank files from NCBI as long as a taxonomical ID was provided (mandatory value)
 */

process entrez {
	conda "${projectDir}/envs/entrez.yaml"
        publishDir "${makeDir}/results/NCBI", mode: 'copy', overwrite: true

	input:
	val gE
	val txid
	path makeDir

	output:

	path '*.gz'


	script:
	"""
	#!/bin/bash
	counter=0
	esearch -db assembly -query "txid${txid}[Organism] AND (latest[filter] AND (complete genome[filter] OR chromosome level[filter]))" | esummary | xtract -pattern DocumentSummary -element FtpPath_RefSeq | while read url; do
        
	if [ "\$counter" -ge "${gE}" ]; then
		break
	fi

	if [ -z "\$url" ]; then
		continue
	fi
        
	fname="\$(basename "\$url")"
	wget "\$url/\${fname}_genomic.gbff.gz"
        
	wget "\$url/\${fname}_genomic.fna.gz"

	counter="\$((counter + 1))"	
	done
	"""
}



process unzipFiles {
	input:
	path gz_files

	output:
	path "*.fna" , emit: fastaFiles
	path "*.gbff" , emit: gffFiles

	script:
	"""
	gunzip -f ${gz_files}
	
	"""
}

process fastaDatabase {
	conda "${projectDir}/envs/biopython.yaml"
	
	input:
	path gffPath

	output:
	path "clustered_sequences.fasta" , emit: theFastaDatabase

	script:

	"""
	#!/bin/bash
	python "${projectDir}"/scripts/parsing_and_contatenating.py $gffPath

	"""
}

workflow {
	dirStructure(resultsDir)
	dwnld = entrez(downloadGenomes, taxID, resultsDir)
	unzipFiles(dwnld)
	gffPath = unzipFiles.out.gffFiles.getParent()
	gffPath.view()
	concatenatedSeqs = fastaDatabase(gffPath)
	concatenatedSeqs.view()
}

My problem is that in the process fastaDatabase, the python script is not able to find the genbank files because it accepts a directory and not a string of file names. I don't know how can I get the PATH for those files as gffPath = unzipFiles.out.gffFiles.getParent()
is not working. Any guidance will be very much appreciated!!!! Thanks. I would like to avoid copying the files to a specific directory and rather use the working directory created by nextflow as the PATH.

[bentsherman]

Hi @mudymudy , I think the best way would be to pass the gffFiles directly into the fastaDatabase and use the stageAs option to stage them as a directory:

process fastaDatabase {
  input:
  path gffFiles, stageAs: 'gff/*'

  // ...

  script:
  """
  parsing_and_contatenating.py gff
  """
}

workflow {
  // ...
  fastaDatabase(unzipFiles.out.gffFiles)
}

If you put your scripts in the bin directory instead of scripts, make the script executable, and add a shebang to the top of the python script, you can call the python script directly as shown above (docs).

By the way, if you ever have questions like this you can also go to https://community.seqera.io/, there are a lot more people who can help you there.

[mudymudy]

Thank you so much Ben. It worked perfectly!!! It's just what I wanted.