Skip to content

Commit

Permalink
Ignore accelerator type for AWS Batch (#4043)
Browse files Browse the repository at this point in the history


Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
  • Loading branch information
bentsherman and pditommaso committed Jun 26, 2023
1 parent 8a89351 commit 263ecca
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 13 deletions.
3 changes: 2 additions & 1 deletion docs/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -1201,9 +1201,10 @@ This directive is only used by certain executors. Refer to the {ref}`executor-pa
:::{note}
The accelerator `type` option depends on the target execution platform. Refer to the platform-specific documentation for details on the available accelerators:

- [AWS](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_)
- [Google Cloud](https://cloud.google.com/compute/docs/gpus/)
- [Kubernetes](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus)

The accelerator `type` option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your Compute Environment. See the [AWS Batch FAQs](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_) for more information.
:::

(process-afterscript)=
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ import nextflow.container.ContainerNameValidator
import nextflow.exception.ProcessSubmitException
import nextflow.exception.ProcessUnrecoverableException
import nextflow.executor.BashWrapperBuilder
import nextflow.executor.res.AcceleratorResource
import nextflow.fusion.FusionAwareTask
import nextflow.processor.BatchContext
import nextflow.processor.BatchHandler
Expand Down Expand Up @@ -715,8 +714,12 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler<String,Job
if( task.config.getCpus() > 1 )
resources << new ResourceRequirement().withType(ResourceType.VCPU).withValue(task.config.getCpus().toString())

if( task.config.getAccelerator() )
resources << createGpuResource(task.config.getAccelerator())
final accelerator = task.config.getAccelerator()
if( accelerator ) {
if( accelerator.type )
log.warn1 "Ignoring task ${task.lazyName()} accelerator type: ${accelerator.type} -- AWS Batch doesn't support accelerator type in job definition"
resources << new ResourceRequirement().withType(ResourceType.GPU).withValue(accelerator.request.toString())
}

if( resources )
container.withResourceRequirements(resources)
Expand All @@ -731,15 +734,6 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler<String,Job
return result
}

protected ResourceRequirement createGpuResource(AcceleratorResource acc) {
final res = new ResourceRequirement()
final type = acc.type ?: 'GPU'
final count = acc.request?.toString() ?: '1'
res.setType(type.toUpperCase())
res.setValue(count)
return res
}

/**
* @return The list of environment variables to be defined in the Batch job execution context
*/
Expand Down

0 comments on commit 263ecca

Please sign in to comment.