Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add concurrent library to ParallelComputingInterface #667

Merged

Conversation

Bartdoekemeijer
Copy link
Collaborator

Feature: add concurrent to ParallelComputingInterface class

This PR adds the ability to use the concurrent library for parallel computing. concurrent is an alternative to the multiprocessing library that is deemed to be more reliable at times, e..g, see the comments in https://stackoverflow.com/questions/20776189/concurrent-futures-vs-multiprocessing-in-python-3.

Related issue

No official issues reported. I have personally experienced some time-out issues with multiprocessing and its interaction with streamlit. Workers may time out due to local errors but that never escalates back up to the main process, and therefore a job may hang forever. Supposedly concurrent is better at handling this.

Impacted areas of the software

Parallel Computing Interface.

Additional supporting information

N/A

Test results, if applicable

The yaw optimization in 12_optimize_yaw_in_parallel.py with interface = "multiprocessing" reports:

===============================================================================
Total time spent for parallel calculation (16 workers): 15.342 s
  Time spent in parallel preprocessing: 0.002 s
  Time spent in parallel loop execution: 15.331 s.
  Time spent in parallel postprocessing: 0.008 s

with interface = "concurrent" I find generally a slight improvement in computation time:

===============================================================================
Total time spent for parallel calculation (16 workers): 14.643 s
  Time spent in parallel preprocessing: 0.002 s
  Time spent in parallel loop execution: 14.632 s.
  Time spent in parallel postprocessing: 0.009 s

@rafmudaf rafmudaf force-pushed the feature/add_concurrent_to_pci branch from 80c3074 to 7f1ca3e Compare July 3, 2023 17:26
@rafmudaf rafmudaf self-requested a review July 3, 2023 18:00
@rafmudaf rafmudaf merged commit 7674287 into NREL:develop Jul 3, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants