Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiprocessing resume_after can apportion post-resume pipeline tables #374

Closed
toliwaga opened this issue Jan 25, 2021 · 0 comments
Closed

multiprocessing resume_after can apportion post-resume pipeline tables #374

toliwaga opened this issue Jan 25, 2021 · 0 comments

Comments

@toliwaga
Copy link
Contributor

When resuming a multiprocess run, mp_tasks.apportion will wrongly apportion tables in pipeline created in a later phase of single-process execution. This only happens if in the prior run, execution proceeded through a multiprocess step and coalesced the results back into the pipeline - and then the resume_after step specified repeats the apportion step. I will be comitting a fix for this forthwith.
toliwaga added a commit that referenced this issue Jan 25, 2021
@toliwaga
multiprocess_steps.slice.exclude: wildcard 'True' in place of list ex…
73c6ad0 
…cludes everything not elseqhere specified

multiprocess_steps.slice.coalesce list of tables (presumably created by subprocess) to coalesce
fix resume_after issue #374 where apportion wrongly apportions tables to subprocess pipeline that
were created during later steps in the original run.
@toliwaga toliwaga mentioned this issue Jan 25, 2021
Merged
bstabler pushed a commit that referenced this issue Feb 3, 2021
@toliwaga
Develop popsim mp support (#375)
6f74101 
* abm agnostic multiprocessing support for popsim

* multiprocess_steps.slice.exclude: wildcard 'True' in place of list excludes everything not elsewhere specified

multiprocess_steps.slice.coalesce list of tables (presumably created by subprocess) to coalesce

fix resume_after issue #374 where apportion wrongly apportions tables to subprocess pipeline that
were created during later steps in the original run.

* warn if PERSON_TYPE_MAP setting not found in cdap.yaml

* cached accessibility support

* fix abm injectable registration issue in cli.run

* add expressions back to eval_utilities tracing
@bstabler bstabler mentioned this issue Jun 9, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@toliwaga @bstabler