Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Joint Tour Chunking - Household Participants Need to Remain in Same Chunk #288

Closed
danielsclint opened this issue Dec 28, 2019 · 4 comments
Closed
Assignees

Comments

@danielsclint
Copy link

In the joint tour participation model, its possible for the chunker to split potential joint tour participants from the same household across two different chunks. This will create an endless loop and eventual assertion error in the joint tour participation model.

def chunked_choosers(choosers, rows_per_chunk):
assert choosers.shape[0] > 0
# generator to iterate over choosers in chunk_size chunks
num_choosers = len(choosers.index)
num_chunks = (num_choosers // rows_per_chunk) + (num_choosers % rows_per_chunk > 0)
i = offset = 0
while offset < num_choosers:
yield i+1, num_chunks, choosers.iloc[offset: offset+rows_per_chunk]
offset += rows_per_chunk
i += 1

If a household is split, so only one person is chosen in the chunk, the following code will never evaluate to true, because x.participants > 1 will always evaluate to False. Or... x.participants > x.adults will always evaluate to False.

satisfaction = (x.composition != 'mixed') & (x.participants > 1) | \
(x.composition == 'mixed') & (x.adults > 0) & (x.participants > x.adults)

Eventually, the mode will crash in the following set of code after the maximum number of iterations for finding participants occurs.

if iter > MAX_ITERATIONS:
logger.warning('%s max iterations exceeded (%s).', trace_label, MAX_ITERATIONS)
diagnostic_cols = ['tour_id', 'household_id', 'composition', 'adult']
unsatisfied_candidates = candidates[diagnostic_cols].join(probs)
tracing.write_csv(unsatisfied_candidates,
file_name='%s.UNSATISFIED' % trace_label, transpose=False)
print(unsatisfied_candidates.head(20))
assert False

@toliwaga
Copy link
Contributor

toliwaga commented Oct 4, 2020

fixed this in develop by adding chunking by chunk_id, and updated tests to catch such errors in the future by forcing singleton full model run test to use minimal chunk size.

This is provisional code as the multiple_zone branch has more extensive changes to chunking (adaptive chunking).

@toliwaga toliwaga closed this as completed Oct 4, 2020
@bstabler
Copy link
Contributor

bstabler commented Oct 8, 2020

@danielsclint to now test

@danielsclint
Copy link
Author

I see the new method in chunk to chunk by chunk id, but I don't see any changes to the joint tour participation code in develop. As this be merged with the main repo or in some other working version of develop?

toliwaga added a commit that referenced this issue Oct 23, 2020
@toliwaga
Copy link
Contributor

Oops! Sorry! It helps to push - which I just did.

danielsclint pushed a commit to danielsclint/activitysim that referenced this issue Dec 15, 2020
bstabler added a commit that referenced this issue Jan 12, 2021
* Provide more flexibility for defining mandatory schedule specifications.

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax.

* code syntax fixes.

* syntax fix.

* Add necessary change to test script settings.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* create develop branch so we can merge features here before merging (releasing) to master

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax fixes.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* Fix overflows

* Complete testing of the flexible time periods.

* pep8

* test some floats too.

* pep8

* start with csvs and write hdf5 outputs if desired  (#290)

* add h5_store option to write_tables

* keep csv files as the default

* read input tables from CSV

* update docs, drop scripts no longer needed, include example files in package

* formatting

Co-authored-by: Blake <brosenthalpdx@gmail.com>

* Update travis.yml

* Pandas 1.0 fixes

* Location and mode choice logsums (#298)

* optional logsums for location_choice and tour destination models

* mode choice logsums for trous and trips

* trip DEST_CHOICE_SAMPLE_TABLE_NAME

* update docs for writing logsums and also a couple small development page updates

* fix small py2 py 3 compatibility issue

* py2 py3 compatibility

Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>

* Deprecate Py2 and add CLI

* Tncs (#14)

* add tncs

* lognormal_for_df broadcasts duplicate indexes

* lognormal_for_df broadcasts duplicate indexes code

* correct lognormal calculations and add functions to calculation lognormal location and scale

* clean-up tnc wait time expressions

* set taxi, tnc wait time standard deviations to zero until we have better data

* add scaled option to random.lognormal_for_df

* add mobility-as-a-service modes to docs

* updated tnc mode constants

* rename maas to ridehail

Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>

* write trip matrices (#311)

require newer pandas

add ability to save trips table to pipeline and begin to finish example

all time periods

fix trip scheduling bug

add support for odt skims as well

Co-authored-by: Ben Stabler <benstabler@yahoo.com>

Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>

* small fixes for semcog deployment (#319)

* improved validation diagnostics in trip_purpose and various windows-related int32/int64 conversions

* pycodestyle

* deprecations in test code

* fix bug in handling of no viable trips case in choose_trip_destination

* add trace folder to example_mtc output folder

Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>

* major work on phase 5 (#325)

* estimation through atwork_subtour_mode_choice

* Tnc updates and notebooks (#18)

* move other resources into folder since examples now part of package as well

* add shapefile

* clean up folder setup

* estimation notebooks for larch (#19)

* multiprocessing related logging and error checking

* trip_destination handle all trips fail

* skim caching with numpy memmap to speed skim loading

* better chunking in vectorize_tour_scheduling

* Cli (#22)

* use activitysim_resources

Fixed auto sufficiency conditions in tour_mode_choice.csv. (Issue #324)

* increment to version 0.9.5

* correct write trip matrices sampling expansion and add vehicle occupancy to the expression file

* updates to documentation for phase 5 work except multiple zone systems

* additional updates for estimation integration  (#328)

* estimation integration updates #327
* drop module no longer needed

* update license, fix univ coeff bug, update estimation example (#339)

* improve example setup for specifying input tables (#27)
* improve LICENSE (#30)
* correct univ coeff template lookup (#28)

* add license to package (#341)

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* correct doc links

* Address Minor Issues Associated with ARC Update to Develop Branch (#342)

* publish v0.9.5 (#340)

* update docs, drop scripts no longer needed, include example files in package

* Pandas 1.0 fixes

* Location and mode choice logsums (#298)

* Tncs (#14)

* write trip matrices (#311)

* small fixes for semcog deployment (#319)

* improved validation diagnostics in trip_purpose and various windows-related int32/int64 conversions

* fix bug in handling of no viable trips case in choose_trip_destination

* major work on phase 5 (#325)

* estimation through atwork_subtour_mode_choice

* Tnc updates and notebooks (#18)

* move other resources into folder since examples now part of package as well

* add example zone shapefile

* estimation notebooks for larch (#19)

* multiprocessing related logging and error checking

* trip_destination handle all trips fail

* skim caching with numpy memmap to speed skim loading

* better chunking in vectorize_tour_scheduling

* Cli (#22)

* use activitysim_resources

Fixed auto sufficiency conditions in tour_mode_choice.csv. (Issue #324)

* correct write trip matrices sampling expansion and add vehicle occupancy to the expression file

* additional updates for estimation integration  (#328)

* improve LICENSE (#30)

* correct univ coeff template lookup (#28)

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* publish (#344)

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* correct doc links

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

Co-authored-by: Ben Stabler <bstabler@users.noreply.github.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Co-authored-by: bstabler <benstabler@yahoo.com>

* fix #288 chunk joint_touyr_participation by chunk_id and update tests to catch such errors

* gitignore trivia

* Support for Multiple Zone Systems and Transit Virtual Path Building (#362)

* TransitVirtualPathBuilder first cut - get_tvpb_logsum computes logsums on demand

* tour_mode_choice logsums for 3-zone working for school and workplace location

* z3 running all models, but with hacked trip_mode_choice (no coefficient templates)

* z3 running all models

* passing tests

* pcodestyle passes

* refactored test teardown

* tvpb tracing

* tvpb tracing roughout

* build multizone test data before running test

* read_settings_file support include_settings

* read_settings_file support include_settings

* move constants to configs yaml and expressions from abm.util to core

* tvpb estimate_chunk_overhead

* adaptive chunking

* fixed bug in trip scheduling when no trips in leg after iteration

* 3 zone tap_tap uniquify

* tvpb TableCache

* tvpb TableCache - all fiiels

* 3 zone with feather cache

* tidy up notes around creating the marin example

* memmap skims

* memmap skims

* correct drive transit cost expression for miles/feet.  Still need to add some additional missing expressions now that everything appears to be working correctly.

* skim docstrings

* disable 3 zone tap_tap caching for traced taps

* initialize_los

* 3 zone mp plumbing untested

* THREE_ZONE tvpb multiprocessing tests and fixes

* THREE_ZONE tvpb multiprocessing tests and fixes

* reanme core tvpb and cache modules

* reanme core tvpb and cache modules

* util.iprod alias for np.prod with dtype int64

* THREE_ZONE cache initialization bug bixes

* multizone bug fixes

* bugs

* multiprocessing mjultizone tweaks

* tvpb cacha\e as array

* complain if apporting with more process than slice rows

* complain if apporting with more process than slice rows

* chunk initialize_tvpb

* shorter trace file names because windows

* refactor pathbuilder sans DYNAMIC

* fix tour_scheduling_calc_row_size

* fix multiprocess resume across mp step bug

* pycodestyle

* minor chunk cleanup

* allow MemMapSkimFactory multiprocessing

* consolidate mtc and multizone configs

* commit updated marin example config files

* some more marin example config updates

* add validation summaries script

* some additional marin example updates

* update example skim time period settings to align with tm1 (#360)

* update example settings to align with tm1

* correct home_is_rural coding since tm1 was incorrect

* corrected skim time periods go in network_los.yaml now

* travis depends on these scripts for testing

* improve logging and increase travis timeout

* #359 (#364)

* updates for CDAP person type mapping

* allow duplicate time labels

* fix a copy small items for #359

* test updates

* pycodestyle

* 3 zone test fix

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>

* longer travis timeout and avoid file permissions between tests related to caching

* test-travis

* ls fix

* add timeout

* fix path

* put in both locations for now

* reduce household sample sizes for tests to avoid travis timeout

* not sure what's taking so long with the second and third tests.  It runs in 40 minutes with py 3.7 on my computer as well

* linux file paths

* cli test depends on py 3.7+

* add coveralls back

* test new architecture

* try build matrix

* setup environments

* correct matrix expansion

* correct duplicate key

* correct key

* correct env usage

* no need to specify architecture

* update inject settings for tests

* updated examples to use activitysim resources repo

* rename folder

* update survey test files so all example_manifest.yaml examples now running

* updated notebooks, default settings for examples, and updated rst files

* fix a couple typos

* add comments to travis script

* ARC Updates (#365)

Add parking_location_choice, trip_departure_choice, and trip_scheduling_choice models.

* add example_mtc_arc_extensions so arc extensions can be run and tested

* small doc update

* fix arc extensions example

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants