Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upsert support #263

Merged
merged 5 commits into from
Apr 1, 2020
Merged

Upsert support #263

merged 5 commits into from
Apr 1, 2020

Conversation

frankmcsherry
Copy link
Member

This PR introduces a prototype "upsert" arrangement operator. Instead of ingesting standard update triples, it receives keyed optional values, where the current value for a key is the most recently received value.

This is substantially less general than differential dataflow allows, but lots of people use this format. The reason to do this here is that if the plan is to arrange the results anyhow, then we can re-use the arrangement and extract the "prior" values associated with keys, and undo the damage done at nominal incremental overhead.

cc @wangandi

@frankmcsherry
Copy link
Member Author

The code is mostly a copy/paste of the Arrange operator, and much of its generality isn't required here, where a total order on timestamps is required. That should probably be cleaned up, but it seemed most likely to work correctly starting from something else that was known (believed) to be correct.

@frankmcsherry frankmcsherry changed the title WIP Upsert support Upsert support Mar 26, 2020
@frankmcsherry frankmcsherry merged commit 60b5e16 into master Apr 1, 2020
@rjnn
Copy link

rjnn commented Apr 1, 2020

🎉!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants