Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve bson iteration performance #2707

Merged
merged 9 commits into from
Feb 29, 2024
Merged

Conversation

tychoish
Copy link
Collaborator

Minor: this avoids casting bson into values until we need to. This
means that if a document in a stream has a field which is not in the
schema, we just skip it.

@universalmind303 universalmind303 changed the title fix: improve bson iteration performance perf: improve bson iteration performance Feb 27, 2024
@vrongmeal
Copy link
Contributor

How does this affect the performance. I can't figure out where do we "skip" if the field doesn't exist?

@tychoish
Copy link
Collaborator Author

How does this affect the performance. I can't figure out where do we "skip" if the field doesn't exist?

It depends on the data, but it makes the process of iterating through a document lazier. If you have a schema that was (a int, b text) but a document that was {"a":1,"c":<...>,"b":"foo"} you could skip reading the c field entirely. Because all fields are length prefixed you just skip ahead... in the previous iterator it would need to parse the value of the field even if we weren't going to use it.

@tychoish tychoish merged commit 2f3b1b9 into main Feb 29, 2024
25 checks passed
@tychoish tychoish deleted the tycho/lazy-bson-iteration branch February 29, 2024 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants